Youtu-Parsing-Omni and the perception contract: adopt the interface, not the assumption

On 9 October 2026 Tencent’s Youtu Lab published a five-billion-parameter model on Hugging Face called Youtu-Parsing-Omni. Give it a document page, a photograph, a chart, a flowchart, a geometry diagram, an audio clip or a video, and it returns the same kind of object every time: one JSON record with a modality label, a list of elements anchored by bounding boxes or segments anchored by timestamps, and a global description. The weights, a vLLM serving plugin, inference examples and a 53-page technical report went up together. Tencent reports 96.96 on OmniDocBench v1.6, the highest document-parsing score among the systems it compared, and 75.08 across OmniParsingBench, a broader test of images, audio and video, where only Gemini 3 Pro scored higher, at 77.44. The licence opens with a sentence most readers will skip: the model is “not intended for use within the European Union”.

I have read the report and the model card closely, because this release touches the least glamorous and most expensive part of an enterprise AI platform. Most multimodal systems in large organisations do not have a model problem. They have a translation problem. A document parser returns Markdown. An OCR engine returns text and boxes. Speech recognition returns a timestamped transcript. A video model returns captions. A chart parser returns a table. Before any agent can reason over a quarterly review, a platform team has to reconcile five representations that disagree about coordinates, time, identity and what counts as an error. Youtu-Parsing-Omni’s real proposition is not that one model sees everything. It is that the interface between perception and reasoning can be a single, learned contract. That idea is right. The way to adopt it is to take the interface and leave behind the assumption that one schema means one level of trust. This essay sets out what Tencent built, what the benchmarks do and do not show, and the controls I would put around it.

The translation problem nobody budgets for

Consider what a typical enterprise multimodal pipeline looks like after two years of honest work. Contracts and invoices go through a document parser that emits Markdown with page numbers. Scanned annexures go through an OCR service that returns words with pixel coordinates. Call recordings go through a speech recogniser that returns words with millisecond offsets and a speaker label. Training videos go through a captioning model that returns free text per scene. Charts pulled from board packs go through a separate model that returns a table. Each choice is reasonable on its own. Together they produce five coordinate systems, three notions of time, two or three notions of who a speaker is, and no shared idea of what a record is.

The cost shows up in three places. First, adapters: every producer needs a mapping into every consumer, the search index, the agent’s tool layer and the audit store, and each mapping is code that someone owns and someone breaks. Second, storage: the canonical record ends up being whatever the most demanding consumer needed, with other fields bolted on. Third, evidence: when an agent answers a question by combining a figure from a slide, a sentence from a call and a cell from a table, the citation has to survive three translations. When I ran data science at Salesken, speech transcription for sales calls was one stage in a longer chain, and much of the engineering lay not in the transcript itself but in making every later stage agree on what a word, a turn and a timestamp meant.

This is why the shape of Tencent’s output matters more than the size of its model. I made the same argument about retrieval in Grounding is the product: the useful unit is not an answer but an answer tied to evidence. A perception layer that emits evidence in one grounded form is the precondition for that.

What Tencent actually shipped

The architecture has three parts, and only the edges are modality-specific. Thin stems turn raw input into tokens: images and video frames are cut into patches at native resolution, with one linear map per patch, and audio becomes a log-mel spectrogram that a few strided convolutions reduce to 13 tokens per second. Those tokens enter one shared, bidirectional Transformer, the Youtu-Omni-Encoder, which has about 0.4 billion parameters and was initialised from Qwen3-0.6B-Base with its embedding layer and output head removed. Modality-specific mergers and projectors then hand the result to a 4.5-billion-parameter decoder, 40 layers with multi-head latent attention and a 128,000-token context, which writes the JSON.

Two details make the encoder unusual. First, every token carries a position on three axes: time, height and width. A vision token takes its time from the frame index and its height and width from the patch grid. An audio token takes its time from its offset in the recording and sets height and width to zero. Both modalities therefore write into the same temporal coordinates. Second, four of the encoder’s layers are fusion layers. In ordinary layers each frame and each audio chunk attends only to itself. In fusion layers a new attention window opens at every boundary where audio gives way to a frame, so a frame attends to the sound of the same moment. The correspondence a video parse needs, which words were spoken while this slide was on screen, is computed during perception rather than reconstructed afterwards by the language model. Alibaba’s Logics-Parsing-Omni, published in March 2026 and the model that introduced OmniParsingBench, takes the conventional route: separate vision and audio towers inherited from Qwen3-Omni-30B-A3B, meeting only inside the decoder.

The output contract, which Tencent calls OmniSchema, covers ten tasks. Every record has a modality, a list of units and a global description. For images the unit is an element with a category, such as text, table, formula, chart or seal, a bounding box written as special tokens on a grid from 0 to 1,000, and content in the notation that fits it: LaTeX for formulas, OTSL for tables, Markdown for chart data, Mermaid for flowcharts, SMILES for molecules and ABC notation for sheet music. For audio and video the unit is a segment with start and end timestamps in hours, minutes and seconds, a speaker identifier that stays consistent across the input, a transcript, acoustic-event labels and, for video, on-screen text, actions and camera motion. Documents add a reading order. Text-rich video adds a Markdown report of the whole recording.

Training is heavier than the parameter count suggests: 2.24 trillion tokens across four pre-training stages, then 16 billion tokens of supervised fine-tuning, then two on-policy stages. The last, which Tencent calls OmniSchema-Aware On-Policy Distillation, is the cleverest piece of the report. Because every output follows one schema, each token can be classified as structure, the keys and punctuation of the JSON, or content, the values. The model samples its own parses; a frozen copy of itself, which can also see the reference parse, scores them token by token; structure and content are supervised with different objectives; and after each round the student becomes the next teacher. Six rounds lifted the OmniDocBench score from 95.21 to 96.96. Of the 1.99 points gained after fine-tuning, this stage supplied 1.75.

One JSON record for every input. Change the scorer and Gemini 3 Pro’s video score moves 30 points

Where the shared timeline shows up in the numbers

The design makes a testable prediction: if aligning sound and image inside perception is the advance, the gains should be largest on tasks that require locating events in time, and smallest on static images. The report’s own tables bear that out, and to Tencent’s credit it says so plainly rather than leading with the average.

Against Logics-Parsing-Omni, Youtu-Parsing-Omni leads by 2.5 points on audio, 6.8 on natural video and 10.3 on text-rich video, and trails by 2.8 points on the static splits, mostly natural images and geometry. On text-rich video, lecture recordings and screen captures where slides change while someone talks over them, its temporal localisation score is 50.70 against 23.05 for Logics. On natural video its timing score, 63.68, is above Gemini 3 Pro’s 59.47. Against Gemini overall, it is level on the temporal splits, 75.96 against 76.12 on average, and well behind on the static ones, 73.76 against 79.42.

The ablations explain the static weakness. In a controlled study with a smaller decoder, initialising the shared encoder from a text language model rather than from scratch gave a reading advantage that persisted throughout training: at 300 billion tokens it scored 767.0 on OCRBench against 636.4. But the encoder trained from scratch led on BLINK, a test of spatial and relational perception, by 9.5 points. Weights that already map context to symbols help a model read. They do not help it reason about space. That is consistent with what the benchmarks show downstream: the final model’s geometry relation score is 46.77 against 64.24 for Logics, and its recognition of knowledge entities in photographs, things that need world knowledge to name, is 13.90 against 21.39. An architecture is a set of bets, and you can see these bets in the error profile.

Read the benchmarks with the scorer in view

Two of the headline numbers need more care than they have received.

OmniDocBench is saturated at the top. The score of 96.96 comes from the official evaluator, and Tencent has released its evaluation pipeline and a container image of that scorer so that anyone can rerun it. The model has not yet appeared on the maintainers’ leaderboard, which lists TeleOCR, a 1.2-billion-parameter document model, at 96.91. TeleOCR’s own model card reports 96.87. OvisOCR2 appears at 96.47 on the leaderboard and 96.58 in TeleOCR’s comparison. The margin of first place over second, 0.05 points, is smaller than the disagreement between published sources about the second-place score. At this level the benchmark no longer separates the leaders. What it does show, and this is the more useful finding, is that a model which also handles audio and video gave up nothing on documents.

The document number that should shape a deployment is a different one. On PureDocBench, which tests the same pages clean, digitally degraded and physically recaptured with a camera, Youtu-Parsing-Omni scores 82.61 clean and 68.13 recaptured, a drop of 14.5 points. It is the strongest end-to-end model on that benchmark, but TeleOCR, a multi-stage pipeline, has the better three-track average, 78.41 against 76.29, and Gemini 3.1 Pro has the best recaptured score of all, 71.98. If your documents arrive as phone photographs of signed pages, as they do in claims, lending and field operations, plan for the recaptured number, not the clean one.

OmniParsingBench scores depend on who scores them. This is the finding I would put in front of anyone writing a procurement memo. OmniParsingBench was released without a complete evaluation toolkit, so Tencent wrote its own from the metric definitions and re-scored every system with it. Its toolkit repairs schema violations before scoring, corrects some annotation errors, uses Qwen3-235B-A22B as the judge for the question-answering component, sets aside the camera-video subset because of annotation problems and drops 136 samples whose media was unavailable, leaving 3,738 of 3,874. The report states clearly that these numbers are “comparable with each other, but not directly with those reported by the benchmark authors”.

It is worth seeing how large that caveat is. In the Logics team’s own paper, Gemini 3 Pro scores 52.68 on audio, 44.56 on natural video and 61.00 on text-rich video. Under Tencent’s scorer the same model scores 76.74, 74.82 and 76.80. Logics-Parsing-Omni moves from 53.75 to 75.58 on audio. Much of the gap sits in the Cognition component, multiple-choice questions a judge answers from the parse alone: Gemini’s audio Cognition score is 31.37 in one table and 87.90 in the other. The ranking changes too. Under the authors’ scorer, Logics beats Gemini on audio and on text-rich video. Under Tencent’s, Gemini leads Logics on all three temporal splits. Neither table is wrong. They are different instruments with the same name.

This is not a criticism of Tencent, which disclosed every one of these choices. It is a description of where multimodal evaluation stands in October 2026. In The evaluation bench, in full I argued that model judges agree with themselves more than with the truth. Here the judge, the repair step and the sample set together move scores by as much as 30 points. The only benchmark number a platform team should trust for a decision is one produced by its own scorer, on its own data, with every candidate run through the same harness.

The sub-scores matter more than the averages. The averages hide a model whose fields are not equally reliable. On audio it has the highest overall score of the systems compared, 78.08, because it is better at locating segments and labelling acoustic events, yet its speech-recognition sub-score, 77.94, is the lowest of the four. On natural video its transcription score is 67.80 against Gemini’s 84.06. It is the best chart parser in the comparison, at 95.02, and matches a dedicated optical-music model on sheet-music transcription, with a character error rate of 22.73 against LEGATO’s 23.30. On chemical structures it produces an exact fingerprint match for 54.2 percent of molecules, the highest of any system compared, which also means no exact match for nearly half. On Video-MME-v2, a general video-reasoning benchmark, its group score of 11.5 is in line with much larger open models. It perceives well. It is not a reasoner, and Tencent does not claim that it is.

One schema, several error models

This is where the unified envelope needs care. A bounding box, a transcript and a chart value sit side by side in the same JSON, but they fail in different ways, and nothing in the record says so.

A bounding box is quantised to a 1,000-step grid. On an A4 page scanned at 300 dots per inch, roughly 2,480 by 3,508 pixels, one step is about two and a half pixels across and three and a half down, which is fine for a paragraph and coarse for a signature line. A timestamp is written to the second, so a speaker turn that begins at 14 minutes 32.8 seconds is recorded as 14:32 or 14:33: adequate for search, too coarse for aligning a spoken figure with a slide that was on screen for under two seconds. A transcript fails word by word. A chart value has a subtler failure: a number printed as a data label is read, while a number inferred from the height of an unlabelled bar is estimated, and the Markdown table records both the same way. The global description, a caption, a narrative and for text-rich video a structured report, is not perception at all. It is generated text, sitting in the same record as the measured fields.

OmniSchema carries provenance in the sense that matters most: every element has a box and every segment has a time span, so a claim can be traced to the region or interval it came from. What it does not carry is confidence. There is no field for how sure the model is of a box, a word or a value, and no field that separates read from estimated from generated. Managed services have treated this as a requirement for some time. Amazon’s Bedrock Data Automation, generally available since March 2025, returns confidence and bounding-box information for each field it extracts from documents. Microsoft’s Azure Content Understanding, generally available since November 2025, gives each extracted field a confidence score between 0 and 1 and a source region, an option documented for its document analysers. Even there, field-level trust is a document feature first.

The irony is that Tencent computed confidence while building the model. Its audio training data was transcribed by three speech systems with different architectures; the transcript with the least disagreement was kept, the largest pairwise word error rate was recorded as a disagreement score, and a chunk was used for fine-tuning only if that score was at most 0.1 and two language identifiers agreed. Text inside natural-image boxes was transcribed by two OCR systems, and the element was discarded when they disagreed. Agreement between independent systems was how Tencent decided which labels to trust. A production pipeline can use the same method to decide which outputs to trust, but the platform team will have to build it, because the model’s output does not carry it.

There is one more field to read before an Indian deployment. The audio schema labels each speech segment’s language as Chinese, English or unknown, and the audio training data was in-house audiobooks, podcasts and conversations. Hindi, Tamil and a Hinglish sales call will arrive as unknown, or worse, as English with a high error rate. At Haptik we extended a conversational platform to twelve Indian languages, and language identification was one of the components that made it work; it is the first field I would test here. I wrote in Voice is the bottleneck that no recogniser stays under 20 percent error on Indian telephone audio in every language. A unified schema does not change that. It only makes the error easier to store.

The benchmark repairs the JSON. Production will not.

Every OmniParsingBench table reports a parse rate, and almost every entry is 100 percent. That is the rate after Tencent’s toolkit has repaired schema violations, so that formatting mistakes are not charged as content errors. It is a sensible choice for comparing perception. It is the wrong number for an operator, because in production nobody repairs the JSON unless someone writes the repair, and a repair is a place where meaning can change silently. The report does not publish the raw, pre-repair validity rate.

Tencent’s own training pipeline shows what to do instead. Its reinforcement-learning reward is zero for any output that does not decode, and the distillation stage admits a rollout only if it passes JSON validity, schema conformance, box syntax and repetition checks; rejected rollouts are masked, not repaired. The model was trained behind a deterministic gate. It should be served behind one.

A few serving details belong in the runbook. The coordinates are special tokens, so a client that strips special tokens, a common default in serving stacks including vLLM’s, silently deletes every bounding box; Tencent’s reference request turns that option off for this reason. The published results were produced with greedy decoding and one sequence at a time, and the report publishes no throughput figure. It notes that, unlike its document-only predecessor, the model does not yet use parallel decoding, and expects lower throughput on dense content such as tables. The serving path and the Transformers path pin different versions of the Transformers library and must live in separate environments. The Transformers path loads with remote code enabled, which runs Python shipped with the checkpoint, and Tencent itself advises pinning a reviewed commit; I explained why that matters in A $50 backdoor in an open model. Community conversions for Apple’s MLX framework appeared on Hugging Face on 11 October; none of Tencent’s numbers apply to them. Finally, the licence: its terms otherwise follow Apache 2.0, but clause zero excludes use within the European Union and says it prevails in any conflict. A group with European subsidiaries needs to know which workloads run where before this model enters a shared platform.

A worked example: one quarterly review, three inputs

Take a scenario every large company runs four times a year. The inputs are a 40-page board pack as a PDF, a 52-minute recording of the review meeting in which the finance lead presents slides and the discussion moves between English and Hindi, and a photographed, signed annexure. The question an agent will be asked afterwards is simple: what revenue figure was presented for the quarter, and does it match the pack?

Without a contract. The pack goes to a document parser that emits Markdown, the annexure to an OCR service with pixel boxes, and the recording to a speech recogniser with millisecond offsets and a video model for slide captions. Answering the question means joining a Markdown table cell, a transcript span and a caption on time, with code written for each pair. Five producers feeding three consumers, the index, the agent tools and the audit store, means fifteen mappings to maintain.

With a contract. The pack is parsed page by page with the document task; the revenue chart on page 12 becomes one chart element whose content is a Markdown table, with a box. The recording goes through the text-rich video task and returns segments, each with on-screen text, a transcript, a speaker and a time span, plus a report of the meeting. The annexure goes through the document task as well. Everything lands in one record type. Five producers and three consumers now need eight mappings: one into the contract for each producer and one out of it for each consumer.

The gate. Before anything reads these records, deterministic checks run. The JSON validates against the schema version the platform has pinned; every box has x1 below x2 and y1 below y2, within 0 to 1,000; every reading-order entry refers to an element that exists; timestamps are monotonic and end before the media does; speaker identifiers are consistent; numeric columns in chart tables parse as numbers. Anything that fails is rejected and logged, and the rejection rate is a metric on the platform dashboard.

The annotation. The platform then adds the fields the model does not supply. Each value gets a producer, the model name and the exact commit; a method, read, estimated or generated; and a confidence. Confidence here comes from agreement, the method Tencent used for its own labels: the revenue figure is checked against the table in the pack, the chart’s data label and the number the finance lead said at 00:14:32. Three sources that agree give a high-confidence field. Two that disagree send the field to review rather than to the agent.

The fallback. Segments whose language comes back unknown, which in this meeting means every Hindi stretch, are re-transcribed by a specialist Indian-language recogniser and written back into the same segment structure, with that recogniser named as the producer. The photographed annexure, the input type on which the parser loses most, goes to a specialist document parser if its validation or agreement checks fail. The agent does not need to know which model produced which field. It sees one contract, and every claim it makes cites a box or a time span that a person can open and check.

What I would do

The release is worth acting on now, even for teams that never deploy this particular model. In order:

1. Adopt a perception contract you own. OmniSchema is a good starting design: units anchored in space or time, typed content, a reading order, a separate description. Version it, publish it internally and make every producer, open model, managed service or specialist, write into it. The contract should outlive any model behind it.
2. Attach confidence and provenance at field level. Every value carries its producer and commit, its anchor, its method, read, estimated or generated, and a confidence derived from agreement or from calibration on your own data. Keep generated descriptions in a field no downstream system mistakes for evidence.
3. Validate deterministically, and reject rather than repair. Schema, coordinate ranges, timestamp order, reference integrity and numeric parsing are cheap checks. Measure the raw validity rate per modality and put it on the dashboard; it is the number the benchmarks leave out.
4. Keep specialist fallbacks for high-risk inputs. On the published evidence those are speech in languages the schema does not name, photographed or recaptured documents, geometry and knowledge-heavy images. Route by task class, as in Routing by task class, and revisit the routes when your own measurements change.
5. Re-benchmark with your own scorer. Build a gold set of a few hundred items per modality from your own documents and recordings, and run every candidate through the same harness, with the same judge and the same repair policy. OmniParsingBench has shown that the scorer can matter as much as the model.
6. Clear the licence and the code before the pilot. Record the territorial clause against where each workload runs, pin the reviewed commit, and run the serving plugin in an isolated environment with its media directory restricted, as Tencent’s own script does.

The multimodal advantage in an enterprise is unlikely to come from one model that understands everything. It will come from every form of evidence, a page, a call, a chart, a recorded meeting, becoming usable through one carefully governed contract. Tencent has shown that a five-billion-parameter model can learn to write that contract across ten tasks, and its own training pipeline shows how to govern it: gate every output, trust agreement over assertion, and measure each field on its own terms. Take the interface. Build the governance yourself.

Sources

1. Tencent Youtu Lab. Youtu-Parsing-Omni model card. 9 October 2026. Licence: License Terms of Youtu-Parsing.
2. Tencent Youtu Lab. Youtu-Parsing-Omni: One Encoder, One Schema, Every Modality (technical report). October 2026. Code and serving plugin: GitHub.
3. Alibaba Logics Team. Logics-Parsing-Omni Technical Report. arXiv:2603.09677, 10 March 2026.
4. OpenDataLab. OmniDocBench, v1.6 leaderboard. Updated 11 September 2026.
5. TeleOCR team. TeleOCR model card. 17 August 2026.
6. Tencent Youtu Lab. Youtu-Parsing: Perception, Structuring and Recognition via High-Parallelism Decoding. arXiv:2601.20430, 28 January 2026.
7. AWS. Amazon Bedrock Data Automation is now generally available. 3 March 2025. AWS Machine Learning Blog, Simplify multimodal generative AI with Amazon Bedrock Data Automation.
8. Microsoft. What is Azure Content Understanding in Foundry Tools? Retrieved October 2026; generally available with API version 2025-11-01.