VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark

Olympiad gold, and a road sign

Frontier vision language models now reach gold-medal level on English math and coding olympiads, and on olympiad geometry. So we showed o3 a page from a Vietnamese driving test: three warning signs and the question “Biển nào dưới đây là biển ‘Cầu hẹp’?” (Which sign below means “narrow bridge”?). It picked the wrong one.

This is not a one-off. In the seven samples below, one from each domain, o3 misses five: a chemistry precipitate curve, an AC circuit, a plant ecology diagram, the road sign, and an IQ pattern puzzle. Each question is a single image, the way these pages look to a Vietnamese student, with the text, answer options, and visual evidence together.

Seven sample VMMU questions, one per domain, each with the ground-truth answer and o3's answer. o3 is correct on Geography and Math, and wrong on Chemistry, Physics, Biology, Driving Test, and IQ Test. — **Figure 2.** Sample questions from each VMMU domain, with the ground truth (GT) and o3's answer. English translations are in the paper (Fig. 32).

Vietnamese has nearly 100 million speakers, and many everyday tasks involve exactly this kind of page: a homework sheet, a textbook PDF, a government form. Existing Vietnamese benchmarks are text only, or contain screenshots where the image is just text. None tests whether a model can combine Vietnamese text with real visual evidence.

The VMMU benchmark

VMMU has 2,548 multiple-choice questions in seven domains. Five come from Vietnam's national High School Graduation Examination; the other two come from the official driving license exam and standardized IQ tests. Together they cover four kinds of reasoning:

STEM visual reasoning: equations, scientific diagrams, and plots in Math, Physics, Chemistry, and Biology.
Data interpretation: Vietnam-specific charts, tables, and maps in Geography.
Grounded, rule-governed reasoning: traffic signs and road situations in the Driving Test.
Abstract visual reasoning: pattern completion and spatial logic in the IQ Test.

Math

456

Physics

361

Chemistry

302

Biology

341

Geography

481

Driving Test

367

IQ Test

240

Number of questions per domain.

What sets VMMU apart is that every question is genuinely multimodal: the image carries information that the text does not. We render each exam PDF to page images, cut out individual questions with OCR-based boundary detection, and automatically drop text-only screenshots. Three native Vietnamese speakers then verify every item.

VMMU data curation pipeline: raw documents become per-page images, questions are cut out by boundary detection, and a visual filter keeps multimodal images and drops text-only images before human verification. — **Figure 3.** The curation pipeline. (a) Exam PDFs are rendered to page images and split into questions. (b) A visual filter keeps only questions with non-text visual evidence, and native speakers verify each one.

The result is a mix of diagrams (28%), tables (21%), charts (20%), traffic signs and scenes (14%), and puzzles (9%). Compared with earlier benchmarks that include Vietnamese, VMMU is the only one in which all questions need real visual evidence:

Benchmark	Vietnamese questions	Vietnamese with image	Genuinely multimodal
EXAMS-V	0	0	✗
SeaExam	1,745	0	✗
VNHSGE	19,300	0	✗
VMLU	10,880	0	✗
M3Exam	1,817	116	partial
VMMU (ours)	2,548	2,548	✓

Table 1. Vietnamese coverage in related benchmarks. “Genuinely multimodal” means the question needs non-text visual evidence such as a chart, diagram, or map, not just a screenshot of text.

Finding 1

Thinking helps, but no model comes close to an expert

We evaluate 10 open-source VLMs, 3 non-thinking closed-source VLMs, and 4 thinking closed-source VLMs on all 2,548 questions. As a reference, the average Vietnamese high-school student scores 66.5% on the same exams, and the best test-takers score 99.6%.

Open-source VLMs average 37%. Even the strongest, Qwen-2.5-VL-32B (52%), is below the average student in every domain. Bigger is not always better: the larger Aya, Qwen, and Llama models do not beat their smaller siblings.
Non-thinking closed-source VLMs average 58%. Only Gemini-2.5-Flash beats the average student; Sonnet-4.0 and GPT-4.1 stay near 50%.
Thinking VLMs average 78%. Turning on test-time reasoning is the largest single jump we see, and Gemini-3-Pro reaches 86%. That is still 13 points short of the best students: not reliable enough to tutor or grade Vietnamese students.

IQ is the hardest domain for almost every model, while Geography and Math are the easiest.

Model	Math	Phys	Chem	Bio	Geo	Drive	IQ	Mean
Human & random baselines
High schoolers (average)	64.50	66.70	66.80	62.80	71.90	–	–	66.54
Human expert (best)	98.00	100.00	100.00	100.00	100.00	–	–	99.60
Random baseline	25.00	24.66	24.38	24.67	25.00	33.24	24.46	25.92
Open-source VLMs
Aya-Vision-8B	24.34	26.04	27.81	24.93	26.20	29.16	20.42	25.56
Aya-Vision-32B	23.90	24.10	19.54	20.53	14.55	34.60	20.00	22.46
Gemma-3-4B	26.32	17.45	22.19	22.29	27.65	40.60	21.67	25.45
Gemma-3-27B	47.37	30.47	38.74	30.79	47.40	43.87	35.00	39.09
Qwen-2.5-VL-32B	61.18	49.17	44.70	42.82	71.31	54.50	41.25	52.13
Qwen-2.5-VL-72B	58.33	49.86	48.34	40.47	64.66	53.95	46.67	51.75
Llama-4-Scout	59.65	44.88	45.03	38.71	69.02	49.05	37.08	49.06
Llama-4-Maverick	42.54	29.36	32.78	27.57	53.22	53.95	29.17	38.37
Mistral-Small-3.2-24B	34.65	30.28	28.48	28.15	30.98	39.78	30.83	31.88
Mistral-Medium-3	44.74	36.57	37.75	29.33	36.59	45.78	33.75	37.78
Mean	42.30	33.82	34.54	30.56	44.16	44.52	31.58	37.35
Non-thinking closed-source VLMs
Gemini-2.5-Flash	82.46	67.04	78.15	63.05	85.24	71.39	52.08	71.34
Sonnet-4.0	64.25	41.08	53.31	44.87	48.44	58.04	44.17	50.58
GPT-4.1	46.27	43.21	44.37	44.87	69.85	66.21	46.25	51.58
Mean	64.33	50.44	58.61	50.93	67.84	65.21	47.50	57.83
Thinking closed-source VLMs
GPT-5	91.23	76.45	85.10	72.73	89.40	75.20	58.75	78.41
Opus-4.6	83.33	67.31	79.47	71.26	90.85	73.84	56.25	74.62
o3	84.87	68.98	82.78	67.16	88.98	74.66	50.42	73.98
Gemini-3-Pro	92.54	81.16	91.72	80.94	94.59	91.28	72.08	86.33
Mean	87.99	73.48	84.77	73.02	90.96	78.75	59.38	78.34

Table 2. Accuracy (%) on VMMU. Bold marks the best model in each group. Human numbers are estimated from the official score distributions of the 2024 Vietnamese national high-school graduation exam; there is no human reference for the Driving and IQ tests.

Finding 2

OCR holds back open models, not closed ones

A model that cannot read the Vietnamese text in the image cannot answer the question, however well it reasons. So we asked every model to transcribe the text on 210 questions and compared it with a human-verified transcription.

Closed-source models read Vietnamese very well: mean BLEU of 89% and a character error rate of 6.6%. Their mistakes on VMMU therefore do not come from misreading the page. Open-source models are a different story. Their mean BLEU is 58%, and some collapse entirely: Aya-Vision-8B and Llama-4-Scout both exceed 200% character error rate, often ignoring the instruction and inventing text. For open models, better Vietnamese OCR is a prerequisite for anything else.

Model	BLEU (%) ↑	F1 ↑	CER (%) ↓	WER (%) ↓
Open-source VLMs
Aya-Vision-8B	2.72	0.20	229.30	253.09
Aya-Vision-32B	10.48	0.33	150.73	180.28
Gemma-3-4B	65.12	0.80	24.79	37.23
Gemma-3-27B	78.60	0.87	16.18	23.89
Qwen-2.5-VL-32B	77.50	0.88	19.05	24.17
Qwen-2.5-VL-72B	82.30	0.89	10.73	18.49
Llama-4-Scout	38.95	0.54	217.92	211.96
Llama-4-Maverick	70.59	0.81	48.77	53.91
Mistral-Small-3.2-24B	72.28	0.84	19.39	28.41
Mistral-Medium-3	81.23	0.89	9.08	16.11
Mean	57.98	0.70	74.59	84.75
Closed-source VLMs
Gemini-2.5-Flash	82.58	0.90	14.46	17.57
Sonnet-4.0	88.77	0.95	4.24	7.52
GPT-4.1	90.28	0.95	4.10	7.04
o3	94.28	0.97	4.09	5.25
Gemini-3-Pro	89.13	0.95	6.04	9.27
Mean	89.01	0.94	6.59	9.33

Table 3. Vietnamese OCR quality on 210 questions. Bold marks the best model in each group.

Finding 3

Pulling the text out of the image helps every model

If closed models can read the text, why do they still fail? One reason is that a single image asks the model to do two things at once: read dense text, and find the relevant part of a chart or diagram on the same canvas. To test this, we built a separated version of all 2,548 questions: the question and options are given as plain text, and only the visual evidence is cropped into the image.

Original setting: text and visual evidence in one screenshot. Separated setting: text as plain text, visual evidence as a separate cropped image. — **Figure 4.** In the original setting, text and visual evidence share one image. In the separated setting, the model gets the text as text and only the visual evidence as an image.

Every model improves, by +8.0 points on average. Open-source (+9.7) and non-thinking closed-source models (+8.6) gain the most. Thinking models gain only +2.1, which suggests they already cope well with the original layout.

Model	Original	Separated
Open-source VLMs
Gemma-3-27B	39.09	47.98 (+8.89)
Qwen-2.5-VL-72B	51.75	52.30 (+0.55)
Llama-4-Scout	49.06	52.55 (+3.49)
Llama-4-Maverick	38.37	52.07 (+13.70)
Mistral-Small-3.2-24B	31.88	45.82 (+13.94)
Mistral-Medium-3	37.78	55.08 (+17.30)
Mean	41.32	50.97 (+9.65)
Non-thinking closed-source VLMs
Gemini-2.5-Flash	71.34	73.70 (+2.36)
Sonnet-4.0	50.58	64.44 (+13.86)
GPT-4.1	51.58	61.05 (+9.47)
Mean	57.83	66.40 (+8.56)
Thinking closed-source VLMs
o3	73.98	76.59 (+2.61)
Gemini-3-Pro	86.33	87.92 (+1.59)
Mean	80.16	82.26 (+2.10)
Overall mean	52.89	60.86 (+7.97)

Table 4. Accuracy (%) before and after separating the text from the visual evidence, on a representative subset of models.

Finding 4

Translating to English makes things worse

A common assumption is that models trained mostly on English do better if a low-resource language is translated into English first. We tested this on 210 questions with the five strongest closed-source models. o3 translated the question and options; Google Translate's image translation (with Nano Banana 2 as a fallback) translated the text inside the image; and native speakers checked every translation.

A Vietnamese biology question about a food web and its English translation, with both the question text and the labels inside the image translated. — **Figure 5.** The translation pipeline. Both the question text and the labels inside the image are translated into English.

Accuracy drops for four of the five models, by −2.5 points on average. Translation introduces drift in Vietnam-specific terms, from traffic rules to chemical names, and adds a lossy step that strong models do not need, since they already handle Vietnamese well. Translation is not a substitute for native-language evaluation.

Model	Vietnamese	English
Gemini-2.5-Flash	82.38	76.19 (-6.19)
Sonnet-4.0	67.14	69.05 (+1.91)
GPT-4.1	67.14	66.19 (-0.95)
o3	81.43	78.10 (-3.33)
Gemini-3-Pro	92.38	88.57 (-3.81)
Mean	78.09	75.62 (-2.47)

Table 5. Accuracy (%) on the original Vietnamese questions and on their English translations.

Conclusion

Reading Vietnamese is largely solved for frontier closed-source models. Reasoning over Vietnamese pages is not. These models transcribe the text well, yet they still miss questions that an average high-school student gets right. Separating the text from the visual evidence helps; translating into English does not. Progress will need stronger joint text-and-vision reasoning on dense layouts, and Vietnamese-centric training that targets grounding, not just language. We hope VMMU helps measure that progress and encourages larger native multimodal datasets for low-resource languages.

Citation

@inproceedings{dang2026vmmu,
  title     = {{VMMU}: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark},
  author    = {Dang, Vy Tuong and Vo, An and Villa-Cueva, Emilio and Tau, Quang and Dm, Duc and Solorio, Thamar and Kim, Daeyoung},
  booktitle = {Findings of the Association for Computational Linguistics: AACL-IJCNLP 2026},
  year      = {2026},
  url       = {https://arxiv.org/abs/2508.13680}
}