Description
Multimodal LLMs claim to read charts and formulas, but measuring how close they are to experts needs a common yardstick. MMMU is a massive multi-discipline multimodal understanding and reasoning benchmark across art, business, science, engineering and medicine, drawn from college exams and textbooks.
The repo provides evaluation code for per-subject scores, plus the harder MMMU-Pro.
Disciplines:Six fields, 30 subjects.
Image questions:Charts, formulas and photos.
Eval code:Scores in one run.
Leaderboard:Compare models.
The repo provides evaluation code for per-subject scores, plus the harder MMMU-Pro.
Features
Disciplines:Six fields, 30 subjects.
Image questions:Charts, formulas and photos.
Eval code:Scores in one run.
Leaderboard:Compare models.
