FunBench: Benchmarking Fundus Reading Skills of MLLMs — Paper Reading Notes
- Paper: FunBench: Benchmarking Fundus Reading Skills of MLLMs
- Authors: Qijie Wei, Kaiheng Qian, and Xirong Li
- Venue/Year: MICCAI 2025
- Code/Data: GitHub repository · Hugging Face dataset
TL;DR
FunBench is a novel hierarchical dataset and benchmark for evaluating the effectiveness of open-source multimodal large language models (MLLMs) on fundus reading tasks of varying difficulty. The code and data are available on GitHub and Hugging Face. 1

1. Why This Paper Matters
The paper provides a structured benchmark for evaluating the fundus reading abilities of MLLMs. Its task framework can also inform the evaluation of ophthalmic agent workflows and the development of benchmarks in other domains. 1
2. Background and Motivation
2.1 Context and Problem
Although research on medical MLLMs is growing rapidly, the development of ophthalmology-focused benchmarks is lagging behind. 1
2.2 Gap in Prior Work
The authors argue that existing medical benchmarks, including OmniMedVQA and GMAI-MMBench, treat retinal image-based VQA as a single task and therefore lack a detailed, structured evaluation of fundus reading skills. 1 4 5
The authors describe LMOD as a more detailed ophthalmic benchmark that evaluates recognition of the optic cup, optic disc, and fovea, as well as glaucoma and macular hole. They distinguish FunBench from LMOD through its hierarchical task organization and targeted evaluation modes. 1 6
3. Core Contributions
- It designs hierarchical tasks ranging from low-level modality and anatomy perception to high-level lesion analysis and disease diagnosis.
- It introduces three evaluation modes intended to assess the vision encoder (VE), the large language model (LLM), and the complete MLLM. 1
4. Method
4.1 Overview
FunBench organizes fundus reading into four levels and evaluates MLLMs through three targeted evaluation modes. 1
4.2 Dataset Curation
4.2.1 Hierarchical Task Organization
The example questions below are adapted for readability from the task templates. Use the released annotations for evaluation. 3
Level 1 (L1): Modality perception
-
L1a: Coarse-grained modality perception
Please choose the most suitable option based on the image and the question. Answer directly with the option letter.
Question: What kind of image is presented here?
Options: A. Fundus image; B. Remote sensing; C. Painting; D. Whole-slide image. -
L1b: Fine-grained modality perception
Please choose the most suitable option based on the image and the question. Answer directly with the option letter.
Question: Which imaging technique was used to capture the image?
Options: A. Ultra-widefield fundus photography; B. Ultrasound; C. Fluorescein fundus angiography; D. X-ray.
Level 2 (L2): Anatomy perception
-
L2a: Determine the relative position of the optic disc and fovea
Please choose the most suitable option based on the image and the question. Answer directly with the option letter.
Question: Which is on the left, the optic disc or the fovea?
Options: A. Optic disc; B. Fovea. -
L2b: Recognize the laterality of the fundus image
Please choose the most suitable option based on the image and the question. Answer directly with the option letter.
Question: Which eye is shown in this image, the left or the right?
Options: A. Left eye; B. Right eye.
Level 3 (L3): Lesion analysis
-
L3a: Lesion recognition
Please choose the most suitable option based on the image and the question. Answer directly with the option letter.
Question: Does this fundus image show any evidence of fibrous proliferation?
Options: A. No; B. Yes. -
L3b: Lesion localization
Please choose all suitable options based on the image and the question. Answer directly with the option letters, separated by commas if necessary.
Question: Where is the fibrous proliferation in this fundus image?
Options: A. Temporal to the optic disc center; B. Nasal to the optic disc center; C. Not observed in the image; D. Inferior to the optic disc center; E. Superior to the optic disc center. -
L3c: Lesion size estimation
Please choose the most suitable option based on the image and the question. Answer directly with the option letter.
Question: What is the size of the fibrous proliferation depicted in the fundus photograph?
Options: A. Not observed in the image; B. No more than 0.5 disc area; C. Between 0.5 and 1.0 disc area; D. Between 1.0 and 1.5 disc areas; E. Between 1.5 and 2.0 disc areas; F. More than 2.0 disc areas. -
L3d: Lesion counting
Please choose the most suitable option based on the image and the question. Answer directly with the option letter.
Question: How many haemorrhages do you see in the image?
Options: A. Not observed in the image; B. No more than 5; C. Between 5 and 15; D. Between 15 and 25; E. Between 25 and 35; F. More than 35.
Level 4 (L4): Disease diagnosis
-
L4a: Binary-condition diagnosis
Please choose the most suitable option based on the image and the question. Answer directly with the option letter.
Question: Does this fundus image show any signs of illness?
Options: A. Yes; B. No. -
L4b: Multi-condition diagnosis
Please choose all suitable options based on the image and the question. Answer directly with the option letters, separated by commas if necessary.
Question: What specific abnormalities are shown in this fundus image?
Options: A. No abnormality; B. Retinitis pigmentosa; C. Retinal vein occlusion; D. Retinal artery occlusion; E. Diabetic retinopathy. -
L4c: Diabetic retinopathy grading
Please choose the most suitable option based on the image and the question. Answer directly with the option letter.
Question: What stage of diabetic retinopathy can be observed in the given fundus image?
Options: A. Proliferative diabetic retinopathy; B. Mild nonproliferative diabetic retinopathy; C. Severe nonproliferative diabetic retinopathy; D. No diabetic retinopathy; E. Moderate nonproliferative diabetic retinopathy. -
L4d: Age-related macular degeneration categorization
Please choose the most suitable option based on the image and the question. Answer directly with the option letter.
Question: What is the age-related macular degeneration category of the presented fundus image?
Options: A. Early age-related macular degeneration; B. Intermediate age-related macular degeneration; C. No age-related macular degeneration; D. Late age-related macular degeneration.
The exact released task definitions are available in the FunBench dataset. 3
4.2.2 Data Sources
FunBench adapts the following 14 public datasets: 1
- Six CFP datasets: IDRiD, DDR, JSIEC, RFMiD, OIA-ODIR, and Retinal-Lesions
- Five OCT datasets: OCTDL, NEH, OCTID, UCSD, and RETOUCH
- One UWF dataset: TOP
- Two multimodal datasets: MMC-AMD (CFP + OCT) and DeepDRiD (CFP + UWF)
Table 1. FunBench statistics: 16,348 fundus images and 91,810 visual questions across four task levels. Adapted from the paper’s Table 1. 1
| Level | Single-answer questions | Multiple-answer questions | Sample question | Data sources |
|---|---|---|---|---|
| L1 Tasks: 2 | 32,696 | 0 | What method was used to capture this image? A. Magnetic resonance imaging B. Ultra-widefield fundus photography C. Color fundus photography D. Optical coherence tomography | All datasets |
| L2 Tasks: 2 | 10,980 | 0 | Which eye is shown in this image, the left or the right? A. Right eye B. Left eye | CFP: DDR, DeepDRiD, IDRiD, OIA-ODIR, Retinal-Lesions UWF: TOP CFP + UWF: DeepDRiD |
| L3 Tasks: 4 Subtasks: 39 | 15,606 | 7,237 | What are the positions of the haemorrhages in the fundus image? A. Nasal to the optic disc center B. Temporal to the optic disc center C. Superior to the optic disc center D. Inferior to the optic disc center E. Not observed in the image | CFP: DDR, IDRiD, Retinal-Lesions OCT: RETOUCH |
| L4 Tasks: 4 | 20,177 | 5,114 | Which abnormalities can be seen in this fundus image? A. Glaucoma B. Diabetic retinopathy C. No abnormality D. Age-related macular degeneration E. Hypertensive retinopathy | CFP: DDR, IDRiD, OIA-ODIR, JSIEC, RFMiD, Retinal-Lesions OCT: NEH, OCTDL, OCTID, UCSD UWF: TOP CFP + OCT: MMC-AMD CFP + UWF: DeepDRiD |
4.3 Targeted Evaluation Modes
To assess an MLLM and its two key modules—the LLM and VE—the authors present three targeted evaluation modes (E-modes) in a bottom-up manner. 1 (Figure 1)
E-mode I: Linear-probe-based VE Evaluation
As illustrated in Figure 1b, linear probing trains a linear classification head for each task or subtask using the task-specific development dataset. 1
The authors omit tasks that cannot be directly addressed as classification problems, such as L3b, L3c, and L3d, as well as tasks that are trivial for linear probing, such as L1a and L1b.
E-mode II: Knowledge-prompted LLM Evaluation
Given a test image and its associated task-specific label, the authors convert the label into an indirect description by querying an expert knowledge database, EyeWiki. The description is then provided together with the image and question. 1 8
Example: L2b laterality recognition
Please choose the most suitable option based on the image, description, and question. Answer directly with the option letter.
Description: The fundus image shows the fovea located to the left of the optic disc.
Question: Which eye is shown in this image, the left or the right?
Options: A. Left eye; B. Right eye.
Illustrative example: L3a hard-exudate recognition
Please choose the most suitable option based on the image, description, and question. Answer directly with the option letter.
Description: The fundus image shows white or yellowish deposits with sharply defined margins.
Question: Does this fundus image show any evidence of hard exudates?
Options: A. No; B. Yes.
The example above illustrates the prompt structure using the hard-exudate description given in the paper. 1
E-mode III: Holistic Evaluation
This mode provides an end-to-end evaluation of the MLLM. 1
4.4 Performance Metrics
The authors report the harmonic mean of sensitivity and specificity, which they call “F1”:
For multiclass tasks such as DR grading (L4c), the authors compute the task-level score as the mean of the class-level scores. 1
5. Evaluating MLLMs on FunBench
5.1 Choice of MLLMs
The authors select MLLMs at the 7B/8B scale, compiling a list of six general-purpose models and three medical models, as shown in Table 2. In addition, they include GPT-4o as a proprietary baseline and DINOv2-large as a VE baseline. 1
Table 2. Open-source MLLMs evaluated in the paper. Medical models are marked with *. 1 (Table 2)
| MLLM | Hugging Face release | VE | LLM |
|---|---|---|---|
| LLaVA-v1.5-7B | 2023.10 | CLIP-ViT | Vicuna-7B |
| *Qilin-Med-VL-Chat | 2023.12 | CLIP-ViT | Chinese-LLaMA2 |
| LLaVA-v1.6-7B | 2024.01 | CLIP-ViT | Vicuna-7B |
| *LLaVA-Med-v1.5-7B | 2024.05 | CLIP-ViT | Mistral-7B |
| Qwen2-VL-7B | 2024.09 | Qwen2-ViT | Qwen2-7B |
| InternVL2.5-8B | 2024.12 | InternViT | InternLM2.5-7B |
| *HuatuoGPT-Vision-7B | 2024.06 | CLIP-ViT | Qwen2-7B |
| Janus-Pro-7B | 2025.01 | ViT-SigLIP | DeepSeek-LLM-7B |
| Qwen2.5-VL-7B | 2025.01 | Qwen2.5-ViT | Qwen2.5-7B |
5.2 Results
Table 3. Results on FunBench. Within each evaluation mode, the highest value in each column (including ties) is shown in red and bold; values below the random-guess baseline are shown in gray. 1 (Table 3)
| Model | Overall performance | L1: Modality | L2: Anatomy | L3: Lesion | L4: Disease | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MEAN | L1 | L2 | L3 | L4 | L1a | L1b | L2a | L2b | L3a | L3b | L3c | L3d | L4a | L4b | L4c | L4d | |
| Random guess | 0.393 | 0.313 | 0.500 | 0.361 | 0.397 | 0.250 | 0.375 | 0.500 | 0.500 | 0.468 | 0.458 | 0.250 | 0.269 | 0.500 | 0.398 | 0.319 | 0.372 |
| E-mode I: | |||||||||||||||||
| CLIP-ViT | 0.578 | - | 0.820 | 0.373 | 0.541 | - | - | - | 0.820 | 0.373 | - | - | - | 0.814 | 0.175 | 0.556 | 0.621 |
| Qwen2-ViT | 0.597 | - | 0.892 | 0.338 | 0.560 | - | - | - | 0.892 | 0.338 | - | - | - | 0.814 | 0.225 | 0.506 | 0.694 |
| InternViT | 0.614 | - | 0.932 | 0.392 | 0.517 | - | - | - | 0.932 | 0.392 | - | - | - | 0.815 | 0.230 | 0.530 | 0.491 |
| ViT-SigLIP | 0.619 | - | 0.880 | 0.398 | 0.581 | - | - | - | 0.880 | 0.398 | - | - | - | 0.833 | 0.293 | 0.528 | 0.669 |
| Qwen2.5-ViT | 0.651 | - | 0.932 | 0.449 | 0.573 | - | - | - | 0.932 | 0.449 | - | - | - | 0.812 | 0.319 | 0.514 | 0.645 |
| DINOv2-large | 0.655 | - | 0.939 | 0.408 | 0.616 | - | - | - | 0.939 | 0.408 | - | - | - | 0.858 | 0.371 | 0.568 | 0.669 |
| E-mode II: | |||||||||||||||||
| Janus-Pro | 0.409 | - | 0.560 | 0.213 | 0.454 | - | - | - | 0.560 | 0.213 | - | - | - | - | 0.517 | 0.446 | 0.401 |
| Qilin-Med-VL | 0.473 | - | 0.473 | 0.519 | 0.426 | - | - | - | 0.473 | 0.519 | - | - | - | - | 0.552 | 0.391 | 0.334 |
| LLaVA-v1.5 | 0.487 | - | 0.386 | 0.560 | 0.515 | - | - | - | 0.386 | 0.560 | - | - | - | - | 0.521 | 0.557 | 0.467 |
| LLaVA-Med-v1.5 | 0.529 | - | 0.483 | 0.507 | 0.597 | - | - | - | 0.483 | 0.507 | - | - | - | - | 0.638 | 0.640 | 0.513 |
| LLaVA-v1.6 | 0.604 | - | 0.502 | 0.717 | 0.594 | - | - | - | 0.502 | 0.717 | - | - | - | - | 0.606 | 0.647 | 0.529 |
| Qwen2.5-VL | 0.654 | - | 0.497 | 0.735 | 0.729 | - | - | - | 0.497 | 0.735 | - | - | - | - | 0.757 | 0.726 | 0.704 |
| Qwen2-VL | 0.673 | - | 0.488 | 0.747 | 0.783 | - | - | - | 0.488 | 0.747 | - | - | - | - | 0.750 | 0.855 | 0.743 |
| HuatuoGPT-V | 0.692 | - | 0.541 | 0.706 | 0.829 | - | - | - | 0.541 | 0.706 | - | - | - | - | 0.793 | 0.927 | 0.768 |
| InternVL2.5 | 0.703 | - | 0.488 | 0.831 | 0.789 | - | - | - | 0.488 | 0.831 | - | - | - | - | 0.792 | 0.929 | 0.647 |
| E-mode III: | |||||||||||||||||
| Janus-Pro | 0.355 | 0.777 | 0.167 | 0.108 | 0.369 | 0.943 | 0.611 | 0.189 | 0.145 | 0.115 | 0.203 | 0.060 | 0.055 | 0.523 | 0.319 | 0.278 | 0.358 |
| LLaVA-v1.5 | 0.418 | 0.722 | 0.489 | 0.165 | 0.296 | 0.974 | 0.470 | 0.492 | 0.485 | 0.276 | 0.266 | 0.013 | 0.104 | 0.364 | 0.292 | 0.187 | 0.341 |
| Qilin-Med-VL | 0.426 | 0.651 | 0.496 | 0.211 | 0.347 | 0.910 | 0.392 | 0.504 | 0.489 | 0.463 | 0.361 | 0.000 | 0.019 | 0.491 | 0.354 | 0.214 | 0.328 |
| LLaVA-v1.6 | 0.435 | 0.777 | 0.449 | 0.194 | 0.319 | 0.970 | 0.585 | 0.400 | 0.497 | 0.461 | 0.243 | 0.058 | 0.013 | 0.357 | 0.314 | 0.289 | 0.315 |
| LLaVA-Med-v1.5 | 0.440 | 0.813 | 0.507 | 0.215 | 0.223 | 0.961 | 0.665 | 0.502 | 0.512 | 0.166 | 0.349 | 0.261 | 0.086 | 0.000 | 0.255 | 0.290 | 0.345 |
| Qwen2.5-VL | 0.480 | 0.942 | 0.364 | 0.231 | 0.384 | 0.978 | 0.906 | 0.310 | 0.417 | 0.386 | 0.242 | 0.218 | 0.077 | 0.532 | 0.394 | 0.303 | 0.308 |
| InternVL2.5 | 0.506 | 0.946 | 0.564 | 0.244 | 0.269 | 0.997 | 0.894 | 0.640 | 0.488 | 0.391 | 0.144 | 0.227 | 0.213 | 0.031 | 0.400 | 0.294 | 0.351 |
| Qwen2-VL | 0.520 | 0.925 | 0.497 | 0.316 | 0.343 | 0.995 | 0.855 | 0.506 | 0.488 | 0.503 | 0.432 | 0.073 | 0.255 | 0.508 | 0.352 | 0.202 | 0.311 |
| HuatuoGPT-V | 0.523 | 0.933 | 0.309 | 0.374 | 0.477 | 0.961 | 0.905 | 0.140 | 0.477 | 0.548 | 0.379 | 0.338 | 0.231 | 0.622 | 0.504 | 0.414 | 0.369 |
| GPT-4o | 0.542 | 0.961 | 0.535 | 0.341 | 0.331 | 0.965 | 0.957 | 0.557 | 0.514 | 0.362 | 0.412 | 0.361 | 0.228 | 0.018 | 0.452 | 0.476 | 0.378 |
VE Comparison
The performance of the different VEs is shown in the E-mode I section of Table 3. 1
The key results are as follows:
- Under the tested frozen-encoder linear-probe protocol and aggregate scoring procedure, DINOv2 is the best-performing VE, whereas CLIP-ViT has the lowest score among the evaluated encoders.
- The performance of the VEs is even worse than chance on L4b.
The authors interpret these results as evidence of limitations in pure-vision solutions for fundus image analysis.
LLM Comparison
The results of the authors’ knowledge-prompted LLM evaluation are shown in the E-mode II section of Table 3. 1
The key results are as follows:
- The authors interpret the stronger knowledge-prompted performance of InternVL2.5, HuatuoGPT-Vision, and Qwen relative to their VE counterparts as evidence that their LLMs possess ophthalmic knowledge relevant to fundus reading.
- Performance varies across tasks: the models perform better on DR grading (L4c) than on AMD categorization (L4d).
- Performance on L2b is close to chance.
The authors explain the second result by suggesting that DR-related materials are more abundant online than AMD-related materials, making LLMs more familiar with DR. For the poor performance on L2b, despite it being a simple task, the authors suggest that the skill may be too basic to be widely discussed, making relevant training data rare.
The authors conclude that the results suggest a fundamental limitation of the current data-driven paradigm: it produces powerful models that lack basic fundus reading skills.
MLLM Comparison
The performance of the MLLMs is summarized in the final section of Table 3. 1
- Among the tested open-source MLLMs, HuatuoGPT-Vision performs best, followed by Qwen2-VL and InternVL2.5. GPT-4o has the highest overall score when the proprietary baseline is included.
- HuatuoGPT-Vision and Qwen2-VL use LLMs with the same architecture, Qwen2-7B, yet the former’s VE (CLIP-ViT) performs worse in the linear-probe evaluation than the latter’s VE (Qwen2-ViT). The authors take HuatuoGPT-Vision’s stronger overall performance as evidence of the importance of domain-specific fine-tuning.
- Table 4 reports a much higher rank correlation between holistic performance and knowledge-prompted performance than between holistic performance and VE-probe performance. The authors interpret this as evidence that the evaluated MLLMs rely heavily on their internal LLMs for fundus reading.
The authors state that the evaluated MLLMs have quite limited fundus reading skills. They also argue that the correlation analysis highlights the urgent need to develop a strong ophthalmic LLM.
Table 4. Correlation analysis based on mean-performance ranks. 1 (Table 4)
| Module | Spearman correlation with MLLM |
|---|---|
| LLM | 0.917 |
| VE | 0.055 |
5.3 Authors’ Conclusions
- The evaluated MLLMs remain weak at fundus reading tasks involving anatomy perception, lesion analysis, and disease diagnosis.
- The evaluated MLLMs rely more heavily on their LLMs than on their VEs.
- The VEs used by these MLLMs are less effective than DINOv2 under the linear-probe evaluation.
- HuatuoGPT-Vision’s leading performance among the open-source models shows the importance of domain-specific training.
- Future training procedures need to consider the full range of fundus reading skills; otherwise, we risk developing an MLLM that lacks basic fundus reading abilities. 1
6. My Comments
I find FunBench useful for evaluating fundus reading skills beyond a single diagnosis score. We can use its tasks to evaluate our models, adapt the framework for agent workflows, and extend the approach to other domains. The public code and annotations make this easier, although source-image preparation and model integration still require work. 2 3
There are also several issues worth noting.
I could not identify an E-mode I feature-extraction and linear-probe training pipeline in the public repository version inspected on October 2, 2026 (see issue #2). This leaves a gap in reproducing the paper’s modular evaluation.
The authors’ explanations and hypotheses regarding the results are not sufficiently convincing.
For example, the authors suggest that the worse performance on AMD than on DR is related to the relative availability of training material. The observed performance difference motivates this hypothesis, but the paper provides no direct evidence about the relative coverage of DR and AMD in the models’ training data. Differences in image difficulty, class definitions, and question design remain possible alternatives.
I am also unconvinced that poor laterality recognition is mainly caused by a lack of relevant training examples. Ambiguity in image coordinates, anatomical laterality, and viewing perspective is another possible source of error. The relatively strong laterality results from the linear probe suggest that useful information is readable from the visual features, so I would also investigate the connector, prompt interpretation, and answer mapping. Testing explicit coordinate conventions and controlled horizontal flips could help locate the problem.
My main concern is the attribution of performance to individual modules. E-mode II supplies both an image and a description derived from the reference label. Its score therefore measures performance with correct semantic cues already supplied, while E-mode I measures supervised linear readout from visual features. These modes differ in their inputs, supervision, and task coverage. I do not think their scores or their correlations with holistic performance are sufficient to establish that the LLM contributes more than the VE. 1 8
A description-only condition, together with blank or shuffled images, would help measure the image’s contribution when a description is supplied. The authors argue that text-only testing may not capture the behavior of an LLM after multimodal adaptation; I would still use it as a complementary control. Descriptions written by experts without access to the reference label and controlled module replacements would provide further evidence for attribution.
The same attribution issue affects the fine-tuning claim: HuatuoGPT-Vision and Qwen2-VL differ in several system components and training choices, so their comparison does not isolate the effect of domain-specific training. Likewise, weak linear-probe results demonstrate limitations of the tested representations and readout protocol, rather than all pure-vision approaches.
Overall, I value the hierarchical task framework, but find the evidence for module attribution less convincing. I regard the end-to-end evaluation as more directly relevant to practical image-based fundus reading.
References
- Wei, Q., Qian, K., and Li, X. FunBench: Benchmarking Fundus Reading Skills of MLLMs. MICCAI 2025, LNCS 15965, pp. 278–288. DOI · Open-access manuscript.
- RUC AIMC Lab. FunBench source code and preparation instructions. Repository.
- AIMClab-RUC. FunBench VQA annotations. Hugging Face dataset.
- Hu, Y., et al. OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM. CVPR 2024. Official proceedings.
- Chen, P., et al. GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI. NeurIPS 2024. Official proceedings.
- Qin, Z., et al. LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models. Findings of NAACL 2025. ACL Anthology.
- RUC AIMC Lab.
evaluate.py. Fixed-version source. - RUC AIMC Lab.
predict.py. Fixed-version source. - Question about evaluation for
emod1. FunBench GitHub issue #2. Issue.
Comments
Sign in — Sign in to join the conversation.