MedGemma improves medical AI performance
MedGemma, a collection of artificial intelligence models trained for medical applications, outperformed its general-purpose base models on several medical text and imaging benchmarks and generally performed better after task-specific fine-tuning, according to a study published in Nature Medicine.
Google researchers evaluated the models across medical knowledge, image classification, clinical reasoning, chest radiograph report generation, and simulated clinical encounters.
Model development and evaluation
Built on Google’s Gemma 3 models, MedGemma includes 2 variants that accept images and text, with 4 billion and 27 billion parameters, and a 27-billion-parameter text-only variant. Parameters are numerical values learned during training. MedGemma’s image interpretation is supported by a medically trained image encoder called MedSigLIP.
Training incorporated medical question-answering datasets, synthetic medical questions, and medical images paired with text from radiology, histopathology, dermatology, and ophthalmology. General-purpose training data were retained to preserve broader capabilities.
Evaluation used data kept separate from MedGemma-specific training. Some evaluations used separate portions of datasets that also contributed to model development; others used entirely different data sources. Comparators included the corresponding Gemma 3 models, other medically specialized models, and larger proprietary models.
Medical knowledge and image classification
On MedQA, a benchmark containing 1,273 medical examination questions, MedGemma 27B Text achieved 86% accuracy using additional computation when generating answers, compared with 75% for Gemma 3 27B. The smaller MedGemma 4B model achieved 64% accuracy vs 52% for Gemma 3 4B.
The retinal, histopathology, and dermatology evaluations used multiple-choice classification tasks without task-specific examples in the prompts. On 3,161 retinal images from EyePACS, accuracy was 75% for MedGemma 27B vs 20% for Gemma 3 27B. Corresponding results for the smaller models were 65% vs 14%. EyePACS also contributed training data, but the evaluation images were kept separate from MedGemma-specific training.
For histopathology classification, MedGemma 27B achieved 72% accuracy vs 42% for its base model. Dermatology classification accuracy was 72% vs 67%, although Gemini 2.5 Pro achieved 81% on that task.
For chest radiograph classification on CheXpert, a dataset not used in MedGemma development, MedGemma 4B achieved a macro F1 score of 0.481 vs 0.326 for Gemma 3 4B. This measure combines sensitivity with precision—the proportion of positive predictions that are correct—and gives equal weight to each of the 5 evaluated findings. Higher scores indicate better performance. Results varied across datasets and tasks, and specialized or larger models remained stronger on some evaluations.
The researchers also assessed open-ended medical responses, with 3 physicians rating responses to each question. Ratings indicating factual inaccuracy decreased from approximately 19% for Gemma 3 4B to 15% for MedGemma 4B. The authors cautioned that ratings could vary among physicians and that evaluation should reflect the intended application.
Report generation and task adaptation
A single US board-certified thoracic radiologist evaluated reports generated by the MedGemma 4B pretrained model using chest radiographs from the MIMIC-CXR test set. The radiologist rated 68% of reports from normal studies and 49% from abnormal studies as equal or superior to the original reports. Overall, 81% were judged likely to result in the same or superior clinical management compared with the original reports. The radiologist knew which reports were AI-generated. The management assessments reflected the radiologist’s judgments rather than observed patient outcomes.
Additional fine-tuning experiments compared MedGemma 4B with Gemma 3 4B on 3 previously unseen datasets for classifying pneumothorax, histopathology patches, and skin lesions. After fine-tuning, MedGemma generally achieved higher balanced accuracy—a measure that gives equal weight to sensitivity in each classification category—than Gemma 3. MedGemma's advantage was particularly apparent when only 10% of available training data were used. Further fine-tuning also improved chest radiograph report generation.
Limitations and further validation
The authors noted that benchmark performance provides only an initial assessment of real-world utility. Although evaluation data were kept separate from MedGemma-specific training, some publicly available benchmark material may have appeared in the base models’ pretraining data.
Differences in evaluation methods also limited comparisons with published results for other models. Several private training and evaluation datasets could not be shared because of licensing restrictions.
The smaller Gemma 3 4B and MedGemma 4B models also had difficulty following system instructions during simulated clinical encounters and were poorly suited to that task.
The authors identified model errors and potential bias in training data as areas requiring continued improvement and validation. They also noted that specialized models may remain better suited to particular tasks.
“Thoughtful validation of safety, performance and reliability for downstream applications remains a critical aspect to advancing the use of multimodal AI models in medicine,” the authors wrote.
The study was funded by Alphabet and/or a subsidiary. The authors disclosed that all authors except Charles Lau and Catherine Kozlowski were current or former employees of Google LLC, a subsidiary of Alphabet, and may own stock as part of their standard compensation packages.
AACE Endocrine AI is published by Conexiant under a license arrangement with the American Association of Clinical Endocrinology, Inc. (AACE®). The ideas and opinions expressed in AACE Endocrine AI do not necessarily reflect those of Conexiant or AACE. For more information, see Policies.