A twenty-minute fine-tune of a 2015 CNN beat GPT-6 and Claude at skin cancer

The most useful number in Magnus Lynch’s skin cancer test is 68,011. That is how many public dermatology images it took to turn Inception v3, a Google network from 2015, into something that named the right diagnosis for 125 of 200 biopsy-proven lesions. GPT-6 Astra and Claude Fable 5.1, given both the clinical photograph and the dermoscopic image, managed 84 and 85. The fine-tune took about twenty minutes on one H100.

I expected the gap to go the other way, or at least to be small. The medical case for general models has leant on exam scores since 2023, when Microsoft researchers reported that GPT-4 cleared the USMLE pass mark by more than 20 points with no special prompting and beat Med-PaLM, a model fine-tuned on medical knowledge. Several model generations later, I assumed a frontier model looking at a mole would at least tie a network from 2015. It lost by twenty percentage points, and it lost worst on the lesions a busy clinic most wants to send home. The CNN recognised 26 of 50 seborrhoeic keratoses; Claude Fable 5.1 recognised 6.

The failure pattern bothers me more than the headline score. GPT-6 Astra called 75 lesions basal cell carcinoma when only 25 were, and 14 of those it mislabelled were melanomas. Claude Fable 5.1 labelled 53 lesions actinic keratosis against 20 real ones. Each model had a favourite answer and kept reaching for it. A benchmark that reports one aggregate accuracy figure averages that habit away. You only see it in a confusion matrix built on a narrow, labelled set shaped like the actual task, and general leaderboards do not publish those.

Lynch’s result fits a pile of earlier work. Pillai and colleagues found GPT-4V got the primary diagnosis on 54% of dermatology images and on 1 of 6 squamous cell carcinomas. Perlmutter’s group put GPT-4’s melanoma sensitivity on the PH2 dataset at 52.5%. A December 2025 arXiv paper found a LoRA-tuned MedGemma-4b averaging 80.37% across six imaging tasks against 69.58% for untuned GPT-4. And Andre Esteva’s Stanford team reported dermatologist-level skin cancer classification with a CNN in Nature in January 2017, so the specialised baseline has been public for nine years.

The obvious reply is that nobody sells a chatbot as a diagnostic device, and that when Lynch asked as a patient, with one photograph and the question of whether to see a doctor, both models said yes for 99 and 98 of 100 melanomas. I don’t find that reassuring. Lynch reports they also told people with harmless lesions to see a doctor in almost every case, so the advice carries close to zero information; a sign on the surgery door would do as well. And in the clinician framing, which is how a doctor would actually use one, the same models were willing to discharge cancers as harmless warty growths.

A fairer objection is that specialised tools have their own problems. DermaSensor, which received FDA de novo clearance, caught 42 of 44 melanomas in its November 2023 clearance study but had 32.5% specificity and a positive predictive value of 16.0%. Its chairman, Maurice Ferre, says getting there took 12 years and tens of millions of dollars. I accept all of that. The difference is that you can read those numbers off a study with confidence intervals and decide whether a tool that flags lots of benign lesions suits your referral pathway. For a frontier LLM there is no such study on your task until a consultant dermatologist builds one. Lynch is candid about his own limits: 30 melanomas give wide error bars, each LLM case ran once, and MILK10k has been public since 2025, which if anything should have helped the LLMs.

So I think medical AI procurement runs in the wrong order. Buyers start from the most capable general model and ask how to make it safe. They should start from the dullest specialised baseline anyone can build and make the expensive model beat it on the buyer’s own biopsy-proven cases. Lynch has published the 200 cases, the prompts, every LLM reply and the scoring script, and his CNN runs in a phone browser with nothing uploaded. Any vendor pitching an LLM for dermatology triage can run that download in an afternoon. If its product cannot clear 125 of 200 overall and 13 of 30 melanomas, take the contract off the table.

Prompted by A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis, lynchlab.net.