Every take-home coding test now grades Claude, so stop scoring them

In the fall of 2025, Claude Code finished the first homework of a Carnegie Mellon course, written reflection included, from nothing more than a pointer to the assignment text. The task was to add two AI features to albumy, a small, clean Instagram clone, and it existed to screen whether students could find their way around unfamiliar code. The instructor, who teaches Machine Learning in Production to 100 to 170 students a term, has written up how he rebuilt his assessments. The most useful thing in his post is an admission tucked near the end: most students get full or nearly full credit on homework, and the grades now separate on the exams.

I expected an educator’s essay about integrity. What I got reads like a grading audit, and it matches what Anthropic reported about its own hiring test almost line for line. Once Claude 3.7 Sonnet arrived, more than half of applicants for its performance-engineering take-home would have scored higher by handing the whole task to the model. By May 2025 Claude Opus 4 was beating nearly all human solutions within the time limit, so Anthropic cut the limit to two hours, and Claude Opus 4.5 then matched its best candidates inside those two hours. Tristan Hume, the team lead, wrote: “Each new Claude model has forced us to redesign the test.”

My position is that a take-home score, in a course or a hiring loop, should now carry zero weight in any decision. If a model can produce a passing submission with no human input, the score mostly tells you whether the candidate pays for a subscription. You can keep setting the exercise. The measuring should happen in a room, where someone explains their code while another person pokes at it.

The CMU course shows what that costs. After each assignment every student books a 15-minute conversation with a TA, worth 20% of the assignment points, graded pass/fail, with penalty-free retries. At a 20:1 student-TA ratio that comes to roughly 300 minutes per TA every two weeks. The TAs found the time because an LLM-as-a-judge now pre-sorts written work into “pass” and “needs review”, which the instructor says cut their grading time by 50 to 80%. Teams also sit 30 to 60 minute debriefs after each milestone and earn bonus points if the TA can ask any member to explain any part of the implementation. I think that rule is the best code review interview I have seen described, and a company could copy it tomorrow.

The serious objection comes from learning research, and the instructor raises it himself. Frequent low-stakes practice beats a few high-stakes exams, and he remembers the German model of a single end-of-semester exam worth 100% of the grade, with failure and drop-out rates of 50 to 80%. Bastani and colleagues, in PNAS in June 2025, found that high school students using a GPT-4 tutor with guardrails raised practice scores by 127% and showed essentially no drop on the exam, which suggests AI-assisted homework can still teach when someone designs it carefully.

I agree, and the same study helps my case. Students with plain GPT-4 scored 48% higher on practice problems, then 17% lower on the exam once the AI was taken away. Practice scores rose in both groups and gave no hint of which students had actually learned anything. So keep homework as practice, grade it pass/fail by machine if you like, and put the decision weight on a live exchange that is short, frequent and retryable, which describes the 15-minute check-in exactly. Nothing about in-person assessment requires it to be a single terrifying exam.

Hiring has moved more slowly. Anthropic still runs a take-home, rewritten with unusual constraints and with AI allowed, and without a time limit humans can still beat Opus 4.5, sometimes by a wide margin. I read that as a test whose shelf life runs to the next model release. Google, after an employee at a February 2025 town hall asked for on-site interviews back, settled on adding at least one in-person round, according to Sundar Pichai, which leaves every other round as open to an agent as before.

Live and remote carries its own hole. The CMU course caught students using Cluely during Zoom check-ins, and the instructor now plans to hold them all in person. Any company running remote live coding rounds should assume the same thing is happening in its own interviews.

Paste your team’s take-home, or your course’s first assignment, into a coding agent with only the brief and no follow-up prompts. If the output passes your rubric, remove that score from the scorecard or gradebook before your next cycle and replace it with 15 minutes of the candidate walking a reviewer through their own diff. I expect at least one large AI lab to announce before the end of 2026 that it has stopped scoring take-homes altogether, because Anthropic has already told us it rewrites its test with every model and that bill only grows.

Prompted by How I changed teaching after AI managed to do all my homework assignments, thelastsoftwareengineer.substack.com.