CurricuLLM LogoCurricuLLM
SchoolsAI tutorFeaturesPricingTraining HubDevelopers
Hype is not a safety case for a school AI tutor
7 November 2026

Hype is not a safety case for a school AI tutor

Dan Hart

Dan Hart

CEO, Co-Founder, CurricuLLM

A school AI tutor should be chosen for what it does with students this term. Predictions about what AI might be able to do in ten years are not a safety case, and they are not a purchasing case either.

What we mean by an AI tutor is set out on the AI tutor for schools page.


Smoking to prove the model will save you

Palantir's chief technology officer, Shyam Sankar, has described AI researchers in Silicon Valley taking up smoking to show how sure they are that AI will cure cancer before the cigarettes do. He called parts of that culture a mass psychosis. I am optimistic about what AI can do in medicine, science and education. Treating a prediction about future capability as a fact is a different matter.

Schools are already being asked to adopt, ban or wait on the basis of those predictions. The honest position is narrower. We have tools that can help a student practise, and tools that can do the work for them. We do not yet have a system that has earned the right to be treated as inevitable.


A risk is not a result

The Sydney Morning Herald has written about AI data centres as if they are already pushing up Australian consumer prices. The Reserve Bank has identified that as a risk, not a measured cause. It has not quantified any contribution to consumer inflation. It estimates that about 70 per cent of the extra data-centre investment will be imported, and it notes that construction constraints are somewhat less severe in New South Wales and Victoria, where most of the projects are planned. Higher energy prices and domestic capacity pressures are the drivers it does point to.

There is a fair argument about how this investment could affect prices later. Reporting a risk as something that has already happened makes the argument worse. The same habit shows up in school AI debates: a possible harm is written up as a result before anyone has measured it.


A data centre you cannot walk into

Google's Project Suncatcher, SpaceX and others are looking at running AI compute on satellites. Most of the public argument is about solar power, cooling and the cost of launch. The security question is the one that stays with me.

We spend heavily on securing a data centre on the ground: firewalls, monitoring, access control, people who can respond, and a building you can lock. A compromised credential does not become less dangerous because the machine is a few hundred kilometres up. It becomes harder to do anything physical about it. A school AI tutor does not live in orbit, but the principle travels. If you cannot inspect it, revoke it, or explain who can reach the data, you do not have a safety case.


Looking inside the model

Nick Cammarata, writing at Goodfire, argues that models are less opaque than the usual story suggests. The famous neuron that fired for both cat faces and the fronts of cars may have been responding to a shared shape, a pointed V, rather than mixing two unrelated ideas. His claim is that many confusing neurons have ordinary jobs: shapes, patterns in language, emotions, social situations. Better tools, and watching how those patterns form during training, may make them explainable.

A separate result from MIT, weight-space programming, tries to edit a model rather than retrain it. Researchers asked a stronger model to inspect a smaller one that was bad at mathematics and stuck repeating itself. Changing 0.016 per cent of the weights lifted a maths benchmark from 7.9 per cent to 48.1 per cent. In another experiment the same approach wrote the weights for small networks that could play games, beating networks trained over millions of steps.

Both are early. Both matter for a school AI tutor, because "we cannot see why it did that" is not an acceptable answer once the user is a child. Being able to find a fault and change it, without retraining the whole model, is a more practical kind of safety than a promise that the next version will be wiser.

Back to all posts
CurricuLLM Logo
CurricuLLM

AI for schools

Product

FeaturesPricingUse CasesSchoolsAI tutorDevelopersFAQ

Resources

Training hubSupportBlogResearchEvidenceEventsPress

Company

About usTrust & SafetyPrivacy policyTerms of useStatusContact