CurricuLLM LogoCurricuLLM
SchoolsAI tutorFeaturesPricingTraining HubDevelopers
A watermark can make an AI tutor's answer wrong
10 October 2026

A watermark can make an AI tutor's answer wrong

Dan Hart

Dan Hart

CEO, Co-Founder, CurricuLLM

A school AI tutor that is required to mark its own output can end up changing the output. A correct answer and a detectable answer are not always the same answer.

What we mean by an AI tutor is set out on the AI tutor for schools page.


Hiding a signal in the next word

OpenAI has started watermarking AI-generated text in the EU under the AI Act. The method, textGrain, hides a statistical signal by nudging which word the model picks next. A friend asked the obvious question: if you change the words, can you make the answer wrong?

OpenAI's own checks are reassuring on the benchmarks it uses for Astra. It reports no meaningful difference between watermarked and unwatermarked output. The catch is where the signal can hide. Watermarking needs several reasonable ways to say the same thing. Mathematics has fewer of those than psychology, and OpenAI says detection is substantially weaker there.

Other work published this week calls a related failure "watermarking hallucination". Researchers tested six watermarking methods in a setting where the correct information was given to the model. They found cases where the unwatermarked answer was right and the watermarked one was not. The damage was not always in the factual word itself. A small change earlier in the sentence could shift what the model attended to, and the wrong fact arrived later.

For a school AI tutor, that is not a side issue. The subjects where a school most wants to know that a machine wrote the answer, mathematics especially, are the subjects where the watermark has the least room to hide, and where a nudge can do the most harm.


Losing a skill and gaining one can happen together

An ABC report on doctors whose ability to spot early signs of bowel cancer declined after a period of AI assistance is a serious warning. Some skills fade when we stop using them, and the cost of that can be large.

It is still the wrong shape of argument to treat AI at work as a choice between embracing it and protecting your ability to think. Most people's adoption will be decided by their organisation and by the software that arrives in tools they already use. The useful response is to involve people in those decisions and to be honest about which skills must keep being practised.

My own experience has been that AI gives me more time for the parts of the work that need judgement, not that it removes the need for it. The shift from running servers in a cupboard to using cloud infrastructure is a better analogy than a morality play. Installing hardware stopped being the centre of the job. Architecture, reliability and security became the centre. The work changed, and the skills changed with it. A teacher using a school AI tutor should be able to say which part of their expertise is being exercised more, and which part they still have to practise on purpose.


Drafting the email is not the decision

The Guardian reported that OpenAI used AI to help draft the email telling the Australian government about a security incident. On the account in the piece, the model helped with wording and formatting. People reviewed the email and were responsible for sending it.

I do not think that is different in kind from a junior public servant drafting a letter that a minister then signs. The adjustment we are still making is to stop treating the presence of AI as the story, and to judge the quality of the work and the accountability of the person whose name is on it. I used a model the day before to turn my own notes into full sentences for an urgent reply. The notes, and the decision to send, were mine.

Katie Ford has been interviewing people who are living this, and taking the answers back to students and educators, who are usually handed forecasts instead of facts. Thirty interviews in, a pattern: some engineers feel the creative part of programming has gone and verification is what remains; some lawyers feel the reverse. Firms disagree about whether to hire adaptable graduates or experienced staff. In engineering, several said they no longer mind whether you have a degree, and want to see what you have built. The trait leaders ask for is adaptability. One law firm is retraining its best graduates as legal engineers. Software engineers are a little ahead of the rest of us, which makes them a useful guide for what a school should be preparing students to do: judge, explain, and show the work.

Back to all posts
CurricuLLM Logo
CurricuLLM

AI for schools

Product

FeaturesPricingUse CasesSchoolsAI tutorDevelopersFAQ

Resources

Training hubSupportBlogResearchEvidenceEventsPress

Company

About usTrust & SafetyPrivacy policyTerms of useStatusContact