An AI tutor is already in classrooms. The useful question is how to keep it safe for children, not whether AI might one day end the world. There is a lot of talk again about AI posing an extinction risk. I think that framing is becoming a distraction from the risks the technology is already creating today. The more useful work this week is quieter: how schools should actually live with an AI tutor, where responsibility sits when children use one, and how you keep standards high when it becomes easy to produce more of everything.
What we mean by an AI tutor is set out on the AI tutor for schools page.
The risks in front of us
People can be misled by confident but wrong answers. Children can form unhealthy relationships with AI systems. Personal information can be exposed, bias can be amplified, safety systems can fail, and organisations can deploy AI without proper oversight.
We should take powerful AI seriously. But if the debate becomes dominated by whether AI might one day "kill us all", we risk taking attention away from the much less dramatic work that actually matters right now: better safeguards, testing, monitoring, transparency, governance and accountability. That work may be less sensational, but it is far more useful. Protecting people from the risks of the technology we already have is a prerequisite for preparing sensibly for what comes next.
An AI tutor needs guardrails, not a digital detox
Lorna Beegan had an interesting piece in the AFR on schools needing AI guardrails rather than a digital detox. The debate is very similar to what is happening inside enterprises right now.
The technology is already there. People are already using it. The real question is how organisations adapt around that reality. What work should still be done without AI? Where does judgement matter most? What controls are sensible? And how do you build capability without creating dependency?
Schools and enterprises are working through much the same problem, just in very different contexts.
A digital duty of care
Australia's proposed Digital Duty of Care feels particularly important for AI used by children. Online services should take reasonable steps to prevent foreseeable harm, rather than waiting for something to go wrong and responding afterwards. Importantly, the proposal reaches beyond social media to services including AI chatbots.
That principle has shaped how we have been building CurricuLLM. Safety is not one filter sitting in front of a model. We use multiple layers of protection and monitor attempts to bypass safeguards. Schools can apply different controls to different groups of students. We also look beyond individual messages. Our Safety Centre can identify patterns across conversations, such as escalation, relationship forming, boundary testing or personal disclosure, and surface them to teachers so they can decide whether a supportive check-in is needed.
The controls are only part of it. We carry out security and AI risk assessments, including Education Services Australia's ST4S and YouthSafe AI, test controls before release, monitor performance and guardrail activation in production, and review incidents through an ISO 42001-aligned process. When something does go wrong, the aim is not just to fix that individual issue. We look for the root cause, assess whether the same risk could exist elsewhere, update our testing and monitoring, and feed those lessons back into the product.
The proposed bill does not make AI risk-free for children. It does change where responsibility sits. If you build technology that children use, anticipating and reducing harm should be part of the product, not something added after launch, and not something done once.
Increase sustainability by maintaining standards
The third principle we set at the start of the year, with help from Karen Simmons, was about sustainability. AI can make it easier to do more, but that does not automatically mean the work gets better. If we are not careful, we can simply create more activity, more complexity and more things to maintain.
For us this means choosing the right tool for the problem, protecting human judgement, and being thoughtful about where automation helps and where it does not. It also means creating ways of working that are reliable, responsible and reusable, so good practice does not depend on a few people remembering how something was done last time. Standards are not there to slow us down. They are there to help us build things that last, and to make it easier for others to trust, adopt and build on what works.
Devices that remember the last 15 seconds
I have been paying close attention to Apple's new audio intelligence features on Watch. Double-press the Digital Crown and the Watch can show the previous 15 seconds of a conversation as text, helping you catch something you missed. I like the feature. It also feels like a careful first step towards a future where our devices can remember more of what happens around us.
What will be interesting is whether Apple's technical approach to privacy also creates a feeling of privacy for the people around us. There is a difference between knowing something is processed privately and instinctively feeling comfortable that a device can recall part of a conversation. Apple may be one of the companies best placed to navigate that gap. The technology is moving quickly. The social norms around it may take a little longer.