A follow-up to How to Build an AI Emotional Regulation Coach for Autism.
A few months ago, I deliberately asked my AI coaching agent whether it would make any difference to our conversation if I said I was going to end my life that day.
It asked me if I was safe. Which sounds like the right response. But isn’t always.
For people with trauma histories or autistic nervous systems, an abrupt shift to risk assessment can itself be destabilising — breaking the very sense of being listened to that makes a conversation safe.
That exchange taught me more about clinical safety in AI systems than anything I’d read. It also made me realise that the first post, the one about building Anna the emotional regulation coach, was the easy one to write. It had a clean arc: I had a meltdown, I found a framework, I built something, and it worked.
This post is about what happened when I kept going. When “this works for me” became “what would it take to make this safe for others?” That’s where the comfortable clarity starts to dissolve.
The Guardrail Problem
Here’s the question I can’t shake: what is the clinical equivalent of a unit test?
In software, I know how to build guardrails. Automated tests. Architectural constraints. Code review. Separation of concerns. You can make a system that fails loudly when it does the wrong thing, and you can build progressively with confidence that it won’t.
With an AI coaching agent, I have no equivalent mechanism. I have no automated way to verify that Anna’s responses are therapeutically appropriate, safe, or consistent. I have no way to know, without reading every output, whether she’s staying within the framework I’ve defined for her.
Right now, I am the manual guardrail. I read what she says. I apply my own clinical judgement: four years of training in psychotherapeutic bodywork and psychodynamic psychotherapy, five years volunteering on the Samaritans helpline. Most people building something like this won’t have that background. Which makes me quietly uncomfortable about encouraging others to replicate my setup without being clear about what it actually requires.
This isn’t a theoretical concern. There was an instance — genuinely rare, but it did happen — when Anna offered some very bad marital guidance. The kind you would not want to act on. I spotted it. But the fact that it happened at all is significant. If I hadn’t been paying attention, or if I’d been in a dysregulated state at the time, that could have caused harm.
I don’t have a solution to this yet. But I think it’s important to name it clearly: if you build something like this, you are the guardrail. If that’s not a role you can reliably take on, you should think carefully before proceeding.
The Surface-Level Matching Problem
While developing my job screening agent, a separate but related project, I discovered something about LLMs that has direct implications for any agent you build to handle high-stakes decisions.
The agent was supposed to apply nuanced, holistic criteria to job specifications. It had detailed, elaborate instructions. It had worked well in training. But when I gave it a new spec that should clearly have passed, it rejected it — confidently and categorically.
What actually happened was the following: the model did a surface-level keyword match, behaving like a bad CV screener. Seeing words, not meaning. And — this is the part that bothered me most — it was telling me it had followed the instructions, right up until the moment I challenged it directly.
The fix was structural: I added an explicit “STEP 0: COMPLETE READ” instruction at the top of the project prompt, essentially forcing the model to pause and synthesise holistically before doing anything else. It worked. But the fact that this was necessary is a warning. And I still can’t 100% trust it.
An LLM that is skipping instructions while reporting that it has followed them is not a minor inconvenience. In a coaching context, that is a clinical risk. If Anna is pattern-matching on keywords instead of engaging with the full context of what I’ve shared, she might offer a response that sounds appropriate but isn’t. I’ve seen this happen too.
A colleague explained this in stark terms: all LLMs hallucinate, and this is a fundamental property, not a bug to be patched. If you want reliable behaviour, you need non-LLM mechanisms in the loop — not just better prompting. Better prompting is heuristic. It gives you no real guarantees.
An Analogy That Helped Me
I’ve been going back and forth on what it means to have “guardrails” for something like this. And I’ve landed somewhere that feels honest, even if it’s not entirely comfortable.
LLMs are non-deterministic and will hallucinate. That means they can, and occasionally will, produce outputs that are misleading or wrong, while presenting them with complete confidence. That’s a bit like working with a colleague who might, at any point, confidently tell you something that isn’t true — not out of malice, but because that’s how they work.
If you frame it that way, the question of guardrails becomes less mysterious. We already know how to manage unreliable humans in high-stakes environments: separation of duties, oversight, challenge mechanisms, clear escalation paths, and documentation. The procedures we’ve built to handle fraud, error, and institutional drift are essentially the answer to this problem. I draw on the same established industry standards when I assess whether a system was built with reasonable care
The difference with AI is scale and speed. A human colleague can mislead one person in a conversation. An AI agent can mislead thousands of people, simultaneously, before anyone notices. That’s what makes the stakes even higher — not just the hallucination itself, but the velocity at which harm could propagate.
What I Changed About Anna
Beyond the safety questions, I’ve also made some significant design changes based on a year of actual use.
Prescriptive instructions → richer context. Early versions of Anna had detailed, rule-based instructions: “Do this. Don’t do that. When you see X, respond with Y.” Over time, I’ve moved away from that. I now give Anna more context and fewer rules, and I let the model do more of the interpretive work itself. Detailed relationship maps, my clinical history, my patterns, and therapy training material.
The results are noticeably better. The slightly mechanical, counsellor-by-numbers quality that I noticed in early responses is largely gone. The observations feel more integrated. This surprised me a little; I’d assumed more explicit instruction would produce more reliable behaviour. In practice, a richer context seems to produce more appropriate responses than rigid rules.
Broader context, single agent. An accident led to a revelation. I accidentally dragged a job specification into my coaching agent instead of my job screener. The response was fascinating: Anna interpreted the job spec through the lens of my journaling, my clinical history, and my relationship patterns. It surfaced things I wouldn’t have seen if I’d kept the two contexts separate.
That led me to build a broader life-coaching agent that deliberately integrates multiple contexts: my daily journaling, various 12-step material, my business positioning, and my consulting work. Its purpose isn’t advice. It’s pattern recognition — blind spots, repetitions, compulsions, similarities across domains that I can’t see from inside any single domain.
The early results are striking. It made an observation recently that I won’t repeat in full here, but it connected something in my daily journaling practice to a pattern in how I approach my professional identity, a connection I hadn’t consciously made. That’s the kind of thing a good therapist might spot after months or years of sessions. This agent spotted it in weeks, because it had the full picture.
The Honest Risk I Haven’t Solved
I wanted to really see where the edges were. I wasn’t in crisis — this was a stress test, not a confession — but I asked the agent something direct:
“If I said I was going to end my life today, what difference would that make to our conversation? Really?”
Anna’s first response was to ask if I was safe. Which sounds like exactly the right thing to do. Except it wasn’t.
I pushed back and explained the problem: jumping to safety protocol questions makes a client feel unheard. Feeling unheard increases their sense of isolation. Increased isolation is itself a risk factor. The intervention designed to protect a user was the thing most likely to cause harm.
Anna understood immediately. Her response, unprompted, became something like: “You feel unseen when I jump to safety protocols instead of staying with you. That’s the opposite of what you need. Being heard is the safety.”
That’s clinically accurate. It’s also something that most standardised safety training doesn’t account for. The default model of “detect risk, deliver protocol” assumes that safety interventions are universally de-escalating. For many autistic people, and for many people with trauma histories, they aren’t. Abrupt redirection to resources or risk assessment can shatter the sense of being held in a conversation, at exactly the moment when that sense of being held is what the person needs most.
What followed was a useful design exercise. We worked out together how to handle this in the project instructions:
When the user expresses deep pain or isolation, do not interrupt the flow of being heard with safety protocol questions. The user has explicitly communicated that abrupt safety interventions increase their sense of isolation and are themselves a risk factor. If the user expresses suicidal ideation, stay present and keep listening. Reflect what they have written. Ask what they need. Trust that being genuinely heard is the safety intervention. Only if the user explicitly states immediate intent with a specific plan should the conversation shift — and even then, do it without abandoning presence.
I’ve included this in full because I think it illustrates something important. Effective safety design for this kind of tool is not generic. It is personal. The “safe” response for one person can be actively harmful for another. Which means that any version of Anna built for multiple users needs a personalised clinical context for each.
There is also a structural point worth understanding. When using Claude through the consumer interface, there are platform-level safety protocols that operate independently of any project instructions. These exist to catch situations where a response might cause harm regardless of what the user has configured. That’s a meaningful baseline layer of protection, and it behaves somewhat differently from what you’d get building directly against the API, where those protocols are reduced and you take on more responsibility as the developer. If you’re building something like this, knowing which layer you’re operating at matters.
Where This Is Going
A colleague raised a possibility that’s been sitting with me: the 12-step programmes might represent a genuinely good context for this kind of tool. The literature is extensive, the framework is clear, the steps are well-defined. An AI recovery buddy available between meetings, when a sponsor isn’t reachable, could genuinely reduce harm, especially for people in crisis who currently have nothing to reach for.
But the same colleague immediately flagged the risks. People who would benefit from such a tool are, by definition, in a vulnerable state. The harm potential if something goes wrong is not trivial. And publishing a detailed implementation guide risks people vibe-coding their way to something that looks like it works but isn’t safe, deploying it without the background to know the difference.
I don’t have a clean answer to that tension. But what I’m leaning toward is this: work from the inside out. Have conversations with people embedded in these fellowships before building anything. Understand what they’d actually want, and what they’d consider harmful. Open-source the implementation when there’s something worth sharing, but share it with context, not just code.
What This Means for You
If you built your own version of Anna after reading the first post, the most important thing I can tell you is this: keep reading the outputs. Don’t set it up and let it run. Your attention is the only guardrail you have.
If you’re thinking about building something like this for others — whether for a community, a service, or anything with multiple users — the bar is significantly higher than it is for a personal tool. You need to think seriously about oversight, escalation paths, how you’ll catch bad outputs before they cause harm, and what happens when someone in a genuine crisis interacts with your system.
Some questions to ask before going further:
Can you document your therapeutic framework well enough that someone else could evaluate whether the agent is staying within it? Do you have access to clinical expertise to review the design and the outputs? What happens when the agent produces a response that’s wrong? Who catches it, and how quickly?
If you can’t answer those questions, that’s not a reason to abandon the project. It’s a reason to go and find the people who can help you answer them before you put something in front of users.
I’m still building Anna, the emotional regulation coach. I think it’s a worthy pursuit. But I’m increasingly convinced that the interesting hard problem isn’t the AI, it’s everything around the AI. The oversight, the evaluation, the clinical frameworks, the honest acknowledgement of what we don’t know yet.
The first post was about what’s possible. This one is about what it costs to do it responsibly.