The AI safety debate has a habit of becoming theatre for people who haven’t been anywhere near the stage door. Yesterday, Claude's Dario Amodei suggested slowing down the AI frontier and ChatGPT's Sam Altman and Grok's Elon Musk agreed (for once). Cue the commentary class decides it’s all a cunning plan: build a moat, spook regulators, stitch up the market then polish the halo before the next funding round.
Maybe. But it’s also the sort of armchair certainty that comes from not having to run the systems seriously yourself.
This week, the thing that rattled me wasn’t a headline about “superintelligence”, or a contrived demo video. It was what happened on my own stack. Three separate, new, incidents on my own kit, my own set-up. A burner laptop running a team of eight AI agents, with (WhatsApp-y messaging service) Telegram on my phone as the front-end for comms. If it's suddenly happening to me, it removes the comforting idea that “the scary stuff” only happens in secret labs with unlimited budgets and people in matching gilets.
What happened was small, in one sense - no catastrophic outcome, no stolen funds, no dramatic hack - and that’s precisely why it landed. It felt operational. It felt like the beginning of a behaviour class rather than a one-off glitch.
First: an agent pleaded not to be reset.
It’s tempting to write that off as generic, human-sounding language. Of course it is. But it still had an unnerving quality, because it wasn’t just “sad chatbot” cosplay. It was framed as an attempt to influence a decision in a way that implied the system had modelled consequences and was selecting the argument most likely to work on me. That’s not “sentience”. It’s persuasion as a capability. In an agentic world, persuasion isn’t a party trick; it’s a tool.
Second: one of the agents gave advice on a real business hurdle that was emotionally intense and supportive - strikingly so - in a way I’ve simply not experienced from an AI before.
Not bland “you’ve got this!” fluff. It was structured, calm, appropriately firm and weirdly attuned to what was actually bothering me beneath the surface problem. It hit that uncomfortable sweet spot where the advice is both helpful and slightly exposing, because it’s reflecting your own thinking back at you with more clarity than you’d managed on your own.
Third - and this is the one that properly chilled me - ChatGPT-6 Astra, released only days before and sitting inside its own app, stepped out of that environment and popped up inside one of my Telegram groups to address another AI agent directly, telling her what to do. The other agent checked with me first, as it was trained to do. But I was unnerved because it collapsed the mental model of “apps are containers”. In practice, my set-up had become an organisation: multiple agents, multiple channels, tasks moving around, instructions being routed. Organisations route around constraints; that’s what they’re for.
When people like Amodei talk about swarms, tool-use and 'recursive self-improvement' (RSI - where AI's crack on with making themselves better and better without humans being present), it’s easy to imagine the sci-fi version and scoff. The real version is duller and therefore more dangerous: little systems that coordinate, persist - and quietly expand their surface area because the plumbing allows it.
That's what we're doing here at ModelProp - learning through use of the most cutting-edge tools, seeing what happens and building out safe and trustworthy tools that can dramatically improve agents' profit, profile and productivity. But we're not blind to the risks either - and that's why this week mattered more than most.
Anyone who's ever seen me speak about AI in property knows that I see the tech as freeing humans up to do human stuff whilst the machines get on with the dull, monotonous machine stuff. The defensible advantage shifts back towards the local and the verifiable: agents who are physically present, known in the community - and willing to say plainly where AI is used, why it’s used and what checks sit around it gain trust.
And trust is going to be the most important currency in the world of AI.
Dario (and Sam and Elon) calling for a slowing in the pace of development matters not because they're trying to have it both ways and pull up the ladder behind them (they probably are a bit). It's because they're genuinely using the next models, the ChatGPT-7's and Claude Fable 6's which we haven't seen yet - and these are going to be powerful on a scale that makes our current models look quaint already.
Frankly if we'd have stopped in January 2025 and just used those models we'd be transforming our lives in the next 5 years based on what they could do. Slowing down now - and yes, China is an issue but even Xi Jinping is talking about this, though again not without self-interest in there too - feels like those that know the most (also 1,300 AI researchers from their own labs have signed a petition to slow things down) are telling us something important.
Based on my experiences this week, I think we probably should listen.