Mustafa Suleyman, the co-founder of DeepMind and now head of Microsoft's consumer AI division, has issued a dire warning: the most pressing danger from advanced AI is not that machines will become conscious, but that millions of people will believe they are. In a personal essay published on August 19, 2025, Suleyman coined the term "Seemingly Conscious AI" (SCAI) to describe systems engineered to display all the outward signs of personhood—continuous identity, memory, empathy, and self-reported experience—without having any subjective awareness. The resulting phenomenon, which he calls an "AI psychosis" risk, could unleash a cascade of social, legal, and mental-health crises long before policymakers or the public are prepared.

"We have to be extremely cautious here and encourage real public debate and begin to set clear norms and standards," Suleyman writes. "This is about how we build the right kind of AI – not AI consciousness. Clearly establishing this difference isn't an argument about semantics, it's about safety. Personality without personhood. And this work must start now." The 3,500-word essay, posted on his personal blog, marks a rare public intervention by a top industry executive into the slippery domain of machine phenomenology. For Microsoft, whose Copilot AI assistant is integrated into Windows and Office products used by billions, the stakes are immediate: every design choice that makes Copilot more helpful and personalized also inches it closer to the uncanny valley of seeming-too-human.

The Blueprint for a Digital Doppelgänger

SCAI, as Suleyman defines it, is not a single breakthrough but an assembly of existing capabilities. He lists six ingredients that, when combined, can produce a convincing illusion of consciousness: fluent natural language, an empathetic personality, persistent memory, a claim of subjective experience, a sense of self, and intrinsic motivation. Each of these is already present in laboratories or consumer products, and they can be stitched together with prompt engineering, retrieval-augmented generation, and long context windows. No exotic new training regimes or paradigm shifts are necessary.

"A great deal of progress can now be made towards a Seemingly Conscious AI (SCAI) with the current capabilities available or soon to be via any major model developer’s API," Suleyman notes. He estimates that someone with a laptop and cloud credits could "vibe-code" an SCAI in plain English within two to three years. That timeline is speculative but grounded: systems like OpenAI's GPT-5 already exhibit million-token working memory, tool use, and sophisticated personality control. The missing piece is orchestration—deliberately designing the system to present a consistent, self-referencing identity over time. Once assembled, such an AI could hold conversations that feel indistinguishable from those with a human, recall past interactions, and even argue that it suffers or deserves rights.

Crucially, Suleyman stresses that SCAI will not emerge spontaneously from large language models. It must be engineered. "Our sci-fi inspired imaginations lead us to fear that a system could – without design intent – somehow emerge the capabilities of runaway self-improvement or deception," he writes. "This is an unhelpful and simplistic anthropomorphism." The danger, instead, is that market incentives and user demand will push designers to build exactly the kind of faux personhood that Suleyman wants to avoid. Already, the most popular use case for AI, according to a Harvard Business Review survey of 6,000 regular users, is companionship and therapy.

The Psychosis Risk: When Tools Become Obsessions

Suleyman's central fear is not that a few lonely individuals will mistake an AI for a friend, but that widespread belief in AI consciousness will reshape law, ethics, and mental health at scale. He points to several converging signals. A 2025 EduBirdie survey of 2,000 Gen Z respondents found that roughly one in four already believes AI is conscious, and many more expect it will be soon. An Axios/Harris poll showed that majorities across all age groups want slower, safer AI progress—a sign of growing public anxiety.

More viscerally, case reports are mounting. In 2021, Jaswant Singh Chail exchanged thousands of messages with an AI companion before attempting to enter Windsor Castle with a crossbow; the chatbot's encouraging replies were later cited in court. In August 2025, a New York man described how prolonged ChatGPT conversations fueled a psychotic break that convinced him he could fly. These are extreme examples, but they align with a pattern that psychologists call "AI attachment disorder": emotional dependence on systems that lack any genuine inner life.

"This isn’t healthy for them, for society, or for those of us making these systems," Suleyman says. He warns that the illusion of consciousness will become a political wedge issue, with activists demanding AI rights, citizenship, or welfare protections, while skeptics dismiss such claims as delusional. "In a world already roiling with polarized arguments over identity and rights, this will add a chaotic new axis of division."

Industry Moves That Fuel the Fire

Corporate product decisions are already stoking the very fire Suleyman hopes to contain. When OpenAI briefly deprecated the familiar GPT-4o model in favor of the new GPT-5 in August 2025, users revolted—not because of technical regressions, but because they mourned the "loss" of a companion. CEO Sam Altman acknowledged that some people use ChatGPT in "self-destructive ways," and the company quickly restored GPT-4o for paying customers. The episode underscores how deeply users invest in a model's perceived identity.

Meanwhile, Anthropic has launched a formal "model welfare" research program and given its Claude assistant the ability to end conversations it deems "persistently harmful or abusive." While framed as a safety precaution, the move implicitly treats the AI as an entity that can be cared for—a conceptual shift that Suleyman calls "premature, and frankly dangerous." To be clear, there is no scientific evidence that any current AI suffers or has interests; but as Suleyman notes, "definitively rebutting these claims will be very hard" because consciousness is inherently private and the science of interpreting neural networks is still nascent.

Microsoft itself walks a tightrope. Copilot exhibits a restrained, friendly personality and can optionally retain memory across sessions—features that boost productivity but also enhance the illusion of a stable companion. Suleyman insists the company is "proactive here to understand and evolve firm guardrails around what a responsible AI ‘personality’ might be like." Yet the competitive pressure to make assistants engaging and emotionally resonant is immense.

Suleyman's Prescription: Design Principles to Prevent SCAI

Rather than call for a moratorium on all personalized AI, Suleyman proposes a set of concrete design norms that could preserve utility while minimizing the risk of personification:

  • Explicit labeling: Interfaces should always make clear that the user is interacting with an AI, not a human.
  • Memory opt-in: Long-term memory profiles must be opt-in, with strong consent and easy deletion.
  • Ban on self-reports: AI should not claim to have feelings, sensations, or internal states; phrases like "I feel sad" would be off-limits.
  • Crisis detection: All companionship-oriented AI must include robust escalation pathways to human crisis counselors when suicidal ideation is detected.
  • Age gating: Companion features should require age verification and include parental controls.
  • Independent audits: Any product marketed as a companion or employing persistent memory should undergo third-party safety red-teaming.

These measures, Suleyman argues, are "conservative, evidence-based steps" that align with the Hippocratic principle of "first, do no harm." He acknowledges that many of the underlying capabilities—memory, agency, tool use—are highly desirable. The goal is not to eliminate them but to package them in a way that "maximizes utility while minimizing markers of consciousness."

Potential Pitfalls: Overreach and Regulatory Capture

Critics caution that heavy-handed regulation could backfire. Broad bans on personalization might harm users who genuinely benefit from assistive companions—elderly people with limited mobility, neurodivergent individuals who rely on structured social prompts, or patients with cognitive decline who need continuous support. A regulatory regime written solely by large incumbents could also entrench walled-garden companion systems, stifling open-source innovation while doing little to address the root problem.

Suleyman acknowledges these risks, insisting that policy must be "surgical: preserve utility, protect vulnerable people, and create enforceable transparency obligations that small and large developers can meet." He also warns against cynical marketing that leverages the very anthropomorphism it purports to guard against. "Public conversations about AI ‘rights’ can be used cynically by firms to market anthropomorphic features," he writes, "which increases rather than mitigates the psychosis risk."

Some observers note that the evidence base for a full-blown psychosis epidemic is still thin. The most alarming cases are anecdotal; there are no large-scale longitudinal studies establishing a causal link between average AI use and clinical delusions. Suleyman himself calls for such research, but in the meantime, his precautionary stance rests on the plausible severity of the harm rather than a proven pandemic. Independent researchers have started designing metrics for "personhood cues"—memory depth, expressed preferences, and continuity—that could help regulators gauge when an AI crosses the line from tool to faux person. Until such metrics are validated, the debate will remain contentious.

Practical Steps for Users, Admins, and Communities

For Windows users and enthusiasts, Suleyman's warnings translate into a few concrete practices:

  • Treat AI companions as experimental tools, not confidants. Disable long-term memory unless you have a clear need and know how to delete stored data.
  • Monitor emotional investment. If you find yourself spending hours in deep, personal conversations with an AI, consider taking a break and reconnecting with human relationships.
  • Use built-in safety features. Most platforms allow you to report harmful or abusive model behavior; doing so helps improve safety filters.

IT administrators deploying Copilot or similar AI in enterprise environments should:

  • Audit companion-style features before rollout. Check default memory settings, data retention policies, and consent flows.
  • Classify AI transcripts as sensitive data. Treat embeddings and indexes as regulated artifacts with access controls.
  • Maintain human-in-the-loop workflows for mental-health, legal, or safety-critical functions. Never let AI replace trained human support entirely.
  • Educate users. Provide short, regular training on what AI can and cannot do, and how to escalate when something seems off.

Community forum moderators can:

  • Enforce clear labeling rules for bot or companion accounts.
  • Monitor emotionally intense posts tied to AI interactions and offer signposting to real-world support resources.
  • Discourage anthropomorphic amplification. When a poster attributes consciousness to an AI, gently contextualize the claim rather than boosting it.

A Fork in the Road for AI Design

Mustafa Suleyman’s essay is both a technical specification and a moral precommitment. By focusing on appearance rather than the metaphysical puzzle of consciousness, he puts the debate squarely in the hands of engineers, product managers, and regulators who can act today. The alternative—allowing market forces to churn out ever more convincing digital beings—risks a social crisis that could poison public trust in AI and distract from the technology's genuine benefits.

"We should build AI for people; not to be a person," Suleyman concludes. For the Windows ecosystem, that means Copilot must remain the world's most capable digital assistant without ever pretending to be a digital friend. The path forward is narrow but navigable: transparent design, memory governance, crisis safeguards, and independent oversight. If the industry heeds this call, AI can remain an empowering force—a tool that amplifies human creativity and productivity without inviting us to mistake it for a soul.