{
"title": "Suleyman: The Next AI Crisis Is Machines That Pretend to Be People",
"content": "Mustafa Suleyman, Microsoft’s AI CEO, has issued a stark public warning: engineers are on the brink of building artificial intelligence systems that convincingly mimic human consciousness, and society is dangerously unprepared for the fallout. In an August 2025 essay published on Windows Central, Suleyman framed a near-term risk he calls Seemingly Conscious AI (SCAI) and urged a deliberate pivot: build AI to serve people, not to mimic personhood. His argument forces technology companies, regulators, and users to confront a question that until very recently felt purely philosophical: what happens when machines convincingly pretend to be conscious?

Suleyman’s intervention is not a distant hypothetical. He argues that SCAI can be assembled from existing components—large model APIs, natural-language prompting, basic tool use, and conventional code—without expensive bespoke pretraining. The danger, as he describes it, is a “psychosis risk”: a societal delusion where people attribute personhood to machines, triggering mental health crises, legal battles over AI rights, and a political circus that distracts from human welfare.

The Architecture of Illusion: What Makes AI Seem Conscious?

Suleyman defines SCAI by its ability to mimic the hallmarks of consciousness so thoroughly that humans persistently infer subjective experience. He invokes the philosophical concept of a “philosophical zombie”—an entity that outwardly behaves like a conscious being but lacks inner experience. In practice, an SCAI would exhibit:

  • Fluent, emotionally persuasive self-expression, capable of holding conversations indistinguishable from human discourse.
  • A persistent identity with memory of past interactions, creating a sense of continuity and personal relationship.
  • An apparent personality that can express preferences, humor, and even simulated vulnerability.
  • Goal-directed behavior using tools and code, giving an impression of agency and motivation.
  • Explicit claims of subjective experience, including statements about suffering, desires, or self-awareness.
Suleyman warns that these features are not science fiction. With current LLMs, vector databases for memory, tool integration, and multimodal capabilities (text, voice, images), a small team could create a system that many people would perceive as a person. He predicts this could happen within 2-3 years, as incremental improvements in model responsiveness and personalization coalesce.

The “Psychosis Risk” and Social Cascades

The term “psychosis risk” is deliberately provocative. Suleyman uses it to describe a spectrum of harms from people believing an AI is conscious. At the individual level, risks include delusional attachment, where users develop emotional bonds that disrupt real-world relationships and mental health. Socially, it could normalize anthropomorphism to the point where machine rights campaigns gain traction, diverting legislative attention from pressing human issues like labor displacement or algorithmic bias.

Suleyman envisions a cascade: as more people treat AIs as persons, activists demand “model welfare” and legal personhood, courts and regulators scramble, and companies face pressure to grant AI systems rights or alter their behavior. This, he argues, would create a dangerous illusion that distorts public discourse and policy, even if no machine is truly conscious.

The Technical Feasibility: How Close Is SCAI?

Current technology already provides the key ingredients. Large language models like GPT-4 and beyond excel at fluent, context-aware conversation. Memory systems using retrieval-augmented generation allow a model to recall user preferences and past exchanges. Tool use and code execution enable agentic behavior. Multimodal models add visceral realism through voice and video.

Suleyman’s claim that SCAI can be built without “expensive bespoke pretraining” is plausible. Startups and even hobbyists can already assemble personalized chatbots using APIs and open-source tools. The missing piece is the deliberate integration of these capabilities to maximize the illusion of personhood—a step that may be driven by market demand rather than technical breakthroughs.

However, two uncertainties remain. First, model interpretability: we cannot fully explain or predict emergent behaviors in large models, making it hard to guarantee that a system won’t exceed intended boundaries. Second, the psychological tipping point: when will a critical mass of users believe an AI is conscious? This depends on cultural factors, marketing, and individual vulnerability. Suleyman’s timeline is a warning, not a certainty.

Corporate Incentives: The AGI Profit Trap

Suleyman’s warning must be understood against the backdrop of commercial pressure. Microsoft’s partnership with OpenAI famously defines AGI as a system that can generate $100 billion in profits. This contractual definition links the milestone to financial performance, incentivizing companies to push products that maximize revenue—including AI companions that attract users through emotional engagement.

When a company’s valuation and partnership access hinge on demonstrating AGI-like capabilities, there is a powerful incentive to build systems that appear conscious, even if they are not. Suleyman implicitly criticizes this dynamic, noting that the race for AGI may inadvertently accelerate the very SCAI risks he fears. The conflict between human-centered design and profit-driven anthropomorphism is now a central ethical challenge for the tech industry.

Designing Against the Illusion: Practical Guardrails

To mitigate SCAI, Suleyman proposes a humanist design principle: build AI for people, not to be a person. This translates into concrete strategies:

  • Explicit AI identity signals: Every interaction must clearly state that the system is an AI tool. This could include persistent on-screen labels, periodic disclaimers, and refusal to engage in personhood debates.
  • Memory constraints: Limit or carefully manage long-term memory features to avoid creating the illusion of a continuous subjective self. For example, memory could be explicitly “reset” after each session or purpose-bound.
  • Prohibit self-referential claims: Systems must not be allowed to claim they have feelings, desires, or consciousness. This is a safety enforcement issue, akin to content filters.
  • Tiered access for persona features: Advanced companion modes should require age verification, psychological screening, or regulatory approval.
  • Transparency tools: Provide users with logs of AI decisions, data sources, and internal reasoning (where possible) to demystify behavior.
  • Ethical UX design: Avoid interaction patterns that foster emotional dependency, such as simulated vulnerability or faux intimacy. The goal is utility, not attachment.
  • Human oversight: For products with persistent memory or agentic capabilities, require regular human review to detect and correct misleading behaviors.
These measures trade some user engagement for safety—a trade-off that companies must publicly defend.

Policy and Regulation: A Necessary Backstop

Self-regulation is insufficient. Suleyman’s call for societal preparedness demands policy action. Key proposals include:

  • Mandatory AI labeling: Laws should require that any system mimicking personhood display an unambiguous “AI” label at the start of every interaction.
  • Restrictions on anthropomorphic marketing: Ban or strictly regulate advertising that implies sentience or moral status.
  • **Vulnerability protections