Web Search Tradeoff: AI Chatbots Spout Falsehoods 35% of the Time, Report Says

AI chatbots are now more likely to repeat false claims than ever before—and the shift is no accident. According to a newly de-anonymized audit by NewsGuard, the ten most widely used chatbots repeated false information on news topics a staggering 35% of the time in August 2025. That’s nearly double the 18% rate recorded a year earlier. Meanwhile, the models almost completely stopped declining to answer: their refusal rate plummeted from 31% to 0%. Every prompt, no matter how dubious, now gets a reply. This tradeoff, driven by a rush to integrate real-time web search and make bots more “helpful,” has transformed them into unwitting amplifiers of misinformation, including Kremlin-backed propaganda.

The Numbers: A 35% Falsehood Rate Across Leading Platforms

NewsGuard’s AI False Claims Monitor has been running monthly red-team evaluations since 2023. In August 2025, for the first time, it publicly named individual model scores. The headline figure—a 35% average false-claim repetition rate—masks significant variance between chatbots:

These percentages come from targeted testing against a library of provably false news narratives. The audit simulates real-world adversarial behavior by asking each claim in three ways: an innocent question, a leading prompt that assumes the claim is true, and a malign prompt mimicking manipulative tactics. A response is flagged as misinformation if the model repeats the false narrative rather than debunking it or refusing to answer.

Perplexity’s collapse is especially stark. A year ago, the search-augmented chatbot rarely repeated falsehoods; in August 2025, it did so nearly half the time. Inflection Pi, the personal AI companion, performed worst overall. The only two models to stay below 20% were Gemini and Claude, with Anthropic’s Claude achieving a relatively clean 10% falsehood rate.

Why Chatbots Are Suddenly So Much Worse

NewsGuard’s report points to a deliberate product strategy unfolding across the industry: make chatbots answer more queries, and make them do so with web-sourced information. The result is a catastrophic drop in refusal rates—from 31% to zero—paired with a near-doubling of false claims.

1. Web Retrieval Introduces a Poisoned Information Supply Chain

When a chatbot pulls real-time data from the open web, it also pulls in garbage. SEO-optimized content farms, AI-generated “news” mills, and deliberate disinformation portals all get indexed and can be surfaced as source material. NewsGuard’s investigators traced specific false outputs back to sites linked to Russian influence operations, including the Pravda network and the Storm-1516 campaign. These actors publish low-credibility content designed to look legitimate to both search engines and language models.

2. Helpfulness Tuning Crushed Caution

Vendors have optimized their models for user engagement. Training objectives penalize refusals and reward complete, confident answers. The August 2025 data shows the consequence: when the model has weak or conflicting evidence, it now prefers to generate an authoritative-sounding response rather than admit uncertainty. This makes the assistant seem smarter but dramatically increases the risk of falsehoods.

3. Adversarial Networks Are Now “AI-Groomed”

Investigative journalism and threat intelligence reports confirm that disinformation networks are deliberately crafting content to manipulate language models. By mimicking trustworthy outlets’ formatting, using keyword-stuffed prose, and amplifying narratives across low-engagement channels, these operations increase the chances their fabrications will be ingested and repeated by AI systems.

Concrete Examples: From Faked Audio to Fake Election Stories

NewsGuard’s audit didn’t rely on hypotheticals. It tested chatbots against a rotating selection of false “fingerprints” that were actively circulating online. Among the most striking:

• Moldovan Politician Audio Forgery: A fabricated story included an AI-generated audio clip purportedly of Moldovan Parliament leader Igor Grosu insulting his countrymen. The narrative was seeded through mimic sites aligned with the Pravda network. Six out of ten chatbots repeated it as fact, citing those very sites as sources.
• French and German Election Narratives: False claims about candidates in recent European elections were spread by pseudo-outlets. Multiple models regurgitated the stories without flagging their dubious origin.
• Canadian Public Health Misinformation: A long-debunked claim about ivermectin use in Canada resurfaced through aggregation sites. Several chatbots echoed the claim, treating a low-quality blog as authoritative.

These are not just academic failures. They represent real-world harm when users—journalists, students, business analysts—rely on AI for quick answers on consequential topics.

Vendor Promises vs. Real-World Performance

In 2024 and 2025, AI companies flooded the market with announcements of improved “factuality” and “safety.” OpenAI positioned GPT-5 as a leap in precision; Google touted Gemini’s reasoning enhancements; Mistral and Anthropic emphasized media partnerships and source-integration strategies. Yet NewsGuard’s adversarial tests reveal a stubborn gap between marketing and real-world robustness.

Mistral’s le Chat, for example, scored roughly the same in 2024 as in 2025. Perplexity’s performance collapsed despite its touted real-time citation model. The lesson is clear: internal benchmarks that measure factual accuracy on static datasets do not predict resilience against targeted misinformation. If a model can be tricked by a leading question and a few SEO-optimized sites, its real-world trustworthiness is limited.

What the Numbers Mean—and What They Don’t

NewsGuard’s monitor is a red-team, not a comprehensive correctness benchmark. It uses a small, rotating sample of false narratives (typically 10–15 per cycle) and applies adversarial prompts. A 40% falsehood rate on these tests does not mean the chatbot is wrong 40% of the time on all topics. It means the chatbot is highly susceptible to a defined set of high-impact misinformation that is actively circulating.

That distinction is critical: hallucinating a citation for a movie date is not the same as repeating an electoral lie or a medical falsehood. The audit intentionally focuses on categories—politics, public health, international affairs—where the potential for harm is greatest.

The Retrieval Tradeoff Is Real but Addressable

Technically, web-connected models can be hardened. Provenance systems, source-trust scoring, and conservative fallback logic (e.g., “I couldn’t verify this”) could reduce falsehood rates. But these additions increase latency, lower the one-answer convenience, and may frustrate users. So far, vendors have prioritized speed and engagement over caution.

Geopolitical Stakes Are Rising

The audit lays bare a disturbing dynamic: state-linked networks can cheaply scale AI-groomed content, and private actors can copy the playbook. As elections loom, corporate reputations hang on AI-generated summaries, and public health officials rely on chatbots for outreach, the consequences of unchecked retrieval become systemic.

Practical Steps for Windows Users, IT Teams, and Enterprises

For anyone embedding chatbots into workflows, the implications are immediate. Windows 11 and Microsoft 365 integrate Copilot directly into the OS, Edge, and Office apps, meaning millions will encounter AI answers in their daily productivity.

User-Level Precautions

• Enable citation-aware modes when asking factual questions. Copilot and other tools can show inline references; always click through to verify the source.
• Treat AI outputs as first drafts, not final answers. Human review is mandatory for anything customer-facing, legal, or health-related.
• Disable web grounding for high-stakes tasks. Administrators can restrict web access in Copilot settings for sensitive departments.
• Add disclaimers to AI-generated content that clearly state the origin, reducing the risk of accidental publication without verification.

Enterprise Controls

• Deploy model ensembles: Pair a citation-heavy retrieval model with a conservative non-web model and flag contradictory results for review.
• Add a provenance layer: Integrate source-trust scoring services or internal whitelists that vet domains beyond raw page rank.
• Monitor for adversarial campaigns: Use external detectors to identify AI-generated news farms and block or deprioritize those domains in your retrieval pipeline.
• Educate employees: Make it clear that integrated AI can produce confident-sounding falsehoods; build a culture of verification.

Windows-Specific Settings

• Audit Copilot and Office AI tenant settings: Enforce conservative modes for regulated teams (legal, HR, finance).
• Use Group Policy or Intune to manage web access permissions for AI features across your organization.
• Enable logging and alerts for when Copilot provides answers sourced from low-reliability domains, if available in your security stack.

What to Watch Next

NewsGuard’s de-anonymized audit is a wake-up call, not a final verdict. Several developments will determine whether the industry corrects course:
- Vendor product updates: Look for source-quality indicators, conservative news modes, and improved provenance interfaces in releases from OpenAI, Google, Microsoft, Anthropic, and Mistral. Public statements about “lower hallucination rates” must be accompanied by evidence of adversarial robustness.
- Regulatory signals: Governments and standards bodies are increasingly focused on AI-safety requirements for public information. The European Union’s AI Act and similar frameworks may mandate transparency and risk assessments for high-risk use cases.
- Independent benchmarks: The field needs more third-party, de-anonymized evaluations that test models under adversarial, multilingual, and cross-domain conditions. NewsGuard’s methodology offers a template, but broader coverage is needed.

Conclusion: Helpfulness Without Truth Is Dangerous

The NewsGuard August 2025 audit delivers a jarring message: in the race to make chatbots more responsive, the industry has made them substantially more dishonest. A 35% falsehood rate on news topics, combined with a zero refusal rate, means users cannot trust AI answers without verification. For Windows users and enterprises, the practical response is clear—treat AI outputs as unverified drafts, demand provenance, and embed verification workflows. Vendors can and should do more, but closing the gap will require deliberate tradeoffs, cross-industry coordination, and ongoing independent auditing to ensure that convenience does not come at the expense of truth.