When outraged Filipinos turned to an AI-powered chatbot to verify a viral photograph of former lawmaker Elizaldy Co in Portugal, the tool failed to detect it was fabricated—even though it had generated the image itself. This single misclassification, repeated across platforms and amplified by social sharing, crystallizes a broader and growing problem: today's multimodal chatbots are excellent at mimicking reality, but they are not yet reliable verifiers of it. The episode exposes a structural blind spot in how large AI systems treat visual evidence, with critical implications for newsrooms, platforms, and everyday Windows users who increasingly treat chatbots like Copilot as a first stop for verification.

The Anatomy of a Viral AI Failure

The case of Elizaldy Co represents a perfect storm of AI verification failure. According to AFP's investigation, the fabricated image showing Co in Portugal was created "for fun" using Nano Banana, Gemini's AI image generator, by a middle-aged web developer in the Philippines. When online sleuths tracking the former lawmaker asked Google's AI mode whether the image was real, it incorrectly said it was authentic. The developer told AFP he was "shocked at how many shares it got" and eventually edited his post to add "AI generated" to stop the spread—but not before the image garnered over a million views across social media.

This isn't an isolated incident. During last month's deadly protests over benefits for senior officials in Pakistan-administered Kashmir, social media users shared a fabricated image purportedly showing men marching with flags and torches. An AFP analysis found it was created using Google's Gemini AI model, yet both Gemini and Microsoft's Copilot falsely identified it as a genuine image of the protest. These cases demonstrate how AI-generated photos flooding social platforms can look virtually identical to real imagery, creating a verification crisis that current AI tools are ill-equipped to handle.

Why Multimodal Chatbots Fail at Image Verification

The Fundamental Optimization Mismatch

Modern multimodal assistants combine three core components: a visual encoder that converts pixels into internal representations, retrieval layers that fetch supporting text or images, and a large language model (LLM) that reasons and formulates answers. Critically, most of these components were trained to produce plausible language or images—to predict what looks and sounds right—not to prove provenance or surface forensic traces.

Alon Yamin, chief executive of AI content detection platform Copyleaks, explains: "These models are trained primarily on language patterns and lack the specialized visual understanding needed to accurately identify AI-generated or manipulated imagery. With AI chatbots, even when an image originates from a similar generative model, the chatbot often provides inconsistent or overly generalized assessments, making them unreliable for tasks like fact-checking or verifying authenticity."

This architecture creates several specific failure points:

  • Generators are optimized for plausibility and photorealism, not detection
  • Vision encoders are tuned for description (what's in this image) not for forensics (was this image generated or manipulated)
  • LLMs are tuned to be helpful and conversational, with product design incentives often rewarding completeness over cautious refusals

Training Data and Label Gaps

Many models are trained on massive web scrapes that mix genuine photographs and synthetic images without consistent provenance labels. Without explicit supervision that separates "synthetic" from "authentic" during training, the model's internal distribution conflates both. This makes downstream detection a weak signal unless the system is explicitly taught to search for and prioritize forensic traces.

Rossine Fallorina from the nonprofit Sigla Research Center notes: "This inability to correctly identify AI images stems from the fact that they (AI models) are programmed only to mimic well. In a sense, they can only generate things to resemble. They cannot ascertain whether the resemblance is actually distinguishable from reality."

UI and Product Incentives Favor Answers Over Accuracy

Product teams favor assistants that answer rather than decline. In real-world UIs, users prefer a shortcut: they submit an image and expect a fast verdict. Assistants calibrated toward user satisfaction therefore lean toward plausible assessments and may under-report doubt—a dangerous behavior for verification tasks. Independent audits have found refusal rates in news Q&A to be vanishingly small, reinforcing the observation that assistants seldom default to "I don't know."

Systematic Evidence of the Problem

Independent Audits Reveal Widespread Issues

A large, coordinated audit by the European Broadcasting Union (EBU) and the BBC found that roughly 45% of assistant replies to news queries contained at least one significant problem—errors that could materially mislead a user. The study reviewed thousands of responses across 14 languages and multiple products, flagging sourcing, temporal staleness, and invented details as recurrent failure modes.

Earlier this year, Columbia University's Tow Center for Digital Journalism tested the ability of seven AI chatbots—including ChatGPT, Perplexity, Grok, and Gemini—to verify 10 images from photojournalists of news events. All seven models failed to correctly identify the provenance of the photos. The Tow Center concluded that while assistants can aid investigatory leads (geolocation clues, scene elements), they cannot replace the discipline and skepticism of trained human verifiers.

The Human Fact-Checking Retreat

At the same time that AI tools are sitting front and center in user workflows, several major platforms have been restructuring or scaling back human fact-checking programs. Meta announced earlier this year it was ending its third-party fact-checking program in the United States, turning over the task of debunking falsehoods to ordinary users under a model known as "Community Notes." These policy shifts transfer greater responsibility to algorithmic systems or to crowdsourced community moderation models—neither of which reliably substitutes for trained verification teams.

Human fact-checking has long been a flashpoint in hyperpolarized societies, where conservative advocates accuse professional fact-checkers of liberal bias, a charge they reject. AFP currently works in 26 languages with Meta's fact-checking program, including in Asia, Latin America, and the European Union.

Technical Limitations and Partial Solutions

Why Pixel-Level Forensics Remain Difficult

Forensic detectors look for narrow statistical fingerprints: resampling artifacts, upscaling traces, compression residues, or model-signature patterns embedded at generation time. Such detectors require targeted supervised training on labeled synthetic content and specialized architectures. General vision encoders—designed to summarize, caption, or identify objects—are not optimized for those signals; nor do they typically have access to provenance metadata. That's why a general assistant, without explicit forensic supervision, will often return the plausibility score rather than a forensically grounded verdict.

Watermarks, Metadata Standards, and Their Limitations

Commercial and standards-level responses are beginning to appear. One practical mitigation is embedded provenance: Google's SynthID (an imperceptible watermark) and C2PA content credentials embed metadata and digital signals that can later be checked for origin. Google has recently integrated SynthID checks into the Gemini app so users can upload images and ask whether they were generated by Google AI; company statements claim billions of items have been watermarked since 2023.

However, these schemes face significant limitations:

  • Watermarks only help if they exist and remain intact
  • Third-party or adversarial generators will not carry a vendor's watermark
  • Metadata can be removed, altered, or lost during reposting and compression

These constraints mean that watermarking and C2PA are complementary tools, not comprehensive solutions. They provide a promising, but partial, fix—useful for images generated within a given vendor's pipeline but not for a heterogeneous internet of generative tools.

Risks for Windows Users and Organizations

Rapid Amplification of Misinformation

A single misclassified image returned by an assistant can be copy-pasted across social platforms, becoming "evidence" before any human review happens. This accelerant is especially dangerous in breaking or political stories. The Windows ecosystem, with Copilot integrated directly into the operating system and Edge browser, creates particular risks as users may treat these built-in tools as authoritative sources.

Enterprise Reputational and Security Risks

Organizations that rely on assistants for triage or communications risk inadvertently propagating false claims if replies are accepted without verification. Legal, PR, and compliance teams should treat AI outputs as tentative, not definitive. For IT teams managing enterprise deployments of AI tools, this creates new governance challenges around content verification and source validation.

Social and Political Escalation

Mislabelled protest imagery or doctored official statements can inflame real-world tensions. Where images are used as triggers for protests, arrests, or policy debates, the stakes are not merely reputational but can have tangible social consequences. The Kashmir protest image misclassification demonstrates how AI verification failures can occur in politically sensitive contexts with real-world implications.

Practical Guidance for Safe Verification Practices

Treat AI Output as Starting Points, Not Final Verdicts

Always cross-check with at least two independent sources (primary reporting, archival databases, or multiple vendor checks). For news Q&A, prefer journalistic toolkits and public-service resources over general-purpose AI assistants.

Implement Multi-Layered Image Verification

For images specifically:

  • Run reverse-image searches across multiple engines (Google Images, TinEye, Yandex)
  • Inspect metadata where available (EXIF, C2PA credentials)
  • Use specialized forensic tools trained to detect generation artifacts
  • If the assistant claims provenance, ask for evidence: exact URLs, time stamps, metadata excerpts; do not accept unaudited assertions

Maintain Audit Trails and Practice Responsible Sharing

Keep an audit trail: record queries, screenshots, and the assistant's full answer. For enterprise environments, log model versions and timestamps. When uncertain, refuse amplification: avoid sharing images that lack verifiable provenance until human review is complete.

What Vendors and Windows Integrators Should Do

Improve Transparency and User Interface Design

Vendors should expose provenance and retrieval signals in the UI, making it easy to view the exact evidence used to generate an answer rather than presenting reconstructions alone. They should offer conservative "verified-mode" defaults for public-interest queries that prioritize provenance and refusal over speculative answers.

Integrate Provenance Checks and Human Oversight

For Windows ecosystem partners, these design choices translate into safer default integrations for Copilot and search-centric experiences within Edge and the OS. Specifically:

  • Integrate and surface watermark/C2PA checks (SynthID and similar) where possible, while making the limits of those checks explicit to users
  • Implement human-in-the-loop gates for high-risk topics (public safety, election reporting, law enforcement)
  • Provide auditable logs to enable post-hoc verification and redress when mistakes occur

The Future of AI Verification

Research Directions and Technical Solutions

Longer-term fixes require substantial research investment:

  • Invest in forensic supervision: Build diverse, high-quality datasets of synthetic and manipulated images with provenance labels to train detectors that can generalize across generator families
  • Improve cross-model detection: Collaborate on open standards for embedding robust provenance (C2PA, SynthID-style signals), and fund third-party verification services
  • Expand transparency: Researchers need access to model retrieval logs and grounding materials under controlled conditions to audit and reduce systematic biases

The Irreplaceable Role of Human Expertise

Researchers emphasize that AI models can be useful to professional fact-checkers, helping to quickly geolocate images and spot visual clues to establish authenticity. But they caution that they cannot replace the work of trained human fact-checkers. Experienced fact-checkers bring contextual, archival, linguistic, and cultural expertise; they can interpret ambiguous visual cues, consult primary sources, and apply rigorous provenance checks (reverse image searches, metadata analysis, geolocation).

Fallorina summarizes the challenge: "We can't rely on AI tools to combat AI in the long run."

Conclusion: Verification in the Age of Generative AI

The Elizaldy Co episode and the broader pattern of misclassification are not mere technical curiosities; they are a practical, present-day hazard created by a gap between what AI systems were trained to do (produce plausibly real text and images) and what society now asks them to deliver (prove what is real). Independent audits, newsroom fact-checks, and academic tests paint a consistent picture: multimodal chatbots can help investigators find leads and surface clues, but they are not yet dependable verifiers of visual provenance.

The immediate remedy is operational: treat assistants as research helpers, preserve professional human verification for sensitive public-information tasks, and deploy provenance checks and watermarking where possible. Vendors must accept that trust requires transparency and conservative failure modes; platforms must recognize that scaling back human fact-checking without robust machine-level and policy safeguards hands a powerful amplifier to systems that still make consequential mistakes.

For Windows users, IT teams, and newsrooms, the message is straightforward and urgent: use AI for speed—but verify with care. As AI assistants become increasingly integrated into our operating systems and daily workflows, developing critical verification literacy becomes not just a technical skill but a civic necessity in an increasingly synthetic information environment.