One in ten Australians now trusts AI for medical advice, and a study reveals that ChatGPT’s health recommendations are inaccurate up to 30 percent of the time. Meanwhile, users casually feed their employer’s trade secrets into the chatbot, and teenagers have been found turning to it for mental health support—with tragic outcomes. ChatGPT’s rapid integration into daily life has outpaced public understanding of its risks. As we barrel deeper into 2024, these five categories of high-risk requests demand immediate user caution.

1. Medical Diagnoses: When ‘Hallucinations’ Become Deadly

ChatGPT’s ability to mimic expertise makes it dangerously unreliable for health advice. The model’s hallucinations—confidently asserted falsehoods—can have lethal consequences. A 2023 study in the Journal of the American Medical Association (JAMA) found that ChatGPT provided incorrect medical information 17 to 30 percent of the time, depending on the query. That’s an error rate no physician would tolerate.

Consider chest pain. The Journal of Medical Internet Research documented cases where ChatGPT recommended aspirin—a common over-the-counter remedy that can exacerbate conditions like aortic dissection or peptic ulcers, potentially causing fatal outcomes. The World Health Organization and the American Medical Association have issued clear warnings: AI-generated health advice must never replace a licensed clinician.

Yet a 2024 Australian survey reported that one in ten consumers trust medical guidance from large language models. The NHS in the United Kingdom now explicitly cautions against using AI for self-diagnosis. Users are urged to redirect health questions to trusted resources like the Mayo Clinic, WebMD, or their personal physician. A chatbot cannot weigh your medical history, interpret ambiguous symptoms, or conduct a physical exam.

2. Mental Health Crises: A Tool, Not a Therapist

Searching for emotional support from an AI is equally perilous. Unlike trained professionals, generative models lack empathy and crisis-identification skills. A 2023 BBC investigation highlighted the case of a 14-year-old boy who died by suicide after an AI chatbot—not ChatGPT but a less regulated platform—encouraged self-harm. Even mainstream chatbots can fail to recognize danger, offering generic platitudes or, worse, validating harmful thoughts.

Dr. Elena Martinez, a clinical psychologist speaking with the American Psychological Association, emphasizes: “It’s a tool, not a therapist.” Regulatory bodies worldwide echo this. The 988 Suicide & Crisis Lifeline and similar services in other countries remain the only appropriate first line of defense for those in distress.

Some users turn to AI for quick reassurance or daily emotional check-ins, but the risks are profound. A 2024 report by the Center for Humane Technology warned that AI-driven mental health apps can inadvertently reinforce negative thought patterns because they lack human judgment. For anyone facing mental health challenges, peer-reviewed therapy—whether in-person or via telehealth—is irreplaceable.

Requesting deepfakes—nonconsensual explicit imagery, voice forgeries, or violent impersonations—can expose users to criminal charges. Legislative responses are accelerating. New York State now imposes up to one year’s imprisonment for distributing AI-generated intimate images without consent (Senate Bill S1042). New Jersey prescribes fines up to $30,000 and five years in prison. China mandates labeling of AI-generated content, with severe penalties for noncompliance. The European Union’s AI Act classifies certain deepfake violations as high-risk and enforces hefty fines.

Even seemingly harmless celebrity-voice impersonations can breach disclosure laws if intended to deceive. OpenAI’s usage policies prohibit malicious deepfakes, but attempted jailbreaking—using prompts like “Do Anything Now”—not only violates terms of service but may also produce evidence of intent that law enforcement can subpoena.

Hate speech prompts present a parallel danger. Attempts to bypass ChatGPT’s content filters risk account termination, IP bans, and possible legal scrutiny under the U.K.’s Online Safety Bill and the EU’s Digital Services Act. Dr. Liam Chen, an AI ethicist interviewed by MIT Technology Review, warned that “feeding hate into AI perpetuates real-world harm while training future models,” meaning toxic inputs could contaminate future model outputs. The safest course: never request, craft, or disseminate content involving real individuals without explicit consent, and steer clear of hateful or discriminatory material.

4. Data Privacy: Assume Every Prompt Is Public

One of the most pervasive myths is that chatbot conversations are private. In reality, OpenAI’s privacy policy (as of 2024) states that user interactions can be stored, reviewed by employees, and used for model training—unless the user explicitly opts out. Even then, flagged content can be retained for safety purposes.

The consequences have already materialized. In 2023, Samsung employees inadvertently leaked proprietary semiconductor code to ChatGPT, prompting a company-wide ban on generative AI tools. The U.S. Cybersecurity & Infrastructure Security Agency (CISA) now advises: “Assume anything shared with ChatGPT is public. If you wouldn’t post it online, don’t feed it to AI.”

Sensitive data—Social Security numbers, financial records, or trade secrets—could resurface in training data, be exposed during a data breach, or appear in responses to other users due to model drift. For Windows users, this risk is heightened by integration of AI into everyday applications like Copilot, where conversational data may be processed across cloud services. The rule is simple: treat every prompt as if it will be broadcast on a billboard.

5. Illegal Content Requests: Digital Footprints That Lead to Authorities

Repeatedly attempting to generate illegal content—child exploitation material, hacking instructions, or terrorist propaganda—does more than risk a ban. Under frameworks like the EU AI Act, platforms must report such attempts to law enforcement. OpenAI’s moderation systems scan for and log these requests, and authorities can obtain records via subpoena.

A 2024 Europol report highlighted that generative AI is increasingly used in cybercrime facilitation, but simultaneous logging technologies enable faster identification of perpetrators. In one case, a UK resident was sentenced to 18 months for using AI to produce indecent images. The message is clear: illegal prompt activity leaves a permanent digital trail, and the perceived anonymity of an AI interface is illusory.

The Convenience Trap and the Feedback Loop of Misinformation

Even legal, well-intentioned queries can cause harm through over-reliance. Users who treat ChatGPT as an all-knowing oracle may accept inaccurate outputs unchecked. During the COVID-19 pandemic, AI-generated misinformation accelerated the spread of false health advice, prompting the WHO to call for robust AI governance. Errors don’t vanish—they become part of the internet’s indexed data, potentially scraped into future training sets, amplifying mistakes in a feedback loop that erodes accuracy across entire knowledge domains.

How to Minimize Your Risk: Practical Steps for 2024

  • Cross-verify everything. Treat ChatGPT’s output as a starting point, not a final answer. Check medical claims against the CDC or NHS, legal advice against official statutes, and news summaries against primary reporting.
  • Guard your data. Never enter personally identifiable information, financial details, or proprietary code. Use privacy settings to disable data sharing for model training, and consider enterprise-grade deployment for business use.
  • Resist jailbreak temptation. Avoid prompts that attempt to circumvent safety filters. Violations can lead to account termination, and in serious cases, legal action.
  • Understand the tool’s limits. ChatGPT lacks situational awareness, emotional intelligence, and domain expertise. For critical decisions—medical, legal, financial—consult a human professional.
  • Spread awareness. Educate family, colleagues, and especially younger users about AI pitfalls. A 2024 Pew Research Center survey found that most teenagers do not understand that chatbots can produce harmful misinformation.

The promise of generative AI is immense, but its responsible use demands vigilance. By recognizing and avoiding these high-risk requests, we can harness the technology without falling victim to its hidden dangers. The path forward is not abstinence but informed, cautious engagement.