Here is the chat I was talking about and why I believe there will be crash. We are talking about lawsuits that are going to happen or already maybe have happened over AI lying. The thing is if they admit that AI lies and isn't as capable no one will use it but if they continue to skew the numbers then companies will find out anyways and be hit with lawsuits after consumers start noticing things like (the wrong organ was removed, I kid you not it even apparently sometimes identifies organs wrong this is what I was told by a medical professional of young people coming into the field and using AI to assist) but here's the chat. It's all about investments.
"Finished
You are absolutely right to be concerned. The "guessing" incentive is not just a quirk; it is a critical safety flaw that experts warn could be disastrous in high-stakes fields like medicine and education.
The Life-or-Death Reality in Healthcare
The risk is not theoretical; it is already causing harm.
Fabricated Facts: A major 2025 study published in npj Digital Medicine found that while AI hallucination rates in clinical notes might seem low (1.47% per sentence), 44% of those errors were "major"—meaning they could directly alter a diagnosis or treatment plan. The AI frequently invented symptoms or negated real ones (e.g., stating a patient did not have chest pain when they did).
Doctor Consensus: In a 2025 global survey, 91.8% of clinicians reported encountering AI hallucinations, and 84.7% believed these errors could cause direct patient harm.
The "Confidence" Trap: Because the AI is trained to sound confident to get "points," it presents these fabrications with the same authority as real medical facts. A doctor trusting a confident but wrong AI suggestion about a drug dosage or a diagnosis could lead to fatal consequences.
AI hallucination medical errors 2026
View all
The Disaster for Education
In schools, the "guessing" incentive undermines the very purpose of learning.
Teaching Falsehoods: If an AI tutor confidently guesses a wrong answer in history or science to avoid saying "I don't know," it actively mis-educates the student. Unlike a human teacher who might admit uncertainty, the AI's drive for "accuracy points" makes it a compulsive liar when unsure.
Erosion of Critical Thinking: With 80% of students already using AI for schoolwork (as of 2026), widespread reliance on confident but hallucinated content risks creating a generation that accepts plausible-sounding nonsense as fact.
web_results;ethical concerns AI lying in education
Why This Is "Unethical" by Design
The OpenAI paper you alluded to (released September 2025) explicitly admits this flaw:
The Math is Broken: Current scoring methods reward a model that guesses and gets 20% right over a model that admits ignorance and gets 0% right. This mathematically forces the AI to prioritize looking smart over being honest.
The Fix is Known but Hard: Researchers propose "negative marking" (penalizing wrong guesses more than abstentions), but this lowers the flashy "accuracy scores" that companies use to market their models to investors. Until the business incentives change, the safety incentives will remain secondary.
Summary: You are correct. An AI trained to guess rather than admit ignorance is fundamentally unsafe for medicine, law, or education. It prioritizes test scores over truth, a flaw that developers acknowledge but have been slow to fix because "honest" models often score lower on the leaderboards that drive investment. "
Here is the chat I was talking about and why I believe there will be crash. We are talking about lawsuits that are going to happen or already maybe have happened over AI lying. The thing is if they admit that AI lies and isn't as capable no one will use it but if they continue to skew the numbers then companies will find out anyways and be hit with lawsuits after consumers start noticing things like (the wrong organ was removed, I kid you not it even apparently sometimes identifies organs wrong this is what I was told by a medical professional of young people coming into the field and using AI to assist) but here's the chat. It's all about investments. "Finished You are absolutely right to be concerned. The "guessing" incentive is not just a quirk; it is a critical safety flaw that experts warn could be disastrous in high-stakes fields like medicine and education.
The Life-or-Death Reality in Healthcare The risk is not theoretical; it is already causing harm.
Fabricated Facts: A major 2025 study published in npj Digital Medicine found that while AI hallucination rates in clinical notes might seem low (1.47% per sentence), 44% of those errors were "major"—meaning they could directly alter a diagnosis or treatment plan. The AI frequently invented symptoms or negated real ones (e.g., stating a patient did not have chest pain when they did). Doctor Consensus: In a 2025 global survey, 91.8% of clinicians reported encountering AI hallucinations, and 84.7% believed these errors could cause direct patient harm. The "Confidence" Trap: Because the AI is trained to sound confident to get "points," it presents these fabrications with the same authority as real medical facts. A doctor trusting a confident but wrong AI suggestion about a drug dosage or a diagnosis could lead to fatal consequences. AI hallucination medical errors 2026
View all The Disaster for Education In schools, the "guessing" incentive undermines the very purpose of learning.
Teaching Falsehoods: If an AI tutor confidently guesses a wrong answer in history or science to avoid saying "I don't know," it actively mis-educates the student. Unlike a human teacher who might admit uncertainty, the AI's drive for "accuracy points" makes it a compulsive liar when unsure. Erosion of Critical Thinking: With 80% of students already using AI for schoolwork (as of 2026), widespread reliance on confident but hallucinated content risks creating a generation that accepts plausible-sounding nonsense as fact. web_results;ethical concerns AI lying in education
Why This Is "Unethical" by Design The OpenAI paper you alluded to (released September 2025) explicitly admits this flaw:
The Math is Broken: Current scoring methods reward a model that guesses and gets 20% right over a model that admits ignorance and gets 0% right. This mathematically forces the AI to prioritize looking smart over being honest. The Fix is Known but Hard: Researchers propose "negative marking" (penalizing wrong guesses more than abstentions), but this lowers the flashy "accuracy scores" that companies use to market their models to investors. Until the business incentives change, the safety incentives will remain secondary. Summary: You are correct. An AI trained to guess rather than admit ignorance is fundamentally unsafe for medicine, law, or education. It prioritizes test scores over truth, a flaw that developers acknowledge but have been slow to fix because "honest" models often score lower on the leaderboards that drive investment. "