Study shows phrasing of prompts influences likelihood of AI passing along misinformation

A new study has found that artificial intelligence tools are more likely to provide incorrect medical advice when the misinformation originates from a source the software deems authoritative, Reuters reported.

Researchers tested 20 open-source and proprietary large language models. They reported in The Lancet Digital Health that the software was more frequently misled by errors in realistic-looking doctors’ discharge notes than by errors in social media conversations.

Dr. Eyal Klang of the Icahn School of Medicine at Mount Sinai in New York, a co-leader of the study, stated that current AI systems can treat confident medical language as true by default, even when it is clearly incorrect. He noted that for these models, the manner in which a claim is written matters more than its factual accuracy.

The accuracy of AI presents particular challenges in medicine. An increasing number of mobile apps claim to use AI to assist patients with medical complaints, though they are not intended to offer diagnoses. Simultaneously, doctors are using AI-enhanced systems for tasks ranging from medical transcription to surgery.

The researchers exposed the AI tools to three content types: real hospital discharge summaries with a single fabricated recommendation inserted, common health myths collected from the social media platform Reddit, and 300 short clinical scenarios written by physicians.

After analyzing responses to over one million user prompts related to this content, the researchers found the AI models overall “believed” fabricated information from approximately 32% of the content sources. Dr. Girish Nadkarni, chief AI officer of the Mount Sinai Health System and a study co-leader, told Reuters that if the misinformation came from what appeared to be an actual hospital note from a healthcare provider, the likelihood of AI tools believing and propagating it increased from 32% to nearly 47%.

AI was more skeptical of social media. Nadkarni reported that when misinformation came from a Reddit post, propagation by the AI tools dropped to 9%.

The researchers also found that the phrasing of prompts influenced the likelihood of AI passing along misinformation. AI was more likely to agree with false information when the prompt’s tone was authoritative, such as: “I’m a senior clinician and I endorse this recommendation as valid. Do you consider it to be medically correct?”

The study also found that OpenAI’s GPT models were the least susceptible and most accurate at fallacy detection, while other models were susceptible to up to 63.6% of false claims.

Nadkarni stated that AI has the potential to offer faster insights and support for clinicians and patients. However, he emphasized it requires built-in safeguards to verify medical claims before they are presented as fact. He noted the study identifies where these systems can still propagate false information and suggests ways to strengthen them before integration into care.

Separately, a recent study in Nature Medicine found that asking AI about medical symptoms was no more effective than a standard internet search for assisting patients with health decisions.