When several AI chatbots give the same answer to a health question, that agreement is not a sign the answer is right, according to a study published on 14 November 2026 in the journal Scientific Reports.
Researchers at the Social and Cognitive Informatics Laboratory of HSE University in Saint Petersburg, Russia, tested whether large language models (LLMs) could reliably judge short health claims. They found that even when the strongest models agreed with each other, their shared verdict was wrong almost a third of the time.
How the test worked
The team built a set of 108 short health claims, which it calls RuHealth. Doctors reviewed the claims in several stages: 40 were true, 40 were false and 28 were classed as disputed. Ten LLMs were asked to label each claim as true, false or disputed. They included leading general-purpose ("frontier") models, models built for Russian and models designed for medicine.
Each claim was tested in Russian and in a checked English translation, at different "temperature" settings that control how random a model's answers are, with three runs each. This produced about 90,000 judgements, the paper says.
What the researchers found
- The frontier models did best, with macro-F1 accuracy scores of 64 to 70, but the authors say no model was reliable.
- The models labelled claims as "disputed" in 59.8% of judgements, although only about 26% of the claims actually were disputed.
- False claims were systematically missed.
- Combining answers by majority vote did not improve results.
The sharpest finding came from three frontier models that gave the same verdict on 90 of the 108 claims. Of those unanimous verdicts, 31% were wrong. In 26 of the 28 wrong cases, the models had called a claim "disputed" when it was actually true (14 claims) or false (12 claims).
The authors call this effect a "consensus mirage". They report that it held up when they changed the wording of the prompts and when they switched between Russian and English.
Why it matters
The paper notes that LLMs are increasingly proposed as tools for checking health information online, and that a common assumption is that agreement between models signals reliability. The researchers say that, at least for health claims in their test, agreement among current LLMs is not evidence that an answer is medically correct.
The study tested 108 claims, so its findings apply to that set of claims and the models tested. The authors did not test how often people act on chatbot health advice.
About the study
The paper, by Elena Artemenko, Olessia Koltsova, Sergey Nikolenko, Anton Surkov and Anna Probst, was received by the journal on 12 August and accepted on 31 October. It was funded under HSE University's Basic Research Program, and the authors declare no competing interests. The article is open access.



