A recent study conducted by a team of AI and medical researchers from the U.K. and the U.S. has raised questions about the reliability of medical advice provided by AI chatbots. The researchers asked 1,298 volunteers to consult chatbots for medical advice and compared the results to advice from other sources or the users’ own intuition.
The convenience of accessing medical advice through chatbots like ChatGPT has become increasingly popular due to its accessibility and cost-effectiveness. However, this study aimed to evaluate the accuracy of such advice provided by Language Model Machines (LLMs).
Previous research has demonstrated the high performance of AI apps on medical benchmarks and licensing exams. Despite this, little research has been conducted to determine if these capabilities translate effectively to practical medical advice. The study revealed that users often failed to provide complete information during their interactions with the chatbots, leading to communication breakdowns.
When comparing the advice given by chatbots to other sources such as online medical sites or the users’ own knowledge, the researchers found mixed results. In some cases, the advice provided by chatbots was similar to other sources, while in others, it was less accurate. There were instances where chatbot users were less likely to correctly identify their ailment and underestimated the severity of their condition.
The study concluded that while chatbots may offer some valuable guidance, it is essential for users to seek advice from trusted sources when it comes to medical concerns. The researchers emphasized the importance of comprehensive and accurate information exchange between users and chatbots to improve the reliability of medical advice provided.
For more information on this study, you can refer to the paper titled “Clinical knowledge in LLMs does not translate to human interactions” published on the arXiv preprint server in 2025. The DOI for the paper is 10.48550/arxiv.2504.18919.
In conclusion, while AI chatbots have the potential to assist users with medical queries, caution should be exercised when relying solely on their advice. It is advisable to seek professional medical guidance from trusted sources for accurate and reliable information on health-related matters.
