Chatbots Advising Asthma Treatment: Can AI Clear the Air?
Journal: Journal of Allergy and Clinical Immunology (IF 11.2)
Article Year: 2025
A cross-sectional study assessed 3 automated intelligence services’ abilities to answer questions regarding the management and prognosis of asthma. Patients waiting at a respiratory clinic in Milan, Italy were asked to write questions they would like to ask their respiratory physician. These responses were aggregated by researchers into fifteen total questions tiered by difficulty (easy, medium, hard) and covered a tremendous amount of ground from the next best step upon feeling breathless to the effect of travel on asthma exacerbations. These aggregate questions were then posed to the free versions of Bard (upgraded by Google and named Gemini at the time of writing), ChatGPT, and Copilot via two devices with IP addresses outside of Milan. (For those keeping score: 3 AI models x 15 questions x 2 devices = 90 responses.) Of the two responses generated per AI model during this study, one response was chosen via random number generator for presentation. A panel of 21 asthma experts then rated the chosen AI responses separately for reliability, accuracy and comprehensiveness, while 16 patient representatives rated the responses for understandability; reliability was rated as either reliable or not reliable whereas the latter three metrics were rated on a scale of 0 to 10, with 0 indicating poor performance and 10 indicating outstanding performance. It should be noted that all three AI models performed remarkably well, especially when stratifying performance by question difficulty (not discussed here). That being said, the following statistically significant differences were found:
Accuracy: ChatGPT > Bard; ChatGPT > Copilot
Comprehensiveness: ChatGPT > Bard; ChatGPT > Copilot; Bard > ChatGPT
Understandability: Bard > ChatGPT; Bard > Copilot
Regarding reliability, ChatGPT generated 0 unreliable responses, Bard generated 4 unreliable responses, and Copilot generated 2 unreliable responses.
The Spin: Industries ranging from film to pharmaceuticals continue to explore the utility of automated intelligence. A rapidly changing technology, it remains unclear where the buck will drop—some envision a drastically automized world with scores facing unemployment while others remain skeptical, citing the dizzying financials of companies in their attempts to scale AI use to ultimately limit the degree of any such displacement. The reality is likely to fall in some intermediary between the two. Further, given the current political landscape friendly to mergers & acquisitions of companies, it remains to be determined which of the many AI chatbots will “win the race,” so to speak, outlasting its competitors and achieving ubiquitous use. For completeness’ sake, I will add that the ethics of AI use remain fuzzy and discussion may dive into great nuance (e.g., environmental impact, creator bias). In any case, the technology continues to have piles of money shoveled in its direction and is therefore not going anywhere anytime soon, so studies like this are of great importance for clinicians as they help us understand where the technology is and where it might go. This is the information our patients often refer to upon discharge: the new “Dr. Google."
To Read More: Nigro, M., Aliverti, A., Angelucci, A., Braido, F., Canonica, G. W., Bossios, A., Pinnock, H., Boyd, J., Powell, P., Aliberti, S., & Artificial Intelligence Responses on Asthma Study Task Force (2025). Artificial Intelligence-Generated Answers to Patients' Questions on Asthma: The Artificial Intelligence Responses on Asthma Study. The journal of allergy and clinical immunology. In practice, 13(9), 2390–2396. https://doi.org/10.1016/j.jaip.2025.04.051