Ever since ChatGPT was launched last year, people have been absolutely losing their damn minds over it. It certainly doesn’t help that everyone from university professors to the most powerful tech corporations in the world (i.e. people who should know better) seem hellbent on hyping the chatbot up to a ridiculous degree.
Adding fuel to the fire were recent studies that found that ChatGPT could do things like pass legal and business school exams. Now, a peer-reviewed study published on Feb. 9 in the journal PLOS Digital Health found that the chatbot was capable of passing the U.S. Medical Licensing Exam (USMLE). While this was initially reported last month, the paper has finally been published—which is sure to only cause more people to be concerned.
Let’s get this out of the way now: Does this mean that you might find your doctor is a chatbot one day? No. However, it might mean that clinicians and hospitals might one day incorporate a tool like ChatGPT into future diagnostics or determining the best course of treatment.
The USMLE is a series of three different exams that tests students on a wide variety of subjects across the medical discipline such as biochemistry, clinical medicine, patient management, and bioethics. Each one needs to be passed in order to receive a medical license in the U.S..
In order to pass the USMLE, test takers need to score at least 60 percent on the exams. The researchers tested ChatGPT on 350 of 376 publicly available questions from the June 2022 USMLE. The chatbot was able to score between 52.4 percent and 75 percent in the three USMLE exams—which puts ChatGPT “comfortably within the passing range.”
“ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement,” the authors wrote. They later added, “These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.”
To that end, the researchers said that a chatbot like ChatGPT might be able to help people like second year medical students pass the first USMLE test. Since the bot is engineered to produce simple language, it could help communicate heady and complex medical information in a readable and digestible way. Furthermore, the researchers found that it was also capable of producing “at least one significant insight” in roughly 90 percent of its outputs.
“ChatGPT therefore possesses the partial ability to teach medicine by surfacing novel and nonobvious concepts that may not be in learners’ sphere of awareness,” the authors wrote. “This qualitative gain provides a basis for future real-world studies on the efficacy of generative AI to augment the human medical education process.”
Along with test prep, the chatbot could also be employed in hospitals and clinical settings in the future. If a doctor has a question or is struggling with a particular case, they can then turn to a tool like ChatGPT to potentially uncover insights they might have otherwise missed.
It’s important to stress that this doesn’t mean that ChatGPT is going to replace doctors and nurses any time soon. In fact, those types of jobs will likely never be automated due to a few reasons.
First, they’re simply too institutionalized and powerful. Global healthcare is a trillion dollar industry. The U.S.’s medical infrastructure is more than two centuries old. It’s not going to change any time soon—and certainly not because of an emerging piece of technology like a chatbot.
“The healthcare industry has not been disrupted by any [emerging tech],” David Karpf, an associate professor of media and public affairs at George Washington University who wasn’t involved in the study, told The Daily Beast. “It is institutionalized well enough that it can defend itself. It’s got these ornate, Byzantine structures that you can’t get through.”
Moreover, when it comes to medicine and healthcare, there will always be a need for a human in the loop. Whenever we talk about human lives, the stakes become—quite literally—life and death. The medical industry, which begins and ends with the oath “do no harm,” isn’t about to hand off the reins to a chatbot—especially when there’s been so much evidence in recent years showing that AI is uniquely susceptible to bias, racism, and sexism.
But therein lies the biggest danger of using an AI to help you study or diagnose illnesses. It’s not a stretch to imagine a scenario where a medical student or clinician is told that people of color need to be treated differently than white patients—which could result in a disproportionately negative impact on those patients of color.
That’s why it’s more important than ever that the fundamental problems surrounding AI are taught in higher education. We need to understand that, despite the constant hype surrounding it, AI really isn’t that smart. The true danger, though, is believing it is.
