You might not realize it, but you can probably be easily duped by AI. This isn’t a knock on you; it’s just human nature. In fact, a study published in April found that OpenAI’s ChatGPT could even greatly influence how you answer the Trolley Problem without you even realizing it.
With the recent boom of AI, we’re still grappling with how exactly these bots might impact society—and a new study found that they could cause harm in hospital rooms.
Researchers from Germany and the Netherlands published a paper Tuesday in the journal Radiology that found AI decision-making systems may impair the decisions of radiologists evaluating mammograms for signs of breast cancer—regardless of the physician’s level of expertise. The findings underscore how automation bias—the penchant for people to favor the decisions of automated decision-making systems—can unknowingly impact highly-educated doctors as well.
“We anticipated that inaccurate AI predictions would influence the decisions made by radiologists in our study, particularly those with less experience,” study lead author Thomas Dratsch, a machine learning researcher at the University of Cologne, Germany, said in a press release. “Nonetheless, it was surprising to find that even highly experienced radiologists were adversely impacted by the AI system’s judgments, albeit to a lesser extent than their less seasoned counterparts.”
As part of the study, 27 radiologists were tasked with examining 50 mammograms. For each mammogram, the doctors gave a Breast Imaging Reporting and Data System (BI-RADS) assessment, which is a scoring system to categorize whether or not evidence of cancer was detected in the mammogram. Each assessment was assisted with an AI system.
The researchers were given two randomized sets of mammograms. The first included results in which the AI correctly assessed the BI-RADS category. The second included results where the AI supposedly suggested the wrong category.
The radiologists were much worse at assigning the correct BI-RADS category when the AI suggested the wrong one. And more experience did not offset this issue. In fact, radiologists with less than 15 years of experience had an accuracy of 20 percent when provided the mistaken category, but scored well in 80 percent of cases where the AI provided the correct one.
By contrast, radiologists with more than 15 years of experience had their accuracy fall from 82 percent to 45.5 percent when the supposed AI misled them.
The implications are grim. Not only are radiologists highly susceptible to automation bias, but even the most experienced ones can fall prey. The study’s authors hope that this acts as a wakeup call for medical systems that are incorporating automated decision-making support into their workflow and diagnostic process. They also suggest implementing safeguards such as showing the radiologists the confidence level of the AI system, educating users about their limitations, and ensuring that they feel accountable for the decisions they ultimately make.
“Given the repetitive and highly standardized nature of mammography screening, automation bias may become a concern when an AI system is integrated into the workflow,” Dratsch explained. “Our findings emphasize the need for implementing appropriate safeguards when incorporating AI into the radiological process to mitigate the negative consequences of automation bias.”
