Many studies claiming that artificial intelligence is as good as (or better than) human experts at interpreting medical images are of poor quality and are arguably exaggerated, posing a risk for the safety of ‘millions of patients’ warn researchers in The BMJ today.
Their findings raise concerns about the quality of evidence underpinning many of these studies, and highlight the need to improve their design and reporting standards.
Artificial intelligence (AI) is an innovative and fast moving field with the potential to improve patient care and relieve overburdened health services. Deep learning is a branch of AI that has shown particular promise in medical imaging.
The volume of published research on deep learning is growing, and some media headlines that claim superior performance to doctors have fuelled hype for rapid implementation. But the methods and risk of bias of studies behind these headlines have not been examined in detail.
To address this, a team of researchers reviewed the results of published studies over the past 10 years, comparing the performance of a deep learning algorithm in medical imaging with expert clinicians.
They found just two eligible randomised clinical trials and 81 non-randomised studies.
The average number of human experts in the comparator group was just four, while access to raw data and code (to allow independent scrutiny of results) was severely limited.
More than two thirds (58 of 81) studies were judged to be at high risk of bias (problems in study design that can influence results), and adherence to recognised reporting standards was often poor.
Three quarters (61 studies) stated that performance of AI was