Abstract
BACKGROUND AND PURPOSE: Interest in artificial intelligence (AI) and machine learning (ML) has been growing in neuroradiology, but there is limited knowledge on how this interest has manifested into research and specifically, its qualities and characteristics. This study aims to characterize the emergence and evolution of AI/ML articles within neuroradiology and provide a comprehensive overview of the trends, challenges, and future directions of the field.
MATERIALS AND METHODS: We performed a bibliometric analysis of the American Journal of Neuroradiology; the journal was queried for original research articles published since inception (January 1, 1980) to December 3, 2022 that contained any of the following key terms: “machine learning,” “artificial intelligence,” “radiomics,” “deep learning,” “neural network,” “generative adversarial network,” “object detection,” or “natural language processing.” Articles were screened by 2 independent reviewers, and categorized into statistical modeling (type 1), AI/ML development (type 2), both representing developmental research work but without a direct clinical integration, or end-user application (type 3), which is the closest surrogate of potential AI/ML integration into day-to-day practice. To better understand the limiting factors to type 3 articles being published, we analyzed type 2 articles as they should represent the precursor work leading to type 3.
RESULTS: A total of 182 articles were identified with 79% being nonintegration focused (type 1 n = 53, type 2 n = 90) and 21% (n = 39) being type 3. The total number of articles published grew roughly 5-fold in the last 5 years, with the nonintegration focused articles mainly driving this growth. Additionally, a minority of type 2 articles addressed bias (22%) and explainability (16%). These articles were primarily led by radiologists (63%), with most (60%) having additional postgraduate degrees.
CONCLUSIONS: AI/ML publications have been rapidly increasing in neuroradiology with only a minority of this growth being attributable to end-user application. Areas identified for improvement include enhancing the quality of type 2 articles, namely external validation, and addressing both bias and explainability. These results ultimately provide authors, editors, clinicians, and policymakers important insights to promote a shift toward integrating practical AI/ML solutions in neuroradiology.
ABBREVIATIONS:
- AI
- artificial intelligence
- ML
- machine learning
- AJNR
- American Journal of Neuroradiology
SUMMARY
PREVIOUS LITERATURE:
Previous studies examining the trends in artificial intelligence (AI) and machine learning (ML) research in medicine has established medical imaging to be a major area of application, accounting for up to 20% of publications. Also, these studies have demonstrated neuroradiology to be a leading area of AI research. The studies that have looked at AI research trends in medical imaging have maintained a broad scope examining growth and research topics, but there is a lack of studies specifically investigating areas that can affect the future integration of this research such as article quality.
KEY FINDINGS:
AI/ML publications in the AJNR have grown roughly five-fold in the last 5 years, mainly driven by developmental research work, and only a minority of articles (21%) investigated end-user applications of AI. Several areas of improvement were identified including external validation, bias, and explainability.
KNOWLEDGE ADVANCEMENT:
Although this study confirms the rapid growth of AI/ML research in neuroradiology, it has also identified areas for improvement in the field. These results ultimately provide authors, editors, clinicians, and policymakers important insights to promote a shift toward integrating practical AI/ML solutions in neuroradiology.
Artificial intelligence (AI) is a field focusing on the development of tools that mimic human cognition, especially its learning, decision, and interpretation capabilities. In recent years, the field has grown rapidly in its knowledge base and complexity. One example of this growth is machine learning (ML), a subfield of AI that uses algorithms and models to enable machines to independently learn from experience or data. The use of AI and ML has become widespread in its applications, especially with recent unprecedented advances in large language models, such as OpenAI’s ChatGPT. AI research in the field of medicine has grown exponentially with more than 60% of all articles that existed in 2018 having been published between 2008 and 2018.1 The main factors driving this growth are recent increases in computing power and the availability of large, high-quality data sets. Additional factors include increased interest in the potential of AI as a scalable solution to today’s demands of service efficiency, intervention efficacy, and personalized health care.2
A recent bibliometric study by Kocak et al3 that examined the trends in AI research in medical imaging showed medical imaging as a major application of AI/ML in medicine, accounting for 20% of AI in medical publications. Additionally, a narrative review by Pesapane et al4 demonstrated that from among the different medical imaging subspecialties, neuroradiology has been consistently identified as leader in AI/ML publications accounting for 34% of the AI/ML articles published in the field of medical imaging. Moreover, a recently published review conducted by Wagner et al5 that outlined current topics in AI and neuroradiology found this to be one of the most heavily researched fields in medical imaging. These studies, however, maintained a broad scope, mostly examining growth, citations, and topics of AI/ML articles rather than more specific indicators of AI/ML article quality.5
The popularity of AI/ML in neuroradiology can partly be explained by: 1) the prominent role of imaging for diagnosis; 2) the availability of brain-specific public data sets; 3) the use of complex multidimensional and multimodal data; and 4) the attraction of researchers to the subspeciality due to the leading role of neuroimaging in the field of big data, and linked to that, advanced data analysis and AI.2,6 However, there is a paucity of studies focused on the emergence and evolution of AI/ML research in the field of neuroradiology. This could help us understand whether this increased interest in AI/ML is reflected in the amount of research published, and also how the characteristics and quality of this research may influence its integration into neuroradiology practice.3,4
To provide an understanding of the emergence and evolution of AI/ML in neuroradiology, we bibliometrically analyzed the American Journal of Neuroradiology (AJNR). First, we assessed the number of research articles focusing on AI/ML integration into day-to-day practice. To gain a better understanding of the factors that may be limiting their publication, we then focused our analysis on the developmental research articles as they should represent the precursor work.
MATERIALS AND METHODS
Study Design
We used the bibliometric method to analyze original articles on AI/ML in AJNR. This method facilitates the examination of a particular field of research by providing a concise overview based on the published literature while also shedding light on details that drive the evolving nature and challenges of the field.
Search Strategy and Eligibility Criteria
We included original research articles published between journal inception (January 1, 1980) to the date the query was conducted (December 3, 2023) that contained any of the following key terms: “machine learning,” “artificial intelligence,” “radiomic,” “deep learning,” “neural network,” “generative adversarial network,” “object detection,” “natural language processing” and fit the definitions of AI and ML as described in Erickson et al7 and Wagner et al8 (Online Supplemental Data). We used the advanced search function on the AJNR website and then queried the journal for original research articles containing the key terms, however this is limited by the fact that it retrieves articles that mention keywords anywhere in the article, irrespective of their use and context. Articles without available full-text and review articles were excluded.
Selection of Studies
The identified articles were then uploaded into Covidence (https://www.covidence.org/) to facilitate screening. Two reviewers (A.B. and M.N., trained academic neuroradiologists with 6 years’ experience and neuroradiology fellows) independently screened titles and abstracts for relevancy based on the inclusion criteria. Conflicts between the 2 reviewers were resolved by a third independent reviewer (V.P., neuroradiology fellow). Articles that successfully passed this stage of the screening process moved on to full-text review. Two reviewers (A.B. and M.N. with V.P. as adjudicator) independently performed the full-text reviews and determined whether these articles completely met the inclusion criteria with reasons for exclusion being recorded.
Data Extraction
The included articles were categorized into 3 types of AI/ML studies: statistical modeling (type 1); AI/ML development (type 2) both representing developmental research work but without a direct clinical integration; or end-user application (type 3), which is the closest surrogate of potential AI/ML integration into day-to-day practice by using the following definitions:Statistical Modelling (Type 1): Article uses statistics to analyze sample data and make predictions. These articles are usually hypothesis-based, with a limited number of candidate biomarkers to be tested using statistical methods.
AI/ML Development (Type 2): Article develops and trains (may additionally test and validate) an AI/ML model.
End-User Applications (Type 3): Article applies pre-existing AI/ML product/software.
Additionally, the year of publication and number of authors were extracted from each article. Because type 2 AI/ML development articles typically precede type 3 end-user application articles, we extracted more data such as the profession and degrees of first/corresponding author(s), geographic location of data source, source of funding, and neuroradiology subfield of the article. We also extracted data from these articles relevant to study quality such as the study design (pro- versus retrospective; multi- versus single-center), types of AI/ML models used, whether testing or validation was done, type of validation (internal and/or external), and whether learning was supervised or unsupervised. Each article was also examined for whether it addressed bias or explainability, a concept relating to whether the features extracted, or predictions made by the authors’ AI/ML models could be reasonably understood by humans. The data were summarized by using descriptive statistical methods.
RESULTS
Our search identified 657 AJNR original research articles of which 116 duplicates were removed. The remaining 541 articles were screened, of which 349 were excluded. The main reason for exclusion was the lack of specificity of the search engine often retrieving key terms used without the study using AI/ML methodology. One hundred ninety-two articles underwent full-text review to confirm their inclusion criteria eligibility. From these 192 articles, 10 articles were excluded because they did not fit the definitions of AI/ML.7,8 The 182 remaining articles were included in our bibliometric analysis (Fig 1).
Flow chart of the article screening process.
Type 3 End-User Application Articles Remain a Minority
The number of articles published has steadily grown since 2007 with a 25- and 5-fold increase in the last 15 and 5 years, respectively. Developmental research articles without direct clinical integration represented 79% of all AI/ML articles (type 1, statistical modeling, n = 53; type 2, AI/ML development, n = 90) with the remaining 21% of articles being integration-focused (type 3, end-user application, n = 39). The proportion of type 1 articles published per year has greatly decreased to levels less than those of the type 3 articles while the number of type 2 articles has rapidly increased in recent years, now representing most articles (Fig 2). The average number of authors per publication was large but similar across article types with type 1, 2, and 3 articles having 9, 10, and 9 authors on average, respectively.
Global results (type 1, 2, and 3). Data characterizing the growth of all AI/ML articles (ie, type 1, 2, and 3). A, The number of all articles published from the first year an AI/ML article was published in the AJNR in 2000 to the article search date on December 3, 2023. *Striped bar in 2023 indicates the extrapolated number of articles published in that year should the search have occurred at the end of the year. B, Pie chart breaking down the share each article type holds among all AI/ML publications. C, Percentile bar graph demonstrating article type share in the last 5 years.
Analysis of Type 2 Articles Sheds Insight into the Lack of Type 3 Articles
We found a consistent increase in the number of type 2 articles published since 2007, with a 4-fold increase in the last 5 years. Most of the 90 type 2 articles focused solely on the adult population (n = 72; 80%) with the remainder focusing on pediatric (n = 14; 16%) or adult and pediatric populations (n = 4; 4%). The most frequent topic representing 40% (n = 36) of all published type 2 articles was oncology with the second being vascular disease (Fig 3B, -C). Regarding the geographic demographics of type 2 studies, we found that a large proportion (n = 42; 47%) were published by authors based in the United States, with South Korea (n = 12; 13%) coming in a distant second. In addition to a variety of countries being represented in the data, 8 articles used public data sets and were not associated with a specific country (Fig 3A).
Qualitative type 2 results. General overview of type 2 publication characteristics (n=90). A, A world map showing where studies were conducted. B, Bar graph demonstrating the number of type 2 articles published in years with at least one article published. *Striped bar in 2023 indicates the extrapolated number of articles published in that year should the search have occurred at the end of the year. C, Graphic display of the patient populations and topics studied. VAS = vascular; DEG = degenerative; H/N = head & neck; INF = inflammation.
Two-thirds (n = 57; 63%) of all type 2 publications had a radiologist as a first or corresponding/senior author and among these radiologist-led type 2 publications, 60% (n = 34) were led by a radiologist with graduate degrees, many of them being PhD and MBA degree holders. Most articles (n = 71; 79%) received some type of funding, most commonly from governmental organizations (eg, National Institutes of Health, Department of Defense, National Cancer Institute), foundations, and universities, with 19 articles receiving no funding. Although there were only 6 industry-funded articles, all of them were recently published in 2022 and 2023.
In terms of the quality of type 2 articles, we found more single-centered (58%) than multicentered designs (42%). However, most were retrospective with only 17% (n = 15/90) being prospective studies (Fig 4A, -B). Our full-text review also revealed that only 14% (n = 13/90) of type 2 articles were externally validated (Fig 4C). Algorithm type was categorized into deep (eg, convolutional neural networks, generative adversarial networks, multilayer perceptrons) or conventional learning (eg, linear regression, random forest, support vector machine) techniques. We found that these publications favored the use of more complex algorithms with deep and conventional learning techniques representing 56% and 44% of articles, respectively (Fig 4D).
Evaluative type 2 results. Methodologic characteristics of type 2 publications (n = 90). Pie charts demonstrating the percentage of type 2 articles that were (A) single- versus multicentered, (B) prospective versus retrospective, (C) externally validated or internally validated their models, and (D) utilized deep learning to train their models or only conventional learning techniques.
Additionally, we found that only 22% (n = 20/90) of type 2 articles mentioned bias as a point of discussion, either as something avoided successfully, a limitation, or recognized as an area for future improvement. Similarly, the concept of explainability or interpretability in AI was addressed in only 16% (n = 14/90) of articles. We also found that type 2 articles published between 2020 and 2023 adequately addressed bias at higher rates (n = 17/68; 25%) than those published before 2020 (n = 3/22; 13.6%). Additionally, type 2 articles published between 2020 and 2023 addressed explainability at slightly higher rates (n = 11/68; 16.2%) than those published before 2020 (n = 3/22; 13.6%).
DISCUSSION
Our results demonstrate that the number of AI/ML publications in the AJNR has been rapidly increasing, with type 2 articles being the major drivers of this overall growth. Only a minority of these new articles focused on end-user application (type 3), indicating AI/ML research in the field is still centered on research rather than clinical integration though this has been growing. Our data point to specific characteristics and deficiencies of the type 2 articles, which provide potential explanations and solutions to improve the quality of research and accelerate its integration.
Our observed rapid increase in AI/ML publications in AJNR is consistent with the overall growth of AI research in medicine, one big exception being the drop in articles published in 2020, most likely due to the coronavirus disease 2019 (COVID-19) pandemic.1,9 The decrease in type 1 and stark increase in type 2 articles is consistent with what we would expect of a relatively new and growing field. As more research is conducted and published, researchers gain more experience and knowledge and can build upon previous publications to produce more complex research. As expected, we found only a small number of type 3 end-user application articles, but we anticipate that they will eventually become the leading type of AI article especially as industries become more interested and begin to increase their funding.10 This shift in article focus over time suggests authors should begin to shift their focus toward research centered on implementing and integrating AI in a manner that is feasible and clinically relevant.
Our analysis of type 2 articles showed a 4-fold increase in the last 5 years compared with the 5-fold increase of all AI/ML articles (type 1, 2, and 3 combined) in the same time period. This trend is in line with the evolving complexity of the field that has moved away from its earlier use of conventional statistical or “shallow learning” methods and toward using deep learning methods. Should a shift to deep learning methods be sustained, we would anticipate seeing a similar shift toward more type 3 end-user application studies because deep learning methods are more suited for tackling contemporary challenges.10 The analysis of type 2 articles also showed characteristics and weaknesses that may limit AI/ML research and its subsequent integration into day-to-day practice. Our results demonstrated a focus on the adult population as well as oncologic and vascular diseases. The limited amount of sufficiently large pediatric data sets restricts the ability of researchers to train high-quality AI models specific to this population. AI models focused on the pediatric population have also been shown to have lower rates of clinically significant outcomes relative to the adult population, which can be explained by the limited amount of data.11,12 It is plausible that the field may be slowed down in the “research phase” with an attempt to solve these issues, resulting in more type 2 articles being published. Additionally, the focus on oncology as a topic of research can be explained by multiple reasons such as the contribution of cancers as the second-most leading cause of deaths worldwide and thus a wide availability of data, but also due to the fact that AI articles focusing on cancers tend to be highly cited and this can act as an additional motivating factor.13 Nonetheless, these findings illustrate areas in which institutions, authors, and journals can fill gaps in the literature, bringing the field closer to clinically significant and feasible AI/ML solutions that could ultimately enhance patient care as a whole, but particularly for pediatric and non-oncology patients.
Furthermore, our analysis across type 2 articles demonstrated that data were derived from only 12 nations with approximately one-half of these articles stemming from the United States indicating geographic centralization. This is consistent with the fact that one-third of all publications on AI in medicine and one-half of all AJNR publications originate from the United States.14 The second largest number of analyzed publications derived from South Korea, which differs from previous AI in medicine data.1 The centralized nature of the articles and the minimal use of public data sources emphasize the need for increased exposure and education in AI and ML for medical trainees, researchers, and physicians around the world. This would hopefully increase international collaboration, propelling the field forward with an increased availability of new and diverse ideas and best practices, but also allow researchers to externally validate and test their models and increase the size of their data sets, increasing the chances of outcomes being replicable, reliable, generalizable, and clinically significant, ultimately promoting the translation of research into practice.
Our results showed an average of 9 to 10 authors per article, which is larger than the 5 and 8 authors reported for publications on AI in medicine and AJNR, respectively.1,14 This can partly be explained by the need for larger data sets, external validation, and testing samples, therefore encouraging researchers to seek out collaborators at other institutions. It also highlights the increasing need for multidisciplinary collaboration among researchers. Additionally, we found that radiologists led 63% of the publications and interestingly, 60% held additional degrees such as PhDs and MBAs. These PhD and MBA degree-holding radiologists have a broader knowledge base and may indicate pre-existing technical skills in domains such as computer science, or an interest in entrepreneurial ventures and health systems leadership. These results demonstrate that although there is room for improvement, the field of neuroradiology is currently doing well with respect to authors’ collaboration and diversity of expertise and training backgrounds.
Our evaluation of type 2 articles revealed weaknesses that may slow down the advancement of the field, and thus implementation of AI/ML solutions into everyday practice. Results showed that only a minority of articles we analyzed were prospective, multicentered, or externally validated, which are important indicators for a real-world clinical application of an AI model.5,15 The lack of external validation may be related to limited data access, lack of time or funding, but also potential misprioritization focused on proving a theory-of-concept rather than the production of a practical solution that has the potential to be implemented. Researchers may find it easier to conduct and ultimately publish type 2 compared with type 3 articles because publishing the latter would require a functional AI/ML tool that can be applied in some way. Going forward, this finding represents an area in which journal publishers and editors can contribute to the field by encouraging the implementation of important indicators of generalizability into study methodologies.
The trust that is required for an effective therapeutic relationship between the patient and health care system is also a requirement for AI applications. In our assessment of potential bias and explainability, we found that only 22% and 16% of articles adequately addressed bias and explainability, respectively. We did however find that type 2 articles published since 2020 were slightly more likely to have adequately addressed bias and explainability. Although attention toward these important topics has been increasing recently, there is marked room for improvement before trust in AI is established, let alone widely integrated into our everyday health care.16
Our results and their interpretation are limited to AJNR publications. It is possible that findings across all relevant medical imaging journals, or other methodologic designs such as a qualitative study examining the experiences and perceptions of AI/ML among neuroradiologists may yield different conclusions. Article inclusion and categorization relied upon published definitions the authors used, and this can ultimately affect the results and analysis of this study. The factors we found to be limiting or contributing to the advancement of the field toward more integration-focused research was primarily gleaned from our analysis of type 2 articles. However, we acknowledge factors outside our scope of analysis such as regulatory challenges, funding constraints, and medico-legal issues may also hinder AI/ML integration.
CONCLUSIONS
Our investigation confirmed the rapid growth of AI research in neuroradiology. An in-depth analysis demonstrated that AI/ML developmental research work without direct clinical integration is the primary driver of publication growth. Improving the quality of this research work will be crucial to promote their integration into day-to-day practice. These results ultimately provide authors, editors, clinicians, and policymakers important insights to promote a shift toward integrating practical AI/ML solutions in neuroradiology.
Footnotes
↵# Alexandre Boutet and Samuel S. Haile are co-first authors that contributed equally.
↵ Farzad Khalvati and Birgit B. Ertl-Wagner are co-supervisors of this article.
Disclosure forms provided by the authors are available with the full text and PDF of this article at www.ajnr.org.
References
- Received October 18, 2023.
- Accepted after revision March 19, 2024.
- © 2024 by American Journal of Neuroradiology
Jump to section
Related Articles
Cited By...
- No citing articles found.