Methodology for testing and monitoring artificial intelligence-based software for medical diagnostics

Cover Image

Cite item

Full Text

Abstract

BACKGROUND: The global amount of investment in companies developing artificial intelligence (AI)-based software technologies for medical diagnostics reached $80 million in 2016, rose to $152 million in 2017, and is expected to continue growing. While software manufacturing companies should comply with existing clinical, bioethical, legal, and methodological frameworks and standards, there is a lack of uniform national and international standards and protocols for testing and monitoring AI-based software.

AIM: This objective of this study is to develop a universal methodology for testing and monitoring AI-based software for medical diagnostics, with the aim of improving its quality and implementing its integration into practical healthcare.

MATERIALS AND METHODS: The research process involved an analytical phase in which a literature review was conducted on the PubMed and eLibrary databases. The practical stage included the approbation of the developed methodology within the framework of an experiment focused on the use of innovative technologies in the field of computer vision to analyze medical images and further application in the health care system of the city of Moscow.

RESULTS: A methodology for testing and monitoring AI-based software for medical diagnostics has been developed, aimed at improving its quality and introducing it into practical healthcare. The methodology consists of seven stages: self-testing, functional testing, calibration testing, technological monitoring, clinical monitoring, feedback, and refinement.

CONCLUSION: Distinctive features of the methodology include its cyclical stages of monitoring and software development, leading to continuous improvement of its quality, the presence of detailed requirements for the results of the software work, and the participation of doctors in software evaluation. The methodology will allow software developers to achieve significant outcomes and demonstrate achievements across various areas. It also empowers users to make informed and confident choices among software options that have passed an independent and comprehensive quality check.

About the authors

Yuri A. Vasiliev

Moscow Center for Diagnostics and Telemedicine

Email: VasilevYA1@zdrav.mos.ru
ORCID iD: 0000-0002-0208-5218
SPIN-code: 4458-5608

MD, Cand. Sci. (Med.)

Russian Federation, Moscow

Anton V. Vlazimirsky

Moscow Center for Diagnostics and Telemedicine

Email: VladzimirskijAV@zdrav.mos.ru
ORCID iD: 0000-0002-2990-7736
SPIN-code: 3602-7120

MD, Dr. Sci. (Med.)

Russian Federation, Moscow

Olga V. Omelyanskaya

Moscow Center for Diagnostics and Telemedicine

Email: OmelyanskayaOV@zdrav.mos.ru
ORCID iD: 0000-0002-0245-4431
SPIN-code: 8948-6152
Russian Federation, Moscow

Kirill M. Arzamasov

Moscow Center for Diagnostics and Telemedicine

Email: ArzamasovKM@zdrav.mos.ru
ORCID iD: 0000-0001-7786-0349
SPIN-code: 3160-8062

MD, Cand. Sci. (Med.)

Russian Federation, Moscow

Sergey F. Chetverikov

Moscow Center for Diagnostics and Telemedicine

Email: ChetverikovSF@zdrav.mos.ru
ORCID iD: 0000-0002-3097-8881
SPIN-code: 3815-8870

Cand. Sci. (Engin.)

Russian Federation, Moscow

Denis A. Rumyantsev

Moscow Center for Diagnostics and Telemedicine

Author for correspondence.
Email: x.radiology@mail.ru
ORCID iD: 0000-0001-7670-7385
SPIN-code: 8734-2085
Russian Federation, Moscow

Maria A. Zelenova

Moscow Center for Diagnostics and Telemedicine

Email: ZelenovaMA@zdrav.mos.ru
ORCID iD: 0000-0001-7458-5396
SPIN-code: 3823-6872
Russian Federation, Moscow

References

  1. Oakden-Rayner L, Palme LJ. Artificial intelligence in medicine: Validation and study design. In: Ranschart E, Morozov S, Algra P, eds. Artificial intelligence in medical imaging. Cham: Springer; 2019. Р. 83–104.
  2. Morozov SP, Zinchenko VV, Khoruzhaya AN, et al. Standardization of artificial intelligence in healthcare: Russia is becoming a leader. Doctor Inform Technol. 2021;(2):12–19. (In Russ). doi: 10.25881/18110193_2021_2_12
  3. Mello AA, Utkin LV, Trofimova TN. Artificial intelligence in medicine: The current state and main directions of development of intellectual diagnostics. Radiation Diagnost Therapy. 2020;(1):9–17. (In Russ). doi: 10.22328/2079-5343-2020-11-1-9-17
  4. Zinchenko VV, Arzamasov KM, Chetverikov SF, et al. Methodology of post-registration clinical monitoring for software using artificial intelligence technologies. Modern Technol Med. 2022;14(5):15–25. (In Russ). doi: 10.17691/stm2022.14.5.02
  5. Tanguay W, Acar P, Fine B, et al. Assessment of radiology artificial intelligence software: A validation and evaluation framework. Can Assoc Radiol J. 2023;74(2):326–333. doi: 10.1177/08465371221135760
  6. Kohli A, Jha S. Why CAD failed in mammography. J Am Coll Radiol. 2018;15(3 Pt B):535–537. doi: 10.1016/j.jacr.2017.12.029
  7. Recht MP, Dewey M, Dreyer K, et al. Integrating artificial intelligence into the clinical practice of radiology: Challenges and recommendations. Eur Radiol. 2020;30(6):3576–3584. doi: 10.1007/s00330-020-06672-5
  8. Higgins DC, Johner C. Validation of artificial intelligence containing products across the regulated healthcare industries. Ther Innov Regul Sci. 2023;57(4):797–809. doi: 10.1007/s43441-023-00530-4
  9. Rudolph J, Schachtner B, Fink N, et al. Clinically focused multi-cohort benchmarking as a tool for external validation of artificial intelligence algorithm performance in basic chest radiography analysis. Sci Rep. 2022;12(1):12764. doi: 10.1038/s41598-022-16514-7
  10. Allen B, Dreyer K, Stibolt R, et al. Evaluation and real-world performance monitoring of artificial intelligence models in clinical practice: Try it, buy it, check it. J Am Coll Radiol. 2021;18(11):1489–1496. doi: 10.1016/j.jacr.2021.08.022
  11. Strohm L, Hehakaya C, Ranschaert ER, et al. Implementation of artificial intelligence (AI) applications in radiology: Hindering and facilitating factors. Eur Radiol. 2020;30(10):5525–5532. doi: 10.1007/s00330-020-06946-y
  12. Sohn JH, Chillakuru YR, Lee S, et al. An open-source, vender agnostic hardware and software pipeline for integration of artificial intelligence in radiology workflow. J Digit Imaging. 2020;33(4):1041–1046. doi: 10.1007/s10278-020-00348-8
  13. Wichmann JL, Willemink MJ, De Cecco CN. Artificial intelligence and machine learning in radiology: Current state and considerations for routine clinical implementation. Invest Radiol. 2020;55(9):619–627. doi: 10.1097/RLI.0000000000000673
  14. Larson DB, Harvey H, Rubin DL, et al. Regulatory frameworks for development and evaluation of artificial intelligence-based diagnostic imaging algorithms: Summary and recommendations. J Am Coll Radiol. 2021;18(3 Pt A):413–424. doi: 10.1016/j.jacr.2020.09.060
  15. Milam ME, Koo CW. The current status and future of FDA-approved artificial intelligence tools in chest radiology in the United States. Clin Radiol. 2023;78(2):115–122. doi: 10.1016/j.crad.2022.08.135
  16. De Silva D, Alahakoon D. An artificial intelligence life cycle: From conception to production. Patterns (NY). 2022;3(6):100489. doi: 10.1016/j.patter.2022.100489
  17. Cerdá-Alberich L, Solana J, Mallol P, et al. MAIC-10 brief quality checklist for publications using artificial intelligence and medical images. Insights Imaging. 2023;14(1):11. doi: 10.1186/s13244-022-01355-9
  18. Vasey B, Novak A, Ather S, et al. DECIDE-AI: A new reporting guideline and its relevance to artificial intelligence studies in radiology. Clin Radiol. 2023;78(2):130–136. doi: 10.1016/j.crad.2022.09.131
  19. Regulations for the preparation of data sets with a description of approaches to the formation of a representative sample of data. Moscow: Scientific and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Department of Health of the City of Moscow; 2022. 40 p. (Best practices in radiological and instrumental diagnostics; Part 1). (In Russ).
  20. Chetverikov S, Arzamasov KM, Andreichenko AE, et al. Approaches to sampling for quality control of artificial intelligence systems in biomedical research. Modern Technol Med. 2023;15(2):19–27. (In Russ). doi: 10.17691/stm2023.15.2.02
  21. Morozov SP, Vladzimirsky AV, Klyashtorny VG, et al. Clinical trials of software based on intelligent technologies (radiation diagnostics). Moscow: Scientific and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Department of Health of the City of Moscow; 2019. 33 р. (In Russ).
  22. Kim DW, Jang HY, Kim KW, et al. Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: Results from recently published papers. Korean J Radiol. 2019;20(3):405–410. doi: 10.3348/kjr.2019.0025

Supplementary files

Supplementary Files
Action
1. JATS XML
Download
2. Fig. 1. Methodology for testing and monitoring artificial intelligence–based software for medical diagnostics.

Download (143KB)
Indexing metadata
3. Fig. 2. Main components of the result of using artificial intelligence–based software with images: A reference example.

Download (184KB)
Indexing metadata
4. Fig. 3. Main components of the result of using artificial intelligence–based software with DICOM SR: A reference example.

Download (247KB)
Indexing metadata
5. Fig. 4. Image clipping of additional series of artificial intelligence–based software: Critical noncompliance with basic functional requirements.

Download (90KB)
Indexing metadata
6. Fig. 5. Overlaying caption texts on images: Critical noncompliance with basic functional requirements.

Download (147KB)
Indexing metadata
7. Fig. 6. Example of a calibration test protocol.

Download (279KB)
Indexing metadata
8. Fig. 7. Form of an internal report on monitoring the operation of artificial intelligence–based software.

Download (337KB)
Indexing metadata
9. Fig. 8. Changes of technological software defects for “chest radiography” modality.

Download (83KB)
Indexing metadata
10. Fig. 9. Example of a technology monitoring report.

Download (200KB)
Indexing metadata
11. Fig. 10. False negative (the subsegmental atelectasis is not detected in the lower lobe of the right lung): Noncritical noncompliance with basic diagnostic requirements.

Download (122KB)
Indexing metadata
12. Fig.11. A feedback window in the user interface.

Download (120KB)
Indexing metadata

Copyright (c) 2023 Eco-Vector

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Согласие на обработку персональных данных

 

Используя сайт https://journals.rcsi.science, я (далее – «Пользователь» или «Субъект персональных данных») даю согласие на обработку персональных данных на этом сайте (текст Согласия) и на обработку персональных данных с помощью сервиса «Яндекс.Метрика» (текст Согласия).