The quality of artificial intelligence algorithms for identifying manifestations of multiple sclerosis on magnetic resonance imaging (systematic review)

Cover Page

Cite item

Full Text

Abstract

A systematic review was undertaken to summarize the data regarding accuracy and effectiveness of artificial intelligence algorithms for identifying MRI manifestations of multiple sclerosis. The review included 39 papers, whose authors put forth a multitude of corresponding algorithms and mathematical models. However, quality assessment of these developments was limited by retrospective testing on repeat data sets. Clinical test results were almost entirely absent, and there were no prospective independent studies of accuracy and applicability. The relatively high values obtained for the main measures (similarity, sensitivity and specificity coefficients, which were 75–85%) were offset by the methodological errors when creating the baseline data sets, and lack of validation using independent data. Due to small sample sizes and methodological errors when measuring the result accuracy, most of the studies did not meet the criteria for evidence-based research. Studies with the highest methodological quality had algorithms that achieved a sensitivity of 51.6–77.0%, with a Sørensen–Dice coefficient of 53.5–56.0%. These numbers are not high, but they indicate that automatic identification of multiple sclerosis manifestations on magnetic resonance imaging may be achievable. Further development of computer-aided analysis requires the creation of clinical use scenarios and testing methodology, and prospective clinical testing.

About the authors

Galina N. Chernyaeva

Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies

Email: a.vladzimirsky@npcmr.ru
https://orcid.org/0000-0002-5066-5997

junior researcher

Russian Federation, Moscow

Sergey P. Morozov

Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies

Email: a.vladzimirsky@npcmr.ru
ORCID iD: 0000-0001-6545-6170
https://orcid.org/0000-0001-6545-6170

D. Sci. (Med), Prof., Director

Russian Federation, Moscow

Anton V. Vladzimirskyy

Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies; I.M. Sechenov First Moscow State Medical University (Sechenov University)

Author for correspondence.
Email: a.vladzimirsky@npcmr.ru
ORCID iD: 0000-0002-2990-7736
https://orcid.org/0000-0002-2990-7736

D. Sci. (Med), Deputy Director for R&D

Russian Federation, Moscow; Moscow

References

  1. Abdurakhmanova R.F., Izzatov Kh.N., Khadibaeva G.R. et al. Multiple sclerosis: etiology, pathogenesis and clinics (part I). Vestnik poslediplomnogo obrazo- vaniya v sfere zdravookhraneniya. 2016;(3):68-74. (In Russ).
  2. Popova E.V., Boyko A.N., Barabanova M.A. et al. Primary progressive multiple sclerosis: current issues of timely diagnosis. Zhurnal nevrologii i psikhiatrii im. S.S. Korsakova. 2017;117(10-12):35-40. (In Russ). doi: 10.17116/jnev- ro201711710235-40.
  3. Howard J., Trevick S., Younger D.S. Epidemiology of multiple sclerosis. Neurol Clin. 2016;34(4):919-939. doi: 10.1016/j.ncl.2016.06.016. PMID: 27720001.
  4. Borodin A.V. Differential diagnostics of multiple sclerosis. Prakticheskaya meditsina. 2018;(10):59-63. (In Russ).
  5. Siva A. Common clinical and imaging conditions misdiagnosed as multiple sclerosis: a current approach to the differential diagnosis of multiple sclerosis. Neurol Clin. 2018;36(1):69-117. doi: 10.1016/j.ncl.2017.08.014. PMID: 29157405.
  6. Solomon A.J., Naismith R.T., Cross A.H. Misdiagnosis of multiple sclerosis: impact of the 2017 McDonald criteria on clinical practice. Neurology. 2019;92(1):26-33. doi: 10.1212/WNL.0000000000006583. PMID: 30381369.
  7. Morozov S.P., Vladzymyrsky A.V., Klyashtorny V.G. et al. Clinical studies of intelligence technology-based software (radiology). Moscow, 2019. 33 p. (In Russ).
  8. Udupa J.K., Wei L., Samarasekera S., Miki Y. et al. Multiple sclerosis lesion quantification using fuzzy-connectedness principles. IEEE Trans Med Imaging. 1997;16(5):598-609. doi: 10.1109/42.640750. PMID: 9368115.
  9. Datta S., Sajja B.R., He R. et al. Segmentation and quantification of black holes in multiple sclerosis. Neuroimage. 2006;29(2):467-474. DOI: 10.1016/j. neuroimage.2005.07.042. PMID: 16126416.
  10. He R., Sajja B.R., Narayana P.A. Implementation of high-dimensional feature map for segmentation of MR images. Ann Biomed Eng. 2005;33(10):1439- 1448. doi: 10.1007/s10439-005-5888-3. PMID: 16240091.
  11. Kawa J., Pietka E. Kernelized fuzzy c-means method in fast segmentation of demyelination plaques in multiple sclerosis. Conf Proc IEEE Eng Med Biol Soc. 2007;2007:5616-5619. doi: 10.1109/IEMBS.2007.4353620. PMID: 18003286.
  12. Ait-Ali L.S., Prima S., Hellier P. et al. STREM: a robust multidimensional parametric method to segment MS lesions in MRI. Med Image Comput Comput Assist Interv. 2005; 8(Pt 1):409-416. doi: 10.1007/11566465_51. PMID: 16685872.
  13. Raff U., Newman F.D. Automated lesion detection and lesion quantitation in MR images using autoassociative memory. Med Phys. 1992;19(1):71-77. doi: 10.1118/1.596853. PMID: 1620061.
  14. Raff U., Newman F.D. Lesion detection in radiologic images using an autoassociative paradigm: preliminary results. Med Phys. 1990; 17(5):926-928. doi: 10.1118/1.596449. PMID: 2233581.
  15. Chase H.S., Mitrani L.R., Lu G.G., Fulgieri D.J. Early recognition of multiple sclerosis using natural language processing of the electronic health record. BMC Med Inform Decis Mak. 2017;17(1):24. doi: 10.1186/s12911-017-0418-4.
  16. Nelson R.E., Butler J., LaFleur J. et al. Determining multiple sclerosis phenotype from electronic medical records. J Manag Care Spec Pharm. 2016;22(12):1377-1382. doi: 10.18553/jmcp.2016.22.12.1377. PMID: 27882837.
  17. Davis M.F., Sriram S., Bush W.S. et al. Automated extraction of clinical traits of multiple sclerosis in electronic medical records. J Am Med Inform Assoc. 2013;20(e2):e334-e340. doi: 10.1136/amiajnl-2013-001999. PMID: 24148554.
  18. Wottschel V., Alexander D.C., Kwok P.P. et al. Predicting outcome in clinically isolated syndrome using machine learning. Neuroimage Clin. 2014;7:281-287. doi: 10.1016/j.nicl.2014.11.021. PMID: 25610791.
  19. Zhao Y., Healy B.C., Rotstein D. et al. Exploration of machine learning techniques in predicting multiple sclerosis disease course. PLoS One. 2017;12(4):e0174866. doi: 10.1371/journal.pone.0174866. PMID: 28379999.
  20. Wu F.Y., Slater J.D., Honig L.S. et al. A neural network design for event-related potential diagnosis. Comput Biol Med. 1993;23(3):251-264. doi: 10.1016/0010-4825(93)90024-u. PMID: 8334865.
  21. Ahmadi A., Davoudi S., Daliri M.R. Computer Aided Diagnosis System for multiple sclerosis disease based on phase to amplitude coupling in covert visual attention. Comput Methods Programs Biomed. 2019;169:9-18. DOI: 10.1016/j. cmpb.2018.11.006. PMID: 30638593.
  22. Kiiski H., Jollans L., Donnchadha S.O. et al. Machine learning EEG to predict cognitive functioning and processing speed over a 2-year period in multiple sclerosis patients and controls. Brain Topogr. 2018;31(3):346-363. doi: 10.1007/s10548-018-0620-4. PMID: 29380079.
  23. Liberati A., Altman D.G., Tetzlaff J. et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62(10):e1- e34. doi: 10.1016/j.jclinepi.2009.06.006. PMID: 19631507.
  24. Wang S.H., Tang C., Sun J. et al. Multiple sclerosis identification by 14-layer convolutional neural network with batch normalization, dropout, and stochastic pooling. Front Neurosci. 2018;12:818. doi: 10.3389/fnins.2018.00818. PMID: 30467462.
  25. Zurita M., Montalba C., Labbe T. et al. Characterization of relapsing-remitting multiple sclerosis patients using support vector machine classifications of functional and diffusion MRI data. Neuroimage Clin. 2018;20:724-730. doi: 10.1016/j.nicl.2018.09.002. PMID: 30238916.
  26. Yoo Y., Tang L.Y.W., Brosch T. et al. Deep learning of joint myelin and T1w MRI features in normal-appearing brain tissue to distinguish between multiple sclerosis patients and healthy controls. Neuroimage Clin. 2017;17:169-178. doi: 10.1016/j.nicl.2017.10.015. PMID: 29071211.
  27. Nakamura K., Fisher E. Segmentation of brain magnetic resonance images for measurement of gray matter atrophy in multiple sclerosis patients. Neuroimage. 2009;44(3):769-776. doi: 10.1016/j.neuroimage.2008.09.059. PMID: 19007895.
  28. Hackmack K., Paul F., Weygandt M. et al. Alzheimer’s disease neuroimaging initiative. Multi-scale classification of disease using structural MRI and wavelet transform. Neuroimage. 2012;62(1):48-58. doi: 10.1016/j.neuroim- age.2012.05.022. PMID: 22609452.
  29. Wei W., Poirion E., Bodini B. et al. Predicting PET-derived demyelination from multimodal MRI using sketcher-refiner adversarial training for multiple sclerosis. Med Image Anal. 2019;58:101546. doi: 10.1016/j.media.2019.101546. PMID: 31499318.
  30. Theocharakis P., Glotsos D., Kalatzis I. et al. Pattern recognition system for the discrimination of multiple sclerosis from cerebral microangiopathy lesions based on texture analysis of magnetic resonance images. Magn Reson Imaging. 2009;27(3):417-422. doi: 10.1016/j.mri.2008.07.014. PMID: 18786795.
  31. Crimi A., Commowick O., Maarouf A. et al. Predictive value of imaging markers at multiple sclerosis disease onset based on gadolinium- and USPIO-enhanced MRI and machine learning. PLoS One. 2014;9(4):e93024. DOI: 10.1371/ journal.pone.0093024. PMID: 24691080.
  32. Aslani S., Dayan M., Storelli L. et al. Multi-branch convolutional neural network for multiple sclerosis lesion segmentation. Neuroimage. 2019;196:1-15. doi: 10.1016/j.neuroimage.2019.03.068. PMID: 30953833.
  33. McKinley R., Wepfer R., Grunder L. et al. Automatic detection of lesion load change in Multiple Sclerosis using convolutional neural networks with segmentation confidence. Neuroimage Clin. 2020;25:102104. DOI: 10.1016/j. nicl.2019.102104. PMID: 31927500.
  34. Jain S., Ribbens A., Sima D.M. et al. D. Two time point MS lesion segmentation in brain MRI: an expectation-maximization framework. Front Neurosci. 2016;10:576. doi: 10.3389/fnins.2016.00576. PMID: 28066162.
  35. Cabezas M., Oliver A., Valverde S. et al. BOOST: a supervised approach for multiple sclerosis lesion segmentation. J Neurosci Methods. 2014;237:108-117. doi: 10.1016/j.jneumeth.2014.08.024. PMID: 25194638.
  36. Gabr R.E., Coronado I., Robinson M. et al. Brain and lesion segmentation in multiple sclerosis using fully convolutional neural networks: A large-scale study. Mult Scler. 2020;26(10): 1217-1226. doi: 10.1177/1352458519856843. PMID: 31190607.
  37. Brosch T., Tang L.Y., Youngjin Yoo et al. Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation. IEEE Trans Med Imaging. 2016;35(5):1229-1239. doi: 10.1109/TMI.2016.2528821. PMID: 26886978.
  38. Valverde S., Cabezas M., Roura E. et al. Improving automated multiple sclerosis lesion segmentation with a cascaded 3D convolutional neural network approach. Neuroimage. 2017;155:159-168. doi: 10.1016/j.neuroimage.2017.04.034. PMID: 28435096.
  39. Bossuyt P.M., Reitsma J.B., Bruns D.E. et al. STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies. Radiology. 2015;277(3):826-832. doi: 10.1148/radiol.2015151516. PMID: 26509226.
  40. Yamamoto D., Arimura H., Kakeda S. et al. Computer-aided detection of multiple sclerosis lesions in brain magnetic resonance images: false positive reduction scheme consisted of rule-based, level set method, and support vector machine. Comput Med Imaging Graph. 2010;34(5):404-413. DOI: 10.1016/j. compmedimag.2010.02.001. PMID: 20189353.
  41. Goldberg-Zimring D., Achiron A., Miron S. et al. Automated detection and characterization of multiple sclerosis lesions in brain MR images. Magn Reson Imaging. 1998;16(3):311-318. doi: 10.1016/s0730-725x(97)00300-7. PMID: 9621972.
  42. Bendfeldt K., Kloppel S., Nichols T.E. et al. Multivariate pattern classification of gray matter pathology in multiple sclerosis. Neuroimage. 2012;60(1):400- 408. doi: 10.1016/j.neuroimage.2011.12.070. PMID: 22245259.
  43. Salem M., Cabezas M., Valverde S. et al. A supervised framework with intensity subtraction and deformation field features for the detection of new T2-w lesions in multiple sclerosis. Neuroimage Clin. 2017;17:607-615. DOI: 10.1016/j. nicl.2017.11.015. PMID: 29234597.
  44. Boudraa A.O., Dehak S.M., Zhu Y.M. et al. Automated segmentation of multiple sclerosis lesions in multispectral MR imaging using fuzzy clustering. Comput Biol Med. 2000;30(1):23-40. doi: 10.1016/s0010-4825(99)00019-0. PMID: 10695813.
  45. Geremia E., Menze B.H., Clatz O. et al. Spatial decision forests for MS lesion segmentation in multi-channel MR images. Med Image Comput Comput Assist Interv. 2010;13(Pt 1):111-118. doi: 10.1007/978-3-642-15705-9_14. PMID: 20879221.
  46. Shiee N., Bazin P.L., Ozturk A. et al. A topology-preserving approach to the segmentation of brain images with multiple sclerosis lesions. Neuroimage. 2010;49(2):1524-1535. doi: 10.1016/j.neuroimage.2009.09.005. PMID: 19766196.
  47. Garcia-Lorenzo D., Lecoeur J., Arnold D.L. et al. Multiple sclerosis lesion segmentation using an automatic multimodal graph cuts. Med Image Comput Comput Assist Interv. 2009;12(Pt 2):584-591. doi: 10.1007/978-3-642-04271- 3_71. PMID: 20426159.
  48. Salem M., Cabezas M., Valverde S. et al. A supervised framework with intensity subtraction and deformation field features for the detection of new T2-w lesions in multiple sclerosis. Neuroimage Clin. 2017;17:607-615. DOI: 10.1016/j. nicl.2017.11.015. PMID: 29234597.
  49. Khayati R., Vafadust M., Towhidkhah F. et al. A novel method for automatic determination of different stages of multiple sclerosis lesions in brain MR FLAIR images. Comput Med Imaging Graph. 2008;32(2):124-133. PMID: 18055174.
  50. Weiss N., Rueckert D., Rao A. Multiple sclerosis lesion segmentation using dictionary learning and sparse coding. Med Image Comput Comput Assist Interv. 2013;16(Pt 1):735-742. doi: 10.1007/978-3-642-40811-3_92. PMID: 24505733.
  51. Gao J., Li C., Feng C. et al. Non-locally regularized segmentation of multiple sclerosis lesion from multi-channel MRI data. Magn Reson Imaging. 2014;32(8):1058-1066. doi: 10.1016/j.mri.2014.03.006. PMID: 24948583.
  52. Bijar A., Khayati R., Penalver Benavent A. Increasing the contrast of the brain MR FLAIR images using fuzzy membership functions and structural similarity indices in order to segment MS lesions. PLoS One. 2013;8(6):e65469. doi: 10.1371/journal.pone.0065469. PMID: 23799015.
  53. Cerasa A., Bilotta E., Augimeri A. et al. A Cellular Neural Network methodology for the automated segmentation of multiple sclerosis lesions. J Neurosci Methods. 2012;203(1):193-199. doi: 10.1016/j.jneumeth.2011.08.047. PMID: 21920384.
  54. Aymerich F.X., Sobrevilla P., Montseny E. et al. Fuzzy approach toward reducing false positives in the detection of small multiple sclerosis lesions in magnetic resonance images. Conf Proc IEEE Eng Med Biol Soc. 2011;2011:5694- 5697. doi: 10.1109/IEMBS.2011.6091378. PMID: 22255632.
  55. Kuwazuru J., Arimura H., Kakeda S. et al. Automated detection of multiple sclerosis candidate regions in MR images: false-positive removal with use of an ANN-controlled level-set method. Radiol Phys Technol. 2012;5(1):105-113. doi: 10.1007/s12194-011-0141-2. PMID: 22139608.
  56. Kanber B., Nachev P., Barkhof F. et al. High-dimensional detection of imaging response to treatment in multiple sclerosis. NPJ Digit Med. 2019;2:49. doi: 10.1038/s41746-019-0127-8. PMID: 31304395.
  57. Freire P.GL, Ferrari R.J. Multiple sclerosis lesion enhancement and white matter region estimation using hyperintensities in FLAIR images. arXiv. 2018:1807.09619.
  58. Feng Y., Pan H., Meyer C. et al. A self-adaptive network for multiple sclerosis lesion segmentation from multi-contrast MRI with various imaging protocols. arXiv. 2018:1811.07491.
  59. Kazancli E., Prchkovska V., Rodrigues P. et al. Multiple sclerosis lesion segmentation using improved convolutional neural networks. In: Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2018). 2018; 4; VISAPP: 260269. doi: 10.5220/0006540902600269.
  60. Roy S., Butman J.A., Reich D.S. et al. Multiple sclerosis lesion segmentation from brain MRI via fully convolutional neural networks. arXiv. 2018:1803.09172v1.
  61. Sacca V, Sarica A., Novellino F. et al. Evaluation of machine learning algorithms performance for the prediction of early multiple sclerosis from resting-state FMRI connectivity data. Brain Imaging Behav. 2019;13(4):1103-1114. doi: 10.1007/s11682-018-9926-9. PMID: 29992392.
  62. Zhang H., Zhang J., Zhang Q. et al. RSANet: Recurrent Slice-wise Attention Network for multiple sclerosis lesion segmentation. arXiv; 2020:2002.12470v1.
  63. Gheshlaghi S.H., Ranjbar A., Suratgar A.A. et al. A superpixel segmentation based technique for multiple sclerosis lesion detection. arXiv; 2020:1907.03109v1.

Supplementary files

Supplementary Files
Action
1. JATS XML
Download
2. Article selection process for the systematic review.

Download (116KB)
Indexing metadata

Copyright (c) 2021 Chernyaeva G.N., Morozov S.P., Vladzimirskyy A.V.

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

This website uses cookies

You consent to our cookies if you continue to use our website.

About Cookies