MarkIt: A Collaborative Artificial Intelligence Annotation Platform Leveraging Blockchain For Medical Imaging Research

Authors

  • Jan Witowski Laboratory of Medical Imaging and Computation, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA https://orcid.org/0000-0001-9284-4830
  • Jongmun Choi Laboratory of Medical Imaging and Computation, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA
  • Soomin Jeon Laboratory of Medical Imaging and Computation, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA https://orcid.org/0000-0003-1009-8227
  • Doyun Kim Laboratory of Medical Imaging and Computation, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA https://orcid.org/0000-0001-5347-3176
  • Joowon Chung Laboratory of Medical Imaging and Computation, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA https://orcid.org/0000-0003-1572-8056
  • John Conklin Division of Emergency Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA https://orcid.org/0000-0001-9921-2086
  • Maria Gabriela Figueiro Longo Division of Emergency Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA https://orcid.org/0000-0001-6819-4890
  • Marc D. Succi Medically Engineered Solutions in Healthcare (MESH) Incubator, Massachusetts General Hospital, Boston, MA, USA https://orcid.org/0000-0002-1518-3984
  • Synho Do Laboratory of Medical Imaging and Computation, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA https://orcid.org/0000-0001-6211-7050

DOI:

https://doi.org/10.30953/bhty.v4.176

Keywords:

artificial intelligence, data annotation, learning from crowds, blockchain, rewarding system

Abstract

Current research on medical image processing relies heavily on the amount and quality of input data. Specifically, supervised machine learning methods require well-annotated datasets. A lack of annotation tools limits the potential to achieve high-volume processing and scaled systems with a proper reward mechanism. We developed MarkIt, a web-based tool, for collaborative annotation of medical imaging data with artificial intelligence and blockchain technologies. Our platform handles both Digital Imaging and Communications in Medicine (DICOM) and non-DICOM images, and allows users to annotate them for classification and object detection tasks in an efficient manner. MarkIt can accelerate the annotation process and keep track of user activities to calculate a fair reward. A proof-of-concept experiment was conducted with three fellowship-trained radiologists, each of whom annotated 1,000 chest X-ray studies for multi-label classification. We calculated the inter-rater agreement and estimated the value of the dataset to distribute the reward for annotators using a crypto currency. We hypothesize that MarkIt allows the typically arduous annotation task to become more efficient. In addition, MarkIt can serve as a platform to evaluate the value of data and trade the annotation results in a more scalable manner in the future. The platform is publicly available for testing on https://markit.mgh.harvard.edu.

Downloads

Download data is not yet available.

References

Abaho M, Bollegala D, Williamson P, Dodd S. Correcting crowdsourced annotations to improve detection of outcome types in evidence based medicine. CEUR Workshop Proc [Internet]. 2019 [cited 10 March 2021]. Available from: https://livrepository.liverpool.ac.uk/3047267

Greenspan H, Van Ginneken B, Summers RM. Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans Med Imaging 2016; 35(5): 1153–9. doi: 10.1109/TMI.2016.2553401

Albarqouni S, Baur C, Achilles F, Belagiannis V, Demirci S, Navab N. AggNet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans Med Imaging 2016; 35(5): 1313–21. doi: 10.1109/TMI.2016.2528120

Abujudeh HH, Boland GW, Kaewlai R, Rabiner P, Halpern EF, Gazelle GS, et al. Abdominal and pelvic computed tomography (CT) interpretation: discrepancy rates among experienced radiologists. Eur Radiol 2010; 20(8): 1952–7. doi: 10.1007/s00330-010-1763-1

Raykar VC, Yu S, Zhao LH, Jerebko A, Florin C, Valadez GH, et al. Supervised learning from multiple experts: whom to trust when everyone lies a bit. Proc Int Conf Mach Learn [Internet]. 2009 [cited 10 March 2021]. Available from: http://portal.acm.org/citation.cfm?doid=1553374.1553488

Raykar VC, Yu S, Zhao LH, Valadez GH, Florin C, Bogoni L, et al. Learning from crowds. J Mach Learn Res 2010; 11(4): 1297–322.

Yan Y, Rosales R, Fung G, Schmidt M, Hermosillo G, Bogoni L, et al. Modeling annotator expertise: learning when everybody knows a bit of something. Proceedings of AISTATS [Internet]. 2010 [cited 10 March 2021]. Available from: http://proceedings.mlr.press/v9/yan10a.html

Raykar VC, Yu S. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. J Mach Learn Res 2012; 13(1): 491–518.

Tanno R, Saeedi A, Sankaranarayanan S, Alexander DC, Silberman N. Learning from noisy labels by regularized estimation of annotator confusion. IEEE/CVF CVPRW 2019 [Internet]. 2019 [cited 10 March 2021]. Available from: https://ieeexplore.ieee.org/abstract/document/8953406

Jodogne S. The Orthanc ecosystem for medical imaging. J Digit Imaging 2018; 31(3): 341–52. doi: 10.1007/s10278-018-0082-y

Irvin J, Rajpurkar P, Ko M, Yu Y, Ciurea-Ilcus S, Chute C, et al. Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. Proc Conf AAAI Artif Intell [Internet] 2019 [cited 10 March 2021]; 33(01): 590–7. Available from: https://ojs.aaai.org//index.php/AAAI/article/view/3834

Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. Chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Conf Comput Vis Pattern Recognit Workshops [Internet]. 2017 [cited 10 March 2021]. Available from: https://ieeexplore.ieee.org/document/8099852

Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-cam: visual explanations from deep networks via gradient-based localization. Proc IEEE Int Conf Comput Vis [Internet]. 2017 [cited 10 March 2021]. Available from: https://ieeexplore.ieee.org/document/8237336

Demirer M, Candemir S, Bigelow MT, Yu SM, Gupta V, Prevedello LM, et al. A user interface for optimizing radiologist engagement in image data curation for artificial intelligence. Radiol Artif Intell 2019; 1(6): e180095. doi: 10.1148/ryai.2019180095

Rubin DL, Ugur Akdogan M, Altindag C, Alkim E. ePAD: an image annotation and analysis platform for quantitative imaging. Tomography 2019; 5(1): 170–83. doi: 10.18383/j.tom.​2018.00055

Urban T, Ziegler E, Lewis R, Hafey C, Sadow C, Van den Abbeele AD, et al. LesionTracker: extensible open-source zero-footprint web viewer for cancer imaging research and clinical trials. Cancer Res 2017; 77(21): e119–22. doi: 10.1158/0008-5472.CAN-17-0334

Philbrick KA, Weston AD, Akkus Z, Kline TL, Korfiatis P, Sakinis T, et al. RIL-contour: a medical imaging dataset annotation tool for and with deep learning. J Digit Imaging 2019; 32(4): 571–81. doi: 10.1007/s10278-019-00232-0

Chen S, Guo J, Wang C, Xu X, Yi Z, Li W. DeepLNAnno: a web-based lung nodules annotating system for CT images. J Med Syst 2019; 43(7): 197. doi: 10.1007/s10916-019-1258-9

Abdullah S, Rothenberg S, Siegel E, Kim W. School of block-review of blockchain for the radiologists. Acad Radiol 2020; 27(1): 47–57. doi: 10.1016/j.acra.2019.06.025

Downloads

Additional Files

Published

2021-05-05

How to Cite

Witowski, J., Choi, J. ., Jeon, S., Kim, D., Chung, J., Conklin, J., Figueiro Longo, M. G. ., Succi, M. D., & Do, S. (2021). MarkIt: A Collaborative Artificial Intelligence Annotation Platform Leveraging Blockchain For Medical Imaging Research. Blockchain in Healthcare Today, 4. https://doi.org/10.30953/bhty.v4.176

Issue

Volume 4, 2021

Section

Proof of Concept/Pilots/Methodologies

License

Copyright (c) 2021 Jan Witowski, Jongmum Choi, Soomin Jeon, Doyun Kim, Joowon Chung, John Conklin, Maria Gabriela Figueiro Longo, Marc D. Succi, Synho Do

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Authors retain copyright of their work, with first publication rights granted to Blockchain in Healthcare Today (BHTY). Read the full Copyright Statement.