Explainable Deep Learning Methods in Medical Image Classification: A Survey

For an interactive version of the table of the survey, allowing sorting by year, interpretability method, and dataset, click here.

Citation

If you use this repository, please cite:

@article{patricio2023explainable,
  title={Explainable Deep Learning Methods in Medical Image Classification: A Survey},
  author={Patr{\'\i}cio, Cristiano and Neves, Jo{\~a}o C and Teixeira, Lu{\'\i}s F},
  journal={ACM Computing Surveys},
  volume={56},
  number={4},
  pages={1--41},
  year={2023}
}

Table of Contents:

• Survey Papers
• XAI Methods in Medical Diagnosis
  • Explanation by Feature Attribution
    • Perturbation-based methods
    • Saliency Methods
  • Explanation by Text
    • Image Captioning
    • Concept Attribution
  • Explanation by Examples
    • Case-Based Reasoning
    • Counterfactual Explanations
    • Prototypes
  • Explanation by Concepts
  • Other Approaches
    • Bayesian Approaches
    • Adversarial Training

Survey Papers

1. Milda Pocevičiute, Gabriel Eilertsen, and Claes Lundström
   Survey of XAI in Digital Pathology
   Artificial Intelligence and Machine Learning for Digital Pathology: State-of-the-Art and Future Challenges, 56–88, 2020
   📄 Paper

2. Erico Tjoa and Cuntai Guan
   A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
   IEEE Transactions on Neural Networks and Learning Systems 32, 11 (2020), 4793–4813
   📄 Paper

3. Amitojdeep Singh, Sourya Sengupta, and Vasudevan Lakshminarayanan
   Explainable Deep Learning Models in Medical Image Analysis
   Journal of Imaging 6, 6 (2020), 52
   📄 Paper

4. Mehmet A Gulum, Christopher M Trombley, and Mehmed Kantardzic
   A Review of Explainable Deep Learning Cancer Detection Models in Medical Imaging
   Applied Sciences 11, 10 (2021), 4573
   📄 Paper

5. Hareem Ayesha, Sajid Iqbal, Mehreen Tariq, Muhammad Abrar, Muhammad Sanaullah, Ishaq Abbas, Amjad Rehman, Muhammad Farooq Khan Niazi, and Shafiq Hussain
   Automatic medical image interpretation: State of the art and future directions
   Pattern Recognition 114 (2021), 107856
   📄 Paper

6. Zohaib Salahuddin, Henry C. Woodruff, Avishek Chatterjee, and Philippe Lambin
   Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods
   Computers in Biology and Medicine 140 (2022), 105111
   📄 Paper

7. Pablo Messina, Pablo Pino, Denis Parra, Alvaro Soto, Cecilia Besa, Sergio Uribe, Marcelo Andía, Cristian Tejos, Claudia Prieto, and Daniel Capurro
   A survey on deep learning and explainability for automatic report generation from medical images
   ACM Computing Surveys (CSUR) 54, 10s (2022), 1–40
   📄 Paper

XAI Methods in Medical Diagnosis

Explanation by Feature Attribution

Perturbation-based methods

1. Avleen Malhi, Timotheus Kampik, Husanbir Pannu, Manik Madhikermi, and Kary Främling
   Explaining Machine Learning-Based Classifications of In-Vivo Gastral Images
   Digital Image Computing: Techniques and Applications (DICTA), 1–7, 2019
   📄 Paper

2. Kyle Young, Gareth Booth, Becks Simpson, Reuben Dutton, and Sally Shrapnel
   Deep Neural Network or Dermatologist?
   Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support, 48–55, 2019
   📄 Paper

3. Fabian Eitel and Kerstin Ritter for the Alzheimer’s Disease Neuroimaging Initiative (ADNI)
   Testing the Robustness of Attribution Methods for Convolutional Neural Networks in MRI-Based Alzheimer’s Disease Classification
   Medical Image Computing and Multimodal Learning for Clinical Decision Support (IMIMIC), 3–11, 2019
   📄 Paper

4. Pavan Rajkumar Magesh, Richard Delwin Myloth, and Rijo Jackson Tom
   An Explainable Machine Learning Model for Early Detection of Parkinson’s Disease using LIME on DaTscan Imagery
   Computers in Biology and Medicine 126 (2020), 104041
   📄 Paper

5. Narinder Singh Punn and Sonali Agarwal
   Automated diagnosis of COVID-19 with limited posteroanterior chest X-ray images using fine-tuned deep neural networks
   Applied Intelligence 51, 5 (2021), 2689–2702
   📄 Paper

6. Sutong Wang, Yunqiang Yin, Dujuan Wang, Yanzhang Wang, and Yaochu Jin
   Interpretability-Based Multimodal Convolutional Neural Networks for Skin Lesion Diagnosis
   IEEE Transactions on Cybernetics (2021)
   📄 Paper

Saliency Methods

1. Pranav Rajpurkar, Jeremy Irvin, Robyn L Ball, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, et al .
   Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists
   PLoS Medicine 15, 11 (2018)
   📄 Paper

2. Rory Sayres, Ankur Taly, Ehsan Rahimy, Katy Blumer, David Coz, Naama Hammel, Jonathan Krause, Arunachalam Narayanaswamy, Zahra Rastegar, Derek Wu, et al.
   Using a Deep Learning Algorithm and Integrated Gradients Explanation to Assist Grading for Diabetic Retinopathy
   Ophthalmology 126, 4 (2019), 552–564
   📄 Paper

3. Tsung-Chieh Lin and Hsi-Chieh Lee
   Covid-19 Chest Radiography Images Analysis Based on Integration of Image Preprocess, Guided Grad-CAM, Machine Learning and Risk Management
   International Conference on Medical and Health Informatics (ICMHI), 281-288, 2020
   📄 Paper

4. Alina Lopatina, Stefan Ropele, Renat Sibgatulin, Jürgen R Reichenbach, and Daniel Güllmar
   Investigation of Deep-Learning- Driven Identification of Multiple Sclerosis Patients Based on Susceptibility-Weighted Images Using Relevance Analysis
   Frontiers in Neuroscience (2020), 1356
   📄 Paper

5. Isabel Rio-Torto, Kelwin Fernandes, and Luís Teixeira
   Understanding the decisions of CNNs: An in-model approach
   Pattern Recognition Letters 133 (2020), 373–380
   📄 Paper

Explanation by Text

Image Captioning

1. Zizhao Zhang, Yuanpu Xie, Fuyong Xing, Mason McGough, and Lin Yang
   MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network
   IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6428–6436, 2017
   📄 Paper

2. Baoyu Jing, Pengtao Xie, and Eric Xing
   On the Automatic Generation of Medical Imaging Reports
   56th Annual Meeting of the Association for Computational Linguistics (ACL), 2018
   📄 Paper

3. Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald Summers
   TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays
   IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 9049–9058, 2018
   📄 Paper

4. Christy Y Li, Xiaodan Liang, Zhiting Hu, and Eric P Xing
   Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
   International Conference on Neural Information Processing Systems (NIPS), 1537–1547, 2018
   📄 Paper

5. Li Sun, Weipeng Wang, Jiyun Li, and Jingsheng Lin
   Study on Medical Image Report Generation Based on Improved Encoding-Decoding Method
   Intelligent Computing Theories and Application, 686–696, 2019
   📄 Paper

6. Catarina Barata, Jorge S Marques, and M Emre Celebi
   Deep Attention Model for the Hierarchical Diagnosis of Skin Lesions
   IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2757–2765, 2019
   📄 Paper

7. Sonit Singh, Sarvnaz Karimi, Kevin Ho-Shon, and Len Hamey
   From Chest X-Rays to Radiology Reports: A Multimodal Machine Learning Approach
   Digital Image Computing: Techniques and Applications (DICTA), 1–8, 2019
   📄 Paper

8. Hyebin Lee, Seong Tae Kim, and Yong Man Ro
   Generation of Multimodal Justification Using Visual Word Constraint Model for Explainable Computer-Aided Diagnosis
   Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support, 21–29, 2019
   📄 Paper

9. William Gale, Luke Oakden-Rayner, Gustavo Carneiro, Andrew P Bradley, and Lyle J Palmer
   Producing Radiologist-Quality Reports for Interpretable Deep Learning
   IEEE International Symposium on Biomedical Imaging (ISBI), 1275–1279, 2019
   📄 Paper

10. Changchang Yin, Buyue Qian, Jishang Wei, Xiaoyu Li, Xianli Zhang, Yang Li, and Qinghua Zheng
    Automatic Generation of Medical Imaging Diagnostic Report with Hierarchical Recurrent Neural Network
    IEEE International Conference on Data Mining (ICDM), 728–737, 2019
    📄 Paper

11. Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi
    Clinically Accurate Chest X-Ray Report Generation
    Machine Learning for Healthcare Conference, 249–269, 2019
    📄 Paper

12. Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan
    Generating Radiology Reports via Memory-driven Transformer
    Conference on Empirical Methods in Natural Language Processing (EMNLP), 1439–1449, 2020
    📄 Paper

13. Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, and Yuexian Zou
    Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation
    IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13753–13762, 2021
    📄 Paper

14. Fenglin Liu, Changchang Yin, Xian Wu, Shen Ge, Ping Zhang, and Xu Sun
    Contrastive Attention for Automatic Chest X-ray Report Generation
    Findings of the Association for Computational Linguistics: ACL-IJCNLP, 269–280, 2021
    📄 Paper

15. Zhanyu Wang, Mingkang Tang, Lei Wang, Xiu Li, and Luping Zhou
    A Medical Semantic-Assisted Transformer for Radiographic Report Generation
    Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part III. 655–664
    📄 Paper

16. Zhanyu Wang, Lingqiao Liu, Lei Wang, and Luping Zhou
    METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens
    IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11558–11567, 2023
    📄 Paper

17. Zhanyu Wang, Lingqiao Liu, Lei Wang, and Luping Zhou
    METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens
    IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11558–11567, 2023
    📄 Paper

18. Alexander Selivanov, Oleg Y Rogov, Daniil Chesakov, Artem Shelmanov, Irina Fedulova, and Dmitry V Dylov
    Medical image captioning via generative pretrained transformers
    Scientific Reports 13, 1 (2023), 4171
    📄   Paper

Concept Attribution

1. Mara Graziani, Vincent Andrearczyk, Stéphane Marchand-Maillet, and Henning Müller
   Concept attribution: Explaining CNN decisions to physicians
   Computers in Biology and Medicine 123 (2020), 103865
   📄 Paper

2. Adriano Lucieri, Muhammad Naseer Bajwa, Stephan Alexander Braun, Muhammad Imran Malik, Andreas Dengel, and Sheraz Ahmed
   ExAID: A Multimodal Explanation Framework for Computer-Aided Diagnosis of Skin Lesions
   Computer Methods and Programs in Biomedicine (2022), 106620
   📄 Paper

Explanation by Examples

Case-Based Reasoning

1. Philipp Tschandl, Giuseppe Argenziano, Majid Razmara, and Jordan Yap
   Diagnostic Accuracy of Content Based Dermatoscopic Image Retrieval with Deep Classification Features
   British Journal of Dermatology 181, 1 (2019), 155–165
   📄 Paper

2. Jean-Baptiste Lamy, Boomadevi Sekar, Gilles Guezennec, Jacques Bouaud, and Brigitte Séroussi
   Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach
   Artificial Intelligence in Medicine 94 (2019), 42–53
   📄 Paper

3. Alina Jade Barnett, Fides Regina Schwartz, Chaofan Tao, Chaofan Chen, Yinhao Ren, Joseph Y Lo, and Cynthia Rudin
   Interpretable Mammographic Image Classification using Case-Based Reasoning and Deep Learning
   Workshop on Deep Learning, Case-Based Reasoning, and AutoML: Present and Future Synergies - IJCAI, 2021
   📄 Paper

4. Catarina Barata and Carlos Santiago
   Improving the Explainability of Skin Cancer Diagnosis Using CBIR
   International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 550–559, 2021
   📄 Paper

Counterfactual Explanations

1. Junho Kim, Minsu Kim, and Yong Man Ro
   Interpretation of Lesional Detection via Counterfactual Generation
   IEEE International Conference on Image Processing (ICIP), 96–100, 2021
   📄 Paper

2. Kathryn Schutte, Olivier Moindrot, Paul Hérent, Jean-Baptiste Schiratti, and Simon Jégou
   Using StyleGAN for Visual Interpretability of Deep Learning Models on Medical Images
   Medical Imaging Meets NeurIPS Workshop, 2021
   📄 Paper

3. Sumedha Singla, Brian Pollack, Stephen Wallace, and Kayhan Batmanghelich
   Explaining the Black-box Smoothly - A Counterfactual Approach
   Medical Image Analysis 84 (2023), 102721
   📄 Paper

Prototypes

1. Eunji Kim, Siwon Kim, Minji Seo, and Sungroh Yoon
   XProtoNet: Diagnosis in Chest Radiography with Global and Local Explanations
   IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 15719–15728, 2021
   📄 Paper

2. Gurmail Singh and Kin-Choong Yow
   An Interpretable Deep Learning Model For Covid-19 Detection With Chest X-ray Images
   IEEE Access 9 (2021), 85198–85208
   📄 Paper

Explanation by Concepts

1. Zhengqing Fang, Kun Kuang, Yuxiao Lin, Fei Wu, and Yu-Feng Yao
   Concept-based Explanation for Fine-grained Images and Its Application in Infectious Keratitis Classification
   ACM International Conference on Multimedia, 700–708, 2020
   📄 Paper

2. Mert Yuksekgonul, Maggie Wang, and James Zou
   Post-hoc concept bottleneck models
   arXiv preprint arXiv:2205.15480 (2022)
   📄 Paper

3. Siyuan Yan, Zhen Yu, Xuelin Zhang, Dwarikanath Mahapatra, Shekhar S Chandra, Monika Janda, Peter Soyer, and Zongyuan Ge
   Towards Trustable Skin Cancer Diagnosis via Rewriting Model’s Decision
   IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11568–11577, 2023
   📄 Paper

4. Cristiano Patrício, João C. Neves, and Luis F. Teixeira
   Coherent Concept-based Explanations in Medical Image and Its Application to Skin Lesion Diagnosis
   IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 3799–3808, 2023
   📄 Paper

Other Approaches

Bayesian Approaches

1. Ponkrshnan Thiagarajan, Pushkar Khairnar, and Susanta Ghosh
   Explanation and Use of Uncertainty Obtained by Bayesian Neural Network Classifiers for Breast Histopathology Images
   IEEE Transactions on Medical Imaging 41, 4 (2021), 815–825
   📄 Paper

2. Mohammad Ehtasham Billah and Farrukh Javed
   Bayesian Convolutional Neural Network-based Models for Diagnosis of Blood Cancer
   Applied Artificial Intelligence (2022), 1–22
   📄 Paper

3. Mahesh Gour and Sweta Jain
   Uncertainty-aware convolutional neural network for COVID-19 X-ray images classification
   Computers in Biology and Medicine 140 (2022), 105047
   📄   Paper

Adversarial Training

1. Andrei Margeloiu, Nikola Simidjievski, Mateja Jamnik, and Adrian Weller
   Improving Interpretability in Medical Imaging Diagnosis using Adversarial Training
   Medical Imaging Meets NeurIPS Workshop, 2020
   📄   Paper