The Multimodal Deep Learning for Diagnosing COVID-19 Pneumonia from Chest CT-Scan and X-Ray Images

Naufal Hilmizen, Alhadi Bustamam, Devvi Sarwinda

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Due to the COVID-19 Pandemic, doctors need to make medical decisions for their patients based on many examinations (e.g., polymerase chain reaction test, temperature test, CT-Scans, or X-rays). However, transfer learning has been used in several researches and focuses on only a single modality of biomarkers (e.g., CT-Scan or X-Ray) for diagnosing Pneumonia. In recent studies, a single modality has its own classification accuracy and every different biomarker may provide complementary information for detecting COVID-19 Pneumonia. The COVID-19 virus can be detected by CT-Scan and X-Ray imaging of the chest. In this work, we propose to use concatenation of two different transfer learning models using an open-source dataset of 2500 CT-Scan images and 2500 X-ray images for classifying CT-Scan images and X-ray images into two classes: normal and COVID-19 Pneumonia. We have used DenseNet121, MobileNet, Xception, InceptionV3, ResNet50, and VGG16 models for image recognition in our work. As a result, we achieve the best classification accuracy of 99.87% of the concatenation of ResNet50 and VGG16 networks. We also achieved the best classification accuracy of 98.00% when using a single modality of CT-Scan ResNet50 networks and classification accuracy of 98.93% for X-Ray VGG16 networks. Our multimodal fusion method shows a better classification accuracy compared to the method of using a single modality of biomarkers.

Original languageEnglish
Title of host publication2020 3rd International Seminar on Research of Information Technology and Intelligent Systems, ISRITI 2020
EditorsFerry Wahyu Wibowo
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages26-31
Number of pages6
ISBN (Electronic)9781728184067
DOIs
Publication statusPublished - 10 Dec 2020
Event3rd International Seminar on Research of Information Technology and Intelligent Systems, ISRITI 2020 - Yogyakarta, Indonesia
Duration: 10 Dec 2020 → …

Publication series

Name2020 3rd International Seminar on Research of Information Technology and Intelligent Systems, ISRITI 2020

Conference

Conference3rd International Seminar on Research of Information Technology and Intelligent Systems, ISRITI 2020
CountryIndonesia
CityYogyakarta
Period10/12/20 → …

Keywords

  • Concatenate
  • COVID-19
  • CT-Scan
  • Multimodal
  • Pneumonia
  • Transfer Learning
  • X-Ray

Fingerprint Dive into the research topics of 'The Multimodal Deep Learning for Diagnosing COVID-19 Pneumonia from Chest CT-Scan and X-Ray Images'. Together they form a unique fingerprint.

Cite this