Convolutional Neural Networks for Medical Image Analysis

Full Training or Fine Tuning?

Nima Tajbakhsh, Jae Y. Shin, Suryakanth R. Gurudu, R. Todd Hurst, Christopher B. Kendall, Michael B. Gotway, Jianming Liang

Research output: Contribution to journalArticle

519 Citations (Scopus)

Abstract

Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch? To address this question, we considered four distinct medical imaging applications in three specialties (radiology, cardiology, and gastroenterology) involving classification, detection, and segmentation from three different imaging modalities, and investigated how the performance of deep CNNs trained from scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner. Our experiments consistently demonstrated that 1) the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch; 2) fine-tuned CNNs were more robust to the size of training sets than CNNs trained from scratch; 3) neither shallow tuning nor deep tuning was the optimal choice for a particular application; and 4) our layer-wise fine-tuning scheme could offer a practical way to reach the best performance for the application at hand based on the amount of available data.

Original languageEnglish (US)
Article number7426826
Pages (from-to)1299-1312
Number of pages14
JournalIEEE Transactions on Medical Imaging
Volume35
Issue number5
DOIs
StatePublished - May 1 2016

Fingerprint

Gastroenterology
Diagnostic Imaging
Cardiology
Radiology
Image analysis
Hand
Tuning
Neural networks
Medical imaging
Imaging techniques
Experiments

Keywords

  • Carotid intima-media thickness
  • computer-aided detection
  • convolutional neural networks
  • deep learning
  • fine-tuning
  • medical image analysis
  • polyp detection
  • pulmonary embolism detection
  • video quality assessment

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Computer Science Applications
  • Radiological and Ultrasound Technology
  • Software

Cite this

Tajbakhsh, N., Shin, J. Y., Gurudu, S. R., Hurst, R. T., Kendall, C. B., Gotway, M. B., & Liang, J. (2016). Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? IEEE Transactions on Medical Imaging, 35(5), 1299-1312. [7426826]. https://doi.org/10.1109/TMI.2016.2535302

Convolutional Neural Networks for Medical Image Analysis : Full Training or Fine Tuning? / Tajbakhsh, Nima; Shin, Jae Y.; Gurudu, Suryakanth R.; Hurst, R. Todd; Kendall, Christopher B.; Gotway, Michael B.; Liang, Jianming.

In: IEEE Transactions on Medical Imaging, Vol. 35, No. 5, 7426826, 01.05.2016, p. 1299-1312.

Research output: Contribution to journalArticle

Tajbakhsh, N, Shin, JY, Gurudu, SR, Hurst, RT, Kendall, CB, Gotway, MB & Liang, J 2016, 'Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?', IEEE Transactions on Medical Imaging, vol. 35, no. 5, 7426826, pp. 1299-1312. https://doi.org/10.1109/TMI.2016.2535302
Tajbakhsh N, Shin JY, Gurudu SR, Hurst RT, Kendall CB, Gotway MB et al. Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? IEEE Transactions on Medical Imaging. 2016 May 1;35(5):1299-1312. 7426826. https://doi.org/10.1109/TMI.2016.2535302
Tajbakhsh, Nima ; Shin, Jae Y. ; Gurudu, Suryakanth R. ; Hurst, R. Todd ; Kendall, Christopher B. ; Gotway, Michael B. ; Liang, Jianming. / Convolutional Neural Networks for Medical Image Analysis : Full Training or Fine Tuning?. In: IEEE Transactions on Medical Imaging. 2016 ; Vol. 35, No. 5. pp. 1299-1312.
@article{62e7a75a375149e9b6e59253ad08e0f3,
title = "Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?",
abstract = "Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch? To address this question, we considered four distinct medical imaging applications in three specialties (radiology, cardiology, and gastroenterology) involving classification, detection, and segmentation from three different imaging modalities, and investigated how the performance of deep CNNs trained from scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner. Our experiments consistently demonstrated that 1) the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch; 2) fine-tuned CNNs were more robust to the size of training sets than CNNs trained from scratch; 3) neither shallow tuning nor deep tuning was the optimal choice for a particular application; and 4) our layer-wise fine-tuning scheme could offer a practical way to reach the best performance for the application at hand based on the amount of available data.",
keywords = "Carotid intima-media thickness, computer-aided detection, convolutional neural networks, deep learning, fine-tuning, medical image analysis, polyp detection, pulmonary embolism detection, video quality assessment",
author = "Nima Tajbakhsh and Shin, {Jae Y.} and Gurudu, {Suryakanth R.} and Hurst, {R. Todd} and Kendall, {Christopher B.} and Gotway, {Michael B.} and Jianming Liang",
year = "2016",
month = "5",
day = "1",
doi = "10.1109/TMI.2016.2535302",
language = "English (US)",
volume = "35",
pages = "1299--1312",
journal = "IEEE Transactions on Medical Imaging",
issn = "0278-0062",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "5",

}

TY - JOUR

T1 - Convolutional Neural Networks for Medical Image Analysis

T2 - Full Training or Fine Tuning?

AU - Tajbakhsh, Nima

AU - Shin, Jae Y.

AU - Gurudu, Suryakanth R.

AU - Hurst, R. Todd

AU - Kendall, Christopher B.

AU - Gotway, Michael B.

AU - Liang, Jianming

PY - 2016/5/1

Y1 - 2016/5/1

N2 - Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch? To address this question, we considered four distinct medical imaging applications in three specialties (radiology, cardiology, and gastroenterology) involving classification, detection, and segmentation from three different imaging modalities, and investigated how the performance of deep CNNs trained from scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner. Our experiments consistently demonstrated that 1) the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch; 2) fine-tuned CNNs were more robust to the size of training sets than CNNs trained from scratch; 3) neither shallow tuning nor deep tuning was the optimal choice for a particular application; and 4) our layer-wise fine-tuning scheme could offer a practical way to reach the best performance for the application at hand based on the amount of available data.

AB - Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch? To address this question, we considered four distinct medical imaging applications in three specialties (radiology, cardiology, and gastroenterology) involving classification, detection, and segmentation from three different imaging modalities, and investigated how the performance of deep CNNs trained from scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner. Our experiments consistently demonstrated that 1) the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch; 2) fine-tuned CNNs were more robust to the size of training sets than CNNs trained from scratch; 3) neither shallow tuning nor deep tuning was the optimal choice for a particular application; and 4) our layer-wise fine-tuning scheme could offer a practical way to reach the best performance for the application at hand based on the amount of available data.

KW - Carotid intima-media thickness

KW - computer-aided detection

KW - convolutional neural networks

KW - deep learning

KW - fine-tuning

KW - medical image analysis

KW - polyp detection

KW - pulmonary embolism detection

KW - video quality assessment

UR - http://www.scopus.com/inward/record.url?scp=84968649810&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84968649810&partnerID=8YFLogxK

U2 - 10.1109/TMI.2016.2535302

DO - 10.1109/TMI.2016.2535302

M3 - Article

VL - 35

SP - 1299

EP - 1312

JO - IEEE Transactions on Medical Imaging

JF - IEEE Transactions on Medical Imaging

SN - 0278-0062

IS - 5

M1 - 7426826

ER -