Human and DNN classification performance on images with quality distortions: A comparative study

Samuel Dodge, Lina Karam

Research output: Contribution to journalArticle

Abstract

Image quality is an important practical challenge that is often overlooked in the design of machine vision systems. Commonly, machine vision systems are trained and tested on high-quality image datasets, yet in practical applications the input images cannot be assumed to be of high quality. Modern deep neural networks (DNNs) have been shown to perform poorly on images affected by blur or noise distortions. In this work, we investigate whether human subjects also perform poorly on distorted stimuli and provide a direct comparison with the performance of DNNs. Specifically, we study the effect of Gaussian blur and additive Gaussian noise on human andDNNclassification performance.We perform two experiments: one crowd-sourced experiment with unlimited stimulus display time, and one lab experiment with 100ms display time. In both cases, we found that humans outperform neural networks on distorted stimuli, even when the networks are retrained with distorted data.

Original languageEnglish (US)
Article number7
JournalACM Transactions on Applied Perception
Volume16
Issue number2
DOIs
StatePublished - Mar 1 2019

Fingerprint

Comparative Study
Machine Vision
Neural Networks
Vision System
Image Quality
Image quality
Computer vision
Experiment
Human Performance
Experiments
Gaussian Noise
Noise
Display
Display devices
Neural networks
Human
Deep neural networks
Design
Datasets

Keywords

  • Deep learning
  • Human study
  • Robust visual recognition

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)
  • Experimental and Cognitive Psychology

Cite this

Human and DNN classification performance on images with quality distortions : A comparative study. / Dodge, Samuel; Karam, Lina.

In: ACM Transactions on Applied Perception, Vol. 16, No. 2, 7, 01.03.2019.

Research output: Contribution to journalArticle

@article{d8cf22ddd291434d8bf38ef49ec5d5c4,
title = "Human and DNN classification performance on images with quality distortions: A comparative study",
abstract = "Image quality is an important practical challenge that is often overlooked in the design of machine vision systems. Commonly, machine vision systems are trained and tested on high-quality image datasets, yet in practical applications the input images cannot be assumed to be of high quality. Modern deep neural networks (DNNs) have been shown to perform poorly on images affected by blur or noise distortions. In this work, we investigate whether human subjects also perform poorly on distorted stimuli and provide a direct comparison with the performance of DNNs. Specifically, we study the effect of Gaussian blur and additive Gaussian noise on human andDNNclassification performance.We perform two experiments: one crowd-sourced experiment with unlimited stimulus display time, and one lab experiment with 100ms display time. In both cases, we found that humans outperform neural networks on distorted stimuli, even when the networks are retrained with distorted data.",
keywords = "Deep learning, Human study, Robust visual recognition",
author = "Samuel Dodge and Lina Karam",
year = "2019",
month = "3",
day = "1",
doi = "10.1145/3306241",
language = "English (US)",
volume = "16",
journal = "ACM Transactions on Applied Perception",
issn = "1544-3558",
publisher = "Association for Computing Machinery (ACM)",
number = "2",

}

TY - JOUR

T1 - Human and DNN classification performance on images with quality distortions

T2 - A comparative study

AU - Dodge, Samuel

AU - Karam, Lina

PY - 2019/3/1

Y1 - 2019/3/1

N2 - Image quality is an important practical challenge that is often overlooked in the design of machine vision systems. Commonly, machine vision systems are trained and tested on high-quality image datasets, yet in practical applications the input images cannot be assumed to be of high quality. Modern deep neural networks (DNNs) have been shown to perform poorly on images affected by blur or noise distortions. In this work, we investigate whether human subjects also perform poorly on distorted stimuli and provide a direct comparison with the performance of DNNs. Specifically, we study the effect of Gaussian blur and additive Gaussian noise on human andDNNclassification performance.We perform two experiments: one crowd-sourced experiment with unlimited stimulus display time, and one lab experiment with 100ms display time. In both cases, we found that humans outperform neural networks on distorted stimuli, even when the networks are retrained with distorted data.

AB - Image quality is an important practical challenge that is often overlooked in the design of machine vision systems. Commonly, machine vision systems are trained and tested on high-quality image datasets, yet in practical applications the input images cannot be assumed to be of high quality. Modern deep neural networks (DNNs) have been shown to perform poorly on images affected by blur or noise distortions. In this work, we investigate whether human subjects also perform poorly on distorted stimuli and provide a direct comparison with the performance of DNNs. Specifically, we study the effect of Gaussian blur and additive Gaussian noise on human andDNNclassification performance.We perform two experiments: one crowd-sourced experiment with unlimited stimulus display time, and one lab experiment with 100ms display time. In both cases, we found that humans outperform neural networks on distorted stimuli, even when the networks are retrained with distorted data.

KW - Deep learning

KW - Human study

KW - Robust visual recognition

UR - http://www.scopus.com/inward/record.url?scp=85063210279&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063210279&partnerID=8YFLogxK

U2 - 10.1145/3306241

DO - 10.1145/3306241

M3 - Article

VL - 16

JO - ACM Transactions on Applied Perception

JF - ACM Transactions on Applied Perception

SN - 1544-3558

IS - 2

M1 - 7

ER -