Generation of movement explanations for testing gesture based co-operative learning applications

Ayan Banerjee, Imane Lamrani, Prajwal Paudyal, Sandeep Gupta

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

The paper proposes an explanation framework for machine learning based gesture recognition systems to increase trust, and also provide users an interface to ask questions about the recognition result. Gestures have three components: a) handshape, b) location, and c) movement. Several techniques exist for handshape and location recognition explanation, but very limited analysis exists on explaining movement. The challenge is that modeling the movement between handshapes in a gesture require dynamic modeling of arm kinematics using differential equations. The arm models can be of various complexity, but many of them may not be explainable. Our approach in this paper, is to mine hybrid system models of gestures using a coalition of hand-shape recognition technology and explainable kinematic models. The hybrid dynamical systems are mined using video data collected from users. Change in dynamics of a test user is expressed using the parameters of the kinematic equations. The parameters are converted into human understandable explanations by experts in movement analysis. The novel outcome is the combination of fault detection in hybrid dynamical systems and machine learning to provide explanation for recognition of continuous events. We have applied our technique on 60 users for 20 ASL gestures. Results show that the mined parameters of the kinematic equations can represent each gesture with precision of 83 %, recall of 80 % and accuracy of 82 %.

Original languageEnglish (US)
Title of host publicationProceedings - 2019 IEEE International Conference on Artificial Intelligence Testing, AITest 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages9-16
Number of pages8
ISBN (Electronic)9781728104928
DOIs
StatePublished - May 17 2019
Event1st IEEE International Conference on Artificial Intelligence Testing, AITest 2019 - Newark, United States
Duration: Apr 4 2019Apr 9 2019

Publication series

NameProceedings - 2019 IEEE International Conference on Artificial Intelligence Testing, AITest 2019

Conference

Conference1st IEEE International Conference on Artificial Intelligence Testing, AITest 2019
CountryUnited States
CityNewark
Period4/4/194/9/19

    Fingerprint

Keywords

  • Artificial intelligence
  • Explainable AI
  • Gesture recognition
  • Testing

ASJC Scopus subject areas

  • Artificial Intelligence
  • Safety, Risk, Reliability and Quality

Cite this

Banerjee, A., Lamrani, I., Paudyal, P., & Gupta, S. (2019). Generation of movement explanations for testing gesture based co-operative learning applications. In Proceedings - 2019 IEEE International Conference on Artificial Intelligence Testing, AITest 2019 (pp. 9-16). [8718241] (Proceedings - 2019 IEEE International Conference on Artificial Intelligence Testing, AITest 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/AITest.2019.00-15