Generation of movement explanations for testing gesture based co-operative learning applications

The paper proposes an explanation framework for machine learning based gesture recognition systems to increase trust, and also provide users an interface to ask questions about the recognition result. Gestures have three components: a) handshape, b) location, and c) movement. Several techniques exist for handshape and location recognition explanation, but very limited analysis exists on explaining movement. The challenge is that modeling the movement between handshapes in a gesture require dynamic modeling of arm kinematics using differential equations. The arm models can be of various complexity, but many of them may not be explainable. Our approach in this paper, is to mine hybrid system models of gestures using a coalition of hand-shape recognition technology and explainable kinematic models. The hybrid dynamical systems are mined using video data collected from users. Change in dynamics of a test user is expressed using the parameters of the kinematic equations. The parameters are converted into human understandable explanations by experts in movement analysis. The novel outcome is the combination of fault detection in hybrid dynamical systems and machine learning to provide explanation for recognition of continuous events. We have applied our technique on 60 users for 20 ASL gestures. Results show that the mined parameters of the kinematic equations can represent each gesture with precision of 83 %, recall of 80 % and accuracy of 82 %.