In order to improve the accuracy and efficiency of emotion recognition, we design a novel system called Learning through Interactive Video and Emotion-aware System (LIVES). LIVES includes data collection, emotion recognition, and result validation, as well as emotion feedback. We adopt transfer learning to label and validate moods in LIVES, while the emotion can be classified into six types of mood in a reasonable accuracy. Through transfer learning, the time-consuming and labor-intensive processing cost on data collection and labeling can also be greatly reduced. In our prototype system, LIVES is used to enhance an emotion-aware robot’s intelligence provided by cloud. LIVES-based emotion recognition is executed in the remote cloud while corresponding result is sent to the robot for emotion feedback. The experimental results demonstrate LIVES significantly improves the accuracy and effectiveness of emotion classification.