Advances in the medical field require new and progressive ways to intensify surgical resident training and surgical skills learning. To address this need, in addition to traditional training using realistic surgical objects, simulator-based training programs have been widely adopted. Both training methods need constant presence of a competent surgeon to assess the surgical dexterity of a trainee, which is not only costly but also very subjective. This paper proposes a novel video-based approach for assessing the trainee's hand and tool movements in minimally invasive surgical training. Videos capturing surgical training actions are analysed to automatically extract major skill measuring features and to provide real-time performance feedback to the trainee. The method has the potential of being extended to skill evaluation in real surgeries at low cost.