Small form factor and low-cost wearable devices enable a variety of applications including gesture recognition, health monitoring, and activity tracking. Energy harvesting and optimal energy management are critical for the adoption of these devices, since they are severely constrained by battery capacity. This paper considers optimal gesture recognition using self-powered devices. We propose an approach to maximize the number of gestures that can be recognized under energy budget and accuracy constraints. We construct a computationally efficient optimization algorithm with the help of analytical models derived using the energy consumption breakdown of a wearable device. Our empirical evaluations demonstrate up to 2.4 x increase in the number of recognized gestures compared to a manually optimized solution.