Multi-touch mobile devices have penetrated into everyday life to support personal and business communications. Secure and usable authentication techniques are indispensable for preventing illegitimate access to mobile devices. This paper presents RhyAuth, a novel two-factor rhythm-based authentication scheme for multi-touch mobile devices. RhyAuth requires a user to perform a sequence of rhythmic taps/slides on a device screen to unlock the device. The user is authenticated and admitted only when the features extracted from her rhythmic taps/slides match those stored on the device. RhyAuth is a two-factor authentication scheme that depends on a user-chosen rhythm and also the behavioral metrics for inputting the rhythm. Through a 32-user experiment on Android devices, we show that RhyAuth is highly secure against various attacks and also very usable for both sighted and visually impaired people.