Federated learning allows a large group of edge workers to collaboratively train a shared model without revealing their local data. It has become a powerful tool for deep learning in heterogeneous environments. User privacy is preserved by keeping the training data local to each device. However, federated learning still requires workers to share their weights, which can leak private information during collaboration. This paper introduces SecureFL, a practical framework that provides end-to-end security of federated learning. SecureFL integrates widely available Trusted Execution Environments (TEE) to protect against privacy leaks. SecureFL also uses carefully designed partitioning and aggregation techniques to ensure TEE efficiency on both the cloud and edge workers. SecureFL is both practical and efficient in securing the end-to-end process of federated learning, providing reasonable overhead given the privacy benefits. The paper provides thorough security analysis and performance evaluation of SecureFL, which show that the overhead is reasonable considering the substantial privacy benefits that it provides.