The wide deployment of IoT systems in smart homes has changed the landscape of networked systems, Internet traffic, and data communications in residential broadband networks as well as the Internet at large. However, recent spates of cyber attacks and threats towards IoT systems in smart homes have revealed prevalent vulnerabilities and risks of IoT systems ranging from data link layer protocols to application services. To address the security challenges of IoT systems in smart homes, this paper introduces IoTArgos, a multi-layer security monitoring system, which collects, analyzes, and characterizes data communications of heterogeneous IoT devices via programmable home routers. More importantly, this system extracts a variety of multi-layer data communication features and develops supervised learning methods for classifying intrusion activities at system, network, and application layers. In light of the potential zero-day or unknown attacks, IoTArgos also incorporates unsupervised learning algorithms to discover unusual or suspicious behaviors towards smart home IoT systems. Our extensive experimental evaluations have demonstrated that IoTArgos is able to detect anomalous activities targeting IoT devices in smart homes with a precision of 0.9876 and a recall of 0.9763.