The ubiquitous deployment of IoT devices in smart homes has led to growing research interests in studying the home network traffic for various applications such as network measurements, device profiling, and IoT device event inference. Recent studies have shown that user activities can be inferred from a home network using extracted device event logs. However, existing solutions for user activity inference such as IoTMosaic and E2AP have limitations when handling ambiguities caused by device malfunctions. In this paper, we first identify the challenges faced by the existing user activity inference algorithms and the root causes of their poor performances on certain types of inputs. We then show that useful information can still be obtained even in situations where device malfunctions introduce ambiguities in user activity patterns. We achieve so by designing an extension to the existing algorithms. We also apply our extension in a digital forensics application. Our extensive experimental evaluations demonstrate that our solutions can effectively provide insights to user activity inference despite the presence of indistinguishable user activity patterns.