DeepUserLog: Deep Anomaly Detection on User Log Using Semantic Analysis and Key-Value Data

Numerous studies have proven that abnormal behaviors related to business and transactions can be detected from user logs. In actual use, we discover that user logs are often formatted in complex ways, and there are challenges to analyzing them: (1) errors in log parsing that necessitate significant human intervention to resolve accurately, and (2) insufficient information mining. These two issues often result in increased human investment and reduced accuracy. To address these challenges, we propose DeepUserLog, a framework for anomaly detection of user logs containing a large number of key-value pairs. Our approach sidesteps the necessity of cumbersome preprocessing and a log parsing step that risks introducing noise. DeepUserLog retrieves the key-value pairs within the log and extracts the semantic features of the content after removing the values in key-value pairs, representing them as semantic vectors. In addition, the framework categorizes key-value pairs into four types while leveraging and identifying the temporal keys to uncover deeper connections between logs. Furthermore, it conducts a more thorough analysis of the information associated with numeric and text-based key-value pairs. DeepUserLog has been validated on real-world user log datasets from industry and public system log datasets, yielding promising results confirming its efficacy.