DeepSyslog: Deep Anomaly Detection on Syslog Using Sentence Embedding and Metadata

Anomaly events indicating the unhealthy status of the computer system are recorded in the system log (Syslog). Therefore, Syslog-based anomaly event detection is crucial for diagnosing system issues and problems. However, existing log-based anomaly detection approaches use raw and unstructured log entries independently and incompletely, i.e., without considering the context of each event and event metadata in the logs. They employ incomplete representation of unstructured log data, limiting the deep learning model’s capacity in the early stage, which tends to omit anomaly events and cause false alarms. In this work, we propose DeepSyslog, which represents Syslog with the context of log events and event metadata in the logs. Inspired by the sequence nature of the log stream, we employ unsupervised sentence embedding to extract the semantic and context information hidden in the log stream, rather than word embedding or one-hot embedding, which only capture the similarities between log words. The sentence embedding is further integrated with event metadata to form complete representations of Syslog, which can distinguish the anomaly caused by the correlated log entries and exceptional event metadata in the log. The simulation results on widely used log datasets show that DeepSyslog achieves high performance compared with the existing log-based anomaly event detection approaches.