Relational reasoning-based approach for network protocol reverse engineering

Extracting the protocol format specifications from packets plays a critical role in many applications, such as application protocol parsing, vulnerability scanning, as well as malware behaviour analysis. In this study, we propose RelaNet, a novel relational reasoning-based method for network protocol reverse engineering. It is based on the key insight that n-grams of packets have context relations. Such relations are especially informative between keywords and can be used to infer protocol formats. RelaNet contains three modules to mimic the analysis strategy used by experts: coarse structure generation, relation learning and fine structure generation. In coarse structure generation, RelaNet first constructs a coarse-grained structure based on the occurrence frequency of n-grams. Relation learning is then performed to discover the context relations between the n-grams in the structure. By using such relations, we can eventually discover the strong context relations that exist between keywords, which can be used to accurately generate a fine-grained structure, namely the protocol format. We implement RelaNet and evaluate it on two publicly available datasets, the experimental results demonstrate the effectiveness and efficiency of RelaNet for protocol format inference. Furthermore, we compare RelaNet with state-of-the-art methods, and the results show our approach outperforms those methods.