Research on Chinese content monitoring technology of darknet based on Scrapy

With the development of anonymous network technology and the intensification of fighting domestic cyber crimes, more criminals tend to use the anonymity of Tor network to trade illegal and contraband goods and release reactionary remarks on the darknet in order to evade the supervision of public security departments, seriously endangering national security. Efficient monitoring of Chinese content on the darknet is of practical significance for obtaining investigation clues on the darknet and monitoring online public opinions. This study designs a batch extraction technology of Chinese darknet content based on Scrapy and obtaining the identification code of dark websites. Methods successfully extracts data from several Chinese Darknet online shops as well as Chinese forums, and makes further statistical analysis of the extracted data. Compared with other data monitoring methods on dark websites, this method can improve the extraction efficiency of target sites and has certain versatility, providing effective monitoring methods for fighting against darknet crimes.