Design and Implementation of the Topic-Focused Crawler Based on Scrapy

E-commerce websites has abundant commercial data. Some very beneficial information to the analysis and prediction of the market can be discovered from these data by applying data mining techniques. The topic-focused web crawler can crawl and gather the subject-related web pages as soon as possible. This thesis has designed and realized the topic-focused crawler based on Scrapy. It firstly introduces the design idea of the crawler and highlights the functions of Scrapys every part. Then, it uses this topic-focused crawler to realize the capture of information from the C2C e-commerce platform, for example TaoBao. At last, it obtains the running result and comparisons of crawling performance between Scrapy based crawler and general crawler.