Due to the project need to grab the sebug of the vulnerability database content, using the scrapy framework simple has written a gripping sebug the crawler, and stored in a database, mysql or mongodb, here to mysql, for example.
Scrapy, Python, development of a quick,high-level screen scraping and web crawling framework, used to crawl the web site and from the page to extract structured data. Scrapy wide range of uses, can be used for data mining, monitoring and automated testing.
Scrapy attractive place is that it is a framework that anyone can easily modify. It also provides various types of reptiles of the base class, such as BaseSpider, sitemap crawler, the latest version also provides web2. 0 crawler support.
Scrach, is to grab the meaning of this Python crawler framework called Scrapy, probably is also this meaning to it, just call it: a small scratch.
Don't know much about scrapy children's shoes see here: http://doc.scrapy.org/en/0.24/intro/overview.html
A picture is worth a thousand words ha: