Using scrapy crawl sebug vulnerability database-vulnerability warning-the black bar safety net

ID MYHACK58:62201559738
Type myhack58
Reporter 佚名
Modified 2015-03-09T00:00:00



Due to the project need to grab the sebug of the vulnerability database content, using the scrapy framework simple has written a gripping sebug the crawler, and stored in a database, mysql or mongodb, here to mysql, for example.

About scrapy

Scrapy, Python, development of a quick,high-level screen scraping and web crawling framework, used to crawl the web site and from the page to extract structured data. Scrapy wide range of uses, can be used for data mining, monitoring and automated testing.

Scrapy attractive place is that it is a framework that anyone can easily modify. It also provides various types of reptiles of the base class, such as BaseSpider, sitemap crawler, the latest version also provides web2. 0 crawler support.

Scrach, is to grab the meaning of this Python crawler framework called Scrapy, probably is also this meaning to it, just call it: a small scratch.

Don't know much about scrapy children's shoes see here:

A picture is worth a thousand words ha:



[1] [2] [3] [4] next