Scrapy Python Crawler: Crawlpy

Python web spider/crawler based on scrapy with support for POST/GET login, variable level of recursions/depth and optionally save to disk. Requirements python 2.7 lxml pip pip install Scrapy Features POST/GET Login prior crawling Can handle logins that requires dynamic CSRF token Variable level o...

Using scrapy crawl sebug vulnerability database-vulnerability warning-the black bar safety net

! Due to the project need to grab the sebug of the vulnerability database content, using the scrapy framework simple has written a gripping sebug the crawler, and stored in a database, mysql or mongodb, here to mysql, for example. About scrapy Scrapy, Python, development of a quick,high-level...

XML External Entity (XXE) Injection

Overview Scrapy is a high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. Affected versions of this package are vulnerable to XML External Entity XXE Injection via the sitemap reader. Details XXE Injection is a type of attack...