Detecting bots using Content Security Policy (CSP) headers

Bots are noisy, like really. And dangerous as well, especially if they can do crawling and increase usage by legitimate operations like items catalog retrieve in the case of e-commerce. I mean, we have a lot of reasons to do not like bots and count this problem as a cybersecurity threat, which...

Simple multi-threaded web crawler: dcrawl

dcrawl is a simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names. How it works? dcrawl takes one site URL as input and detects all links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the sa...