dcrawl

dcrawl dcrawl is a simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names. How it works? dcrawl takes one site URL as input and detects all links in the site's body. Each found link is put into the queue. Successively, each queued link is crawled in...

dcrawl

dcrawl dcrawl 是一个简单但智能的多线程网页爬虫，用于随机收集海量独特域名列表。 工作原理是什么？ dcrawl 以一个网站 URL 作为输入，检测该网站页面中的所有 链接。每个找到的链接都会被放入队列中。随后，每个队列中的链接会以同样的方式被爬取，通过每个网页上的链接分支到更多 URL。 智能爬取 的工作方式： 仅对每个主机名分支到预定义数量的链接。 每个域名允许的最大不同主机名数量（避免子域名爬取地狱，例如 blogspot.com）。 可在相同域名列表下重新启动——上次保存的域名会被添加到 URL 队列中。 仅爬取在 HEAD 响应中返回 text/html...

Simple multi-threaded web crawler: dcrawl

dcrawl is a simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names. How it works? dcrawl takes one site URL as input and detects all links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the sa...