267 matches found
dcrawl - Simple, But Smart, Multi-Threaded Web Crawler For Randomly Gathering Huge Lists Of Unique Domain Names
dcrawl is a simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names. How it works? dcrawl takes one site URL as input and detects all links in the site's body. Each found link is put into the queue. Successively, each queued link is crawled in the sa...
Domain Analyzer
Domain analyzer is a security analysis tool which automatically discovers and reports information about the given domain. Its main purpose is to analyze domains in an unattended way. Domain analyzer takes a domain name and finds information about it, such as DNS servers, mail servers, IP addresse...
Simple multi-threaded web crawler: dcrawl
dcrawl is a simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names. How it works? dcrawl takes one site URL as input and detects all links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the sa...
massExpConsole - Collection of Tools and Exploits with a CLI UI
Collection of Tools and Exploits with a CLI UI What does it do? an easy-to-use user interface cli execute any adapted exploit with process-level concurrency crawler for baidu and zoomeye a simple webshell manager some built-in exploits automated more to come... Requirements GNU/Linux or MacOS, WS...
Cross-site Scripting (XSS)
github.com/koding/koding is vulnerable to cross-site scripting XSS attacks. The attacks exist since it does not encode the user input value in crawler to avoid the attack...
Shodan’ Malware Hunter to Expose Command & Control Centers of Botnets
By Jahanzaib Hassan Recorded Future, a threat intelligence company together with Shodan, which is an internet search engine connecting services to internet devices, have created an online crawler which they call the Malware Hunter. What does it do? The Malware Hunter is a program that sends out...
swarm - A Modular Distributed Penetration Testing Tool
Swarm is an open source modular distributed penetration testing Tool that use distributed task queue to implement communication in the master-slave mode system and use MongoDB for data storage. It consists of a distributed framework and function modules. The function module can be an entirely new...
Scrapy Python Crawler: Crawlpy
Python web spider/crawler based on scrapy with support for POST/GET login, variable level of recursions/depth and optionally save to disk. Requirements python 2.7 lxml pip pip install Scrapy Features POST/GET Login prior crawling Can handle logins that requires dynamic CSRF token Variable level o...
Web Crawler Access Detection - MJ12bot
Binary data 9185.prm...
Web Crawler Access Detection - Facebook
Binary data 9180.prm...
Web Crawler Access Detection - Baiduspider
Binary data 9183.prm...
Web Crawler Access Detection - meanpathbot
Binary data 9184.prm...
Web Crawler Access Detection - Googlebot
Binary data 9181.prm...
Web Crawler Access Detection - Bingbot
Binary data 9182.prm...
Web Crawler Access Detection - SemrushBot
Binary data 9205.prm...
Web Crawler Access Detection via HTTP
Binary data 9179.prm...
Web Crawler Access Detection - DotBot
Binary data 9188.prm...
Web Crawler Access Detection - AhrefsBot
Binary data 9206.prm...
Web Crawler Access Detection - YandexBot
Binary data 9186.prm...
Web Crawler Access Detection - Exabot
Binary data 9187.prm...