spider

Spider The fastest web crawler and scraper for Rust. spider.cloud · Guides · Docs · Examples · Discord Spider is a concurrency-first crawling engine written in Rust. It streams pages as they arrive, renders JavaScript only on pages that need it, and scales from a single script to a distributed...

grubcrawler

Agentic web crawler with anti-detection, vision fallback, and peer-to-peer mesh. Full-stack web crawling engine with JavaScript rendering, Camoufox anti-detect browser, per-request proxy routing, and autonomous agent loops. Converts pages to clean markdown with a native Rust extraction engine. Wh...

JAW

J A W Website | Docs | Setup | Crawler | Quick Start Docker Example JAW An open-source, prototype implementation of property graphs for JavaScript based on the esprima parser, and the EsTree SpiderMonkey Spec. JAW can be used for analyzing the client-side of web applications and JavaScript-based...

evine

Evine Interactive CLI Web Crawler. Evine is a simple, fast, and interactive web crawler and web scraper written in Golang. Evine is useful for a wide range of purposes such as metadata and data extraction, data mining, reconnaissance and testing. If you like the project, give it a star. It forces...

dirhunt

.. image:: https://raw.githubusercontent.com/Nekmo/dirhunt/v0.2.0/images/dirhunt.png | .. image:: https://raw.githubusercontent.com/Nekmo/dirhunt/pip-rating-badge/pip-rating-badge.svg :target: https://github.com/Nekmo/dirhunt/actions/workflows/pip-rating.yml :alt: pip-rating badge .. image::...

wxpath

wxpath - declarative web graph traversal with XPath NEW: TUI - Interactive terminal interface powered by Textual for testing wxpath expressions and exporting data. Install Requires Python 3.10+. pip install wxpath For TUI support: pip install "wxpathtui" Immediately launch the TUI via uv: uvx...

dcrawl

dcrawl dcrawl is a simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names. How it works? dcrawl takes one site URL as input and detects all links in the site's body. Each found link is put into the queue. Successively, each queued link is crawled in...

hakrawler

Hakrawler Fast golang web crawler for gathering URLs and JavaScript file locations. This is basically a simple implementation of the awesome Gocolly library. Example usages Single URL: echo https://google.com | hakrawler Multiple URLs: cat urls.txt | hakrawler Timeout for each line of stdin after...

sqlifinder

SQL Injection scanner made with python. Features • Install • Usage • sqlifinder is a tool with the function of detecting GET-based sql injection vulnerabilities in web applications using waybackurls, web crawlers and sql injection payloads. Features Fast and powerful scanner Include web crawler a...

EasySpider

易采集/EasySpider: Visual Code-Free Web Crawler 一个完全免费 （包括商业使用和二次开发 ）的可视化浏览器自动化测试/数据采集/爬虫软件，可以使用图形化界面，无代码可视化的设计和执行任务。只需要在网页上选择自己想要操作的内容并根据提示框操作即可完成任务的设计和执行。同时软件还可以单独以命令行的方式进行执行，从而可以很方便的嵌入到其他系统中。 A completely free including for commercial use and secondary development visual browser automation...

PenCrawLer

PenCrawLer An Advanced Web Crawler and DirBuster PeNCrawLer is an advanced webcrawler and dirbuster designed to use in penetration testing based on Windows Os. To read more about features please read the wiki: https://github.com/TheM4hd1/PenCrawLer/wiki...

SpiderSuite

SpiderSuite is an advance cross-platform and multi-feature web crawler designed for security professionals. For more information visit SpiderSuite's website. Installation and Usage Spider Suite is designed for easy installation and usage even for first timers. First, download the package of your...

xepor

Xepor Xepor pronounced /ˈzɛfə/ , zephyr is a web routing framework for reverse engineers and security researchers. It provides a Flask-like API for hackers to intercept and modify HTTP requests and/or HTTP responses in a human-friendly coding style. This project is meant to be used with mitmproxy...

domain_analyzer

Domain Analyzer v0.8.3 What Domain analyzer is a security analysis tool that automatically discovers and reports information about a given domain. Its main purpose is to analyze domains in an unattended way. It has many crazy features, such as getting more domains from DNS zones, automatic nmap,...

ache

ACHE Focused Crawler ACHE is a focused web crawler. It collects web pages that satisfy some specific criteria, e.g., pages that belong to a given domain or that contain a user-specified pattern. ACHE differs from generic crawlers in sense that it uses page classifiers to distinguish between...

Bashter

Bashter 4.1 Web Crawler, Scanner, and Analyzer Framework Shell-Script based...

sql-global-injector

Ultimate-Dork web crawler And Scan SQL injection vulnerability Usage & Installation - Installation apt-get install python python3 git git clone https://github.com/nu11secur1ty/sql-global-injector.git python3 -m pip install requests mechanicalsoup cd sql-global-injector - Usage python3...

xcrawl3r

xcrawl3r xcrawl3r es una utilidad de línea de comandos diseñada para rastrear recursivamente páginas web en busca de URLs. Funciona recorriendo activamente los sitios web, siguiendo enlaces incrustados en páginas web y analizando archivos incluyendo sitemaps y robots.txt para descubrir cada URL. ...

evine

Evine Web Crawler CLI interactivo. Evine es un web crawler y web scraper simple, rápido e interactivo escrito en Golang. Evine es útil para una amplia gama de propósitos, como extracción de metadatos y datos, minería de datos, reconocimiento y pruebas. Si te gusta el proyecto, dale una estrella...

gospider

GoSpider GoSpider - Rápido rastreador web escrito en Go ¿Integrar Gospider sin complicaciones en tu flujo de trabajo de reconocimiento? este proyecto fue parte de Osmedeus Engine. Mira cómo fue integrado en @OsmedeusEngine Instalación Instalación con GO GO111MODULE=on go install...