wxpath

wxpath - declarative web graph traversal with XPath NEW: TUI - Interactive terminal interface powered by Textual for testing wxpath expressions and exporting data. Install Requires Python 3.10+. pip install wxpath For TUI support: pip install "wxpathtui" Immediately launch the TUI via uv: uvx...

sqlifinder

SQL Injection scanner made with python. Features • Install • Usage • sqlifinder is a tool with the function of detecting GET-based sql injection vulnerabilities in web applications using waybackurls, web crawlers and sql injection payloads. Features Fast and powerful scanner Include web crawler a...

Pyison

Pyison Pyison is a tarpit for AI webcrawlers Introduction Like all web crawlers, AI bots request pages from webservers, and then follow links on the page to other pages. By doing this, they can build an index of an entire website. Unlike other web crawlers, though, AI bots present some unique...

nuubi

Nuubi Tools Scanner, Recon. Details Options: -h/--help | Show help message and exit Arguments: -b/--banner | Banner grabing of target ip address -s/--subnet | Subnetlookup of target -c/--cms | Cms detect with headers -d/--dns | Dnslookup of target domain -e/--extract | Extract links from target...

domain_analyzer

Domain Analyzer v0.8.3 What Domain analyzer is a security analysis tool that automatically discovers and reports information about a given domain. Its main purpose is to analyze domains in an unattended way. It has many crazy features, such as getting more domains from DNS zones, automatic nmap,...

Web-Cache-Vulnerability-Scanner

Web Cache Vulnerability Scanner WCVS is a fast and versatile CLI scanner for web cache poisoning and web cache deception developed by Hackmanit and Maximilian Hildebrand. The scanner supports many different web cache poisoning and web cache deception techniques, includes a crawler to identify...

dirmap

Dirmap 中文 An advanced web directory scanning tool that will be more powerful than DirBuster, Dirsearch, cansina, and Yu Jian Demand analysis After a lot of research, summarizing an excellent web directory scanning tool has at least the following features: Concurrency engine Can use the dictionary...

festin

FestIN the powered S3 bucket finder and content discover What is FestIn Why Festin Install Using Python Using Docker Full options Usage Configure search domains Concurrency HTTP Crawling configuration Manage results Proxy usage DNS Options Full Text Support Running as a service or watching mode...

PenCrawLer

PenCrawLer An Advanced Web Crawler and DirBuster PeNCrawLer is an advanced webcrawler and dirbuster designed to use in penetration testing based on Windows Os. To read more about features please read the wiki: https://github.com/TheM4hd1/PenCrawLer/wiki...

secator

The pentester's swiss knife. Features • Supported commands • Installation • Usage • Documentation • Join us on Discord ! secator is a task and workflow runner used for security assessments. It supports dozens of well-known security tools and it is designed to improve productivity for pentesters a...

mec

mec MassExploitConsole Intended for mass exploiting Follow thewiki to get started Features Easy-to-use CLI UI Execute adpated exploits with process-level concurrency Built-in exploits automated Hide your IP using proxychains4 and proxypool ZoomEye host scan 10 threads Simple Baidu crawler...

JAW

J A W Website | Docs | Setup | Crawler | Quick Start Docker Example JAW An open-source, prototype implementation of property graphs for JavaScript based on the esprima parser, and the EsTree SpiderMonkey Spec. JAW can be used for analyzing the client-side of web applications and JavaScript-based...

Photon

Photon Incredibly fast crawler designed for OSINT. Photon Wiki • How To Use • Compatibility • Photon Library • Contribution • Roadmap Sponsored By Key Features Data Extraction Photon can extract the following data while crawling: URLs in-scope & out-of-scope URLs with parameters...

dcrawl

dcrawl dcrawl is a simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names. How it works? dcrawl takes one site URL as input and detects all links in the site's body. Each found link is put into the queue. Successively, each queued link is crawled in...

hakrawler

Hakrawler Fast golang web crawler for gathering URLs and JavaScript file locations. This is basically a simple implementation of the awesome Gocolly library. Example usages Single URL: echo https://google.com | hakrawler Multiple URLs: cat urls.txt | hakrawler Timeout for each line of stdin after...

evine

Evine Interactive CLI Web Crawler. Evine is a simple, fast, and interactive web crawler and web scraper written in Golang. Evine is useful for a wide range of purposes such as metadata and data extraction, data mining, reconnaissance and testing. If you like the project, give it a star. It forces...

SpiderSuite

SpiderSuite is an advance cross-platform and multi-feature web crawler designed for security professionals. For more information visit SpiderSuite's website. Installation and Usage Spider Suite is designed for easy installation and usage even for first timers. First, download the package of your...

xepor

Xepor Xepor pronounced /ˈzɛfə/ , zephyr is a web routing framework for reverse engineers and security researchers. It provides a Flask-like API for hackers to intercept and modify HTTP requests and/or HTTP responses in a human-friendly coding style. This project is meant to be used with mitmproxy...

grubcrawler

Agentic web crawler with anti-detection, vision fallback, and peer-to-peer mesh. Full-stack web crawling engine with JavaScript rendering, Camoufox anti-detect browser, per-request proxy routing, and autonomous agent loops. Converts pages to clean markdown with a native Rust extraction engine. Wh...

Pinkerton

🕵️ Pinkerton Investigating JavaScript files since 1850 Python project to crawl for JavaScript files and search for secrets like API keys, authorization tokens, hardcoded passwords, or related data. ⚡ Getting started A quick guide on how to install and use Pinkerton. 1. Clone the repository with:...