Inventus

Inventus Inventus is a spider designed to find subdomains of a specific domain by crawling it and any subdomains it discovers. It's a Scrapy spider, meaning it's easily modified and extendable to your needs. Demo asciicast Requirements Linux -- I haven't tested this on Windows. Python 2.7 or Pyth...

smbcrawler

SmbCrawler SmbCrawler is no-nonsense tool that takes credentials and a list of hosts and 'crawls' or 'spiders' through those shares. Features: takes host names, IP addresses, IP ranges, or an nmap xml file as input checks permissions check for 'write' permissions is opt-in, because it requires...

gospider

GoSpider GoSpider - Fast web spider written in Go Painless integrate Gospider into your recon workflow? this project was part of Osmedeus Engine. Check out how it was integrated at @OsmedeusEngine Installation GO install GO111MODULE=on go install github.com/jaeles-project/gospider@latest Docker...

HTTPLoot

HTTPLoot An automated tool which can simultaneously crawl, fill forms, trigger error/debug pages and "loot" secrets out of the client-facing code of sites. Usage To use the tool, you can grab any one of the pre-built binaries from the Releases section of the repository. If you want to build the...

MANSPIDER

MANSPIDER Crawl SMB shares for juicy information. File content searching + regex is supported! What's New in v2.0 Manspider 2.0 is here! This brings significant improvements: New and improved text extraction powered by Kreuzberg - now supporting PDF, DOCX, XLSX, PPTX, images with OCR, and many...

Photon

Photon Incredibly fast crawler designed for OSINT. Photon Wiki • How To Use • Compatibility • Photon Library • Contribution • Roadmap Sponsored By Key Features Data Extraction Photon can extract the following data while crawling: URLs in-scope & out-of-scope URLs with parameters...

webvulscan

WebVulScan Synopsis WebVulScan is a web application vulnerability scanner. It is a web application itself written in PHP and can be used to test remote, or local, web applications for security vulnerabilities. As a scan is running, details of the scan are dynamically updated to the user. These...

domain_analyzer

Domain Analyzer v0.8.3 What Domain analyzer is a security analysis tool that automatically discovers and reports information about a given domain. Its main purpose is to analyze domains in an unattended way. It has many crazy features, such as getting more domains from DNS zones, automatic nmap,...

robin

Robin: AI-Powered Dark Web OSINT Tool Robin is an AI-powered tool for conducting dark web OSINT investigations. It leverages LLMs to refine queries, filter search results from dark web search engines, and provide an investigation summary. Installation • Robin as MCP • Troubleshooting • Contributi...

Scrapling

Effortless Web Scraping for the Modern Web -DFE0E5 !Static Badgehttps://img.shields.io/badge/Skill-black?style=flat&label=Agent&...

AIL-framework

AIL Framework Open-source framework for the collection, crawling, processing, and analysis of unstructured information. AIL framework is an open-source platform to collect, crawl, process and analyse unstructured data from the clear web, Tor, I2P, chats, files and external feeds. Originally...

Pyison

Pyison Pyison is a tarpit for AI webcrawlers Introduction Like all web crawlers, AI bots request pages from webservers, and then follow links on the page to other pages. By doing this, they can build an index of an entire website. Unlike other web crawlers, though, AI bots present some unique...

katana

A next-generation crawling and spidering framework Features • Installation • Usage • Scope • Config • Filters • Join Discord Features Fast And fully configurable web crawling Standard and Headless mode JavaScript parsing / crawling Customizable automatic form filling Scope control - Preconfigured...

burp-dom-scanner

Burp DOM Scanner It's a Burp Suite's extension to allow for recursive crawling and scanning of Single Page Applications. It runs a Chromium browser to scan the webpage for DOM-based XSS. It can also collect all the requests XHR, fetch, websockets, etc issued during the crawling allowing them to b...

dcrawl

dcrawl dcrawl is a simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names. How it works? dcrawl takes one site URL as input and detects all links in the site's body. Each found link is put into the queue. Successively, each queued link is crawled in...

Egyscan

EgyScan V3.4 "Ultra" : Your Digital Fort of Invincibility 🏰 Greetings, Guardians of the Digital Realm! 🛡️ Prepare to embark on an extraordinary journey with EgyScan V3.0 , the ultimate sentinel that transforms your websites into impregnable fortresses against relentless cyber threats. Let's explor...

xsscrapy

xsscrapy Fast, thorough, XSS/SQLi spider. Give it a URL and it'll test every link it finds for cross-site scripting and some SQL injection vulnerabilities. See FAQ for more details about SQLi detection. From within the main folder run: ./xsscrapy.py -u http://example.com If you wish to login then...

akca

AKCA Advanced Web Security Scanner Discover endpoints. Test web applications. Inspect the evidence. Installation · Usage · Workflow · Profiles · Coverage · Reports · Support · Features · Changelog AKCA is an open-source, evidence-oriented Dynamic Application Security Testing DAST scanner written ...

spider

Spider The fastest web crawler and scraper for Rust. spider.cloud · Guides · Docs · Examples · Discord Spider is a concurrency-first crawling engine written in Rust. It streams pages as they arrive, renders JavaScript only on pages that need it, and scales from a single script to a distributed...

domdig

DOMDig DOMDig is a DOM XSS scanner that runs inside the Chromium web browser and it can scan single page applications SPA recursively. Unlike other scanners, DOMDig can crawl any webapplication including gmail by keeping track of DOM modifications and XHR/fetch/websocket requests and it can...