katana

A next-generation crawling and spidering framework Features • Installation • Usage • Scope • Config • Filters • Join Discord Features Fast And fully configurable web crawling Standard and Headless mode JavaScript parsing / crawling Customizable automatic form filling Scope control - Preconfigured...

domain_analyzer

Domain Analyzer v0.8.3 What Domain analyzer is a security analysis tool that automatically discovers and reports information about a given domain. Its main purpose is to analyze domains in an unattended way. It has many crazy features, such as getting more domains from DNS zones, automatic nmap,...

xsssniper

XSSSNIPER xsssniper is an handy xss discovery tool with mass scanning functionalities. Usage: root@kitploit: Usage: xsssniper.py options Options: -h, --help show this help message and exit -u URL, --url=URL target URL --post try a post request to target url --data=POSTDATA post data to use...

smbcrawler

SmbCrawler SmbCrawler is no-nonsense tool that takes credentials and a list of hosts and 'crawls' or 'spiders' through those shares. Features: takes host names, IP addresses, IP ranges, or an nmap xml file as input checks permissions check for 'write' permissions is opt-in, because it requires...

Scrapling

Effortless Web Scraping for the Modern Web العربيه | Español | Português Brasil | Français | Deutsch | 简体中文 | 日本語 | Русский | 한국어 Selection methods · Fetchers · Spiders · Proxy Rotation · CLI · MCP Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a...

Instagram-Crawler

Instagram-Crawler Non API. Crawling post photo, likes, comments, date ... by username, hashtags Installation 1. Make sure you have Chrome browser installed. 2. Download chromedriver and put it into driver folder: ./driver/chromedriver 3. Install requirements pip install -r requirements.txt...

protections-artifacts

Protections Artifacts Elastic Security prevents ransomware and malware, detects advanced threats, and arms responders with vital context. It’s free and open, ready for every endpoint. Protections-Artifacts is the home of our detection logic rules, yara, etc for Elastic Security for endpoint. At...

subcrawl

SubCrawl SubCrawl is a framework developed by Patrick Schläpfer, Josh Stroschein and Alex Holland of HP Inc’s Threat Research team. SubCrawl is designed to find, scan and analyze open directories. The framework is modular, consisting of four components: input modules, processing modules, output...

dcrawl

dcrawl dcrawl is a simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names. How it works? dcrawl takes one site URL as input and detects all links in the site's body. Each found link is put into the queue. Successively, each queued link is crawled in...

gospider

GoSpider GoSpider - Fast web spider written in Go Painless integrate Gospider into your recon workflow? this project was part of Osmedeus Engine. Check out how it was integrated at @OsmedeusEngine Installation GO install root@kitploit: GO111MODULE=on go install...

burp-dom-scanner

Burp DOM Scanner It's a Burp Suite's extension to allow for recursive crawling and scanning of Single Page Applications. It runs a Chromium browser to scan the webpage for DOM-based XSS. It can also collect all the requests XHR, fetch, websockets, etc issued during the crawling allowing them to b...

MANSPIDER

MANSPIDER Crawl SMB shares for juicy information. File content searching + regex is supported! What's New in v2.0 Manspider 2.0 is here! This brings significant improvements: New and improved text extraction powered by Kreuzberg - now supporting PDF, DOCX, XLSX, PPTX, images with OCR, and many...

Pyison

Pyison Pyison is a tarpit for AI webcrawlers Introduction Like all web crawlers, AI bots request pages from webservers, and then follow links on the page to other pages. By doing this, they can build an index of an entire website. Unlike other web crawlers, though, AI bots present some unique...

AIL-framework

AIL Framework Open-source framework for the collection, crawling, processing, and analysis of unstructured information. AIL framework is an open-source platform to collect, crawl, process and analyse unstructured data from the clear web, Tor, I2P, chats, files and external feeds. Originally...

Photon

Photon Incredibly fast crawler designed for OSINT. Photon Wiki • How To Use • Compatibility • Photon Library • Contribution • Roadmap Key Features Data Extraction Photon can extract the following data while crawling: URLs in-scope & out-of-scope URLs with parameters example.com/gallery.php?id=2...

Egyscan

EgyScan V3.4 "Ultra" : Your Digital Fort of Invincibility 🏰 Greetings, Guardians of the Digital Realm! 🛡️ Prepare to embark on an extraordinary journey with EgyScan V3.0 , the ultimate sentinel that transforms your websites into impregnable fortresses against relentless cyber threats. Let's explor...

SourceWolf

SourceWolf ​ Tested environments: Windows, MAC, linux, and windows subsystem for linux WSL Sponsors Support this project by becoming a sponsor. Checkout these awesome sponsors: Oxylabs provides premium proxies with 100M+ IPs across 195 countries and ready-to-use Scraper APIs. You can use this...

socialhunter

socialhunter !IMPORTANT This tool is no longer working, and we won't be able to fix it as it has become difficult to bypass social media protections. Crawls the given URL and finds broken social media links that can be hijacked. Broken social links may allow an attacker to conduct phishing attack...

domdig

DOMDig DOMDig is a DOM XSS scanner that runs inside the Chromium web browser and it can scan single page applications SPA recursively. Unlike other scanners, DOMDig can crawl any webapplication including gmail by keeping track of DOM modifications and XHR/fetch/websocket requests and it can...

katana

下一代爬取与爬虫框架 特性 • 安装 • 用法 • 范围 • 配置 • 过滤器 • 加入Discord 特性 快速且完全可配置的网页爬取 标准 和无头 模式 JavaScript 解析/爬取 可自定义的自动表单填写 范围控制 - 预配置字段/正则 知识库 - 机器学习页面类型/表单分类（自动下载模型） 可自定义的输出 - 预配置字段 输入 - STDIN 、URL 和LIST 输出 - STDOUT 、FILE 和JSON 安装 katana需要Go...