wxpath

wxpath - declarative web graph traversal with XPath NEW: TUI - Interactive terminal interface powered by Textual for testing wxpath expressions and exporting data. Install Requires Python 3.10+. root@kitploit: pip install wxpath For TUI support: pip install "wxpathtui" Immediately launch the TUI...

dcrawl

dcrawl dcrawl is a simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names. How it works? dcrawl takes one site URL as input and detects all links in the site's body. Each found link is put into the queue. Successively, each queued link is crawled in...

ache

ACHE Focused Crawler ACHE is a focused web crawler. It collects web pages that satisfy some specific criteria, e.g., pages that belong to a given domain or that contain a user-specified pattern. ACHE differs from generic crawlers in sense that it uses page classifiers to distinguish between...

EasySpider

易采集/EasySpider: Visual Code-Free Web Crawler 一个完全免费 （包括商业使用和二次开发 ）的可视化浏览器自动化测试/数据采集/爬虫软件，可以使用图形化界面，无代码可视化的设计和执行任务。只需要在网页上选择自己想要操作的内容并根据提示框操作即可完成任务的设计和执行。同时软件还可以单独以命令行的方式进行执行，从而可以很方便的嵌入到其他系统中。 A completely free including for commercial use and secondary development visual browser automation...

PenCrawLer

PenCrawLer An Advanced Web Crawler and DirBuster PeNCrawLer is an advanced webcrawler and dirbuster designed to use in penetration testing based on Windows Os. To read more about features please read the wiki: https://github.com/TheM4hd1/PenCrawLer/wiki...

hakrawler

Hakrawler Fast golang web crawler for gathering URLs and JavaScript file locations. This is basically a simple implementation of the awesome Gocolly library. Example usages Single URL: root@kitploit: echo https://google.com | hakrawler Multiple URLs: root@kitploit: cat urls.txt | hakrawler Timeou...

domain_analyzer

Domain Analyzer v0.8.3 What Domain analyzer is a security analysis tool that automatically discovers and reports information about a given domain. Its main purpose is to analyze domains in an unattended way. It has many crazy features, such as getting more domains from DNS zones, automatic nmap,...

dirhunt

.. image:: https://raw.githubusercontent.com/Nekmo/dirhunt/v0.2.0/images/dirhunt.png | .. image:: https://raw.githubusercontent.com/Nekmo/dirhunt/pip-rating-badge/pip-rating-badge.svg :target: https://github.com/Nekmo/dirhunt/actions/workflows/pip-rating.yml :alt: pip-rating badge .. image::...

sqlifinder

SQL Injection scanner made with python. Features • Install • Usage • sqlifinder is a tool with the function of detecting GET-based sql injection vulnerabilities in web applications using waybackurls, web crawlers and sql injection payloads. Features Fast and powerful scanner Include web crawler a...

SpiderSuite

SpiderSuite is an advance cross-platform and multi-feature web crawler designed for security professionals. For more information visit SpiderSuite's website. Installation and Usage Spider Suite is designed for easy installation and usage even for first timers. First, download the package of your...

second-order

Second Order 扫描Web应用程序以寻找二次子域名劫持漏洞，通过爬取应用并收集符合特定规则或以特定方式响应的URL（及其他数据）。 安装 从二进制文件安装 从发布页面下载预构建的二进制文件并解压。 从源码安装 推荐使用Go 1.17版本。 root@kitploit: go install -v github.com/mhmdiaa/second-order@latest Docker root@kitploit: docker pull mhmdiaa/second-order 命令行选项 root@kitploit: -target string 目标URL -config...

spider

Spider The fastest web crawler and scraper for Rust. spider.cloud · Guides · Docs · Examples · Discord Spider is a concurrency-first crawling engine built in Rust. It streams pages the moment they arrive, renders JavaScript only when a page demands it, and scales from a single script to a...

xcrawl3r

xcrawl3r xcrawl3r 是一个命令行工具，用于递归爬取网页以获取 URL。它通过主动遍历网站——跟踪网页中的链接并解析文件（包括站点地图和 robots.txt）——来发现所有 URL。 与 xurlfind3r 不同，xcrawl3r 不直接与目标交互，而是通过实时爬取页面来直接与目标交互。这种主动方式使其能够发现可能被隐藏或未索引的 URL，从而提供网站导航流程和内容分布的完整视图。这使得 xcrawl3r 成为安全研究人员、IT 专业人员以及任何希望深入了解网站相关 URL 的用户的强大工具。 资源 功能 安装 安装发布版二进制文件（未安装 Go） 从源码安装（已安装...

grubcrawler

Agentic web crawler with anti-detection, vision fallback, and peer-to-peer mesh. Endpoints · Mesh · Anti-Detection · Ghost Protocol · Live Stream · MCP Tools · Quick Start · Benchmarks · Architecture Full-stack web crawling engine with JavaScript rendering, Camoufox anti-detect browser, per-reque...

sql-global-injector

Ultimate-Dork web crawler And Scan SQL injection vulnerability Usage & Installation root@kitploit: - Installation apt-get install python python3 git git clone https://github.com/nu11secur1ty/sql-global-injector.git python3 -m pip install requests mechanicalsoup cd sql-global-injector - Usage...

evine

Evine 交互式 CLI 网络爬虫。 Evine 是一个用 Golang 编写的简单、快速且交互式的网络爬虫和网页抓取工具。 Evine 适用于多种用途，例如元数据和数据提取、数据挖掘、侦察和测试。 如果您喜欢这个项目，请给它一个星标。这能激励我继续开发该项目！ 安装 从二进制文件 预编译的二进制发布版也提供（建议使用）。 从源码 root@kitploit: go get github.com/saeeddhqan/evine "$GOPATH/bin/evine" -h 从 GitHub root@kitploit: git clone...

evine

Evine Interactive CLI Web Crawler. Evine is a simple, fast, and interactive web crawler and web scraper written in Golang. Evine is useful for a wide range of purposes such as metadata and data extraction, data mining, reconnaissance and testing. If you like the project, give it a star. It forces...

JAW

J A W Website | Docs | Setup | Crawler | Quick Start Docker Example JAW An open-source, prototype implementation of property graphs for JavaScript based on the esprima parser, and the . JAW can be used for analyzing the client-side of web applications and JavaScript-based programs...

hakrawler

Hakrawler 用于收集URL和JavaScript文件位置的高速Go语言网络爬虫。这基本上是优秀的Gocolly库的一个简单实现。 使用示例 单个URL： root@kitploit: echo https://google.com | hakrawler 多个URL： root@kitploit: cat urls.txt | hakrawler 每行标准输入的超时时间设为5秒： root@kitploit: cat urls.txt | hakrawler -timeout 5 通过代理发送所有请求： root@kitploit: cat urls.txt | hakrawle...

gospider

GoSpider GoSpider - 用Go编写的快速网络爬虫 无需痛苦地将Gospider集成到你的侦察工作流中？ 该项目是Osmedeus引擎的一部分。查看它在@OsmedeusEngine上的集成方式 安装 GO安装 root@kitploit: GO111MODULE=on go install github.com/jaeles-project/gospider@latest Docker root@kitploit: 克隆仓库 git clone https://github.com/jaeles-project/gospider.git 构建容器 docker build...