urx

从 OSINT 档案中提取 URL，助力安全洞察。 Urx 是一款命令行工具，专为从 OSINT 档案（如 Wayback Machine 和 Common Crawl）中收集 URL 而设计。它使用 Rust 构建以追求高效，利用异步处理快速查询多个数据源。该工具简化了为指定域名收集 URL 信息的过程，提供全面的数据集，可用于多种用途，包括安全测试和分析。 功能特性 并行从多个来源获取 URL（Wayback Machine、Common Crawl、OTX、Arquivo.pt） 通过 --cdx-endpoint URL 接入任何其他 CDX 索引服务器——国家网络档案、私有...

gau

getallurls gau getallurls gau 从 AlienVault 的 Open Threat Exchange、Wayback Machine、Common Crawl 和 URLScan 获取给定域名的已知 URL。灵感来源于 Tomnomnom 的 waybackurls。 资源 使用 安装 ohmyzsh 说明 使用： 示例： $ printf example.com | gau $ cat domains.txt | gau --threads 5 $ gau example.com google.com $ gau --o example-urls.txt...

warcannon

WARCannon - Catastrophically powerful parallel WARC processing WARCannon was built to simplify and cheapify the process of 'grepping the internet'. With WARCannon, you can: Build and test regex patterns against real Common Crawl data Easily load Common Crawl datasets for parallel processing Scale...

warcannon

WARCannon - 灾难级强大的并行WARC处理工具 WARCannon 旨在简化和降低“搜索互联网”这一过程的成本与复杂度。 通过 WARCannon，你可以： 构建并针对真实的 Common Crawl 数据测试正则表达式模式 轻松加载 Common Crawl 数据集以进行并行处理 扩展计算能力，以相当不合理的吞吐量异步处理 WARC 文件 存储并轻松检索结果 工作原理 WARCannon 巧妙利用 AWS 技术实现水平扩展至任意规模，通过竞价实例集群和同区域数据传输降低成本，以惊人速度（每节点高达 100Gbps）从 S3 拉取数据，跨数百个 CPU 核心并行处理，通过...

xurlfind3r

xurlfind3r xurlfind3r is a command-line utility designed to discover URLs for a given domain in a simple, efficient way. It works by gathering information from a variety of passive sources, meaning it doesn't interact directly with the target but instead gathers data that is already publicly...

paskto

Paskto - 被动式Web扫描器 Paskto 将使用 Common Crawl 互联网索引被动扫描Web，既可以按需下载索引，也可以解析本地系统中的数据。URL 随后通过 Nikto 和已知 URL 列表进行处理，以识别感兴趣的内容。哈希签名也用于识别某些物联网设备或Web应用程序的已知默认内容。 选项 -d, --dir-input directory 包含 Common Crawl 索引文件（.gz 扩展名）的目录。例如：-d "/tmp/cc/" -v, --ia-dir-input directory 包含互联网档案馆索引文件（.gz 扩展名）的目录。例如：-v...

paskto

Paskto - Passive Web Scanner Paskto will passively scan the web using the Common Crawl internet index either by downloading the indexes on request or parsing data from your local system. URLs are then processed through Nikto and known URL lists to identify interesting content. Hash signatures are...

Sigurlfind3R - A Reconnaissance Tool, It Fetches URLs From AlienVault's OTX, Common Crawl, URLScan, Github And The Wayback Machine

sigurlfind3r is a passive reconnaissance tool, it fetches known URLs from AlienVault's OTX , Common Crawl , URLScan , Github and the Wayback Machine. DiSCLAIMER: fetching urls from github is a bit slow. Usage sigurlfind3r -h This will display help for the tool. | |/ | | / / | |/ | | | | '| | || |...

Sigurls - A Reconnaissance Tool, It Fetches URLs From AlienVault's OTX, Common Crawl, URLScan, Github And The Wayback Machine

sigurls is a reconnaissance tool, it fetches URLs from AlienVault's OTX , Common Crawl , URLScan , Github and the Wayback Machine. Usage To display help message for sigurls use the -h flag: $ sigurls -h | | / | |/ | | | | '| / | \ \ | | | || | | | \ \ |/|, |,|| ||/ v1.3.1 |/ USAGE: sigurls...

CWFF - Create Your Custom Wordlist For Fuzzing

CWFF is a tool that creates a special High quality fuzzing/content discovery wordlist for you at the highest speed possible using concurrency and it's heavily inspired by @tomnomnom's Who, What, Where, When, Wordlist NahamCon2020. Usage CWFF -h --threads --github --subdomains --recursive...