AIL-framework

AIL Framework Open-source framework for the collection, crawling, processing, and analysis of unstructured information. AIL framework is an open-source platform to collect, crawl, process and analyse unstructured data from the clear web, Tor, I2P, chats, files and external feeds. Originally...

HTTPLoot

HTTPLoot An automated tool which can simultaneously crawl, fill forms, trigger error/debug pages and "loot" secrets out of the client-facing code of sites. Usage To use the tool, you can grab any one of the pre-built binaries from the Releases section of the repository. If you want to build the...

Photon

Photon Incredibly fast crawler designed for OSINT. Photon Wiki • How To Use • Compatibility • Photon Library • Contribution • Roadmap Key Features Data Extraction Photon can extract the following data while crawling: URLs in-scope & out-of-scope URLs with parameters example.com/gallery.php?id=2...

SourceWolf

SourceWolf ​ Tested environments: Windows, MAC, linux, and windows subsystem for linux WSL Sponsors Support this project by becoming a sponsor. Checkout these awesome sponsors: Oxylabs provides premium proxies with 100M+ IPs across 195 countries and ready-to-use Scraper APIs. You can use this...

festin

FestIN - パワフルなS3バケットファインダー＆コンテンツ発見 FestInとは なぜFestInか インストール Pythonを使う Dockerを使う 全オプション 使い方 検索ドメインの設定 同時実行数 HTTPクロール設定 結果の管理 プロキシの使用 DNSオプション 全文検索サポート サービス（監視モード）として実行 例：FestIn + DnsReconの組み合わせ ステップ1 - ターゲットドメインに対してdnsreconを希望のオプションで実行し、出力を保存 ステップ2 - 前の生成ファイルをFestInに与える準備 ステップ3 -...

jshole

jshole RetireJSに基づく、シンプルなJavaScriptコンポーネントの脆弱性スキャナー。 JSholeをRetireJSの代わりに使う理由 デフォルトではRetireJSは1ページのみを検索しますが、JSholeはすべてのページをクロールしようとします。 はじめに 必要条件 requests インストール git clone https://github.com/callforpapers-source/jshole.git cd jshole pip3 install -r requirements python3 jshole.py usage: jshole -h ...

subcrawl

SubCrawl SubCrawl은 HP Inc의 Threat Research 팀 소속 Patrick Schläpfer, Josh Stroschein 및 Alex Holland이 개발한 프레임워크입니다. SubCrawl은 공개 디렉토리를 찾고, 스캔하며, 분석하도록 설계되었습니다. 프레임워크는 모듈식으로, 입력 모듈, 처리 모듈, 출력 모듈 및 코어 크롤링 엔진의 네 가지 구성 요소로 이루어져 있습니다. URL이 주요 입력 값이며, 프레임워크는 이를 파싱하여 큐 시스템에 추가한 후 크롤링합니다. URL 파싱은 중요한 첫 단계로,...

katana

차세대 크롤링 및 스파이더링 프레임워크 기능 • 설치 • 사용법 • 범위 • 설정 • 필터 • Discord 참여 기능 빠르고 완전히 구성 가능한 웹 크롤링 Standard 및 Headless 모드 JavaScript 파싱 / 크롤링 사용자 정의 가능한 자동 폼 채우기 범위 제어 - 사전 구성된 필드 / Regex 지식 베이스 - ML 페이지 유형 / 폼 분류 자동 다운로드 모델 사용자 정의 가능한 출력 - 사전 구성된 필드 INPUT - STDIN , URL 및 LIST OUTPUT - STDOUT , FILE 및 JSON...

domain_analyzer

Domain Analyzer v0.8.3 What Domain analyzer is a security analysis tool that automatically discovers and reports information about a given domain. Its main purpose is to analyze domains in an unattended way. It has many crazy features, such as getting more domains from DNS zones, automatic nmap,...

domain_analyzer

Domain Analyzer v0.8.3 概要 Domain Analyzerは、指定されたドメインに関する情報を自動的に発見し報告するセキュリティ分析ツールです。主な目的は、無人でドメインを分析することです。 DNSゾーンからさらに多くのドメインを取得する、自動nmap、Webクローラー、世界征服モードなど、多数のクレイジーな機能があります。機能を確認してください。 nmapでより多くのポートをスキャンし、スクリプトを実行し、それらのWebサイトでクローラーを実行したい場合は、root権限が必要です。 デフォルトオプションの例...

dark-web-osint-tools

🕵️‍♂️ ダークウェブ向けOSINTツール 🔎 ダークウェブ検索エンジンツール Katana - https://github.com/adnane-X-tebbaa/Katana OnionSearch - https://github.com/megadose/OnionSearch Darkdump - https://github.com/josh0xA/darkdump Ahmia Search Engine - ahmia.fi, https://github.com/ahmia/ahmia-site Darkus -...

gospider

GoSpider GoSpider - Fast web spider written in Go Painless integrate Gospider into your recon workflow? this project was part of Osmedeus Engine. Check out how it was integrated at @OsmedeusEngine Installation GO install GO111MODULE=on go install github.com/jaeles-project/gospider@latest Docker...

responsible-disclosure-email-gathering

責任ある開示メールの収集 目的 ホストに脆弱性を発見したが、そのホスト/組織がHackerOneやBugCrowdでアクティブなプログラムを持っていない場合があります。そのような場合、このワークフローが役立ちます！大まかに言うと、ウェブサイトをクロールし、調査結果を提出できるセキュリティ関連のメールを探します。 前提条件 Katana Nuclei アクティブなインターネット接続 プロセス このワークフローは以下を行います: 1. Katanaを使用してターゲットホストをクロールする 2. Katanaからの出力がNucleiに解析される 3...

Lurkstrain

Lurkstrain An all in one admin panel crawler for pentesters...

CVE-2026-84366

Scrapy is a high-level web crawling and scraping framework for Python. Prior to 2.17.0, in scrapy/core/downloader/handlers/s3.py, Scrapy's S3DownloadHandler converts an S3-scheme bucket and key request into a plaintext HTTP request to the corresponding S3 endpoint unless request.meta"issecure" is...

UBUNTU-CVE-2026-84366

Scrapy is a high-level web crawling and scraping framework for Python. Prior to 2.17.0, in scrapy/core/downloader/handlers/s3.py, Scrapy's S3DownloadHandler converts an S3-scheme bucket and key request into a plaintext HTTP request to the corresponding S3 endpoint unless request.meta"issecure" is...

CVE-2026-76993

A vulnerability was determined in GreyDGL PentestGPT up to 1.0.0. This vulnerability affects unknown code of the component Web-Page Crawling. Executing a manipulation of the argument Traceback can lead to injection. The attack can be executed remotely. A high complexity level is associated with...

EUVD-2026-63377

A vulnerability was determined in GreyDGL PentestGPT up to 1.0.0. This vulnerability affects unknown code of the component Web-Page Crawling. Executing a manipulation of the argument Traceback can lead to injection. The attack can be executed remotely. A high complexity level is associated with...

CVE-2026-76993 GreyDGL PentestGPT Web-Page Crawling injection

A vulnerability was determined in GreyDGL PentestGPT up to 1.0.0. This vulnerability affects unknown code of the component Web-Page Crawling. Executing a manipulation of the argument Traceback can lead to injection. The attack can be executed remotely. A high complexity level is associated with...

PT-2026-79029

A vulnerability was determined in GreyDGL PentestGPT up to 1.0.0. This vulnerability affects unknown code of the component Web-Page Crawling. Executing a manipulation of the argument Traceback can lead to injection. The attack can be executed remotely. A high complexity level is associated with...