Xurlfind3R - A CLI Utility To Find Domain'S Known URLs From Curated Passive Online Sources

xurlfind3r is a command-line interface (CLI) utility to find domain’s known URLs from curated passive online sources.

Features

• Fetches URLs from curated passive sources to maximize results:
  • AlienVault’s OTX *BeVigil *Common Crawl *URLScan *Github *Intelligence X *Wayback Machine
• With Wayback Machine, Parses URLs from robots.txt snapshots.
• With Wayback Machine, Parses URLs from webpages’ snapshots.
• Cross-Platform (Windows, Linux & macOS)
• Supports URLs match and filter

Installation

Install release binaries (Without Go Installed)

Visit the releases page and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with wget or curl:

• …with wget:

   wget https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz
  

• …or, with curl:

   curl -OL https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz
  

…then, extract the binary:

tar xf xurlfind3r-<version>-linux-amd64.tar.gz

> TIP: The above steps, download and extract, can be combined into a single step with this onliner
>
>
> curl -sL https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz | tar -xzv

NOTE: On Windows systems, you should be able to double-click the zip archive to extract the xurlfind3r executable.

…move the xurlfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

sudo mv xurlfind3r /usr/local/bin/

NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurlfind3r to their PATH.

Install source (With Go Installed)

Before you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.

go install ...

go install -v github.com/hueristiq/xurlfind3r/cmd/xurlfind3r@latest

go build ... the development Version

• Clone the repository

   git clone https://github.com/hueristiq/xurlfind3r.git 
  

• Build the utility

   cd xurlfind3r/cmd/xurlfind3r && \  
  

  go build .

• Move the xurlfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

   sudo mv xurlfind3r /usr/local/bin/
  

NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurlfind3r to their PATH.

NOTE: While the development version is a good way to take a peek at xurlfind3r’s latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.

Post Installation

xurlfind3r will work right after installation. However, BeVigil,GithubandIntelligence Xrequire API keys to work,URLScan supports API key but not required. The API keys are stored in the $HOME/.hueristiq/xurlfind3r/config.yaml file - created upon first run - and uses the YAML format. Multiple API keys can be specified for each of these source from which one of them will be used.

Example config.yaml:

version: 0.2.0  
sources:  
    - bevigil  
    - commoncrawl  
    - github  
    - intelx  
    - otx  
    - urlscan  
    - wayback  
keys:  
    bevigil:  
        - awA5nvpKU3N8ygkZ  
    github:  
        - d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39  
        - asdsd54bbc1aabb208c9acfbd2dd41ce7fc9db39  
    intelx:  
        - 2.intelx.io:00000000-0000-0000-0000-000000000000  
    urlscan:  
        - d4c85d34-e425-446e-d4ab-f5a3412acbe8

Usage

To display help message for xurlfind3r use the -h flag:

xurlfind3r -h

help message:

                 _  __ _           _ _____        
__  ___   _ _ __| |/ _(_)_ __   __| |___ / _ __   
\ \/ / | | | '__| | |_| | '_ \ / _` | |_ \| '__|  
 >  <| |_| | |  | |  _| | | | | (_| |___) | |  
/_/\_\\__,_|_|  |_|_| |_|_| |_|\__,_|____/|_| v0.2.0  
  
USAGE:  
  xurlfind3r [OPTIONS]  
  
TARGET:  
 -d, --domain string              (sub)domain to match URLs  
  
SCOPE:  
     --include-subdomains bool    match subdomain's URLs  
  
SOURCES:  
 -s,  --sources bool              list sources  
 -u,  --use-sources string        sources to use (default: bevigil,commoncrawl,github,intelx,otx,urlscan,wayback)  
      --skip-wayback-robots bool  with wayback, skip [parsing](<https://www.kitploit.com/search/label/Parsing> "parsing" ) [robots.txt](<https://www.kitploit.com/search/label/Robots.txt> "robots.txt" ) snapshots  
      --skip-wayback-source bool  with wayback   , skip parsing source code snapshots  
  
FILTER & MATCH:  
 -f, --filter string              regex to filter URLs  
 -m, --match string               regex to match URLs  
  
OUTPUT:  
     --no-color bool              no color mode  
 -o, --output string              output URLs file path  
 -v, --verbosity string           debug, info, warning, error, fatal or silent (default: info)  
  
CONFIGURATION:  
 -c,  --configuration string      configuration file path (default: ~/.hueristiq/xurlfind3r/config.yaml)  

Examples

Basic

xurlfind3r -d hackerone.com --include-subdomains

Filter Regex

# filter images  
xurlfind3r -d hackerone.com --include-subdomains -f '`^https?://[^/]*?/.*\.(jpg|jpeg|png|gif|bmp)(\?[^\s]*)?$`'

Match Regex

# match js URLs  
xurlfind3r -d hackerone.com --include-subdomains -m '^https?://[^/]*?/.*\.js(\?[^\s]*)?$'

Contributing

Issues and Pull Requests are welcome! Check out the contribution guidelines.

Licensing

This utility is distributed under the MIT license.

Download Xurlfind3R