xcrawl3r

xcrawl3r xcrawl3r is a command-line utility designed to recursively spider webpages for URLs. It works by actively traversing websites - following links embedded in webpages, and parsing files including sitemaps & robots.txt - to uncover every URL. Unlike xurlfind3r that doesn't interact directly...