Parsero

什么是 Parsero？ Parsero 是一个用 Python 编写的免费脚本，用于读取 Web 服务器的 Robots.txt 文件并查看 Disallow 条目。Disallow 条目告知搜索引擎，Web 服务器上托管的哪些目录或文件不得被索引。例如，"Disallow: /portal/login" 意味着 www.example.com/portal/login 上的内容不允许被 Google、Bing、Yahoo 等爬虫索引。这是管理员不向搜索引擎分享敏感或私人信息的一种方式。 但有时这些在 Disallow 条目中输入的路径，用户无需使用搜索引擎，只需访问 URL...

Parsero v0.75 - Attacking Robots.txt Files

Parsero is a free script written in Python which reads the Robots.txt file of a web server and looks at the Disallow entries. The Disallow entries tell the search engines what directories or files hosted on a web server mustn't be indexed. For example, "Disallow: /portal/login" means that the...

[Parsero] Robots.txt audit tool

.PNG Parsero is a free script written in Python which reads the Robots.txt file of a web server and looks at the Disallow entries. The Disallow entries are the URL path of directories or files hosted on a web server which the administrators don't want to be indexed by the crawlers. For example,...