Parsero

Parseroとは？ ParseroはPythonで書かれた無料のスクリプトで、WebサーバーのRobots.txtファイルを読み込み、Disallowエントリを調べます。Disallowエントリは、Webサーバー上にあるどのディレクトリやファイルがインデックスされるべきでないかを検索エンジンに伝えます。例えば、"Disallow:...

Parsero v0.75 - Attacking Robots.txt Files

Parsero is a free script written in Python which reads the Robots.txt file of a web server and looks at the Disallow entries. The Disallow entries tell the search engines what directories or files hosted on a web server mustn't be indexed. For example, "Disallow: /portal/login" means that the...

[Parsero] Robots.txt audit tool

.PNG Parsero is a free script written in Python which reads the Robots.txt file of a web server and looks at the Disallow entries. The Disallow entries are the URL path of directories or files hosted on a web server which the administrators don't want to be indexed by the crawlers. For example,...