If you use HttpAuthMiddleware
(i.e. the http_user
and http_pass
spider attributes) for Splash authentication, any non-Splash request will expose your credentials to the request target. This includes robots.txt
requests sent by Scrapy when the ROBOTSTXT_OBEY
setting is set to True
.
Upgrade to scrapy-splash 0.8.0 and use the new SPLASH_USER
and SPLASH_PASS
settings instead to set your Splash authentication credentials safely.
If you cannot upgrade, set your Splash request credentials on a per-request basis, using the splash_headers
request parameter, instead of defining them globally using the HttpAuthMiddleware
.
Alternatively, make sure all your requests go through Splash. That includes disabling the robots.txt middleware.
If you have any questions or comments about this advisory:
CPE | Name | Operator | Version |
---|---|---|---|
scrapy-splash | eq | 0.6 | |
scrapy-splash | eq | 0.3 | |
scrapy-splash | eq | 0.7.1 | |
scrapy-splash | eq | 0.7 | |
scrapy-splash | eq | 0.6.1 | |
scrapy-splash | eq | 0.7.2 | |
scrapy-splash | eq | 0.4 | |
scrapy-splash | eq | 0.2 | |
scrapy-splash | eq | 0.5 |
github.com/scrapy-plugins/scrapy-splash
github.com/scrapy-plugins/scrapy-splash/commit/2b253e57fe64ec575079c8cdc99fe2013502ea31
github.com/scrapy-plugins/scrapy-splash/releases/tag/0.8.0
github.com/scrapy-plugins/scrapy-splash/security/advisories/GHSA-823f-cwm9-4g74
nvd.nist.gov/vuln/detail/CVE-2021-41124