Your tool is great!
When I run it, it reports a half-dozen websites return HTTP status 403. When accessed with curl, they also return status 403. But when accessed from a web browser, they return status 200. Thus, it appears these sites are blocking crawlers.
I'd like to be able to add 403 as a successful status code, to eliminate these false negatives. I'm sure there are other scenarios where it would be useful to be able to set the list of "successful" status codes.
Your tool is great!
When I run it, it reports a half-dozen websites return HTTP status 403. When accessed with
curl, they also return status 403. But when accessed from a web browser, they return status 200. Thus, it appears these sites are blocking crawlers.I'd like to be able to add 403 as a successful status code, to eliminate these false negatives. I'm sure there are other scenarios where it would be useful to be able to set the list of "successful" status codes.