Possible HTTP/2 Incompatibility

[sparked435]

This might not be a lua-http issue; but only lua-http shows this behavior on sites using the Fastly CDN. The following works, outputting "200", as expected:

local req = httpreq.new_from_uri( 'https://www.bbc.com/robots.txt' )
req.version = 1

local headers, body = req:go(5)
print(headers:get(':status'))

Switching req.version to either nil or 2 returns a 403. I spent some time ensuring Curl and lua-http were sending identical headers; lua-http gets 403 for HTTP/2 and Curl always gets a 200 regardless of version. Web browsers consistently succeed as well.

Environment is Lua 5.4, lua-http 0.4, cqueues 20200726.54, and luaossl 20220711.

[daurnimator]

It appears as though BBC have opted into some kind of fastly "bot-prevention" feature?

[sparked435]

Whatever Fastly is doing was probably was intended to be something like that.

That said, I can run a few hundred requests with lua-http returning 403, and then run Curl with the exact same headers from the exact same IP and get a 200 - so it's upset at something lua-http specific and not the IP appearing to act like a bot/scraping/hitting an denied URL.

Is there a way to force lua-http to speak HTTP/2 but over unencrypted TCP? That way I could use stunnel and try to debug further with a packet sniffer.

[daurnimator]

Is there a way to force lua-http to speak HTTP/2 but over unencrypted TCP? That way I could use stunnel and try to debug further with a packet sniffer.

sure. you should be able to use a http url and then set

req.version = 2