content app take a very long time to render HTML directory listing

[dkliban]

Version
3.103.1

Describe the bug
We have a very large Pulp deployment. The first time a directory is listed in the browser (cache MISS), the request takes 35 to 50 seconds to be fulfilled.

[dralley]

This was reproduced at a likely smaller scale with pulp/pulp_rpm#4287

In that case we reduced the # of directories.

[sdherr]

Be aware that directory listings are not very cacheable in a CDN. When repos are republished the directory listings will need to change too. Actual clients (dnf, apt, etc) don't care because they don't use directory listings, but any human browsing the repo or any web-crawler cloning tool will be unstable with cached directory listings.

The default TTL of Redis cache content is only 10 minutes which is probably not long enough to really help in this case, but if that were made longer that would probably be the best solution. It'll still be slow the first time though.