Cache initialization fails when deploying Apache Traffic Server via HashiCorp Nomad (Docker driver)

[youssefbennour]

Description

When deploying ATS using HashiCorp Nomad's Docker driver, the cache consistently fails to initialize, rendering caching entirely non-functional.
The same setup works perfectly in a local Docker Compose environment, where cache initialization succeeds and responses are cached as expected.

Logs

[Apr  8 09:40:05.967] [ET_AIO 7:26] WARNING: cache disk operation failed READ 0 1
[Apr  8 09:40:05.967] [ET_NET 2] WARNING: could not read disk header for disk /mount/disk_1/cache.db: declaring disk bad
[Apr  8 09:40:05.967] [ET_NET 2] WARNING: failed operation: READ (opcode=1), span: /mount/disk_1/cache.db (fd=26)
[Apr  8 09:40:05.967] [ET_NET 2] WARNING: failed to initialize the cache for http: cache disabled

Steps to Reproduce

1. Deploy ATS using Nomad's Docker driver with volumes enabled on the client
2. Configure a cache span pointing to a mounted volume (e.g. /mount/disk_1/cache.db)
3. Start the Nomad job — ATS starts but cache fails to initialize

What I've Tried

1. Mounted host volume via Nomad Docker driver

Configured a persistent cache volume by mounting a host path into the container through
Nomad's config.volumes. Initially resulted in a permission denied error. After
correcting the ownership and permissions on the host directory (chown/chmod on the
host path to match the ATS process user inside the container), the permission error
was resolved — but the original cache initialization error reappeared, suggesting
the issue is deeper than filesystem permissions.

Notably, cache.db is being created on disk, so ATS does reach the point of
creating the file, but fails when attempting to read the disk header from it.

2. Adjusted filesystem permissions on the mounted volume

Applied broad ownership and permission changes on the host mount path to rule out any
access restriction:

chown -R nobody /mount/disk_1
chmod -R 777 /mount/disk_1

The same cache initialization error persisted regardless, which strongly suggests that
permissions are not the root cause of this issue.

3. Removed volume mount entirely (ephemeral local storage)

To isolate the problem, I removed the volume mount altogether and pointed the cache
to a path local to the container, with no host mount involved. The cache initialization
error persisted identically, which suggests the issue is not related to volume
mounting or host-level permissions, but rather something in how ATS interacts with
the filesystem in a Nomad-managed container environment.

4. Ran the container in privileged mode

Enabled privileged mode in the Nomad Docker driver config to rule out any
capability-related restrictions. The error remained unchanged.

Questions

• Is there a way to enable more verbose diagnostics around the READ operation that
  fails at cache startup?
• Are there any recommended records.config or storage configuration adjustments
  for containerized environments?
• Do you have any other suggestion to further debug the issue?