Cache for tasks

[Matios102]

#52

This pull request introduces a file caching mechanism for test case files to reduce redundant downloads and improve efficiency in the worker service. The main changes include adding a persistent file cache with versioning and TTL, integrating it into the packager stage, updating configuration and dependencies, and providing mock support for testing.

File caching and integration:

• Added a new FileCache interface and implementation in internal/storage/cache.go, which manages cached files with versioning, TTL, and metadata storage. The cache is initialized at startup and cleans expired entries automatically.
• Updated the packager component to use the new file cache: test case files are now retrieved from cache if available and valid, or downloaded and cached otherwise. This is handled in the new downloadOrCopyFromCache and downloadAndCache methods. [1] [2] [3] [4] [5] [6]
• The TaskQueueMessage struct now includes a TaskFilesVersion field to track versioning of task files for cache validation.

Configuration and environment updates:

• Added cache-related constants (cache directory, TTL, metadata file) to pkg/constants/constants.go and mounted the cache directory as a Docker volume in docker-compose.yaml for persistence across container restarts. [1] [2] [3]

Testing and dependency updates:

• Added the FileCache interface to the mock generation script and provided a generated mock for use in tests. [1] [2]
• Updated go.mod to include new dependencies required for testing and cache implementation. [1] [2] [3] [4]

[HermanPlay]

• Please limit amount of cached entries to not overflow memory
• add a mechanism to invalidate cache manually for backend, for example when task test cases change
• what's the point of metadata file if cache is in memory and is deleted on restart the same way as metadata file is deleted

[Matios102]

@HermanPlay

• Please limit amount of cached entries to not overflow memory
• add a mechanism to invalidate cache manually for backend, for example when task test cases change
• what's the point of metadata file if cache is in memory and is deleted on restart the same way as metadata file is deleted

1. ok
2. There is a TaskFilesVersion in the TaskQueueMessage struct which is used to detect changes so you need to store updated_at or something like that
3. The cache is mounted on a volume so he json file is used to restore in memory entries after a restart

[HermanPlay]

@HermanPlay

• Please limit amount of cached entries to not overflow memory

• add a mechanism to invalidate cache manually for backend, for example when task test cases change

• what's the point of metadata file if cache is in memory and is deleted on restart the same way as metadata file is deleted

2. There is a TaskFilesVersion in the TaskQueueMessage struct which is used to detect changes so you need to store updated_at or something like that

I am not storing version but I have updated at field. Do you want me to send it to you?

3. The cache is mounted on a volume so he json file is used to restore in memory entries after a restart

Do we need it?