Cache from s3 instead of fuse #1138
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
mrge found 7 issues across 7 files. View them in mrge.io
| ForcePathStyle: c.config.ImageService.Registries.S3.Primary.ForcePathStyle, | ||
| }, blobcache.StoreContentOptions{ | ||
| CreateCacheFSEntry: true, | ||
| RoutingKey: sourcePath, |
There was a problem hiding this comment.
The sourcePath construction is changed but still used as the routing key, which might cause inconsistency with previously cached items
| nodeFilters: | ||
| - server:0 | ||
| - volume: $PWD/.k3d/storage:k3s-storage | ||
| - volume: $HOME/.beta9-k3d/storage:k3s-storage |
There was a problem hiding this comment.
Should probably merge main into this branch. I think you already merged this change in right?
| client := c.fileCacheManager.GetClient() | ||
|
|
||
| // Remove the leading "/" from the checkpoint path | ||
| sourcePath := checkpointPath[1:] |
There was a problem hiding this comment.
I actually don't think this // Remove the leading "/" from the checkpoint path is necessary anymore. But maybe leave it for now.
pkg/worker/image.go
Outdated
| operation := func() error { | ||
| baseBlobFsContentPath := fmt.Sprintf("%s/%s", baseFileCachePath, sourcePath) | ||
| if _, err := os.Stat(baseBlobFsContentPath); err == nil && c.cacheClient.IsPathCachedNearby(ctx, "/"+sourcePath) { | ||
| if _, err := os.Stat(baseBlobFsContentPath); err == nil && c.cacheClient.IsPathCachedNearby(ctx, "/"+imageKey) { |
There was a problem hiding this comment.
I would triple check that the routing key and the image key are exactly the same. This caused an issue in prod last night because the routing key and path passed here were different. I think below you're using the sourcePath (the full s3 qualifier), and up here in the filesystem you're using the image key.
Resolve BE-2569
Summary by mrge
Switched file caching to use S3 as the source instead of FUSE, improving reliability and simplifying the caching process.
Requires beam-cloud/blobcache-v2#49
beam-cloud/geesefs#5