Embed image metadata into OCI index #42
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: luke <luke@smartshare.io>
|
Cursor Agent can help with this pull request. Just |
luke-lombardi
deleted the
cursor/embed-image-metadata-into-oci-index-3d35
branch
luke-lombardi
added a commit
that referenced
this pull request
Nov 1, 2025
* feat: Add OCI image support and clipctl CLI This commit introduces support for OCI container images, enabling lazy loading and zero data duplication. It also includes the `clipctl` command-line tool for indexing and mounting these images. Co-authored-by: luke <luke@smartshare.io> * feat: Implement OCI image indexing and lazy loading Co-authored-by: luke <luke@smartshare.io> * Remove clipctl CLI and promote programmatic API Co-authored-by: luke <luke@smartshare.io> * fix test * Fix: Handle symlink targets and add tests Co-authored-by: luke <luke@smartshare.io> * Refactor: Stabilize overlayfs mounts and remove test files Add delay and verification for FUSE mounts. Remove unused symlink tests. Co-authored-by: luke <luke@smartshare.io> * feat: Add debug script for FUSE mount issues Co-authored-by: luke <luke@smartshare.io> * Checkpoint before follow-up message Co-authored-by: luke <luke@smartshare.io> * feat: Add OCI content caching and v2 indexing Introduces layer caching for OCI storage and enables direct indexing from OCI registries for Clip v2. This significantly speeds up image builds and reduces archive sizes. Co-authored-by: luke <luke@smartshare.io> * Refactor OCI storage to simplify caching and improve testability Co-authored-by: luke <luke@smartshare.io> * feat: Add comprehensive OCI v2 format verification tests Co-authored-by: luke <luke@smartshare.io> * feat: Optimize OCI indexing and add performance tests Co-authored-by: luke <luke@smartshare.io> * Fix: Exclude runtime directories /proc, /sys, /dev from OCI index Co-authored-by: luke <luke@smartshare.io> * Fix: Ensure parent directories exist for all node types Co-authored-by: luke <luke@smartshare.io> * Refactor: Simplify OCI indexing by removing parent dir creation Co-authored-by: luke <luke@smartshare.io> * feat: Add FUSE metadata testing strategy and tests Co-authored-by: luke <luke@smartshare.io> * Remove overlayfs and runtime dir filtering Co-authored-by: luke <luke@smartshare.io> * Fix: Set Nlink attribute for directories and files Co-authored-by: luke <luke@smartshare.io> * Fix: Add missing FUSE attributes for OCI indexing Co-authored-by: luke <luke@smartshare.io> * Checkpoint before follow-up message Co-authored-by: luke <luke@smartshare.io> * Checkpoint before follow-up message Co-authored-by: luke <luke@smartshare.io> * feat: Implement disk and remote layer caching Replaces in-memory layer cache with disk cache and integrates with remote content cache. This significantly reduces memory usage and improves performance by avoiding repeated decompression. Co-authored-by: luke <luke@smartshare.io> * feat: Implement cross-image disk cache sharing Co-authored-by: luke <luke@smartshare.io> * feat: Audit OCI indexing and confirm correctness Co-authored-by: luke <luke@smartshare.io> * feat: Add checkpointing analysis and benchmarks Co-authored-by: luke <luke@smartshare.io> * feat: Add content-defined checkpoints for faster reads Co-authored-by: luke <luke@smartshare.io> * feat: Use content hash for remote cache keys Co-authored-by: luke <luke@smartshare.io> * Fix: Add complete FUSE attributes to root node Co-authored-by: luke <luke@smartshare.io> * remove some old files * feat: Implement ContentCache range reads for lazy loading Co-authored-by: luke <luke@smartshare.io> * feat: Implement ContentCache range reads for lazy loading Co-authored-by: luke <luke@smartshare.io> * feat: Implement ContentCache range reads and CI fixes Co-authored-by: luke <luke@smartshare.io> * Skip FUSE and Docker integration tests in CI Co-authored-by: luke <luke@smartshare.io> * Fix: Skip FUSE and Docker integration tests in CI Co-authored-by: luke <luke@smartshare.io> * feat: Implement lazy OCI image FUSE with range reads Co-authored-by: luke <luke@smartshare.io> * remove md files * remove more files * more cleanup * remove unused tests * remove comment * move metrics * wip * add some verbose logs * Consolidate and structure all logging (#31) * Refactor: Remove verbose flag, use debug logging Co-authored-by: luke <luke@smartshare.io> * feat: Add SetLogLevel function and logging documentation Removes verbose flags and adds a function to control log levels. Co-authored-by: luke <luke@smartshare.io> * feat: Add OCI image indexing progress reporting Co-authored-by: luke <luke@smartshare.io> * remove logging test * remove usage * remove test --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Luke Lombardi <luke@beam.cloud> * Audit and improve oci image caching behavior (#32) * Refactor: Improve OCI caching memory efficiency and consistency Co-authored-by: luke <luke@smartshare.io> * remove files --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Luke Lombardi <luke@beam.cloud> * move tests * move some tests around * cleanup * Fix OCI indexing cache lookup order (#35) * Refactor: Fix OCI cache lookup and hash generation Co-authored-by: luke <luke@smartshare.io> * feat: Pass ContentCache to OCI storage layer Fixes ContentCache passthrough issue, enabling cluster-wide layer sharing. Co-authored-by: luke <luke@smartshare.io> * Refactor: Standardize OCI cache key format to sha256_hash Co-authored-by: luke <luke@smartshare.io> * Refactor: Use pure hex hash for cache keys Use just the hex hash for cache keys, enabling true content-addressing and deduplication. Co-authored-by: luke <luke@smartshare.io> * Refactor: Use decompressed hash for content caching This change introduces a new cache for mapping layer digests to their decompressed hashes. This allows for true content-addressable storage and sharing of decompressed layers across different CLIP images. The cache key for remote content cache is now the decompressed hash, enabling efficient range reads and deduplication. Co-authored-by: luke <luke@smartshare.io> * feat: Compute and store decompressed layer hashes during indexing Co-authored-by: luke <luke@smartshare.io> * Refactor: Improve OCI indexer and layer caching tests (#37) This commit refactors the OCI indexer for better performance and readability. It also enhances layer caching tests and fixes existing storage tests. Co-authored-by: Cursor Agent <cursoragent@cursor.com> * clean up --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Luke Lombardi <luke@beam.cloud> * feat: Add minimum gap between OCI checkpoints (#38) Co-authored-by: Cursor Agent <cursoragent@cursor.com> * clean up logs * Enable gzip checkpoint seeking for oci layers (#39) * feat: Add checkpoint-based OCI layer decompression Co-authored-by: luke <luke@smartshare.io> * Refactor: Move NearestCheckpoint to common and enable checkpoints Co-authored-by: luke <luke@smartshare.io> * remove md files --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Luke Lombardi <luke@beam.cloud> * add content cache available flag * Make registry authentication pluggable (#40) * feat: Implement pluggable OCI registry authentication Co-authored-by: luke <luke@smartshare.io> * cleanup * more cleanup --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Luke Lombardi <luke@beam.cloud> * Fix OCI credential provider for beta9 integration (#41) * Refactor OCI auth to use authn.FromConfig Co-authored-by: luke <luke@smartshare.io> * feat: Add ECR and AWS credential providers Introduces new credential providers for AWS ECR and general AWS credentials. Enhances static provider with wildcard pattern matching for registries. Improves credential type detection and parsing logic. Co-authored-by: luke <luke@smartshare.io> * Refactor credential provider logic for better registry support Co-authored-by: luke <luke@smartshare.io> * Refactor: Improve credential parsing and logging This commit enhances the credential parsing logic to handle nested JSON structures, particularly those generated by Beta9. It also refines logging within the ECR provider and OCI storage components for better visibility into credential retrieval and usage. Co-authored-by: luke <luke@smartshare.io> * Refactor ParseCredentialsFromJSON to support beta9 format Co-authored-by: luke <luke@smartshare.io> * Refactor logging for credential provider creation Co-authored-by: luke <luke@smartshare.io> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> * tidy * Embed image metadata into OCI index (#42) * feat: Embed OCI image metadata in clip index Co-authored-by: luke <luke@smartshare.io> * remove md --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Luke Lombardi <luke@beam.cloud> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Luke Lombardi <luke@beam.cloud>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request contains changes generated by a Cursor Cloud Agent