Feature: Add Git LFS Support

[terrabitz]

Overview

This PR adds Git LFS support to the GitHub mode adapter. That way we can more easily add and version images to repos without cluttering up the Git history. It's enabled via a new opt-in flag.

Note

I am a bit of a noob with React sites, and did use LLM-powered tooling to assist with this PR. I have reviewed all the output and tested it to the best of my ability, but I definitely may have missed something. I feel like I have a decent understanding of all the components I touched, but I don't know what I don't know.

Background

I love Git LFS, and use it pretty much any large binaries so I don't bloat my Git history. I recently setup a Keystatic+Astro site, and was using Git LFS for some of my website image assets. But once it came to adding images via Keystatic, I started to see errors that looked like this:

Encountered 1 file that should have been a pointer, but wasn't:
	src/assets/content/my-image.png

Even though I had setup LFS and .gitattributes correctly, the GitHub content API that Keystatic was using didn't automatically support LFS. This lead to some images not getting LFS tracked, even though most were.

It looks like the community has been wanting cloud image assets support for a while (#491). It looks like the paid Keystatic Cloud option supports this with a nice image processing pipeline through Cloudflare. But I suspect for a lot of people, Git LFS would be a nice middle ground so that all the assets are tracked in Git via pointer files. GitHub offers a free tier of 10 GB LFS storage and more for paid users, which is probably enough for most people's purposes.

Implementation Notes

LFS support can be enabled via a new opt-in config called lfs

  storage: {
    kind: 'github',
    repo: {
      owner: REPO_OWNER,
      name: REPO_NAME,
    },
+    lfs: true,
  }

When enabled, Keystatic will gather information about what LFS patterns are configured in the .gitattributes. It does so via a new LfsPatternsContext.Provider in data.tsx. It looks at a file in the target repo/commit called .gitattributes, and parses it to determine which patterns have been added via lfs track. For now, I made the behavior fairly strict, such that it'll only use patterns where filter, diff, and merge are all set to LFS.

Once we have these patterns, that affects the upload path: it checks for the existence of these patterns and the LFS config, and will process any matching files through a separate LFS path. Instead of directly uploading these to the GitHub API, it instead uses a new backend API called github/lfs/upload that uses the LFS basic transfer API to upload to LFS and commit the pointer file.

Note

We needed a new backend here because the LFS APIs in GitHub don't have any CORS headers set

The download path works slightly differently: it looks at the content of each images, and determines which ones are actually an LFS pointer file. It does so by looking at the first few bytes to see if they match version https://git-lfs.github.com/spec/v1. If it is a pointer file, then we resolve it through the github/lfs/download endpoint. Once resolved, we override the existing cache entry to hold the resolved image bytes instead of just the pointer. That way subsequent pulls can use the image cache directly without having to re-resolve them.

Design Notes

There were a few design decisions I made here:

• The feature is gated with a new lfs flag. That way existing sites should not be affected at all by the new code paths. Realistically we don't need a new flag, since the existence of a .gitattributes file with LFS entries would also be a pretty good signal. But I opted to do so to stay as conservative as possible.
• The github/lfs/download endpoint does single file downloads, while the github/lfs/upload endpoint does all writes in one batch. The former is good to make sure the images are all downloaded at the same time, so that the initial resolution is as fast as possible. Downside is this is unbounded, so I don't know how it'll perform if we have hundreds of images, each making its own API call.
• The LFS file download path doesn't depend on the lfs:true config flag being set. It will always resolve LFS files regardless of this setting. This allows both forward and backwards compatibility: you can use non-LFS files in LFS mode, and vise-versa. The LFS mode only affects how files are uploaded. This makes it more robust, since you can enable and disable the LFS setting without losing any images you have previously uploaded.

Testing Notes

Apart from a few small unit tests I added for some of the new parsing logic, most of my testing was manual. I also didn't see a whole lot of end-to-end testing, so I opted not to write any. I tested in both a public and private repository to confirm there weren't any differences wrt authentication.

All of this is tested through a minimal Keystatic+Astro+Cloudflare site

• Repo: https://github.com/terrabitz/keystatic-lfs
• Deployment: https://keystatic-lfs.tjtaubit.workers.dev/keystatic
• The .gitattributes is setup so PNGs are LFS-managed, JPGs are not
• Note, I would be happy to add anyone as git collaborator to that repo so you can test out the changes yourself without having to setup your own deployment

After enabling LFS mode, previously-uploaded non-LFS files are still resolved

Adding LFS-managed files works when lfs: true is set, and we have our .gitattributes setup

Adding non-LFS managed files still works

After disabling LFS mode, the previously-uploaded LFS files are still resolved, while new files don't go through LFS

[changeset-bot]

🦋 Changeset detected

Latest commit: f48527b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages

Name	Type
@keystatic/core	Minor
@keystar/docs	Patch
@example/astro-content	Patch
@example/astro	Patch
@example/localization	Patch
@example/next-app	Patch
@example/next-block-builder	Patch
@example/next-pages	Patch
@keystatic/remix-test-app	Patch
keystatic-docs	Patch
@keystatic/astro	Major
@keystatic/next	Major
@keystatic/remix	Major
@keystatic/templates-astro	Patch
@keystatic/templates-nextjs	Patch
@keystatic/templates-remix	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR