Feature Request: availibity of tar and zip in the docker image to use with git-archive

[NBardelot]

It seems that a reasonable usage of --exechook-command / $GITSYNC_EXECHOOK_COMMAND is to run a git-archive command on the local repository after it has been synced from the original source (for example, it's a very convenient way to build a backup).

See the git-archive manpage: the --format= of such an archive is usually some very classical tar, or zip, or tgz.

Thus, it would seem helpful that the git-sync container image provides the tar, gzip and or zip (ideally all of them). Especially since it is hard to add them afterwards. For example, when building a custom image based on git-sync the environnement is already minimal, with no package management tool ready to use, and very few dynamic libraries.

Another option would be to continue having a basic git-sync image, but offer a complementary image with busybox tools on top of it, more tailored to the needs of those who run exec hooks and need a little bit of tooling.

[thockin]

The more tools we include in the image, the more opportunities there are for bugs and the more I have to rebuild just for CVEs. By default it runs as non-root, so it's not ACTUALLY a big deal, but the way scanners work today ignores that, and people's dashboards light up anyway.

I don't think I understand the use-case -- why do you need a backup of a git repo that you, just moments ago, synced from an upstream?

[NBardelot]

@thockin thanks for the quick answer. Here are two git centric real world use-cases where we'd want git-archive running as a hook:

1. Running git-archive as a post-sync hook in order to keep an archive of the flattened content a complex git projet with multiple submodules. Such an archive is easier to store as a short-term backup that does not rely on the central git repository.
2. Serving the same archive from a shared ephemeral RAMFS volume using git-daemon as a local cache, decoupling the consumers from connecting to the central repository, while not changing the consumers behavior from an architecture standpoint (we don't want to switch to object storage, with consumers needing to switch API/SDK right now).

As a user of git-sync, I feel it's made to help git centric designs in Kubernetes, and git-archive without tar or tgz means having to rebuild a container image from git-sync and adding tar as a 100% static binary. Which means building from sources with cc, something like this:

ARG CACHE_REGISTRY
ARG GITSYNC_IMAGE_VERSION="latest"

# Use Alpine to build a fully statically linked tar binary,
# because we can't rely on the libs in the git-sync image
FROM ${CACHE_REGISTRY}alpine:latest AS static_tar_builder
ENV FORCE_UNSAFE_CONFIGURE=1
COPY resources/tar-1.34.tar.gz tar-1.34.tar.gz
RUN apk add --no-cache build-base && \
    tar -xzf tar-1.34.tar.gz && \
    cd tar-1.34 && \
    ./configure CFLAGS='-static' --prefix=/static && \
    make && \
    make install LDFLAGS='-static' && \
    strip /static/bin/tar

FROM ${CACHE_REGISTRY}git-sync:$GITSYNC_IMAGE_VERSION
COPY --from=static_tar_builder /static/bin/tar /usr/local/bin/tar

I find it very, very ugly to start from a really nice helper image and paradoxically end up maintaining such an image ^^

Impact of adding GNU TAR alone would mean:

• Adding ~850KB to the image.
• Exposing the image to ~10 CVE in a decade (i found less than 10 CVE between 2016 and now).

It's a core utility after all.

[thockin]

Can you help me understand how this is being used? Is it running as a sidecar which produces a tgz which is consumed by another container in the pod? Or is it re-serving the tgz through some other mechanism to clients which only know how to pull a tgz? Or something else?

I guess I would be open to looking at a PR if it included a reasonable e2e test

[NBardelot]

The design is one pod, with two associated volumes, one ephemeral and one persistent (NFS storage, with poor I/O performance).

In the pod:

• A git-sync is executed once (and allowed to fail) as an init container, and syncs the repository in the persistent NFS volume.
• Then a main git-sync container is executed infinitely (and allowed to fail) and keeps the repository in sync in the persistent NFS volume at regular intervals.
• Both git-sync executions use the same post-sync hook using a small shell script mounted via a configmap:
  • they make an archive of the repository with git-archive if the hash has changed or if the ephemeral volume is empty;
  • they store the archive's content in the ephemeral volume;
  • in the ephemeral volume they update a symbolic link to the archive's content last hash (to have a somewhat atomic update).
• In parallel a second container runs git-daemon after a small initial delay:
  • if the ephemeral storage is empty it initializes it by building an archive first using git-archive the same way the hooks do (NOTE : this is needed when/if the git-sync cannot run correctly, for example if the remote repository is offline and the pod has been moved from one node to another);
  • and then it serves the last archive's content (using the symlink of the last hash) locally in the namespace.

Thus, if the remote repository is offline and/or the git-sync fails to init, the daemon keeps serving the archive from NFS.

The clients in the namespace are able to make heavy use of the local repository since the RAMFS of the ephemeral volume provides good I/O performances. And the consumption of the repo content is decoupled from the remote repository.

And finally, the complex submodule structure of the remote repository is simply hidden to the clients in the namespace. Which would not be possible with a single git-sync, because then clients would try to init the submodules and this would imply serving those submodules on top of the main repo (and having to tweak some insteadOf stuff in the git config...).

[thockin]

If you already have a git-daemon container, why can't you run git archive from there? Or from a 3rd container which just does that task? Why does it need to be a git-sync hook?

[NBardelot]

Coding it in the same container than the git-daemon would be a bad container architecture (1 container .. 1 main process design, and the main process of the daemon is already to serve the repository, nothing else).

Adding yet another container to listen for changes made by git-sync (using a webhook instead of an exec hook) would be possible. But why make things this much harder and complicated? It would necessitate an image with a web server, equiped with a git SDK or CLI... While git-sync is already totally designed to provide a git-centric synchronization loop with the right hook and a git CLI. Adding tar , such a simple and omnipresent core utility, to make the hooks able to run git-archive seems a far simpler and consistent solution.

[thockin]

For the sake of argument, you don't need a webserver, just git, tar, gzip, and something which can loop and periodically examine the filesystem - a shell or a simple Go program would suffice. Any time the git-sync link changes (shared volume), run git archive.

Anyway, as mentioned I am open to PRs here, if they come with a great e2e test case. I don't have a lot of time to do it myself in the very near future.