Add github actions workflow (with caching)

[Peter0x44]

This is a continuation of #72, the changes I added were to make caching work, so the CI doesn't have to download tarballs on every run.

I am not sure if the changes to the Dockerfile are necessary. I initially made them because exporting the image to layers or something like that was hanging, and it fixed that. Docker is not something I have a lot of knowledge about. My changes were LLM authored, but I did test them as thoroughly as I could.

They are not. Removed.

[Peter0x44]

The changes to the Dockerfile seem to be required. I removed them in https://github.com/Peter0x44/w64devkit/actions/runs/18278436300/job/52035735401 and it hung.

[Peter0x44]

I managed to remove the changes to the Dockerfile, it still works.

[Peter0x44]

OK. There are still a few issues with this.

for PRs, it runs 2x the builds it needs to.

this might just be an issue with docker in general, but I had to update one of gcc's patches, but it did not invalidate the cache for x-gcc and continued using the previously cached build.

[Peter0x44]

I fixed the duplicate runs on pull requests. Should be good to merge @skeeto .

[Peter0x44]

No. I can't get that working.
But at least it cancels the jobs quickly after they start.

[Peter0x44]

@skeeto any objections left? I would love to get this in, I'm using it heavily.
It's being used in all of my experimental/research branches.

[Peter0x44]

Pinging this again...

[skeeto]

Sorry, I've been putting this off because CI/CD is unpleasant for me, and it's difficult to find motivation, especially when the results are unimportant to me. I barely tolerate it even when I'm being paid to deal with it. Embedding a programming language within YAML syntax, and the resulting syntactical nesting, has got to be one of the worst ideas in computer science to gain widespread adoption.

The rename step seems superfluous, but I see that you deliberately added it in order to cache. It seems the rename still isn't actually necessary, and a "breather" step between "build" and 'run" is what really matters? I'm just trying to understand that part.

I looked at your Actions result to see what this all looks like in the end, and I noticed:

JSON arguments recommended for ENTRYPOINT/CMD

While this issue with signals doesn't matter to our use case, the warning is annoying and it would be nice not to have it. I'm fine with dropping $PREFIX from this last step and just hard-coding a name so we can use JSON syntax. Should we do that?

--- a/Dockerfile
+++ b/Dockerfile
@@ -525,4 +525,3 @@ RUN printf "id ICON \"$PREFIX/src/w64devkit.ico\"" >w64devkit.rc \
  && echo $VERSION >$PREFIX/VERSION.txt \
- && 7z a -mx=9 -mtm=- $PREFIX.7z $PREFIX
-ENV PREFIX=${PREFIX}
-CMD cat /7z/7z.sfx $PREFIX.7z
+ && 7z a -mx=9 -mtm=- w64devkit.7z $PREFIX
+CMD ["cat", "/7z/7z.sfx", "w64devkit.7z"]

Dropping that ENV removes a layer, too, as a bonus.

Just as a result of making that branch I've got my first Actions result:

https://github.com/skeeto/w64devkit/actions/runs/19975315723/job/57289946957

I haven't set "Read and write permissions" yet, so I expect this to fail. It's still going as I write this. How have you confirmed that the cache works?

The automatic build and insert into a draft on tags seems like it could be convenient, though it depends on how long that typically takes. I can build locally from scratch pretty quickly, and I test my builds before I even push the tag, so generally the delay between pushing the tag and creating the formal release with artifacts is the time it takes to upload the builds plus source.tar. If the build comes after the tag, I have to wait, then download and test, and then if something's wrong with the build I've already pushed bad tag. It kind of flips things upside down. I don't think this is solvable, just an inherent limitation.

Is the primary convenience for you having the .github/ workflow already checked in, instead of you clumsily adding it on top of each of your own changes?

---

What follows is a failed attempt to make this easier on myself, and it's not so important to the PR.

It's probably a personal shortcoming, but my frustration increases quadratically on the time between action and feedback, and with CI/CD that means committing, pushing, then waiting minutes on even the slightest change. (One reason I like C so much: It's one of the precious few programming ecosystems where we measure the time between action and result in milliseconds!) I soon found nektos/act, which is a neat little utility for running GitHub Actions locally, and would ideally minimize that time. I rebased and squashed your changes to 6ea4c17, then tried it on Windows with:

$ act --artifact-server-path ~/Downloads

So far most of the time Docker Desktop crashes on the final export step, causing act to hang indefinitely, and I have to restart Docker Desktop manually, after which almost no caching occurs and act starts over from scratch. It also runs the architecture builds in parallel, so it ends up downloading everything twice, in parallel. (I deliberately organized the Dockerfile so that both architectures can share a download cache, and this isn't taking advantage of it.) One run eventually succeeded and I got these files:

$ du -sh 1/*/*
0	    1/skeeto~w64devkit~J4H7Z8.dockerbuild/skeeto~w64devkit~J4H7Z8.dockerbuild.zip
0	    1/skeeto~w64devkit~WGY9D3.dockerbuild/skeeto~w64devkit~WGY9D3.dockerbuild.zip
16.7M	1/w64devkit-x64-2.5.0-2-g6ea4c17.exe/w64devkit-x64-2.5.0-2-g6ea4c17.exe.zip
15.3M	1/w64devkit-x86-2.5.0-2-g6ea4c17.exe/w64devkit-x86-2.5.0-2-g6ea4c17.exe.zip

Obviously they're a little under half the size they're supposed to be. I see a PK header, a DOS stub (meaning w64dk is in there, uncompressed), and the end of the file has a PK table, but otherwise the zip is corrupt. busybox-w32 unzip errors with a "short read". It's not simply truncated, as I initially suspect, but half the bytes in the middle have gone missing. I suspect some PowerShell-related bug is processing it as text and mangling it. Looking online I see Windows isn't supported by act, and so it's not supposed to work anyway.

Next I try building in my mac, and I immediately get warnings about running on Apple Silicon, and it seems it's not well-supported either. It successfully builds and "uploads" an x64 package, but the x86 package fails (maybe due to a Git error: "failed to identify reference (tag/branch) for the checked-out revision" but there are lots of these?). The x64 package seems to work fine, so a small success. But the cache does not work, and every build is done from scratch. Perhaps caching isn't solvable with act.

[Peter0x44]

First of all, thanks for getting around to it. I understand it isn't a fun subject and I pretty much agree with your assessment. Yaml is truly horrific. Related: https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell

But regarding CI itself, I think it's extremely valuable and would do it for any library/executable I make. There is no easier way to get testing on a variety of operating systems and environments. The pain is nearly always worth it, IMO.

Is the primary convenience for you having the .github/ workflow already checked in, instead of you clumsily adding it on top of each of your own changes?

Yeah, I'm getting tired of lugging that commit around to all of my experiment branches and having to rebase it constantly.
Maybe it's a selfish reason.

How have you confirmed that the cache works?

painfully. I just push a commit changing something after the toolchain, and observe that it doesn't take as long (~5 minutes), and the log doesn't do as much.

I don't have any evidence to show other than "trust me bro". Unfortunately.

The rename step seems superfluous, but I see that you deliberately added it in order to cache. It seems the rename still isn't actually necessary, and a "breather" step between "build" and 'run" is what really matters? I'm just trying to understand that part.

I don't remember that being the motivation. I just like having the commit hash in the file name so it's easy to track which commit a specific w64devkit.exe was built from.

While this issue with signals doesn't matter to our use case, the warning is annoying and it would be nice not to have it. I'm fine with dropping $PREFIX from this last step and just hard-coding a name so we can use JSON syntax. Should we do that?

probably. I don't care about this warning personally, but if it can be fixed without consequence, we should do it.

[Peter0x44]

While this issue with signals doesn't matter to our use case, the warning is annoying and it would be nice not to have it. I'm fine with dropping $PREFIX from this last step and just hard-coding a name so we can use JSON syntax. Should we do that?

I added this change.

[Peter0x44]

To prove that the caching works, I added a commit simply changing something in the README.md as you suggested:

For example, changing a word in README.md should only take a couple of minutes to rebuild. Just the time to recompress the 7z archive with the new README.md, because that file is copied in at the last step. Requiring a full rebuild is locust behavior.

That run took only 5 minutes, the majority of which was spent downloading the cached layers.

Is there anything more I need to do to convince you that it works?

Full build:

rebuild:

[Peter0x44]

Pinging this again.