CI/CD to build and sanity check docker container and release executables

[CMCDragonkai]

The final built executables coming out of vercel/pkg are not tested when uploaded as pre-release executables.

We should run a final test on those, maybe a simplified test that runs just the --version or --help page to ensure that the executable actually runs on the target environment. This can help us know whether the final packaged executables are actually working mostly.

This requires our CI/CD to expand to include all environments:

• Linux
• Windows - https://about.gitlab.com/blog/2020/01/21/windows-shared-runner-beta/
• Mac - @joshuakarp knows the details about this in Secret Vault Schema #4

We have these stages:

  stages:
    - check
    - build
    - release

This might work as an extra stage: qa to mean "quality assurance". And it needs to use raw environments (not based on our Nix image that we use to build things).

Tasks

1. - Investigate adding an extra stage qa that enables Linux, Windows, and Mac environment runners, that means this stage runs on an entirely different runner compared to check, build and release.
2. - Download the executables from the release point. This tests whether our release point is up and running. If the release point is down, the test should retry after some time, as this depends on network availability.
3. - Run a sanity check over the executables such as --help and --version to get the help page and the version page respectively.
4. - Ensure that this job is interruptible.

---

Regarding docker:

1. - Create the docker image with nix-build ./release.nix -A docker
   • test that this container works with docker run -it name-version:hash.
   • example: docker run -it matrixai_typescript-demo-lib-1.0.0:f3wf1wxvkdbbmrhxx9rhr5qw2skbrah6
   • see TypeScript-Demo-Live README.md for more details
   • the TMPDIR may be used by PK, if so, use this https://gist.github.com/CMCDragonkai/94e6550302aa7ebc175c06819927605a
   • inspect the image contents for correctness by using this https://gist.github.com/CMCDragonkai/d7a652ada27289ab59e28bc914c942f6
   • do we need a cacert and tzdata?
     • cacert may be needed in case we want to contact external websites, but this is not a thing for the PK agents hosted on testnet.polykey.io
     • tzdata shouldn't be necessary since we don't operate on any timezones, we assume OS time is UTC time

[CMCDragonkai]

Remember Gitlab provides many CI/CD runners. But I suspect this will require running a separate runner on the last stage. I'm not sure if the configuration allows this. But we must investigate this.

Also consider what additional costs there is, and whether it is free for Open Source, and we have replicated this to the Open Source group.

[CMCDragonkai]

Should also do a hash check to ensure that the expected hashes of the released executables are in fact what we expect. That means during the release stage, we should be calculating the hashes from nix-build and then publishing those as well on the release page. Note that the hashes used in the naming currently are nix hashes which are hashes of the input derivation expression, not the build outputs.

[CMCDragonkai]

This maybe relevant too https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix3-store-make-content-addressable.html

Checking the hash of the release is just about ensuring integrity but it doesn't ensure authenticity. A signature would ensure authenticity. However combining the hash with an "authentic" trusted source is sufficient to ensure authenticity and integrity. Of course end-to-end authenticity is enhanced with a proper signature applied.

To dog-food Polykey, it would make sense to do a code-sign from the Polykey organisation gestalt rather than GPG since we stopped using GPG.

[CMCDragonkai]

The CI/CD check here should actually be done on the container image build. We should also do a check against the release executable builds while we're at it.

Note that the release.nix is what has all the specifications. The current docker attribute bundles up the cacert package. For the testnet and mainnet. I believe the cacert won't be required for NAT functionality as contacting external websites is only needed in the identities domain.

If however we are going to join all of the testnet nodes as part of the Matrix AI gestalt and mainnet could be in a different gestalt, it may in fact be required. We can keep the dependency in just for now.

[CMCDragonkai]

Useful gist notes about using docker and nix:

• https://gist.github.com/CMCDragonkai/94e6550302aa7ebc175c06819927605a
• https://gist.github.com/CMCDragonkai/d7a652ada27289ab59e28bc914c942f6
• https://gist.github.com/CMCDragonkai/f7ca9b3ce8a64aa21ab68cf69dfc9f04

[CMCDragonkai]

Pushing this Nov 11 finish from Nov 10 as an additional last sanity check should be done once #269 is done. But this can be started without #269.

[CMCDragonkai]

Make sure to check that bundled files JSON and protobuf files are being put into the ./dist directory properly. Except of course the ./package.json.

See: MatrixAI/TypeScript-Demo-Lib#34 (comment) for discussion and fix in TypeScript-Demo-Lib.

If there are future files that need to be copied over, it's also important to consider the npm postbuild hook to copy over those non-transpiled files. These stages of "building" should be documented.

[CMCDragonkai]

Checking the build executables can be done in a number of ways automatically.

Application Emulation

The idea here is to directly execute the built artifacts in the same CI/CD container environment which currently is a "nix" container running on Linux. The advantage here is probably less resources being used, and all of this could be tested on our local development machines which are NixOS (or linux with Nix).

Disadvantage is that it's not a full test, we don't know for sure what might happen on the other platforms.

• Linux - linux executables will be compiled for non-NixOS, this means we would need to be able to run the executables and simulate a non-NixOS environment: https://unix.stackexchange.com/questions/522822/different-methods-to-run-a-non-nixos-executable-on-nixos
• Windows - WINE is the proper of doing this, it can do application emulation and we should be able to run the pk CLI via wine
• MacOS - this is the toughest one, but there's a project called https://github.com/darlinghq/darling which aims to do WINE but for mac, we could try this and it might work
• iOS/Android - not sure but the CLI isn't intended for these locked down platforms, although android does have an emulation ecosystem

Example: https://jake-shadle.github.io/xwin/#2-specify-runner This guy uses WINE to test a cross-compiled Rust executable.

System Emulation/Virtualisation

This relies on virtualisation, such as using QEMU to run a full system. This is closer to the real deal, but more complex and expensive.

Atm, I don't believe the containers inside Gitlab CI/CD are allowed to execute QEMU so we don't automate it that way. Instead this will rely on the CI/CD system itself to provide these extra environments. Gitlab has MacOSX and Windows guests available to be used, and these can then run tests against our built executables. Note that this means the tests are not running inside the Nix container anymore. Which leads us to our next problem.

What kind of tests?

These tests that run against the final built executable are not like the tests we have currently inside our Jest testing. Jest itself is just a test runner, it uses Node executes JS at the end of the day. The JS itself right now is importing code directly from src and testing this. Even now our tests/bin tests don't actually do a child process execution. They are lighterweight by calling main function directly and mocking the STDOUT/STDERR... etc. And even if they were to actually do a proper process exec, it's again done against the src/ code, and not against the final built executables.

So for testing the final executables, the tests have to shell out out and execute the commands and then testing what is the expected side-effects. This can be done in jest by using child process exec, and all we are doing is using the JS to run child processes.

If we use a library like https://github.com/raingerber/jest-shell-matchers, this allows us to create expectations and matchers that can execute external executables.

However there are 2 challenges with this:

1. We may need to copy some or alot of the tests in tests/bin to instead test external executables. Some of the tests can work without any change since all we do is swap out the usage of main with executing an external executable.
2. Using this means all the platforms we're doing tests in will need to be able to run jest and node. That is the test utilities has to work in that environment.

If abandon jest entirely, then we would need a new test harness, and then platform-specificity is an issue. For example if we were to write tests in a shell language like bash, this may work easily on Linux and MacOS, but not as easily in Windows which doesn't have bash. We could create platform specific ways for every platform, Linux and Mac can use bash, while Windows uses powershell, but this just ends up duplicating our tests/bin 2-times.

We know that node and jest can work on Windows, Mac and Linux. But we won't be able to use Nix in most of these extra environments, since these are meant to be non-Nix platforms. So it's just a matter of installing npm and node on those platforms and proceeding from there. I'm sure the CI/CD scripts have some suggested way of installing Node on those platforms. Windows may have chocolatey, and Mac may have homebrew. See the listed methods here: https://nodejs.org/fa/download/package-manager/

Also Gitlab docs may have some suggested ways of doing things too.

Portability of tests/bin

So does this mean we have to write some new tests like tests/bin-external? Or could we re-use some of the tests in tests/bin?

Surely we can swap out execution of main with direct execution of the real thing right?

That may be the case, but if we are using jest-mock-process, then that won't work, since the real process is not using the Node process mocks.

However if all of the mocking used by https://github.com/EpicEric/jest-mock-process is done within a wrapper function like pkWithStdio, then it may be possible to swap that out with the real implementation in which case the side-effects are real, and STDOUT, STDERR and exit code can be captured as well. Then a parameter can be passed into tests/bin to run the same tests on the external executable.

I'm sure some tests may not work though, and this requires a careful review.