Automate MacOS Code-Signing in CICD

[CMCDragonkai]

Specification

During the integration stage, the final macos executables must be code-signed and notarised in order for them to be a drop-in runnable on end-user systems. This is the case even for applications distributed outside of the app store.

I have succesfully done this manually inside the matrix-mac-1. However this should be done by our CICD during integration:macos job before we do a prerelease or release.

The exact process of doing this may change if we were to integrate Nix: MatrixAI/TypeScript-Demo-Lib-Native#1. In this case, it would be better to have the Nix produce the final output, and have the code signing and notarisation part of the Nix process. However notarisation involves contacting the Apple mothership, and this may mean we require an "impure" build shell which basically means disabling the sandbox.

Additional context

• WIP: Demonstrating Native Addons TypeScript-Demo-Lib#38 (comment) - initial manual work doing code-signing and notarisation
• WIP: Demonstrating Native Addons TypeScript-Demo-Lib#38 (comment) - additional resources on how to automate this into our CICD
  • macos runners have passwordless sudo, however the keychain password is entirely different, we don't know what the main keychain password is, however it appears we can use a temporary keychain and not require to know the login keychain password
• We will need to inject our Developer ID Application key and certificate into our CICD system, this must be a protected variable
• We will continue to use the zip format, but if the pkg format allows to staple the notarization onto the executable, then it should be investigated further, because we could also release the pkg format, while having the bare executable around

Tasks

1. ...
2. ...
3. ...

[CMCDragonkai]

This issue is in TypeScript-Demo-Lib because it affects anything that is macos application.

[CMCDragonkai]

I think we should be using fastlane. But we will see more about this when we create a mobile application react-native.

[CMCDragonkai]

At this point in time we are using the Developer ID Application certificate for distributing the CLI executable outside the Mac app store.

If we want to package it as a pkg file, then you would also need the Developer ID Installer which is used to sign just the installer package.

Now if you want to go into the Mac app store, then we have another 2 kinds of certificates. In the new xcode 11 we can create one that works for all:

• Apple Development - works for macos, ios, tvos, watchos
• Apple Distribution - for both app store AND ad-hoc distribution

The ad-hoc distribution is used for test device deployment (outside of the mac app store). Basically distributing to beta testers.

For mobile development, the ios apps are .ipa files "iOS App Store Package".

The ad hoc method allows you to distribute your app to Test Flight and other sources and enables "designated" devices to run the app, including devices that aren't specifically registered to your Apple Developer Program account to run the app. (think public TestFlight testing)

Test flight is the system provided by Apple, it allows other users of iphone to download the app easily and test it. It's better than distributing to devices designated in the developer account.

So it appears the purpose of adhoc provisioning is to send to beta testers which are all orchestrated and registered under testflight. I will need third party expertise to setup the testflight account details. Testers use the testflight app to install these beta-builds and can provide feedback.

It appears fastlane is just a "CLI tool" that therefore can also "upload" to testflight. To setup testflight, you also need an "App Store Connect" account. This is part of the Apple developer account.

Anyway:

• Fastlane is therefore used at the CI/CD stage to interact with apple mothership, but can also automate things like codesigning
• Apple mothership includes things like the notarisation process and stapling process (necessary for macos developer id application), but also app store connect and therefore testflight integration

[CMCDragonkai]

So we can use fastlane in our Gitlab CI/CD, provide the necessary secrets in our CI/CD which will then do code signing, notarisation, and stapling (requiring .pkg) for our macos application. And this would be distributed outside the mac app store simply because this is a demo app, but also because it is a CLI-application. The app store only supports GUI applications.

For the GUI application which would make sense to distribute on the app store, one would then instead use the Apple Development and Apple Distribution certificate.

This makes me wonder. What the is purpose of Apple Development certificates? They are not part of ad-hoc distribution, perhaps they are part of some sort of "deployment" profile that allows them to execute on devices registered as part of the apple developer account. I can see that's how they would work.

So for a GUI app destined for the app store, one may use:

• Apple Development - for local testing
• Developer ID Application - for non-mac app store distribution
• Apple Distribution - for both ad hoc distribution (beta testing) AND app store submission

The usage of Apple Development certificate might be useless when doing integration testing on CI/CD, because we don't actually "control" the device. Unless the CI/CD server instance were to disable the gatekeeper and thus could run GUI apps without any constraints.

Is it possible to codesign the same app with all 3 certificates?

More details here: https://docs.fastlane.tools/best-practices/continuous-integration/

[CMCDragonkai]

I think the consensus is that you're supposed to use Apple Development certificates for day to day development, assuming you're on an apple machine that needs it. Will it run on the CI/CD system? Probably... not sure, atm the builds are running on the Gitlab macos instances without any codesigning, but they are still using the intel macs, not the new arm macs.

Once you finish development, you make a choice:

• Use Developer ID Application and/or Developer ID Installer when you want to distribute a CLI app, or just outside the app store
• Use Apple Distribution certificate if you want to distribute into testflight for beta testing via ad-hoc AND for app store submission

Perhaps both could be done, so you could also download the app from outside the app store and run it too. This seems to only apply to MacOS though, there's no official way of doing side-loading on ios.

[CMCDragonkai]

I believe the Apple Distribution certificate is needed to code sign for both ad-hoc distribution and testflight. Testflight is actually different from ad-hoc distribution, in that there's a static analysis and a review process. Whereas ad-hoc distribution has no review process.

This means you can use the development certificate to run on our local device, or local devices (possibly within the development team). And then use distribution certificate for ad-hoc distribution to a closer group of beta testers, testflight for more larger scale beta testers and finally app store. The last 3 stages are all using the Apple Distribution certificate.

In xcode, they don't just refer to certificates, they refer to "profiles", which probably select the certificates along with some other settings. https://help.apple.com/xcode/mac/current/#/dev31de635e5

[CMCDragonkai]

Android builds don't require code signing, the apk can be downloaded from the internet and executed.

However once they go into the google play store, they will require code signing and packaging into an .aab "Android Application Bundle". This process also requires a Google Play Developer Account.

I think the beta equivalent for Testflight on Google Play is just Google Play beta, it's not really a service, you're just pushing up builds with the ability for users to opt in and download beta versions of the application.

[CMCDragonkai]

More details here:

• https://gregoryszorc.com/blog/2022/08/08/achieving-a-completely-open-source-implementation-of-apple-code-signing-and-notarization/
• https://news.ycombinator.com/item?id=32386762

Seems like it has good information for automating this on Linux instead of automating all the MacOS steps.

[CMCDragonkai]

This will now be in Polykey-CLI, and done there instead.

[linear]

ENG-388 Automate MacOS Code-Signing in CICD

[CMCDragonkai]

The TypeScript-Demo-Lib isn't really relevant anymore. We don't really use it as a guide for any repos.