feat: install Kubernetes to Flatcar from sysexts on MAR #7680

chewi · 2026-01-19T11:01:13Z

What this PR does / why we need it:

Following #7125, this installs kubelet, kubectl, and acr-credential-provider to Flatcar from systemd system extensions hosted on MAR.

Flatcar is immutable so cannot use traditional packages. The sysexts are downloaded using ORAS at build time and the desired version is later "merged" by simply creating symlinks and calling systemd-sysext refresh.

If a sysext for the requested Kubernetes version cannot be found locally at deployment time, ORAS is used to try and fetch one. Failing that, it falls back to downloading the regular binary tarballs.

Sysexts are a systemd concept, so they have been published using systemd's architecture names rather than Docker's even though they are hosted in an OCI registry.

The sysexts are built using Dalec from a distribution other than Flatcar, which is currently Azure Linux 3.

Renovate has been configured to consider azlinux3 part of the version so that different Kubernetes releases could potentially be built from different distributions. The use of matchCurrentVersion ensures that tags containing azlinux3 (e.g. not noble) will only be applied against artifacts already using azlinux3. Switching distribution should be a manual change.

This also adjusts the "aks-sysext" hack currently applied to Flatcar that allows writing to /usr/local/bin. Despite efforts to use /opt/bin instead, it is likely that waagent will still need to write here for the time being. aks-sysext now allows writes to all of /usr/local rather than just /usr/local/bin. The underlying files are now written under /mnt rather than /etc because Flatcar is going to start using confexts, making /etc an overlayfs, which would cause issues here.

The rest is just renames, refactoring, and fixes to support the above.

Which issue(s) this PR fixes:

Fixes Flatcar for AKS work item #34435556

chewi · 2026-01-21T14:38:01Z

I am seeing a few persistent CI failures relating to sudo, which I haven't touched, at least not in this PR.

+ sudo sshd -t -f /tmp/tmp.0M7Kkzjjsv
sudo: Account or password is expired, reset your password and try again
Changing password for root.
sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: unable to change expired password: Authentication token manipulation error
sudo: a password is required

vhdbuilder/packer/test/linux-vhd-content-test.sh

djsly · 2026-01-21T16:34:04Z

.github/renovate.json

+      "matchDatasources": [
+        "docker"
+      ],
+      "matchPackageNames": [


who would be the owner of such packages ? I'm not sure if they would have automatic assigned in the PR

probably Chewi + Jeremi as primary owners, and maybe us as secondary?

I'm not sure that we need to be owners. These sysexts are published in tandem with their .deb and .rpm counterparts from the very same Dalec YAML definition. Adding sysext support is usually just a case of adding one line.

djsly · 2026-01-21T16:36:00Z

parts/linux/cloud-init/artifacts/flatcar/cse_install_flatcar.sh

+        shift 3
+    done
+
+    systemd-sysext --no-reload refresh


where can I learn more on this, those sysext droppin how do they work ?

also curious

The sysext man page is a good place to start: https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
Also, Lennart wrote a few blog posts around sysext use but these tend to be pretty specific. Here's one on creating extension overlays on the fly: https://0pointer.net/blog/testing-my-system-code-in-usr-without-modifying-usr.html (using subdirectories, not actual images like we do).

@djsly @cameronmeissner I'm also happy to jump on a call and talk you through the general concept if we get a few folks interested. I held a number of presentations on sysexts (the most recent one at FrOSCon 2025: https://app.media.ccc.de/v/froscon2025-3330-immutable_turtles_all_the_way_down ) and would be happy to ramp you up!

parts/linux/cloud-init/artifacts/flatcar/cse_install_flatcar.sh

djsly

left a few comments :)

jepio · 2026-01-21T16:40:50Z

vhdbuilder/packer/flatcar-customdata.yaml

+  - path: /mnt/aks-sysext-rw/usr/local
+  - path: /mnt/aks-sysext-rw/usr/local/bin
+  - path: /mnt/aks-sysext-rw/usr/local/lib
+  - path: /mnt/aks-sysext-rw/usr/local/lib64
+  - path: /mnt/aks-sysext-rw/usr/local/sbin
+  - path: /mnt/aks-sysext-rw/usr/local/share


why are we making all these paths mutable/read-write? the only one we discussed potentially needing is /usr/share/ca-certificates i think, but not any of these /usr/local.

Ah, I was thinking it was /usr/local/share/ca-certificates. I thought I'd cover the rest of /usr/local just in case it helps with other extensions. I can extend this to /usr/share/ca-certificates. Up to you whether we cover the whole of /usr/local, but I think it doesn't hurt.

jepio · 2026-01-21T16:42:03Z

vhdbuilder/packer/flatcar-customdata.yaml

  files:
-  - path: /etc/extensions/aks-sysext/usr/lib/extension-release.d/extension-release.aks-sysext
-    mode: 0644
+  - path: /mnt/aks-sysext/usr/lib/extension-release.d/extension-release.aks-sysext


It feels like we should keeps this together with the other AKS bits in /opt, rather than putting it in /mnt. That shouldn't run into the overlay stacking issues because we guarantee we won't be applying a sysext to /opt, as doing so would make it read-only.

jepio · 2026-01-21T16:44:23Z

vhdbuilder/packer/flatcar-customdata.yaml

+  - path: /etc/extensions/aks-sysext
+    target: /mnt/aks-sysext
+    hard: false
+  - path: /mnt/aks-sysext/usr/local
+    target: /mnt/aks-sysext-rw/usr/local
+    hard: false


what is the thinking behind this double indirection via symlinks? why /etc/extension/aks-sysext -> /mnt/aks-sysext -> /mnt/aks-sysext-rw rather than one level of symlink?

It has to be a symlink rather than a real directory under /etc once confext comes into play. If it was /etc/extension/aks-sysext -> /mnt/aks-sysext with /mnt/aks-sysext/usr/local being a real directory, then it wouldn't be writeable.

parts/linux/cloud-init/artifacts/flatcar/cse_install_flatcar.sh

cameronmeissner · 2026-01-21T16:50:03Z

parts/linux/cloud-init/artifacts/flatcar/cse_install_flatcar.sh

+        local seName=$1 seURL=$2 desiredVer=$3 seMatch
+
+        seMatch=$(matchLocalSysext "${seName}" "${desiredVer}" "${seArch}")
+        if ! test -f "${seMatch}"; then


why using test directly? if [ ! -f "${seMatch}" ]; then would be preferred

I wrote this months ago and can't remember. It possibly involved Copilot. 😅

Ah, now I remember! It was so that I could mock the call in cse_install_flatcar_spec.sh. If you mock [, but then you can't use it inside the mock. Doing so possibly also had other explosive effects.

cameronmeissner · 2026-01-21T16:50:26Z

parts/linux/cloud-init/artifacts/flatcar/cse_install_flatcar.sh

+            fi
+
+            seMatch=$(matchLocalSysext "${seName}" "${desiredVer}" "${seArch}")
+            if ! test -f "${seMatch}"; then


same question here

cameronmeissner · 2026-01-21T16:51:39Z

parts/linux/cloud-init/artifacts/flatcar/cse_install_flatcar.sh

    fi
 }

+installCredentialProviderPackageFromBootstrapProfileRegistry() {


do we really need this helper?

Not having it would make the confusing logic in ensureKubelet even more confusing.

installStandaloneContainerd should never be called in this case.

We almost always pass `${OS}` so make this the default.

There are only really four outcomes here, install kubelet and kubectl from a URL, bootstrap profile registry, or package, or do nothing.

There are only really three outcomes here, install the credential provider from a URL, bootstrap profile registry, or package.

Flatcar will use the same logic but install these from MAR instead, so make the name a bit more generic.

Flatcar is immutable so cannot use traditional packages. The sysexts are downloaded using ORAS at build time and the desired version is later "merged" by simply creating symlinks and calling systemd-sysext refresh. If a sysext for the requested Kubernetes version cannot be found locally at deployment time, ORAS is used to try and fetch one. Failing that, it falls back to downloading the regular binary tarballs. Sysexts are a systemd concept, so they have been published using systemd's architecture names rather than Docker's even though they are hosted in an OCI registry. The sysexts are built using Dalec from a distribution other than Flatcar, which is currently Azure Linux 3. Renovate has been configured to consider `azlinux3` part of the version so that different Kubernetes releases could potentially be built from different distributions. The use of `matchCurrentVersion` ensures that tags containing `azlinux3` (e.g. not `noble`) will only be applied against artifacts already using `azlinux3`. Switching distribution should be a manual change.

See previous commit for details.

Functions like installKubeletKubectlFromPkg exist in multiple distro scripts, so one script will shadow another. Only load the applicable scripts when required by grouping the tests for each distro. Defining BeforeEach as a function is not the correct way to use it, so such usage was doing nothing. You need to give it code as a string. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

This was mostly written by Copilot, but it did a good job.

Hosting a directory-based sysext from /etc is a problem because Flatcar will use confext soon, which will make /etc an overlayfs. It is now hosted from /mnt/aks-sysext instead. Pointing the /usr/local/bin symlink at /opt/bin is also confusing because AKS now installs several binaries here. It just needs to point somewhere writeable, so this now points under /mnt/aks-sysext-rw. The symlink now covers the whole of /usr/local, not just /usr/local/bin, as there may be components that need to write to other directories. Nothing in Agent Baker should write to /usr/local now, but VM extensions may still do this. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

Flatcar is breaking the OSProfile limit again, so savings need to be made somewhere. This helper function is only used in that one file. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

It's a legacy file with .deb package data, and Flatcar needs to save space in OSProfile. Also the `MANIFEST_FILEPATH` references in the tests were unused. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

chewi self-assigned this Jan 19, 2026

chewi had a problem deploying to test January 19, 2026 11:01 — with GitHub Actions Failure

github-actions bot added the components This pull request updates cached components on Linux or Windows VHDs label Jan 19, 2026

chewi changed the title ~~Install K8s from systemd system extensions on Flatcar~~ feat: install Kubernetes to Flatcar from sysexts on MAR Jan 19, 2026

chewi force-pushed the chewi/aks-flatcar branch from 0f01d7a to bb63179 Compare January 19, 2026 17:29

chewi had a problem deploying to test January 19, 2026 17:29 — with GitHub Actions Failure

chewi force-pushed the chewi/aks-flatcar branch from bb63179 to 1edf82e Compare January 20, 2026 21:40

chewi temporarily deployed to test January 20, 2026 21:40 — with GitHub Actions Inactive

chewi force-pushed the chewi/aks-flatcar branch from 1edf82e to 9fafc40 Compare January 21, 2026 11:23

chewi temporarily deployed to test January 21, 2026 11:23 — with GitHub Actions Inactive

chewi force-pushed the chewi/aks-flatcar branch from 9fafc40 to e77233d Compare January 21, 2026 11:40

chewi temporarily deployed to test January 21, 2026 11:40 — with GitHub Actions Inactive

chewi marked this pull request as ready for review January 21, 2026 14:38

chewi requested review from AbelHu, Devinwong, YaoC, awesomenix, cameronmeissner, djsly, ganeshkumarashok, juan-lee, junjiezhang1997, lilypan26, phealy, r2k1, timmy-wright, yewmsft and zachary-bailey as code owners January 21, 2026 14:38

djsly reviewed Jan 21, 2026

View reviewed changes

vhdbuilder/packer/test/linux-vhd-content-test.sh Show resolved Hide resolved

djsly reviewed Jan 21, 2026

View reviewed changes

parts/linux/cloud-init/artifacts/flatcar/cse_install_flatcar.sh Show resolved Hide resolved

djsly reviewed Jan 21, 2026

View reviewed changes

jepio reviewed Jan 21, 2026

View reviewed changes

cameronmeissner reviewed Jan 21, 2026

View reviewed changes

parts/linux/cloud-init/artifacts/flatcar/cse_install_flatcar.sh Outdated Show resolved Hide resolved

cameronmeissner reviewed Jan 21, 2026

View reviewed changes

chewi force-pushed the chewi/aks-flatcar branch from e77233d to 4bb4239 Compare January 22, 2026 16:33

chewi temporarily deployed to test January 22, 2026 16:33 — with GitHub Actions Inactive

chewi added 15 commits January 23, 2026 16:43

fix: explicitly exclude containerd from Flatcar in components.json

0306d0c

installStandaloneContainerd should never be called in this case.

fix: testPackagesInstalled logic for Flatcar and Azure Linux OS Guard

3642583

style: make "isOS" shell helpers less cumbersome to use and add isUbuntu

2fddd42

We almost always pass `${OS}` so make this the default.

refactor: simplify configureKubeletAndKubectl logic

fbe78db

There are only really four outcomes here, install kubelet and kubectl from a URL, bootstrap profile registry, or package, or do nothing.

refactor: simplify credential provider installation logic

2c6b56f

There are only really three outcomes here, install the credential provider from a URL, bootstrap profile registry, or package.

style: rename installKubeletKubectlPkgFromPMC to *FromPkg

904153b

Flatcar will use the same logic but install these from MAR instead, so make the name a bit more generic.

rename: installCredentialProviderFromPMC to *FromPkg

c91d847

Flatcar will use the same logic but install these from MAR instead, so make the name a bit more generic.

feat: install credential provider to Flatcar from sysext on MAR

d82f62b

See previous commit for details.

test: add shellspec tests for new Flatcar and sysext code

0f704dd

This was mostly written by Copilot, but it did a good job.

fix: move apt_get_download to Ubuntu-specific script file

e1454ee

Flatcar is breaking the OSProfile limit again, so savings need to be made somewhere. This helper function is only used in that one file. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

fix: only provision Ubuntu with /opt/azure/manifest.json

44b6428

It's a legacy file with .deb package data, and Flatcar needs to save space in OSProfile. Also the `MANIFEST_FILEPATH` references in the tests were unused. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

chore: update test data

b96f8d1

chewi force-pushed the chewi/aks-flatcar branch from 4bb4239 to b96f8d1 Compare January 23, 2026 16:44

chewi temporarily deployed to test January 23, 2026 16:44 — with GitHub Actions Inactive

feat: install Kubernetes to Flatcar from sysexts on MAR #7680

Are you sure you want to change the base?

feat: install Kubernetes to Flatcar from sysexts on MAR #7680

Uh oh!

Conversation

chewi commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chewi commented Jan 21, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cameronmeissner Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

t-lo Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

djsly left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jepio Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

chewi commented Jan 19, 2026 •

edited

Loading

cameronmeissner Jan 21, 2026 •

edited

Loading

t-lo Jan 22, 2026 •

edited

Loading

jepio Jan 21, 2026 •

edited

Loading