[Feature] MicroVM support

[ahmedtd]

For Alpha, we need to support running actors using a MicroVM hypervisor, preferably without regressing on gVisor support.

Open Questions:

• Should we dispatch to Kata, or directly interface with Cloud Hypervisor? Kata apparently does not currently implement the checkpoint/restore parts of the containerd shim API.
• Should we revive NoVM, if we think we will need to be patching the hypervisor?
• MicroVM checkpoints will be larger, since the hypervisor doesn't have a granular understanding of which pages are in use.
  • One alternative is to launch MicroVMs without checkpoint/restore. The actor always boots from scratch, and we just restore a data directory (or mount it via NFS or similar protocol).

[dims]

Taahir Ahmed (@ahmedtd) are we going to have say a ateom-firecracker similar to ateom-gvisor? (one for every micro vm we support?)

Can we come up with a check list of here are things we need from this microvm for us to have BEFORE we consider supporting it?

[EItanya]

Taahir Ahmed (@ahmedtd) are we going to have say a ateom-firecracker similar to ateom-gvisor? (one for every micro vm we support?)

Can we come up with a check list of here are things we need from this microvm for us to have BEFORE we consider supporting it?

This is a great question

I think generally having different ateom-* means that the actual ateom impl can be pluggable, but as you say that also means we need to come up with a clean API and checklist of features which it needs to implement. The first thing that comes to mind is snapshotting, firecracker can do it, but I don't know off the top of my head which others can. Or do we do the dreaded gvisor in microvm 😆

The pluggability may also become complicated with things like #126 and other networking related features

[ahmedtd]

Taahir Ahmed (@ahmedtd) are we going to have say a ateom-firecracker similar to ateom-gvisor? (one for every micro vm we support?)

I think this is yet undecided. If we have multiple types of ateoms, then we will need to do capacity management between them. If we have one ateom that understands how to handle all backends, then capacity management is much simpler. But probably, no matter what, people are going to want the ateom interface to be pluggable.

One other dimension that's still undecided is how much communication goes through atelet, vs how much is direct from ate-api-server to ateoms.

Like, a very clean ateom interface might just be to have the current atelet Run/Checkpoint/Restore workload RPCs surfaced on ateom. And ateom is responsible for everything, including image pulling (probably still sharing the cache in the a shared host folder).

[EItanya]

MicroVM checkpoints will be larger, since the hypervisor doesn't have a granular understanding of which pages are in use.
One alternative is to launch MicroVMs without checkpoint/restore. The actor always boots from scratch, and we just restore a data directory (or mount it via NFS or similar protocol).

This is a very interesting idea, and especially prudent for agents which store things in the fileystem anyway.

Should we dispatch to Kata, or directly interface with Cloud Hypervisor? Kata apparently does not currently implement the checkpoint/restore parts of the containerd shim API.

This actually brings up a core question for me. Do we want the core "workload" semantic here to be containers. The current actor template re-uses the k8s-api Container shape, but is that out of expediency, or because containers are the end-state of these workloads? Happy to try and elaborate further if that doesn't quite make sense

[ahmedtd]

In one of my original proof-of-concepts with Cloud Hypervisor, I just punted on container support and had the user specify a qcow disk image that supported cloud-init as their application.

I think we probably want the container to be how we specify the application, just so we're not breaking totally new ground. Individual backends can then repack the container image into a different format (qcow or whatever) if they need.

Using the K8s shape is mostly just because it's likely well thought through.

[EItanya]

Makes sense to me. I'm more familiar with the firecracker side of the house, but from what I've seen it's mostly the same. The main question that I've had when thinking about this is multiple containers and what the boundary is between them within the VM, if there is any. However, maybe that's up to the implementation here.