Replace actor eth0 move with worker-owned veth networking

[EItanya]

Today ateom-gvisor gives an active actor network connectivity by moving the worker pod's real Kubernetes eth0 into the actor/gVisor network namespace. That works for basic actor execution, but it has an important side effect: while the actor is running, the worker pod no longer owns its pod network interface.

This makes the worker pod partially disappear from the normal Kubernetes networking model during actor activation. It also prevents us from cleanly adding pod-local networking components, such as an egress proxy or policy enforcement sidecar, because those components remain in the worker pod network namespace while actor traffic leaves through an interface that has been moved elsewhere.

We should replace the eth0 move approach with an explicit veth pair between the worker pod namespace and the actor/gVisor namespace.

Rationale

Keeping the pod's real eth0 in the worker namespace gives us a stable networking boundary:

• The worker pod keeps normal Kubernetes network connectivity while an actor is running.
• Sidecars or worker-local helper processes can remain reachable and can participate in actor networking.
• Actor networking becomes explicit and owned by Substrate instead of depending on moving the CNI-provided interface.
• This creates the foundation for transparent egress capture and policy enforcement.
• It avoids coupling actor lifecycle to destructive mutation of the pod's primary network interface.

This is especially important for planned transparent egress policy work. We want actor traffic to be captured without requiring application opt-in via HTTP_PROXY or HTTPS_PROXY. To do that cleanly, actor traffic needs to cross a worker-owned interface where Substrate can install forwarding, NAT, and later proxy-capture rules.

Proposed approach

For each active actor, ateom-gvisor should create a point-to-point veth pair:

• Worker pod namespace side: ateom0, for example 10.200.0.1/30.
• Actor/gVisor namespace side: renamed to eth0, for example 10.200.0.2/30.
• Actor default route points to the worker-side veth IP.
• The Kubernetes-provided pod eth0 remains in the worker pod namespace.

For compatibility with the current routing model, the initial implementation can install temporary nftables rules:

• Masquerade actor egress behind the worker pod IP.
• DNAT inbound traffic from the worker pod IP and actor service port to the actor veth IP.
• Enable forwarding between the actor veth and pod eth0.

These rules are intended as a compatibility bridge, not the final egress policy implementation. The later egress work should replace broad actor egress NAT with transparent capture into AgentGateway and default-deny policy rules.

Expected outcome

After this change:

• Actors still start, checkpoint, restore, and receive inbound traffic as before.
• The worker pod retains network connectivity during actor execution.
• ateom-gvisor no longer needs to scrape, move, or restore the pod's real eth0.
• Substrate has a cleaner networking foundation for transparent egress capture, policy enforcement, and future worker-local AgentGateway integration.

Validation

The implementation should be validated with:

• Existing focused Go tests for ateom-gvisor and related networking/server boot packages.
• The demo e2e flow that exercises actor startup, golden snapshot creation, restore, and inbound routing.
• A runtime check that the worker pod still has its Kubernetes eth0 while an actor is active.
• A runtime check that actor traffic routes through the veth pair and inbound traffic still reaches the actor.

[ahmedtd]

I think this is in general a good direction to go. Moving the pod's eth0 was the simplest thing we could get done before our launch deadline, but the fact that ateom doesn't have any networking is quite restrictive.

Some questions / comments:

• What IP(s) should we pick for the actor's veth pair. Can linux handle it if they all have the same IP, as long as they are inside the independent pod network namespaces?
• "DNAT inbound traffic from the worker pod IP and actor service port to the actor veth IP" -- I am assuming this means we will need to require the ActorTemplate to declare any listening ports for the actor up front?
• One of the things I have discussed with Fabricio Voznika (@fvoznika) and nybidari is that gVisor is probably going to get the ability to send packets directly to Envoy as a TCP/UDP proxy, rather than injecting them into the host kernel networking stack with AF_PACKET. But we will probably always need this setup for microvm-based workloads.

[BenTheElder]

Can linux handle it if they all have the same IP, as long as they are inside the independent pod network namespaces?

Yes.

"DNAT inbound traffic from the worker pod IP and actor service port to the actor veth IP" -- I am assuming this means we will need to require the ActorTemplate to declare any listening ports for the actor up front?

If we keep all other aetom traffic on UDS we could DNAT all inbound traffic from the worker pod IP to the actor veth.

[nybidari]

Yes, we are investigating a method to communicate directly with Envoy over UDS with gvisor. This is still in the research phase.

[thockin]

The failure mode that I am worried about is this:

• ateom receives traffic for itself on port 12345
• actor decides to listen() on port 12345
• actor sees no error but does not actually receive any traffic on 12345

I am also a little worried about connection-tracking. It's a finite kernel resource which consumes precious RAM and needs to be configured.

[EItanya]

I agree that we cannot have arbitrary listening ports.

When I built something like this in the past I bypassed this by having the ateom equivalent bind and then proxy all traffic into the actor equivalent via various mechanisms. IIRC in the Firecracker case it was a TAP device but I would need to double check.

In this case the atelet could potentially have a list of reserved ports which it could dole out to the actors, and then the ateom could proxy traffic appropriately.

I'm not as familiar with gVisor, but because firecracker is a microVM, the application can bind to whatever port it wants, so it's all transparent to the application.

One thing I want to stress is that perfect should NOT be the enemy of good for this PR/issue, I'm glad we're thinking through the long-term networking design, but maybe that belongs in an EPIC so that we can make incremental progress.