IPv6 private addressing strategy for project networking

[scotwells]

Every project on the platform needs a unique private IPv6 address space so that networks across projects can communicate without address conflicts — for example, when connected via private peering or inter-project networking.

When a user enables networking services in a project, the platform automatically allocates a unique private (ULA) /48 prefix from a platform-managed pool. All networks, subnets, and individual instance addresses within that project are drawn from this prefix.

Allocation hierarchy

Resource	Block size
Per-project root	/48
Per-subnet (network + location)	/64
Per-instance address	/128

Each project supports up to 65,536 subnets — no artificial cap on how many networks or locations a project can have. The association between a subnet and its network and location is tracked by the platform, not encoded in the address itself.

Policy-mode networks

When a user creates a network with a manually specified address range (policy mode), multiple networks within the same project may intentionally use the same private address space — for example, two isolated test networks both using the same range. This gives users flexibility to create isolated networks with intentionally overlapping address spaces for advanced use cases such as multi-tenant environments or staged migrations. The platform should support this without treating it as a conflict. This requires a capability enhancement to the IPAM service (milo-os/ipam#23).

Implementation note

The per-project /48 is allocated by the network services operator when a project's networking ServiceEntitlement becomes active. The full technical design is tracked in enhancements/ipam-integration.md.

Public IPv6 addressing (globally routable GUA prefixes) is out of scope for this issue and will be tracked separately.

[scotwells]

@privateip / @slindseysr could use your eyes on how we're looking at doing private IPv6 address allocation within a project. Above was based on some research I did with claude.

[slindseysr]

In general I think /48 per project is too large. We should also talk through if/how the prefixes will be exposed to the underlay and externally as there are management and scaling factors we must be mindful.

[scotwells]

@slindseysr wouldn't this be in the overlay because it's private VPC network space?

[slindseysr]

Its not fully clear to me if these are fully contained in the overlay or if will get exposed to the underlay and ultimately externally. If fully contained in the overlay I would still be concerned that /48 per project is over allocating. We can always allocate more to projects that really need them. The only compelling reason for /48 I can come up with is that it is short enough to advertise to the public Internet routing table. Even if we need to expose them publicly we would still want to summarize public advertisement in a larger aggregate i.e. /44 or 40. I recommend we consider /56 (256x /64s) as a default per project and have option to make larger allocation by user request. The other thing to think about especially if we are going to break things down to small subnets or host routes i.e. /128 is which routing tables need to carry them. If these small bits are only user visible inside the overlay then less of a concern, but if the overlay RRs or worse the underlay RRs need to carry them we should find a way to allocate in ways that we can summarize and protect route scale resources.

[slindseysr]

We had a sidebar meet and discussed this. The per project /48 allocation will be from ULA space fd00::/8 and never exposed outside of the overlay which nulls most concerns stated above. The only future concern I have is how they are assigned and the scalability of managing the routes. They should be allocated in such a way that they could be summarized from sites, regions, etc in the future if ever needed.

[privateip]

Per our discussions today, the plan is to allocate three IPv6 /48 subnets to each PoP as follows:

• One /48 to be used for private VPC addressing to come from the IPv6 ULA address space (fc00::/8)
• One /48 to be used for SID Location out of a registered /33 block
• One /48 to draw loopback addresses from. From that /48, it will be split into two /49s. One /49 will allocate loopbacks for the underlay BGP router and one /49 will allocate loopbacks for the overlay BGP router

[scotwells]

@privateip think we're missing one detail from here on what we were going to carve out of the /48 in the location. If I'm remembering correctly, it was a /64 for each consumer VPC to allocate out of?

[privateip]

Yes thats right. For each VPC we will carve out a /64 that will come from the /48 ULA space. I'm working on writing up an addressing plan to add to our docs that has all of the details in it instead of embedding in comments here. I should have the PR ready in https://github.com/datum-cloud/enhancements early next week (Monday or Tuesday)-ish

[scotwells]

You read my mind. Thank you!