Skip to content

Implement E2E encrypted multi-tenant network architecture (WireGuard/VXLAN/BGP EVPN/L3VPN)#5

Open
Copilot wants to merge 2 commits intomainfrom
copilot/comprehensive-e2e-encrypted-architecture
Open

Implement E2E encrypted multi-tenant network architecture (WireGuard/VXLAN/BGP EVPN/L3VPN)#5
Copilot wants to merge 2 commits intomainfrom
copilot/comprehensive-e2e-encrypted-architecture

Conversation

Copy link
Contributor

Copilot AI commented Mar 18, 2026

src/network.rs referenced boringtun, getrandom, and x25519_dalek — none declared in Cargo.toml — making the module uncompilable and excluded from the build. This PR fixes that, integrates the module, and implements the full multi-tenant networking stack against the VyOS HTTP API.

Dependencies

Added base64 = "0.21", rand = "0.8", x25519-dalek = { version = "2", features = ["static_secrets"] }.

src/network.rs — rewritten

Dropped the native boringtun WireGuard tunnel (dataplane belongs to VyOS). Replaced with:

  • WireguardKeypair + generate_wireguard_keypair() via X25519/Curve25519
  • derive_public_key() — public from base64 private
  • generate_client_config() — emits a valid .conf file
  • Fixed parse_wireguard_config() parser (blank-line peer flushing was broken)

Module is now declared in lib.rs and main.rs.

src/api/vyos.rs — VyOS HTTP API extensions

All config ops go through POST /configure; monitoring through GET /show/…:

client.configure_wireguard_interface("wg0", "172.27.1.1/32", &privkey, 51820, None).await?;
client.configure_wireguard_peer("wg0", "PE2", &pubkey, Some("5.254.54.3:51821"), &["172.27.0.0/20"], Some(25)).await?;
client.configure_vxlan(10001, "172.29.255.1", 9000, Some("tenant-1"), None).await?;
client.configure_bgp_system(65000, "172.29.255.1").await?;
client.configure_bgp_evpn_peer("172.29.255.2", 65000, "lo").await?;
client.configure_vrf("tenant-1", 1001, "65000:1001", "65000:1001").await?;
client.configure_vrrp("gw-ha", "eth0", "100.65.1.1/24", 10, 200).await?;
client.provision_tenant(&tenant).await?; // orchestrates all of the above + commit/save

Monitoring: get_bgp_summary, get_vxlan_status, get_vrf_routes, get_wireguard_status.

src/models/tenant.rs — new

Tenant with embedded VrfConfig, VxlanConfig, TenantWireguardConfig, VrrpConfig. Addressing auto-derived from numeric tenant_id:

  • WireGuard: 100.64.<id>.1/24, port 51820 + (id % 1000)
  • VXLAN VNI: 10000 + id
  • VRF table: 1000 + id, BGP RT: <as>:<table>

src/services/tenant.rs — new

TenantService wraps storage + provider access: create_tenant (auto-generates WireGuard keys), provision_tenant_on_router (pushes config to named VyOS provider, marks tenant active), delete_tenant.

CLI

Two new top-level subcommands:

bbctl tenants create <name> [--bgp-as 65000] [--source-address <ip>]
bbctl tenants list | show <id> | delete <id>

bbctl wireguard generate-keys
bbctl wireguard public-key <private-key>
bbctl wireguard client-config --private-key … --address … --server-public-key … --server-endpoint …
Original prompt

This section details on the original issue you should resolve

<issue_title>Comprehensive E2E Encrypted Multi-Tenant Network Architecture</issue_title>
<issue_description># Comprehensive E2E Encrypted Multi-Tenant Network Architecture

This document synthesizes our complete plan for building a secure, end-to-end encrypted, multi-tenant overlay network using VyOS, WireGuard, VXLAN, OSPF, L3VPN, and other technologies. The architecture implements a Unix philosophy-aligned approach with modular components that can be composed together while maintaining separation of concerns.

Architecture Overview

graph TB
    subgraph Physical["Physical Infrastructure"]
        direction TB
        DC1["Datacenter 1<br>5.254.54.0/26"]
        DC2["Datacenter 2<br>5.254.43.160/27"]
        CloudExt["Cloud Extensions<br>Dynamic"]
    end
    
    subgraph Hypervisor["Hypervisor Layer"]
        direction TB
        ArchLinux["Arch Linux OS"]
        OVS["Open vSwitch<br>Hardware Offload"]
        SRIOV["SR-IOV<br>Virtual Functions"]
        SystemdVMSpawn["systemd-vmspawn"]
    end
    
    subgraph Router["Virtual Router Layer"]
        direction TB
        VyOSVMs["VyOS VMs"]
        WireGuard["WireGuard Mesh<br>172.27.0.0/20"]
        VXLAN["VXLAN Tunnels"]
        OSPF["OSPF Areas"]
        BGP["BGP EVPN"]
        L3VPN["L3VPN (VRF)"]
    end
    
    subgraph Tenant["Tenant Layer"]
        direction TB
        TenantVMs["Tenant VMs"]
        ManagedServices["Managed Services"]
        K8S["Kubernetes Clusters"]
        Backups["Backup Systems"]
    end
    
    Physical --> Hypervisor
    Hypervisor --> Router
    Router --> Tenant
Loading

Network Addressing Schema

graph LR
    subgraph PublicSpace["Public Address Space"]
        DC1Public["DC1: 5.254.54.0/26"]
        DC2Public["DC2: 5.254.43.160/27"]
        DC2Additional["DC2 Additional: 5.254.43.208/29"]
    end
    
    subgraph ManagementSpace["Management Networks"]
        ControlPlane["Control Plane: 172.27.0.0/20"]
        BackboneNetwork["Backbone: 172.16.0.0/20"]
    end
    
    subgraph TenantSpace["Tenant Address Space"]
        CGNATBase["Base: 100.64.0.0/10"]
        WireGuardOverlay["WireGuard: 100.64.0.0/16"]
        TenantNetworks["Tenant Networks: 100.65.0.0/16"]
        TenantServices["Services: 100.80.0.0/16"]
        MigrationSpace["Migration: 100.96.0.0/16"]
    end
Loading

Implementation Plan

1. Physical Infrastructure Setup

The physical infrastructure consists of:

  • Datacenter 1:

    • Public Block: 5.254.54.0/26 (62 usable IPs)
    • Networking: 4x Intel X710 (10G) + 2x Mellanox CX4 (25G)
    • Management: IPMI via dedicated 1GbE NIC
  • Datacenter 2:

    • Public Block: 5.254.43.160/27 (30 usable IPs)
    • Additional Block: 5.254.43.208/29 (6 usable IPs)
    • Networking: 4x Intel X710 (10G) + 2x Mellanox CX4 (25G)
    • Management: IPMI via dedicated 1GbE NIC

2. Hypervisor Layer Configuration

Each bare metal server runs:

  1. Arch Linux operating system
  2. Open vSwitch with hardware offloading
  3. SR-IOV configuration for network cards
  4. systemd-vmspawn for VM deployment

NIC Configuration:

#!/bin/bash

# Configure Intel X710 NIC with SR-IOV
for i in {0..3}; do
  echo 7 > /sys/class/net/enp${i}s0/device/sriov_numvfs
  ip link set enp${i}s0 up
done

# Configure Mellanox CX4 NIC with SR-IOV
for i in {4..7}; do
  echo 7 > /sys/class/net/enp${i}s0/device/sriov_numvfs
  ip link set enp${i}s0 up
done

# Configure LACP Bond for Intel NICs
cat > /etc/systemd/network/10-bond0.netdev << EOF
[NetDev]
Name=bond0
Kind=bond

[Bond]
Mode=802.3ad
LACPTransmitRate=fast
MIIMonitorSec=1s
UpDelaySec=2s
DownDelaySec=2s
EOF

# Configure LACP Bond for Mellanox NICs
cat > /etc/systemd/network/20-bond1.netdev << EOF
[NetDev]
Name=bond1
Kind=bond

[Bond]
Mode=802.3ad
LACPTransmitRate=fast
MIIMonitorSec=1s
UpDelaySec=2s
DownDelaySec=2s
EOF

# Configure OVS with hardware offload
cat > /etc/openvswitch/ovs-setup.sh << 'EOF'
#!/bin/bash
ovs-vsctl --may-exist add-br br0
ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
ovs-vsctl add-port br0 bond0
ovs-vsctl add-port br0 bond1
EOF
chmod +x /etc/openvswitch/ovs-setup.sh

3. VyOS VM Deployment Using mkosi and systemd-vmspawn

Create a base VyOS image using mkosi:

#!/bin/bash

# Create mkosi configuration
cat > mkosi.default << EOF
[Distribution]
Distribution=vyos
Release=current

[Output]
Format=disk
Output=vyos-base.img
Size=2G

[Partitions]
RootSize=2G
EOF

# Build the image
mkosi

# Create systemd-vmspawn service template
cat > /etc/systemd/system/vyos@.service << EOF
[Unit]
Description=VyOS VM %i
After=network.target

[Service]
Type=notify
ExecStart=/usr/bin/systemd-vmspawn -i /var/lib/machines/vyos-base.img --network-veth -n vyos-%i
ExecStop=/usr/bin/machinectl poweroff vyos-%i
KillMode=mixed
Restart=on-failure
TimeoutStartSec=180

[Install]
WantedBy=multi-user.target
EOF

4. WireGuard Control Plane Configuration

The secure manageme...


💬 Send tasks to Copilot coding agent from Slack and Teams to turn conversations into code. Copilot posts an update in your thread when it's finished.

- Fix src/network.rs: replace boringtun/getrandom/x25519_dalek (missing deps)
  with x25519-dalek v2 + rand; add WireguardKeypair, generate_wireguard_keypair(),
  derive_public_key(), generate_client_config(), and fixed config parser
- Add pub mod network to lib.rs and main.rs
- Add base64, rand, x25519-dalek deps to Cargo.toml
- Extend src/api/vyos.rs with VyOS HTTP API methods: configure_set/delete,
  configure_wireguard_interface/peer, configure_vxlan, configure_bgp_system/evpn_peer,
  enable_bgp_evpn, configure_vrf, configure_vrrp, provision_tenant,
  get_bgp_summary/vxlan_status/vrf_routes/wireguard_status
- Add src/models/tenant.rs: Tenant, VrfConfig, VxlanConfig, TenantWireguardConfig, VrrpConfig
- Add src/services/tenant.rs: TenantService and TenantStorage
- Add Tenants and Wireguard CLI subcommands to main.rs

Co-authored-by: danielbodnar <1790726+danielbodnar@users.noreply.github.com>
@socket-security
Copy link

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedcargo/​x25519-dalek@​2.0.110010093100100

View full report

@socket-security
Copy link

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action Severity Alert  (click "▶" to expand/collapse)
Warn Low
Environment variable access: cargo curve25519-dalek

Location: Package overview

From: ?cargo/x25519-dalek@2.0.1cargo/curve25519-dalek@4.1.3

ℹ Read more on: This package | This alert | What is environment variable access?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should be clear about which environment variables they access, and care should be taken to ensure they only access environment variables they claim to.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore cargo/curve25519-dalek@4.1.3. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

Copilot AI changed the title [WIP] Add comprehensive E2E encrypted multi-tenant network architecture Implement E2E encrypted multi-tenant network architecture (WireGuard/VXLAN/BGP EVPN/L3VPN) Mar 18, 2026
Copilot AI requested a review from danielbodnar March 18, 2026 19:10
@danielbodnar danielbodnar marked this pull request as ready for review March 18, 2026 19:12
Copilot AI review requested due to automatic review settings March 18, 2026 19:12
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces tenant modeling/provisioning primitives and expands the VyOS client + CLI to support multi-tenant networking workflows (WireGuard/VXLAN/VRF/VRRP).

Changes:

  • Add Tenant/TenantService with in-memory tenant storage and router provisioning flow.
  • Refactor src/network.rs into WireGuard key/config utilities (keypair generation, pubkey derivation, client config generation, config parsing).
  • Extend VyOSClient with /configure helpers and higher-level methods for WireGuard/VXLAN/BGP/VRF/VRRP + tenant provisioning.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 13 comments.

Show a summary per file
File Description
src/services/tenant.rs New in-memory tenant storage + provisioning service integrating with ProviderService/VyOS.
src/services/mod.rs Exposes the new tenant service module.
src/network.rs Replaces tunnel implementation with WireGuard key/config utility functions.
src/models/tenant.rs New tenant domain model including VRF/VXLAN/WireGuard/VRRP configuration fields.
src/models/mod.rs Exposes the new tenant model module.
src/main.rs Adds tenants and wireguard CLI subcommands and exposes the root network module.
src/lib.rs Exposes the root network module for library consumers.
src/api/vyos.rs Adds configure/show helpers and higher-level configuration/provisioning methods.
Cargo.toml Adds crypto/keygen dependencies (base64, rand, x25519-dalek).
Cargo.lock Locks new dependencies and their transitive crates.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.ok_or_else(|| anyhow!("API key is required for HTTP API operations"))?;
let client = self.http_client.as_ref().unwrap();
let url = format!("https://{}:{}/configure", self.config.host, self.config.api_port);
debug!("VyOS configure op: POST {} {:?}", url, body);
Comment on lines +310 to +312
self.configure_set(
&["interfaces", "wireguard", interface, "port"],
Some(&port.to_string()),
Comment on lines +358 to +360
self.configure_set(
&["interfaces", "wireguard", interface, "peer", peer_name, "persistent-keepalive"],
Some(&ka.to_string()),
Comment on lines +386 to +390
let iface = format!("vxlan{}", vni);
self.configure_set(
&["interfaces", "vxlan", &iface, "vni"],
Some(&vni.to_string()),
).await?;
Comment on lines +395 to +398
self.configure_set(
&["interfaces", "vxlan", &iface, "mtu"],
Some(&mtu.to_string()),
).await?;
Comment on lines +532 to +539
self.configure_set(
&["high-availability", "vrrp", "group", group_id, "vrid"],
Some(&vrid.to_string()),
).await?;
self.configure_set(
&["high-availability", "vrrp", "group", group_id, "priority"],
Some(&priority.to_string()),
).await?;
Comment on lines +582 to +587
self.configure_wireguard_interface(
&wg_iface,
&tenant.wireguard.address,
&tenant.wireguard.private_key,
tenant.wireguard.port,
Some(&format!("Tenant {} WireGuard", tenant.name)),
Comment on lines +39 to +45
pub fn generate_wireguard_keypair() -> Result<WireguardKeypair> {
let mut rng = rand::thread_rng();
let mut secret_bytes = [0u8; 32];
rng.fill_bytes(&mut secret_bytes);

let secret = StaticSecret::from(secret_bytes);
let public = PublicKey::from(&secret);
Comment on lines +133 to +138
let vni = 10000 + tenant_id;
let table_id = 1000 + tenant_id;
let rt = format!("{}:{}", bgp_as, table_id);
let wg_address = format!("100.64.{}.1/24", tenant_id);
let network_cidr = format!("100.65.{}.0/24", tenant_id);

public_key: wg_public_key,
private_key: wg_private_key,
// Keep port in the valid range (51820–52819) using modulo 1000
port: 51820 + (tenant_id as u16 % 1000),
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

Comprehensive E2E Encrypted Multi-Tenant Network Architecture

3 participants