Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions _topic_maps/_topic_map.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1872,6 +1872,14 @@ Topics:
File: disabling-route-advertisements
- Name: Example route advertisements setup
File: example-route-advertisement-setup
- Name: BGP EVPN for user-defined networks
Dir: bgp_evpn_udn
Distros: openshift-enterprise, openshift-origin
Topics:
- Name: About BGP EVPN for primary cluster user-defined networks
File: about-bgp-evpn-user-defined-networks
- Name: Enabling BGP EVPN for primary cluster user-defined networks
File: enabling-bgp-evpn-primary-cudn
- Name: Using PTP hardware
Dir: ptp
Distros: openshift-enterprise,openshift-origin
Expand Down
104 changes: 104 additions & 0 deletions modules/nw-bgp-evpn-about.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
// Module included in the following assemblies:
//
// * networking/advanced_networking/bgp_evpn_udn/about-bgp-evpn-user-defined-networks.adoc

:_mod-docs-content-type: CONCEPT
[id="nw-bgp-evpn-about_{context}"]
= Overview of BGP EVPN with OVN-Kubernetes

[role="_abstract"]
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The abstract here was compiled from the following resources:

Border Gateway Protocol (BGP) Ethernet Virtual Private Network (EVPN) extends BGP with address families that distribute Ethernet and IP VPN routes, such as MAC reachability in a MAC-VRF and IP prefixes in an IP-VRF.

With the OVN-Kubernetes network plugin on {product-title}, you can attach primary cluster user-defined networks (CUDNs) to an external EVPN fabric. This integration lets you extend segmentation and reachability for tenant namespaces beyond the cluster boundary by using the same BGP-based standards as your physical or virtual network infrastructure.

OVN-Kubernetes drives EVPN integration with FRR-K8s: FRR establishes BGP EVPN sessions toward your fabric, while OVN-Kubernetes programs overlay and underlay state for the selected CUDNs. A `VTEP` custom resource defines how the cluster exposes virtual tunnel endpoints (VTEPs) for EVPN encapsulation, and `RouteAdvertisements` objects select which primary CUDNs are advertised and which FRR configuration to use.

[NOTE]
====
For primary CUDNs that use EVPN transport, traffic carried over the EVPN fabric typically uses VXLAN encapsulation rather than the Geneve overlay that OVN-Kubernetes commonly uses for east-west traffic on user-defined networks without EVPN. The cluster default pod network is not modeled as an EVPN VPN in this design; EVPN applies to primary CUDNs that you configure for EVPN.
====

[id="bgp-evpn-primary-use-cases_{context}"]
== BGP EVPN with primary CUDN use cases

Common reasons to deploy BGP EVPN with primary CUDNs include the following:

Fabric integration:: Map tenant networks to the same route-target and VRF semantics that you already use on an external BGP EVPN fabric.

Layer 2 stretch for mobility:: Extend a layer 2 segment into the cluster so that virtual machines or pods can keep addresses stable. EVPN MAC mobility can help when workloads move between nodes.

Simpler attachment than per-node local wiring:: Reduce dependence on host-specific local attachment patterns when the fabric can carry many tenant VNIs over shared uplinks.

Multi-cluster extension:: Stretching the same logical segment across clusters might be possible in some designs; treat this as advanced and confirm support and caveats for your release.

[id="about-bgp-evpn-other-features_{context}"]
== About BGP EVPN, BGP routing, and route advertisements

BGP EVPN for primary CUDNs is not a separate routing stack. It requires the same platform pieces as OVN-Kubernetes BGP:

* FRR-K8s must be deployed through Cluster Network Operator `additionalRoutingCapabilities` so that `FRRConfiguration` objects can express BGP and EVPN toward peers on the provider network.

* Route advertisements must be enabled in the OVN-Kubernetes configuration so that `RouteAdvertisements` CRs can associate fabric-facing FRR configuration with primary CUDNs that use EVPN transport.

You configure BGP neighbors, address families, and related policies in `FRRConfiguration` CRs. You then use `RouteAdvertisements` to tie those configurations to `ClusterUserDefinedNetwork` objects that are labeled or selected according to your design.

Plan BGP sessions to the underlay or route reflectors (typically on every node that participates) before you rely on EVPN reachability. If VTEP addresses are not taken from the machine network, ensure that those addresses, such as loopbacks, are routable on the underlay so that remote VTEPs can encapsulate to them.

[id="bgp-evpn-gateway-mode_{context}"]
== Gateway mode and node forwarding

EVPN for primary CUDNs is supported when the cluster uses local gateway mode. The `spec.defaultNetwork.ovnKubernetesConfig.gatewayConfig.routingViaHost` field must be `true` in the `Network` Operator API. Invalid combinations are rejected through resource status conditions.

[id="bgp-evpn-supported-platforms_{context}"]
== Supported platforms

BGP EVPN for primary CUDNs is intended for on-premises deployments where nodes peer with an external BGP EVPN fabric. Bare-metal clusters are the common case.

[id="bgp-evpn-compatible-features_{context}"]
== Compatible features

The following cluster networking features are compatible with EVPN on primary CUDNs:

* Egress Firewall
* Egress QoS
* Network policy, including AdminNetworkPolicy

Kubernetes `Service` objects need special consideration on EVPN:

Cluster IP:: Pods on the CUDN can use the cluster IP of services in their namespaces as usual.

NodePort, External IP, and LoadBalancer:: Support is limited compared with cluster-internal service access. MetalLB does not provide per-VRF advertisements to the same BGP peer in all configurations, so advertising external IPs or `LoadBalancer` IPs onto an EVPN fabric might require manual `FRRConfiguration` in FRR-K8s. Advertising cluster IPs into the EVPN fabric can also require explicit FRR-K8s configuration.

Multicast:: Multicast in a MAC-VRF (layer 2) context is supported with the expected EVPN multicast route behavior. Multicast in an IP-VRF (layer 3) context is not supported because kernel routing is not configured for that case.

[id="bgp-evpn-limitations_{context}"]
== Limitations

The following cluster networking features are incompatible with EVPN on primary CUDNs:

* Multiple external gateways (MEG). MEG requires shared gateway mode and EVPN is supported only in local gateway mode.
* EgressIP
* Egress Service
* IPsec encryption for OVN-Kubernetes
* No-overlay mode
* IPv6 VTEPs are not supported

[id="bgp-evpn-cudn-lifecycle_{context}"]
== Cluster user-defined network lifecycle

You enable EVPN by setting EVPN-related fields on a primary `ClusterUserDefinedNetwork` resource, for example, EVPN transport, VNI, and optional route target settings. If you omit a route target, implementations can default it from your autonomous system number and VNI. You cannot retroactively enable EVPN on an existing CUDN whose specification is already immutable. You must plan EVPN when you create the network or create a new CUDN for EVPN workloads.

[id="bgp-evpn-custom-resources_{context}"]
== Custom resources involved in EVPN

The following custom resources are commonly used in an EVPN deployment:

`FRRConfiguration`:: Defines BGP peers, autonomous system numbers, and EVPN-related address-family configuration toward the provider fabric. Namespaced; used with FRR-K8s.

`VTEP`:: Describes Virtual Tunnel Endpoint parameters (for example, a CIDR and `managed` or `unmanaged` mode). In `managed` mode, OVN-Kubernetes assigns a VTEP address from the range to each node. In `unmanaged` mode, each node uses an address on the host that falls inside the CIDR. You can configure host addressing with a `NodeNetworkConfigurationPolicy` resource from the Kubernetes NMState Operator, if you use it. The CIDR is often not the machine network. If you use a dedicated range, ensure that it is reachable on the underlay.

`RouteAdvertisements`:: Selects which networks to advertise (including primary CUDNs), which `FRRConfiguration` instances to use, and optional node selectors.

`ClusterUserDefinedNetwork`:: Defines the primary CUDN topology (layer 2 or layer 3), attaches namespaces, and carries EVPN transport settings that link the logical network to the configured VTEP and VPN identifiers.

Together, these objects separate fabric peering policy (FRR), what to advertise (`RouteAdvertisements`), tunnel endpoint context (`VTEP`), and tenant network definition (CUDN).
14 changes: 14 additions & 0 deletions modules/nw-bgp-evpn-enable.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
// Module included in the following assemblies:
//
// * networking/bgp_routing/enabling-bgp-evpn-primary-cudn.adoc

:_mod-docs-content-type: PROCEDURE
[id="nw-bgp-evpn-routing-config_{context}"]
= Enabling Border Gateway Protocol (BGP) EVPN routing

[role="_abstract"]
As a cluster administrator, you can. . .

.Prerequisites

.Procedure
1 change: 1 addition & 0 deletions networking/advanced_networking/bgp_evpn_udn/_attributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
:_mod-docs-content-type: ASSEMBLY
[id="about-bgp-evpn-user-defined-networks"]
= About BGP EVPN for primary cluster user-defined networks
include::_attributes/common-attributes.adoc[]
:context: about-bgp-evpn-user-defined-networks

toc::[]

[role="_abstract"]
Copy link
Copy Markdown
Member Author

@stevsmit stevsmit Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lines 1-14 are based on the following sections from the OKEP-5088 resource:

Problem statement

Goals

Introduction

EVPN/UDN background (MAC-VRF / IP-VRF / pod)

User stories

Not everything is verbatim from those resources. Information was compressed (e.g., the second sentence of line 10 layer 2 mac VRF (switched/bridged behavior), layer 3 IP-VRF (routed behavior) and other information (VPN overlay and VXlAN information included in the EVPN/UDN background section). This is a dense feature @_@

Ethernet Virtual Private Network (EVPN) is a Border Gateway Protocol (BGP) control-plane technology commonly used in data center fabrics. When you enable this capability on {product-title}, you can connect primary cluster user-defined networks (CUDNs) to an external BGP EVPN fabric so that pod and virtual machine traffic on those networks participates in the same routing and bridging semantics as the rest of your network.

Integrating BGP EVPN with primary CUDNs lets you align tenant segmentation with your existing BGP operations and reduce reliance on per-node static route configuration on hosts when you combine EVPN with route advertisements and local gateway mode. The feature builds on the same OVN-Kubernetes components as BGP routing and route advertisements: FRR-K8s, `RouteAdvertisements`, and `ClusterUserDefinedNetwork` objects configured for EVPN transport.
Copy link
Copy Markdown
Member Author

@stevsmit stevsmit Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the following true?
Integrating BGP EVPN with primary CUDNs lets you align tenant segmentation with your existing BGP operations

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reduce reliance on per-node static route configuration on hosts is referring to VRF-Lite (here and here)

Copy link
Copy Markdown
Member Author

@stevsmit stevsmit Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...combine EVPN with route advertisements and local gateway mode . There could be more to this, but my understanding is that by combining EVPN with route advertisements and local gateway mode means EVPN runs only in local gateway mode, and RouteAdvertisements define what is advertised over BGP/EVPN to the fabric.

Obtained from

enhancement limits the scope of EVPN support to local gateway mode and will specify what routes should be advertised via EVPN


Network administrators might use BGP EVPN with primary CUDNs to include layer 2 extension (MAC-VRF) for workloads that share a broadcast domain, layer 3 extension (IP-VRF) for routed VPN segments, and attachment to fabrics that already run BGP EVPN for redundancy and multihoming patterns such as BFD on the underlay.
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taken from slide 16 of the Enablement deck (I won't link like that here unless given permission).


[IMPORTANT]
====
If you are using the MetalLB Operator and there are existing `FRRConfiguration` custom resources (CRs) in the `metallb-system` namespace that were created by cluster administrators or by third-party components other than the MetalLB Operator, you must ensure that they are copied to the `openshift-frr-k8s` namespace or that those components use the `openshift-frr-k8s` namespace. For more information, see xref:../../../networking/advanced_networking/bgp_routing/migrating-frr-k8s-resources.adoc#migrating-frr-k8s-resources[Migrating FRR-K8s resources].
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This note is taken from a note that is currently in our documentation.

====

include::modules/nw-bgp-evpn-about.adoc[leveloffset=+1]

[role="_additional-resources"]
[id="additional-resources_about-bgp-evpn-user-defined-networks"]
== Additional resources
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ensure relevancy of included Additional resources.


* xref:../../../networking/advanced_networking/bgp_routing/about-bgp-routing.adoc#about-bgp-routing[About BGP routing]

* xref:../../../networking/advanced_networking/route_advertisements/about-route-advertisements.adoc#about-route-advertisements[About route advertisements]

* xref:../../../networking/multiple_networks/primary_networks/about-user-defined-networks.adoc#about-user-defined-networks[About user-defined networks]

* link:https://docs.frrouting.org/en/latest/bgp.html[FRRouting User Guide: BGP]

* link:https://docs.frrouting.org/en/latest/evpn.html[FRRouting User Guide: EVPN]
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
:_mod-docs-content-type: ASSEMBLY
[id="enabling-bgp-evpn-primary-cudn"]
= Enabling BGP EVPN for primary cluster user-defined networks
include::_attributes/common-attributes.adoc[]
:context: enabling-bgp-evpn-routing

toc::[]

[role="_abstract"]
As a cluster administrator, you can enable OVN-Kubernetes EVPN Border Gateway Protocol (BGP) routing support for your cluster.

include::modules/nw-bgp-evpn-enable.adoc[leveloffset=+1]
1 change: 1 addition & 0 deletions networking/advanced_networking/bgp_evpn_udn/images
1 change: 1 addition & 0 deletions networking/advanced_networking/bgp_evpn_udn/modules
1 change: 1 addition & 0 deletions networking/advanced_networking/bgp_evpn_udn/snippets