Feature Request: Report PCIe P2P Platform Routing Health

[3ricj]

Feature Request: Report PCIe P2P Platform Routing Health

Summary

Please add a driver diagnostic which reports whether PCIe GPU-to-GPU peer traffic is likely to be routed directly, or degraded by platform ACS/IOMMU configuration.

For PCIe P2P-capable GPU pairs, the driver should report:

ACS: OK|WARNING, <details>. IOMMU Modes: OK|WARNING, <details>.

This should be visible in a support-friendly location such as kernel logs, nvidia-smi topo, nvidia-smi -q, /proc/driver/nvidia, or UVM peer debug output.

Motivation

PCIe P2P can be functionally available but perform badly when the platform routes peer traffic through the CPU/root complex or through translated IOMMU paths. Today, users often only discover this after running a bandwidth benchmark and then manually checking BIOS settings, kernel command line options, IOMMU groups, and PCIe ACS registers.

A clear driver-side diagnostic would make these failures much easier to recognize and report.

Conditions To Detect

ACS

For each PCIe P2P GPU pair, inspect the upstream PCIe path between both endpoints. If any bridge/root port on the path advertises and enables ACS P2P redirect bits, report a warning.

Relevant ACS control bits:

PCI_ACS_RR - P2P Request Redirect
PCI_ACS_CR - P2P Completion Redirect

Example OK output:

P2P platform routing check for GPU0 (0000:06:00.0) <-> GPU1 (0000:09:00.0): ACS: OK, P2P Request/Completion Redirect not detected. IOMMU Modes: OK, identity/identity.

Example warning output:

P2P platform routing check for GPU0 (0000:06:00.0) <-> GPU1 (0000:09:00.0): ACS: WARNING, P2P Request/Completion Redirect enabled on PCIe bridge 0000:00:03.0 (ACSCap=0xNN ACSCtl=0xNN); GPU-to-GPU PCIe traffic may be routed upstream through the root complex, reducing bandwidth. IOMMU Modes: OK, identity/identity.

IOMMU

For each GPU endpoint, report whether the active IOMMU/DMA domain is identity/passthrough or translated.

Example OK output:

IOMMU Modes: OK, identity/identity

Example warning output:

IOMMU Modes: WARNING, translated-dma/identity; translation is active for at least one GPU. If P2P bandwidth is low, boot with IOMMU passthrough, for example iommu=pt with the platform-specific IOMMU enabled.

Suggested Behavior

• Do not fail P2P solely because ACS redirect or IOMMU translation is detected.
• Treat this as a performance diagnostic, not a correctness gate.
• Report at most once per physical GPU pair per driver/module load to avoid log spam.
• Key de-duplication by physical PCI BDF pair, not by CUDA ordinal or transient UVM parent GPU ID.
• Include both endpoint BDFs in the message.
• Include the ACS bridge BDF and ACS capability/control values when warning.
• Include the IOMMU mode for both endpoints.

Why This Should Be In The Driver

The driver already knows when a GPU pair is considered P2P-capable and which path is being used, such as PCIe BAR1 P2P versus NVLink. It is in the best position to explain that a pair is functionally supported but may be routed suboptimally by platform policy.

This also makes bug reports more actionable. A support log would immediately show whether the platform has:

• ACS redirect enabled on the path.
• IOMMU identity/passthrough mode.
• IOMMU translated DMA mode.
• A clean direct-routing configuration.

Prototype

A local prototype was implemented in UVM's parent peer initialization path:

• Hook: after RM reports a parent GPU pair link type of PCIE or PCIE_BAR1.
• ACS check: walk the upstream PCIe bridge path between both endpoints and inspect PCI_EXT_CAP_ID_ACS, PCI_ACS_CAP, and PCI_ACS_CTRL.
• IOMMU check: inspect the Linux IOMMU domain for each GPU endpoint and classify as identity, none, translated-dma, translated, or unavailable.
• De-duplication: sorted physical PCI BDF pair.

On a 4-GPU system, exercising all pairs produced exactly one line for each physical pair:

0000:05:00.0 <-> 0000:06:00.0
0000:05:00.0 <-> 0000:09:00.0
0000:05:00.0 <-> 0000:0d:00.0
0000:06:00.0 <-> 0000:09:00.0
0000:06:00.0 <-> 0000:0d:00.0
0000:09:00.0 <-> 0000:0d:00.0

All pairs reported:

ACS: OK, P2P Request/Completion Redirect not detected. IOMMU Modes: OK, identity/identity.

Prototype Patch

The following patch shows the local prototype implementation against kernel-open/nvidia-uvm/uvm_gpu.c:

diff --git a/kernel-open/nvidia-uvm/uvm_gpu.c b/kernel-open/nvidia-uvm/uvm_gpu.c
index 4dedd4e3..d4ee56d2 100644
--- a/kernel-open/nvidia-uvm/uvm_gpu.c
+++ b/kernel-open/nvidia-uvm/uvm_gpu.c
@@ -46,6 +46,9 @@
 #include "uvm_mmu.h"
 #include "uvm_kvmalloc.h"
 
+#include <linux/iommu.h>
+#include <linux/pci_regs.h>
+
 #define UVM_PROC_GPUS_PEER_DIR_NAME "peers"
 
 // The uvm_peer_copy module parameter enables to choose from "phys" or "virt".
@@ -2438,6 +2441,237 @@ static NV_STATUS init_procfs_parent_peer_files(uvm_parent_gpu_t *parent_gpu0, uv
     return NV_OK;
 }
 
+typedef struct
+{
+    bool valid;
+    NvU64 endpoint0;
+    NvU64 endpoint1;
+} uvm_pcie_p2p_platform_routing_report_t;
+
+static uvm_pcie_p2p_platform_routing_report_t
+    g_uvm_pcie_p2p_platform_routing_reported[UVM_MAX_UNIQUE_PARENT_GPU_PAIRS];
+
+static NvU64 uvm_pci_dev_bdf_key(struct pci_dev *pci_dev)
+{
+    return (((NvU64)(NvU32)pci_domain_nr(pci_dev->bus)) << 32) |
+           (((NvU64)pci_dev->bus->number) << 16) |
+           (NvU64)pci_dev->devfn;
+}
+
+static bool uvm_pcie_p2p_platform_routing_already_reported(struct pci_dev *pci_dev0, struct pci_dev *pci_dev1)
+{
+    NvU64 endpoint0 = uvm_pci_dev_bdf_key(pci_dev0);
+    NvU64 endpoint1 = uvm_pci_dev_bdf_key(pci_dev1);
+    size_t i;
+
+    if (endpoint0 > endpoint1) {
+        NvU64 tmp = endpoint0;
+        endpoint0 = endpoint1;
+        endpoint1 = tmp;
+    }
+
+    for (i = 0; i < NV_ARRAY_ELEMENTS(g_uvm_pcie_p2p_platform_routing_reported); i++) {
+        uvm_pcie_p2p_platform_routing_report_t *report = &g_uvm_pcie_p2p_platform_routing_reported[i];
+
+        if (!report->valid) {
+            report->valid = true;
+            report->endpoint0 = endpoint0;
+            report->endpoint1 = endpoint1;
+            return false;
+        }
+
+        if (report->endpoint0 == endpoint0 && report->endpoint1 == endpoint1)
+            return true;
+    }
+
+    return false;
+}
+
+static struct pci_dev *uvm_pcie_common_upstream_bridge(struct pci_dev *pci_dev0, struct pci_dev *pci_dev1)
+{
+    struct pci_dev *bridge0;
+
+    for (bridge0 = pci_upstream_bridge(pci_dev0); bridge0 != NULL; bridge0 = pci_upstream_bridge(bridge0)) {
+        struct pci_dev *bridge1;
+
+        for (bridge1 = pci_upstream_bridge(pci_dev1); bridge1 != NULL; bridge1 = pci_upstream_bridge(bridge1)) {
+            if (bridge0 == bridge1)
+                return bridge0;
+        }
+    }
+
+    return NULL;
+}
+
+static bool uvm_pcie_bridge_has_acs_redirect(struct pci_dev *bridge, NvU16 *acs_cap, NvU16 *acs_ctrl)
+{
+    int pos;
+
+    if (bridge == NULL)
+        return false;
+
+    pos = pci_find_ext_capability(bridge, PCI_EXT_CAP_ID_ACS);
+    if (pos == 0)
+        return false;
+
+    if (pci_read_config_word(bridge, pos + PCI_ACS_CAP, acs_cap) != 0)
+        return false;
+
+    if (pci_read_config_word(bridge, pos + PCI_ACS_CTRL, acs_ctrl) != 0)
+        return false;
+
+    return (*acs_cap & *acs_ctrl & (PCI_ACS_RR | PCI_ACS_CR)) != 0;
+}
+
+static bool uvm_pcie_path_has_acs_redirect(struct pci_dev *pci_dev0,
+                                           struct pci_dev *pci_dev1,
+                                           struct pci_dev **redirect_bridge,
+                                           NvU16 *acs_cap,
+                                           NvU16 *acs_ctrl)
+{
+    struct pci_dev *common_bridge = uvm_pcie_common_upstream_bridge(pci_dev0, pci_dev1);
+    struct pci_dev *bridge;
+
+    *redirect_bridge = NULL;
+    *acs_cap = 0;
+    *acs_ctrl = 0;
+
+    for (bridge = pci_upstream_bridge(pci_dev0); bridge != NULL; bridge = pci_upstream_bridge(bridge)) {
+        if (uvm_pcie_bridge_has_acs_redirect(bridge, acs_cap, acs_ctrl)) {
+            *redirect_bridge = bridge;
+            return true;
+        }
+
+        if (bridge == common_bridge)
+            break;
+    }
+
+    for (bridge = pci_upstream_bridge(pci_dev1); bridge != NULL; bridge = pci_upstream_bridge(bridge)) {
+        if (uvm_pcie_bridge_has_acs_redirect(bridge, acs_cap, acs_ctrl)) {
+            *redirect_bridge = bridge;
+            return true;
+        }
+
+        if (bridge == common_bridge)
+            break;
+    }
+
+    return false;
+}
+
+static bool uvm_parent_gpu_iommu_is_translated(uvm_parent_gpu_t *parent_gpu, const char **mode)
+{
+#if defined(CONFIG_IOMMU_API)
+    struct iommu_domain *domain;
+
+    if (parent_gpu->pci_dev == NULL) {
+        *mode = "unknown";
+        return false;
+    }
+
+    domain = iommu_get_domain_for_dev(&parent_gpu->pci_dev->dev);
+    if (domain == NULL) {
+        *mode = "none";
+        return false;
+    }
+
+    if (domain->type == IOMMU_DOMAIN_IDENTITY) {
+        *mode = "identity";
+        return false;
+    }
+
+#if defined(NV_IOMMU_IS_DMA_DOMAIN_PRESENT)
+    if (iommu_is_dma_domain(domain))
+        *mode = "translated-dma";
+    else
+#endif
+        *mode = "translated";
+
+    return true;
+#else
+    *mode = "unavailable";
+    return false;
+#endif
+}
+
+static void uvm_report_pcie_p2p_platform_routing(uvm_parent_gpu_t *parent_gpu0,
+                                                 uvm_parent_gpu_t *parent_gpu1,
+                                                 uvm_gpu_link_type_t link_type)
+{
+    struct pci_dev *redirect_bridge;
+    const char *iommu_mode0;
+    const char *iommu_mode1;
+    NvU16 acs_cap;
+    NvU16 acs_ctrl;
+    bool acs_redirect_enabled;
+    bool iommu0_translated;
+    bool iommu1_translated;
+
+    if (link_type != UVM_GPU_LINK_PCIE && link_type != UVM_GPU_LINK_PCIE_BAR1)
+        return;
+
+    if (parent_gpu0->pci_dev == NULL || parent_gpu1->pci_dev == NULL)
+        return;
+
+    if (uvm_pcie_p2p_platform_routing_already_reported(parent_gpu0->pci_dev, parent_gpu1->pci_dev))
+        return;
+
+    acs_redirect_enabled = uvm_pcie_path_has_acs_redirect(parent_gpu0->pci_dev,
+                                                          parent_gpu1->pci_dev,
+                                                          &redirect_bridge,
+                                                          &acs_cap,
+                                                          &acs_ctrl);
+    iommu0_translated = uvm_parent_gpu_iommu_is_translated(parent_gpu0, &iommu_mode0);
+    iommu1_translated = uvm_parent_gpu_iommu_is_translated(parent_gpu1, &iommu_mode1);
+
+    if (acs_redirect_enabled) {
+        UVM_INFO_PRINT("P2P platform routing check for %s (%s) <-> %s (%s): "
+                       "ACS: WARNING, P2P Request/Completion Redirect enabled on PCIe bridge %s "
+                       "(ACSCap=0x%x ACSCtl=0x%x); GPU-to-GPU PCIe traffic may be routed "
+                       "upstream through the root complex, reducing bandwidth. "
+                       "IOMMU Modes: %s, %s/%s.\n",
+                       uvm_parent_gpu_name(parent_gpu0),
+                       pci_name(parent_gpu0->pci_dev),
+                       uvm_parent_gpu_name(parent_gpu1),
+                       pci_name(parent_gpu1->pci_dev),
+                       pci_name(redirect_bridge),
+                       acs_cap,
+                       acs_ctrl,
+                       (iommu0_translated || iommu1_translated) ? "WARNING" : "OK",
+                       iommu_mode0,
+                       iommu_mode1);
+    }
+
+    if (iommu0_translated || iommu1_translated) {
+        UVM_INFO_PRINT("P2P platform routing check for %s (%s) <-> %s (%s): "
+                       "ACS: %s%s%s. IOMMU Modes: WARNING, %s/%s; "
+                       "translation is active for at least one GPU. If P2P bandwidth is low, boot with IOMMU "
+                       "passthrough, for example iommu=pt with the platform-specific "
+                       "IOMMU enabled.\n",
+                       uvm_parent_gpu_name(parent_gpu0),
+                       pci_name(parent_gpu0->pci_dev),
+                       uvm_parent_gpu_name(parent_gpu1),
+                       pci_name(parent_gpu1->pci_dev),
+                       acs_redirect_enabled ? "WARNING, P2P Request/Completion Redirect enabled on PCIe bridge " : "OK",
+                       acs_redirect_enabled ? pci_name(redirect_bridge) : "",
+                       acs_redirect_enabled ? "" : ", redirect not detected",
+                       iommu_mode0,
+                       iommu_mode1);
+    }
+
+    if (!acs_redirect_enabled && !iommu0_translated && !iommu1_translated) {
+        UVM_INFO_PRINT("P2P platform routing check for %s (%s) <-> %s (%s): "
+                       "ACS: OK, P2P Request/Completion Redirect not detected. "
+                       "IOMMU Modes: OK, %s/%s.\n",
+                       uvm_parent_gpu_name(parent_gpu0),
+                       pci_name(parent_gpu0->pci_dev),
+                       uvm_parent_gpu_name(parent_gpu1),
+                       pci_name(parent_gpu1->pci_dev),
+                       iommu_mode0,
+                       iommu_mode1);
+    }
+}
+
 static NV_STATUS parent_peers_init(uvm_parent_gpu_t *parent_gpu0,
                                    uvm_parent_gpu_t *parent_gpu1,
                                    uvm_parent_gpu_peer_t *parent_peer_caps)
@@ -2499,6 +2733,8 @@ static NV_STATUS parent_peers_init(uvm_parent_gpu_t *parent_gpu0,
     if (parent_peer_caps->bar1_p2p_dma_size[0] || parent_peer_caps->bar1_p2p_dma_size[1])
         UVM_ASSERT(link_type == UVM_GPU_LINK_PCIE_BAR1);
 
+    uvm_report_pcie_p2p_platform_routing(parent_gpu0, parent_gpu1, link_type);
+
     return NV_OK;
 
 cleanup:

Acceptance Criteria

• The diagnostic identifies ACS redirect state for each PCIe P2P-capable GPU pair.
• The diagnostic identifies IOMMU mode for each endpoint.
• Output clearly labels each field as OK or WARNING.
• The diagnostic does not change P2P enablement behavior.
• The diagnostic is de-duplicated per physical GPU pair.
• The diagnostic appears in NVIDIA bug reports or another standard support collection path.