Support for External Jumbo Frames (RFD 689) by rcgoodfellow · Pull Request #10472 · oxidecomputer/omicron

rcgoodfellow · 2026-05-20T18:44:33Z

Implements the Omicron side of RFD 689.

Depends on:

A CLI/SDK build to drive this new functionality is here

incorporate jumbo frames into CLI/SDK oxide.rs#1445

Docs for this new feature are here

https://github.com/oxidecomputer/docs/pull/821

Some other fixes included here.

Updates lldpd to fix client/server version mismatch.
Don't squash lldpd 404 errors into 500 Oxide API errors.
Add support for static IPv4 routes over IPv6 nexthops (this was an oversight in initial IPv6 routing support and something I needed for my testing setup).

Testing

Lab Environment Performance

I've tested this on berlin. When the fleet level jumbo flag is set and instances have opted into jumbo, they come up with an 8500 byte MTU in linux. The numbers I see for TCP from berlin running through one of the external d4x4 routers is.

1500 MTU
- ~6 gbps single stream
- ~10 gbps 16 streams (-P 16 on the iperf3 client)
8500 MTU
- ~12 gbps single stream
- ~30 gbps 16 streams

For UDP

1500 MTU
- ~1 gbps single stream
- ~3 gbps 16 streams
8500 MTU
- ~3 gbps single stream
- ~11 gbps 16 streams

I also tested that a pair of clients with and without jumbo enabled can communicate over both TCP and UDP.

Control Bits

The fleet-level jumbo enable and instance-level jumbo enable have been tested in combination. For an instance to have jumbo frames enabled, both the fleet level setting and the instance level setting must be set to true. I've tested all combinations of these settings with an instance restart in between changes to pick up the setting, and things behave as expected.

rcgoodfellow · 2026-05-22T03:02:57Z

    curl -fL -o "$P5P_PATH" "$P5P_URL"

    RC=0
-    pfexec pkg install -g "$P5P_PATH" driver/network/opte || RC=$?


This pattern was not working for me in a4x2 with Helios 3. The new form does work.

FelixMcFelix

Thanks Ry, I mainly have questions around the organisation of versioned types.

FelixMcFelix · 2026-05-27T11:53:34Z

+            let settings = self
+                .db_datastore
+                .system_networking_settings_view(opctx)
+                .await?;


It's been a while since I've paged some of the authz stuff in, but this datastore method first checks the user has read access to the fleet. Will a user of an instance create/update endpoint necessarily have that? If not we'd need to read the value directly or construct a new OpContext at the right auth level.

This would apply to the other callsites of this function in instance.rs as well, if it needs handling.

Good catch. Removed the fleet read requirement from system_networking_settings_view and replace with comment explaining why there is not an opctx.authorize there.

FelixMcFelix · 2026-05-27T12:36:21Z

+        query_params: Query<
+            PaginatedByNameOrId<latest::project::ProjectSelector>,
+        >,
+    ) -> Result<HttpResponseOk<ResultsPage<Instance>>, HttpError> {


Again I haven't been on the versioned API surface much front (having had my head in illumos-gate for so long), but the bare Instance type as contrasted with latest::instance::Instance feels like it's out of the general pattern for most other endpoints:

Suggested change

) -> Result<HttpResponseOk<ResultsPage<Instance>>, HttpError> {

) -> Result<HttpResponseOk<ResultsPage<v2025_11_20_00::instance::Instance>>, HttpError> {

But looking into it it seems like any prior change to, e.g., fn instance_update has just exposed the bare Instance type. I don't know if this is maybe because it's the first time Instance itself has actually been updated since we started handling versioning this way?

My read on this is that we need to move external::Instance into versions::initial::instance, and then update those references.

Yeah that's probably a better way to do it. Moved to this approach, the reference updates are most of what is expanding the footprint of this PR in the latest commit.

FelixMcFelix · 2026-05-27T12:53:56Z

+            let settings = self
+                .db_datastore
+                .system_networking_settings_view(opctx)
+                .await?;
+            if !settings.external_jumbo_frames_opt_in_enabled {


Ditto on fleet read authz.

Handled this inside system_networking_settings_view

FelixMcFelix · 2026-05-27T12:54:05Z

+            let settings = self
+                .db_datastore
+                .system_networking_settings_view(opctx)
+                .await?;
+            if settings.external_jumbo_frames_opt_in_enabled {


Ditto on fleet read authz.

Handled this inside system_networking_settings_view

FelixMcFelix · 2026-05-27T12:56:20Z

+/// Default Ethernet MTU for external-facing OPTE ports, in bytes. Used when
+/// the per-instance jumbo-frames opt-in is unset or when the fleet-wide opt-in
+/// is disabled.
+pub const EXTERNAL_DEFAULT_MTU: u32 = 1500;


We're not actually reading this anywhere. Should we be sending this down rather than None? Obviously OPTE will pick the same value, but I being explicit about this as a choice the control plane is making seems beneficial to me.

FelixMcFelix · 2026-05-27T13:18:32Z

          "auto_restart_policy": {
            "nullable": true,
-            "description": "The auto-restart policy for this instance.\n\nThis policy determines whether the instance should be automatically restarted by the control plane on failure. If this is `null`, any explicitly configured auto-restart policy will be unset, and the control plane will select the default policy when determining whether the instance can be automatically restarted.\n\nCurrently, the global default auto-restart policy is \"best-effort\", so instances with `null` auto-restart policies will be automatically restarted. However, in the future, the default policy may be configurable through other mechanisms, such as on a per-project basis. In that case, any configured default policy will be used if this is `null`.",
+            "description": "The auto-restart policy for this instance.",


It looks as though something's happened to lose the extra docs information here (and on a few more fields of Instance)?

The extra docs are back now.

omicron/openapi/nexus/nexus-2026052000.0.0-8543ef.json

Lines 22690 to 22698 in f331e2d

"auto_restart_policy": {

"nullable": true,

"description": "The auto-restart policy configured for this instance, or `null` if no explicit policy has been configured.\n\nThis policy determines whether the instance should be automatically restarted by the control plane on failure. If this is `null`, the control plane will use the default policy when determining whether or not to automatically restart this instance, which may or may not allow it to be restarted. The value of the `auto_restart_enabled` field indicates whether the instance will be auto-restarted, based on its current policy or the default if it has no configured policy.",

"allOf": [

{

"$ref": "#/components/schemas/InstanceAutoRestartPolicy"

}

]

},

Not sure what happened there. Maybe it was a consequence of the composition with serde flatten that was there before the review feedback edits?

rcgoodfellow added 3 commits May 20, 2026 18:43

implement RFD 689

9c8d3c3

fix install_opte.sh override case for helios 3

653923d

bump maghemite

96476c4

david-crespo reviewed May 21, 2026

View reviewed changes

Comment thread openapi/nexus/nexus-2026052000.0.0-7e307e.json Outdated

rcgoodfellow added 2 commits May 20, 2026 20:25

bump maghemite

9f71c30

deal with maghemite api refactor fallout

9a6dd5a

rcgoodfellow force-pushed the ry/opte-984 branch from bbb7dd5 to 9a6dd5a Compare May 21, 2026 07:41

cleanup, bump lldpd

b5b2d28

rcgoodfellow commented May 22, 2026

View reviewed changes

Comment thread sled-agent/src/services.rs

rcgoodfellow commented May 22, 2026

View reviewed changes

rcgoodfellow added 4 commits May 22, 2026 15:25

add jumbo to instance view

7d96ac3

dont squash lldpd 404 into 500

df26643

support for v4 over v6 static routes

51b1185

proper versioning for instance view

057d562

rcgoodfellow marked this pull request as ready for review May 24, 2026 05:39

rcgoodfellow mentioned this pull request May 24, 2026

incorporate jumbo frames into CLI/SDK oxidecomputer/oxide.rs#1445

Open

rcgoodfellow changed the title ~~implement RFD 689~~ Support for Jumbo Frames (RFD 689) May 24, 2026

rcgoodfellow changed the title ~~Support for Jumbo Frames (RFD 689)~~ Support for External Jumbo Frames (RFD 689) May 26, 2026

rcgoodfellow mentioned this pull request May 26, 2026

Set OPTE maximum port MTU during creation oxidecomputer/opte#984

Open

rcgoodfellow requested a review from FelixMcFelix May 26, 2026 19:22

FelixMcFelix reviewed May 27, 2026

View reviewed changes

kyle's review feedback

f331e2d

rcgoodfellow mentioned this pull request May 28, 2026

Add jumbo frame support for control plane services #10501

Open

	) -> Result<HttpResponseOk<ResultsPage<Instance>>, HttpError> {
	) -> Result<HttpResponseOk<ResultsPage<v2025_11_20_00::instance::Instance>>, HttpError> {

	"auto_restart_policy": {
	"nullable": true,
	"description": "The auto-restart policy configured for this instance, or `null` if no explicit policy has been configured.\n\nThis policy determines whether the instance should be automatically restarted by the control plane on failure. If this is `null`, the control plane will use the default policy when determining whether or not to automatically restart this instance, which may or may not allow it to be restarted. The value of the `auto_restart_enabled` field indicates whether the instance will be auto-restarted, based on its current policy or the default if it has no configured policy.",
	"allOf": [
	{
	"$ref": "#/components/schemas/InstanceAutoRestartPolicy"
	}
	]
	},

Conversation

rcgoodfellow commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Lab Environment Performance

Control Bits

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FelixMcFelix left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rcgoodfellow commented May 20, 2026 •

edited

Loading