database support for PendingMgsUpdate for SP updates#8291
Merged
davepacheco merged 17 commits intomainfrom Jun 30, 2025
Merged
Conversation
jgallagher
reviewed
Jun 17, 2025
Collaborator
Author
|
The cleanup really wanted a rustfmt pass, but rustfmt hasn't been working on this code path. Fixing that added a bunch of noise so I split that into #8394 and made this PR dependent on that one. |
jgallagher
reviewed
Jun 23, 2025
| pending_mgs_updates, | ||
| sleds: sled_configs, | ||
| parent_blueprint_id, | ||
| internal_dns_version, |
Contributor
There was a problem hiding this comment.
It's below the diff, but:
a) I think we also need to update blueprint_delete, right?
b) I'm surprised we don't have any tests that caught this; do we need a test like we have for inventory that checks that deletion doesn't leave any leftovers behind in any of the bp_* tables?
Collaborator
Author
There was a problem hiding this comment.
As you saw, I implemented a test in #8439. I merged that in here. It correctly caught that the new table wasn't checked by the test, but once I fixed that, it did not immediately catch that I hadn't fixed blueprint_delete. That's because it only checks a relatively boring blueprint. I've updated the test to check deletion for a blueprint that we've deliberately constructed to contain a variety of values. (That blueprint is currently used to exercise that we can write a blueprint and read it back and get the same thing so it's deliberately trying to cover various values. Fortunately, I had remembered to update that here.)
We could augment this with something like we do for the inventory tests, which is to test that all the expected tables are populated after we insert a "representative" blueprint. I filed #8455 for this.
karencfv
reviewed
Jun 26, 2025
Comment on lines
+505
to
+518
| let (expected_active_version, expected_inactive_version) = | ||
| match &update.details { | ||
| PendingMgsUpdateDetails::Sp { | ||
| expected_active_version, | ||
| expected_inactive_version, | ||
| } => ( | ||
| expected_active_version, | ||
| expected_inactive_version, | ||
| ), | ||
| PendingMgsUpdateDetails::Rot { .. } | ||
| | PendingMgsUpdateDetails::RotBootloader { | ||
| .. | ||
| } => continue, | ||
| }; |
Contributor
There was a problem hiding this comment.
How will this fit with the RoT and RoT bootloader? Specifically with the RoT, will we only want the active/inactive versions like it's represented here? Or will we want to have all of the other additional fields (active slot, persistent boot preference, etc)?
If it's the latter, is the idea to have additional columns for these (active_slot, slot_a_version etc for RoT, and stage0_version, stage0_next_version for bootloader) and leave expected_active_version and expected_inactive_version empty? Or do we want completely different tables for these two (bp_pending_mgs_update_rot, bp_pending_mgs_update_rot_bootloader)?
Collaborator
Author
There was a problem hiding this comment.
My plan is to use separate tables for those and just duplicate the common fields.
Collaborator
Author
|
I finally managed to test this today on dublin and I believe it's worked sufficiently. It also requires #8466 to work. What I tested was commit 1f900db from the tip of that PR's branch, not this one. I will plan to land both as close in time as I can (but the latter will probably require a manual git resolution and then re-running CI). Here's what I did, based on the current Update Bringup Script. I used I saved an omdb-printed summary of inventory collections at this point as a baseline. I then used Then I applied the usual TUF trust workaround and enabled the planner background task and restart Nexus. Then I enabled the current target blueprint. Then I uploaded the TUF repo for commit 1f900db (the same one that this system was installed with) and set the target release. I was surprised to find that it did kick off some switch updates. But that appears correct: the switches on this system were running 1.0.35 (probably a result of previously trying this on Friday) and the TUF repo contains 1.0.39. (This might be a bug in rkadm if it's not updating the switch SPs.) Then it updated the three sleds that it should have. Then it started updating zones. That's still going on and I will wait for it to finish but at this point it's looking good. So far, I've got these three inventory collections: Here's the changes from my downgrade: # diff inventory-initial-sps.txt inventory-downgrade-sp.txt
1,4c1,4
< collection: 07983eb5-d389-4c8d-8553-298102b5d57c
< collector: 59530bc5-684a-42ac-930a-d0bbd90600ca (likely a Nexus instance)
< started: 2025-06-30T17:27:23.861Z
< done: 2025-06-30T17:27:35.447Z
---
> collection: 903364c6-a754-4a69-9a9d-6ec4d13e3b93
> collector: ce7cda27-045e-4191-be02-8f26c00d8d1d (likely a Nexus instance)
> started: 2025-06-30T17:43:58.275Z
> done: 2025-06-30T17:44:08.161Z
6,23c6,23
< error 0: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "aba7a9fd-a30f-4ed2-945d-3b0a3b15566a", "content-length": "247", "date": "Mon, 30 Jun 2025 17:27:29 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image caboose does not contain 'VERS'", request_id: "aba7a9fd-a30f-4ed2-945d-3b0a3b15566a" }
< error 1: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot1: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "51a776c6-72a3-492a-82b8-3c9f0ee15504", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:29 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image does not include a caboose", request_id: "51a776c6-72a3-492a-82b8-3c9f0ee15504" }
< error 2: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose RotSlotB: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "9ce9cba5-e519-403b-b295-ad65f28b6d55", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:29 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: failed to read data from the caboose", request_id: "9ce9cba5-e519-403b-b295-ad65f28b6d55" }
< error 3: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "9eaf6a5c-ed12-4614-b9d9-5c0534fab98f", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:29 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "9eaf6a5c-ed12-4614-b9d9-5c0534fab98f" }
< error 4: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0Next: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "d17871bd-ff59-45e8-b530-89e6e3a4b8f8", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:29 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "d17871bd-ff59-45e8-b530-89e6e3a4b8f8" }
< error 5: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page Cmpa: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "9403210d-2edc-466b-b587-1e1a223a1397", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:29 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "9403210d-2edc-466b-b587-1e1a223a1397" }
< error 6: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaActive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "da58527e-a089-4d92-afde-b19546048a45", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:30 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "da58527e-a089-4d92-afde-b19546048a45" }
< error 7: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaInactive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "95a0cdc6-2dc0-4743-9589-ec833626169d", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:30 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "95a0cdc6-2dc0-4743-9589-ec833626169d" }
< error 8: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaScratch: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "aad3a5d1-26f6-42e0-9e84-964a8783f970", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:30 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "aad3a5d1-26f6-42e0-9e84-964a8783f970" }
< error 9: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "a446f0a3-1364-4df3-a91e-ce5d5d43ad3e", "content-length": "247", "date": "Mon, 30 Jun 2025 17:27:33 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image caboose does not contain 'VERS'", request_id: "a446f0a3-1364-4df3-a91e-ce5d5d43ad3e" }
< error 10: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot1: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "4c8490a1-b779-4f9d-ae10-f09be0b20fd4", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:33 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image does not include a caboose", request_id: "4c8490a1-b779-4f9d-ae10-f09be0b20fd4" }
< error 11: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose RotSlotB: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "5bf06e27-40c5-4970-9fe9-a904171834e5", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:33 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: failed to read data from the caboose", request_id: "5bf06e27-40c5-4970-9fe9-a904171834e5" }
< error 12: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "f153361e-b921-4cf0-8ec4-586faff25706", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:33 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "f153361e-b921-4cf0-8ec4-586faff25706" }
< error 13: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0Next: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "f4e52c1e-7c5f-47e7-9b6e-45f7b4b4e2b0", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:33 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "f4e52c1e-7c5f-47e7-9b6e-45f7b4b4e2b0" }
< error 14: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page Cmpa: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "06f7d51b-f609-4db2-8ac6-9a5559fb9faa", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:33 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "06f7d51b-f609-4db2-8ac6-9a5559fb9faa" }
< error 15: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaActive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "edaacfcb-9ac2-405e-b7d2-a3c6c8d4ddd5", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:33 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "edaacfcb-9ac2-405e-b7d2-a3c6c8d4ddd5" }
< error 16: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaInactive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "823844c3-8b60-44b2-8145-6f30e50ada35", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:33 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "823844c3-8b60-44b2-8145-6f30e50ada35" }
< error 17: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaScratch: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "e1f0e775-45b9-4031-9e7e-ba126981c25a", "content-length": "242", "date": "Mon, 30 Jun 2025 17:27:33 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "e1f0e775-45b9-4031-9e7e-ba126981c25a" }
---
> error 0: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "e4535ab7-65a1-452e-82e0-b92529e4625d", "content-length": "247", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image caboose does not contain 'VERS'", request_id: "e4535ab7-65a1-452e-82e0-b92529e4625d" }
> error 1: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot1: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "5794e863-abf8-4b62-9b88-4db47fa1157d", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image does not include a caboose", request_id: "5794e863-abf8-4b62-9b88-4db47fa1157d" }
> error 2: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose RotSlotB: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "3f642d6f-5c1a-4783-aac0-0181f4b8417d", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: failed to read data from the caboose", request_id: "3f642d6f-5c1a-4783-aac0-0181f4b8417d" }
> error 3: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "b5844cae-774d-4a66-9d38-8c09236c6c00", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "b5844cae-774d-4a66-9d38-8c09236c6c00" }
> error 4: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0Next: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "2f3caf74-897e-4191-9be8-2e5ca6f1d968", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "2f3caf74-897e-4191-9be8-2e5ca6f1d968" }
> error 5: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page Cmpa: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "5a299aa8-174b-45e9-94db-cd70a5c64ffd", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "5a299aa8-174b-45e9-94db-cd70a5c64ffd" }
> error 6: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaActive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "c65cff3e-b62a-4c34-9392-d6303c050674", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "c65cff3e-b62a-4c34-9392-d6303c050674" }
> error 7: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaInactive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "3b9a48d0-72bd-478c-8779-2fd14f98c5c9", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "3b9a48d0-72bd-478c-8779-2fd14f98c5c9" }
> error 8: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaScratch: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "f4c063e2-b8a4-47ea-a58d-c4ee60e2a207", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "f4c063e2-b8a4-47ea-a58d-c4ee60e2a207" }
> error 9: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "cc9b63b4-32ec-4944-9f2a-850e2f47c854", "content-length": "247", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image caboose does not contain 'VERS'", request_id: "cc9b63b4-32ec-4944-9f2a-850e2f47c854" }
> error 10: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot1: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "dde7a768-f004-4d93-b761-60a68c3b46f8", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image does not include a caboose", request_id: "dde7a768-f004-4d93-b761-60a68c3b46f8" }
> error 11: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose RotSlotB: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "fc373a2e-0136-4f4a-b2e9-b6d4eda5d5df", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: failed to read data from the caboose", request_id: "fc373a2e-0136-4f4a-b2e9-b6d4eda5d5df" }
> error 12: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "46b5a3d2-7ffc-4fc5-9584-f0bc9fe4b3d8", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "46b5a3d2-7ffc-4fc5-9584-f0bc9fe4b3d8" }
> error 13: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0Next: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "1b300061-35af-42ef-9337-17b591349c45", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "1b300061-35af-42ef-9337-17b591349c45" }
> error 14: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page Cmpa: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "234e56a5-63d3-4c72-a82a-599fa9df53b1", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "234e56a5-63d3-4c72-a82a-599fa9df53b1" }
> error 15: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaActive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "5456ac62-6968-450a-a93f-ffffda217791", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "5456ac62-6968-450a-a93f-ffffda217791" }
> error 16: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaInactive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "a4b9f693-92c7-4ba4-a4b1-bf1c1f72b091", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "a4b9f693-92c7-4ba4-a4b1-bf1c1f72b091" }
> error 17: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaScratch: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "51023fa0-1280-4917-bd6c-2d8601ab5b8a", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "51023fa0-1280-4917-bd6c-2d8601ab5b8a" }
30c30
< found at: 2025-06-30 17:27:25.018537 UTC from http://[fd00:1122:3344:103::2]:12225
---
> found at: 2025-06-30 17:43:59.682939 UTC from http://[fd00:1122:3344:103::2]:12225
33,34c33,34
< SpSlot0 gimlet-c gimlet-c 1.0.38 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
< SpSlot1 gimlet-c gimlet-c 1.0.34 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
---
> SpSlot0 gimlet-c gimlet-c 1.0.34 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
> SpSlot1 gimlet-c gimlet-c 1.0.38 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
57c57
< found at: 2025-06-30 17:27:28.551105 UTC from http://[fd00:1122:3344:103::2]:12225
---
> found at: 2025-06-30 17:44:03.555192 UTC from http://[fd00:1122:3344:103::2]:12225
60,61c60,61
< SpSlot0 gimlet-e gimlet-e 1.0.38 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
< SpSlot1 gimlet-e gimlet-e 1.0.34 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
---
> SpSlot0 gimlet-e gimlet-e 1.0.34 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
> SpSlot1 gimlet-e gimlet-e 1.0.38 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
84c84
< found at: 2025-06-30 17:27:23.888888 UTC from http://[fd00:1122:3344:103::2]:12225
---
> found at: 2025-06-30 17:43:58.301368 UTC from http://[fd00:1122:3344:103::2]:12225
111c111
< found at: 2025-06-30 17:27:26.248046 UTC from http://[fd00:1122:3344:103::2]:12225
---
> found at: 2025-06-30 17:44:02.423975 UTC from http://[fd00:1122:3344:103::2]:12225
114,115c114,115
< SpSlot0 gimlet-d gimlet-d 1.0.38 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
< SpSlot1 gimlet-d gimlet-d 1.0.34 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
---
> SpSlot0 gimlet-d gimlet-d 1.0.34 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
> SpSlot1 gimlet-d gimlet-d 1.0.38 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
138c138
< found at: 2025-06-30 17:27:30.562341 UTC from http://[fd00:1122:3344:103::2]:12225
---
> found at: 2025-06-30 17:44:04.708406 UTC from http://[fd00:1122:3344:103::2]:12225
156c156
< found at: 2025-06-30 17:27:31.040778 UTC from http://[fd00:1122:3344:103::2]:12225
---
> found at: 2025-06-30 17:44:05.063202 UTC from http://[fd00:1122:3344:103::2]:12225
183c183
< found at: 2025-06-30 17:27:32.589292 UTC from http://[fd00:1122:3344:103::2]:12225
---
> found at: 2025-06-30 17:44:06.210617 UTC from http://[fd00:1122:3344:103::2]:12225More importantly, here's the result after all the SP upgrades (zone upgrades are still ongoing): root@oxz_switch1:~# diff -U 5 inventory-downgrade-sp.txt inventory-middle-sp.txt
--- inventory-downgrade-sp.txt Mon Jun 30 17:45:03 2025
+++ inventory-middle-sp.txt Mon Jun 30 18:35:34 2025
@@ -1,39 +1,39 @@
-collection: 903364c6-a754-4a69-9a9d-6ec4d13e3b93
-collector: ce7cda27-045e-4191-be02-8f26c00d8d1d (likely a Nexus instance)
-started: 2025-06-30T17:43:58.275Z
-done: 2025-06-30T17:44:08.161Z
+collection: 7c443a20-be90-46f8-9ad6-428078d23574
+collector: 59530bc5-684a-42ac-930a-d0bbd90600ca (likely a Nexus instance)
+started: 2025-06-30T18:35:20.900Z
+done: 2025-06-30T18:35:29.926Z
errors: 18
- error 0: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "e4535ab7-65a1-452e-82e0-b92529e4625d", "content-length": "247", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image caboose does not contain 'VERS'", request_id: "e4535ab7-65a1-452e-82e0-b92529e4625d" }
- error 1: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot1: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "5794e863-abf8-4b62-9b88-4db47fa1157d", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image does not include a caboose", request_id: "5794e863-abf8-4b62-9b88-4db47fa1157d" }
- error 2: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose RotSlotB: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "3f642d6f-5c1a-4783-aac0-0181f4b8417d", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: failed to read data from the caboose", request_id: "3f642d6f-5c1a-4783-aac0-0181f4b8417d" }
- error 3: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "b5844cae-774d-4a66-9d38-8c09236c6c00", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "b5844cae-774d-4a66-9d38-8c09236c6c00" }
- error 4: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0Next: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "2f3caf74-897e-4191-9be8-2e5ca6f1d968", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "2f3caf74-897e-4191-9be8-2e5ca6f1d968" }
- error 5: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page Cmpa: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "5a299aa8-174b-45e9-94db-cd70a5c64ffd", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "5a299aa8-174b-45e9-94db-cd70a5c64ffd" }
- error 6: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaActive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "c65cff3e-b62a-4c34-9392-d6303c050674", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "c65cff3e-b62a-4c34-9392-d6303c050674" }
- error 7: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaInactive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "3b9a48d0-72bd-478c-8779-2fd14f98c5c9", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "3b9a48d0-72bd-478c-8779-2fd14f98c5c9" }
- error 8: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaScratch: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "f4c063e2-b8a4-47ea-a58d-c4ee60e2a207", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:04 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "f4c063e2-b8a4-47ea-a58d-c4ee60e2a207" }
- error 9: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "cc9b63b4-32ec-4944-9f2a-850e2f47c854", "content-length": "247", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image caboose does not contain 'VERS'", request_id: "cc9b63b4-32ec-4944-9f2a-850e2f47c854" }
- error 10: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot1: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "dde7a768-f004-4d93-b761-60a68c3b46f8", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image does not include a caboose", request_id: "dde7a768-f004-4d93-b761-60a68c3b46f8" }
- error 11: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose RotSlotB: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "fc373a2e-0136-4f4a-b2e9-b6d4eda5d5df", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: failed to read data from the caboose", request_id: "fc373a2e-0136-4f4a-b2e9-b6d4eda5d5df" }
- error 12: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "46b5a3d2-7ffc-4fc5-9584-f0bc9fe4b3d8", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "46b5a3d2-7ffc-4fc5-9584-f0bc9fe4b3d8" }
- error 13: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0Next: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "1b300061-35af-42ef-9337-17b591349c45", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "1b300061-35af-42ef-9337-17b591349c45" }
- error 14: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page Cmpa: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "234e56a5-63d3-4c72-a82a-599fa9df53b1", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "234e56a5-63d3-4c72-a82a-599fa9df53b1" }
- error 15: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaActive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "5456ac62-6968-450a-a93f-ffffda217791", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "5456ac62-6968-450a-a93f-ffffda217791" }
- error 16: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaInactive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "a4b9f693-92c7-4ba4-a4b1-bf1c1f72b091", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "a4b9f693-92c7-4ba4-a4b1-bf1c1f72b091" }
- error 17: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaScratch: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "51023fa0-1280-4917-bd6c-2d8601ab5b8a", "content-length": "242", "date": "Mon, 30 Jun 2025 17:44:07 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "51023fa0-1280-4917-bd6c-2d8601ab5b8a" }
+ error 0: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "d3e8c89c-6307-412c-a92a-049ca0648f62", "content-length": "247", "date": "Mon, 30 Jun 2025 18:35:26 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image caboose does not contain 'VERS'", request_id: "d3e8c89c-6307-412c-a92a-049ca0648f62" }
+ error 1: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot1: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "7516b6ea-a7ec-41e4-adee-09889260a535", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:26 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image does not include a caboose", request_id: "7516b6ea-a7ec-41e4-adee-09889260a535" }
+ error 2: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose RotSlotB: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "e498cd6c-4815-4572-8f7e-6f5b7067c1dd", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:26 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: failed to read data from the caboose", request_id: "e498cd6c-4815-4572-8f7e-6f5b7067c1dd" }
+ error 3: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "ad211d92-2f9d-4e37-8a8c-3689816f9a41", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:26 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "ad211d92-2f9d-4e37-8a8c-3689816f9a41" }
+ error 4: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0Next: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "945d2d12-8171-4e15-bfcf-15117af7204b", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:26 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "945d2d12-8171-4e15-bfcf-15117af7204b" }
+ error 5: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page Cmpa: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "01585241-6833-4738-9e8b-8f820cd0732e", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:26 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "01585241-6833-4738-9e8b-8f820cd0732e" }
+ error 6: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaActive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "72940416-c83a-4988-966a-dcb0e2078bea", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:26 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "72940416-c83a-4988-966a-dcb0e2078bea" }
+ error 7: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaInactive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "fe893e56-6017-435d-aa6b-051e6ff37651", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:26 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "fe893e56-6017-435d-aa6b-051e6ff37651" }
+ error 8: MGS "http://[fd00:1122:3344:103::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaScratch: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "e6679193-43a3-49c3-86be-de7124720e7c", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:26 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "e6679193-43a3-49c3-86be-de7124720e7c" }
+ error 9: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "1eea416f-812a-43f5-aec6-14cfc4e26709", "content-length": "247", "date": "Mon, 30 Jun 2025 18:35:28 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image caboose does not contain 'VERS'", request_id: "1eea416f-812a-43f5-aec6-14cfc4e26709" }
+ error 10: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose SpSlot1: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "05d865d2-e1dd-4298-9ce7-c226c94b4401", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:28 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: the image does not include a caboose", request_id: "05d865d2-e1dd-4298-9ce7-c226c94b4401" }
+ error 11: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose RotSlotB: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "d570df1d-9387-4719-ac2b-777315c0bbfc", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:28 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: failed to read data from the caboose", request_id: "d570df1d-9387-4719-ac2b-777315c0bbfc" }
+ error 12: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "0d796f00-75d4-45a3-82e5-db73609eb7bc", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:28 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "0d796f00-75d4-45a3-82e5-db73609eb7bc" }
+ error 13: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: caboose Stage0Next: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "b4d30918-c2d6-4677-af25-2ade56100abb", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:28 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "b4d30918-c2d6-4677-af25-2ade56100abb" }
+ error 14: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page Cmpa: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "1cc645cb-eb67-4730-abda-8f478f4ae3b2", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:28 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "1cc645cb-eb67-4730-abda-8f478f4ae3b2" }
+ error 15: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaActive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "41616f37-6e54-4d3f-a26f-6fb461c843c6", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:28 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "41616f37-6e54-4d3f-a26f-6fb461c843c6" }
+ error 16: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaInactive: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "b300ced3-e99a-4ce3-bdf8-80f22cb8f705", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:28 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "b300ced3-e99a-4ce3-bdf8-80f22cb8f705" }
+ error 17: MGS "http://[fd00:1122:3344:101::2]:12225": SP SpIdentifier { slot: 0, type_: Power }: rot page CfpaScratch: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "73e65af8-ed56-4b53-8077-fd0172455028", "content-length": "242", "date": "Mon, 30 Jun 2025 18:35:28 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Power, slot: 0 }: Error response from SP: sprot: failed to deserialize message", request_id: "73e65af8-ed56-4b53-8077-fd0172455028" }
Sled BRM42220026
part number: 913-0000019
power: A0
revision: 6
MGS slot: Sled 14 (cubby 14)
- found at: 2025-06-30 17:43:59.682939 UTC from http://[fd00:1122:3344:103::2]:12225
+ found at: 2025-06-30 18:35:22.958370 UTC from http://[fd00:1122:3344:103::2]:12225
cabooses:
SLOT BOARD NAME VERSION GIT_COMMIT SIGN
- SpSlot0 gimlet-c gimlet-c 1.0.34 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
- SpSlot1 gimlet-c gimlet-c 1.0.38 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
+ SpSlot0 gimlet-c gimlet-c 1.0.38 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
+ SpSlot1 gimlet-c gimlet-c 1.0.34 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
RotSlotA oxide-rot-1 oxide-rot-1 1.0.35 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
RotSlotB oxide-rot-1 oxide-rot-1 1.0.35 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
Stage0 oxide-rot-1 oxide-rot-1 1.4.0 bdf56dd950b934360df596ed5b2d8b8813c92168 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
Stage0Next oxide-rot-1 oxide-rot-1 1.4.0 bdf56dd950b934360df596ed5b2d8b8813c92168 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
RoT pages:
@@ -52,15 +52,15 @@
Sled BRM27230037
part number: 913-0000019
power: A0
revision: 13
MGS slot: Sled 15 (cubby 15)
- found at: 2025-06-30 17:44:03.555192 UTC from http://[fd00:1122:3344:103::2]:12225
+ found at: 2025-06-30 18:35:25.226216 UTC from http://[fd00:1122:3344:103::2]:12225
cabooses:
SLOT BOARD NAME VERSION GIT_COMMIT SIGN
- SpSlot0 gimlet-e gimlet-e 1.0.34 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
- SpSlot1 gimlet-e gimlet-e 1.0.38 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
+ SpSlot0 gimlet-e gimlet-e 1.0.38 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
+ SpSlot1 gimlet-e gimlet-e 1.0.34 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
RotSlotA oxide-rot-1 oxide-rot-1 1.0.35 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
RotSlotB oxide-rot-1 oxide-rot-1 1.0.35 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
Stage0 oxide-rot-1 oxide-rot-1 1.4.0 bdf56dd950b934360df596ed5b2d8b8813c92168 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
Stage0Next oxide-rot-1 oxide-rot-1 1.4.0 bdf56dd950b934360df596ed5b2d8b8813c92168 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
RoT pages:
@@ -79,11 +79,11 @@
Sled BRM23230018
part number: 913-0000019
power: A0
revision: 11
MGS slot: Sled 16 (cubby 16)
- found at: 2025-06-30 17:43:58.301368 UTC from http://[fd00:1122:3344:103::2]:12225
+ found at: 2025-06-30 18:35:20.985590 UTC from http://[fd00:1122:3344:103::2]:12225
cabooses:
SLOT BOARD NAME VERSION GIT_COMMIT SIGN
SpSlot0 gimlet-d gimlet-d 1.0.38 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
SpSlot1 gimlet-d gimlet-d 1.0.34 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
RotSlotA oxide-rot-1 oxide-rot-1 1.0.35 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
@@ -106,15 +106,15 @@
Sled BRM23230010
part number: 913-0000019
power: A0
revision: 11
MGS slot: Sled 17 (cubby 17)
- found at: 2025-06-30 17:44:02.423975 UTC from http://[fd00:1122:3344:103::2]:12225
+ found at: 2025-06-30 18:35:24.097977 UTC from http://[fd00:1122:3344:103::2]:12225
cabooses:
SLOT BOARD NAME VERSION GIT_COMMIT SIGN
- SpSlot0 gimlet-d gimlet-d 1.0.34 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
- SpSlot1 gimlet-d gimlet-d 1.0.38 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
+ SpSlot0 gimlet-d gimlet-d 1.0.38 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
+ SpSlot1 gimlet-d gimlet-d 1.0.34 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
RotSlotA oxide-rot-1 oxide-rot-1 1.0.35 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
RotSlotB oxide-rot-1 oxide-rot-1 1.0.35 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
Stage0 oxide-rot-1 oxide-rot-1 1.4.0 bdf56dd950b934360df596ed5b2d8b8813c92168 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
Stage0Next oxide-rot-1 oxide-rot-1 1.4.0 bdf56dd950b934360df596ed5b2d8b8813c92168 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
RoT pages:
@@ -133,11 +133,11 @@
Power BRM11230018
part number: 913-0000003
power: A2
revision: 9
MGS slot: Power 0
- found at: 2025-06-30 17:44:04.708406 UTC from http://[fd00:1122:3344:103::2]:12225
+ found at: 2025-06-30 18:35:26.374021 UTC from http://[fd00:1122:3344:103::2]:12225
cabooses:
SLOT BOARD NAME VERSION GIT_COMMIT SIGN
RotSlotA oxide-rot-1 oxide-rot-1 0.0.0-git 886df1648f6eeeb587181fc82cd82a3760599cf0 n/a
RoT pages:
SLOT DATA_BASE64
@@ -151,15 +151,15 @@
Switch BRM23230002
part number: 913-0000006
power: A0
revision: 10
MGS slot: Switch 0
- found at: 2025-06-30 17:44:05.063202 UTC from http://[fd00:1122:3344:103::2]:12225
+ found at: 2025-06-30 18:35:26.714343 UTC from http://[fd00:1122:3344:103::2]:12225
cabooses:
SLOT BOARD NAME VERSION GIT_COMMIT SIGN
- SpSlot0 sidecar-c sidecar-c 1.0.35 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
- SpSlot1 sidecar-c sidecar-c 1.0.38 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab n/a
+ SpSlot0 sidecar-c sidecar-c 1.0.39 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
+ SpSlot1 sidecar-c sidecar-c 1.0.35 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
RotSlotA oxide-rot-1 oxide-rot-1 1.0.35 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27
RotSlotB oxide-rot-1 oxide-rot-1 1.0.35 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27
Stage0 oxide-rot-1 oxide-rot-1 1.4.0 bdf56dd950b934360df596ed5b2d8b8813c92168 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27
Stage0Next oxide-rot-1 oxide-rot-1 1.4.0 bdf56dd950b934360df596ed5b2d8b8813c92168 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27
RoT pages:
@@ -178,15 +178,15 @@
Switch BRM31230002
part number: 913-0000006
power: A0
revision: 10
MGS slot: Switch 1
- found at: 2025-06-30 17:44:06.210617 UTC from http://[fd00:1122:3344:103::2]:12225
+ found at: 2025-06-30 18:35:27.974470 UTC from http://[fd00:1122:3344:103::2]:12225
cabooses:
SLOT BOARD NAME VERSION GIT_COMMIT SIGN
- SpSlot0 sidecar-c sidecar-c 1.0.35 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
- SpSlot1 sidecar-c sidecar-c 1.0.38 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab n/a
+ SpSlot0 sidecar-c sidecar-c 1.0.39 35bd6493ede62be7751ba0c9c8f9ff9f25b8c5c0 n/a
+ SpSlot1 sidecar-c sidecar-c 1.0.35 441b09270d8715275d6b3c3b57d84dba539e34d0 n/a
RotSlotA oxide-rot-1 oxide-rot-1 1.0.35 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27
RotSlotB oxide-rot-1 oxide-rot-1 1.0.35 6edf9b5e6aa5c928a5462bda1f7a4c6f3caa40ab 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27
Stage0 oxide-rot-1 oxide-rot-1 1.4.0 bdf56dd950b934360df596ed5b2d8b8813c92168 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27
Stage0Next oxide-rot-1 oxide-rot-1 1.4.0 bdf56dd950b934360df596ed5b2d8b8813c92168 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27
RoT pages: Here's the current history of blueprints: I'm going to paste the output of The noise I'm talking about is that we have a bunch of blueprints in quick succession that alter the existing |
Collaborator
Author
|
Note the zones part of the update got stuck, but I think this is expected: when the zones hosting Nexus zones restarted as part of the SP update, the config change I made to enable the autoplanner got undone (expected). It got stuck at the point where it had expunged and re-added the zone on sled 16. That one's new and also had the autoplanner disabled. I've re-enabled the autoplanner and it marched along to the end: |
This was referenced Jun 30, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I'd like to implement support for RoT updates too to make sure we're not missing something important that would require an annoying database migration to get past. But otherwise I believe this is ready for review.
Depends on #8394.