Skip to content

[wip] basically fix all the sequencer IRQ handling#2390

Draft
hawkw wants to merge 16 commits intoeliza/namespace-fmc-regsfrom
eliza/mapo-irq
Draft

[wip] basically fix all the sequencer IRQ handling#2390
hawkw wants to merge 16 commits intoeliza/namespace-fmc-regsfrom
eliza/mapo-irq

Conversation

@hawkw
Copy link
Member

@hawkw hawkw commented Feb 14, 2026 •
edited
Loading

Depends on #2392

@hawkw
Copy link
Member Author

hawkw commented Feb 14, 2026 •
edited
Loading

whew okay i think i this Cosmo just did everything i wanted it to do and nothing i didn't want it to do!

eliza@alfred ~ $ pfexec humility pmbus -r V12_SYS_A2 -w VOUT_COMMAND=8 \
  && pfexec humility pmbus -r V12_SYS_A2 -w VOUT_COMMAND=12 \
  && pfexec humility ringbuf cosmo_seq \
  && pfexec humility pmbus -s
humility: WARNING: archive in environment variable overriding archive in environment file
humility: attached to 1fc9:0143:XJACFXSEKTQJS via CMSIS-DAP
humility: I2C4, port F, dev 0x67: successfully wrote VOUT_COMMAND
humility: WARNING: archive in environment variable overriding archive in environment file
humility: attached to 1fc9:0143:XJACFXSEKTQJS via CMSIS-DAP
humility: I2C4, port F, dev 0x67: successfully wrote VOUT_COMMAND
humility: WARNING: archive in environment variable overriding archive in environment file
humility: attached to 1fc9:0143:XJACFXSEKTQJS via CMSIS-DAP
humility: ring buffer drv_cosmo_seq_server::__RINGBUF in cosmo_seq:
   TOTAL VARIANT
   14105 RegStateValues
     534 ContinueBitstreamLoad
     112 CPUPresent
       3 SetState(InitialPowerOn)
       2 PmbusAlert
       2 EreportSent
       1 FpgaInit
       1 WaitForDone
       1 Programmed
       1 Startup
       1 SequencerInterrupt
       1 Coretype
       1 ResetCounts
       1 NicMapoInterrupt
 NDX LINE      GEN    COUNT PAYLOAD
  76  601        5        1 CPUPresent(true)
  77  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  78  601        5        1 CPUPresent(true)
  79  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  80  601        5        1 CPUPresent(true)
  81  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  82  601        5        1 CPUPresent(true)
  83  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  84  601        5        1 CPUPresent(true)
  85  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  86  601        5        1 CPUPresent(true)
  87  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  88  601        5        1 CPUPresent(true)
  89  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  90  601        5        1 CPUPresent(true)
  91  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  92  601        5        1 CPUPresent(true)
  93  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  94  601        5        1 CPUPresent(true)
  95  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  96  601        5        1 CPUPresent(true)
  97  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  98  601        5        1 CPUPresent(true)
  99  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
 100  601        5        1 CPUPresent(true)
 101  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
 102  601        5        1 CPUPresent(true)
 103  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
 104  601        5        1 CPUPresent(true)
 105  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
 106  601        5        1 CPUPresent(true)
 107  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
 108  601        5        1 CPUPresent(true)
 109  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
 110  601        5        1 CPUPresent(true)
 111  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
 112  601        5        1 CPUPresent(true)
 113  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
 114  601        5        1 CPUPresent(true)
 115  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
 116  601        5        1 CPUPresent(true)
 117  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
 118  601        5        1 CPUPresent(true)
 119  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
 120  601        5        1 CPUPresent(true)
 121  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
 122  601        5        1 CPUPresent(true)
 123  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
 124  601        5        1 CPUPresent(true)
 125  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
 126  601        5        1 CPUPresent(true)
 127  530        5        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
   0  601        6        1 CPUPresent(true)
   1  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
   2  601        6        1 CPUPresent(true)
   3  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
   4  601        6        1 CPUPresent(true)
   5  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
   6  601        6        1 CPUPresent(true)
   7  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
   8  601        6        1 CPUPresent(true)
   9  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  10  601        6        1 CPUPresent(true)
  11  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  12  601        6        1 CPUPresent(true)
  13  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  14  601        6        1 CPUPresent(true)
  15  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(GroupAPgAndWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  16  601        6        1 CPUPresent(true)
  17  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  18  601        6        1 CPUPresent(true)
  19  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  20  601        6        1 CPUPresent(true)
  21  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  22  601        6        1 CPUPresent(true)
  23  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  24  601        6        1 CPUPresent(true)
  25  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  26  601        6        1 CPUPresent(true)
  27  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  28  601        6        1 CPUPresent(true)
  29  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  30  601        6        1 CPUPresent(true)
  31  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  32  601        6        1 CPUPresent(true)
  33  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  34  601        6        1 CPUPresent(true)
  35  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  36  601        6        1 CPUPresent(true)
  37  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  38  601        6        1 CPUPresent(true)
  39  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  40  601        6        1 CPUPresent(true)
  41  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  42  601        6        1 CPUPresent(true)
  43  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  44  601        6        1 CPUPresent(true)
  45  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  46  601        6        1 CPUPresent(true)
  47  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  48  601        6        1 CPUPresent(true)
  49  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  50  601        6        1 CPUPresent(true)
  51  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  52  601        6        1 CPUPresent(true)
  53  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  54  601        6        1 CPUPresent(true)
  55  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(EnableGrpA) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(RtcClkWait) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  56  601        6        1 CPUPresent(true)
  57  530        6        2 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(Sp5FinalCheckpoint) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(WaitPwrok) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
  58  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(Done) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(Done) }, nic_api_status: NicApiStatusView { nic_sm: Ok(EnablePower) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(WaitForPgs) } }
  59  632        6        1 Coretype { coretype0: true, coretype1: false, coretype2: true, sp5r1: true, sp5r2: false, sp5r3: false, sp5r4: false }
  60  530        6    13232 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(Done) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(Done) }, nic_api_status: NicApiStatusView { nic_sm: Ok(NicReset) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Done) } }
  61  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(Done) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(Done) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Done) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Done) } }
  62  556        6        1 SetState { prev: Some(A0), next: A0PlusHP, why: InitialPowerOn, now: 0x2272b }
  63  530        6       88 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(Done) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(Done) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Done) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Done) } }
  64  530        6        1 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(Done) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(Done) }, nic_api_status: NicApiStatusView { nic_sm: Ok(NicReset) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Done) } }
  65  530        6        2 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(Done) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(Done) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Done) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Done) } }
  66 1059        6        1 SequencerInterrupt { our_state: A0PlusHP, seq_state: Ok(Done) }
  67  830        6        1 SequencerIfr(IfrView { fanfault: false, thermtrip: false, smerr_assert: false, a0mapo: false, nicmapo: false, amd_pwrok_fedge: false, amd_rstn_fedge: false, fan_central_hsc_alert: false, fan_east_hsc_alert: false, fan_west_hsc_alert: false, ibc_alert: false, m2_hsc_alert: false, nic_hsc_alert: false, v12_ddr5_abcdef_hsc_alert: false, v12_ddr5_ghijkl_hsc_alert: false, v12_mcio_a0hp_hsc_alert: false, main_hsc_alert: false, vr_v1p8_sys_to_fpga1_alert: true, vr_v3p3_sys_to_fpga1_alert: true, vr_v5p0_sys_to_fpga1_alert: true, v0p96_nic_to_fpga1_alert: true, pwr_cont1_to_fpga1_alert: true, pwr_cont2_to_fpga1_alert: false, pwr_cont3_to_fpga1_alert: false })
  68  878        6        1 PmbusAlert { now: 0x38a11 }
  69 1134        6        1 EreportSent(0x87)
  70 1134        6        1 EreportSent(0x88)
  71  830        6        1 SequencerIfr(IfrView { fanfault: false, thermtrip: false, smerr_assert: false, a0mapo: false, nicmapo: true, amd_pwrok_fedge: false, amd_rstn_fedge: true, fan_central_hsc_alert: false, fan_east_hsc_alert: false, fan_west_hsc_alert: false, ibc_alert: false, m2_hsc_alert: false, nic_hsc_alert: false, v12_ddr5_abcdef_hsc_alert: false, v12_ddr5_ghijkl_hsc_alert: false, v12_mcio_a0hp_hsc_alert: false, main_hsc_alert: false, vr_v1p8_sys_to_fpga1_alert: true, vr_v3p3_sys_to_fpga1_alert: true, vr_v5p0_sys_to_fpga1_alert: true, v0p96_nic_to_fpga1_alert: true, pwr_cont1_to_fpga1_alert: true, pwr_cont2_to_fpga1_alert: true, pwr_cont3_to_fpga1_alert: false })
  72  878        6        1 PmbusAlert { now: 0x38a47 }
  73  910        6        1 ResetCounts { rstn: 0x1, pwrokn: 0x0 }
  74  916        6        1 NicMapoInterrupt
  75  530        6      666 RegStateValues { seq_api_status: SeqApiStatusView { a0_sm: Ok(Done) }, seq_raw_status: SeqRawStatusView { hw_sm: Ok(Done) }, nic_api_status: NicApiStatusView { nic_sm: Ok(Idle) }, nic_raw_status: NicRawStatusView { hw_sm: Ok(Idle) } }
humility: ring buffer drv_cosmo_seq_server::vcore::__RINGBUF in cosmo_seq:
 NDX LINE      GEN    COUNT PAYLOAD
   0  150        1        1 Initializing
   1  159        1        1 LimitsLoaded
   2  205        1        1 StatusWords { vddcr_cpu0: Ok(0x0), vddcr_cpu1: Ok(0x0) }
   3  170        1        1 Initialized
   4  220        1        1 PmbusAlert { timestamp: 0x38a11, vrms: Vrms { pwr_cont1: true, pwr_cont2: false } }
   5  330        1        1 StatusWord(VddcrCpu0, Ok(0x2001))
   6  370        1        1 RegulatorStatus { rail: VddcrCpu0, power_good: true, faulted: true }
   7  393        1        1 StatusInput(VddcrCpu0, Ok(0x20))
   8  405        1        1 StatusVout(VddcrCpu0, Ok(0x0))
   9  409        1        1 StatusIout(VddcrCpu0, Ok(0x0))
  10  414        1        1 StatusTemperature(VddcrCpu0, Ok(0x0))
  11  418        1        1 StatusCml(VddcrCpu0, Ok(0x0))
  12  423        1        1 StatusMfrSpecific(VddcrCpu0, Ok(0x0))
  13  330        1        1 StatusWord(VddcrCpu1, Ok(0x2001))
  14  370        1        1 RegulatorStatus { rail: VddcrCpu1, power_good: true, faulted: true }
  15  393        1        1 StatusInput(VddcrCpu1, Ok(0x20))
  16  405        1        1 StatusVout(VddcrCpu1, Ok(0x0))
  17  409        1        1 StatusIout(VddcrCpu1, Ok(0x0))
  18  414        1        1 StatusTemperature(VddcrCpu1, Ok(0x0))
  19  418        1        1 StatusCml(VddcrCpu1, Ok(0x0))
  20  423        1        1 StatusMfrSpecific(VddcrCpu1, Ok(0x0))
  21  285        1        1 Reading { timestamp: 0x38a1d, vddcr_cpu0_vin: Volts(9.97), vddcr_cpu1_vin: Volts(9.960001) }
  22  285        1        1 Reading { timestamp: 0x38a1e, vddcr_cpu0_vin: Volts(9.830001), vddcr_cpu1_vin: Volts(9.81) }
  23  285        1        1 Reading { timestamp: 0x38a20, vddcr_cpu0_vin: Volts(9.670001), vddcr_cpu1_vin: Volts(9.650001) }
  24  285        1        1 Reading { timestamp: 0x38a22, vddcr_cpu0_vin: Volts(9.540001), vddcr_cpu1_vin: Volts(9.490001) }
  25  285        1        1 Reading { timestamp: 0x38a23, vddcr_cpu0_vin: Volts(9.39), vddcr_cpu1_vin: Volts(9.340001) }
  26  285        1        1 Reading { timestamp: 0x38a25, vddcr_cpu0_vin: Volts(9.22), vddcr_cpu1_vin: Volts(9.190001) }
  27  285        1        1 Reading { timestamp: 0x38a27, vddcr_cpu0_vin: Volts(9.080001), vddcr_cpu1_vin: Volts(9.030001) }
  28  285        1        1 Reading { timestamp: 0x38a28, vddcr_cpu0_vin: Volts(8.920001), vddcr_cpu1_vin: Volts(8.900001) }
  29  285        1        1 Reading { timestamp: 0x38a2a, vddcr_cpu0_vin: Volts(8.790001), vddcr_cpu1_vin: Volts(8.740001) }
  30  285        1        1 Reading { timestamp: 0x38a2c, vddcr_cpu0_vin: Volts(8.630001), vddcr_cpu1_vin: Volts(8.6) }
  31  285        1        1 Reading { timestamp: 0x38a2d, vddcr_cpu0_vin: Volts(8.4800005), vddcr_cpu1_vin: Volts(8.450001) }
  32  285        1        1 Reading { timestamp: 0x38a2f, vddcr_cpu0_vin: Volts(8.320001), vddcr_cpu1_vin: Volts(8.290001) }
  33  285        1        1 Reading { timestamp: 0x38a30, vddcr_cpu0_vin: Volts(8.160001), vddcr_cpu1_vin: Volts(8.13) }
  34  285        1        1 Reading { timestamp: 0x38a32, vddcr_cpu0_vin: Volts(8.040001), vddcr_cpu1_vin: Volts(8.02) }
  35  285        1        1 Reading { timestamp: 0x38a34, vddcr_cpu0_vin: Volts(7.9900007), vddcr_cpu1_vin: Volts(8.01) }
  36  285        1        1 Reading { timestamp: 0x38a35, vddcr_cpu0_vin: Volts(8.000001), vddcr_cpu1_vin: Volts(8.01) }
  37  285        1        1 Reading { timestamp: 0x38a37, vddcr_cpu0_vin: Volts(7.9900007), vddcr_cpu1_vin: Volts(8.01) }
  38  285        1        1 Reading { timestamp: 0x38a39, vddcr_cpu0_vin: Volts(7.9900007), vddcr_cpu1_vin: Volts(8.000001) }
  39  285        1        1 Reading { timestamp: 0x38a3a, vddcr_cpu0_vin: Volts(7.9900007), vddcr_cpu1_vin: Volts(8.01) }
  40  285        1        1 Reading { timestamp: 0x38a3c, vddcr_cpu0_vin: Volts(7.9900007), vddcr_cpu1_vin: Volts(8.01) }
  41  285        1        1 Reading { timestamp: 0x38a3e, vddcr_cpu0_vin: Volts(7.9800005), vddcr_cpu1_vin: Volts(8.02) }
  42  285        1        1 Reading { timestamp: 0x38a3f, vddcr_cpu0_vin: Volts(7.9800005), vddcr_cpu1_vin: Volts(8.02) }
  43  285        1        1 Reading { timestamp: 0x38a41, vddcr_cpu0_vin: Volts(7.9800005), vddcr_cpu1_vin: Volts(8.02) }
  44  285        1        1 Reading { timestamp: 0x38a43, vddcr_cpu0_vin: Volts(7.9900007), vddcr_cpu1_vin: Volts(8.02) }
  45  285        1        1 Reading { timestamp: 0x38a44, vddcr_cpu0_vin: Volts(7.9800005), vddcr_cpu1_vin: Volts(8.030001) }
  46  205        1        1 StatusWords { vddcr_cpu0: Ok(0x2001), vddcr_cpu1: Ok(0x2001) }
  47  220        1        1 PmbusAlert { timestamp: 0x38a47, vrms: Vrms { pwr_cont1: true, pwr_cont2: true } }
  48  205        1      297 StatusWords { vddcr_cpu0: Ok(0x2001), vddcr_cpu1: Ok(0x2001) }
  49  205        1        1 StatusWords { vddcr_cpu0: Ok(0x2001), vddcr_cpu1: Ok(0x0) }
  50  205        1        1 StatusWords { vddcr_cpu0: Ok(0x0), vddcr_cpu1: Ok(0x0) }
humility: ring buffer drv_i2c_devices::bmr491::__RINGBUF in cosmo_seq:
 NDX LINE      GEN    COUNT PAYLOAD
   0  186        1        1 MitigationApplied(AlreadyApplied)
humility: ring buffer drv_oxide_vpd::__RINGBUF in cosmo_seq:
humility: ring buffer drv_packrat_vpd_loader::__RINGBUF in cosmo_seq:
humility: WARNING: archive in environment variable overriding archive in environment file
humility: attached to 1fc9:0143:XJACFXSEKTQJS via CMSIS-DAP
DEVICE      RAIL               PG? #FLT       VIN      VOUT      IOUT    TEMP_1
tps546b24a  V3P3_SP_A2           Y    2   11.984V    3.309V    0.421A  38.000°C
tps546b24a  V5_SYS_A2            Y    2   12.000V    4.986V    1.229A  35.250°C
tps546b24a  V1P8_SYS_A2          Y    2   12.000V    1.795V   -0.453A  31.750°C
raa229620a  VDDCR_CPU0_A0        Y    0   11.970V    0.889V   26.900A  37.000°C
raa229620a  VDDCR_SOC_A0         Y    0   11.980V    0.900V   25.200A  37.000°C
raa229620a  VDDCR_CPU1_A0        Y    0   12.050V    0.890V   25.100A  40.000°C
raa229620a  VDDIO_SP5_A0         Y    0   12.040V    1.100V    1.100A  38.000°C
isl68224    V1P1_SP5_A0          Y    0   11.970V    1.099V    5.700A  34.000°C
isl68224    V1P8_SP5_A1          Y    0   11.960V    1.800V    1.500A  31.000°C
isl68224    V3P3_SP5_A1          Y    0   11.970V    2.750V    0.100A  33.000°C
tps546b24a  V0P96_NIC_VDD_A0HP   N    1    0.000V    0.012V   -0.129A  30.250°C
bmr491      V12_SYS_A2           Y    0   48.000V   11.997V   12.750A  36.750°C
lm5066i     V54P5_FAN_EAST       Y    0   48.178V   48.218V         -  31.125°C
lm5066i     V54P5_FAN_CENTRAL    Y    0   48.092V   48.110V         -  29.250°C
lm5066i     V54P5_FAN_WEST       Y    0   48.070V   48.131V         -  27.188°C
adm127x     V54P5_IBC_A3       --  error: can't read VOUT_MODE: NoRegister  --
eliza@alfred ~ $

@hawkw
Copy link
Member Author

hawkw commented Feb 14, 2026 •
edited
Loading

i regret to inform you that

eliza@alfred ~ $ pfexec humility ringbuf thermal
humility: WARNING: archive in environment variable overriding archive in environment file
humility: attached to 1fc9:0143:XJACFXSEKTQJS via CMSIS-DAP
humility: ring buffer drv_i2c_devices::emc2305::__RINGBUF in thermal:
humility: ring buffer drv_i2c_devices::max31790::__RINGBUF in thermal:
humility: ring buffer task_thermal::__RINGBUF in thermal:
   TOTAL VARIANT
    1641 ControlPwm
      99 SensorReadFailed
       3 AutoState(Boot)
       2 AutoState(Running)
       1 AutoState(Critical)
       1 AutoState(Uncontrollable)
       6 FanAdded
       3 MiscReadFailed
       2 PowerModeChanged
       1 Start
       1 ThermalMode(Auto)
       1 PowerDownDueTo
       1 CriticalDueTo
       1 FanControllerInitialized
       1 PowerDownAt
       1 SetFanWatchdogOk
 NDX LINE      GEN    COUNT PAYLOAD
   3 1031        7        1 SensorReadFailed(SensorId(0x38), I2cError(NoDevice))
   4 1401        7        1 CriticalDueTo { sensor_id: SensorId(0x38), temperature: Celsius(80.326996) }
   5 1409        7        1 AutoState(Critical)
   6 1352        7        1 ControlPwm(0x64)
   7 1031        7        1 SensorReadFailed(SensorId(0x38), I2cError(NoDevice))
   8 1352        7        1 ControlPwm(0x64)
   9 1031        7        1 SensorReadFailed(SensorId(0x38), I2cError(NoDevice))
  10 1352        7        1 ControlPwm(0x64)
  11 1031        7        1 SensorReadFailed(SensorId(0x38), I2cError(NoDevice))
  12 1352        7        1 ControlPwm(0x64)
  13 1031        7        1 SensorReadFailed(SensorId(0x38), I2cError(NoDevice))
  14 1352        7        1 ControlPwm(0x64)
  15 1031        7        1 SensorReadFailed(SensorId(0x38), I2cError(NoDevice))
  16 1352        7        1 ControlPwm(0x64)
  17 1031        7        1 SensorReadFailed(SensorId(0x38), I2cError(NoDevice))
  18 1352        7        1 ControlPwm(0x64)
  19 1031        7        1 SensorReadFailed(SensorId(0x38), I2cError(NoDevice))
  20 1352        7        1 ControlPwm(0x64)
  21 1031        7        1 SensorReadFailed(SensorId(0x38), I2cError(NoDevice))
  22 1352        7        1 ControlPwm(0x64)
  23 1031        7        1 SensorReadFailed(SensorId(0x38), I2cError(NoDevice))
  24 1352        7        1 ControlPwm(0x64)
  25 1031        7        1 SensorReadFailed(SensorId(0x38), I2cError(NoDevice))
  26 1276        7        1 PowerDownDueTo { sensor_id: SensorId(0x38), temperature: Celsius(85.326996) }
  27 1478        7        1 OverheatedFor(0x2710)
  28 1481        7        1 CriticalFor(0x2710)
  29 1444        7        1 AutoState(Uncontrollable)
  30 1356        7        1 PowerDownAt(0x504a3)
  31 1100        7        1 PowerModeChanged(PowerBitmask(0b1))
   0  924        8        1 AutoState(Boot)
   1 1388        8        1 AutoState(Running)
   2 1352        8     1320 ControlPwm(0x0)
eliza@alfred ~ $

...the sequencer task taking us from A0+HP back to A0 (normal) does not make the thermal loop Stop Doing That.

so this fixes, uh, everything except the actual bug (#2384)

@hawkw hawkw mentioned this pull request Feb 14, 2026
Open
hawkw added a commit that referenced this pull request Feb 14, 2026
@hawkw
thermal: only poll T6 temperatures in A0+HP
cc63c6d 
Presently, the thermal loop on Gimlet and Cosmo will attempt to read the
T6 temperature sensors whilst in the A0 *or* A0+HP power states. This is
wrong, as the NIC is only known to be powered in the A0+HP state (since
its sequencing is controlled by the host OS).

This means that if the T6 MAPOs but the host processor is still up, the
thermal loop will continue trying to read temperatures from the T6
whilst it is not powered. Because it's not powered, we will just read
I2C NACKs from it. This causes the logic in the thermal loop that
attempts to extrapolate a worst-case temperature increase for devices
which have transient sensor read errors to imagine a T6 temperature
which continues to increase, eventually leading to a thermal shutdown.
This is described in #2384.

This commit adds a bit in the Gimlet and Cosmo thermal tasks'
`PowerBitmask`s to differentiate between the A0 and A0+HP power states,
and only consider the T6 temperature if the A0+HP bit is set. This will
fix #2384...alongside #2390, which is necessary to fix Cosmo ignoring
the sequencer IRQ that would tell it to transition from A0+HP back to A0
if the NIC MAPOs.
@hawkw hawkw mentioned this pull request Feb 14, 2026
Open
@hawkw
Copy link
Member Author

hawkw commented Feb 14, 2026 •
edited
Loading

@rmustacc informs me that this may actually be Cosmo-only as T6's temperature sensor may be powered in the A0 domain on Gimlet --- if that's the case, we can probably back this out on Gimlet, as it would just add an additional reset of some of the thermal loop parameters for no real reason.

EDIT: er, this comment was meant to be left on #2391, which is why it doesn't really make sense here. The Gimlet handling of the RAA229618 PMBus alert is also wrong and the subset of this change that applies to gimlet is indeed valid.

@hawkw
Copy link
Member Author

hawkw commented Feb 15, 2026

I think that I shall pull the FMC peripheral codegen changes out into its own PR so that that can be reviewed separately.

@hawkw hawkw mentioned this pull request Feb 15, 2026
Open
@hawkw hawkw changed the base branch from master to eliza/namespace-fmc-regs February 15, 2026 17:57
@hawkw hawkw mentioned this pull request Feb 15, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@hawkw @nathanaelhuffman