Sudden loss of control, upon resume Controller Ports were shuffled

[jlmcmchl]

At the beginning of Stemley, our robot was behaving normally. We play Quals 2 without issue.

In Quals 7 is where misbehavior begins. Auton went weird, but we root caused it to a broken sensor in the coral mech.

Field clock 1:09:
Halfway through teleop, the driver swings the robot around to the back of the reef, and suddenly, the robot stops responding to driver input. I find it unusual that the robot didn't properly align to the reef given our in-season driver is on the sticks, but I'm not convinced it is related.

Field clock 0:52
20 seconds later, control is restored. The drivers controls now manually operate the arm, which is a behavior bound to the operator's controller.

This entire time, the RSL is blinking consistently (Seen just above the FIRST logo on the scorebug). There's no sign that it ever disabled.

After the match, the team took the robot back to the pit w/o restart to replicate. They were able to replicate the loss of control across multiple driver stations (including loaners), across multiple controllers, across power cycles.

Another symptom observed by those in attendance was that the SC's internal storage had reached 50MB remaining. It is unknown if it had been like that for some time, or if this just occurred.

The resolution to this issue was to re-deploy code.

Video: https://youtu.be/wlQgRg98SpY?si=hSFOWLQAol9_Zesm (1:25 is the incident)

Canbus logs + Wpilogs for this incident are lost. I will be getting dmesg logs this week.

[PeterJohnson]

Can you be more detailed on the replication back in the pits? Did it occur a similar amount of time after being enabled in teleop? Did the controls swap? I’m assuming this was without clicking “restart robot code” or power cycling at any point?

[jlmcmchl]

The effect of the controller mismatch was permanent until code was re-deployed. Regardless of change of hardware or power cycling the robot, the issue remained.

Here's the record of events from a member in attenance yesterday:

In Match 7:

• As noted, auto went weirdly, was later traced to a beam break issue that was unrelated
• Lost control with about ~60 seconds left on the clock (can watch video to see exactly)
• Notably, as soon as we lost control I looked down at the driver station, and we had lost robot code, but had not lost comms or joysticks
• Robot code came back up. I'm not sure amount of time .... 10-20 seconds?
• Once code came back up we still had no control, we still had not ever lost comms or joysticks
• The driver tried to drive and the only thing that moved was the algae arm (which isn't even on the driver controller)
• The driver took operator controller (thinking maybe controllers had been swapped) and no buttons did anything
• We went to USD devices tab and confirmed both controllers were recognized, in the correct ports, and sending inputs
• Unplugged the driver controller and plugged it back in, rescanned, and moved it back to port 0 - no change, still algae arm only
• match ended

Between match 7 & 10

• Left robot powered on when we took it off the field
• Took it back to the pits, connected to Systemcore via wifi
• Confirmed the same behavior of controllers, and again confirmed that controllers were recognized, in their correct ports, and sending inputs
• Power cycled and tested again
• Same behavior
• Went to car to get the spare driver station (with second laptop and set of controllers)
• Swapped battery and bumpers
• Powered on with new battery - same behavior
• USB devices tab still indicating controllers were recognized, in correct ports, and sending inputs
• Went to the field for match 10

Prematch for Match 10

• Setup as per usual routine
• Connected to field with usual driver station and controllers, selected auto
• Got spare controller, did mapping sequence on it
• I'm not 100% sure why, but we also swapped laptops at this point a bunch of times - Two FTA laptops, our own spare DS laptop, and our code laptop. I don't recall which one we ran the match on, but it wasn't our usual one.
• I think maybe at this point we were suspicious of the USB ports?
• Match started, auto didn't run, but there was a lot going on pre match and laptops swapped a lot, so hard to say what caused that
• In teleop same problem - Controllers were connected, in the correct ports, and sending inputs, but the only thing that ever moved was algae arm from driver controllers

Post Match 10:

• Went back to pits
• Tried different laptops again
• Tried both guilkit controllers
• Borrowed another team's Xbox controller and tried that
• Tried our controller in every port
• Pulled up phoenix tuner and checked to make sure all the devices were still recognized (they were)
• Additional good ideas were posted in a thread that nobody on sight saw, so we did not try to control motors or blink motors from tuner
• Took the log stick out of System Core

^None of those attempts changed anything. Power cycles after each attempt.

It was at this point we got into the Systemcore UI and identified the <50MB memory issue, and having tried everything we could think of with various controllers and laptops, started down this path instead.
Clearing memory reduced disk usage, but a function check with lower disk usage resulted in same behavior as previously, so that led to redeploying code (and the resulting struggles, but all root causes there have been worked out I think)

[jlmcmchl]

I've finally had access to the Systemcore unit to check journal logs - and they are entirely missing.

A week prior, at C3, is when journal logs stopped. These are the journals created that day:

The final journal:
boot-c77a3b05852d4a73bc1bc311df1376ee.log

The second-to-last journal:
boot-db3680362bfa437ea3a256f144a0430c.log

The last messages in the log all indicate that storage had reached a critical level, with 5MB remaining.

Oct 26 13:45:31 robot bash[1058]: DataLog: Stopped logging due to low free space (4.9 MiB available)
Oct 26 13:45:32 robot bash[1058]: [phoenix] Signal Logger: Available disk space (4 MB) below 5 MB, stopping log (Network: can_s0)
Oct 26 13:45:32 robot bash[1058]: [phoenix] Signal Logger Stopped (Network: can_s0)

Today, the system has about 50% storage free, and yet, the system journal remains empty.