"Sticky" `: 1` after failing command: suggestions for debugging?

[mmesiti]

Context: I am trying to use this kernel to write part f the user documentation of our HPC cluster, to make it testable with nbval in a CI fashion.

Unfortunately, one of the bash commands that I need to show partially "breaks" the kernel, in the sense that:

• The output and the actions of the command seem to happen normally, but the kernel reports an error (: 1 after the executed cell)
• it does not fail when run in the terminal, on the same machine where the kernel would run (exit status is 0)
• after that command, all subsequent commands display their output correctly but still report a failure with : 1 after they have run in their cell (even a simple ls command)
• this particular command takes some minutes to run and might also need quite some memory
• Also the variants

  <command> &> outputfile
  

  and

  <command> &> outputfile || true
  

  still cause the same behviour.
• when running jupyter with --debug, I get, among all the messages:

[D 2025-08-12 16:52:34.177 ServerApp] activity on 1bb662fb-23b5-4d79-bfa6-d453b6bf4f38: stream
[D 2025-08-12 16:52:34.185 ServerApp] activity on 1bb662fb-23b5-4d79-bfa6-d453b6bf4f38: stream
[D 2025-08-12 16:52:34.238 ServerApp] activity on 1bb662fb-23b5-4d79-bfa6-d453b6bf4f38: error
[D 2025-08-12 16:52:34.243 ServerApp] activity on 1bb662fb-23b5-4d79-bfa6-d453b6bf4f38: status 

But I do not get additional information about the what the failure really is.
How can I get more detailed information?

(Thank you)

[takluyver]

There's probably something configured in bash which triggers some command to run after the command you've sent from Jupyter, so when we ask bash to echo $? to get the status of the last command, it's the wrong last command.

There's something similar in #147, with a possible workaround for that specific case. I think resetting PS0 inside bash_kernel might be a more general fix, but I'm not really using bash_kernel myself at the moment, so I haven't got around to trying it.

[mmesiti]

Thank you. For the record, I have tried:

• to run the same thing but on a my laptop (ubuntu) instead of the cluster login node (RHEL9), using the same software versions everywhere. Things worked fine here.
• the workaround mentioned in ]777;preexec\ before and : 1 after the output #147 (setting .env.VTE_VERSION = "" in the kernel spec). This did not change anything.
• switching back to python3.11 instead of python3.12. This has solved an error message provoked by https://github.com/takluyver/bash_kernel/blob/master/bash_kernel/kernel.py#L130, where python3.12 complains with invalid escape sequence '\[', but still the behaviour I am concerned about is unchanged
• completely ignoring the system bashrc in pexpect/bashrc.sh: this has changed nothing

I am trying to produce a logfile (pexpect.spawn() does have a logfile kwarg), but I have not managed yet.

[mmesiti]

An update: the logfile looks perfectly fine (I mean, the one that is created by passing the logfile argument to spawn() here: https://github.com/takluyver/bash_kernel/blob/master/bash_kernel/kernel.py#L123). But for some reason, the output of (see https://github.com/takluyver/bash_kernel/blob/master/bash_kernel/kernel.py#L227)

self.bashwrapper.run_command('{ echo $?; } 2>/dev/null')

is not just a number on an empty line any more, after a problematic command is executed,
but includes the command to get the value itself:

{ echo $?; } 2>/dev/null
0

so the parsing fails, as self.bashwrapper.run_command('{ echo $?; } 2>/dev/null').rstrip().split("\r\n")[0] evaluates actually to
{ echo $?; } 2>/dev/null.

It looks to me like a problem in pexpect.
Restarting the kernel resets everything and everything starts working again.

[mmesiti]

Debugging this has been so far a rabbit hole that, although interesting, might not be super useful and a little wasteful of everyone's time.

I have a simple proposal that bypasses the need for deep understanding of the problem: changing
https://github.com/takluyver/bash_kernel/blob/master/bash_kernel/kernel.py#L227

from

        exitcode = int(self.bashwrapper.run_command('{ echo $?; } 2>/dev/null').rstrip().split("\r\n")[0])        

to

        exitcode = int(self.bashwrapper.run_command('{ echo $?; } 2>/dev/null').rstrip().split("\r\n")[-1])        

does solve my problem, and I do not see potential problems coming from this fix, BUT I might be missing something - perhaps someone else know why there's a 0 instead of a -1 there.

Perhaps - sorry for pointing fingers and disturbing people - @kdm9 might know, since they added the .split("\r\n")[0] there.