[WIP] fix: rollout recorder not adding turns to rollout after an error until next restart#15095
Draft
[WIP] fix: rollout recorder not adding turns to rollout after an error until next restart#15095
Conversation
…r until next restart I have been chasing this bug for quite some time. I noticed that sometimes when I closed the Codex app, or I tried to resume a CLI thread, that a bunch of messages were missing. Turns out that this can happen if the rollout recorder hits an error (for example if you’re out of disk space). After that happens, it will stop writing to disk. I’m not confident about the Codex-written fix, but I’m sharing it as-is at least to highlight the problem and start the conversation!
JaviSoto
force-pushed
the
dev/javi/rollout-recorder-stop-on-failure
branch
from
1d36c78 to
0143adc
Compare
charley-oai
reviewed
Mar 19, 2026
| #[cfg(test)] | ||
| fail_next_writes: Option<Arc<AtomicUsize>>, | ||
| #[cfg(test)] | ||
| fail_next_flushes: Option<Arc<AtomicUsize>>, |
Collaborator
There was a problem hiding this comment.
Can we avoid adding test-only fields to a prod type? Or is there a reason we need this for tests?
| } | ||
|
|
||
| async fn write_rollout_items(&mut self, rollout_items: &[RolloutItem]) -> std::io::Result<()> { | ||
| #[cfg(test)] |
Collaborator
There was a problem hiding this comment.
This makes the prod code a bit hard to read, do we have to nest test-only code in a prod function like this?
|
|
||
| let result = async { | ||
| self.file.write_all(json.as_bytes()).await?; | ||
| #[cfg(test)] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I have been chasing this bug for quite some time. I noticed that sometimes when I closed the Codex app, or I tried to resume a CLI thread, that a bunch of messages were missing.
Turns out that this can happen if the rollout recorder hits an error (for example if you’re out of disk space). After that happens, it will stop writing to disk.
I’m not confident about the Codex-written fix, but I’m sharing it as-is at least to highlight the problem and start the conversation!