feat(kitchen-sink): add mock agentic lifecycle loop

[NathanFlurry]

Description

Please include a summary of the changes and the related issue. Please also include relevant motivation and context.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[NathanFlurry]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• chore(rivetkit): note duplicate admission logic #4948
• test(rivetkit): cover bypass fetch header forwarding #4947
• feat(kitchen-sink): add actor probe controls #4946
• fix(gateway): split actor response wait errors #4945
• fix(guard): handle bypass preflight before query parse #4944
• feat(kitchen-sink): add mock agent event log copy #4943
• fix(kitchen-sink): default mock agent to local endpoint #4942
• test(rivetkit): enable driver lifecycle regressions #4941
• test(rivetkit): verify cached websocket db refs exceed grace #4940
• fix(rivetkit-core): keep sleeping actors reachable during grace #4939
• test(rivetkit): cover async websocket close db access #4938
• fix(rivetkit): keep actors awake until keepAwake work finishes #4937
• fix(rivetkit): persist native onSleep state on errors #4936
• fix(rivetkit-core): allow dispatch during sleep grace #4935
• fix(rivetkit): make duplicate sleep requests idempotent #4934
• fix(rivetkit): align logs with engine logfmt #4933
• fix(ci): standardize publish rust toolchain #4931
• SPLIT UP: agentic loop test #4932
• chore(rivetkit): doc skip ready wait #4927
• fix(guard): parse gateway bypass query booleans #4926
• feat(depot-client): add vfs staging cache ttl #4930
• feat(depot-client): wire sqlite optimization flags #4923
• feat(rivetkit): support gateway bypass client overrides #4928
• test(depot-client): stabilize fault harness #4924
• chore(depot-client): split sqlite vfs transports #4922
• feat(rivetkit): support gateway bypass probes #4915
• feat(kitchen-sink): add mock agentic lifecycle loop #4914 👈 (View in Graphite)
• feat(kitchen-sink): add mock agentic loop #4891
• feat(rivetkit): support gateway bypass client requests #4917
• fix(pegboard): restore hibernating request ids #4898 : 2 other dependent PRs (#4899 , #4916 )
• chore(guard): configure local timeout knobs #4913
• docs(rivetkit): raise serverless start payload limit #4911
• test(kitchen-sink): add sqlite memory soak harness #4910
• feat(rivetkit): expose runtime diagnostics #4909
• chore(depot-client): migrate to worker thread #4907 : 2 other dependent PRs (#4912 , #4918 )
• test(driver): select matrix cells with vitest filters #4906
• fix(pegboard): restore hibernating requests on serverful start #4904 : 1 other dependent PR (#4905 )
• fix(rivetkit): drain shutdown work before sleep #4902 : 1 other dependent PR (#4903 )
• feat(sqlite): wire remote sqlite execute handlers #4897
• test(serverless): avoid pinning header validation order #4896
• chore(driver-test-runner): add native/wasm runtime matrix to skill #4884 : 3 other dependent PRs (#4892 , #4894 , #4901 )
• chroe: rename rivetkit-sqlite to depot-client #4883
• fix(rivetkit-wasm): fix mem leaks #4880
• chore(rivetkit): wasm support #4860
• feat(sqlite): add cold read benchmarks and simplify optimizations #4857
• feat(sqlite): benchmark cold reads #4848 : 1 other dependent PR (#4864 )
• docs(actors): document manual lifecycle controls #4847
• test(sqlite): expand db stress coverage #4846 : 1 other dependent PR (#4849 )
• test(raw-websocket): add serverless smoke coverage #4845
• feat(serverless): configure drain grace period #4844
• fix(publish): enable frontend build for engine target and use turbo build instead of turbo build:engine #4843
• feat(rivetkit): expose conn handle in onWebSocket context #4842
• fix(rivetkit): normalize waitUntil/keepAwake promises to null to avoid serde undefined error #4841
• fix(rivetkit): remove requirement for token #4839 : 1 other dependent PR (#4840 )
• fix(pegboard-envoy): allow sqlite requests across lifecycle states #4838
• fix(pegboard): route scoped actor key reads by runner dc #4837
• fix(sqlite-storage): let sqlite open take over stale owners #4834
• fix(engine-tests): parallelize multi-DC startup + enable 6 multi-DC tests #4833
• fix(api): honor cursor in actor_ids list path #4832
• fix(api): silently filter invalid IDs from actor_ids query instead of 400 #4831
• fix(api-peer): make actors_delete idempotent for already-destroyed actors #4830
• fix(pegboard): use Iterator streaming mode in list_names so limit param is honored #4829
• chore(engine-tests): convert envoy tests to Sleep crash policy + triage ignored #4828
• feat(engine-tests): wire envoy actor names through prepopulate_actor_names #4827
• chore(depot): fault injection tests #4868
• feat(sqlite): pitr & forking #4859 : 1 other dependent PR (#4863 )
• chore(sqlite): stateless storage refactor #4856
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[claude]

PR Review: feat(kitchen-sink): add mock agentic lifecycle loop

Overview

This PR substantially expands the mock agentic lifecycle test harness in examples/kitchen-sink. The script gains the ability to auto-spawn a local kitchen-sink server, run a concurrent probe loop validating actor liveness during sleep/reconnect cycles, track close events, validate full history consistency at teardown, and report aggregate statistics. The actor gains onSleep delay, ping/pong WebSocket support, and a verifyAll action.

---

Issues

1. Timer leak in runProbeAttempt

The timeout() helper creates a setTimeout that is never cancelled when Promise.race resolves via a competing branch. Each probe attempt leaks up to two pending timers (open and pong phases). The inner pong promise already uses cleanup() correctly -- apply the same pattern to the outer timeout() helper.

2. Platform-specific process management

pidsWithEnvValue reads from a Linux-only pseudo-filesystem, failing silently or erroring on macOS. listenerPids uses lsof flags that differ across platforms. Add a runtime platform guard (return empty array on non-Linux) or document the assumption.

3. Hard-coded 2-second sleeps in cleanup helpers

stopListeners and stopProcessesWithEnvValue sleep 2 seconds after SIGTERM then force-kill. Polling the port check with a short interval would be more reliable.

4. DEFAULT_ON_SLEEP_DELAY_MS = 15_000 has no comment

This 15-second delay in onSleep is non-obvious and extends test runtime. Per CLAUDE.md, add a comment explaining why it exists (presumably to let probe connections observe the actor stopping).

5. IIFE for final history validation is hard to read

The inline self-invoking async function around line 1121 should be extracted as a named function (e.g. getFinalHistory()).

6. Double cast as unknown as ActionVerifier

Bypasses the type system entirely. If actor action signatures change there is no compile-time check. Define the interface against the typed handle's action surface.

7. Unbounded backlog in RawSession

The #backlog array has no size cap. Add a cap (~500 entries) and shift off the oldest when exceeded.

8. Repeated reconnect-time check

if (reconnectMs > MAX_RECONNECT_MS) throw appears three times in runWorkload. Extract into a helper.

---

Minor

• Removed verifier param from runInference: good -- verification is now explicit at the call site with layered checks.
• connect() returns 0 when already open (reads as "opened instantly" vs "was already open"). A named result would be clearer.
• --import args in startLocalKitchenSinkServer: add a comment that @rivetkit/sql-loader is required for SQLite.

---

Summary

Architecture is sound; the probe loop + sleep-close observation pattern is a solid approach for validating lifecycle behavior under reconnect. Main blocking issue: timer leak in runProbeAttempt. Linux-only assumptions and the onSleep delay need documentation. IIFE and double-cast are maintainability items.

[claude]

PR Review: feat(kitchen-sink) — add mock agentic lifecycle loop

Overview

This PR substantially expands the kitchen-sink mock agentic loop test harness with:

• Auto-managed local server: optionally spawns/stops the kitchen sink server + engine in-process so the script is fully self-contained
• Probe loop: continuously fires WebSocket ping/pong connections in parallel to the main workload, observing expected actor.stopping closes during sleep transitions
• Richer history validation: validates full history (row counts, contiguity, expected request IDs) on every reconnect and at the end
• Sleep-close observation: the main session now explicitly waits for and validates the close event triggered by sleep posts
• verifyAll action: new bulk verification action that validates all rows in a single DB query
• onSleep hook with configurable delay: adds a sleep(delayMs) inside onSleep to give the probe loop time to observe the stopping close

---

Issues

Hard: Logic / Correctness

sleepPostsObservedByMain guard can silently skip a close observation

There are two separate places that check sleepStats.posts > sleepPostsObservedByMain. If the connection drops before runInference starts (the !session.open branch fires first), sleepPostsObservedByMain is bumped. The post-inference guard then skips, so waitForClose(SLEEP_CLOSE_TIMEOUT_MS) is never called and the close event goes unvalidated. Consider a single unified reconnect path rather than two parallel guards.

IIFE in runWorkload is hard to follow and has a missing finally

validateHistory(await (async () => {
    const finalSession = new RawSession(...);
    ...
    finalSession.close();   // not called if an exception is thrown first
    return history;
})(), expectedRequests);

Extract this into a named async function (e.g. requestFinalHistory). The inline IIFE is difficult to read and finalSession.close() is not in a finally block, so the session leaks on error.

Unbounded #backlog in RawSession

Messages pushed to #backlog are only consumed if a subsequent waitFor finds them. Stray progress or done events from a prior inference can accumulate and never be drained. For a long-running workload this is a slow memory leak. Consider capping the backlog or logging when messages are silently dropped.

---

Medium: Platform Assumptions

Linux-only: pidsWithEnvValue reads the proc filesystem

The function iterates over /proc entries and reads each process's environment file to find processes by env-var value. This is Linux-only: it silently returns nothing on macOS and fails on Windows. lsof in listenerPids has a similar constraint (macOS has lsof, Windows does not). These are test-only scripts so impact is limited, but a comment explaining the Linux assumption would prevent confusion when running locally on macOS.

resolveEngineBinary only looks at the debug build path

target/debug/rivet-engine is the only fallback. A release build (target/release/rivet-engine) is not tried. Developers who built with --release get a confusing "binary not found" error. Either check both paths or update the error message to mention RIVET_ENGINE_BINARY and --release as alternatives.

---

Medium: Style / Conventions

numberFromEnv duplicated between script and actor

The function appears verbatim in both scripts/mock-agentic-loop.ts and src/actors/testing/mock-agentic-loop.ts. A shared utility module would avoid drift.

Error event in waitForOpen drops the error detail

const onError = () => {
    cleanup();
    reject(new Error("websocket error"));
};

The ErrorEvent argument is discarded. Include event.message in the rejection message to aid debugging.

Probe timeout helper leaks its setTimeout handle

const timeout = (phase: string) =>
    new Promise<never>((_resolve, reject) => {
        setTimeout(() => reject(...), PROBE_TIMEOUT_MS);
    });

The setTimeout is never cleared if the Promise.race resolves on another branch first. The timer fires, calls reject on a settled promise (harmless but noisy), and holds a reference until it fires. Store and clear the handle, or use AbortSignal.timeout.

verifyAll action discards validation return values

stringValue(request.requestId, "requestId");
positiveInteger(request.seconds, "seconds");

These are called only for their throw-on-invalid side effect. A comment or renaming to assertStringValue / assertPositiveInteger would make the intent clear.

---

Low: Minor Notes

• stopLocalKitchenSinkServer calls stopListeners for each port sequentially; each call sleeps 2 s between SIGTERM and SIGKILL. Use Promise.all to run the two port cleanups in parallel.
• appendServerLog uses Array.shift() in a while loop to maintain a 200-entry cap; shift is O(n). For 200 items this is negligible, but a single splice(0, logs.length - 200) is cleaner.
• connect() return type changed from void to number (openMs) without a type annotation update. Existing callers that ignore the value still compile, but the changed contract is invisible at call sites.
• The final [done] log line is already long; splitting probe stats into a separate [done-probe] line would improve readability.

---

Summary

The core design is solid: the probe loop and sleep-close observation are good additions that catch real liveness regressions. The main asks before merging are (1) unifying the two sleepPostsObservedByMain guards so every sleep close is validated, (2) extracting the final-session IIFE into a named function with proper finally cleanup, and (3) adding comments about the Linux-only platform assumptions.

[NathanFlurry]

Landed in main via stack-merge fast-forward push. Commits are in main; closing to match.