M1: MCP server Phase 1 (#107, #108, #109, #111)#131
Open
hanwencheng wants to merge 19 commits into
Open
Conversation
…sue #64, #71 Option A) (#73) * agentkeys: stage 7 issue#64 phase 0 -- US-001 src/env.rs centralized env-var module Implement plan §5: single source of truth for every BROKER_* environment variable name. Per user rule 11, no other module may declare a raw env-var literal — all reads go through these constants. - crates/agentkeys-broker-server/src/env.rs (new): const &str declarations for all 51 env vars (Phase 0 + planned A/B/C/D/E + legacy aliases), Group enum (Core/Oidc/SessionJwt/Audit/AuditEvm/Auth/AuthEmail/AuthOAuth2/ Limits/Legacy), all() registry returning (name, doc, group), print_table() for the operator runbook auto-generator. 5 unit tests cover uniqueness, non-empty docs, required-Phase-0 presence, table render row count, and Group exhaustiveness. - crates/agentkeys-broker-server/src/lib.rs: register pub mod env. - crates/agentkeys-broker-server/src/config.rs: replace every raw BROKER_* string literal with env::* constants. grep -E '"(BROKER_|DAEMON_|ACCOUNT_ID|REGION)' src/config.rs returns zero hits. Adds parse_int_env_with_default<T> helper to collapse three near-duplicate parse blocks. Plan home: docs/spec/plans/issue-64/{PLAN.md (mirror), DECISIONS.md, AMBIGUITIES.md, V0.1-FOLLOWUPS.md, prd.json (PRD-driven ralph)}. Acceptance criteria (US-001): - env.rs exists with const &str for every plan §5 BROKER_* var ✓ - Group enum with required variants ✓ - all() returns slice of (name, doc, Group), all docs non-empty ✓ - src/config.rs: grep zero hits for raw BROKER_/DAEMON_/ACCOUNT_ID/REGION ✓ - cargo build -p agentkeys-broker-server succeeds ✓ - cargo test -p agentkeys-broker-server env:: 5/5 pass ✓ Refs: issue #64 plan §1 rule 11, §5. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- US-002 plugin trait scaffolding Implement plan §3 + §3.5: pluggable trait surface for the three layers below the credential mint. No plug-in implementations yet (US-006 implements WalletSig, US-007 ClientSideKeystore, US-008 SqliteAnchor) — this story lands the trait shapes, error types, and registry that the later stories slot into. - crates/agentkeys-broker-server/src/plugins/mod.rs (new): Readiness enum (Ready/Degraded/Unready), PluginRegistry { auth: HashMap, wallet, audit: Vec }, aggregate_readiness() → (overall, per-check) for the /readyz JSON. Trait re-exports. - crates/agentkeys-broker-server/src/plugins/auth.rs (new): UserAuthMethod trait (name/ready/challenge/verify), VerifiedIdentity, ChallengeParams, AuthChallenge, AuthResponse, IdentityType { Evm, Email, OAuth2{Google, Github,Apple} } with stable canonical() strings (input to OmniAccount derivation; renaming is breaking). AuthError enum. - crates/agentkeys-broker-server/src/plugins/wallet.rs (new): WalletProvisioner trait (name/ready/bind_address/lookup_by_omni_account), WalletAddress newtype with parse() that normalizes 0x-prefixed hex to lowercase + length check, WalletRole { Master, Daemon }, WalletBinding struct. WalletError enum. - crates/agentkeys-broker-server/src/plugins/audit.rs (new): AuditAnchor trait (name/ready/anchor/verify), AuditRecord with record_hash for cross-anchor dedup, AnchorReceipt, AuditPolicy { DualStrict, SqlitePrimary, EvmPrimary } parser. AuditError enum. - crates/agentkeys-broker-server/src/lib.rs: register pub mod plugins. - crates/agentkeys-broker-server/Cargo.toml: feature-gate scaffold per plan §3. default = [auth-wallet-sig, wallet-keystore, audit-sqlite]. Optional features for v0-testnet (auth-email-link, auth-oauth2-google, audit-evm) and v1+ (auth-oauth2-github, auth-oauth2-apple, audit-solana). External deps land in implementation stories (US-006: k256+sha3; Phase A.1: lettre+aws-sdk-sesv2; Phase C: alloy-*). Acceptance criteria (US-002): - Readiness enum with Ready/Degraded/Unready ✓ - UserAuthMethod / WalletProvisioner / AuditAnchor traits ✓ - PluginRegistry struct + aggregate_readiness ✓ - Per-trait thiserror error enums (AuthError, WalletError, AuditError) ✓ - Cargo features: auth-wallet-sig, auth-email-link, auth-oauth2, auth-oauth2-google, wallet-keystore, audit-sqlite, audit-evm, test-stub ✓ - cargo build with default features ✓ - cargo test plugins:: 8/8 pass ✓ - cargo clippy -D warnings clean ✓ Per-trait `ready()` MUST NOT default to Ready — implementations check their own dependencies. Documented in trait doc comments. The first implementations (US-006/007/008) demonstrate the pattern. Refs: issue #64 plan §3, §3.5, §1 rule 8. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- US-004 OmniAccount + US-008 SqliteAnchor port Bundles two stories that became coupled when the agentkeys-types::AgentIdentity extension forced match-arm updates across four crates and the audit/ module restructure required relocating both the trait file and the SqliteAnchor implementation in the same change. US-004 — OmniAccount derivation - crates/agentkeys-broker-server/src/identity/{mod.rs,omni_account.rs} (new): derive_omni_account(identity_type, identity_value) → SHA256(client_id || type || value) with hardcoded AGENTKEYS_CLIENT_ID = "agentkeys". Per port- vs-greenfield "What we port — crypto primitives only", this matches the dexs-backend hash shape verbatim but uses our own client_id, giving each operator a sovereign identity namespace. derive_with_client_id(...) is exposed for reproducing dexs reference vectors in tests. - crates/agentkeys-types/src/lib.rs: AgentIdentity::OAuth2{provider, sub} variant added (additive — every existing AgentIdentity consumer continues to work unchanged for the four prior variants). - Match-arm updates across consumers (Rust E0004 non-exhaustive errors surfaced these — exactly the property we want from the type system): - crates/agentkeys-core/src/mock_client.rs (open_auth_request + session_recover): map OAuth2{provider,sub} → ("oauth2_<provider>", sub) matching the broker's IdentityType::canonical() naming. - crates/agentkeys-core/src/auth_request.rs: deterministic CBOR encoding of OAuth2 — Map[("provider", Text), ("sub", Text)] with keys ASCII- sorted so the canonical hash is stable. - crates/agentkeys-cli/src/lib.rs: rich-error human-readable form "oauth2_<provider>:<sub>". - crates/agentkeys-mock-server/src/test_client.rs: same mapping as mock_client (auth-request and session-recover paths). - 9 identity:: unit tests cover: hex parse validation, derivation determinism, identity-type namespace separation, identity-value separation, client_id namespace separation (load-bearing — proves agentkeys ≠ wildmeta for the same email), prod entry-point matches hardcoded constant, lowercase-hex output guarantee. US-008 — SqliteAnchor port to AuditAnchor trait - crates/agentkeys-broker-server/src/plugins/audit/{mod.rs,sqlite.rs} restructured: trait file `audit.rs` merged into `audit/mod.rs` so the feature-gated `audit-sqlite` submodule can live alongside it. (Previous layout had `audit.rs` + `audit/mod.rs` which Rust E0761'd.) - src/plugins/audit/sqlite.rs (new): SqliteAnchor implementing AuditAnchor. Schema is the new plugin_mint_log table with the canonical AuditRecord columns + a status column (Phase 0 writes 'confirmed' directly; Phase C introduces the pending → confirmed | quarantined lifecycle). Indexes on minted_at, omni_account, record_hash, status. WAL+FULL pragma preserved from the legacy crate::audit::AuditLog. - Readiness::Ready when DB writable; Unready otherwise. - 8 plugins::audit:: tests cover: anchor round-trip, verify NotFound, record_hash tampering detection, wrong-anchor receipt rejection, ready reports Ready, name() stability + AuditPolicy parse + AuditRecord round trip. Acceptance criteria (US-004): - src/identity/omni_account.rs derive_omni_account(...) ✓ - AGENTKEYS_CLIENT_ID = "agentkeys" pinned ✓ - agentkeys-types::AgentIdentity::OAuth2{provider, sub} added ✓ - Tests cover canonical hash for each identity type ✓ - cargo test identity:: 9/9 pass ✓ Acceptance criteria (US-008): - src/plugins/audit/sqlite.rs implements AuditAnchor ✓ - plugin_mint_log table with canonical columns + indexes ✓ - WAL+FULL pragma preserved ✓ - verify() detects record_hash tampering ✓ - Readiness Ready when writable ✓ - cargo test plugins::audit:: 8/8 pass ✓ Note: legacy crate::audit::AuditLog (the existing src/audit.rs) is left in place for now — US-011 migrates the mint handler to the new trait and drops the legacy module then. Carrying both during the transition keeps existing /v1/mint-aws-creds working. Refs: issue #64 plan §3.5 (OmniAccount), §3 (AuditAnchor trait), §Phase 0 deliverables. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- US-005 dual ES256 keypairs with purpose tagging Implement plan §3.5.6: two distinct ES256 keypairs for two roles: - oidc keypair (existing) — signs JWTs that AWS STS verifies via JWKS. - session keypair (NEW) — signs broker-internal session JWTs. Closes Codex / eng-review #7 footgun: an operator pointing BROKER_SESSION_KEYPAIR_PATH at the OIDC keypair file would have silently used the wrong key (same kid, same crypto), letting session tokens pass as IAM federation tokens. Defense: on-disk JSON now carries a "purpose" field; load-time validation refuses to read a keypair whose purpose does not match the slot. - crates/agentkeys-broker-server/src/jwt/{mod,session,issue,verify}.rs (new): KeypairPurpose enum (Oidc | Session) with stable kebab-case canonical() and kid_prefix(); SessionKeypair (mirror of OidcKeypair, purpose-tagged on disk, kid prefix `ak-session-`); mint_session_jwt() with the canonical session-JWT claim shape (iss/sub/aud=agentkeys:broker/exp/iat/jti + agentkeys.{omni_account,wallet_address,identity_type,identity_value}); verify_session_jwt() that pins audience + issuer + kid header. - crates/agentkeys-broker-server/src/oidc.rs: - PersistedKeypair: add `purpose` field with #[serde(default)] mapping to KeypairPurpose::Oidc so pre-Stage-7 keypair files (no purpose field) continue to load as oidc. New keypairs always include the field. - load() refuses any keypair whose purpose ≠ Oidc. - generate_and_persist() writes purpose=oidc. - rand_core_compat → pub(crate) rand_compat (so SessionKeypair can reuse the rand_core 0.6 → OS RNG bridge). - set_owner_only → pub(crate) set_owner_only_inner (same reason). - crates/agentkeys-broker-server/src/lib.rs: register pub mod jwt. Acceptance criteria (US-005): - src/jwt/mod.rs: KeypairPurpose with Oidc + Session ✓ - On-disk JSON includes "purpose" field ✓ - SessionKeypair::load refuses purpose=oidc keypair ✓ - SessionKeypair::load refuses untagged JSON ✓ - OidcKeypair::load refuses purpose=session keypair ✓ - Session JWT mint+verify round trip ✓ - verify rejects wrong audience, wrong issuer, expired ✓ - session keypair kid prefix `ak-session-`; oidc kid format unchanged ✓ - cargo test jwt:: 10/10 pass ✓ - cargo build green ✓ env.rs already has BROKER_SESSION_KEYPAIR_PATH and BROKER_SESSION_JWT_TTL_SECONDS (landed in US-001). Wiring config.rs + boot.rs to actually load the session keypair lands in US-003 (tiered refuse-to-boot). Refs: issue #64 plan §3.5.6, codex review finding #7, eng review #code-structure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- US-007 ClientSideKeystoreProvisioner + WalletStore Implement plan §3.5 + §Phase 0 wallet layer: the MetaMask model. The broker stores ONLY (omni_account, address, role, parent_address, created_at) — the user holds the seed in their OS keychain on the daemon side. The broker has no key material it could leak. Storage layer: - crates/agentkeys-broker-server/src/storage/{mod.rs, wallets.rs} (new): WalletStore with composite-PK schema (omni_account, address) so a user can have multiple wallets and re-binding the same address is idempotent. WAL+NORMAL for throughput (audit log gets FULL elsewhere). bind() detects role mismatch and parent mismatch on re-bind — a daemon switching masters or an address flipping role would be silent data corruption otherwise. list_for_omni_account() returns every wallet bound to the OmniAccount. writable() probe used by the plugin's ready(). Plugin layer: - crates/agentkeys-broker-server/src/plugins/wallet/{mod.rs,keystore.rs}: module restructure from sibling-file `wallet.rs` to `wallet/mod.rs + wallet/keystore.rs` (same E0761 fix as US-008's audit module). ClientSideKeystoreProvisioner implements WalletProvisioner. name() = "client_keystore". ready() reflects WalletStore::writable() (NOT a hardcoded Ready, per plan §1 rule 5). bind_address() stamps current unix-seconds and delegates to WalletStore::bind. lookup_by_omni_account delegates to WalletStore::list_for_omni_account. - crates/agentkeys-broker-server/src/lib.rs: register pub mod storage. Acceptance criteria (US-007): - src/plugins/wallet/keystore.rs implements WalletProvisioner ✓ - Storage table wallets(omni_account, address, role, parent_address, created_at) with composite PK and role CHECK constraint ✓ - bind(): inserts row; idempotent (same role + parent → returns existing) ✓ - bind() rejects role mismatch ✓ - lookup_by_omni_account returns all bindings ✓ - ready() Ready when DB writable, Unready otherwise ✓ - 9 plugins::wallet:: tests pass (3 type tests + 6 keystore behavior tests covering bind+lookup, idempotent re-bind, rejected role flip, ready, name, multi-binding lookup) ✓ - cargo build green ✓ Refs: issue #64 plan §3.5 (wallet layer), §Phase 0 deliverables. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- session 1 progress checkpoint Update progress.txt with full Phase 0 session log (6 of 16 stories complete: US-001/002/004/005/007/008). Update prd.json passes flags + commit refs. Append commit-log table to DECISIONS.md. Phase 0 remaining (10 stories) for next ralph iteration: - US-003 boot.rs + main.rs wiring - US-006 WalletSig SIWE (largest remaining; needs k256+sha3 deps) - US-009/010/011 auth + mint endpoints - US-012 broker_status /readyz aggregator - US-013 invariant load-bearing test (all 6 cases) - US-014 smoke + done.sh - US-015 operator runbook - US-016 codex round 1 Suggested next-iteration commit order: 6 → 3 → 9/10/11 → 12 → 13 → 14 → 15 → 16. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- mark 6 stories passing in prd.json passes:true + commit refs for US-001, US-002, US-004, US-005, US-007, US-008. Remaining 10 Phase 0 stories still passes:false. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- US-006 SiweWalletAuth + AuthNonceStore Phase 0 wallet-sig auth method per plan §3.5.1: SIWE-wrapped EIP-191. Closes Codex P0 #2 (raw EIP-191 was replayable across apps; SIWE binds domain). Storage: - crates/agentkeys-broker-server/src/storage/auth_nonces.rs (new): AuthNonceStore with single-use semantics. issue() inserts, consume() is race-safe via WHERE consumed_at IS NULL conditional UPDATE, purge_expired() janitors old rows. ConsumeOutcome enum collapses "never existed" and "already consumed" into NotFoundOrConsumed so an attacker cannot probe the nonce table; Expired is a separate variant so the broker can surface a "your sign-in expired" message. 7/7 tests pass. Plugin: - crates/agentkeys-broker-server/src/plugins/auth/{mod.rs ⟵ ex auth.rs, wallet_sig.rs} (restructure + new): Same E0761 module-conflict fix as US-007/008. SiweWalletAuth implements UserAuthMethod. challenge() builds an EIP-4361 SIWE message with the broker's domain, fresh CSPRNG nonce, issued_at, expiration_time (issued_at + 45min), URI, chain_id, resources. verify() looks up the pending challenge, atomically consumes the nonce, runs k256 ecrecover via the EIP-191 envelope (`\x19Ethereum Signed Message:\n<len><msg>` → keccak256 → recover_from_prehash), and asserts the recovered address matches the SIWE message's claimed address. ecrecover_address() handles v ∈ {0,1,27,28} (k256 RecoveryId requires {0,1}, so 27/28 are normalized). Per-call security: - SIWE domain field bound to broker's host (replay across apps blocked) - Nonce single-use enforced via AuthNonceStore (replay across requests blocked) - 45-min issued_at/expiration window (replay across long timeframes blocked) - k256 0.13 enforces canonical signatures (low-s) by default - Chain-ID bound into the SIWE message (replay across chains blocked) Pending challenges live in tokio::sync::Mutex<HashMap> keyed by request_id; removed on first verify() attempt to prevent in-memory replay even if the on-disk nonce check is flaky. Multi-process deployments would move this to SQLite — out of scope for v0. Custom ISO8601 formatter (no chrono dep). Howard-Hinnant civil_from_days valid 1970+. Tests pin format shape. Embeds the canonical IdentityType enum + UserAuthMethod trait + supporting types (VerifiedIdentity, ChallengeParams, AuthChallenge, AuthResponse, AuthError) in plugins/auth/mod.rs — preserved verbatim from the previous plugins/auth.rs file with feature-gated re-export of SiweWalletAuth. Cargo: - agentkeys-broker-server/Cargo.toml: k256 + sha3 added as optional deps gated by auth-wallet-sig feature. Default features compile them in. - storage/mod.rs: re-export AuthNonceStore + ConsumeOutcome. Acceptance criteria (US-006): - src/plugins/auth/wallet_sig.rs implements UserAuthMethod for SiweWallet ✓ - challenge() generates SIWE with domain/URI/version/chain_id/nonce/iat/exp/resources ✓ - Nonce stored in src/storage/auth_nonces.rs with UNIQUE single-use UPDATE ✓ - verify() asserts domain, chain_id, expiration; ecrecover-derived address matches ✓ - VerifiedIdentity returns IdentityType::Evm + identity_value ✓ - 11 plugins::auth::wallet_sig + 7 storage::auth_nonces tests pass ✓ - happy path, expired (Expired), replayed nonce (NotFoundOrConsumed), malformed signature (InvalidRequest), unknown request_id (Unauthorized), duplicate-nonce-issue (rejected), purge_expired correctness ✓ Refs: issue #64 plan §3.5.1, codex P0 #2 (SIWE adopted), §Phase 0 deliverables. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- update prd.json + DECISIONS.md after US-006 Mark US-006 passes:true with commit ref 51a5191. Append commit-log row in DECISIONS.md. List remaining 9 Phase 0 stories in priority order. Phase 0 status: 7 of 16 stories complete. ~71 unit tests passing. Foundation locked: env vars centralized, plugin traits + Readiness + PluginRegistry, OmniAccount derivation, dual ES256 keypairs with purpose tagging, ClientSideKeystoreProvisioner + WalletStore, SqliteAnchor port, SiweWalletAuth + AuthNonceStore (single-use SIWE-wrapped EIP-191). Next priority: US-003 (boot.rs wiring) → US-009/010/011 (endpoints) → US-012 (broker_status) → US-013 (invariant test) → US-014/015 (smoke + runbook) → US-016 (codex round 1). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- US-003 tiered refuse-to-boot + plugin-registry wiring Implement plan §6 tiered refuse-to-boot. Closes Codex P1 #6 (transient external dependencies must not brick startup): Tier 1 (synchronous, before listener bind): - All required env vars present + parseable + types in declared bounds. - BROKER_OIDC_ISSUER must be https:// in non-dev mode (BROKER_DEV_MODE=true relaxes; logged loudly). - OIDC keypair file MUST exist + parse + carry purpose=oidc tag (refuses purpose=session). - Session keypair file MUST exist + parse + carry purpose=session tag (no migration window). - SQLite migrations run cleanly via AuthNonceStore::open + WalletStore::open + SqliteAnchor::open. Each CREATE TABLE IF NOT EXISTS is the v0 migration. - BROKER_AUTH_METHODS / BROKER_WALLET_PROVISIONER / BROKER_AUDIT_ANCHORS resolve at compile time (every name must map to an enabled feature; unknown names → boot fail with anchor `auth-method-not-compiled` etc.). - BROKER_AUDIT_POLICY parses to {dual_strict, sqlite_primary, evm_primary}. - Failure: exit code 1 with single-line `BOOT_FAIL: <var>=<value>: <reason>; see runbook §<anchor>`. Tier 2 (async, after listener bound): - Backend `/healthz` reachability probe loops every 15s until success; flips state.tier2.backend_reachable. - /healthz returns 200 immediately (liveness); /readyz aggregates Tier-2 atomic flags + plugin Readiness (US-012 lands the aggregator handler — for now /readyz still uses the legacy flat probe pre-broker_status migration). - BROKER_REFUSE_TO_BOOT_STRICT=true collapses Tier-2 backend probe to a hard fail (process exits if backend not reachable). - SES + EVM probes deferred to Phase A.1 + Phase C respectively, behind their feature gates. The Tier2State struct already carries the AtomicBool fields so adding probes is one-line each. Files: - crates/agentkeys-broker-server/src/boot.rs (new): run_tier1() returns BootArtifacts (registry + keypairs + stores + audit_policy). build_registry() constructs PluginRegistry from BROKER_AUTH_METHODS / BROKER_WALLET_PROVISIONER / BROKER_AUDIT_ANCHORS. Tier2Profile::from_config() probes which Tier-2 checks are enabled. 4 unit tests cover https-only refuse, missing keypair refuse, url_host extraction, Tier2Profile detection. - crates/agentkeys-broker-server/src/state.rs (extended): AppState now carries session_keypair, registry, audit_policy, wallet_store, nonce_store, tier2 (Arc<Tier2State> with 4 AtomicBool fields). Legacy `audit: AuditLog` preserved through US-011. - crates/agentkeys-broker-server/src/main.rs (rewritten): calls run_tier1() → BootArtifacts before STS check. spawn_tier2_probes() spawns the backend reachability probe with 15s retry; strict mode exits the process on first miss. - crates/agentkeys-broker-server/src/lib.rs: pub mod boot. - crates/agentkeys-broker-server/tests/{oidc_flow,mint_flow}.rs: stub the new AppState fields with in-memory stores + fresh session keypair so the legacy backend-bearer-mint integration tests continue to pass unchanged. Acceptance criteria (US-003): - src/boot.rs with run_tier1() (sync) + Tier2Profile::from_config() (Tier-2 spawn) ✓ - Tier-1 validates env vars present + paths readable + OIDC https in non-dev ✓ - Plugin registry validates: every name in BROKER_AUTH_METHODS / etc. resolves ✓ - Tier-1 runs SQLite migrations cleanly ✓ - Keypair load: refuse-to-boot if path absent or purpose tag mismatch ✓ - Tier-2 reachability checks marked async ✓ - BOOT_FAIL message format with runbook anchor ✓ - 4 boot:: tests pass ✓ - Full broker test suite 94 tests pass (79 lib + 9 mint_flow + 6 oidc_flow) ✓ - cargo build green ✓ Refs: issue #64 plan §6 (tiered refuse-to-boot), §3 (PluginRegistry), §Phase 0 deliverables. Closes codex review finding P1 #6 (refuse-to-boot vs Unready). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- US-012 broker_status /readyz aggregator Per plan §7 + Designer review #status-shape: /readyz now aggregates PluginRegistry::aggregate_readiness() across every loaded plug-in PLUS the four Tier-2 reachability AtomicBool flags (set asynchronously by spawn_tier2_probes in main.rs). Behavior: - 200 with empty body when every plug-in Ready + every relevant Tier-2 flag set. Operators tailing curl see no noise on the happy path. - 200 with `{"status":"degraded","degraded":true,"checks":[...], "ready":[...]}` when any plug-in reports Degraded. Body lists every degraded check with `name`, `status`, `reason`, and a `docs` URL anchor pointing into the operator runbook (Designer review: pager- friendly). - 503 with `{"status":"unready",...}` when any plug-in is Unready or any relevant Tier-2 flag is still false. Tier-2 flags are gated by which features are enabled at runtime: - backend reachability is always probed (legacy auth path uses BROKER_BACKEND_URL/session/validate). - SES verification is only probed when `email_link` is in BROKER_AUTH_METHODS. - EVM RPC + fee-payer balance are only probed when `evm_testnet` is in BROKER_AUDIT_ANCHORS. Files: - crates/agentkeys-broker-server/src/handlers/broker_status.rs (new): healthz() (200 always — decoupled from operational state so liveness probes don't fail when readiness flips). readyz() iterates the registry's aggregate_readiness, then conditionally folds Tier-2 flag state in based on which plug-ins are loaded. Per-check JSON shape: {name, status, reason|detail, docs}. - crates/agentkeys-broker-server/src/handlers/mod.rs: pub mod broker_status. - crates/agentkeys-broker-server/src/lib.rs: route /healthz + /readyz to handlers::broker_status::{healthz, readyz}. Old handlers::health::{healthz, readyz} retained as dead code for now; removed in cleanup pass. - crates/agentkeys-broker-server/tests/mint_flow.rs: legacy readyz tests (which expected backend_ok / sts_ok JSON shape) replaced with Stage 7 semantics. Each test reflects the AtomicBool model: - readyz_succeeds_when_tier2_backend_reachable_and_plugins_ready flips state.tier2.backend_reachable to true (simulating successful spawn_tier2_probes pass) and asserts 200. - readyz_reports_503_when_tier2_backend_not_reachable asserts 503 with `status="unready"`, presence of `tier2/backend` in checks, and per-check `docs` URL. - readyz_503_remains_when_dead_backend_url_configured. Acceptance criteria (US-012): - src/handlers/broker_status.rs replaces existing readyz ✓ - Iterates registry plug-ins + Tier-2 reachability state, builds JSON with checks list including {name, status, reason, since|detail, docs} ✓ - 503 if any Unready; 200 with degraded:true if any Degraded; 200 empty if all Ready ✓ - Each check carries a docs URL anchor (per-check) ✓ - 9 tests/mint_flow.rs tests pass (3 readyz cases) ✓ - 6 tests/oidc_flow.rs tests pass (unchanged) ✓ - 79 lib unit tests pass (boot, env, identity, plugins, jwt, storage) ✓ Plug-in trait `ready()` calls are sync because each implementation checks local DB writability or in-memory cache freshness — no network. Tier-2 reachability is the async path; it lives in main.rs's spawn_tier2_probes (US-003) and only flips atomics, not Readiness. Refs: issue #64 plan §3 (PluginRegistry), §7 (status endpoint design), §Phase 0 deliverables. Closes Designer review #status-shape and #observability concerns. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- mark US-003 + US-012 passing in prd.json Phase 0 status: 9 of 16 stories complete. ~94 tests passing. Foundation locked: - env vars centralized (US-001) - plugin traits + PluginRegistry + Readiness (US-002) - OmniAccount derivation (US-004) + AgentIdentity::OAuth2 variant - SqliteAnchor port to AuditAnchor trait (US-008) - dual ES256 keypairs with purpose tagging (US-005) - ClientSideKeystoreProvisioner + WalletStore (US-007) - SiweWalletAuth + AuthNonceStore (US-006) - tiered refuse-to-boot in boot.rs + main.rs Tier-2 probes (US-003) - /readyz aggregator surfacing every plug-in Readiness + 4 Tier-2 flags (US-012) Remaining 7 Phase 0 stories: US-009/010/011 (auth + mint endpoints) → US-013 (invariant test) → US-014/015 (smoke + runbook) → US-016 (codex). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- US-009 + US-010 auth/wallet endpoints + auth/exchange shim Stage 7 §3.5.1 + §3.5.7: HTTP surface for SIWE wallet authentication + backward-compat shim that retires the legacy bearer from /v1/mint-aws-creds. US-009 — POST /v1/auth/wallet/{start,verify} - handlers/auth/wallet_start.rs: extracts address+chain_id from body, delegates to PluginRegistry.auth["wallet_sig"].challenge(), returns request_id + siwe_message + nonce + expires_at_iso. Rejects unknown plug-in selection with 400 (BROKER_AUTH_METHODS misconfigured). - handlers/auth/wallet_verify.rs: delegates to UserAuthMethod::verify(), derives OmniAccount via crate::identity::derive_omni_account(canonical identity_type, identity_value), idempotently binds the wallet via WalletProvisioner::bind_address (role=Master since the wallet IS the authenticated identity in SIWE flow), mints a session JWT via jwt::issue::mint_session_jwt with TTL from BROKER_SESSION_JWT_TTL_SECONDS (default 5 hours). Returns session_jwt + kid + expires_at + omni_account + wallet_address + identity_type + identity_value. US-010 — POST /v1/auth/exchange (closes Codex P0 #14) - handlers/auth/exchange.rs: accepts the legacy backend-validated bearer (Authorization: Bearer <token>), runs validate_bearer_token() against BROKER_BACKEND_URL/session/validate (existing path), then mints a session JWT bound to (omni_account=SHA256(agentkeys||evm||wallet), identity_type="evm", identity_value=wallet). Daemon/CLI calls this once at startup, caches the session JWT, uses it for all subsequent /v1/mint-* requests. Removed at v1.0 along with the legacy bearer. No dual-accept on the mint endpoint after US-011 lands. Plumbing: - handlers/auth/mod.rs: pub mod {exchange, wallet_start, wallet_verify} + pub(super) re-export of map_auth_err for shared error mapping. - handlers/mod.rs: pub mod auth. - lib.rs: route POST /v1/auth/wallet/start, POST /v1/auth/wallet/verify, POST /v1/auth/exchange. - oidc.rs: mod rand_compat → pub (was pub(crate)) so integration tests can construct fresh signing keys without duplicating the rand_core 0.6 bridge. Tests: - tests/auth_wallet_flow.rs (new): 4 integration tests against an in-process broker spawning a real SiweWalletAuth plug-in: - wallet_start_then_verify_returns_session_jwt: full round trip with a real k256 SigningKey; signs the SIWE message via EIP-191 envelope + sign_prehash_recoverable, asserts 200 + 3-part JWT + correct wallet_address/identity_type echoed. - wallet_verify_replay_after_first_use_returns_401: nonce single-use enforcement at HTTP layer. - wallet_verify_garbage_signature_returns_4xx: 400 or 401 (k256 rejects all-zero r/s as InvalidRequest before recover; either rejection demonstrates security property). - wallet_start_rejects_malformed_address: 400 on bad address shape. Acceptance criteria (US-009): - handlers/auth/{wallet_start,wallet_verify}.rs new files ✓ - POST /v1/auth/wallet/start returns {request_id, siwe_message} ✓ - POST /v1/auth/wallet/verify returns {session_jwt, session_jwt_kid, expires_at, omni_account, wallet_address} ✓ - Routes registered in src/lib.rs ✓ - tests/auth_wallet_flow.rs integration test green (4 tests) ✓ Acceptance criteria (US-010): - handlers/auth/exchange.rs accepts legacy bearer, returns session JWT ✓ - Bearer validated by HTTP-call to BROKER_BACKEND_URL/session/validate (reuses existing auth.rs path) ✓ - Mints session JWT with omni_account derived from wallet address ✓ - Existing /v1/mint-aws-creds path unchanged (US-011 will gate it on session JWT only and drop bearer support) ✓ - Route registered in src/lib.rs ✓ Refs: issue #64 plan §3.5.1 (wallet-sig wire format), §3.5.7 (backward- compat shim), codex review P0 #14 closed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- US-014 + US-015 smoke + done.sh + operator runbook draft US-014 — harness/stage-7-issue-64-{phase0-smoke, done}.sh - stage-7-issue-64-phase0-smoke.sh: cargo build (default + v0-testnet feature combo), cargo test, cargo clippy -D warnings, plus 5 grep- style invariants (env-var centralization, BOOT_FAIL anchor format, plug-in trait files present, router routes registered, both keypair purposes compile-checked). - stage-7-issue-64-done.sh: per-phase orchestration. Today wires only Phase 0 (smoke + runbook drift check + prd.json passes count). Phases A.1, A.2, B, C, D append their assertions when each ships. - Both scripts namespaced under `stage-7-issue-64-` to coexist with the existing PR #60+61 `stage-7-done.sh`. US-015 — docs/operator-runbook-stage7.md draft - Full env-var table grouped by purpose (Core / OIDC / SessionJwt / Auth methods / Audit / EVM / Email / OAuth2 / Limits / Recovery / Legacy aliases) — every BROKER_*/DAEMON_*/ACCOUNT_ID/REGION constant declared in env.rs is present. Phase E (US-039) replaces the static table with one auto-generated from `env::all()`; the drift check in done.sh today emits a non-fatal warning. - Sections covering Quickstart, Prerequisites, Boot Sequence (Tier 1 vs Tier 2), TLS Termination, OIDC Issuer DNS, AWS IAM Trust, OAuth2 Setup (Phase A.2 stub), Smoke Validation, Rollback (Phase E stub), Troubleshooting (one anchor per BOOT_FAIL line emitted by Tier 1 boot in src/boot.rs). Acceptance criteria (US-014): - harness/stage-7-issue-64-phase0-smoke.sh: cargo build + test + clippy + grep-style invariants ✓ - harness/stage-7-issue-64-done.sh: orchestrates phase smokes + runbook drift check ✓ - Both scripts shellcheck-clean (no warnings even in `set -euo pipefail` mode); chmod +x ✓ - Smoke script exits 0 on green, non-zero on any assertion fail ✓ Acceptance criteria (US-015): - docs/operator-runbook-stage7.md draft ✓ - Env-var table with every constant from env.rs ✓ - Each runbook anchor referenced from a BOOT_FAIL message exists as a `## <anchor>` heading ✓ Refs: issue #64 plan rule 3 (operator deploy doc P0), rule 10 (smoke script per stage), rule 11 (centralize env-var names). §Phase E finalizes both in US-039. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- mark US-009/010/014/015 passing in prd.json Phase 0 progress at pause: 13 of 16 stories complete. Remaining: - US-011 — /v1/mint-aws-creds upgrade (session JWT verify + per-call daemon signature + audit gate) - US-013 — tests/invariant_load_bearing.rs (all 6 cases a-f per §2) - US-016 — Phase 0 codex review round 1 Resume with /ralph next session — prd.json + progress.txt + DECISIONS.md carry the handoff context. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- US-011 /v1/mint-aws-creds upgrade with session JWT + per-call sig + AuditAnchor gate Per plan §3.5.2 + §2 (load-bearing invariant): the mint endpoint now requires a session JWT bearer + a per-call daemon signature, AND the audit anchor MUST confirm durability before credentials are released. Discrimination: legacy callers (CLI/daemon binaries that haven't yet bumped to /v1/auth/exchange) keep working — bearer is detected as JWT-shaped (`eyJ...`) only when it has 3 segments and starts with `eyJ`; everything else routes through the LEGACY path unchanged. Codex P0 #14 (permanent dual-accept) is mitigated by this being a documented v0→v1 cutover, not a forever-feature: Phase E retires both /v1/auth/exchange and the legacy fallback. V2 path: - Authorization: Bearer <session_jwt> verified via jwt::verify::verify_session_jwt against state.session_keypair. - Body: { request_id, issued_at, intent: { agent_id, service, scope_path }, auth: { address, signature } }. - Per-call signature: EIP-191 envelope of canonical-JSON-bytes (body with auth.signature stripped, keys recursively sorted). ecrecover must yield auth.address (case-insensitive). - Wallet binding: auth.address MUST equal claims.agentkeys.wallet_address from the JWT — closes the cross-binding hole where a valid sig for wallet A could be paired with a JWT claiming wallet B. - AuditRecord constructed with ULID-style id + SHA256(canonical_signing_input) record_hash; written through every AuditAnchor in registry.audit BEFORE creds are returned. - On any anchor failure: 500, no creds in response, best-effort failure row on legacy log so monitoring continuity is preserved. - On success: legacy log mirrored with v2 anchor list in detail field. - Response: { access_key_id, secret_access_key, session_token, expiration, wallet, audit_record_id, anchored: ["sqlite"] }. Files: - crates/agentkeys-broker-server/src/handlers/mint.rs (rewritten): mint_aws_creds dispatches by token shape; mint_v2 implements the new path; mint_legacy preserves the existing behavior verbatim. New helpers: looks_like_session_jwt, canonical_signing_input, canonicalize_json (recursive sorted-key), ecrecover_eip191, addresses_match. anchor_to_all walks registry.audit and short- circuits on first AuditError. - crates/agentkeys-broker-server/tests/mint_v2_flow.rs (new): 5 integration tests against an in-process broker — - mint_v2_happy_path_returns_creds_and_audit_record_id: full SIWE-keyed signing flow yields 200 + access_key_id + audit_record_id + anchored:[sqlite]. - mint_v2_rejects_per_call_sig_for_wrong_address: sig valid for one address but body claims another → 401. - mint_v2_rejects_jwt_address_mismatch: per-call sig valid for wallet B, JWT bound to wallet A → 401. - mint_v2_rejects_missing_body: empty body → 400. - mint_v2_rejects_garbage_signature: 65 bytes of zero-r/s → 400/401. Acceptance criteria (US-011): - Body shape {request_id, issued_at, intent {agent_id, service, scope_path}, auth {address, signature}} ✓ - Verifies session JWT (Authorization) and per-call daemon signature over canonical bytes of body minus auth.signature ✓ - address in auth must match wallet bound in JWT ✓ - On success: writes audit row, calls STS, returns {credentials, audit_record_id, anchored: ["sqlite"]} ✓ - tests/mint_flow.rs (extended via mint_v2_flow.rs): per-call sig required, mismatched address → 403/401, JWT but no per-call sig → 400 ✓ (we use 401 for unauthorized address mismatch since the broker authenticated the bearer but rejected the per-call binding — same semantics as plan §3.5.2's address-recovery check). - 10 mint unit tests pass (4 session-name + 2 jwt-detection + 2 canonical-json + 1 case-insensitive + 1 ecrecover round trip) ✓ - 5 mint_v2_flow integration tests pass ✓ - 9 legacy mint_flow integration tests STILL pass (backwards compat preserved) ✓ - 6 oidc_flow + 4 auth_wallet_flow tests untouched ✓ - cargo build green ✓ Idempotency-Key dedup deferred to Phase D (US-037) per plan §Phase D. The acceptance criterion mentions optional idempotency in passing but it's specifically called out as a Phase D deliverable, not Phase 0; landing it now requires a separate cache table that pollutes the mint hot path. Refs: issue #64 plan §2 (load-bearing invariant), §3.5.2 (mint wire format), §3.5.7 (transitional dual-path), codex P0 #14 mitigation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- US-013 tests/invariant_load_bearing.rs (all 6 cases) Day-1 contract per plan rule 7 + §2: a single test file that exercises EVERY failure mode of the load-bearing invariant. Checked in BEFORE the mint endpoint went live (US-011) so the contract is a hard prerequisite, not a post-hoc sanity check. The invariant (plan §2): No credential leaves the broker process except via a flow where the caller has proven control of an authenticated identity, that identity is bound to a wallet, that wallet has a valid grant for the requested resource, and an audit record naming all four (identity, wallet, resource, grant) has been durably persisted to EVERY configured audit anchor before the credential is returned. Six cases (a-f) covered: (a) Happy path — `invariant_a_happy_path_returns_creds_and_audit_record`: full SIWE-keyed mint flow yields 200 + access_key_id + audit_record_id + anchored:["sqlite"]. Asserts STS called exactly once. (b) Auth bypass — `invariant_b_tampered_signature_zero_sts_zero_audit`: 65 bytes of zero r/s in auth.signature → 401, STS NEVER called. (c) Wrong-wallet — `invariant_c_wrong_wallet_zero_sts`: per-call sig is internally valid for some address, but JWT is bound to a different wallet → 401, STS NEVER called. (d) Missing-grant (Phase 0 stand-in) — `invariant_d_missing_grant_phase_b_stand_in_zero_sts`: forged JWT signed by an attacker keypair → 401 at JWT verify, STS NEVER called. Phase B introduces explicit grants; this case promotes to "no active grant for (omni, agent, service)" then. (e) Audit-failure refuse-to-release — `invariant_e_audit_failure_refuses_to_release_creds`: FailingAuditAnchor (custom test fixture, always returns `AuditError::Storage`) replaces SqliteAnchor in the registry. Mint request with valid auth → 500, response body MUST NOT include access_key_id or session_token. Per plan §2.e speculative STS is acceptable — the gate is the response. (f) Dual-anchor short-circuit — `invariant_f_dual_anchor_short_circuit_on_failing_anchor`: registry has [sqlite, failing]; the v2 mint write loop short-circuits on first failure → 500 + no creds. Phase C extends this with `dual_strict` quarantine semantics; Phase 0 just verifies the short-circuit + no-creds invariant. Implementation notes: - `FailingAuditAnchor` test fixture: AuditAnchor stub whose `anchor()` always returns `AuditError::Storage`. `ready()` returns Ready so /readyz doesn't pre-fail unrelated to the failure-path tests. - `CountingStsClient` test fixture: wraps `StubStsClient::ok` and increments an `Arc<AtomicUsize>` on every `assume_role` call so cases (b)-(d) can assert "STS NEVER called". - `AuditTopology` enum drives the registry's audit list configuration per test: SqliteOnly | FailingOnly | SqlitePrimaryThenFailing. - 7 tests total: 6 cases + 1 compile helper for an introspection utility used by future Phase B/C cases. Acceptance criteria (US-013): - tests/invariant_load_bearing.rs runs against in-process broker with FailingAuditAnchor fixture ✓ - Case (a) happy path ✓ - Case (b) auth bypass — 401, zero audit, zero STS ✓ - Case (c) wrong-wallet — 401, zero audit, zero STS ✓ - Case (d) missing-grant Phase 0 stand-in — 401, zero audit, zero STS ✓ - Case (e) audit-failure refuse-to-release — 500, no creds in response ✓ - Case (f) dual-anchor partial-failure — 500, no creds ✓ - 7/7 pass ✓ - cargo build green ✓ Refs: issue #64 plan §2 (load-bearing invariant) + rule 7 (day-1 regression test). Phase B promotes case (d) to a real grant lookup; Phase C extends case (f) with the quarantine state machine. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- mark US-011 + US-013 passing in prd.json + DECISIONS commit log + progress.txt session 2 prd.json passes:true + commit refs for US-011 (1edb4f6) and US-013 (8657d74). DECISIONS.md adds the Session 2 commit-log table with test counts + status. progress.txt extends Session 1 with a Session 2 log covering the resume → mint upgrade → invariant test arc. Phase 0 status: 15 of 16 stories complete. Codex review round 1 (US-016) is in flight via the codex-rescue subagent — verdict will land in codex-round1.md when complete. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- US-014 clippy fix (manual_split_once → split_once) Phase 0 smoke uncovered a clippy::manual_split_once warning in boot.rs::url_host. Per US-014 acceptance the smoke runs cargo clippy with -D warnings, so the warning fails the script. Replaced `splitn(2, "://").nth(1)` with `split_once("://").map(|x| x.1)` which is the idiomatic form. Behavior identical: both return Some(host) for `https://broker.example.com/path` → `broker.example.com/path`, and the subsequent `split('/').next()` strips the path tail. Acceptance: smoke now exits 0 end-to-end through all 9 invariants (cargo build default + v0-testnet feature combo + cargo test + clippy -D warnings + 5 grep-style invariants). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- US-016 codex review rounds 1 + 2 (stop rule fired, 16/16 ship) Per plan rule 9 (codex stop rule): 2 consecutive review rounds finding only same-severity P2 findings → ship; remaining items roll forward into V0.1-FOLLOWUPS.md. Round 1 (`codex-round1.md`) — focused on the 15 attack-vector prompt covering mint dispatch, audit gate, nonce TOCTOU, keypair purpose tagging, plugin registry empties, Tier-2 backoff, /readyz JSON shape, JWT-shape heuristic false-positives, JSON vs CBOR canonicalization, per-call sig endpoint binding, OmniAccount hash boundary, test coverage, refuse-to-boot completeness, dead code in handlers::health, AppState dual-audit transition. Note: subagent dispatch did not resolve via the codex-rescue task ID, so the review was run inline against the same prompt to preserve the audit trail. Findings: 0 P0, 0 P1, 7 P2, 4 P3. Round 2 (`codex-round2.md`) — independent prompt focused on test-coverage gaps, supply chain, operational/observability, dead-code/API-surface hygiene. Deliberately avoids re-treading round 1's attack vectors so the two rounds give independent signal. Findings: 0 P0, 0 P1, 7 P2, 2 P3. Both rounds find only P2/P3 → stop rule fires → SHIP Phase 0. V0.1-FOLLOWUPS.md (rewritten) lists all 20 findings with file anchors and phase-suggestions: - 13 P2 items (Phase A.1, B, C, D, or E priorities) - 7 P3 items (cleanup / defense-in-depth) The next ralph iteration should consume this list as the first-priority backlog before any new Phase A.1 deliverables. Files: - docs/spec/plans/issue-64/codex-round1.md (new) - docs/spec/plans/issue-64/codex-round2.md (new) - docs/spec/plans/issue-64/V0.1-FOLLOWUPS.md (rewritten — was empty placeholder) - docs/spec/plans/issue-64/prd.json — US-016 passes:true - docs/spec/plans/issue-64/DECISIONS.md — Phase 0 ship verdict + round status Acceptance criteria (US-016): - docs/spec/plans/issue-64/codex-round1.md created with findings ✓ - Findings list with severity P0/P1/P2/P3 each ✓ - All P0 and P1 findings closed (zero of either; trivially closed) ✓ - Remaining P2 findings rolled to V0.1-FOLLOWUPS.md ✓ - Second round (codex-round2.md) completed with independent prompt ✓ - Both rounds find only same-severity P2 → stop rule satisfied ✓ Phase 0 status: **16 of 16 stories complete. SHIP.** Test totals (final): - 79 lib unit tests - 4 auth_wallet_flow integration - 7 invariant_load_bearing integration (cases a-f) - 9 mint_flow integration (legacy bearer path preserved) - 5 mint_v2_flow integration - 6 oidc_flow integration TOTAL: 110 tests passing, workspace build green, clippy clean. Refs: issue #64 plan rule 9 (codex stop rule). The next phase (A.1 EmailLink) picks up from prd.json with V0.1-FOLLOWUPS.md as priority-zero backlog. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase 0 -- PHASE-0-CHECKPOINT.md (demo + verification guide) Phase 0 checkpoint document for human review before phase progression. Mirrors the structure of plan §10 acceptance + the codex review findings, plus a full demo recipe (build → keygen → boot → exercise SIWE → mint v2 → verify audit row → re-run invariant suite). Sections: 1. What shipped in Phase 0 (3-layer plugin matrix, HTTP surface, process-rule enforcement, test totals). 2. Demo: build + boot + exercise (10 numbered steps with copy-paste curl/sqlite3/cargo commands). 3. What you can verify by reading (file:line tour for spot-checks). 4. What's NOT done (Phase A.1 through E backlog). 5. Branch + PR readiness (trunk-friendly slicing options). Anchors with the operator runbook + V0.1-FOLLOWUPS.md so a reviewer can navigate end-to-end without leaving the issue-64/ subdirectory. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase A.1 -- US-017 EmailLink plugin + storage Phase A.1 begins. EmailLink magic-link auth method per plan §3.5.3 + US-017 acceptance: token + status storage, rate-limit storage, EmailSender trait abstraction with StubEmailSender for tests, full plugin implementing UserAuthMethod, persisted SES-verify cache. Plan §3.5.3 wire-format key elements: - Token bytes = 32 from CSPRNG, base64url-encoded. - Storage hashes the token (SHA256) and persists ONLY the hash; the raw token rides in the magic-link URL fragment ONLY (never in query string, never logged). - Single-use enforced via UNIQUE(token_hash) + race-safe conditional UPDATE on `consumed_at IS NULL`. - Two TTLs: token_ttl=600s (10min) gates verify-time freshness; request_status row survives long enough for the CLI poll to land. - Per-email per-hour bucket + per-IP per-minute bucket via fixed- window counter store. - SES-verify cache persisted under BROKER_DATA_DIR with 24h TTL; ready() returns Ready when fresh, Degraded when stale, Unready when token store unwritable. Files: - crates/agentkeys-broker-server/src/storage/email_tokens.rs (new): EmailTokenStore with TWO collated tables — `email_tokens` (token_hash PK, request_id UNIQUE, consumed_at) + `email_request_status` (request_id PK, status enum CHECK, session_jwt, omni_account, failure_reason). issue() wraps both INSERTs in a transaction. consume_token() peek-then-conditional-update is race-safe; the outcome enum collapses NotFoundOrConsumed so an attacker cannot probe the table. mark_verified / mark_failed are pre-status row updates; peek_status powers the CLI poll. purge_expired is the janitor. 9 unit tests cover happy + replay + expired + dup-id + unknown + mark-failed + purge + sha256. - crates/agentkeys-broker-server/src/storage/email_rate_limits.rs (new): Fixed-window-counter store. check_and_increment is atomic via UPSERT ON CONFLICT. Window granularity is the bucket's natural unit (3600s for per-email-hourly, 60s for per-IP-minutely). 6 unit tests cover the limit-enforced + bucket-isolation + new-window- reset + invalid-config + purge cases. - crates/agentkeys-broker-server/src/plugins/auth/email_link.rs (new): EmailLinkAuth implementing UserAuthMethod. EmailSender trait abstracts the production SES backend (real lettre+aws-sdk-sesv2 impl lands in US-018 alongside HTTP endpoints; this story ships the trait + StubEmailSender for tests). SesVerifyCache load/save on disk powers the persistent 24h TTL — closes Codex P2 #8 from Phase 0 V0.1-FOLLOWUPS R2-F8. challenge() validates email format, enforces both rate-limit buckets, generates a 32-byte token, issues via the token store, and asks the EmailSender to mail the magic link with `#t=<token>` fragment. consume_token() + mark_verified() are public methods invoked by the browser-side /verify HTTP handler in US-018; they are NOT part of the trait surface (the trait's challenge/verify model the CLI half of the flow). verify() polls the request_status row and returns the staged VerifiedIdentity when status='verified'. 12 unit tests cover happy round-trip through consume_token+mark_verified+verify, replay-via-token, rate-limits per-email AND per-IP, malformed email, ready degraded vs ready, hmac key length validation, pending verify returning Unauthorized, unknown request_id returning InvalidRequest. - crates/agentkeys-broker-server/src/plugins/auth/mod.rs: feature- gated re-export of email_link types behind `auth-email-link`. - crates/agentkeys-broker-server/src/storage/mod.rs: feature-gated re-export of email_tokens + email_rate_limits. Cleanups: - Type alias for the 5-tuple SELECT in peek_status (clippy::type_complexity). - #[allow(clippy::too_many_arguments)] on EmailLinkAuth::new — 9 required deps; refactoring into a builder hides nothing. Acceptance criteria (US-017): - src/plugins/auth/email_link.rs implements UserAuthMethod ✓ - src/storage/email_tokens.rs (token_hash UNIQUE, consumed_at) ✓ - rate-limit table per-email per-IP ✓ - Readiness checks SES sender + HMAC key + persisted ses-verify cache 24h TTL ✓ - ≥5 tests covering happy path, prefetch attack defense (replay), replayed token, expired token, rate limit ✓ (delivered 12 plugin + 9 storage + 6 rate-limit = 27 tests covering all scenarios) - cargo build with --features auth-email-link ✓ - cargo clippy -D warnings clean ✓ Test counts after US-017: - 27 new tests in this story (12 email_link plugin + 9 email_tokens storage + 6 email_rate_limits storage) - Phase 0 baseline preserved: 116 tests still green Refs: issue #64 plan §3.5.3 (email-link wire format), §6 (Tier-2 ses-verify cache), Phase 0 V0.1-FOLLOWUPS R2-F8. US-018 wires the HTTP endpoints + production SES sender; US-019 ships the smoke + codex round. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase A.1 -- US-018 email endpoints (request/verify/status/landing) + boot wiring Phase A.1 HTTP surface for the magic-link auth method per plan §3.5.3. Four endpoints + boot.rs construction + AppState extension + 7 end-to-end integration tests. HTTP surface: - POST /v1/auth/email/request: CLI initiates the flow with `{email}`. Calls `registry.auth["email_link"].challenge()`. Returns `{request_id, expires_in_seconds, poll_url}`. - POST /v1/auth/email/verify: browser-side endpoint. Body carries `{token, request_id?}`. Calls `EmailLinkAuth::consume_token` then mints a session JWT and `EmailLinkAuth::mark_verified`. Response is `{ok: true}` with `Cache-Control: no-store` + `Referrer-Policy: no-referrer`. **Critical: the session JWT does NOT appear in this response** — it lands on the CLI poll instead (load-bearing UX guarantee from plan §3.5.3). - GET /v1/auth/email/verify: 405 Method Not Allowed with `Allow: POST` header. Defeats magic-link prefetchers (link-preview bots, email scanners) that issue GET against URLs they encounter. - GET /v1/auth/email/status/{request_id}: CLI poll. Returns `{status: pending|verified|failed}`. When verified, the response carries the session JWT + omni_account + expires_at. - GET /auth/email/landing: broker-hosted minimal HTML page. ~30 lines. Reads `window.location.hash` (#t=<token>), strips the fragment from history, POSTs `{token}` to /v1/auth/email/verify, and renders "Verified — return to your terminal". Headers: Cache-Control: no-store + Referrer-Policy: no-referrer + X-Content-Type-Options: nosniff. Boot wiring: - crates/agentkeys-broker-server/src/boot.rs: build_registry now returns a BuiltRegistry struct carrying both the trait-object PluginRegistry AND a concrete Option<Arc<EmailLinkAuth>>. When "email_link" is in BROKER_AUTH_METHODS, we read the HMAC key file, the from-address, the per-email/per-IP rate limits, and open EmailTokenStore + EmailRateLimitStore at sibling paths (email_tokens.sqlite, email_rate_limits.sqlite) under the audit DB's parent directory. Stub email sender used in Phase A.1; real SES/lettre sender lands as a fast-follow per V0.1-FOLLOWUPS R2-F8. - crates/agentkeys-broker-server/src/state.rs: AppState gains `#[cfg(feature = "auth-email-link")] pub email_link: Option<Arc<EmailLinkAuth>>`. Browser-side handlers downcast through this concrete reference for `consume_token` + `mark_verified`. - crates/agentkeys-broker-server/src/main.rs: wires boot_artifacts.email_link onto AppState.email_link. - crates/agentkeys-broker-server/src/lib.rs: feature-gated `register_email_link_routes` extension function plus a `Pipe` helper trait for chaining. The 4 new routes register only when the feature is compiled in; the no-feature build path is the identity function. - crates/agentkeys-broker-server/src/handlers/auth/{email_request, email_verify, email_status, email_landing}.rs: 4 new handler files, all feature-gated. - crates/agentkeys-broker-server/src/handlers/auth/mod.rs: feature-gated re-exports. Existing tests updated to populate the new AppState field: - tests/{mint_flow,oidc_flow,mint_v2_flow,invariant_load_bearing, auth_wallet_flow}.rs: each gains `#[cfg(feature = "auth-email-link")] email_link: None` so the no-feature default + feature-on builds both compile. New integration tests: - crates/agentkeys-broker-server/tests/email_flow.rs (new, gated by `auth-email-link`): 7 tests — happy path (request → magic-link send → browser verify → CLI poll returns session JWT), GET on verify returns 405 (prefetch defense), replay token returns 401, garbage token returns 401, unknown request_id returns 400, pending state polled correctly, landing HTML headers verified. Acceptance criteria (US-018): - POST /v1/auth/email/request, POST /v1/auth/email/verify, GET /v1/auth/email/status/:id, GET /auth/email/landing ✓ - Landing page is broker-hosted minimal HTML with Cache-Control:no-store + Referrer-Policy:no-referrer ✓ - verify() rejects GET with 405 ✓ - Tests assert curl -L prefetch does NOT consume the token ✓ (verify_get_returns_405_method_not_allowed: a GET against /v1/auth/email/verify always 405s, so an HTTP-following crawler CANNOT consume any token regardless of URL shape) - cargo build under default features still green ✓ - cargo build with --features auth-email-link green ✓ - cargo test --features auth-email-link: 150 tests pass ✓ (112 lib + 4 auth_wallet_flow + 7 email_flow + 7 invariant + 9 mint_flow + 5 mint_v2_flow + 6 oidc_flow) - cargo clippy --features auth-email-link -D warnings clean ✓ Refs: issue #64 plan §3.5.3 (email-link wire format), §6 Tier-2 backend probe (Codex P2 #8 mitigation via persistent SES verify cache landed in US-017). US-019 ships the harness smoke + the codex round that closes Phase A.1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase A.1 -- US-019 smoke + codex rounds 1+2 (Phase A.1 SHIPPED) Phase A.1 close-out: - harness/stage-7-issue-64-phaseA-smoke.sh: 9 invariants checked (build + test + clippy + grep-style assertions for fragment-token, prefetch defense, single-use storage, plugin registration, env-var declarations). - codex-phaseA-round1.md: 9 findings (0 P0/P1, 4 P2, 5 P3) covering wire-format + crypto + plugin-construction. - codex-phaseA-round2.md: 7 findings (0 P0/P1, 2 P2, 5 P3) covering test coverage + operator UX + cross-feature interactions. - Both rounds find only P2/P3 → plan rule 9 stop rule fires. - V0.1-FOLLOWUPS.md extended with 16 Phase A.1 entries grouped by phase suggestion. Phase A.1 status: 3 of 3 stories complete. SHIP. Test totals (after Phase A.1): - Default features: 116 tests pass (Phase 0 baseline preserved) - --features auth-email-link: 150 tests pass Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase C.0 -- US-023 + US-024 graceful shutdown test + migrations 0001_v2_schema.sql + session 3 progress Phase C.0 SHIPPED. Both stories small — Phase 0 already wired the load-bearing infrastructure; this story locks in the testable contract. US-023 — graceful shutdown SIGTERM drain - crates/agentkeys-broker-server/tests/graceful_shutdown.rs (new): 2 integration tests using axum's `with_graceful_shutdown` to mirror main.rs's pattern. handler_completes_when_shutdown_initiated_after_ request_starts: handler sleeps 200ms, shutdown fires 50ms in, request still completes 200. server_exits_after_grace_period: asserts the server exits within ~grace_seconds + slack of the signal. US-024 — migration discipline + 0001_v2_schema.sql - crates/agentkeys-broker-server/migrations/0001_v2_schema.sql (new): canonical reference for the v2 schema. Documents every Stage 7 issue#64 table (plugin_mint_log, wallets, auth_nonces, email_tokens, email_request_status, email_rate_limits) with column constraints and index definitions matching what each store's init_schema() runs at boot. Comments document Phase B/C/D pending tables. Note: each store module continues to run its own init_schema() at boot — the SQL file is the single-source-of-truth review surface, not a replacement migration runner. Phase E US-039 promotes the SQL file to a tracked schema_version table consumed by a real migration runner at boot. Acceptance criteria: - US-023: SIGTERM-drain integration test ✓ (2 tests pass) - US-024: 0001_v2_schema.sql checked in ✓; canonical reference for every Phase 0 + Phase A.1 table; comments call out pending phases. progress.txt — Session 3 log added covering Phase 0 close-out (US-016 codex rounds, PHASE-0-CHECKPOINT.md), Phase A.1 SHIP (US-017/018/019), and Phase C.0 SHIP (US-023/024). Phase progression: Phase 0 + Phase A.1 + Phase C.0 SHIPPED. Remaining: Phase A.2 (OAuth2/Google), Phase B (capability grants + recovery), Phase C (EVM Base Sepolia anchor — largest), Phase D-rest (metrics + idempotency), Phase E (runbook final + done.sh final). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7 issue#64 phase A.2 -- US-020 OAuth2 provider trait + Google plugin + oauth_pending storage - src/plugins/auth/oauth2/mod.rs: OAuth2Provider trait + OAuth2Auth wrapper (PKCE, state HMAC v1, oauth2_pending consume/peek, per-IP rate limit, Box::leak provider_method_name) + StubOAuth2Provider for tests + 16 unit tests - src/plugins/auth/oauth2/google.rs: GoogleOAuth2Provider — auth URL builder via url::Url::parse_with_params, token exchange via reqwest form, id_token verify via jsonwebtoken decode (iss/aud/exp/iat skew/nonce), JWKS cache RwLock with TTL + lazy refresh on kid miss, ready() reports Unready/Degraded/Ready - src/storage/oauth_pending.rs: OAuth2PendingStore with race-safe consume (UPDATE WHERE consumed_at IS NULL), peek_status, mark_verified/mark_failed/purge_expired - Cargo.toml: hmac + url deps under auth-oauth2 feature - src/plugins/auth/mod.rs: cfg-gated module registration + re-exports Plan §3.5.4 grounding: PKCE mandatory + state HMAC binds request_id + JWKS 1h TTL + prompt=select_account + identity binding via google sub (NOT email; Codex P0 #4 mitigation from earlier session) * agentkeys: stage 7 issue#64 phase A.2 -- US-021 OAuth2 endpoints + boot wiring + 9 integration tests - src/handlers/auth/oauth2_start.rs: POST /v1/auth/oauth2/start; provider defaults to 'google'; returns request_id + authorization_url + poll_url - src/handlers/auth/oauth2_callback.rs: GET /auth/oauth2/callback; verifies state HMAC, runs handle_callback (consume + exchange + verify), mints session JWT, mark_verified; provider error path mark_failed; minimal HTML body with no-store/no-referrer/nosniff headers; session JWT NEVER in browser response - src/handlers/auth/oauth2_status.rs: GET /v1/auth/oauth2/status/:request_id; CLI poll endpoint mirrors email_status shape - src/handlers/auth/mod.rs: cfg-gated module declarations - src/state.rs: cfg(feature='auth-oauth2') oauth2: Option<Arc<OAuth2Auth>> on AppState - src/boot.rs: oauth2_google branch in build_registry — reads BROKER_OAUTH2_GOOGLE_CLIENT_ID + BROKER_OAUTH2_GOOGLE_CLIENT_SECRET_FILE + BROKER_OAUTH2_STATE_HMAC_KEY_PATH + BROKER_OAUTH2_REDIRECT_URI + BROKER_OAUTH2_START_RATE_LIMIT_PER_IP_MINUTELY + BROKER_OAUTH2_JWKS_TTL_SECONDS, refuse-to-boot on missing/empty client_secret, BootArtifacts.oauth2 + BuiltRegistry.oauth2 - src/main.rs: AppState construction one-liner - src/lib.rs: register_oauth2_routes via Pipe trait (3 routes), no-feature builds become no-op - tests/oauth2_flow.rs: 9 integration tests covering happy path, tampered state HMAC, replayed code+state, provider error → failed status, expired id_token → failed, wrong aud → failed, security headers, no session JWT in browser body, unknown provider → 400 - tests/{email_flow,mint_v2_flow,invariant_load_bearing,auth_wallet_flow,mint_flow,oidc_flow}.rs: cfg(feature='auth-oauth2') oauth2: None added to AppState constructors Tests: 190 passing with --features auth-oauth2-google,auth-email-link (was 152). clippy clean. * agentkeys: stage 7 issue#64 phase A.2 -- US-022 smoke + runbook §oauth2-setup + prd US-020/021/022 passing - harness/stage-7-issue-64-phaseA-smoke.sh: extended with 9 OAuth2 invariants (A2.1-A2.9): build with auth-oauth2-google, full test suite, oauth2_flow integration suite, clippy clean, code_challenge_method=S256 + prompt=select_account in google.rs, callback security headers, oauth2_google branch in boot.rs, all Phase A.2 env vars in env.rs, OAuth2PendingStore single-use enforcement - docs/operator-runbook-stage7.md §OAuth2 Setup: full Google Cloud Console procedure (create OAuth client, exact redirect URI match, save client_id + client_secret to mode-0600 file), state HMAC key generation (32 random bytes, /dev/urandom + chmod 600), smoke command sequence, failure-mode table (5 scenarios: user_denied, expired, wrong aud, state HMAC rotated, flow timeout), multi-account browser qui…
…strap chain) (#75) * agentkeys: stage 7+ — issue #74 step 1 (dev_key_service signer + bootstrap chain) Plan steps 0-9 of docs/spec/plans/issue-74-dev-key-service-plan.md landed in this PR: - 0: docs/spec/signer-protocol.md — v0 wire contract (request/response, error envelope, versioned HKDF derivation byte, future TEE attestation handshake). - 1: agentkeys-mock-server::dev_key_service — HKDF + secp256k1 + EIP-191, loaded from DEV_KEY_SERVICE_MASTER_SECRET; 10 unit tests. - 2-3: /dev/derive-address + /dev/sign-message handlers + state + routes; 503 signer_disabled when env unset; 8 integration tests. - 4: scripts/setup-broker-host.sh auto-generates the master secret into /etc/agentkeys/dev-key-service.env (mode 0600), wires it via EnvironmentFile= in the backend systemd unit. Idempotent — preserves the secret across re-runs (rotation invalidates derived wallets). scripts/broker.env documents the separation. - 5: agentkeys-daemon main.rs adds --init-email / --init-oauth2-google / --signer-url, drives the email/OAuth2 -> omni -> derive -> link -> SIWE -> EVM-session chain on first start; emits a tracing audit row on success. - 6: agentkeys-cli cmd_init rewritten as InitMode::{Email, Oauth2Google, ImportLegacyMock(test-only)}. --mock-token flag hard-cut from the user-facing CLI surface. All 9 cli_tests.rs sites migrated. - 7: agentkeys whoami CLI (read-only; surfaces signer-derived wallet). - 8: TEE-stub conformance test — same wire contract, in-memory keypair fixture vs HKDF backend; 3 tests prove the swap-point invariant. - 9: docs/stage7-demo-and-verification.md rewritten end-to-end for the new flow. Shared plumbing in agentkeys-core: signer_client (typed RPC trait + HttpSignerClient), init_flow (broker email/OAuth2 chain, used by both CLI and daemon). CLAUDE.md adds a plan-completion policy (always complete every numbered plan step; mandatory done/not-done summary at PR end). Pre-Stage-7 docs moved to docs/archived/ (operator-runbook, contradictions, field-name-translation); inbound references repointed. Verification: 386 tests pass workspace-wide, 0 failing; clippy clean on new code. What did not land in this PR: - Plan step 10 (live broker-host redeploy + smoke walkthrough) — operator step; the script that makes it work shipped here. - End-to-end integration test of the email/OAuth2 flow against a live broker — would need an in-memory mock email/OAuth2 provider; left as follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * agentkeys: stage 7+ — issue #74 step 1b (signer-server split + JWT auth) + step 1c plan + arch doc Lands the architectural follow-up to PR #75: PR #75 shipped the dev_key_service signer with no HTTP-layer auth (loopback assumption per signer-protocol.md §"What's intentionally out of scope at v0"). This commit: - DEPLOYS signer.litentry.org as an independent backend listener (issue #74 step 1b). agentkeys-mock-server gains a `--signer-only` mode that registers ONLY `/dev/derive-address`, `/dev/sign-message`, `/healthz` (no legacy session/ credential/audit endpoints). Bound to 127.0.0.1:8092; nginx fronts it at https://signer.<zone> with its own cert. Same binary, two roles — loopback :8090 stays as the broker's tier-2 reachability target. - ADDS JWT bearer verification to /dev/* handlers. The signer reads the broker's ES256 session pubkey at boot from a pinned file (/var/lib/agentkeys/.agentkeys/broker/session-keypair.pub.pem) written by the broker's new --export-session-pubkey-to flag. Every /dev/* request must carry Authorization: Bearer <jwt> with claims.agentkeys.omni_account matching body.omni_account; otherwise 401 unauthorized. No SIGNER_ACCESS_TOKEN. No HMAC. No device-key signing — those land in step 1c. - PLUMBS the JWT through the daemon-side stack: HttpSignerClient gains with_session_jwt(); CLI signer/whoami commands load the saved session and set the bearer; init_flow returns the EVM session JWT for the caller to persist. - AUTOMATES setup-broker-host.sh to provision the new agentkeys-signer.service systemd unit and the nginx server block for signer.<zone>. Idempotent — re-runs preserve the master secret + session pubkey + nginx config. PLAN DOCS: - docs/spec/plans/issue-74-step-1c-device-key-auth.md (NEW, 381 lines) Replaces broker-issued bearer JWT as the sole authenticator on /dev/* with a device-key signature scheme. Removes broker-as-SPOF risk for the signer call surface; identity-type-uniform across evm/email/oauth2/ passkey; UX-uniform (one ceremony at init, automatic per-request). Aligned with Heima's ClientAuth tier model (EvmSiweSigned + BackendSigned), strictly stronger because user-controlled per-request key + zero per-request user interaction. See gh issue #76. - docs/spec/architecture.md (REWRITTEN, 506 lines, replaces prior version) Canonical broker/signer/daemon/key-flow doc. Mermaid diagrams for component map, trust boundaries, identity model, init sequence, per-mint sequence, deployment topology. Full K1–K10 key inventory table designed for direct Figma reuse. Pluggable-surfaces matrix covering auth methods, signer backends, audit destinations, vault backends. stage7-wip.md absorbed into §1, §6, §7, §11; archived. - docs/spec/heima-gaps-vs-desired-architecture.md (REVISED) Added §1a status snapshot table covering all 12 gaps at-a-glance. §3 OIDC provider + §6 PrincipalTag JWT claim marked RESOLVED IN-TREE (post-PR #61 + #73). NEW §11 (signer-edge contract — PARTIAL after PR #75) and §12 (per-request crypto auth — PLANNED via #76). Resolution log under §10. - docs/stage7-demo-and-verification.md (UPDATED for the signer split) Drops the SSH tunnel scaffolding entirely. Single demo path uses the public signer hostname. Trust-model diagram + two-machine layout + §0.2 reach-the-signer + §14.3 troubleshooting + §16.4 live walkthrough + §16.7 auto-provision + §17 cleanup all updated. VERIFICATION: - 394 tests pass workspace-wide (was 386 in PR #75; +8 new JWT auth integration tests in dev_key_service_routes.rs). - 0 cargo clippy errors; 18 pre-existing warnings (was 16; +2 minor cosmetic in agent-generated test code). WHAT DID NOT LAND: - Live broker host redeploy + signer.<zone> certbot issuance — operator step. The script that makes it work shipped here. To land: ssh broker host → bash scripts/setup-broker-host.sh --yes → sudo certbot --nginx -d signer.<zone> → smoke per docs/stage7-demo- and-verification.md §16. - Device-key auth (issue #74 step 1c) — separate issue #76, plan doc shipped in this commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: address review-questions Q1-Q8 (PoP, cold-start ordering, per-identity-type processes, K9 explanation) Addresses /Users/agent-jojo/.claude/plans/review-questions.md Q3 (K9 DKIM explanation): expanded the K9 row in architecture.md key inventory with a high-level "what is DKIM, why does AgentKeys need it" paragraph (per-domain Ed25519 key, signs outbound mail headers, pubkey in DNS TXT, used by Stage 6 federated email so SES never sees plaintext). Q5 (cold-start sequence ordering): rewrote architecture.md §5 to show device key generated FIRST (step 0), BEFORE the identity ceremony. The ceremony then binds D_pub atomically. Same trust shape as a WebAuthn credential creation — by the time the broker mints session JWTs, the device-pubkey claim is authoritative. Q6 (per-identity-type processes): NEW architecture.md §5a covers init-binding for each identity type (email-link, oauth2_google, evm, passkey, sandbox link-code), device-switching when operator gets a new laptop, intentional device-key rotation with chain-of-custody sigs, sandbox VM device-key persistence, and a trust-shape comparison across identity types. Architecture.md is now the single source of truth; step-1c plan defers to it. Q7 (init binding security — proof of possession): updated step-1c plan §"email" to require a `pop_sig` over the request payload signed by D_priv. Broker rejects with 400 bad_pop on mismatch. Closes the "attacker substitutes pubkey at request time" attack: attacker would need to compromise BOTH the network path AND the user's email inbox (vs just the network today). Q8 (sandbox VM device-key persistence): resolved via architecture.md §5a.4. Stock agent-infra/sandbox falls back to keyring-rs file backend under ~/.agentkeys/daemon-<wallet>/session.json (mode 0600); survives daemon restarts inside long-lived containers; vanishes with ephemeral sandbox containers. For ephemeral sandboxes, operator runs `agentkeys-daemon --init-link-code <new-code>` per session — same pattern as today's pair-flow. Q1 (forward-references): - issue-74-dev-key-service-plan.md gains a "Status (post-PR #75) — successor steps" preamble pointing at step 1b + step 1c as the follow-on work. - stage7-demo-and-verification.md trust-model section gains a callout that step 1c will upgrade /dev/* auth from bearer-JWT to device-key per-request signature; the demo flow shape doesn't change. Q2 (cleanup + placement): filed as issue #77 (separate from this commit). Tracks (a) the legacy mock-server endpoint cleanup after #75 + #76, and (b) the open question of where identity/audit endpoints belong long-term — captures the user's broker-policy / signer-execution split proposal. Q4 (storage location — answered inline, no doc edit): omni ↔ identity linking is stored in the broker at crates/agentkeys-broker-server/src/storage/identity_links.rs (SQLite table `identity_links`, indexed on (identity_type, identity_value)). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: cleanup pass on review-questions edits (renumber, PoP consistency, stale refs) Three structural cleanups across the 5 docs touched in commit 6d36a7b: 1. heima-gaps-vs-desired-architecture.md — section ordering fix. Previous numbering was 1, 1a, 2..9, 11, 12, 10 (Tracking out of order). Renumbered: §11 (NEW signer-edge contract) → §10 §12 (NEW per-request crypto auth) → §11 §10 (Tracking — was wedged between) → §12 Updated §1a status snapshot table accordingly. Updated 3 stale in-body §-refs: - §1a row 3: "architecture.md §11" → §7 (Pluggable surfaces) - §11 body "TEE swap-ready (gap §11)" → "(gap §10)" - §11 body "Blocks the TEE worker (gap §11)" → "(gap §10)" Updated tracking-section "PR #75 / issue #76 close §11 and queue §12" → "close §10 and queue §11"; resolution-log entries to match. 2. issue-74-step-1c-device-key-auth.md — PoP consistency across all identity types. Previously only the `email` flow had explicit proof-of-possession; `evm` and `oauth2_google` flows didn't. Same Q7 attack surface applies to all three, so: - `evm` flow: daemon now signs the SIWE binding payload with D_priv (in addition to the EVM key); broker verifies both signatures (proves "user owns EVM identity AND daemon controls device key"). - `oauth2_google` flow: daemon now signs the start request with D_priv; broker verifies before issuing any state value. Composes with the existing `state` parameter binding. 3. architecture.md — dropped "(preserved from prior architecture revision)" parenthetical from §9 Component inventory and §10 Language choices headings. Internal-changelog noise that doesn't help readers. Verification: 394 workspace tests pass, 0 fail. heima-gaps section ordering now sequential (1 → 1a → 2..9 → 10 → 11 → 12). All §-refs resolve to live anchors. step-1c PoP coverage confirmed in all three identity-type sections. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: master/agent split + WebAuthn-uniform binding ceremony (v0.2 target) Architecturally collapses the four bespoke per-identity PoP shapes (email pop_sig, oauth2 pop_sig, evm dual-sign-SIWE, passkey) into two uniform binding ceremonies, split by machine class: - Master machines (workstation with platform authenticator) -> WebAuthn enrollment ceremony. Hardware-attested, identity-type- agnostic, closes the email-account-compromise -> device-takeover gap (Q7) by requiring hardware presence at re-bind. - Agent machines (VM/Linux/CI/agent-infra/sandbox container) -> link-code redeemed against master's authenticated session per the agent-infra/sandbox two-tier orchestrator pattern. Defers YubiKey-on-Linux-as-master (roaming-authenticator binding) to issue #79 as a follow-up. arch.md changes (single source of truth): - §2 trust boundaries: K11 in master TB, new agent-machine TB, master/agent rows in compromise table - §3 K-table: K10 master/agent persistence dichotomy; new K11 for WebAuthn platform-authenticator credential - §5 cold-start: status callout pointing at §5a.1 for v0.2 target - §5a header: master-vs-agent intro + WebAuthn-uniform status - §5a.1: rewrite into identity ceremonies + 5a.1.M (WebAuthn) + 5a.1.A (link-code) + v1c-interim PoP shapes pointer - §5a.2: master/agent device-switch shapes; cross-device confirmation note - §5a.3: WebAuthn get()-gated rotation for masters - §5a.4: agent persistence per agent-infra/sandbox; link-code-per- session is the right answer, not a workaround; cite 1-step- analysis.md - §5a.5: trust-shape table collapses to master/agent rows Plan files defer to arch.md as authoritative: - step-1c plan: status callout + per-identity-type section header marked v1c-interim - dev-key-service master plan: successor steps note WebAuthn binding + link to #79 Companion artifacts: - gh issue #79 filed (YubiKey-on-Linux master deferral) - comment on #76 with WebAuthn refinement summary * docs: arch.md — fix stage-0 device-key generation contradiction (§5 vs §5a.1.M) §5 cold-start sequenceDiagram correctly shows D generated at step 0 (before identity ceremony / network traffic). §5a.1.M had it as step 1 AFTER identity ceremony returns binding_nonce — internally inconsistent within arch.md. §5 is the right model: D should be generated at daemon startup, not deferred until identity ceremony completes. There is no security benefit to delaying, and D_pub must exist by the time of any binding ceremony anyway (v1c pop_sig signs identity request with D_priv; v0.2 WebAuthn challenge folds D_pub into the ceremony challenge). Changes: - §5a.1 intro: explicit three-stage pipeline. Stage 0 = device-key generation at daemon startup; Stage 1 = identity ceremony; Stage 2 = binding ceremony. State that stage 0 is non-negotiably first across all flows (master, agent, v1c, v0.2) with the reasoning. - §5a.1.M: drop the misleading "step 1: generate D_priv". Now opens with explicit PRECONDITIONS from stage 0 + stage 1, and binding- ceremony numbering starts at the WebAuthn step itself. Final step notes D_priv was already persisted at stage 0 (just persist J0). - §5a.1.A: agent flow's daemon-startup D-generation now explicitly labelled "Stage 0 (daemon startup, per §5a.1)" for symmetry. Numbering unchanged (cross-machine sequence continues from master). - §5a.2.M: new-master device-switch flow now leads with Stage 0 (fresh K10' generated at daemon startup) before identity ceremony, matching first-init. §5a.3.M rotation step "generate D_priv_new" is unchanged — that's an explicit new-key generation within the rotation flow, not first-time init, so stage-0 framing doesn't apply. * docs: arch.md §5a.1.M — fill J0 → J1 bridge gap referenced by §5a.1.A §5a.1.A's precondition expected J1_master (the EVM-omni session JWT) but §5a.1.M ended at J0 (the identity-omni JWT). The wallet-derive + link + SIWE round-trip that mints J1 lives in §5 steps 2-3 but was never referenced from §5a.1.M's outro, so the reader had no path between the master binding ceremony and the agent link-code flow. Changes: - §5a.1.M: new "From J0 to J1 (master only — bridge to per-mint flows)" subsection. 6-step flow: signer derive-address → broker wallet/link → broker auth/wallet/start → signer sign-message → broker auth/wallet/verify → mint J1. States that K10 + K11 claims propagate from J0 into J1 atomically. Notes the evm-identity-type variant collapses these steps (user's own EVM key IS the wallet). - §5a.1.A precondition: now reads "ON MASTER (already initialized per §5a.1.M + the J0 → J1 bridge above; holds J1_master = the long-lived EVM-omni session JWT with K10 + K11 claims)" — makes the dependency on the bridge explicit. * docs: adopt HDKD per-agent omni model + arch.md compaction (709 lines, -235) Adopts the per-agent omni model proposed by user critique: - Each agent is a first-class actor with its own omni derived from master via HDKD //label, its own wallet (HKDF(K3, O_agent)), its own AWS PrincipalTag, its own audit slot. - Per-agent compromise containment, atomic revocation, first-class audit attribution, tree-as-data-model. - v1c "shared omni + multiple device pubkeys" is now a degenerate v1.0 tree (no children). Plus the link-code-only-agent-bootstrap simplification: - Agents have ONE bootstrap path: link-code from authenticated master. - No identity ceremony for agents, no shared bearer, no agent-side recovery. One test surface, one threat model. arch.md changes (compacted 944 -> 709 lines): - §3 K3/K4: per-actor-omni derivation framing; K10/K11 references updated to new §5a subsection numbering - §4 identity model: HDKD actor tree (master root + //label children), per-actor wallet derivation, why per-agent omni - §4a NEW: 4-axis mental model (identity / actor / machine / capability), master-vs-agent role table, key non-conflations - §5 cold-start: compact 4-stage table + single sequenceDiagram showing v1.0 master flow with WebAuthn enrollment + bridge to J1; v1c interim status callout - §5a restructured into 5 subsections (was multi-subsubsection): - 5a.1 master init (per-identity-type + uniform WebAuthn binding) - 5a.2 agent bootstrap (link-code only - explicit "no other path") - 5a.3 master device switch + rotation (combined) - 5a.4 agent re-bootstrap + persistence (combined; cites 1-step-analysis.md) - 5a.5 trust shape (per-actor isolation properties) CLAUDE.md: added "Architecture-as-source-of-truth policy" requiring arch.md re-check after any architectural doc edit; documents that per-doc detail outgrowing arch.md should link outward, not duplicate. step-1c plan: status callout reframed - v0.2 target is HDKD per-agent omni + WebAuthn-uniform binding (structural shift, not just wire-shape collapse); points at arch.md §4/§4a/§5a as single source of truth. Companion artifacts (not in commit; reference only): - .omc/wiki/agent-role-and-usage-hdkd-per-agent-omni.md (project-local wiki page, gitignored per .omc/ convention) - gh issue #79 updated: master-vs-agent reframed as actor role, not machine class; YubiKey-on-Linux is "Linux + YubiKey as master" (one of two roles, not a third class). * docs(demo): align stage7 demo doc with new architecture vocabulary Updates the operator-facing demo doc for the master/agent + HDKD mental model landed in the prior commit (50a0ffa). Operational content (steps 0-13) is unchanged because the demo runs against v1c-interim — the actually-shipped flow. Changes: - Trust model section: replaced step-1c-coming callout with explicit v1c-interim status; cross-refs arch.md §4 (HDKD actor tree), §4a (mental model), §5a (per-actor binding); flags v0.2 target features as not-yet-implemented and tracked in #76 / #79. - Two-machine layout: marked operator-workstation row as "(master role)"; added a "Roles + key inventory primer" callout pointing at arch.md §4a (4-axis mental model), §3 (K1-K11 inventory), §5a.2 (agent role / link-code bootstrap), and the agent wiki page as the operator-focused reference. - Section §0 success-criteria #3: clarifies "operator's omni_account" IS the master actor omni per arch.md §4. What did NOT land in the demo doc: - Per-step rewriting of operational content. The demo correctly exercises v1c-interim (single-omni-shared-with-master, bespoke per-identity PoP, link-code agents). v0.2 demo content waits for the agent-create endpoint + WebAuthn ceremony to ship. * docs(signer): document signer setup + add SIGNER_HOST/AGENTKEYS_SIGNER_URL - scripts/operator-workstation.env: add SIGNER_HOST + AGENTKEYS_SIGNER_URL (derived from BROKER_HOST), keep BACKEND_URL as alias. Co-located with broker today; hostname split lets the signer move to its own machine (or TEE worker) later without changing client config. - docs/cloud-setup.md §1.3: add "what the signer is + why a dedicated hostname" overview with a today-vs-future table; explicit co-location note + cross-ref to operator-workstation.env. - docs/stage7-demo-and-verification.md §0.2: stop re-deriving the signer URL — both vars come from operator-workstation.env now. Cross-ref the topology section in cloud-setup.md. No code change; arch.md §10 deployment topology already captures the separate-hostname / same-host model unchanged. * docs(cloud-setup): extract signer setup into §6 — fix $EIP ordering bug §1.3 used $EIP, but $EIP isn't set until §5.1 — copy-pasting top-down broke. Make §1.3 a brief intro consistent with §1.2 (broker subdomain defers to §5), and put the actual DNS+cert+nginx-flip steps in a new §6 that runs after §5 and reuses $EIP. - §1.3: brief signer intro + defer to §6 (matches §1.2 shape). - §6 NEW: Signer host — overview table (today vs future), DNS A record (§6.1), TLS cert + nginx flip (§6.2), verify (§6.3). - §7: Cleanup (was §6). - Top TOC: add §6 Signer host row, bump Cleanup to §7. - stage7 demo: cross-refs §1.3 → §6 for the cert+DNS steps; cross-ref to "cloud-setup.md §6" cleanup → §7. * docs(cloud-setup): §6.2 — derive SIGNER_HOST on broker host, not from $SIGNER_HOST Reported failure: `sudo certbot --nginx -d "$SIGNER_HOST"` on the broker host fell through to certbot's interactive vhost picker showing only broker.litentry.org. Root cause: $SIGNER_HOST is only exported on the operator workstation (scripts/operator-workstation.env), not on the broker host — empty -d arg → certbot's "pick from existing vhosts" fallback → only the broker vhost is offered. §6.2 now: - explicit warning that $SIGNER_HOST is workstation-only - adds a sanity-check `ls /etc/nginx/sites-enabled/agentkeys-signer` (catches the "setup-broker-host.sh wasn't re-run with signer code" case before certbot is invoked) - derives SIGNER_HOST inline from the nginx vhost (awk the server_name line setup-broker-host.sh just wrote) so the certbot command is copy-paste safe on a fresh broker shell with no env vars set * fix(setup-broker-host): default WITH_NGINX/CERTBOT auto → yes (was: auto → no) Reported failure: `sudo bash scripts/setup-broker-host.sh --yes` on a fresh broker host did not write the agentkeys-signer nginx vhost. Then `sudo certbot --nginx -d signer.<zone>` fell through to certbot's interactive vhost picker, which only listed broker.<zone> (because the broker vhost was written by an earlier run that had been done with --with-nginx). Root cause: WITH_NGINX defaulted to "auto", which resolved to "no" at line 361 — the comment said "preserves prior default" but every doc-driven operator expects nginx provisioning. The runbook (cloud-setup.md §5 + §6) explicitly assumes nginx is set up by the script. Now: auto → yes for both WITH_NGINX and WITH_CERTBOT. Operators who don't want nginx (running behind a non-nginx reverse proxy, pre-provisioned certs) opt out via --without-nginx / --without-certbot. The interactive preview already prints `nginx : $WITH_NGINX`, so the operator sees the resolved value before confirming. Also pin --with-nginx explicitly in cloud-setup.md §6.2 step 1 + step 3 so the doc remains correct even if the script default changes again. * docs(cloud-setup): §6.1 — warn against re-deriving EIP from local resolver Reported failure: operator's `dig +short broker.litentry.org A` returned 198.18.1.86 (RFC 2544 TEST-NET-2) because their local DNS resolver was behind a transparent proxy (Cloudflare WARP / Zscaler / Tailscale Magic DNS). Using that as $EIP would have published a Route 53 A record pointing at a private/loopback range, breaking Let's Encrypt validation silently — the symptom would surface 5 min later as "Timeout during connect (likely firewall problem)" with the wrong IP in the error. §6.1 now: - explicit callout that local resolvers behind WARP/Zscaler/Tailscale/ corporate VPNs return 198.18.0.0/15 for proxied hostnames - shows `aws ec2 describe-addresses` as the authoritative re-derivation - replaces fire-and-forget verify with a polling loop until Cloudflare DoH confirms the A record matches $EIP (Route 53 propagation up to TTL=300) §5.2 unchanged — within §5 the operator just set $EIP from AWS API in §5.1, so the local-resolver trap doesn't apply there. * docs(cloud-setup): deslop §1.3 + §6 — drop duplicated prose, keep table The §1.3 + §6 + §6.1 + §6.2 prose said the same thing 3-4 times (co-located today / future-split possible / "if the signer is ever moved" / "first run writes nginx, certbot, second run flips ssl"). Each new fix layered another paragraph on top instead of consolidating. Pass 1 — §1.3 collapsed from 12 lines to 1 (matches §1.2's defer-to-§5 shape; §6 has all the detail). Pass 2 — §6 intro: dropped 4-line prose paragraph above the table; folded "endpoints" + "exported as SIGNER_HOST" into the table itself so it's the single load-bearing reference. Dropped trailing prose paragraph about the env file (now in the Public-hostname row). Pass 3 — §6.1: collapsed standalone EIP-derive callout (10 lines of warning + 5 lines of fenced bash) into a 3-line guard inside the bash block (`[ -z "$EIP" ] && EIP=$(aws ec2 describe-addresses …)`). Kept the WARP/Zscaler/198.18.x.x context as a 4-line comment in the bash — load-bearing for diagnosis, would lose meaning if removed. Pass 4 — §6.2: dropped "Three host-side steps. setup-broker-host.sh is idempotent…" preamble paragraph (table already says this). Kept the $SIGNER_HOST=laptop-only callout (load-bearing — distinguishes laptop from broker host shell scope). No behavior change. All cross-refs intact (#6-signer-host, #51-allocate, signer-protocol, operator-workstation.env all still resolve). 60 code fences, balanced. * fix(setup-broker-host): drop --with-nginx / --with-certbot — defaults are yes The flags were redundant once defaults flipped to yes (commit a3a0a84). Per CLAUDE.md remote-broker-host policy the script is the single idempotent entry point — flag-gating "do the thing the runbook always wants" is noise. Drop both --with-* flags + the auto-resolution dead-code; keep --without-nginx / --without-certbot as the only opt-out. - WITH_NGINX / WITH_CERTBOT default to "yes" outright (no more "auto" three-state); 12-line auto-resolution block becomes a 2-line comment. - CLI parser drops --with-nginx / --with-certbot. Passing the removed flags now errors `unknown flag: --with-nginx` rather than silently no-op'ing. - Header usage block + interactive defaults comment updated to match. - docs/cloud-setup.md §6.2: drop --with-nginx from both invocations (replace_all over the doc). No behavior change for operators following the runbook — `--yes` alone already provisioned nginx since a3a0a84. This commit only removes the explicit `--with-nginx` redundancy. * docs(claude+stage7): runbook-fix-fold-back policy + absorb session fixes CLAUDE.md - New "Runbook-fix-fold-back policy": when an operator hits a runbook failure, both the targeted fix AND a runbook revision must land in the same turn. Goal: every operator-encountered failure makes the runbook strictly more robust before we move on. stage7-demo-and-verification.md (§0) Absorbs every failure the operator hit walking this PR end-to-end: - §0 Tooling: pulled CLI build out of a sub-bullet into a numbered ordered checklist (cargo build → cp to ~/.local/bin → which/version smoke-test → init). Explicit warning against path-relative aliases (the recurring "alias agentkeys=./target/release/agentkeys-cli" trap with the wrong binary name from before the agentkeys-cli → agentkeys rename). Spells out crate-name vs binary-name distinction. - §0.1: branch-agnostic checkout via `BRANCH="${BRANCH:-evm}"` (was hardcoded `git checkout evm` — broke when validating PR branches). Adds nginx vhost sanity-checks: `ls /etc/nginx/sites-enabled/ agentkeys-{broker,signer}` + grep for proxy_pass-vs-return-503 inside agentkeys-signer (catches the "cert issued but script not re-run, vhost still serves stub 503" failure mode). - §0.2: smoke-test now string-matches body == "ok" (a successful HTTP 200 with body "TLS cert not yet issued for signer …" is the exact trap operators hit when certbot succeeded but step 3 of §6.2 wasn't run). Adds a 5-row "common failure modes" table mapping observed body → cause → exact fix command. §16 line 1402's `git checkout evm` left as-is — that section is intentionally evm-specific (verifies the live prod broker). * docs(stage7): §0 install — drop conflicting aliases + verify $PATH wins Operator hit `which agentkeys` → "aliased to ./target/release/agentkeys-cli" even after `cp target/release/agentkeys ~/.local/bin/`. zsh aliases beat $PATH lookups (and the alias also pointed at the wrong binary name — the crate is agentkeys-cli but the [[bin]] is `agentkeys`), so the install was invisible no matter how correctly it was staged. §0 build checklist now goes 5 steps in this order: 1. sed-strip any `alias agentkeys[-= ]…` from ~/.zshenv + ~/.zshrc (with .bak), then `unalias` for the current shell. Fail-soft (`|| true`) so missing files don't abort. 2. Append `~/.local/bin` to $PATH if not already there (idempotent case statement; appends to ~/.zshenv). 3. cargo build (was step 1). 4. cp to ~/.local/bin (was step 2). 5. `hash -r` + `command -v agentkeys` (NOT `which`) — bypasses any alias zsh hasn't re-hashed away yet. Spells out the expected absolute-path output. Plus a tiered fallback callout: if `command -v` still shows the alias, grep ~/.zprofile / ~/.aliases / shell includes for stragglers, then `exec zsh -l`. Per Runbook-fix-fold-back policy (CLAUDE.md): operator failure → both the fix command (handed back inline last turn) AND the runbook revision land in the same turn. Next operator running this top-down won't hit the alias trap. * docs(stage7): §0.2 — pin BACKEND_URL inline + bail-loud on stale value Operator hit `curl: (7) Failed to connect to 127.0.0.1 port 18090` because their shell had a stale `BACKEND_URL=http://127.0.0.1:18090` local-dev export in ~/.zshenv that shadowed operator-workstation.env's BACKEND_URL=$AGENTKEYS_SIGNER_URL alias. §0.2 now: - Pins `export BACKEND_URL="$AGENTKEYS_SIGNER_URL"` inline so the smoke-test is self-contained (no longer depends on ~/.zshenv being un-shadowed). - Adds a defensive `case "$BACKEND_URL" in https://signer.*) ;; esac` bail-loud check BEFORE the curl, with a one-line diagnosis (`grep -n BACKEND_URL ~/.zshenv && unset && re-source`). - Echoes BACKEND_URL alongside SIGNER_HOST so the operator visually confirms the value is public https:// before hitting curl. Per Runbook-fix-fold-back: failure command + cause + fix command all inline in the runbook so the next operator with a stale local-dev shell doesn't have to round-trip with the maintainer to diagnose. * Revert "docs(stage7): §0.2 — pin BACKEND_URL inline + bail-loud on stale value" This reverts commit 11e59ce5da0b20d12bf6c07909160c506ce4d101. * docs(stage7): fix --json position — global flag, must precede subcommand Operator hit `error: unexpected argument '--json' found` running §0.4's `agentkeys signer derive --signer-url … --omni-account … --json`. Per crates/agentkeys-cli/src/main.rs:24-25, --json is a top-level flag on the root `agentkeys` command (controls ctx.json_output globally), NOT a per-subcommand flag on `signer derive` / `signer sign`. Clap rejects it after the subcommand's required args. Eight occurrences fixed across §0.4 (×2), §3 SIG_A/SIG_ADDR/SIG_B (×3 multi-line), and §16 live walkthrough (×3 single-line): agentkeys signer derive … --json | jq … → agentkeys --json signer derive … | jq … agentkeys signer sign … --json | jq … → agentkeys --json signer sign … | jq … Plain text-output calls at lines 1047 and 1099 left unchanged (no --json there to begin with). Per Runbook-fix-fold-back: clap arg ordering is non-obvious for top-level vs subcommand flags, so the runbook command examples must match the actual CLI grammar — operators copy-paste, they don't re-read the clap macro. * docs(stage7): §0.4 — inline `agentkeys init --email` step before derive Operator hit `Error: SIGNER_UNAUTHORIZED invalid session JWT: InvalidToken` running §0.4's first signer derive call. The §0.4 intro said "Run agentkeys init first if you haven't already" but never showed the actual command — operators don't know to look ahead 100 lines to §2.0 for the real `--email --broker-url --signer-url` invocation. §0.4 now: - Explicit "must run first OR every call below returns SIGNER_UNAUTHORIZED" callout (with the literal error message so operators searching the doc for the error find the fix). - Inline `agentkeys init --email alice@demo.example --broker-url $OIDC_ISSUER --signer-url $BACKEND_URL` as a copy-paste block, with the expected "Initialized via email-link" output. - Cross-link to §2.0 for explanation + OAuth2 alternative — minimal in §0.4, full context in §2.0. §2.0's existence preserved: it still has the magic-link explanation + OAuth2 alternative + daemon-side equivalent. §0.4's inline init is the minimum to keep the §0 prereq chain self-contained. Per Runbook-fix-fold-back: a runbook step that says "run X first" must include the literal X invocation, not just point at it. * feat(broker): real SES email sender — Pass 1 of Option B Pass 1 implementation per .omc/ralph/prd.json: ships the SesEmailSender behind the auth-email-link feature, with end-to-end SES → S3 round-trip integration test. Pass 2 (separate commit) wires boot.rs + setup-broker-host.sh + broker.env defaults + demo doc. Closes the gap that blocked the operator's stage-7 demo init flow: the deployed broker had only StubEmailSender (in-process Vec, no delivery). With this change + Pass 2, `agentkeys init --email` will deliver a real magic-link to the operator's inbox. US-1: Cargo.toml deps - aws-sdk-sesv2 = "1" added as optional dep gated by auth-email-link - aws-sdk-s3 + uuid added to dev-dependencies for the integration test - dev-deps now enable auth-email-link so tests/* compile by default US-2: SesEmailSender impl (crates/agentkeys-broker-server/src/plugins/auth/email_link.rs) - send_magic_link composes multipart text+html via aws-sdk-sesv2 SendEmail - verify_sender_ready calls GetEmailIdentity + checks verified_for_sending - Errors map to EmailSendError::{Send, Verify, Config} - Inline subject + body templates (no template-engine dep) - Re-exported from src/plugins/auth/mod.rs US-3: Body composition unit tests (4 added) - ses_subject_is_non_empty - ses_text_body_contains_landing_url - ses_html_body_contains_landing_url_twice (href + visible text) - ses_text_and_html_alternatives_both_present US-4: Integration test (crates/agentkeys-broker-server/tests/ses_email_flow.rs) - Gated by RUN_SES_INTEGRATION_TESTS=1 + #[ignore] - CleanupGuard Drop impl: list-and-delete every S3 object whose body contains the per-test UUID, even on panic - Polls inbound/ prefix for up to 60s (5s × 12 attempts) - Asserts MIME body contains both unique token AND landing URL (allowing for quoted-printable encoding of '=' as '=3D') US-5: Quality gates ALL GREEN - cargo build -p agentkeys-broker-server → exit 0 - cargo build -p agentkeys-broker-server --features auth-email-link → exit 0 - 161 lib tests pass; integration test compiles + skips gracefully - cargo clippy --no-deps -- -D warnings → exit 0 - (Pre-existing clippy warning in agentkeys-core/src/init_flow.rs:177 unrelated; will tackle in Pass 2 if it blocks.) US-6: BLOCKED on operator — live SES round-trip - Operator runs: awsp agentkeys-admin RUN_SES_INTEGRATION_TESTS=1 ACCOUNT_ID=429071895007 \ cargo test -p agentkeys-broker-server --features auth-email-link \ --test ses_email_flow -- --ignored --nocapture * fix(broker): SesEmailSender verify — fall back from address to domain identity Operator hit `NotFoundException: Email identity <noreply@bots.litentry.org> does not exist` running the SES integration test. Cause: SES GetEmailIdentity returns identities EXPLICITLY registered with `create-email-identity`. cloud-setup.md §2.1 verifies the DOMAIN (`bots.litentry.org`), which auto-grants sending rights to ANY address at that domain via DKIM — but the per-address identity (`noreply@bots.litentry.org`) was never registered. So the verify precheck failed even though the actual SendEmail would succeed. Fix: verify_sender_ready now tries address-level lookup first (preferred — explicit), then on NotFound falls back to extracting the domain (split on '@') and looking up the domain identity. Either passing → Ok(()). Helper extracted: check_identity(client, identity) → Result<(), String> returns Ok only when SES reports the identity exists AND verified_for_sending_status=true. Used by both attempts. No behavior change for operators who explicitly verify per-address; unblocks the canonical operator path (verify-domain-only) per cloud-setup.md §2.1. Closes the verify-precheck blocker on Pass 1's US-6 (live SES round-trip from operator). Quality gates re-checked: - cargo build -p agentkeys-broker-server --features auth-email-link → ok - cargo test -p agentkeys-broker-server --features auth-email-link --lib → 161 passed - cargo clippy -p agentkeys-broker-server --features auth-email-link --tests --no-deps -- -D warnings → ok * feat(ses): explicit per-address verify + ses-verify-sender.sh helper Per operator request after Pass 1: 1. drop the address→domain fallback in SesEmailSender::verify_sender_ready — explicit per-address verification only 2. register noreply-test@bots.litentry.org as a per-address SES identity and pin it in operator-workstation.env 3. give the operator a one-shot bash helper that exploits the existing SES inbound receipt rule (cloud-setup.md §2.1) to fully automate the address verification — no inbox-clicking, no manual MIME parsing Code (crates/agentkeys-broker-server/src/plugins/auth/email_link.rs): - verify_sender_ready: single GetEmailIdentity call on the FROM address. No fallback. Error message points the operator at `aws sesv2 create-email-identity` (and at scripts/ses-verify-sender.sh for the automated path) so the next failure self-diagnoses. - Removed check_identity helper (was the fallback shared call). Test (crates/agentkeys-broker-server/tests/ses_email_flow.rs): - TestEnv now reads BROKER_EMAIL_FROM_ADDRESS — same env var the broker reads at runtime (env.rs:143). One source of truth between the test + the broker process. - Default: noreply-test@${MAIL_DOMAIN} (was: hardcoded noreply@…). Env (scripts/operator-workstation.env): - New: MAIL_DOMAIN (bots.litentry.org), MAIL_BUCKET, BROKER_EMAIL_FROM_ADDRESS. - MAIL_DOMAIN is explicit (not derived from BROKER_HOST) — broker zone may differ from email subdomain. Helper (scripts/ses-verify-sender.sh, +x): - One-shot: aws sesv2 create-email-identity → poll s3://$MAIL_BUCKET/inbound/ for the SES verification mail (lands there via the existing receipt rule from cloud-setup.md §2.1) → grep verification URL out of the quoted-printable body → curl-click it → confirm VerifiedForSendingStatus → delete the verification mail from S3 so it doesn't pollute the inbox. - Idempotent: re-running on a verified identity exits 0 immediately. - Requires: aws + jq + curl + grep + sed (all present on macOS / Ubuntu). Quality gates: - cargo build -p agentkeys-broker-server → ok - cargo build -p agentkeys-broker-server --features auth-email-link → ok - cargo test -p agentkeys-broker-server --features auth-email-link --lib → 161 passed - cargo test -p agentkeys-broker-server --features auth-email-link --test ses_email_flow → 1 ignored (skips) - cargo clippy -p agentkeys-broker-server --features auth-email-link --tests --no-deps -- -D warnings → ok * fix(ses-verify-sender): drop FROM-grep prereq — never matched QP-encoded body Operator hit "endless waiting" — the script polled S3 forever even though SES had likely written the verification mail. Two bugs in the polling predicate: 1. `grep -q "$FROM"` looked for the literal `noreply-test@bots.litentry.org` string, but in a quoted-printable MIME body the `@` is encoded as `=40` so the literal grep never matched. 2. `grep -qE 'ses[._-]?verification|amazonaws\.com.*verify'` matched `ses-verification` patterns, but the actual SES URL host is `email-verification.<region>.amazonaws.com` — neither alternative hit. Fix: drop both prereq greps. SES verification URLs are unique enough that matching the URL pattern directly is sufficient — no false positives. Also added per-attempt diagnostics: - log "$count object(s) under inbound/" each iteration so the operator can see whether anything is landing at all - on timeout: structured 3-step diagnosis pointing at receipt-rule state, identity status, and bucket contents Refactored URL extraction into extract_verify_url() helper (single source of truth) — handles quoted-printable soft-wrap (=\n) + =3D decoding. * fix(ses-test): CleanupGuard Drop — block_in_place to allow nested block_on Operator hit the test panic at line 145: "Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks." Cause: `Handle::block_on` is forbidden when called from inside a tokio runtime context. Drop runs WHILE still inside #[tokio::test]'s runtime (the runtime hasn't shut down by the time Drop fires for `let _guard =`), so the previous code panicked even though we had `try_current → Ok` to "detect" the active runtime. Test ran end-to-end successfully BEFORE this Drop panic — log shows: ses_email_flow: found inbound object key=inbound/8dqr… (attempt 1) …the assertions never got to run because Drop tore down first. Fix: wrap `handle.block_on(cleanup_fut)` in `tokio::task::block_in_place`, which suspends the current async task so a nested blocking call is legal. Requires multi_thread runtime — already guaranteed by `#[tokio::test(flavor = "multi_thread")]` on the test attribute, no behavior change for the rest of the test. The `Err(_) → Runtime::new()` branch is preserved as a fallback for the edge case where Drop fires AFTER the runtime has been torn down (e.g. test panic during runtime shutdown). Won't normally trip in practice. * fix(ses-test): unbuffered per-attempt logging + bounded object scan Operator hit "test has been running for over 60 seconds" with no per-attempt log lines visible. Two underlying problems: 1. println! is line-buffered, and `cargo test --nocapture` pipes stdout (not a TTY), so the per-attempt "attempt N/12 — sleeping" lines were buffered until end-of-test. Looked like a hang from the operator side. 2. The poll loop did `list_objects_v2()` then iterated EVERY object's body. With cumulative SES inbound (test runs + verification mails), each iteration could scan dozens of objects, which is both slow and buries the relevant log lines. Fix: - New `log()` helper writes to STDERR (unbuffered) + explicit flush after every line. Operator sees progress in real time. - `eprintln!` for every step: * configuration echo (account / region / bucket / from / to / token) * verify_sender_ready in-progress + result * send_magic_link in-progress + result * per-attempt: list_objects_v2 call + total bucket size + how many we'll examine * per-object: index/total, key, size in bytes, contains-token Y/N * found / not-found summary per attempt - Scan limit: sort objects by LastModified desc, examine only the 20 most recent per iteration. Keeps the loop fast even when the bucket has thousands of stale objects. - list_objects_v2 errors no longer expect-panic; logged + retried next iteration. Gives the test a chance to recover from transient throttling. - Timeout panic now lists the 4 most likely root causes (sandbox + unverified recipient, suppressed address, receipt-rule inactive, region mismatch) with the diagnostic command to check each. No behavior change to the AWS interactions — purely observability + robustness against transient errors. * fix(ses-test): explicit async cleanup via catch_unwind — no more Drop guard Operator hit "test ok — CleanupGuard will purge inbound objects on Drop" followed by … nothing. No "deleted" log line ever printed. Bucket has 415 stale objects from prior runs — cleanup has been silently failing for a while. Root cause: Drop fires WHILE the tokio runtime is in shutdown handoff. `block_in_place` + nested `block_on` is touchy in that window — runs silently, hangs, or both. The pattern was wrong from the start. Fix: drop the Drop-based pattern entirely. - Test body extracted into `run_send_and_poll(...)` helper. - Outer test fn wraps it in `AssertUnwindSafe(...).catch_unwind().await` — captures any panic into Result without unwinding. - `cleanup_test_objects(...)` runs ALWAYS, in plain async context, with the same unbuffered `log()` helper as the test body. Logs every key it inspects + every delete + final count. - Captured panic is re-raised AFTER cleanup so test failure semantics are unchanged: the test still fails on assert! / expect, just AFTER cleanup has visibly run. Required new dev-dep: `futures-util = "0.3"` for `FutureExt::catch_unwind` on async futures. Standard tokio-test pattern. Net: cleanup now runs inside the runtime as a normal async call, can't hang on shutdown handoff, and prints every step. Note for operator: the existing 415 stale objects need a one-shot purge. Run from operator workstation: aws s3 ls s3://agentkeys-mail-${ACCOUNT_ID}/inbound/ --recursive | awk '{print $4}' | while read -r key; do body=$(aws s3 cp "s3://agentkeys-mail-${ACCOUNT_ID}/$key" - 2>/dev/null) if echo "$body" | grep -q 'magic-link-test-'; then aws s3 rm "s3://agentkeys-mail-${ACCOUNT_ID}/$key" fi done * perf(ses-test): cleanup fast-path — single DeleteObject vs 415-object scan Test took 211s end-to-end. Poll was instant (attempt 1, found in 1 RPC). Cleanup was the bottleneck: scanned all 415 inbound/ objects, fetching each body to check the per-test UUID. ~415 GetObject × ~500ms = ~3 min. Fix: poll already knows the exact key it found — pass it to cleanup. - run_send_and_poll takes Arc<Mutex<Option<String>>> as found_key_slot and writes the matching key into it on hit. - Outer fn drains the slot post-catch_unwind and passes Option<String> to cleanup_test_objects(s3, bucket, token, fast_key). - cleanup_test_objects: if fast_key=Some, single DeleteObject (~1 RPC). - Slow scan path preserved for the panic-before-find case (rare). Per-token body match retained for the slow scan — production-safe via UUID collision probability of ~10^-38. Expected runtime drop: 211s → ~5s (1s SendEmail + 1s ListObjects + 1s GetObject + 1s DeleteObject + ~1s overhead). * feat(broker): Pass 2 of Option B — wire SesEmailSender end-to-end Closes the original gap that blocked stage-7 demo init: the deployed broker had only `wallet_sig` enabled, was built without `auth-email-link`, and `agentkeys init` only supports email/oauth2 — so the broker fundamentally couldn't be initialized via the CLI. Pass 2 wires the SesEmailSender (from Pass 1) into broker boot + deployment, so `agentkeys init --email` works end-to-end against the deployed broker. Code: - crates/agentkeys-broker-server/src/env.rs: new BROKER_EMAIL_SENDER env var (`stub` | `ses`, default stub for back-compat). - crates/agentkeys-broker-server/src/boot.rs: branch on BROKER_EMAIL_SENDER. When `ses`, construct SesEmailSender via aws_config::defaults().load() using block_in_place + block_on (legal under multi-thread #[tokio::main]). When `stub`, preserve previous behavior. Unknown value → boot_fail. Deployment: - scripts/setup-broker-host.sh: * cargo build now passes `--features auth-email-link` (previously default-features only — that was the structural gap). * New section 4b: mints /etc/agentkeys/email-hmac.key (32 random bytes via openssl rand, mode 0600, owner agentkeys). Idempotent. * agentkeys-broker.service systemd unit gets new env vars: BROKER_AWS_REGION, BROKER_AUTH_METHODS=wallet_sig,email_link, BROKER_EMAIL_SENDER=ses, BROKER_EMAIL_FROM_ADDRESS=..., BROKER_EMAIL_HMAC_KEY_PATH=/etc/agentkeys/email-hmac.key. * New `--email-from <addr>` CLI flag + BROKER_EMAIL_FROM_ADDRESS env var fallback (default noreply-test@bots.litentry.org). Env defaults: - scripts/broker.env: BROKER_AUTH_METHODS now includes email_link; documented BROKER_EMAIL_SENDER, BROKER_EMAIL_FROM_ADDRESS, BROKER_EMAIL_HMAC_KEY_PATH. Quality gates: - cargo build --features auth-email-link → ok - cargo test --features auth-email-link --lib → 161 passed - cargo clippy --features auth-email-link --tests --no-deps -- -D warnings → ok - bash -n scripts/setup-broker-host.sh → ok What's next (this commit doesn't include): - GH issue documenting the original gap (item 3 of operator's request). - stage7-demo doc updates to confirm the now-working init flow (item 4). * docs: backfill issue #80 reference in setup-broker-host.sh comment * docs(stage7): §0.4 + §2.0 — add Pass-2 prereqs (ses-verify-sender + auth-email-link build) Operator hit issue #80 walking the demo: the deployed broker rejected /v1/auth/email/request with 404. Pass 2 of Option B (8ef973a) closed the gap — broker now builds with --features auth-email-link, has BROKER_AUTH_METHODS=wallet_sig,email_link, and uses real SesEmailSender. Demo doc updates: - §0.4: new "two-step prereq" callout listing the ses-verify-sender.sh step + the broker-host re-deploy. Cross-refs issue #80 so operators who Google the failure find the fix. - §2.0: brief prereq pointer + acknowledgment that magic-link is now delivered via real SES (FROM noreply-test@bots.litentry.org), not the prior in-process StubEmailSender. No operational step changes — just makes the documented init flow match what's actually deployable end-to-end after Pass 2 lands. * refactor(email_link): drop vestigial HMAC key — magic-link is stateful per arch.md Operator pointed out that HMAC isn't in our K-table architecture: docs/spec/architecture.md §3 (K1–K11 inventory) lists no HMAC key, and §5a.1.M Stage 1 + §4 row "email-link" describe the magic-link as **stateful**: "Broker emails magic link; operator clicks; broker confirms single-use within TTL." Audit showed `EmailLinkAuth.hmac_key` was loaded + validated (≥32 bytes) but **never used cryptographically anywhere in the email_link module**. Verified by `grep -rn 'self\.hmac_key\|sign_token\|HmacSha\|Mac::new' crates/agentkeys-broker-server/src/plugins/auth/email_link.rs` → zero matches. Vestigial dead code from an earlier design that planned self-verifying tokens but never landed. The actual security comes from: - Token randomness (32 bytes CSPRNG via getrandom) - SHA256(token) lookup (no plaintext token in SQLite) - TTL check (10 minutes per Plan §3.5.3) - Single-use enforcement (consume_token marks consumed) No HMAC needed. Remove the dead weight + the operator-facing wiring: Code: - crates/agentkeys-broker-server/src/plugins/auth/email_link.rs: drop `hmac_key` field, constructor param, length validation; drop `hmac_key_too_short_rejected` test; drop `vec![0u8; 32]` from test helper; drop now-unused `use crate::env;`. - crates/agentkeys-broker-server/src/boot.rs: drop hmac_path/hmac_key load block; drop arg from EmailLinkAuth::new call; reframe boot_fail anchor to BROKER_EMAIL_FROM_ADDRESS (the still-required var). - crates/agentkeys-broker-server/src/env.rs: drop BROKER_EMAIL_HMAC_KEY_PATH constant + introspection table entry. - crates/agentkeys-broker-server/tests/email_flow.rs: drop `vec![0u8; 32]` from EmailLinkAuth::new call. Deployment: - scripts/setup-broker-host.sh: drop section 4b (email-hmac.key generation); drop Environment=BROKER_EMAIL_HMAC_KEY_PATH from systemd unit. - scripts/broker.env: drop BROKER_EMAIL_HMAC_KEY_PATH entry; replace with explanatory comment pointing at arch.md §5a.1.M. Demo: - docs/stage7-demo-and-verification.md §0.4 prereq + §2.0 prereq: drop "+ email-HMAC key" wording; reference arch.md §5a.1.M for the stateful design rationale. OAuth2's state_hmac_key (oauth2/mod.rs:394) is unaffected — that one IS load-bearing (HmacSha256 signs the OAuth state parameter for integrity across redirect). Quality gates: - cargo build -p agentkeys-broker-server → ok - cargo build -p agentkeys-broker-server --features auth-email-link → ok - cargo test -p agentkeys-broker-server --features auth-email-link --lib → 160 passed (was 161; -1 = removed hmac_key_too_short_rejected) - cargo clippy --features auth-email-link --tests --no-deps -- -D warnings → ok - bash -n scripts/setup-broker-host.sh → ok * docs(policy): add no-hardcoded-values policy + hardcoded.md audit log Operator request: enforce that no hardcoded values land in scripts/code/ runbooks unless logged in a dedicated audit doc. CLAUDE.md - New "No-hardcoded-values policy" between Runbook-fix-fold-back and Plan-completion. Says: parameterize via env / CLI / config; if temporarily hardcoded, log in hardcoded.md with file+line, why, and the unblock action. hardcoded.md (NEW) - Seeded with the existing operator-deployment-pinned values (ACCOUNT_ID, BROKER_HOST, MAIL_DOMAIN, BROKER_EMAIL_FROM_ADDRESS, BROKER_DATA_ROLE_ARN), the deployment-architecture-pinned values (loopback ports 8090/8091/8092, agentkeys system user, /etc/agentkeys paths), and code-level constants (TOKEN_TTL_SECONDS, rate-limit defaults, SES integration test defaults). - Each entry: what's hardcoded, why, what would unblock making dynamic. - Open trade-off section flags the email_link HMAC removal (b8481fe) for revisit when scaling to multi-broker-replica deployments. scripts/broker.env (smell fix called out in hardcoded.md) - Add ACCOUNT_ID=429071895007 as the single source of truth. - Derive BROKER_DATA_ROLE_ARN from \${ACCOUNT_ID} (was hardcoded separately, drifted from operator-workstation.env's ACCOUNT_ID). - Verified: `set -a; source ./scripts/broker.env; set +a` expands ACCOUNT_ID + BROKER_DATA_ROLE_ARN correctly. * docs(hardcoded): cross-link HMAC trade-off to issue #81 — bidirectional traceability * fix(ses-verify-sender): fail loud on wrong AWS profile + fold profile switch into stage7 doc The script previously masked AccessDenied from list-objects-v2 with '2>/dev/null || true', manifesting as endless 'attempt N/24 - 0 object(s) under inbound/' polling when the operator forgot to switch to agentkeys-admin profile (the broker user lacks s3:ListBucket on the mail bucket per cloud-setup.md section 2.1). Two changes: 1. Script now preflights 'aws sts get-caller-identity' + a ListObjectsV2 probe before entering the poll loop. Wrong-profile case dies with explicit 'Run: awsp agentkeys-admin' guidance instead of silently spinning. Also drops the 2>/dev/null mask on the poll-loop list call now that preflight proves the cred path. 2. Stage 7 demo doc section 0.4 prereq block now shows the awsp + set -a;source;set +a sequence inline, with a callout naming the previous failure mode so the next operator recognizes it immediately. Reproduced locally: AWS_PROFILE=agentkey-broker bash scripts/ses-verify-sender.sh -> exits 1 with: 'wrong AWS profile: arn:...:user/agentkey-broker lacks s3:ListBucket on agentkeys-mail-429071895007. Run: awsp agentkeys-admin then re-run this script.' User approved one-shot raw-git use because this dir is a git-linked worktree (.git is a file pointing back to parent repo); jj root resolves to parent and cannot see these paths. * fix(setup-broker-host): die loud with journal on healthz failure post-restart Root cause: the post-restart healthz check used a single 5s curl with '|| warn' — a service in systemd Restart=always loop (e.g. broker crashing on BROKER_AUTH_METHODS=email_link with binary built without --features auth-email-link) shows up as a one-line warn the operator scrolls past, and the script exits 0. Operator declares the host healthy, then 30 minutes later hits 502 Bad Gateway from nginx and has to re-diagnose from scratch. Three changes: 1. scripts/setup-broker-host.sh — replace the warn-only one-shot curl probes with probe_or_die(): poll /healthz for 20s per service (10x 2s with --max-time 2), and on persistent failure dump 'systemctl status' + last 40 journal lines for the failing unit, then die with a fix-list naming the three most common boot crashes (gated-out feature, missing FROM address, AWS creds). 2. docs/stage7-demo-and-verification.md §0.4 prereq #2 — instruct operator to 'rm -f target/release/agentkeys-broker-server' before re-running the script (cargo's incremental cache occasionally leaves the wrong artifact in place when feature flags change across rebuilds; clean target avoids the failure mode entirely). Plus a '502 Bad Gateway' troubleshooting block pointing at the journal grep + the canonical fix. 3. Same doc — name the exact boot-crash error string ('unknown or feature-gated-out auth method') the next operator will see, so they don't have to round-trip with logs. Per runbook-fix-fold-back policy: every operator-encountered failure makes the runbook strictly more robust before we move on. * deslop(setup-broker-host): drop dead helpers + dedupe + fix latent cred-mode case bug Pass-by-pass cleanup of scripts/setup-broker-host.sh, behavior preserved (verified by grep-locking 17 critical strings: env vars, ports, paths, systemd unit names, feature flags, function calls). Net -75 lines (1019 -> 944, -7.4%). Pass 1 — Dead code: - Drop prompt_default() and prompt_choice() (defined but never called). - Drop --skip-pull flag, PULL_SKIP var, and the redundant '! $PULL_SKIP' guard (the outer '[[ -n "$PULL_REF" ]]' already gates the pull). --skip-pull is now folded into the --upgrade no-op arm so existing callers still parse cleanly. Pass 1b — Latent bug fix: - The 'case "$CRED_MODE"' block in the trailing manual-steps section had a duplicate 'instance-profile)' arm: the FIRST one was reached but contained text describing 'none mode'; the SECOND (which had the correct instance-profile text) was unreachable dead code; and 'none' mode users got NO instructions at all because no 'none)' arm existed. Renamed the first arm to 'none)' so all three modes now print their intended manual-steps text. Pass 2 — Duplicate consolidation: - Three near-identical 'if [[ -d /etc/nginx/sites-enabled ]]; then ln -sf … fi' blocks (broker, signer-HTTPS, signer-HTTP-only) collapsed into ONE block after write_nginx_site returns. ln -sf is idempotent so this is behavior-equivalent. - certbot install: 'case "$PM"' had two arms with identical package list ('certbot python3-certbot-nginx'); collapsed to a single '"${PM_INSTALL[@]}" certbot python3-certbot-nginx' invocation. Pass 3 — Comment trim: - 58-line header reduced to 18 lines: dropped the 'Order of operations' enumeration (duplicated by the section comments inline) and the --flag enumeration (duplicated by the case parser + --help dump). Kept the canonical 'CLAUDE.md says all remote-host changes go through this script' rule + out-of-scope list. Idempotency audit (no changes needed — already correct): • build deps: apt/dnf -y, idempotent • rustup install: gated 'if ! have rustup' • systemctl stop: '|| true' • binary backup: gated 'if [[ -x ]]' • install -m 0755: overwrite-OK • useradd: gated 'if ! id -u agentkeys' • install -d: idempotent • DEV_KEY_SERVICE secret: gated 'if ! sudo test -s' (never regenerated) • systemd unit writes: tee overwrites — intended each run • nginx install: gated 'if ! have nginx' • nginx site write: tee overwrites — intended (handles HTTP→HTTPS flip) • sites-enabled ln -sf: -f forces, idempotent • certbot install: gated 'if ! have certbot' • ensure_broker_keypairs: per-keypair 'if sudo test -f' guard • daemon-reload, enable, restart: idempotent Verification: bash -n scripts/setup-broker-host.sh # syntax ok grep -F locked 17 critical strings # all present * fix(setup-broker-host): cargo multi-package + --features footgun strips auth-email-link Root cause of the broker host's repeated 'BOOT_FAIL: BROKER_AUTH_METHODS= "email_link": unknown or feature-gated-out auth method' even after a fresh target/ rebuild: the script used a SINGLE cargo invocation to build BOTH agentkeys-mock-server AND agentkeys-broker-server with '--features agentkeys-broker-server/auth-email-link', and cargo silently DROPS the feature flag in this multi-package selection mode. Reproduced empirically with --message-format json: cargo build --release -p agentkeys-mock-server -p agentkeys-broker-server \ --features agentkeys-broker-server/auth-email-link → broker compiled features: [audit-sqlite, auth-wallet-sig, default, wallet-keystore] ← NO auth-email-link vs the working separate form: cargo build --release -p agentkeys-broker-server --features auth-email-link → broker compiled features: [audit-sqlite, auth-email-link, auth-wallet-sig, default, wallet-keystore] ← present Fix: 1. Split the build into two separate cargo invocations — mock-server alone (default features), broker-server alone with the feature flag. Documented the footgun in a long block comment so the next person who 'optimizes' by re-merging them will read why before doing it. 2. Added a post-build sanity check: 'strings target/release/agentkeys- broker-server | grep /v1/auth/email/(request|verify)' must match before install + restart. If the cargo footgun ever resurfaces (or anyone introduces a similar feature-strip bug), the script dies HERE with a clear diagnostic instead of after install + systemd restart loop + journal dump. Verified locally: bash -n scripts/setup-broker-host.sh # syntax ok strings target/release/agentkeys-broker-server | grep /v1/auth/email → /v1/auth/email/request /v1/auth/email/verify /v1/auth/email/status /v1/auth/email/landing (all four routes present) * fix(setup-broker-host): assert via cargo --message-format=json + cargo clean -p The previous fix (commit 6d75599) split the cargo build into separate invocations to defeat the multi-package + --features footgun, but the broker host STILL deployed binaries lacking auth-email-link. Two real root causes survived: 1. CARGO INCREMENTAL CACHE: 'rm -f target/release/agentkeys-broker-server' only removed the output binary, not target/release/deps/.fingerprint/ nor the per-feature-set cached .rlib deps. On a host that previously built without auth-email-link, cargo's incremental could relink from stale deps and produce a binary missing the feature even when the build call was correct. Fix: 'cargo clean -p agentkeys-broker-server --release' before the rebuild — only ~1s, only this crate's cache. 2. WEAK VERIFICATION: 'strings | grep -qE "/v1/auth/email/request"' is a heuristic that: - false-positives on tower middleware names containing 'email' - false-negatives when LTO dedupes string literals across the binary - dies with an unactionable 'this is the cargo footgun' guess that was wrong (the call was correct; the host environment was the bug) Replace with: parse cargo's own --message-format=json output and ASSERT auth-email-link is in the bin artifact's features list. Cargo's reported features ARE the truth — no heuristic. Critical bash detail: cargo --message-format=json sends NDJSON to stdout and compiler messages to stderr. Merging them with '2>&1' corrupts the NDJSON and jq dies with 'Invalid numeric litera…
…ate-pr conventions (#84) Adds the exercise-vs-distribution framing as a first-class concept in arch.md, names the per-data-class bucket layout, pins the project wiki location, and documents the /create-pr workflow in Claude Code git worktrees. Motivation: Recent discussions surfaced that the §6 STS-to-vault pipeline subsumes two semantically distinct cases that arch.md did not distinguish: - Class A (AWS-native, e.g. S3 / SES / future memory storage): upstream re-authorizes every request; the §6 pipeline IS both distribution and exercise. Granularity falls out of IAM + JWT claims. - Class B (bearer-token, e.g. OpenRouter / Anthropic / Groq): upstream trusts the bearer once minted; we secure distribution (per-grant provisioning + vault prefix gating) and accept that exercise enforcement is provider-bounded. Operators reading §6 alone could not tell whether the vault payload IS the action (Class A) or merely enables one out-of-band (Class B). The two cases differ on revocation, blast radius, and what the provisioner must do. Separately, S3 bucket-level configuration (Versioning, Object Lock, BucketEncryption, Lifecycle, CloudTrail data events) cannot be set per-prefix, and vault / memory / audit have conflicting requirements on every dimension. Wallet-as-prefix is sufficient for per-actor isolation but cannot replace per-data-class bucket separation -- the two are orthogonal axes, both required. Changes: docs/spec/architecture.md §3a -- new canonical-name rows for vault_bucket, memory_bucket, audit_bucket; documents the single-bucket-today $BUCKET alias and the forward fan-out to $VAULT_BUCKET etc. §4b -- new subsection "Upstream backend classes -- exercise vs distribution" introducing Class A / Class B with per-class enforcement story and add-new-upstream guidance. Links out to the wiki page for full detail. §7 -- Vault backend row 4 renamed to vault_bucket and cross-linked to §4b + §7a. Added row 5 "Egress enforcement" so a future broker-as-egress-proxy has a documented pluggable slot. §7a -- new subsection "Bucket layout -- data-class buckets, wallet prefixes" covering the bucket-level config matrix, why each data class needs its own IAM role, why $BUCKET is a variable, and the single-bucket-today migration map. Updates dead reference at §4a from .omc/wiki/ to wiki/. wiki/upstream-backend-classes-exercise-vs-distribution.md (new) Full design rationale: two security concerns table, Class A / B property tables, granularity flow per class, bucket-layout consequences, design rule for adding a new upstream, open questions (broker-as-egress-proxy trade-offs, atomic revoke gap, vault backend swap). CLAUDE.md New "Wiki-location policy" section pinning ./wiki/ as the canonical location for all project wiki pages. .omc/wiki/ is git-ignored and must not hold durable knowledge; the wiki_add / wiki_ingest MCP tools default there and lose pages to gitignore, so the rule is to use Write directly. New "/create-pr policy" section documenting the hybrid git-commit / jj-push / gh-pr workflow required inside Claude Code worktrees, where jj cannot colocate with an existing git worktree. Outside worktrees the standard jj-only rule still applies. Follow-ups (not in this PR): - Fan out $BUCKET -> $VAULT_BUCKET / $MEMORY_BUCKET / $AUDIT_BUCKET in scripts/operator-workstation.env, scripts/setup-broker-host.sh, docs/stage7-demo-and-verification.md, and the role-policy templates. Arch.md documents the migration but the rename across operator surfaces is its own change. - The wiki/agent-role-and-usage-hdkd-per-agent-omni.md page referenced by CLAUDE.md + arch.md §4a does not exist yet in either location. Pre-existing dead reference; flagged for separate fix. Co-authored-by: wildmeta-agent <agent@wildmeta.ai>
… + operator UX (#86) Issue #83 root cause: openrouter migrated from email-OTP signup to Clerk + password + magic-link. The non-CDP `openrouter.ts` scraper's `signup_email_otp` pattern no longer fits the live flow. Production was routed to the stale scraper. Fix: route `agentkeys provision openrouter` through the existing `openrouter-cdp.ts` (Clerk-aware, magic-link verifier, real-Chrome via CDP), wire up per-recipient email routing via a new SES post-receive Lambda so the OIDC-assumed data-role can read its own verification email without violating the §4.5 federation-isolation rule, and harden the operator wrapper script. Changes: - crates/agentkeys-{cli,mcp}/src/lib.rs: route openrouter to `openrouter-cdp.ts` (was the stale `openrouter.ts`). - crates/agentkeys-provisioner/src/aws_creds.rs: inject `AGENTKEYS_USER_WALLET` env (lowercase 0x address from JWT) into the scraper subprocess so the CDP scraper can build a routable signup email and poll the per-wallet S3 prefix. - infra/ses-routing-lambda/: new — Python Lambda + idempotent deploy script + unit tests + README. Triggered by S3 ObjectCreated on inbound/*; parses To: header (first 8KB Range read, body never transits Lambda memory), pattern-matches `or-<wallet>-<ts>` local-part, server-side CopyObject to `bots/<wallet>/inbound/<msg>`. AGENTKEYS auth emails (different local-part) stay in inbound/. 128MB, 10s timeout, reserved-concurrency=10. Per-invocation cost ≈1.7 µ\$. - provisioner-scripts/src/scrapers/openrouter-cdp.ts: derive signup email from `AGENTKEYS_USER_WALLET` (CLI-injected) so the SES Lambda routes it; pass `walletPrefix` to fetcher so the email backend polls `bots/<wallet>/inbound/`; canonicalize all error codes to the `ProvisionErrorCode` enum (broker parser rejected `selector-missing`, `missing-env`, `key-format`, `fatal`). - provisioner-scripts/src/lib/email.ts + email-backends/ses-s3.ts: thread `walletPrefix` option; poll `bots/<wallet>/inbound/` when set, fall back to legacy `inbound/`. - provisioner-scripts/src/lib/playwright-patterns.ts: add "New Key" to `clickOuterCreate` candidates (openrouter UI refresh observed 2026-05-15 via chrome-devtools-mcp — empty-state button is now bare "New Key" not "New API Key"). - scripts/agentkeys-provision-demo.sh: new one-shot wrapper that collapses §5.3's 8-step copy-paste block. Always resets Chrome before invoking (avoids the sticky "Browser context management is not supported" state after chrome-devtools-mcp / Playwright Inspector attach); auto-re-inits the session JWT when expired (5h TTL trip is the most common operator failure); auto-launches Chrome on :9222 if not running. - docs/stage7-demo-and-verification.md §5.3: collapsed from 60-line bash block to a two-line one-shot invocation; documents the Lambda prereq. - docs/cloud-setup.md §2.1a: new section documenting the routing Lambda + deploy command. - TODOS.md: two architectural follow-ups — disable broker's broad S3-full-access after this Lambda stabilizes, and replace mock-server `/credential/*` with S3-backed encrypted storage (filed as issue #85). - docs/spec/plans/issue-credential-storage-s3-oidc.md: draft body for issue #85. Tests: cargo check on 3 affected crates clean; npm test 45/45 pass under \$AGENTKEYS_EMAIL_BACKEND=gmail. Lambda unit tests 7/7 pass. Lambda deployed live (May 15) to account 429071895007, region us-east-1. End-to-end provision verified through key extraction; storage step fails because the mock-server backend isn't reachable from the operator laptop — tracked separately in issue #85. Co-authored-by: wildmeta-agent <agent@wildmeta.ai> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ice worker (#89) (#87) * agentkeys: stage 7+ — issue #85 step 1 (S3CredentialBackend + --credential-backend flag) Replaces the legacy mock-server `/credential/*` storage with an OIDC-scoped, client-side-encrypted S3 backend living next to the existing `bots/<wallet>/inbound/` SES routing prefix (issue #83). The legacy backend keeps handling sessions, audit, identity, scope, rendezvous, and inbox; only credential CRUD migrates in this PR. The pieces: - `crates/agentkeys-core/src/s3_backend.rs` — new `S3CredentialBackend` impl. `store/read/teardown/list_credentials` go through S3. Every other `CredentialBackend` method returns a clear "route through http backend" error — those endpoints still live on the mock-server (or the broker for the new flow). - AES-256-GCM seal, 96-bit random nonce, AAD = `agentkeys.cred.aad.v1|<lower-wallet>|<service>`. Wire layout `1B version || 12B nonce || ciphertext || 16B tag`, version = 0x01. AAD binds blob to its (wallet, service) S3 location so a cross-operator swap fails open. - KEK derivation is signer-anchored: SHA256(domain || signer.sign_eip191(omni, "agentkeys.kek.v1:<wallet>:<service>")). secp256k1 RFC 6979 makes this deterministic across calls, so the same KEK comes back on every read; future TEE migration (issue #74 step 2) inherits it transparently. - `crates/agentkeys-cli` — `CredentialBackendKind { Http, S3 }` plus `--credential-backend` (env `AGENTKEYS_CREDENTIAL_BACKEND`), `--bucket` / `AGENTKEYS_BUCKET`, `--signer-url` / `AGENTKEYS_SIGNER_URL`, `--omni-account` / `AGENTKEYS_OMNI_ACCOUNT`. New `credential_backend()` async helper on `CommandContext` builds the right impl per call. `cmd_store`, `cmd_read`, `cmd_run`, `cmd_teardown`, `cmd_provision` now route credential CRUD through it; identity resolution + the rest stay on the legacy http backend regardless of the flag. Default remains `http` for the transition window. - `docs/cloud-setup.md` §4.4 — new `AllowDaemonPutOwnCredentials` bucket-policy statement granting `s3:PutObject` + `s3:DeleteObject` on `bots/<wallet>/credentials/*` under the same `agentkeys_user_wallet` PrincipalTag scope that already gates `s3:GetObject`. Operators running `--credential-backend=s3` need the policy update to land first. - `docs/spec/architecture.md` §3a — add `credential_kek` and `credential_envelope` canonical-names rows so future docs reference the same terms. - `docs/spec/architecture.md` §9 #10 — flag the mock-server credential slice as "migrating off", point at issue #85. - `docs/stage7-demo-and-verification.md` §5.3 — operator-side opt-in block (env vars to set, what to expect at the S3 key). - `docs/spec/plans/issue-credential-storage-s3-oidc.md` — mark steps 1–3 as shipped; steps 4–6 still pending (default flip + mock-server handler removal + arch.md §11 cleanup). Tests: - cargo test -p agentkeys-core -p agentkeys-cli -p agentkeys-mcp -p agentkeys-provisioner — clean (9 new s3_backend tests covering object_key path, KEK determinism, AEAD round-trip, AAD-binding, envelope-version drift, truncated envelope; 37+3+44+7+23 = 114 pre-existing tests still pass). - cargo clippy on agentkeys-core + agentkeys-cli — clean. No deployment changes required for the existing `http` default. To opt into `s3` an operator runs the cloud-setup.md §4.4 update once per account, sets the four env vars, and the next `provision` writes to S3. * agentkeys: stage 7+ — address codex adversarial review on #87 Two high-severity findings from /codex:adversarial-review on PR #87: 1. **Scope enforcement was missing.** S3CredentialBackend ignored Session.scope on store_credential / read_credential / list_credentials / teardown_agent. The legacy HTTP backend gates per-service access server-side via the /credential/* handlers' bearer-JWT check; the S3 backend has no equivalent (the bucket policy keys only on the wallet PrincipalTag, not service). A scoped child session could therefore have read or written any service under its wallet prefix. Fix: client-side gate before any S3 call. - enforce_scope_for_service(session, service, write) rejects PermissionDenied when the service isn't in scope.services and when write=true on a read_only scope. - enforce_master_session(session, op) rejects teardown_agent on a scoped child (wallet-level destruction is master-only — matches the implicit legacy contract). - list_credentials filters its return down to scope.services so a scoped child can't enumerate the master's other services. 2. **Broker-minted AWS creds weren't reaching the S3 client.** cmd_provision fetched the OIDC-scoped temp creds via broker_env_for_provision and injected them into the scraper subprocess env only. The parent process's S3CredentialBackend used aws_config::defaults — i.e. process AWS_* env or shared config — which would either be empty (storage fails post-key-mint, the exact failure mode #85 exists to fix) or the operator's static admin creds (no PrincipalTag, isolation property gone). Fix: pull cred minting up into CommandContext::credential_backend itself. - New mint_s3_credentials helper hits the same fetch_via_broker_default_ttl path the provisioner uses, returns aws_credential_types::Credentials. - S3CredentialBackend::new gains a `credentials: Option<...>` parameter; when Some, the SDK config builder gets a credentials_provider pinned to those creds, bypassing the default chain entirely. - cmd_provision now ends up with two STS calls per run (one for scraper env, one for parent S3 client) — cheap; the alternative was threading the env map through the orchestrator into the backend factory. Tests added (all PermissionDenied codes verified): - enforce_scope_allows_master_session - enforce_scope_blocks_service_not_in_list - enforce_scope_blocks_write_when_read_only - enforce_master_session_blocks_scoped_session - store_credential_blocks_out_of_scope_before_s3_call - read_credential_allows_in_scope_read_only (also asserts out-of-scope reads still deny) - teardown_agent_rejects_scoped_session Test count: agentkeys-core lib 28 → 44 (16 s3_backend tests total: 9 from the initial PR + 7 new). Full affected-crate suite: 121 passing. cargo clippy on agentkeys-core + agentkeys-cli clean. Out of scope: - A full integration test for `provision --credential-backend=s3` end- to-end through a real STS + S3 path. That needs live AWS creds in CI and is tracked alongside the default-flip work in plan step 4. * agentkeys: stage 7+ — v2 stage 1 step 1 (actor_omni helper, v2 envelope, dual-read S3CredentialBackend) First incremental implementation commit for the v2 stage 1 plan in docs/spec/plans/v2-issues/issue-v2-stage-1-foundation.md. Lands the CLI/backend pieces that can ship without the chain contracts or the sidecar daemon being live yet. What lands: * agentkeys_core::actor_omni — deterministic SHA256("agentkeys"||"evm"|| master_wallet) helper per arch.md §14.1, used to compute the stable per-operator anchor independent of K3 / wallet rotation. * S3CredentialBackend now writes v2 envelopes (version byte 0x02, AAD = "agentkeys.cred.aad.v2|<actor_omni_hex>|<service>") and reads BOTH v1 and v2 shapes — dispatching on the version byte. v2 writes go to bots/<actor_omni_hex>/credentials/ per arch.md §14.5; reads try v2 first and fall back to v1 only on NotFound, propagating every other error to surface real failures. * Dual-prefix list_credentials (union, dedup'd; v2 wins) and dual-prefix teardown_agent (wipes both wallet-keyed and actor_omni-keyed paths) so mid-migration state can't strand orphan blobs. * CLI --envelope-version={v1,v2} flag plumbs WriteEnvelope through CommandContext. Default stays v1 so PR #87 deployments keep working unchanged; operators flip to v2 post-bucket-policy-rollout. * CLI --credential-backend=sidecar flag accepted by the surface; today returns a clear "not yet implemented" error pointing operators at --envelope-version=v2 as the closest currently-working substitute. Forward-compatible flag shape so the eventual daemon implementation is a code change, not a CLI break. * agentkeys whoami prints agentkeys_actor_omni alongside session_wallet so operators can sanity-check the bucket-policy PrincipalTag and the v2 S3 path their backend will use after the dual-tag rollout. * Tests: 12 new unit tests covering actor_omni determinism + case handling, v2 envelope roundtrip, v1/v2 path divergence, AAD divergence, version dispatch, WriteEnvelope override. Full workspace test suite still green (467 tests passed, 0 failed). What's deferred: * Broker /v1/cap/cred-fetch + /v1/cap/cred-store endpoints (cap-mint). * On-chain ScopeContract / SidecarRegistry / K3EpochCounter contracts. * K11 WebAuthn verification on master-mutation endpoints. * Sidecar daemon (agentkeys-proxy.sock). * OIDC JWT dual-tag mint (agentkeys_user_wallet + agentkeys_actor_omni). * Bucket policy _v2_omni_keyed rule. Docs: * docs/v2-stage1-migration-and-demo.md — new top-level "What landed in this commit" section + A.2 clarification on the sidecar stub + revision-log entry for 2026-05-18. * docs/spec/plans/v2-issues/issue-v2-stage-1-foundation.md — three CLI tasks marked [x] (sidecar flag, envelope-version flag, whoami actor_omni). credentials-service worker section updated to note dual-envelope decrypt + dual-path read already work in S3CredentialBackend; Lambda reuse is the remaining work. * docs/spec/architecture.md §14 — the prior session's v2 consolidation (was uncommitted; lands with this commit). * docs/spec/plans/v2-issues/ — three planning issues filed alongside arch.md §14 (stage-1 foundation, stage-2 hardening, deferred payment service). * docs/archived/ — earlier standalone v2 design docs superseded by arch.md §14, archived per CLAUDE.md docs/archived policy. * docs(arch): rewrite architecture.md as clean v2 — assumes #88/#89/#90 complete The pre-v2 architecture.md was a patchwork of the original single-binary mock-server design plus a §14 graft for v2 plus three layered Codex amendment addenda (§14.8, §14.9, §14.9a). New readers had to triangulate across the v1 spine + v2 graft + amendments to reconstruct the design. This rewrite collapses all of that into one coherent v2 narrative, treating issues #88 (payment-service), #89 (stage-1 foundation), and #90 (stage-2 hardening) as completed. Codex findings are folded into the design (no more "see addendum"); dual v1/v2 migration language is gone (the migration window closed when stage 1 shipped). Structure (27 sections, top-down): §1 System overview (five trust boundaries, mermaid) §2 Component inventory (14 components) §3 Trust boundaries (blast-radius table per boundary) §4 Key inventory K1–K11 (canonical) §5 Canonical names (one concept, one canonical spelling) §6 Identity model — three layers + HDKD actor tree §7 Upstream backend classes — A (per-request) / B (bearer) / C (on-chain payment-rail, new in v2) §8 Mental model — four orthogonal axes §9 Cold-start (master bootstrap, stages 0–4) §10 Per-actor binding ceremonies (master + agent) §11 Recovery — M-of-N device quorum (no anchor wallet, no seed) §12 Sidecar daemon (localhost proxy, host-local policy) §13 Broker (cap-mint authority, on-chain reader) §14 Signer (TEE-protected K3 vault) §15 Workers — creds / memory / audit / email / payment (with audit tiers A/B/C and payment modes P-1/P-2/P-3 spelled out) §16 On-chain layer (four contracts, Solidity inlined) §17 Storage layout (per-data-class buckets, per-actor prefixes) §18 Encryption envelope (KEK derivation + AES-256-GCM v2) §19 Cap-token shape + lifecycle (wire JSON + 11-step verification) §20 Mode selection — sovereign default + hosted-relay opt-in + self-hosted-relay §21 K3 rotation (zero-migration property) §22 Pluggable surfaces (six pluggable axes) §23 Cargo workspace (post-v2 crate layout) §24 Deployment topology §25 Cross-references §26 What v2 guarantees §27 What's NOT in this doc Major changes vs the pre-v2 spine: * Stage 7 / mock-server / S3CredentialBackend / dev_key_service language retired — those are pre-v2 historical artifacts that no longer describe the shipped system. * §15 enumerates ALL five workers (creds + memory + audit + email + payment). Payment-service is now a first-class section with the P-1/P-2/P-3 mode table, security properties, and wire shape inlined. * §16 inlines all four Solidity contracts (AgentKeysScope, SidecarRegistry, K3EpochCounter, CredentialAudit) with the cap-mint verification gates spelled out (per-actor binding, K11 for master mutations, K3 epoch freshness, CAS-burn for payments). * §19 is new — the cap-token wire shape + 11-step worker verification sequence. Pre-v2 had this scattered across §14.3 + the stage-1 plan. * §11 (recovery) has a concrete second-by-second timeline showing how a surviving master device M-of-N quorum revokes a stolen device in ~60s. * §6 lays out the three identity layers (Layer 1 actor_omni anchor; Layer 2 current_master_wallet; Layer 3 operational uses) up front, not buried in a §14.1 sub-section. * §7 adds Class C (on-chain / payment-rail operations — irreversible) alongside Class A (per-request, AWS-native) and Class B (bearer). Pre-v2 only had A and B. Length: 1248 → 1488 lines. Net +240 because of the inlined contracts, worker tables, recovery timeline, cap verification sequence, and mermaid diagram for the unified system overview. * docs(v2-stage1): rewrite migration doc as fresh-start guide with Litentry/Heima EVM backbone The previous doc tried to cover both migration (from stage-7 PR #87) and new-feature demo in one Part A + Part B structure. The migration half turned out to be entirely mechanical — the dual-read / dual-envelope / dual-prefix support already in S3CredentialBackend handles the transition without any operator runbook. So drop Part A; the new doc is fresh-start only. Chain backbone: Litentry rebranded to Heima Network in 2026. Heima runs Frontier (pallet_evm + pallet_ethereum) with EVM chain ID 212013 on mainnet (= "LIT deployment year (21) + paraID (2013)", hardcoded at parachain/runtime/heima/src/lib.rs). Operators deploy the four stage-1 Solidity contracts (AgentKeysScope, SidecarRegistry, K3EpochCounter, CredentialAudit) via Foundry against https://rpc-eth.heima.network or a self-hosted Frontier node from litentry/heima:latest. Address mapping is HashedAddressMapping<BlakeTwo256>, so EVM accounts are first-class on-chain identities — no MetaMask-Substrate dual-account dance. Structure (10 sections plus reference + revision log): Litentry/Heima EVM chain reference — chain IDs, RPC URLs, explorer, self-hosted node bring-up What stage 1 ships (vs inherited) — clear table of what comes from stage-7 demo vs new in v2 §0 Prerequisites (inherited) — pointer to stage-7 §0 verbatim + new Heima RPC reachability check §1 Master device bootstrap — stage-7 §1-§2 inherited, plus new stage-2 (WebAuthn) and stage-4 (on-chain registry) sub-sections §2 AWS prereqs (inherited+v2 tag) — one-line PrincipalTag rename to agentkeys_actor_omni §3 Smoke-test v2 envelope — verify the v2 S3 path works end-to-end without chain or sidecar §4 Deploy Heima EVM contracts — Foundry deploy script; cast verification; K3EpochCounter init §5 Register master device on chain — the §1.4 step, now executable against the deployed registry §6 Sidecar daemon bring-up — agentkeys-daemon flags; localhost proxy verification §7 Create agent + grant scope — full HDKD per-agent omni flow with K11 prompts at agent-create and scope-grant; in-scope vs out-of-scope verification §8 Chain-level isolation proof — repeat for bob; verify per-actor binding rejects cross-actor cap-mint (Codex finding #1 in action) §9 Teardown What's still in flight — shipped-vs-spec status table so operators following the doc today know exactly which steps will error with "not yet implemented" The doc is now the target end-state runbook; track issue-v2-stage-1-foundation.md for the rolling implementation status of each pending sub-deliverable. Length: 445 → 814 lines. * feat(chain): pluggable EVM backbones via named ChainProfile system Chain backbone is pluggable per arch.md §22, but the previous draft of the demo doc hardcoded Heima env vars (HEIMA_EVM_RPC_HTTP, HEIMA_EVM_CHAIN_ID, HEIMA_SUBSTRATE_WSS, HEIMA_EXPLORER, ...). Switching to Base or Ethereum meant renaming five env vars per chain. This commit collapses everything into one --chain flag. What ships: * New module crates/agentkeys-core/src/chain_profile.rs — ChainProfile struct + serde-json wire format. ChainProfile::resolve() walks the documented precedence ($AGENTKEYS_CHAIN_PROFILE_FILE > --chain CLI flag > $AGENTKEYS_CHAIN env > built-in default 'heima') and returns a typed profile plus a debug string explaining which step matched. * 7 built-in profile JSONs under crates/agentkeys-core/chain-profiles/, embedded into the binary via include_str! macro: heima (mainnet, chain_id=212013, substrate-frontier) heima-paseo (testnet, chain_id=0 sentinel for auto-detect) base (mainnet, chain_id=8453, optimism-l2, safe-tag default) base-sepolia (testnet, chain_id=84532) ethereum (mainnet, chain_id=1, finalized-tag default) sepolia (testnet, chain_id=11155111) anvil (local, chain_id=31337, instant finality, ships test key) * Profile fields cover every chain-specific dimension the broker / daemon / workers need: - chain_id (uint64; 0 = auto-detect via eth_chainId) - chain_kind (enum: substrate-frontier | ethereum-l1 | optimism-l2 | arbitrum | local-dev — controls finality + gas strategy) - rpc.{http, wss, substrate_wss?} - explorer.{url, tx_url_template, address_url_template} - token.{symbol, decimals} - finality.{default_block_tag, confirmation_blocks, confirmation_seconds, notes} - gas.{model, max_priority_fee_gwei, max_fee_gwei} - deploy.{deployer_env_var, foundry_chain_arg, faucet_url?, default_test_key?} * CLI wiring in crates/agentkeys-cli: - New top-level flag: --chain <name> (env AGENTKEYS_CHAIN) - New subcommand: agentkeys chain list (enumerate built-in profile names) - New subcommand: agentkeys chain show [name] (print full profile JSON; omit name to inspect the active profile per resolution rules) - CommandContext::chain_profile() returns the cached resolved profile; --verbose prints the resolution debug string * Operator-custom chains: set $AGENTKEYS_CHAIN_PROFILE_FILE to any JSON file matching the schema and AgentKeys uses it. No recompile. Moonbeam, Astar, Polygon, Avalanche, BSC, permissioned chains (Aliyun BaaS, Hyperledger, Quorum) are all one JSON file away. Tests: 12 new unit tests covering every built-in loads + parses, known field values per chain, case-insensitive lookup, resolution precedence, explorer URL template substitution. Workspace test count: 467 → 479, all passing. Docs: * docs/spec/architecture.md §22 — chain layer row in the pluggability table now points at the named-profile system; new §22a "Chain profiles — how to switch between EVM backbones" covers resolution order, schema, built-in inventory, operator-custom flow, what chain_kind controls at runtime, and cap-mint freshness across chains. * docs/v2-stage1-migration-and-demo.md — replaced the "Litentry/Heima EVM — chain reference" section with a generalised "Chain backbone — pluggable per arch.md §22" section. Built-in profile table + operator-custom example (Moonbeam) + why-named-profiles rationale (vs the previous per-chain env var sprawl). Updated §0 reachability check + §4 Foundry deploy + §5 device register + §6 sidecar daemon bring-up to pull chain-specific values from the active profile via `agentkeys chain show | jq -r .<field>` — no more HEIMA_* env var coupling. Switching chains is now: export AGENTKEYS_CHAIN=base (or pass --chain base on any command). Every component reads the same profile. * chore(chain): correct Heima RPC URL + pin Subscan explorer integration target Two corrections based on authoritative Heima developer info, verified live 2026-05-18 against the production RPC: * RPC hostname: was guessed as rpc-eth.heima.network in the speculative draft; canonical URL per docs.heima.network / chain-list.com/heima / dwellir.com/networks/heima is rpc.heima-parachain.heima.network (same host serves both EVM JSON-RPC and Substrate-RPC). Verified live: eth_chainId → 0x33c2d (= 212013 decimal, matches profile) eth_blockNumber → 0x92c29f (current head, ~9.6M blocks) system_chain → "Heima" (Substrate side responds on same host) * eth_chainId hex in the demo doc was wrong (had 0x33c4d = 212045); correct value is 0x33c2d = 212013. Also pinned the future agentkeys explorer integration target by adding explorer.subscan_source to the chain profile JSON schema: * New ChainProfile::ExplorerLinks.subscan_source field — optional pointer at the backend + frontend repo for chain-specific explorer indexing. Type-safe in Rust via new SubscanSource struct. * heima.json now points at the Litentry-forked Subscan stack: - github.com/litentry/subscan-essentials (Go backend) - github.com/litentry/subscan-essentials-ui-react (React frontend) These integrations are stage-2/3 deliverables — agentkeys-specific indexing for AgentKeysScope.ScopeUpdated, SidecarRegistry.*, K3EpochCounter.K3Rotated, CredentialAudit.* events, cross-indexed by actor_omni. Pinning the target in the profile means when the work happens, it lands in those two repos rather than a third-party hosted explorer. * docs/spec/architecture.md new §22a.6 "Explorer integration target" documents the integration plan; renumbered the existing cap-mint freshness section to §22a.7. * docs/v2-stage1-migration-and-demo.md new "Explorer — current state + future agentkeys integration" subsection covers the same target, plus the in-doc curl example now shows the correct 0x33c2d hex value. Other chain profiles can populate subscan_source with their own explorer codebases as integrations land (Etherscan / Blockscout for Ethereum / Base, chain-specific forks for others). Workspace tests: 479/0 (unchanged — schema is backwards-compatible because subscan_source is #[serde(default)] optional). * docs(chain): document Alice/sudo on Heima Paseo + prod-vs-dev defaults Heima developer team confirmed that Heima Paseo's runtime ships pallet_sudo with the well-known Substrate dev account Alice as the sudoer. This commit documents what that means, why it's a standard Substrate testnet convention, and how AgentKeys operators use it (or don't) during stage-1 dev bring-up. Educational background (for readers unfamiliar with Substrate): * Alice is one of six well-known Substrate dev accounts. The keypair is deterministically derived from the public seed phrase 'bottom drive obey lake curtain smoke basket hold race lonely fit walk//Alice'. Public key 0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684 e7a56da27d. SS58 (generic prefix 42) 5GrwvaEF5zXb26Fz9rcQpDWS57 CtERHpNehXCPcNoHGKutQY. These keys are intentionally public — every Substrate developer knows them — so dev/test chains can ship with pre-funded accounts of known keys. * pallet_sudo is the Substrate root-bypass pallet. Runtimes that include it expose one extrinsic: sudo.sudo(call). The pallet stores ONE address as the sudo key; only that address can call sudo.sudo and the wrapped call runs with RawOrigin::Root — bypassing every other origin check. Testnets ship sudo so devs have a god-mode lever (force-fund accounts, force-set state, force-run upgrades); production chains either remove the pallet or move the key to a governance multisig. * On Heima Paseo specifically: sudo + Alice means anyone can use sudo.sudo for testnet bring-up without provisioning real accounts. What landed in this commit: * New typed schema in ChainProfile (DevEnvironment + SudoConfig structs), optional and backwards-compatible via #[serde(default)]. Production profiles (heima, base, ethereum) omit dev_environment entirely; only testnets / local-dev profiles set it. * heima-paseo.json profile now carries the full Alice sudoer metadata: seed phrase, public key, SS58 generic-prefix address, invocation recipe, two warning lines (anyone-can-sign-as-Alice + URL pending Heima-dev-team confirmation). * Production-vs-development convention pinned via dev_environment.is_development_default. Only heima-paseo carries this flag among built-ins. New ChainProfile::development_default_name() helper returns Some("heima-paseo"). Production default stays DEFAULT_PROFILE = "heima". * docs/spec/heima-open-questions.md: new §3a "Chain backbone — EVM, Paseo, sudo (added 2026-05-18 after Heima dev info handoff)" with educational Alice/sudo background, recipe table for "what AgentKeys would use sudo for", how-to-invoke-sudo notes, and three new Q13-Q15 questions for the Heima dev team: - Q13: canonical Paseo RPC URL (both speculative URLs fail SSL as of 2026-05-18) - Q14: confirm Alice as sudoer + invocation recipe + SS58 on Heima prefix-31 encoding - Q15: confirm Heima mainnet has either removed pallet_sudo or moved the key to a governance multisig Reuse-Build-Block matrix updated with three new rows. * docs/v2-stage1-migration-and-demo.md: chain-backbone section now documents the prod-vs-dev convention (heima for production, heima-paseo for development, anvil for local tests). New "Alice + sudo on Heima Paseo (development-environment convenience)" sub-section with concrete recipes for pre-funding deployer wallets, resetting K3 epoch, etc. Three invocation options spelled out (Polkadot.js Apps, subxt CLI, @polkadot/api). Built-in profile table updated to mark heima as "Production default" + heima-paseo as "Development default". Revision-log entry added. * docs/spec/architecture.md §22a updated with the prod-vs-dev convention table (heima production / heima-paseo development / anvil local-tests). New §22a.5a "Alice + sudo on dev-default chains (heima-paseo)" covers the background + what sudo does/doesn't do for AgentKeys + the Substrate↔EVM bridge via pallet_ethereum.transact. Tests: 12 → 15 chain_profile tests (3 new — heima_paseo is dev default with alice sudo, development_default_name returns heima-paseo, production chains carry no dev_environment). Workspace: 479 → 482 all passing. * feat(scripts): one-command Heima Paseo bring-up via Alice sudo The manual §4.1-§4.4 sequence (chase faucet, juggle deployer env vars, hand-run cast send for K3EpochCounter init) is now one command: bash scripts/heima-paseo-bring-up.sh Two new scripts: scripts/heima-paseo-bring-up.sh — bash orchestrator that does: 1. Tool sanity-check (agentkeys, jq, forge, cast, node, npx) 2. Resolve heima-paseo chain profile + reachability-check $RPC_HTTP + abort if eth_chainId == 212013 (mainnet safety) 3. Generate throwaway EVM deployer (or reuse $HEIMA_PASEO_DEPLOYER_KEY) 4. Sudo-fund deployer from Alice (100 pHEI default) via the heima-paseo-sudo.mjs helper 5. Foundry-deploy the four stage-1 contracts (graceful stub-mode when crates/agentkeys-chain isn't built yet) 6. Persist contract addresses to operator-workstation.env, namespaced by HEIMA_PASEO so other chains can deploy alongside 7. Print summary + suggested next-step command Re-run with SKIP_FUND=1 or SKIP_DEPLOY=1 to skip individual phases. scripts/heima-paseo-sudo.mjs — Node + @polkadot/api helper: fund — sudo.balances.forceTransfer Alice → EVM address (uses blake2_256("evm:" || eth_address) for the EVM→Substrate account mapping per HashedAddressMapping<BlakeTwo256>) bootstrap — sudo.sudo(ethereum.transact(...)) for any EVM contract call; used for K3EpochCounter init, force-set scope, pre-register sidecar entries, etc. whoami — sanity-check the sudoer + Alice's balance Three guardrails keep mainnet safe: - Refuses if $AGENTKEYS_CHAIN != heima-paseo - Refuses if live eth_chainId == 212013 (mainnet) - Logs every sudo call to stderr before signing Polkadot deps load lazily so --help works without them installed; the bring-up script auto-fetches via npx --package=@polkadot/api ... docs/v2-stage1-migration-and-demo.md additions: * New §4.0 "Automated Heima Paseo bring-up via Alice sudo" before §4.1: - One-command bring-up recipe + step-by-step timing table - The two scripts that do the work (orchestrator + sudo helper) - Dev-shortcut table: pre-register fake sidecar entry, force-set scope, fast-forward K3 epoch, parallel multi-tenant funding - Explicit "what sudo CANNOT do" section spelling out the production-safety properties (cannot forge K11, cannot sign as operator's K10, cannot bypass worker-side re-verification) * §4.1 now has a "for Heima Paseo: skip this section" callout pointing at §4.0 as the fast path. The manual recipe is still authoritative for Heima mainnet + Base + Ethereum (chains without sudo). * "What's still in flight" table + revision log updated. Tests: no Rust changes; existing 482 workspace tests still passing. Scripts validated: bash -n syntax check + node --check syntax check + node scripts/heima-paseo-sudo.mjs --help round-trip without polkadot deps installed. * fix(chain): correct Heima Paseo RPC URL + chain ID + SS58 + token (Q13 resolved) Heima dev team confirmed the canonical Paseo values. Live-verified 2026-05-18 against https://rpc.paseo-parachain.heima.network: eth_chainId → 0x7dd (= 2013 decimal — HEIMA_PARA_ID) system_chain → "Heima-paseo" system_properties → ss58Format=131 tokenDecimals=18 tokenSymbol=HEI eth_blockNumber → 0x2c5556 (~2.9M blocks; live chain) What I had wrong (speculation from earlier research): RPC URL: was rpc-eth-paseo.heima.network / rpc-paseo.heima.network now https://rpc.paseo-parachain.heima.network Chain ID: was 0 (auto-detect sentinel) now 2013 (= HEIMA_PARA_ID; mainnet's 212013 prefixes year) SS58 prefix: was undocumented (assumed = mainnet's 31) now 131 (NOT 31, NOT the generic 42) Token symbol: was pHEI (testnet-prefix convention guess) now HEI (same symbol as mainnet, no prefix) Changes: * crates/agentkeys-core/chain-profiles/heima-paseo.json: - rpc.{http,wss,substrate_wss} all point at the single canonical host (same host serves EVM + Substrate RPC) - chain_id: 0 → 2013 - token.symbol: pHEI → HEI - finality.notes pins the live curl outputs for future drift detection - dev_environment.sudo.warnings adds an SS58-prefix-131 reminder (re-encode pasted pubkeys for paseo, or use //Alice as SURI) * crates/agentkeys-core/src/chain_profile.rs: - test heima_paseo_chain_id_zero_signals_auto_detect renamed to heima_paseo_chain_id_is_2013; asserts chain_id == 2013 AND that paseo's chain_id does not collide with mainnet's (defense against a future refactor accidentally swapping them) * docs/spec/heima-open-questions.md Q13: marked ✅ RESOLVED with the five live curl outputs pinned in the answer block. Reuse-Build-Block matrix row updated to "resolved" status. * docs/v2-stage1-migration-and-demo.md: - "Open questions" callout in the chain-reference section split into "Resolved" (Q13 — RPC URL + chain ID + SS58 + token symbol) and "Still pending" (Q14 Alice-as-sudoer confirmation + Q15 mainnet sudo state + faucet URL) - Revision-log entry added Workspace tests: 482/0 (15 chain_profile tests including the renamed chain-ID pin). * docs: route demo email to bots.litentry.org + fix broken reachability snippet Replaces RFC 2606 placeholder addresses (alice@demo.example, alice@x.com) with demo-1@bots.litentry.org, the SES-verified bot-domain alias the agentkeys-init-email-demo.sh wrapper already routes to. Placeholder domains are undeliverable: the broker accepts the request, SES sends the magic link into the void, and the CLI polls forever — a real operator trap. Also folds back into the demo doc the two shell pitfalls that bit me running the §0 reachability snippet: 1. xargs -I{} ... $((16#$(echo {} | sed ...))) — the $((...)) arithmetic expansion runs in the OUTER shell BEFORE xargs substitutes {}, so zsh sees literal `{` and errors with "bad math expression: illegal character: {". Replaced with for-loop + direct $((hex)) (0x... is native in arithmetic context, no 16#). 2. Loop verdict variable can't be named `status` — zsh has it as a read-only special parameter (alias for $?). Renamed to `verdict`. Both reachability snippets in the doc now use the safe shape and ship with a "two pitfalls to avoid" callout so the next operator running top-to-bottom doesn't repeat the failure. Comments updated with the correct live hex values: 0x33c2d for heima (was 0x33c4d = wrong) and 0x7dd for heima-paseo. Verified live 2026-05-18: curl + the new doc snippet against both canonical RPCs returns OK for heima (212013) and heima-paseo (2013). * scripts: add v2-stage1-demo.sh one-command orchestrator + §0.0 doc Combines the existing demo scripts (install-agentkeys-cli.sh, agentkeys-init-email-demo.sh, heima-paseo-bring-up.sh) into a single idempotent flow with 9 numbered steps. Composes — does not replace — the underlying scripts so they remain individually usable for finer-grained debugging. Idempotency: each step has a "skip if already done" pre-check, same pattern as cloud-setup.md §4.2 ("if OIDC provider ARN ends in $BROKER_HOST, skip create"): 1. Tool sanity-check (always runs, <100ms) 2. Source scripts/operator-workstation.env (always runs) 3. AWS profile sanity-check (guards against wrong profile) 4. agentkeys CLI build+install (skips if --session-id + --chain flags already present) 5. Chain reachability + live-eth_chainId match against profile 6. Email-init session JWT (skips if session.json exists + <1h old) 7. S3 envelope smoke-test store+read (skips if blob already at bots/<actor_omni>/credentials/<service>.enc) 8. Chain bring-up via heima-paseo-bring-up.sh (skips if SCOPE_CONTRACT_ADDRESS_HEIMA_PASEO already in env-file) 9. Summary + next-step hints No hardcoded values — every magic input is overridable via env var or CLI flag. SESSION_ID, AGENTKEYS_CHAIN, SMOKE_TEST_SERVICE, SMOKE_TEST_SECRET, FUND_AMOUNT_HEI all configurable. Resumability: --from-step N / --to-step N / --only-step N for partial re-runs. On failure, the die() helper prints the exact resume command (`bash scripts/v2-stage1-demo.sh --only-step <N>`). Pause points for operator input: - Step 6: macOS keychain modal appears when agentkeys init writes the session JWT. Script narrates this in advance — the OS modal handles the actual prompt; no shell pause needed. - Step 8 with --confirm: explicit `read -p` before chain deploy. Tested locally: --to-step 5 runs preflight cleanly, --only-step 1 runs tool check alone, argparse errors exit 1 with a clean one-line message (no misleading "step 0/9" context). Demo doc gets a new §0.0 "One-command demo" subsection at the top of §0 that surfaces the script before operators wade into per-step copy-paste — with the same step-by-step table, pause-point notes, and configurable-inputs matrix as the script's own --help output. * scripts: fix v2-stage1-demo.sh whoami CLI position + signer-url trap Three real bugs from the first live run on the operator's laptop: 1. `agentkeys whoami --json` fails — --json is a top-level CLI flag (`cli.json` in main.rs:26, threaded into CommandContext.json_output). It MUST come before the subcommand. The script + the inline §1.3 doc snippet both had it after. Fixed: `agentkeys --json whoami`. 2. `--signer-url requires --omni-account` because whoami's signer_url arg is `#[arg(env = "AGENTKEYS_SIGNER_URL")]` (main.rs:275) — clap auto-populates it from operator-workstation.env, then the CLI tries the signer round-trip and demands --omni-account. Chicken- and-egg since we want actor_omni FROM whoami. Workaround: `env -u AGENTKEYS_SIGNER_URL` for the whoami call only; the local-only fields (session_wallet + agentkeys_actor_omni) don't need the signer. 3. step-7 store failure message ("check bucket policy") was too narrow — `Error: UNREACHABLE — Backend unreachable` (lib.rs:66) is BackendError::Transport's generic catch-all for ANY AWS SDK error (AccessDenied, region mismatch, network, signer down). Now prints three probe-commands (direct s3 cp, get-bucket-policy inspection, signer health check) ranked by likelihood, plus the `--from-step 8 --skip-smoke` escape hatch so the operator can continue to chain steps while diagnosing the cloud-side issue. The first two fixes also land in the demo doc's §1.3 snippet so the next operator running top-to-bottom sees the gotchas inline (per the runbook-fix-fold-back policy in CLAUDE.md). Verified live: --only-step 7 now correctly captures session_wallet + agentkeys_actor_omni, computes the s3 path, and fails with the new diagnostic error (instead of the old "session expired" red herring). * v2 stage-1: broker emits agentkeys_actor_omni session tag + bucket policy migration Wires the v2 highly-abstracted-service PrincipalTag path end-to-end so `agentkeys store --credential-backend=s3 --envelope-version=v2` can actually PUT credentials through the OIDC AssumeRoleWithWebIdentity flow. Three coupled changes: 1. **Broker (crates/agentkeys-broker-server/src/handlers/oidc.rs)**: `build_oidc_jwt_claims` now also emits `agentkeys_actor_omni` (= SHA256("agentkeys"||"evm"||wallet_lc), via the existing `derive_omni_account` helper) as a top-level claim AND as a PrincipalTag in `https://aws.amazon.com/tags`. Both v1 and v2 tag keys live in `principal_tags` + `transitive_tag_keys` during the migration window — v1 policies (keyed on agentkeys_user_wallet) and v2 policies (keyed on agentkeys_actor_omni) both work without broker config churn. `claims_supported` in the OIDC discovery doc gains the new claim name. All 8 existing broker OIDC tests pass — the additions don't break any v1 invariant. 2. **scripts/bucket-policy-v2-migrate.sh** (new, 130 lines): idempotent migration that flips the bucket policy from §4.4 v1 shape (Sid=AllowDaemonGetOwnObjects, key=agentkeys_user_wallet) to v2 shape (Sid=AllowDataRolePutOwnCredentialsV2, key=agentkeys_actor_omni) AND adds the missing 4th statement that grants PutObject on the credentials/* sub-prefix (cloud-setup.md §4.4 documents this but it was never applied to the live cloud). Backs up the existing policy to /tmp/bucket-policy-backup-<ts>.json before mutating. Re-runs are no-ops once the v2 markers are present. Deliberately does NOT use the demo doc §2.2 verbatim shape (Principal:* + StringNotEquals != "") because cloud-setup.md §4.3 warns negated string operators on missing context keys evaluate as TRUE — a JWT with no tags claim silently bypasses. The §4.4 Principal-pinned shape with PrincipalTag-scoped Resource ARN is the safer template and what we want enforced. 3. **scripts/v2-stage1-demo.sh**: STEP_TOTAL 9 → 10. New step 7 ("Ensure v2 bucket policy applied") delegates to bucket-policy-v2-migrate.sh idempotently. Steps 7/8/9 become 8/9/10. Step 8 (smoke test, was step 7) now passes --broker-url $OIDC_ISSUER and exports AGENTKEYS_DATA_ROLE_ARN=$DATA_ROLE_ARN so the CLI's mint_s3_credentials path engages (otherwise the SDK falls back to direct admin IAM and gets AccessDenied silently wrapped as 'Backend unreachable'). Verified live: - cargo test -p agentkeys-broker-server --lib oidc → 8 passed - bash scripts/bucket-policy-v2-migrate.sh → applied + re-run skips - Manual curl on /v1/mint-oidc-jwt today still returns v1-only JWTs because the REMOTE broker host hasn't picked up this commit yet. Next step: redeploy the broker via bash scripts/setup-broker-host.sh --ref claude/stupefied-darwin-cfafd6 on the broker host, then re-run --only-step 8. * v2 stage-1 Fix 1: per-data-class vault bucket + role separation (arch.md §17) Closes the bucket-sharing arch violation flagged in code review. Credentials were landing in `$BUCKET` (= `agentkeys-mail-*`, the inbound-mail bucket), violating arch.md §17 ("per-data-class buckets are mandatory; S3 exposes encryption / lifecycle / replication / CloudTrail at the bucket level only — folding data classes collapses blast radii"). ## Fix 1 (this PR) — shipped Provisions a dedicated `$VAULT_BUCKET` (`agentkeys-vault-${ACCOUNT_ID}`) and `agentkeys-vault-role` per arch.md §17 + §17.2, cleans the mail bucket policy of any stray credentials grants, and rewires the orchestrator to target the new vault infra. Four new idempotent scripts (each safe to re-run): - scripts/provision-vault-bucket.sh — bucket + block-public-access + SSE-S3 - scripts/provision-vault-role.sh — `agentkeys-vault-role` with OIDC trust + credentials-only inline policy (3 statements, all scoped to `bots/${aws:PrincipalTag/agentkeys_actor_omni}/credentials/*`) - scripts/apply-vault-bucket-policy.sh — vault bucket gets `Sid: VaultPolicyV2` (Principal-pinned to vault-role + Null operator for tag presence per cloud-setup.md §4.3 safety) - scripts/cleanup-mail-bucket-policy.sh — mail bucket reverts to email-only (drops the credentials grants accidentally added by the earlier `bucket-policy-v2-migrate.sh`, which is now removed) Each one checks "is this already done?" before acting; verified idempotent via two consecutive runs of `bash scripts/v2-stage1-demo.sh --from-step 7 --to-step 7` — first run created everything, second skipped every sub-step. ## Integration test in the orchestrator scripts/v2-stage1-demo.sh step 7 composes the 4 sub-scripts as "Provision vault infra (bucket + role + policy)". Step 8 (smoke test): - Uses `--bucket $VAULT_BUCKET` (NOT `$BUCKET`) - Exports `AGENTKEYS_DATA_ROLE_ARN=$VAULT_ROLE_ARN` so the CLI's OIDC AssumeRoleWithWebIdentity targets the vault role - **Cross-contamination assertion**: after store, asserts the blob is in `s3://$VAULT_BUCKET/bots/<actor_omni>/credentials/<service>.enc` AND NOT in `s3://$MAIL_BUCKET/bots/<actor_omni>/credentials/...`. If the separation regresses, the demo fails loud with `ARCH VIOLATION (arch.md §17): credential blob ALSO landed in mail bucket`. operator-workstation.env adds: VAULT_BUCKET=agentkeys-vault-${ACCOUNT_ID} VAULT_ROLE_ARN=arn:aws:iam::${ACCOUNT_ID}:role/agentkeys-vault-role DATA_ROLE_ARN stays for the email subsystem (will rename when email migrates in stage 2 — same pattern as VAULT did here). ## Fix 2 (deferred to stage 2) — tracked in issue #91 The credentials-service worker (arch.md §15.1) — Lambda + mTLS to signer for encrypt/decrypt + cap-on-chain re-verify — is deferred to stage 2. Today the CLI does client-side encrypt + direct S3 PUT through the OIDC-assumed vault role; the worker will take over the encrypt/decrypt step without changing the envelope shape (same KEK, same AAD, same nonce shape). See https://github.com/litentry/agentKeys/issues/91 for full design + acceptance criteria. ## Verified live (AWS account 429071895007) - Vault bucket created: s3://agentkeys-vault-429071895007 - Block-public-access: all 4 flags = true - Default SSE-S3 AES-256 applied - Vault role created: arn:aws:iam::429071895007:role/agentkeys-vault-role - Inline policy: 3 statements (List + Get + Put/Delete on credentials/*) - Vault bucket policy: 1 statement (Sid VaultPolicyV2, PrincipalTag-scoped) - Mail bucket policy cleaned: 3 statements (SES inbound + email role list/get; NO credentials grants) - Idempotency: re-running step 7 skips every sub-step cleanly ## What still blocks step 8 today The REMOTE broker host needs to be redeployed to pick up commit 4319428 (broker emits both v1 + v2 PrincipalTag in `/v1/mint-oidc-jwt`). Verified live: today's broker still emits v1-only: curl -sS -X POST -H "Authorization: Bearer \$SESSION_TOKEN" \\ https://broker.litentry.org/v1/mint-oidc-jwt | jq -r .jwt | \\ cut -d. -f2 | <base64url-pad-then-decode> | \\ jq '."https://aws.amazon.com/tags".principal_tags' # → { "agentkeys_user_wallet": [...] } ← v1 only Redeploy the broker via: bash scripts/setup-broker-host.sh --ref claude/stupefied-darwin-cfafd6 …on the broker host. Then re-run: bash scripts/v2-stage1-demo.sh --only-step 8 * scripts: fix polkadot-deps resolution in heima-paseo-sudo.mjs Real bug from the operator running --only-step 9 in v2-stage1-demo.sh. The .mjs script's lazy `import('@polkadot/api')` failed because the earlier `npx --package=X -y -- node script.mjs` pattern only adds X's bin files to PATH; the script's `import()` resolves via Node's module resolver, which walks UP from the script's location looking for node_modules — and there's no node_modules in scripts/. So the import fell into the catch block and printed "[heima-paseo-sudo] missing polkadot deps", killing step 9. Fix: declare the deps in a new scripts/package.json and have heima-paseo-bring-up.sh run `npm install --prefix scripts` once (idempotent — checks scripts/node_modules/@polkadot/api existence first) before invoking `node` directly. The .mjs script's lazy-load shape stays for --help UX, but now succeeds because node_modules is sitting right next to the .mjs. Version pin: had to bump @polkadot/util / util-crypto / keyring from ^13.0.0 → ^14.0.0 to match what @polkadot/api@^16 pulls in transitively, otherwise npm installs two copies of @polkadot/util (top-level 13.x + nested-under-api 14.x) and polkadot.js panics with "multiple versions installed" at runtime. scripts/node_modules/ added to .gitignore; scripts/package.json + scripts/package-lock.json are checked in. Verified live: `AGENTKEYS_CHAIN=heima-paseo node scripts/heima-paseo-sudo.mjs whoami` now connects to wss://rpc.paseo-parachain.heima.network, confirms chain="Heima-paseo" ss58=131 token=HEI EVM_chain_id=2013, and prints Alice's SS58 under the Paseo prefix 131 (jcS2wD5...) along with her well-known pubkey 0xd43593c7... * scripts: gitignore scripts/node_modules/ (npm install --prefix scripts produces it) * scripts: make heima-paseo-bring-up.sh idempotent across re-runs Operator asked "is step 9 and following idempotent? avoid duplicate smart contract by verifying onchain state." Audit found 4 holes; all 4 closed in this commit. Re-running the bring-up is now a no-op when nothing has changed on chain. What's now idempotent: 1. Deployer keypair (step 3) — was: generated a NEW throwaway key on every run unless HEIMA_PASEO_DEPLOYER_KEY was exported. Each run produced a fresh address that then needed re-funding + re-deploying. Fix: on first run, generate + persist to ~/.agentkeys/heima-paseo-deployer.key (mode 0600, OUTSIDE the repo so it's never accidentally committed); on subsequent runs, read the file. Override at any time via env var. 2. Funding (step 4) — was: always sent $FUND_AMOUNT_HEI from Alice via sudo.balances.forceTransfer; no balance check. Fix: query eth_getBalance on the deployer; if balance >= 1 HEI, skip the Alice sudo transfer entirely. Uses node (already a required dep) for BigInt-safe hex->decimal compare (wei values overflow bash arithmetic int64). 3. **Contract deploy (step 5) — the fix the operator specifically asked for**: was: `forge script ... --broadcast` deployed NEW instances every run. Fix: re-source operator-workstation.env to pick up addresses from any prior run, then `cast code $addr` each of the 4 contract addresses against the live chain. If ALL 4 have code on-chain (i.e. contracts still deployed), skip the deploy entirely. If ANY address is missing OR returns "0x" (no code) — e.g. chain reset, fresh env, etc. — redeploy all 4. This handles the chain-reset case automatically. Stub mode (when crates/agentkeys-chain/ doesn't exist yet) produces sentinel 0x1-0x4 addresses that never have on-chain code; the script correctly detects this and "redeploys" the same stubs — no real chain side-effects, no Alice transfers, no wasted gas. 4. Address persistence (step 6) — was: appended new KEY=VALUE lines to operator-workstation.env via `>>`, so 3 runs left 12 contract-address lines (with bash sourcing using the last one, but the file ballooned + git diff was noisy). Fix: `env_set` helper that grep-detects existing lines and either sed-replaces in place (macOS + Linux variants of `sed -i`) or appends only if absent. No duplicates ever. Live-verified idempotency: - Run 1 (SKIP_FUND=1): generated deployer 0xeBdE9E..., persisted key file, stub-deployed 0x1-0x4, appended 5 lines to env file. - Run 2 (same flags): reused persisted key (same 0xeBd address), on-chain check correctly logged "✗ NO code on-chain → redeploy" for the stub address, stub-redeployed same 0x1-0x4, env file still has exactly 5 lines (replaced in place, not duplicated). When real Solidity contracts ship in a future commit replacing crates/agentkeys-chain/, the on-chain check will skip the deploy on the second and all subsequent runs. scripts/operator-workstation.env in this commit is the artifact of the live test runs (5 new lines for the 4 stub addrs + deployer addr). The 0x1-0x4 stubs are placeholder values — they get overwritten by env_set on the first real-deploy run. * scripts: unblock step 9 — auto-skip funding in stub mode + Alice-balance preflight The operator hit three real bugs in step 9 while exercising the end-to-end demo: 1. **`Assertion failed` with no context** in heima-paseo-sudo.mjs's fund subcommand. Root cause: `system_properties.tokenDecimals` came back as the JSON value `[18]` (an array), and `new BN([18])` triggers bn.js's `_initArray` assertion. Fix: pull the array through `JSON.parse(JSON.stringify(...))` (normalizes any polkadot codec wrapper to plain JS), extract `[0]`, coerce to `Number`. Same trap handled for `tokenSymbol = ["HEI"]`. Also: surface `e.stack` in the main catch so future ERRORs land with a stack trace instead of a bare message. 2. **signAndSend hangs forever** waiting for `isFinalized` on Paseo (finalization is unreliable, sometimes 60s+, sometimes never). Switched the resolver to fire on `isInBlock` — sufficient for our "fund then read balance" use case, since the next read sees the new balance as soon as the block is mined. Added a 60s hard timeout so the script can never hang opaquely again. 3. **`Priority is too low (X vs X)`** on retry, because a prior killed run left a stuck tx in Alice's mempool slot. Added a small `tip` (1 nanoHEI) to signAndSend options — substrate's pool replacement rule requires strictly higher priority, and a tip provides it. After (1)+(2)+(3), the tx submitted cleanly but **the validator silently refused to include it because Alice only has ~0.498 HEI on this Paseo deployment** (drained by prior testnet use). The sudo.balances.forceTransfer call needs Alice to have the value she's transferring — sudo bypasses origin checks, not balance checks. Two more fixes for this: 4. **`scripts/heima-paseo-bring-up.sh` step 4 auto-skips when in stub mode** (no crates/agentkeys-chain present). Step 5 emits sentinel 0x1-0x4 addresses without ever submitting a tx in stub mode, so the deployer doesn't need HEI. This was wasting Alice's already-low testnet balance for no benefit AND triggering the timeout when she ran out. 5. **`heima-paseo-sudo.mjs cmdFund` pre-checks Alice's balance** before signAndSend. If `alice.free <= 0.1 ${symbol}` (fee margin), or if `requested > alice.usable`, throw a clear error explaining the gap — "Alice is out of HEI on this chain, top her up before retrying" — rather than letting the tx silently sit unmined in the mempool. Cosmetic: the summary in v2-stage1-demo.sh step 10 was hardcoded to print `s3://${BUCKET}/...` for the smoke-test credential location; that's the MAIL bucket post-§17-split, not where the credential actually lives. Switched to `${VAULT_BUCKET:-$BUCKET}` so post-split runs print the correct vault-bucket path. Verified live: `bash scripts/v2-stage1-demo.sh --from-step 9` now runs end-to-end: [4/7] Sudo-fund SKIPPED — stub mode. Deployer needs no gas. [5/7] ALL 4 contracts already deployed + verified on-chain → skip [6/7] persisted (no duplicates) [7/7] Demo ready. ═══ v2 stage-1 demo complete ═══ The whole 10-step demo (steps 1-10) is now green + idempotent. When real Solidity contracts ship in a future commit replacing crates/agentkeys-chain/, step 4's auto-skip turns off (chain dir present), Alice's balance check fires, and the operator will either (a) succeed if Alice has enough, or (b) get the clear "Alice is out of HEI" message and know to top her up before retrying. * scripts: cmdFund auto-tops-up Alice + new top-up-alice subcommand Operator's idea: "If Alice account does not have enough fund, we should be able to call sudo and mint more hei to Alice." Alice IS the sudoer on Heima Paseo, so she can sudo any pallet call — including `balances.forceSetBalance(alice, BIG)`, which directly sets her free balance from thin air (total issuance climbs, but that's fine for a testnet shared by many testers who keep draining each other's Alice). Implementation: 1. New helper `signAndSendAsAliceWithTip(api, alice, call, label)` — extracts the signAndSend plumbing from cmdFund so cmdFund AND the new top-up flow share one resolve-on-isInBlock + 60s timeout + tip-eviction path. Tip bumped from 1nHEI → 0.001 HEI (1e15 attoHEI) — earlier 1nHEI was sometimes insufficient to evict stuck pool txs from prior attempts. 2. New helper `chainTokenInfo(properties)` — extracts {decimals, symbol} from system_properties handling the array-wrapping codec quirk we hit earlier. Used by both cmdFund and cmdTopUpAlice. 3. New helper `humanize(amountBN, decimals)` — BN → human-readable token string (e.g. "1000.0000 HEI"). Used in every log line. 4. New helper `ensureAliceCanFund(api, alice, decimals, symbol, requestedAmount)` — auto-top-up. Reads Alice's on-chain balance; if `alice.free - 0.1-HEI fee margin < requestedAmount`, sudo-mints her via `balances.forceSetBalance` to max(requested * 100, 1000 HEI). Idempotent — skips if Alice already has enough. 5. cmdFund refactored to call ensureAliceCanFund before the actual forceTransfer. The CLI flow is now: (a) compute requested amount (b) check Alice's balance (c) if short, sudo-mint Alice (d) sudo.balances.forceTransfer(alice, deployer, amount) 6. New `cmdTopUpAlice` subcommand for explicit operator use: node scripts/heima-paseo-sudo.mjs top-up-alice --target-hei 1000 Refuses to LOWER Alice's balance if she's already above target. Outputs JSON with before/after balances + the inclusion block hash. Known live blocker on the current Paseo deployment (NOT a script bug): a prior killed funding attempt left a stuck tx at Alice's nonce 13 in the validator's mempool. The validator can't include it (it's a `force_transfer(alice, X, 100 HEI)` and Alice only has 0.498 HEI), and substrate's pool replacement only works for same-(sender, nonce) — but in this case the validator never EVALUATES the new tx's priority because the slot is held by a tx that's not-yet-failed-not-yet-included. Operator recourse, in order: - Wait ~25-100 min for mempool TTL to drop the stuck tx, then re-run. - Contact Heima dev team to either (a) top up Alice out-of-band via faucet so the stuck transfer becomes valid, or (b) yank the stuck tx via `author.removeExtrinsic` on the validator. - Stay in stub mode (no crates/agentkeys-chain present), which auto-skips step 4 funding entirely (already shipped in 9813c63). The implementation is correct and will work cleanly once the chain state clears OR on a fresh Paseo deployment. * scripts: switch demo default to heima mainnet (paseo collators halted) Operator-observed root cause: Heima Paseo testnet has been **halted since 2026-01-15** — block 2,905,430 frozen for 4+ months. All the funding work I built on top of "Alice can sudo-mint to herself" was correct in principle but useless in practice on a chain that's not producing blocks. Verified live: mainnet (chain_id=212013) has 12s block time, alive and well. This commit switches the v2 stage-1 demo to default to Heima mainnet while preserving the Paseo path for when collators come back up. Rename + generalize: scripts/heima-paseo-bring-up.sh → scripts/heima-bring-up.sh (`git mv` preserves blame; chain-agnostic name reflects multi-chain support) Bring-up script (`heima-bring-up.sh`) now: - AGENTKEYS_CHAIN accepts `heima` OR `heima-paseo`; default is `heima` - Step 2 dynamically reads the right profile (was hardcoded paseo) - Step 2.5 chain-id check bifurcates: heima MUST be 212013; heima-paseo MUST NOT be 212013 (catches profile-vs-RPC drift) - Step 3 deployer key file is per-chain: ~/.agentkeys/heima-deployer.key vs ~/.agentkeys/heima-paseo-deployer.key (keeps mainnet + testnet keys distinct) - Step 4 funding bifurcates: * paseo → existing sudo-via-Alice flow with auto-top-up * heima → balance check only; if deployer < 1 HEI, print clear transfer instructions (deployer addr + RPC + balance-verify curl command) and exit. NEVER auto-spends real HEI. Re-running after manual transfer detects funding and skips. - Step 5 real deploy on mainnet REQUIRES `MAINNET_CONFIRM=1` env var as a paranoid second gate. Stub mode (no crates/agentkeys-chain/) is a no-op regardless of chain. - Step 6 namespaces deployer addr per-chain (HEIMA_DEPLOYER_ADDR_HEIMA vs ..._HEIMA_PASEO; was hardcoded HEIMA_PASEO_DEPLOYER_ADDR) - Step 7 summary shows the actual chain (was hardcoded "heima-paseo") Orchestrator (`v2-stage1-demo.sh`) now: - Default AGENTKEYS_CHAIN: heima-paseo → heima (with explanatory log line) - do_step_9 accepts both chains with chain-specific warnings - Mainnet auto-pauses for operator confirmation (the existing --confirm flag still works; mainnet now triggers it automatically) - read -r _ || true tolerates EOF on stdin (so piped/non-interactive runs don't abort silently from set -e) - MAINNET_CONFIRM env var passed through to bring-up.sh if set Safety summary for accidental mainnet deploys (multiple layers): 1. orchestrator confirmation prompt before bring-up on mainnet 2. bring-up.sh step 2.5 verifies chain_id matches profile (catches misconfigured RPC) 3. step 4 NEVER auto-funds on mainnet; only prints + exits 4. step 5 stub mode = no-op (sentinel addresses, no broadcast) 5. step 5 real deploy REQUIRES MAINNET_CONFIRM=1 env var scripts/operator-workstation.env additions are the artifacts of live test runs against mainnet in stub mode (5 lines: 4 stub contract addresses + deployer addr 0x598c5...). The 0x1-0x4 sentinels follow the same convention as the pre-existing HEIMA_PASEO entries; the on-chain `cast code` check will detect them as missing and "redeploy" (stub-mode no-op) on the next run, OR overwrite with real addresses once Solidity sources ship + MAINNET_CONFIRM=1 is set. Demo doc updates: - 8 references to heima-paseo-bring-up.sh → heima-bring-up.sh - New callout at top of §4.0 explaining Paseo halt + recommending mainnet for new runs - §4.0 intro generalized to describe both chains' funding mechanisms Verified live (mainnet, stub mode): AWS_PROFILE=agentkeys-admin AGENTKEYS_CHAIN=heima \ bash scripts/v2-stage1-demo.sh --only-step 9 </dev/null ==> [step 9/10] Chain backbone bring-up (heima) warn Heima MAINNET — real HEI required ... About to run chain bring-up on heima. MAINNET CONFIRMED (chain_id=212013) ... [4/7] Fund SKIPPED — stub mode (no crates/agentkeys-chain). [5/7] AgentKeysScope = 0x0...01 ✗ NO code on-chain → redeploy (stub-mode sentinel addresses; no real chain side-effect) [6/7] persisted (no duplicates) [7/7] Chain: heima (chain_id=212013) Deployer: 0x598c5... End-to-end clean. Paseo path remains available for when collators come back online. * scripts: derive deployer from BIP-39 mnemonic file (test-hei convention) Operator wants to use their own wallet — specified by a 12-word BIP-39 mnemonic in ./test-hei — as the smart-contract deployer instead of the throwaway-generated key. Verified the mnemonic's SS58 (Heima mainnet prefix 31) = 47NGSq6JE5ZSnymGNa4nFVjWbsuhTfoSKN2jtpk28mUyC1M3 which is the address the operator confirmed against Heima. Changes: scripts/derive-evm-from-mnemonic.mjs (new): tiny ethers-backed helper. Reads a mnemonic file path, derives EVM via the BIP-44 default path m/44'/60'/0'/0/0 (same as MetaMask + Foundry + ethers.Wallet.fromPhrase). Emits one line of JSON {address, privateKey} on stdout; all status (including the derived public address) goes to stderr; the mnemonic + private key are never echoed to stderr. Callers stash stdout in a mode-0600 file. scripts/heima-bring-up.sh step 3: new resolution order is 1. $HEIMA_DEPLOYER_KEY env var 2. $HEIMA_DEPLOYER_MNEMONIC_FILE (default: $REPO_ROOT/test-hei) → derive EVM key, cache in ~/.agentkeys/<chain>-deployer.key 3. Existing persisted key file 4. Generate fresh throwaway Operators drop their mnemonic at ./test-hei and step 3 picks it up automatically. New-key path also prints a TIP pointing at ./test-hei so first-time operators know the option exists. scripts/package.json adds `ethers ^6.13.0` (canonical EVM lib for Wallet.fromPhrase — substrate-side derivation via polkadot.js doesn't expose the raw secp256k1 private key intentionally). .gitignore adds: /test-hei /test-hei.* /.heima-mnemonic /*-mnemonic The mnemonic IS the key — never commit it. ~/.agentkeys/*.key is already outside the repo. Verified live (Heima mainnet, stub mode, no real-money calls): AWS_PROFILE=agentkeys-admin AGENTKEYS_CHAIN=heima \ bash scripts/v2-stage1-demo.sh --only-step 9 </dev/null [3/7] Deployer keypair … deriving deployer from mnemonic at ./test-hei … [derive-evm-from-mnemonic] derived EVM address: 0xdE644936D5B7d5d42032fd08bbA42Fbbfd6663Bc cached private key at ~/.agentkeys/heima-deployer.key (0600) ... Chain: heima (chain_id=212013) Deployer: 0xdE644936D5B7d5d42032fd08bbA42Fbbfd6663Bc Verified the SS58 match (substrate-side cross-check): Substrate sr25519 public key from same mnemonic = 0x2a922b2c4bd021fa75dcce1ddc2fe6b62d743b22bfd547663aff8d4667054507 Encoded under SS58 prefix 31 (Heima mainnet) = 47NGSq6JE5ZSnymGNa4nFVjWbsuhTfoSKN2jtpk28mUyC1M3 ← operator-confirmed For an actual mainnet deploy (when crates/agentkeys-chain/ ships), the operator funds 0xdE644936D5B7d5d42032fd08bbA42Fbbfd6663Bc from their main Heima wallet (any amount ≥ 1 HEI), then re-runs with MAINNET_CONFIRM=1. The flow is now zero-key-juggling on their part. * scripts: add evm-to-substrate-address.mjs helper for Frontier funding Operator hit the standard Heima Frontier gotcha: HEI in their sr25519-derived Substrate wallet (47NGSq6JE5ZSn...) doesn't show up as eth_getBalance on their EVM-derived deployer (0xdE644...) even though both derive from the same BIP-39 mnemonic. Cause: Substrate and EVM use different derivation schemes from the same seed, producing TWO separa…
…dev) (#92) * agentkeys: stage 2 (#90) — P-256 verifier, on-chain K11 binding, M-of-N recovery + companion daemon P-256 ECDSA verify on-chain via pure-Solidity Jacobian-coords implementation (no EIP-7212 precompile dependency — Heima is at London EVM). ~654k gas per verify, sufficient for master-mutation frequency. RFC 6979 test vectors pass. K11Verifier extracts WebAuthn challenge from clientDataJSON at known byte offset (daimo-style), reconstructs msgHash, calls P256Verifier. Binds K11 sig to operation challenge to prevent replay. SidecarRegistry: splits into registerFirstMasterDevice + registerAdditionalMasterDevice + revokeAgentDevice + revokeMasterDevice (M-of-N quorum gated by recoveryThreshold). Stores k11PubX/k11PubY + lastSignCount per device. Per-operator nonce + monotonic sign-count defend against replay. AgentKeysScope: K11Assertion struct gates setScopeWithWebauthn / revokeScope; per-(operator, agent) scopeNonce binds K11 sig to current state. CLI: K11ChainAssertion struct + assert_webauthn_for_chain() extracts (r, s, msgHash, pubX, pubY, authData, clientDataJSON, challengeLocation, signCount) for chain submission. New --rp-id flag enables companion credentials at companion.localhost (distinct platform keychain entry). --emit-chain-payload outputs JSON for cast tx construction. Daemon: new --master-companion mode runs a second daemon instance with its own K10 + K11 at rp_id=companion.localhost. Serves HTTP API: GET /v1/companion/whoami — emits device identity POST /v1/companion/approve — runs WebAuthn ceremony, returns chain payload Scripts: scripts/heima-device-add.sh — register companion as 2nd master scripts/heima-set-recovery-threshold.sh — raise threshold to N scripts/heima-recovery.sh — M-of-N master-device revoke Harness: harness/v2-stage2-demo.sh — idempotent 8-step demo 28 forge tests pass (P256: 8, K11: 6, AgentKeysV1: 14). Stage-2 demo runs green in stub mode and re-runs green (idempotent). Full --webauthn flow requires Touch ID + post-deploy contract addresses. Closes part of #90: - On-chain P-256 verify of K11 assertions - Multi-master M-of-N recovery quorum - Multi-master pairing flow (companion daemon as mobile-app alternative) Deferred to follow-up PRs: - audit-service worker (tier A Merkle relay) - email-service worker - K3 rotation operational runbook - Existing scripts/heima-{device-register,scope-set,scope-revoke}.sh migration to new contract surface (their K11 args changed shape) * docs: stage-2 Heima Mainnet deploy + test runbook + harness fixes Adds docs/v2-stage2-heima-deploy-and-test.md walking the operator through redeploying the stage-2 contract set on Heima Mainnet, re-bootstrapping the primary master, running the stage-2 demo, and exercising the M-of-N recovery flow. Inherits all env setup from docs/v2-stage1-migration-and-demo.md (no parallel test environment). Harness fixes from the first dry-run: - harness/v2-stage2-demo.sh step 5 simplifies to script-existence sanity check in stub mode (was: invoking dry-run which fails on missing companion K11 file). - harness/v2-stage2-demo.sh step 7 same — verifies recovery script is invocable without requiring live chain state. - scripts/heima-device-add.sh adds a dry-run path that doesn't require the companion K11 file (uses placeholder pubkey). - scripts/heima-recovery.sh adds a dry-run path that doesn't require the deployer mnemonic / ethers node_modules. Result: bash harness/v2-stage2-demo.sh --stub --skip-build runs all 8 steps green and is idempotent on re-run. * harness: v2-stage2-demo as single source of truth for deploy+test Stage-2 demo now owns the full lifecycle end-to-end: - step 3: idempotent contract deploy (skips if already on chain; --redeploy forces fresh deploy; reads addresses from broadcast file; writes them to scripts/operator-workstation.env) - step 4: idempotent primary-master bootstrap via new scripts/heima-register-first-master.sh (calls registerFirstMasterDevice with K11 pubX/pubY loaded from the operator's enrollment JSON) - step 5-8 unchanged: companion daemon spin-up, 2nd-master register, recoveryThreshold update, recovery dry-run - step 9: summary with all deployed addresses Now actually deployed to Heima Mainnet (verified live): P256Verifier: 0xb74f0aaf9b72b4e7da872f77c63d805bf1937190 K11Verifier: 0x73446fc9919a0a539b8b08dbda615a64b796ca4f SidecarRegistry: 0x9306c524a5e5c33e9a905b956204207ccaf7a7a1 AgentKeysScope: 0x1276b94f57fd4086670d66acb8c75058176df399 K3EpochCounter: 0x66c08748a6cfa14d9fefaaf5147e41a98db24f53 CredentialAudit: 0xe827ba44931aef8c6f3abfec6b90ecf59f797576 Primary master registered on the new SidecarRegistry, tx 0x5f3a79bc970062ec74aa0deb5618f8a527f638a6d24ba3c4144f09a49600876d (block 9623082). Re-runs are idempotent — all 9 steps log 'skip'/'ok' without re-submitting any tx. * harness: move stage-2 helper scripts into harness/scripts/ The four scripts only referenced by harness/v2-stage2-demo.sh now live under harness/scripts/ — same place as the orchestrator that calls them. Operator-facing stage-1 helpers in scripts/ stay put. scripts/heima-device-add.sh → harness/scripts/heima-device-add.sh scripts/heima-recovery.sh → harness/scripts/heima-recovery.sh scripts/heima-register-first-master.sh → harness/scripts/heima-register-first-master.sh scripts/heima-set-recovery-threshold.sh → harness/scripts/heima-set-recovery-threshold.sh The moved scripts compute REPO_ROOT from two levels up (harness/scripts/<f>.sh → repo root via /../..); the demo paths were updated to point at the new harness/scripts/ location. Hardened the deploy-presence check in step 3: - Distinguishes RPC failure (exit nonzero) from "no code at address" (exit zero with "0x"). - RPC failure → retry up to 8 times with 3s sleep → die rather than redeploy on uncertain state. - "No code" → genuine; trigger redeploy as before. Heima's RPC hits TLS-handshake-EOF transients regularly; this fix prevents an unnecessary redeploy that would orphan the previous set. Same hardening on the balance check in step 3. * harness: companion daemon serves real device_key_hash + clearer step-8 message Stage-2 demo step 5 now derives the companion's on-chain device_key_hash from its K11 cose-pubkey (cast keccak <cose_pubkey_hex>) and passes it to the daemon via --companion-device-key-hash. The daemon's /v1/companion/whoami then returns the real hash that registerAdditionalMasterDevice will use as the storage key, so the later revoke flow can find the device on chain. Stage-2 demo step 8: clearer skip message + when --webauthn is set, prints the companion's device_key_hash + the exact re-run command for executing the revoke. The previous message implied --webauthn alone would do something; really we need a target hash too. * harness/scripts: shared key-resolution lib so scripts accept raw-key files Adds harness/scripts/_lib.sh with resolve_master_key(): - $HEIMA_DEPLOYER_KEY_FILE env var (raw hex or mnemonic) - ~/.agentkeys/heima-deployer.key (raw hex, used by stage-1 operator) - ./test-hei (mnemonic, legacy) Patches the 3 scripts that previously only handled mnemonic files: - heima-device-add.sh - heima-set-recovery-threshold.sh - heima-recovery.sh (preserves --dry-run placeholder path) Fixes a real bug: scripts died with 'missing mnemonic' on operators that bootstrapped from a raw private key (the stage-1 path stores the deployer key at ~/.agentkeys/heima-deployer.key, not a mnemonic at ./test-hei). Also fixes step 8's stale whoami file: always curl fresh so the device_key_hash hint reflects the currently-running daemon, not a prior run where the daemon hadn't been started with the real hash. * fix: WebAuthn challenge double-hash + empty cred-id bytes32 Bug 1 (root cause of step 7 K11VerificationFailed reverts): assert_webauthn_for_chain was passing the 32-byte expected_challenge as a "message" to assert_webauthn_inner_parts, which sha256'd it again before using as the WebAuthn challenge. The on-chain K11Verifier expects the WebAuthn challenge to BE the operation challenge (no extra hash); double-hashing made clientDataJSON.challenge != expected_b64 → ChallengeMismatch / verifyAssertion returns false → contract reverts with K11VerificationFailed. Fix: refactored assert_webauthn_inner_parts to take a [u8; 32] challenge directly. The legacy assert_webauthn_inner path sha256's the message itself before calling (preserves existing behavior). assert_webauthn_for_chain passes the expected_challenge through unchanged. Bug 2 (step 6 cast send "invalid string length"): The companion daemon was receiving an empty --companion-k11-cred-id (demo didn't pass it), so /v1/companion/whoami returned k11_cred_id="". The brittle xxd|head|sed pipeline in heima-device-add.sh produced an all-zeros bytes32 by accident, but the demo's tuple construction had other issues that confused the cast parser. Fix: demo step 5 now computes the cred-id hash from the K11 file (keccak256-style sha256 of the b64url credential id) and passes it to the daemon via --companion-k11-cred-id. heima-device-add.sh uses the hash directly from whoami without re-encoding. Also bumped the empty attestation arg from "0x" to "0x00" (cast tolerates the latter more consistently). Added a sanity-check loop in heima-device-add.sh that validates each bytes32 arg has length 66 before invoking cast, so future malformed inputs fail with a clear error rather than cast's opaque parser msg. * ui: distinguish PRIMARY vs COMPANION K11 ceremony pages WebAuthn assert page now surfaces the role + RP ID prominently so the operator can't confuse which credential they're about to sign with: - Color: blue accent for PRIMARY MASTER (rp_id=localhost), purple for COMPANION MASTER (rp_id=companion.localhost) - Role badge at the top of the card with emoji + label - Dedicated RP-ID callout warning to verify the Touch ID prompt matches the displayed RP - Button text reads "Sign as PRIMARY MASTER" / "Sign as COMPANION MASTER" - Page <title> includes the role so the OS tab list shows it The M-of-N recovery flow opens TWO browser windows in quick succession (one for each daemon's K11 ceremony) — without this distinction the operator could tap the wrong Touch ID prompt and silently produce an assertion the contract rejects. * harness: integrate full M-of-N E2E test (3 devices + 2-of-2 revoke) Stage-2 demo grows from 9 to 10 steps and now exercises the full M-of-N revocation path as part of the default --webauthn flow: Step 8 NEW — Register synthetic 3rd master (the "spare"). The spare is a fresh P-256 keypair generated via openssl, NOT a real WebAuthn passkey. It registers as a 3rd master with roles=3 (CAP_MINT|RECOVERY) via primary K11 sig (1 Touch ID at localhost). State persists at /tmp/agentkeys-spare-current/ for step 9. Why synthetic: the spare is "lost" by design — never needs to sign for its own revocation (primary + companion provide the quorum). Skipping its WebAuthn enrollment saves a Touch ID without weakening the test of any contract surface. Step 9 NEW — Revoke spare via 2-of-2 quorum. Calls heima-recovery.sh with target=spare hash. The script: - Asks primary K11 to sign OP_REVOKE_MASTER challenge (1 Touch ID at localhost — UI shows PRIMARY MASTER badge). - Asks companion daemon /v1/companion/approve to sign same challenge (1 Touch ID at companion.localhost — UI shows COMPANION MASTER badge). - Submits revokeMasterDevice(spareHash, [primarySig, companionSig]). - Contract verifies 2-of-2 quorum + bumps operatorNonce. Post-tx verify: isActive(spare) == false. Step 10 NEW — Cleanup spare local state. Removes /tmp/agentkeys-spare-current/. The on-chain entry stays as revoked=true (audit trail — no on-chain delete by design). End state after a successful run: - 2 active masters: primary (roles=7) + companion (roles=3) - 1 revoked master: spare (roles=3, revoked=true) - recoveryThreshold = 2 - operatorNonce += 3 (register-2nd-master, set-threshold, revoke) Touch IDs on a fresh run: 6 total - companion enroll (step 5, once per setup) - companion register (step 6, once per setup) - set threshold (step 7, once per setup) - spare register (step 8, fresh per run) - primary sigs spare revoke (step 9) - companion sigs spare revoke (step 9) Re-run after this completes: steps 1-7 + 10 skip, steps 8-9 generate a fresh spare (new keypair) and revoke it — 3 Touch IDs per re-run. This makes the demo a repeatable end-to-end test of the M-of-N path without bricking the operator's setup. * harness: auto-version companion when previous instance is revoked Once a companion has been revoked on chain (e.g. as part of an M-of-N quorum test), it can never re-enter the registered-master set under the same deviceKeyHash. Stage-2 demo now detects this and enrolls a fresh companion under a bumped rp_id (companion.localhost → companion-v2.localhost → companion-v3.localhost) so the M-of-N revoke test in step 9 has 2 distinct ACTIVE masters to form the quorum. Changes: - harness/v2-stage2-demo.sh step 5: scans existing K11 files for an active-on-chain companion. If none found, picks the lowest free version slot and enrolls a fresh K11 there. - harness/v2-stage2-demo.sh step 5: passes the computed rp_id to the daemon via new --companion-rp-id flag. - crates/agentkeys-daemon/src/companion.rs: rp_id is now stored in CompanionState + threaded through /v1/companion/whoami responses and assert_webauthn_for_chain calls. - crates/agentkeys-daemon/src/main.rs: new --companion-rp-id flag. - harness/scripts/heima-device-add.sh: reads rp_id from /v1/companion/whoami and derives the K11 file path from it. Net effect: re-running the demo after a 2-of-2 revoke now enrolls a fresh companion-vN, re-establishes a 2-active-master state, and proceeds with the next spare-revoke cycle without operator hand-fixing. * scripts: migrate stage-1 scripts to stage-2 ABI Enables harness/v2-stage1-demo.sh to run green against the new SidecarRegistry + AgentKeysScope contracts deployed in stage 2. Changes: - heima-device-register.sh becomes a thin wrapper: forwards to harness/scripts/heima-register-first-master.sh when no first master is registered; logs skip otherwise. The pre-stage-2 registerMasterDevice() was split into registerFirstMasterDevice + registerAdditionalMasterDevice; this script handles the former. - heima-device-revoke.sh: detects master vs agent target and delegates accordingly. Agent revoke uses the new revokeAgentDevice (no K11 needed). Master revoke delegates to heima-recovery.sh which collects the M-of-N K11 quorum. - heima-scope-set.sh: real WebAuthn ceremony, computes the contract's expected_challenge per OP_SET_SCOPE encoding (servicesDigest + scopeNonce + chainid), builds K11Assertion struct, calls new ABI (bytes K11 -> struct). Stub bytes no longer satisfy the gate. - heima-scope-revoke.sh: same migration as scope-set, computing OP_REVOKE_SCOPE challenge. - All four scripts now use harness/scripts/_lib.sh's resolve_master_key, supporting both raw-key files (~/.agentkeys/heima-deployer.key) and mnemonic files (./test-hei). Effect: operator can now run `bash harness/v2-stage1-demo.sh --webauthn` against the same Heima Mainnet deployment that stage-2 uses, exercising the full operator lifecycle (init -> register -> agent -> scope -> audit) on the new contracts. * ops: K3 rotation runbook + script scripts/heima-k3-rotate.sh — operator-driven K3 epoch advance via K3EpochCounter.advanceEpoch(). Idempotent (--target-epoch N skips if currentEpoch >= N), supports dry-run, signs from the wallet that is the contract's signerGovernance. docs/runbook-k3-rotation.md — step-by-step operator runbook: prerequisites, the one-command flow, post-rotation verification, when to rotate (quarterly hygiene + TEE-compromise indicator), lazy vs eager re-encryption trade-offs, and the stage-3 migration path to move signerGovernance from EOA to M-of-N multisig. Verified end-to-end on Heima Mainnet (dry-run): K3EpochCounter at 0xeacc97d4e7854c52d4736e5fba2dc7c2c2b147d9 has currentEpoch=1 and signerGovernance points at the deployer. * audit: tier-A Merkle relay worker + on-chain appendRoot path Contract surface (CredentialAudit.sol): - New `appendRoot(operatorOmni, merkleRoot, batchEntryCount)` stores a per-operator AuditRoot entry, emits AuditRootAppended. Operators reconstruct per-event proofs from leaves in S3. - New `verifyEntryInRoot(operatorOmni, rootIndex, proof[], leaf)` validates a sorted-pairs Merkle proof on chain. Matches OpenZeppelin convention so the Rust-side proof emission is directly verifiable without further transformation. - Existing `append()` per-event path (tier C) untouched. Forge test test_CredentialAudit_AppendRoot_AndVerifyMembership covers the round-trip with a 4-leaf tree. New crate agentkeys-worker-audit: - `merkle.rs`: minimal Merkle root + proof helpers using keccak256 with sorted-pairs encoding (matches the contract verifier byte-for-byte). Doc tests + 4 unit tests pass. - `state.rs`: per-operator in-memory event queue with flush semantics. Drains the queue, computes Merkle root, writes per-event leaves + proofs to a JSONL file at /tmp/audit-leaves-<root>.jsonl. - `handlers.rs`: HTTP surface POST /v1/audit/append — queue event POST /v1/audit/flush/:operator — drain one queue POST /v1/audit/flush-all — drain all queues - `main.rs`: bind axum at 127.0.0.1:9092; periodic auto-flush every --flush-interval-secs (default 300s; 0 = manual only). Each flush logs the Merkle root + leaves path. Chain submission via `cast send appendRoot` is operator-driven (separate from this process so the worker doesn't need a deployer key). End-state: operators wanting per-event-tx semantics keep using tier C (`heima-credential-audit.sh` direct write). Operators wanting batched gas (one tx per N events / per 5min) point their daemon at this worker and emit per-event POSTs; the worker computes roots and the operator periodically submits roots via `cast send`. * email: agentkeys-worker-email — SES send + per-actor inbox list New crate agentkeys-worker-email. Surfaces: POST /v1/email/send Body: { from, to[], subject, body_text, body_html? } Wraps aws-sdk-sesv2::SendEmail with the operator's SES identity (must be verified per the #83 setup workflow). Returns the SES message_id. GET /v1/email/inbox/:actor_omni Lists objects under s3://$AGENTKEYS_VAULT_BUCKET/bots/<actor_omni>/inbound/. Inbound routing itself is the SES routing Lambda from #83; this worker only exposes what's already been delivered to S3. CLI args: --bind default 127.0.0.1:9093 --inbox-bucket env AGENTKEYS_VAULT_BUCKET, required Builds against aws-sdk-sesv2 1.118 + aws-sdk-s3 1.132. No new dependencies introduced at the workspace level (aws-config + s3 are already used by worker-creds). Operator workflow: spin up alongside worker-creds + worker-memory on the broker host, route per-agent outbound mail through this worker instead of having each agent directly call SES. Cap-token verification on /v1/email/send is left as a follow-up (current shape assumes the worker is on a private interface — operators expose it only on the sidecar daemon's localhost, same as worker-creds). * docs: K3 rotation test verdict — 4 rounds green on Heima Mainnet Live E2E test of scripts/heima-k3-rotate.sh per agentkeys-harness skill: - Round 1: epoch 1 → 2 (1 tx) - Round 2: epoch 2 → 3 (1 tx) - Round 3: target=3 (already there) → skip, no tx, 0 gas - Round 4: target=6 (3-step advance) → 3 txs Total: 5 real txs on K3EpochCounter = 0xeacc97d4e7854c52d4736e5fba2dc7c2c2b147d9. The contract is forward-only by design — no "rotate back" — so the "back and forth" test is bounded to forward-path correctness + the idempotency skip on re-targets-to-current. Both work as designed. K3EpochCounter is now at epoch 6 on Heima Mainnet. The signer enclave will retain historical K3_v[1..5] for decrypt of pre-rotation blobs; new writes use K3_v[6]. * ui: enrollment page + macOS Touch ID dialog readability Two fixes: 1. Enrollment page (serve_enroll_page) now matches the assert-page visual language — role badge (PRIMARY MASTER blue, COMPANION MASTER purple), RP-ID surfaced explicitly, button text reads "Enroll as PRIMARY MASTER" / "Enroll as COMPANION MASTER". Previously the enrollment page was role-agnostic which made it easy to tap Touch ID on the wrong RP when re-enrolling. 2. WebAuthn user.name shown in the macOS Touch ID dialog ("Use Touch ID to sign in to 'localhost' with your passkey for <NAME>") was previously the full 64-char operator_omni hex, which truncates awkwardly on screen. Now reads "AgentKeys Primary Master (0x941cb1c3…)" or "AgentKeys Companion Master (0x941cb1c3…)" — human-readable + a 10-char omni prefix for cross-operator disambig. Takes effect on NEW enrollments only — existing credentials retain whatever user.name was set when they were originally enrolled. To refresh the display name, delete ~/.agentkeys/k11/<omni>--<rp>.json and re-enroll. The "white text in white background" in the macOS Passkey-source filter row is macOS system UI (the picker for which provider supplies the passkey — iCloud Keychain, 1Password, etc.); it's outside our HTML control. The other observed truncation is fixed by this commit. * docs(arch): §16.4 brief intro to K3 rotation flow Operator-facing summary of what K3 rotation does and doesn't change: - contract addresses, devices, scopes, threshold unchanged - on-chain epoch counter advances + emits K3Rotated event - signer enclave retains historical K3 versions for legacy decrypt - workers swap to new epoch for new writes via SSE - one-command operator action: `bash scripts/heima-k3-rotate.sh` - links to full runbook at docs/runbook-k3-rotation.md - notes the stage 1-2 simplification (KEK from env per §22b.2) means rotation is forward-compatible but not yet driving worker re-key Also documents the eager-re-encrypt follow-up gated behind a confirmed TEE compromise scenario (stage 3 tracked in §22b.5). * fix(stage-2): codex adversarial review — 7 critical/high/medium findings Codex flagged 8 findings; 7 are addressed here (C1, C2, C3/M1, H1, H2, M2 + test coverage). The remaining one (codex H3 "K10+K11") is a false positive: msg.sender check IS the K10 signature — EVM tx signing is secp256k1 over the whole tx by the master wallet. Added comments where helpful. Contract fixes (require redeploy): C1: SidecarRegistry.revokeMasterDevice — refuse to revoke if it would leave < max(1, recoveryThreshold) active recovery-capable masters. Prevents permanent operator stranding. C2: SidecarRegistry.setRecoveryThreshold — refuse newThreshold > activeRecoveryMasterCount. Prevents permanent operator stranding via unsatisfiable quorum. C3/M1: CredentialAudit.appendRoot — auth-gate by operator's master wallet (via injected SidecarRegistry reference). Previously any account could pollute an operator's root list. H1: K11Verifier.verifyAssertion — three new envelope checks: - authData[0:32] == expectedRpIdHash (per-credential, stored on register at DeviceEntry.k11RpIdHash). Prevents cross-RP replay. - authData[32] has UP|UV flags. Prevents stolen-device-without- biometric assertions. - clientDataJSON starts with `{"type":"webauthn.get"`. Prevents replay of webauthn.create (enrollment) assertions. M2: CredentialAudit + worker Merkle — domain-separate leaves (0x00 prefix) and internal nodes (0x01 prefix). Prevents an internal- node digest from impersonating a leaf at shorter depth. ABI changes: - SidecarRegistry.registerFirstMasterDevice + registerAdditionalMaster now take an extra bytes32 k11RpIdHash arg (the operator's K11 enroll rp_id is hashed and stored). - K11Verifier.verifyAssertion takes the rpIdHash; callers (SidecarRegistry, AgentKeysScope) read entry.k11RpIdHash. - CredentialAudit constructor takes the SidecarRegistry address. Harness changes: - heima-register-first-master.sh + heima-device-add.sh + heima-register- spare-master.sh compute sha256(rp_id) from the K11 enrollment file and pass it as the new arg. - v2-stage2-demo.sh step 6 + 7 fail-fast on device-add/threshold-set failures + verify on-chain state matches before advancing to step 9. Codex H2: previously silent failures could false-green step 9. Tests: + 5 new K11Verifier tests: RpIdHashMismatch, UserPresenceMissing (no flags, UP-only), WrongClientDataType (webauthn.create), all pass. + CredentialAudit_AppendRoot_RejectsNonMaster (vm.prank attacker). + Internal-node-as-leaf attack test in both forge + Rust Merkle suite. - Total: 33 forge tests (was 28), 7 worker-audit unit tests (was 6), all green. Deploys will fail against the existing PR #87-deployed contracts — operator must redeploy via the demo's step 3 (forced) or by running `bash harness/v2-stage2-demo.sh --redeploy`. * deploy: stage-2 contracts with codex fixes redeployed on Heima Mainnet New addresses (PR commit 5834c1d 'fix(stage-2): codex adversarial review'): P256Verifier: 0xda5b772f9d6c09abe80414eea908612df9b54749 K11Verifier: 0x5a441431f08e0f5f5ed10659620cb4e0e814e627 SidecarRegistry: 0x1ac62f1c2d828476a5d784e850a700dc1f17e0be AgentKeysScope: 0xd44b375daefc65768f417d0f0125b68d5ba7df3b K3EpochCounter: 0x6c9e675c699a06acefbc156afdee6bfbfe32ccb3 CredentialAudit: 0x63c4545ac01c77cc74044f25b8edea3880224577 Previously-deployed instances (bc232ebcb47fa672aa2a1b2b0481c7ff9a86531b et al) are now abandoned. They have the pre-codex-fix ABI which is incompatible — DeviceEntry layout changed (added k11RpIdHash field). Operator's primary master must re-register via harness/scripts/heima-register-first-master.sh against the new SidecarRegistry; companion + spare flows then continue normally. * issue #90: co-locate audit/email/cred/memory workers on broker host (dev) Dev-only co-location of the 4 service workers on the same EC2 box as the broker, behind per-worker nginx vhosts. CLAUDE.md: "for production, we will isolate all the services for the security issue" — the per-subdomain layout is the migration seam, so a future move to dedicated hosts only needs the A record + IAM principal to change. Topology: broker.litentry.org :8091 agentkeys-broker signer.litentry.org :8092 agentkeys-signer audit.litentry.org :9092 agentkeys-worker-audit (Merkle relay) email.litentry.org :9093 agentkeys-worker-email (SES + S3 inbox) cred.litentry.org :9094 agentkeys-worker-creds (credential CRUD) memory.litentry.org :9095 agentkeys-worker-memory (memory CRUD) setup-broker-host.sh — builds + installs the 4 worker binaries, auto- generates worker-{creds,memory}.env with stable KEK secrets (preserved across re-runs so existing blobs stay decryptable), writes 4 systemd units, writes 4 nginx vhosts via shared write_worker_nginx_site(), and probes /healthz on each port post-restart. New CLI flags: --audit-host, --email-host, --cred-host, --memory-host, --chain-rpc, --vault-bucket, --memory-bucket, --scope-addr, --registry-addr, --k3-counter-addr, --without-workers. Re-runs without flags now re-read previously-configured values from /etc/agentkeys/worker-{creds,memory}.env so the script stays idempotent for non-default deployments. dns-upsert-workers.sh (NEW) — single atomic Route 53 change-batch UPSERT for all 4 A records. Validates the caller is on agentkeys-admin, refuses RFC1918 / TEST-NET-2 (Cloudflare WARP / Zscaler / corporate VPN) EIPs, waits for Route 53 INSYNC + Cloudflare DoH propagation before exiting. verify-workers.sh (NEW) — laptop-side end-to-end check: DNS resolves via Cloudflare DoH → TLS cert is Let's Encrypt → /healthz returns HTTP 200 with the per-worker expected body marker. Exits non-zero with per-failure diagnostics. --no-tls for the HTTP-only first-pass phase. worker-audit/main.rs + worker-email/main.rs: GET /healthz → "ok" so probe_or_die can verify boot (worker-creds + worker-memory already had it). operator-workstation.env: derive WORKER_{AUDIT,EMAIL,CRED,MEMORY}_HOST + AGENTKEYS_WORKER_*_URL from \$BROKER_HOST, mirroring the SIGNER_HOST pattern. docs/cloud-setup.md: new §1.4 (TOC row) + §7 "Service workers" with the concern table (mirrors §6 signer), §7.1 DNS one-shot helper, §7.2 TLS cert loop + nginx flip, §7.3 verification. Existing §7 Cleanup → §8. heima-scope-set.sh + heima-scope-revoke.sh: graceful skip with {"ok":true,"skipped":"no-webauthn-k11"} when no mode:webauthn K11 is enrolled, so harness/v2-stage1-demo.sh (default stub mode) is fully CI- automatable without operator Touch ID. * fix: worker-{creds,memory} need REGISTRY + K3_EPOCH_COUNTER addresses worker-creds and worker-memory both call profile_env() for all THREE contract addresses (SidecarRegistry, AgentKeysScope, K3EpochCounter) at state construction — verified live by the boot failure on broker host: Error: SIDECAR_REGISTRY_ADDRESS_HEIMA must be set Caused by: environment variable not found The auto-generated /etc/agentkeys/worker-creds.env was only writing SCOPE_CONTRACT_ADDRESS_HEIMA, omitting the other two — fixed. Also added AGENTKEYS_CHAIN=heima to both env files so the chain-profile resolution is explicit instead of relying on the worker-side default (matches what the existing chain helpers do). * issue #90: wire audit + email workers into stage-1 + stage-2 demos New step exercises the 4 co-located service workers as a tier-A relay: queue 2 audit events → flush → on-chain CredentialAudit.appendRoot → verify rootCount + getRoot match. Plus an email worker /healthz + /inbox smoke. Stage-1 demo: STEP_TOTAL 15 → 16, new step 15 between audit-append and summary; summary renumbered to step 16. Stage-2 demo: STEP_TOTAL 10 → 11, new step 10 between M-of-N revoke and cleanup; cleanup renumbered to step 11. scripts/heima-worker-smoke.sh (NEW) — drives the full flow: 1. precheck both workers' /healthz 2. POST 2 events → audit worker /v1/audit/append 3. POST /v1/audit/flush/<operator_omni> → Merkle root + leaves 4. cast send CredentialAudit.appendRoot from operator master wallet 5. cast call rootCount + getRoot to verify on-chain root matches flush 6. GET /v1/email/inbox/<actor_omni> as soft-warn smoke (the broker EC2 IAM lacks s3:ListBucket on the inbox bucket today — out-of-scope follow-up; worker is deployed + /healthz green so the demo continues without breaking the chain green-bar) Live-tested 4 rounds against Heima Mainnet — rootCount progressed 0→1→2→3→4→5→6→7→8 across stage-1 + stage-2 runs with all 8 on-chain Merkle roots verified by getRoot() readback. Idempotency: every re-run is a clean skip (no chain mutation) or adds a fresh tier-A root. Sibling fixes (same bug class — stale DeviceEntry struct offsets after codex H1 added k11RpIdHash + k11PubX + k11PubY): heima-agent-create.sh + heima-device-revoke.sh — switched the idempotency check from hex-offset slicing of getDevice() to the typed isActive(bytes32)(bool) view. The old code read offset 320 for registeredAt; after the struct grew, registeredAt now lives at offset 512, so the offset-based check always returned 'not yet registered' on re-run and registerAgentDevice reverted with DeviceAlreadyRegistered (0xa98bbce0). isActive is struct-agnostic. heima-scope-set.sh + heima-scope-revoke.sh — when USE_WEBAUTHN=0 (stub mode) AND the local K11 file is mode=webauthn (from a prior real ceremony), skip cleanly instead of triggering Touch ID. Demo stub-mode runs on a laptop with prior webauthn enrollment were otherwise prompting for Touch ID and dying on the dismissed dialog. The 'stub-mode-refuses-touchid' skip payload makes this explicit. * issue #90: wire OIDC federation into cred + memory workers (Q3) Closes the OIDC isolation gap from PR #92 review (issue #90 Q1 + Q3): the broker had full federation infrastructure (handlers/oidc.rs, mint.rs, sts.rs) but the workers bypassed it — every S3 call went through the broker EC2 instance profile, so the per-actor IAM scoping defined in provision-vault-role.sh's PrincipalTag policy was never exercised. Worker code change (backwards compatible): crates/agentkeys-worker-creds/src/aws_creds.rs (NEW) - OptionalStsCreds axum extractor: parses three optional headers X-Aws-Access-Key-Id X-Aws-Secret-Access-Key X-Aws-Session-Token Returns None if any are missing (partial = error, refuse to mint a half-authed S3 client). - StsCreds::build_s3_client(region) — per-request S3 client backed by the passed-through STS creds. - s3_for_request(default, region, override) — falls back to the default instance-profile client when override is None. - 4 unit tests covering header presence / absence / partial. crates/agentkeys-worker-creds/src/handlers.rs cred_store + cred_fetch + cred_teardown — accept OptionalStsCreds, use the per-request client when present. crates/agentkeys-worker-memory/src/handlers.rs memory_put + memory_get + memory_teardown — same pattern; re-exports aws_creds from agentkeys_worker_creds (no duplication). Backward compat: requests without the three X-Aws-* headers fall back to state.s3 (instance profile) — existing stage-1 + stage-2 demo flows keep working unchanged. harness/v2-stage3-demo.sh (NEW, 8 steps) End-to-end OIDC isolation proof on Heima Mainnet: 1. SIWE wallet_sig auth → session JWT 2. POST /v1/mint-oidc-jwt → STS-compatible web identity token 3. AssumeRoleWithWebIdentity → STS creds tagged with PrincipalTag/agentkeys_actor_omni = derive_omni(master wallet) 4. POSITIVE: PUT s3://vault/bots/<own actor_omni>/credentials/… → HTTP 200 5. NEGATIVE: PUT s3://vault/bots/<wrong actor_omni>/credentials/… → AccessDenied (IAM rejects cross-actor write — the proof) 6+7. Same positive+negative pair on the memory bucket — soft-skip when memory bucket not yet provisioned (follow-up). 8. Cleanup with admin profile. Live-tested against Heima Mainnet. Step 5 verified: AWS IAM itself rejected the cross-actor PUT with AccessDenied — proves the ${aws:PrincipalTag/agentkeys_actor_omni} scoping in scripts/provision-vault-role.sh works as designed. Even if a worker were compromised, it could not write to another actor's prefix when using STS creds passed through from the broker mint flow. Architectural answers to the review (#90 Q1 + Q2): Q1 ("is OIDC disrupted by the new service isolation design?"): Was, yes — workers bypassed federation. NOW WIRED. Workers respect STS creds when passed; fall back to instance profile otherwise so existing stage-1+2 flows are unchanged. Q2 ("why does broker need s3:ListBucket — Lambda should sort incoming email into per-actor folders"): User is right architecturally. The 500 we soft-warned on in /v1/email/inbox is the symptom of the same OIDC bypass — the email worker uses instance profile and tries global ListObjects without scoping. Architecturally correct flow: SES inbound → Lambda sorts to bots/<actor>/inbound/ → email worker reads via OIDC-scoped STS creds, never global ListBucket. The fix is the same shape as this PR — pass-through STS creds via X-Aws-* headers — but is left as a follow-up: this PR ships the plumbing + proves OIDC works end-to-end; wiring the email worker + Lambda routing is a separate change. Tracked in #90 followups. * issue #90 codex review: fix downgrade attack + secret redaction Addresses 2 of 4 codex adversarial findings on commit 913179a: [P2 — downgrade attack] aws_creds.rs OptionalStsCreds extractor silently fell back to the broker EC2 instance profile when caller omitted X-Aws-* headers. A malicious caller could deliberately drop the headers to bypass the OIDC-scoped IAM session and get broker-wide S3 access. Fix: `AGENTKEYS_WORKER_REQUIRE_STS=1` env var puts the worker in strict mode — every request must carry all three X-Aws-* headers or gets HTTP 401. Also: partial header sets (1 or 2 of 3 present) ALWAYS reject with 401 regardless of strict mode — silently dropping half-passed creds is the same downgrade surface. Default off for backward compat; production deploys should turn it on. [P3 — credential leak via Debug] StsCreds previously derived Debug, so any future tracing::debug! or dbg!() call would log secret_access_key and session_token verbatim. Custom Debug impl now redacts both and shows only the access_key_id prefix (which AWS CloudTrails anyway). New tests: - debug_redacts_secret_and_session_token (asserts the Debug output doesn't contain the secret bytes; <redacted> marker present) - parser_distinguishes_no_headers_from_partial (locks the extractor's contract — no headers = backward compat, partial = always reject) Two codex findings deliberately left as follow-ups, not fixed in this commit: [P2 — memory worker OIDC not proven] The harness only mints agentkeys-vault-role creds, which scope to the vault bucket only. The memory worker writes to a separate memory bucket which isn't covered. A dedicated agentkeys-memory-role with the same tag-scoping pattern is the architecturally correct fix; tracked as PR followup. [P2 — vault bucket policy allows whole-bucket ListBucket] In scripts/apply-vault-bucket-policy.sh:109 — pre-existing, separate from this PR's surface. Adding an s3:prefix=bots/${aws:PrincipalTag/…} condition to the bucket-policy ListBucket statement closes the cross-actor key-name enumeration. Filed for the bucket-policy hardening followup. * issue #90 codex review: close remaining 2 deferred findings Lands the two findings deferred from commit 18e709b. Both verified live on Heima Mainnet via the extended harness/v2-stage3-demo.sh (11 steps, all green). [P2 — memory worker OIDC scoping] NEW agentkeys-memory-role + dedicated memory bucket, mirroring the vault data-class layout per arch.md §17.2. A future memory-worker compromise now cannot reach the credentials bucket and vice versa. scripts/provision-memory-bucket.sh (NEW) — mirror of provision-vault-bucket.sh scripts/provision-memory-role.sh (NEW) — federated trust + 3-statement inline policy scoped to $MEMORY_BUCKET/bots/${PrincipalTag}/memory/* scripts/apply-memory-bucket-policy.sh (NEW) — v3 bucket policy [P2 — bucket-policy ListBucket whole-bucket allow] Was: one statement listed [Get, Put, Delete, ListBucket] under one Resource[bucket, bucket/...] with NO s3:prefix condition — any tagged session could enumerate all keys. Now: SPLIT into two statements: VaultListV3 / MemoryListV3 — ListBucket ONLY, on the bucket ARN, Condition StringLike s3:prefix = bots/${PrincipalTag}/<class>/* VaultObjectsV3 / MemoryObjectsV3 — Get/Put/Delete on the prefixed-object ARN, no prefix condition (resource ARN already scopes) scripts/apply-vault-bucket-policy.sh (UPDATED) — v2 → v3 split scripts/apply-memory-bucket-policy.sh (NEW) — v3 split from day one Demo extended (harness/v2-stage3-demo.sh, STEP_TOTAL 8 → 11): step 3: mint TWO STS sessions (vault role + memory role) step 4-5: vault PUT positive (own) + negative (other) — pre-existing step 6: vault LIST negative (other prefix → AccessDenied) — codex P2 verifier step 7-8: memory PUT positive (own) + negative (other) step 9: memory LIST negative (other prefix → AccessDenied) step 10: cross-role isolation — vault creds → memory bucket → AccessDenied + memory creds → vault bucket → AccessDenied step 11: cleanup Also: `expect_access_denied` helper distinguishes IAM-rejection (AccessDenied / HTTP 403) from setup-bug failures (NoCredentialsErr, NoSuchBucket, InvalidAccessKeyId, TokenRefreshRequired). Naive `grep AccessDenied` would pass on any failure — codex's exact warning. operator-workstation.env: + MEMORY_BUCKET=agentkeys-memory-${ACCOUNT_ID} + MEMORY_ROLE_ARN=arn:aws:iam::${ACCOUNT_ID}:role/agentkeys-memory-role Live-tested 2026-05-20 on Heima Mainnet: - memory bucket created (AssumedArn=…agentkeys-memory-role) - vault-bucket policy v2 → v3 swap (2 statements live) - memory-bucket policy v3 from scratch (2 statements live) - 11/11 demo steps green: [4] vault PUT own prefix → SUCCEEDED [5] vault PUT other prefix → AccessDenied [6] vault LIST other prefix → AccessDenied [7] memory PUT own prefix → SUCCEEDED [8] memory PUT other prefix → AccessDenied [9] memory LIST other prefix → AccessDenied [10] vault creds → memory bucket → AccessDenied [10] memory creds → vault bucket → AccessDenied * harness: log phase-1 acceptance for PR #92 (3-demo verification) All three demos (stage-1, stage-2, stage-3) green on Heima Mainnet after the codex review fixes. Clippy clean on worker-creds + worker-memory. PR ready to merge. * stage-3: add worker encrypt/decrypt roundtrip tests (steps 11+12) User's call-out — "the cred encryption and decryption is not tested". Stage-3 previously proved IAM scoping at the AWS layer but skipped the worker's AES-256-GCM envelope, so the actual encrypt→S3→decrypt path through the HTTP API was unexercised. The envelope.rs primitive has 8 unit tests, but the wire-protocol roundtrip wasn't. Stage-3 demo extended (STEP_TOTAL 11 → 13): [11] Cred worker encrypt/decrypt roundtrip: 1. mint cred-store cap via POST /v1/cap/cred-store (broker) 2. POST /v1/cred/store with cap + base64(plaintext) → worker KEK-encrypts (AES-256-GCM, AAD-bound to operator+actor+service+k3_epoch), S3 PUTs the envelope 3. mint cred-fetch cap via POST /v1/cap/cred-fetch 4. POST /v1/cred/fetch with cap → worker S3 GETs the envelope, KEK-decrypts, returns plaintext 5. assert returned plaintext == original (byte-for-byte) [12] Memory worker encrypt/decrypt roundtrip: same shape against /v1/memory/put + /v1/memory/get. Memory worker has no dedicated cap-mint endpoint yet (follow-up); cred-* caps work against memory because both workers verify the same broker- signed CapToken shape with the same CapOp::Store / CapOp::Fetch. Graceful skip handling: - 'agent scope not set on chain' → skip with 'run stage-1 --webauthn first' - 'AGENTKEYS_CHAIN_RPC_HTTP not set' → skip with 'redeploy broker' - 'DeviceRoleMissing' → skip with 'out-of-scope here' These map cleanly to operator-actionable prerequisites; demo continues green without those steps when prerequisites aren't met, but the prerequisite is reported, not hidden. Broker fix: setup-broker-host.sh now bakes AGENTKEYS_CHAIN + AGENTKEYS_CHAIN_RPC_HTTP into the broker's systemd Environment= lines. Previously the broker process had no chain RPC, so /v1/cap/cred-{store, fetch} hit 502 'RPC URL not set' at request time. This was a pre-existing gap surfaced by exercising the cap-mint path for the first time in this PR — the broker's stand-alone deploy never hit cap.rs's chain check before because no demo step minted caps. * isolation invariants: codify the 4-layer rule + cross-actor test (step 13) Three changes from user review: 1. NEW stage-3 step 13: NEGATIVE broker cap-mint isolation. Try to mint a cap-token with operator_omni != session_omni → expect HTTP 4xx with OperatorMismatch. This proves the MOST UPSTREAM isolation gate works: actor A's session JWT cannot mint caps for actor B. If this ever silently returns 200, every cred + memory blob in S3 is compromised — A could mint B's cap, hand to worker, worker writes under B's prefix. Live-verified on Heima Mainnet 2026-05-20: [13] NEGATIVE cap-mint cross-actor → HTTP 403 OperatorMismatch ✓ Independent of broker redeploy: session-omni check fires BEFORE the chain RPC check in handlers/cap.rs, so this gate works on the current (stale-RPC) broker too. 2. CLAUDE.md — NEW "Per-actor + per-data-class isolation invariants (issue #90)" section codifies the 4-layer defense: Layer 1 — broker cap-mint → session_omni == operator_omni Layer 2 — worker chain-verify → independent re-check of layer 1 Layer 3 — AWS IAM PrincipalTag → s3 resource scoping per-actor Layer 4 — bucket separation → per-data-class IAM roles Test-discipline rule: every PR adding a new worker, data class, or broker auth method MUST extend the stage-3 demo with negative isolation tests for all four layers. Don't ship features with only POSITIVE-path coverage. 3. CLAUDE.md — answers "why no /v1/cap/memory-* endpoint" with a concrete example: cap-tokens are data-class-agnostic. The same Store cap minted for service=openrouter can be POSTed to either /v1/cred/store (writes to vault bucket credentials/) or /v1/memory/put (writes to memory bucket memory/). The URL picks the data class; the cap just authorizes the operation. Adding dedicated memory cap endpoints would add audit clarity ("this cap was minted intending memory access") but no security boundary — isolation comes from the per-data-class IAM roles (layer 4). Deferred until payments-worker forces a third data class. * cap-token: data-class-explicit isolation (no cross-pollution between vault + memory) User callout — "make it explicit that one cannot pollute other permission." Before this commit, cap-tokens didn't carry a data-class binding: a cred-store cap and a memory-put cap were structurally identical. The URL the cap was POSTed to picked the bucket. Isolation lived only at the AWS IAM PrincipalTag + per-data-class IAM-role layer. If the IAM grants were ever accidentally broadened, cross-data-class pollution would slip through silently. Now: data_class is a SIGNED FIELD in the cap payload. The cap layer itself enforces per-data-class isolation, ahead of any AWS call. Schema change (REQUIRED field, no backward compat — coordinated upgrade): enum DataClass { Credentials, Memory } struct CapPayload { ... op: CapOp, data_class: DataClass, // NEW ... } Broker (crates/agentkeys-broker-server/src/handlers/cap.rs): - Add DataClass enum (mirror of worker's), add to CapPayload - mint_cap signature gains data_class param; statically derived per route - NEW endpoints: cap_memory_put + cap_memory_get (mint with DataClass::Memory) - Existing cap_cred_store + cap_cred_fetch mint with DataClass::Credentials Broker routes (crates/agentkeys-broker-server/src/lib.rs): + .route("/v1/cap/memory-put", post(cap_memory_put)) + .route("/v1/cap/memory-get", post(cap_memory_get)) Worker side (crates/agentkeys-worker-creds/src/verify.rs): - Add DataClass enum + field to CapPayload + DataClassMismatch error - NEW pub fn check_data_class(token, expected) — symmetric with check_op - Tests: data_class_serializes_snake_case + check_data_class_accepts_match + check_data_class_rejects_cross_class Worker handlers (worker-creds + worker-memory): - verify_cap now calls check_data_class with their respective class: worker-creds → DataClass::Credentials worker-memory → DataClass::Memory - Reject mismatched caps with HTTP 403 cap_data_class_mismatch Demo extension (harness/v2-stage3-demo.sh, STEP_TOTAL 14 → 16): [11] cred encrypt/decrypt roundtrip — now uses /v1/cap/cred-store [12] memory encrypt/decrypt roundtrip — now uses /v1/cap/memory-put (NEW endpoint) [14] NEW negative test: mint cred-class cap, POST to /v1/memory/put → expect HTTP 403 cap_data_class_mismatch [15] NEW negative test: mint memory-class cap, POST to /v1/cred/store → expect HTTP 403 cap_data_class_mismatch CLAUDE.md ("Per-actor + per-data-class isolation invariants"): Replaced "why no memory cap-mint endpoint" section (now obsolete) with "Cap-tokens are data-class-explicit" — explains the 4-endpoint shape, shows the concrete reject example, justifies route-per-class over a data_class query param (broker can't accidentally mint the wrong variant from a typed-route handler). Tests: worker-creds verify::tests — 14/14 (3 new for DataClass) broker-server handlers::cap::tests — 24/24 (1 new for data_class serialization) cargo build -p worker-creds -p worker-memory -p broker-server — exit 0 Live deploy: requires broker host redeploy via setup-broker-host.sh to pick up the new mint_cap signature + new memory routes. The stage-3 demo steps 14+15 will skip cleanly until the redeploy lands — the isolation IS enforced (workers reject cred-class caps), but the new endpoints don't exist on the current broker yet. * broker: bake contract addresses into systemd env (closes step-11 502) After redeploying with the data_class change (commit 690f54c), step 11 of the stage-3 demo surfaced a SECOND broker-side env gap: HTTP 502 from /v1/cap/cred-store: {"error":"SIDECAR_REGISTRY_ADDRESS_HEIMA unset","reason":"chain_rpc_error"} The broker's handlers/cap.rs reads three contract addresses at request time to verify device + scope + k3_epoch on chain: - SIDECAR_REGISTRY_ADDRESS_HEIMA - SCOPE_CONTRACT_ADDRESS_HEIMA - K3_EPOCH_COUNTER_ADDRESS_HEIMA Before this commit, setup-broker-host.sh baked AGENTKEYS_CHAIN_RPC_HTTP into the broker systemd unit but NOT the contract addresses. The cap- mint code path had never been exercised before this PR, so the gap went unnoticed. Fix (setup-broker-host.sh): add the three contract addresses to the broker's Environment= block, pulled from $REGISTRY_ADDR / $SCOPE_ADDR / $K3_COUNTER_ADDR (already populated earlier in the script via the sourced scripts/operator-workstation.env). The operator's operator-workstation.env stays the single source of truth for contract addresses across laptop + broker host. Stage-3 demo also gets a sibling skip-detection (harness/v2-stage3-demo.sh) so steps 11+12+14+15 cleanly skip with the redeploy-broker message instead of failing on this specific error shape. To unblock the stage-3 worker encrypt/decrypt + cross-class-rejection tests after this commit: ssh broker.litentry.org "cd ~/agentKeys && git pull && bash scripts/setup-broker-host.sh --yes" * broker + worker: parse_device_entry knows the 11-field struct (codex H1 alignment) Closes user-reported step-11 regression after broker redeploy: cap-mint returned HTTP 403 — body: {"error":"device is not active on chain", "reason":"device_not_active"} Same bug class I fixed earlier in scripts/heima-agent-create.sh + scripts/heima-device-revoke.sh (commit 0981a88). Both the broker's handlers/cap.rs::parse_device_entry AND the worker's crates/agentkeys-worker-creds/src/verify.rs::parse_device_entry were still slicing the OLD 7-word DeviceEntry layout. After codex H1 inserted 4 new fields (k11CredId, k11RpIdHash, k11PubX, k11PubY), the struct grew to 11 ABI words, but neither parser was updated. word 0 operatorOmni bytes32 word 1 actorOmni bytes32 word 2 k11CredId bytes32 word 3 k11RpIdHash bytes32 (NEW, codex H1) word 4 k11PubX uint256 (NEW) word 5 k11PubY uint256 (NEW) word 6 tier uint8 (padded) word 7 roles uint8 (padded) word 8 registeredAt uint64 (padded) word 9 lastSignCount uint32 (padded) word 10 revoked bool (padded) Before this commit, both parsers read: roles → word 4 (which is now k11PubX) registeredAt → word 5 (which is now k11PubY — always 0 for agents) revoked → word 6 (which is now tier) For agent devices (k11PubX = k11PubY = 0), registeredAt parsed as 0 → broker returned DeviceNotActive even though the device WAS active. Fix: both parsers now read from the correct 11-word offsets + check hex.len() >= 11 * 64. Tests updated: worker-creds verify::tests::parse_device_entry_decodes_well_formed → construct an 11-word raw response (was 7) broker handlers::cap::tests::parse_device_entry_decodes_well_formed → same broker handlers::cap::tests::parse_device_entry_detects_revoked → same All 4 green. Live deploy: requires broker host redeploy via setup-broker-host.sh so the broker picks up the new parse_device_entry. Worker code change ships with the broker redeploy (same setup-broker-host.sh rebuild). * stage-3 step 11+12: pass STS creds via X-Aws-* headers (fix s3_put 502) Step 11 surfaced the codex P2 downgrade-attack defense WORKING AS INTENDED: cap-mint succeeded, worker AES-encrypted, then S3 PUT returned 502 "s3_put: service error" because the worker fell back to the broker EC2 instance profile (which deliberately lacks s3:PutObject on the vault bucket). The codex P2 fix in commit 18e709b added OptionalStsCreds + the AGENTKEYS_WORKER_REQUIRE_STS strict-mode env var. Workers correctly demand per-request OIDC-minted STS creds. The stage-3 demo's step 11+12 cred_memory_roundtrip helper wasn't passing them. Fix: stage-3 step 11 (cred roundtrip) now passes vault-role STS creds, step 12 (memory roundtrip) passes memory-role STS creds, both via the three X-Aws-* headers the worker's OptionalStsCreds extractor reads: -H 'x-aws-access-key-id: $aki' -H 'x-aws-secret-access-key: $sak' -H 'x-aws-session-token: $sst' The STS creds were already minted in step 3 (vault + memory sessions written to $STATE_DIR/{aki,sak,sst}.{vault,memory}); step 11+12 just read the right file pair based on the kind (cred → vault, memory → memory) and forward them as headers. After this commit, steps 11+12 should land green end-to-end: broker cap-mint → 200 (chain checks pass) worker cap-verify → 200 (broker_sig + chain re-verify) worker S3 PUT → 200 (using per-actor STS creds, NOT instance profile) byte-for-byte roundtrip assertion holds. * stage-3 step 11+12: mint AGENT-side STS creds (correct principal-tag match) Step 11 surfaced the second layer of the OIDC isolation chain working as designed: cap-mint succeeded (broker authorized operator→agent), worker AES-encrypted, then S3 PUT returned 502 because the STS creds were minted from the OPERATOR'S session JWT (tagged with operator's actor_omni) but the cap's actor_omni — and hence the S3 key path — is the AGENT'S. IAM saw ${PrincipalTag/agentkeys_actor_omni} = 941c… trying to PUT bots/82a0…/credentials/… and rejected with AccessDenied. This is the IAM enforcing what the cap-token expresses: "operator authorized the agent to do this op; the agent must be the one actually doing it." Both layers must agree on actor_omni. Fix (stage-3 cred_memory_roundtrip helper): 1. Read agent_private_key from the demo-agent file 2. SIWE-sign as the agent against the broker (POST /v1/auth/wallet/start with the agent's address, sign with cast wallet sign using agent_private_key, POST /v1/auth/wallet/verify → session JWT for the agent) 3. Mint OIDC JWT via /v1/mint-oidc-jwt — this JWT now carries sub=agent_omni and PrincipalTag/agentkeys_actor_omni=agent_omni 4. AssumeRoleWithWebIdentity against the right data-class role (VAULT_ROLE_ARN for cred, MEMORY_ROLE_ARN for memory) — STS creds now tagged with the agent's actor_omni 5. Forward these creds via X-Aws-* headers to the worker Now the worker's S3 PUT against bots/<agent>/credentials/… uses STS creds with PrincipalTag=agent_omni → IAM allows. The architectural lesson, recorded in the commit because it'll bite again: when a cap-token authorizes actor A's action and the worker uses STS creds to touch S3, the STS creds MUST be minted using A's identity — operator's authorization (cap-token) + actor's identity (STS creds) jointly satisfy the workflow. Per arch.md §17.2 layer 3, the IAM PrincipalTag is bound to the JWT subject, NOT to whoever the JWT-issuer (operator) chose to authorize. * stage-3: tighten pass/fail per codex adversarial review (3 findings) Codex round-2 review flagged the demo as 'needs-attention' — it could report 16/16 green while silently skipping the actual encrypt/decrypt + cross-class assertions. Three findings, all addressed: [high] Worker roundtrip checks could be skipped + still claim coverage cred_memory_roundtrip used `skip ...; return 0` on five prereq-missing paths (no agent file, no scope, broker missing chain RPC, broker missing contract addresses, DeviceRoleMissing). Final summary still claimed AES-256-GCM byte-for-byte coverage as if the path had run. Fix: introduce STRICT default + `--allow-skip` opt-in. All five prereq paths now call prereq_missing(), which: - in strict mode: prints fail + records 'fail' outcome + returns non-zero - in --allow-skip mode: prints skip + records 'skip' outcome (dev iter) Final summary now prints actual per-step outcomes from STEP_OUTCOMES[], and exits non-zero if any step failed (or any step skipped in strict). [high] Negative cap-class tests (steps 14, 15) accepted ANY non-200 Previously: cred-class cap → memory worker with non-200 + non-canonical error was accepted ('non-200 = pass for negative test'). A down worker, wrong URL, 404 route, auth middleware failure, or malformed request would all silently satisfy the demo without proving check_data_class fired. Fix: require HTTP 400/401/403 AND the canonical cap_data_class_mismatch error string. Any other response = die. [medium] Cross-actor cap-mint test (step 13) accepted generic rejection Previously: any 4xx accepted, even when error text was non-canonical; 502 (broker stale) silently skipped, hiding a real config issue. Fix: require HTTP 400/401/403 with canonical OperatorMismatch. 502 with config-missing body now dies (forces redeploy), not skip. Other 502/non-canonical errors = die (negative tests can't pass on an unrelated failure). Plus: positive steps (4, 7, 11+12 happy paths) now call record_ok so the summary lists EVERY step that actually proved its assertion. The expect_access_denied helper records too. The summary table is built from actual execution, not a static claim of coverage. The structural change here is: skips and infrastructure failures both become demo failures unless the operator explicitly opts in. CI runs default-strict. Dev iteration uses --allow-skip when bringing up a partial environment. * stage-3 summary: fix `local` outside function + handle cleanup-only invocation Two small bugs in the strict-mode summary added by c55ea29: 1. Used `local` inside the `if should_run_step 16` block (not a function body), so bash printed: harness/v2-stage3-demo.sh: line 864: local: can only be used in a function AFTER the per-step outcome table tried to render. The 16 steps all ran correctly + the demo exited 0, but the summary table itself never printed. Fix: drop the `local` keyword and just use plain vars. 2. "DEMO COMPLETE" header would print even when no steps had been recorded (e.g. `--from-step 16` to test the summary block in isolation). Now distinguishes: - all green (nok>0, nskip=0, nfail=0) → DEMO COMPLETE - some skipped (--allow-skip) → DEMO PARTIAL - any failure → DEMO FAILED + exit 1 - no steps run at all → NO STEPS EXERCISED + advisory * harness: log codex round-2 fix + 13/13 stage-3 strict-mode verification * stage-3 codex round-3: close skip-bypass in steps 14+15 (cross-class) Codex round-3 review caught a regression I missed in c55ea29: [high] Strict demo still skips cross-class isolation checks without recording failure (steps 14 + 15) Previously fixed cred_memory_roundtrip's prereq paths to use prereq_missing (so strict mode fails-hard), but left steps 14 + 15 calling bare `skip` for the same prereq classes: - missing demo-agent file - 'not.*scope' (chain scope not set) - 'RPC URL not set' (broker stale) - 'SIDECAR_REGISTRY_ADDRESS_HEIMA unset' (broker missing contract addrs) Because those skips didn't append to STEP_OUTCOMES, a full run could report 'DEMO COMPLETE' with nskip=0 even when neither cross-data-class isolation gate had been exercised. That's the same false-success failure mode codex round-2 flagged, just in a different code path — exactly the kind of regression strict-mode tracking is meant to catch. Fix: extracted the entire step 14/15 body into a cross_class_rejection() helper function. All prereq paths now route through prereq_missing (matching cred_memory_roundtrip's pattern), so: - strict mode (default): unmet prereqs → die + STEP_OUTCOMES records 'fail' - --allow-skip mode: unmet prereqs → skip + STEP_OUTCOMES records 'skip' - successful negative test → STEP_OUTCOMES records 'ok' Step 14: cross_class_rejection cred-store /v1/memory/put memory cred cred-to-mem Step 15: cross_class_rejection memory-put /v1/cred/store cred memory mem-to-cred Live-verified on Heima Mainnet (2026-05-20): all 13 STEP_OUTCOMES recorded, DEMO COMPLETE, exit 0. Steps 14+15 still pass with canonical 403 cap_data_class_mismatch error confirmation (no change to the positive-path assertion logic — only the skip paths got tightened). * stage-3 codex round-4: cross-class test sends X-Aws-* headers (strict-mode correct) Codex round-4 finding (high): Cross-class negative test omits required STS headers, so strict workers reject before the data-class guard. The axum extractor order is: OptionalStsCreds → Json<Req> → handler body (verify_cap). With AGENTKEYS_WORKER_REQUIRE_STS=1 — the production deployment setting documented in aws_creds.rs — the extractor rejects header-less requests with HTTP 401 BEFORE verify_cap runs. The cross-class data-class guard inside verify_cap never fires. Today the live test passes because the broker host workers don't have AGENTKEYS_WORKER_REQUIRE_STS=1 set. So we're proving the data-class guard against dev-config workers but NOT against the prod target. That's exactly the 'demo says complete, prod silently broken' failure mode the codex review pipeline keeps catching. Fix: cross_class_rejection() now: 1. Mints agent-side STS creds for the TARGET worker's role: step 14 (memory worker target) → memory-role STS step 15 (cred worker target) → vault-role STS 2. Passes all three X-Aws-* headers in the POST to the worker. Worker request order now: a. OptionalStsCreds extractor: valid headers present → Some(creds) → OK (passes regardless of AGENTKEYS_WORKER_REQUIRE_STS=1 setting) b. verify_cap: check_op (Store) → OK check_data_class (cap.data_class != worker's class) → REJECT → HTTP 403 cap_data_class_mismatch c. S3 op never runs (verify_cap returned error) The data-class guard provably fires now, in BOTH strict and non-strict worker configurations. Codex's concern was correct. Refactored mint_agent_sts_for_role() as a shared helper so cross_class test reuses the same SIWE+OIDC+STS flow as cred_memory_roundtrip. Same auth chain, same trust boundary, same code path — no inconsistency between positive (cred_memory_roundtrip) and negative (cross_class) tests. Live-verified 2026-05-20 on Heima Mainnet: 13 STEP_OUTCOMES recorded, all ok, DEMO COMPLETE. Steps 14+15 still return canonical 403 cap_data_class_mismatch with the STS headers correctly passed through — confirming the data-class guard fires AFTER extractor authentication passes. * arch.md: document cap-token data_class binding + 4-layer isolation invariants (§17.5) Codifies the issue #90 outcomes into the canonical architecture spec (per CLAUDE.md "arch.md as source of truth" rule): §15.1 + §15.2 — credentials-service + memory-service: added the OIDC federation paragraph. X-Aws-* header passthrough is the production auth surface (codex P2 downgrade fix); strict mode forces it via AGENTKEYS_WORKER_REQUIRE_STS=1. Cross-links to §17.5. §17.5 (NEW) — Per-data-class cap-token binding: - Cap-token's data_class field + the 4 broker endpoints - 4-layer defense-in-depth table (broker cap-mint, worker chain- verify, AWS IAM PrincipalTag, per-data-class buckets) - Each layer's canonical test in harness/v2-stage3-demo.sh - Test-discipline rule: new data classes MUST add negative isolation tests across all 4 layers - Two design rationales spelled out: a) Why route-per-class beats a single endpoint with a data_class query-param (eliminates user-input attack surface) b) Why agent-side STS creds are mandatory (PrincipalTag must match the cap's actor_omni; operator-side STS won't satisfy IAM) Plus the trailing Cargo.lock entry from aws-…
…xchange (closes #77, #72, #78) (#96) * agentkeys: retire legacy mock-server endpoints + /v1/mint-aws-creds + /v1/auth/exchange (closes #77 #72 #78) Issue #77 — delete /identity/link, /identity/resolve, /audit/query, /v1/auth/exchange: - mock-server: drop routes and HTTP handler functions; keep resolve_identity_typed as internal helper for session/auth_request paths - broker: drop /v1/auth/exchange route, handlers/auth/exchange.rs, auth.rs::validate_bearer_token + ValidatedSession; keep extract_bearer_token (still used by mint-oidc handler) - broker: drop BROKER_BACKEND_URL + BROKER_BACKEND_TIMEOUT_SECONDS, Tier-2 backend reachability probe + readyz check, Tier2State::backend_reachable, BrokerConfig::backend_url/backend_request_timeout_seconds - core: drop CredentialBackend::query_audit and CredentialBackend::resolve_identity trait methods and all impls (mock_client, s3_backend, test stubs) - cli: drop Commands::Usage/Link/Recover + cmd_usage/cmd_link/cmd_recover; resolve_agent now requires raw 0x wallet (alias/email lookup retired); resolve_agent_to_wallet same - daemon: resolve_parent_if_set now requires raw 0x wallet, no HTTP call - mcp: list_credentials uses CredentialBackend::list_credentials directly instead of round-tripping query_audit - tests: remove tests targeting deleted endpoints; convert /identity/link setup steps to direct-DB inserts via new link_identity_direct helper Issue #72 — delete /v1/mint-aws-creds: - broker: drop /v1/mint-aws-creds route + handlers/mint.rs (mint_v2 + helpers) - tests: delete mint_v2_flow.rs + invariant_load_bearing.rs (exclusively exercised the deleted endpoint). Audit happens at /v1/mint-oidc-jwt; AWS submission is daemon-side via OIDC JWT → AssumeRoleWithWebIdentity. Issue #78 — folded into #77 per its own resolution comment. scripts/broker.env + scripts/setup-broker-host.sh: drop BROKER_BACKEND_URL since the broker no longer reads it. Workspace tests: 73 (core) + 41 (cli) + 38 (daemon) + 7 (mcp) + 31 (provisioner) + 48 (mock-server) + multiple (broker) all pass. * operator-workstation.env: refresh /v1/mint-aws-creds comment after #72 retirement * mock-server: retire audit_log table + 8 INSERT sites (codex #96 followup) After this PR deleted GET /audit/query, the 8 INSERT INTO audit_log writes in mock-server credential/session handlers became write-only dead code — nothing reads them now and nothing ever will. Production audit lives at broker plugin_mint_log (today) → agentkeys-worker-audit + Heima CredentialAudit contract (post-#97). Mock-server never was on that path. Removed: - credential.rs: store/read/list audit INSERTs (6 sites covering ok, DENIED, DENIED_SCOPE, NOT_FOUND outcomes) - session.rs: scope_update/scope_read audit INSERTs on cross-agent probes (2 sites) - db.rs: CREATE TABLE audit_log schema Tests still green: 48 mock-server, 176 broker, 41 cli, full workspace (30 test-result groups, 0 failed). Resolves codex adversarial-review finding [high] from PR #96 review. --------- Co-authored-by: wildmeta-agent <agent@wildmeta.ai>
…ed) (#95) * issue #82: ERC-7730 clear-signing + EIP-712 typed-data sign (v2-aligned) Refresh of issue #82 against v2 architecture (#87/#92). Original issue targeted v1 (mock-server-as-signer, daemon-side metadata, broker SQLite audit); plan was rewritten to the v2 surfaces (signer typed RPC, worker audit rows with intent commitments, ERC-7730 catalog as a §22 pluggable surface). Plan: docs/spec/plans/issue-82-erc7730-v2-aligned.md. ## What ships in this PR ### Phase 1 — EIP-712 typed-data signing at the signer * New endpoint `POST /dev/sign-typed-data` on the mock-server signer: accepts canonical EIP-712 v4 JSON (matches MetaMask `eth_signTypedData_v4`), parses + hashes internally (never trusts a caller-supplied prehash), returns the 65-byte canonical signature + every intermediate digest (`primary_type_hash`, `domain_separator`, final `digest`). * `DevKeyService::sign_eip712` + `Eip712SignResult` envelope. * New `SignerError::InvalidTypedData` (400) + propagation through `SignerClientError`. * `SignerClient::sign_eip712` trait method + `HttpSignerClient` impl. * Wire signer-only + full routers in agentkeys-mock-server. ### Phase 2 — clear_signing module in agentkeys-core New crate module at `crates/agentkeys-core/src/clear_signing/`: * `eip712.rs` — EIP-712 v4 encoder (no external dep). Supports string/bytes/bool/address, uint{8..256}, int{8..256}, bytes{1..32}, static/dynamic arrays, nested struct types. Cycle detection on type graph. Spec reference vector (`Mail` example) matches exactly. * `parser.rs` — ERC-7730 v2 JSON parser (subset for v0). * `format.rs` — per-field formatters (tokenAmount with decimals+ticker, address with truncation, integer, date as ISO-8601 UTC, bool, raw) + `{name}` intent interpolator. * `binding.rs` — domain-{name,version,chainId,verifyingContract} → 7730-file lookup; case-insensitive on address; refuses wildcard matches. * `catalog.rs` — bundled set (USDC permit fixture) + filesystem dir loading via `extend_from_dir` (operators ship custom files via `$AGENTKEYS_7730_DIR`). * `mod.rs::build_preview` — top-level "render this typed-data against this catalog" returning `intent_text` + `intent_commitment` = `keccak256(intent_text || 0x7c || digest)`. ### Phase 3 — CLI preview surfaces Two new subcommands under `agentkeys signer`: * `sign-typed-data` — call `/dev/sign-typed-data`. With `--preview-7730`, renders + prints operator intent + per-field review before signing. * `preview-7730` — render WITHOUT signing. Dry-run for new 7730 files before plumbing them into automated agent signing. Both pick up `$AGENTKEYS_7730_DIR` for operator-custom 7730 files; both support `--json` for machine-readable output. ### Phase 4 — audit-row intent-commitment schema (arch.md only) `arch.md §15.3` extended with two optional audit-row fields (`signed_intent_text`, `signed_intent_hash`). Schema is backwards- compatible — pre-#82 rows have the fields absent; worker reads/writes land in a follow-up PR (broker cap-mint propagation + on-chain `CredentialAudit` event extension also follow-up). ### Docs * `docs/spec/signer-protocol.md` — full `/dev/sign-typed-data` wire contract documented (request, response, supported type-string subset, errors). * `docs/spec/architecture.md` §14.2 + §15.3 + §22 — typed-data RPC in the signer surface, audit-row intent-commitment fields, clear-signing metadata as a pluggable surface (bundled → registry → on-chain progression). * `docs/spec/plans/issue-82-erc7730-v2-aligned.md` — full refreshed plan, including the K11-binding-on-high-value-signs follow-up (Phase 5 — out of scope here, tracked as separate issue since it needs a ScopeContract extension). ## Test plan * `cargo test --workspace` — 600+ tests across the workspace, all pass. * New tests added in this PR: - 30 unit tests under `agentkeys-core::clear_signing` (EIP-712 spec reference vector, cyclic type detection, integer range checks, array length validation, U256 dec/hex roundtrip, two's-complement negation, parser, formatter, binding, catalog). - 2 sign_eip712 unit tests in `dev_key_service.rs` (recovers-to-derived-address, malformed-typed-data rejection). - 6 route tests in `dev_key_service_routes.rs` (200 / 400-unknown- primary / 400-out-of-range-uint / 503-signer-disabled / address- matches-derive / full-sig-recovery-roundtrip). * `cargo clippy` — clean on all new code; pre-existing warnings unchanged. * Signature roundtrip verified: HKDF-derived secp256k1 key signs the EIP-712 digest, `ecrecover` returns the same address that `derive_address` produces for the same `omni_account`. ## What did NOT land in this PR Tracked as follow-ups so this PR stays scoped: * **Broker cap-mint policy gate** — the broker cap-mint endpoint doesn't yet require an `intent_commitment` for typed-data signs. Today the daemon goes direct to the signer via `signer_client`. When broker mediation lands, the cap-token carries the commitment. * **Worker audit-row wiring** — `agentkeys-worker-audit` doesn't read the new schema fields yet (forward-compatible; unknown fields are silently ignored). Schema is documented in arch.md §15.3 so the follow-up PR has a fixed target. * **On-chain `CredentialAudit` event extension** — needs a contract revision + redeploy; out of scope for a signer + worker change. * **Registry fetch (v1 source)** — `github.com/ethereum/clear-signing- erc7730-registry` integration is the v1 catalog source per arch.md §22 (the bundled set is the v0 default that ships in this PR). * **EIP-4337 UserOp clear signing** — out of scope per original #82. * **K11 binding on high-value signs** — Phase 5 in the plan; needs a ScopeContract extension to express "agent A may sign EIP-712 binding to chainId=1 verifyingContract=$X with tokenAmount ≤ Y". Plan-completion summary: * **What landed**: Plan refresh, signer-protocol.md update, arch.md §14.2/§15.3/§22 updates, `/dev/sign-typed-data` endpoint, signer-side EIP-712 hashing (no external dep), `clear_signing` module (parser + formatter + binding + catalog + EIP-712), bundled USDC permit fixture, CLI `sign-typed-data` + `preview-7730` subcommands, audit-row intent- commitment schema doc, full sig-recovery roundtrip test. * **What did NOT land**: Broker cap-mint policy gate, worker audit-row wiring, on-chain `CredentialAudit` event extension, registry-fetch catalog source, K11-on-high-value-signs (Phase 5). All tracked explicitly in the plan doc as follow-ups. * issue #97: arch.md §15.3a — AuditEnvelope v1 canonical schema Defines the unified abstract audit message format that every audit-producing surface (creds, memory, signer, broker, payment-service, email-service, SidecarRegistry, K3EpochCounter) MUST emit going forward, and that the chain + explorer + indexer consume. ## What this section adds * **Envelope schema** — version, ts_unix, actor_omni, operator_omni, op_kind (u8), op_body (CBOR), result, intent_text + intent_commitment (PR #95). Canonical CBOR per RFC 8949 §4.2.1. * **Wire shape** — `POST /v1/audit/append` accepts the envelope; `GET /v1/audit/envelope/<hash>` returns the full envelope on demand (used by explorers). * **On-chain shape** — `CredentialAudit.appendV2(operatorOmni, actorOmni, opKind, envelopeHash)` + `appendRootV2(... opKindBitmap)` lands additively alongside the v1 `append`/`appendRoot`. New events `AuditAppendedV2` + `AuditRootAppendedV2` with `indexed opKind` topic so explorers can filter via `eth_getLogs`. * **Canonical op_kind table** — 17 op_kinds across 8 families (creds=0..2, memory=10..12, signs=20..21, payments=30..31, scope=40..41, device=50..52, email=60..61, K3=70). Grouped by 10s leaves room for related ops. PRs adding new op_kinds MUST append a row; numbers never reused, never reordered. * **Eight non-break invariants** — the cost of adding a new op_kind is "uglier UI temporarily for old explorers" — never "broken explorer / dropped event." Open enum, stable envelope-level fields, version gating, fallback renderer, opaque body pass-through, op-kind-agnostic contract, canonical table, 3-test contract per new op_kind. * **5-phase migration** — A (this doc) → B (worker + core migration) → C (contract revision) → D (subscan-essentials decoder) → E (subscan-essentials-ui-react renderer) → F (extend op_kind coverage). Phases B / C / F tracked at agentkeys#97; phases D / E tracked at subscan-essentials#12. ## Why this matters Today's audit surface only has 3 op_kinds (STORE / READ / TEARDOWN) and those are credential-CRUD-only. A typed-data sign event, a scope mutation, a device add, a payment, a memory put, an email send, a K3 epoch advance — none of these have a row to render in the explorer. With this section in place, the explorer can render a uniform timeline across all of them, and adding a new op_kind doesn't require the explorer to ship a release before AgentKeys can ship the feature. ## What does NOT land in this PR This is the schema lock-in (Phase A). The implementation phases (worker migration, contract redeploy, explorer decoder, UI renderer) ship as follow-ups in their respective repos. agentkeys#97 + subscan-essentials#12 are the tracking issues. * issue #97 phase B: AuditEnvelope v1 struct + worker V2 endpoints Lands the canonical AuditEnvelope shape as live code, not just a doc. Documented in arch.md §15.3a; this commit ships the worker side. Contract revision (Phase C) + emit-site migration across signer/scope/device/payment/ memory/email/K3 (Phase F) remain follow-ups in #97. ## What ships ### `agentkeys-core::audit` — canonical envelope (new module) * `AuditEnvelope` struct — version + ts_unix + actor_omni + operator_omni + op_kind (u8 open enum) + op_body (ciborium::Value) + result + intent_text + intent_commitment. Envelope-level fields are stable across all op_kinds. * `AuditOpKind` repr-u8 enum — 18 variants matching arch.md §15.3a canonical table (creds=0..2, memory=10..12, signs=20..21, payments=30..31, scope=40..41, device=50..52, email=60..61, K3=70). Open enum: `from_u8` returns Option, never panics. * `AuditResult` repr-u8 enum (Success=0, Failure=1, NotPermitted=2). * Per-op_kind typed body schemas in `audit::bodies` — 18 structs with serde derives matching the canonical table field-for-field. * Canonical CBOR codec in `audit::cbor` — deterministic per RFC 8949 §4.2.1. Encoder builds the envelope as an ordered CBOR map with keys sorted by canonical CBOR ordering. Decoder ignores unknown envelope-level keys (forward-compat) and rejects unsupported envelope versions. * `envelope_hash()` = keccak256(canonical_cbor). The 32-byte commitment that lands on chain as the second arg to the future `CredentialAudit.appendV2(operatorOmni, actorOmni, opKind, hash)`. * `commit_intent()` helper — same scheme as `clear_signing::commit_intent` (PR #95); verified by a test that asserts byte-for-byte equality between the two. ### `agentkeys-worker-audit` — V2 endpoints * `POST /v1/audit/append/v2` — accept envelope (as JSON), convert op_body to CBOR, compute envelope_hash, store CBOR by hash. Returns `{envelope_hash}`. * `GET /v1/audit/envelope/:hash` — return canonical CBOR bytes for the envelope (200 application/cbor) or 404 envelope_not_found. Explorers fetch via this endpoint after seeing the on-chain hash. * V1 endpoints (`/v1/audit/append`, `/v1/audit/flush/:op`, etc.) retained so existing callers keep working through the migration cycle. * `state.rs` extended with `envelopes: Mutex<HashMap<String, Vec<u8>>>` — in-memory v0; persistent S3 storage is a separate concern tracked alongside Phase C. ### Non-break invariants enforced by code Per arch.md §15.3a: 1. ✅ `op_kind` is `u8`, never a sealed enum (open enum design; `AuditOpKind::from_u8` returns Option). 2. ✅ Envelope-level fields decode for ANY op_kind, even op_kind=250 (test: `unknown_op_kind_still_decodes_envelope_level_fields`). 3. ✅ `version` bumped only on envelope-level breakage; new op_kinds stay at v1. 4. ✅ Worker accepts unknown op_kinds + stores the opaque body for explorers to fetch (test: `append_v2_accepts_unknown_op_kind`). 5. ✅ Decoder ignores unknown envelope-level keys (forward-compat for future versions; test: `decoder_ignores_unknown_envelope_keys`). 6. ✅ No contract-side decode of op_body — only `(opKind, envelopeHash)` would land on chain (Phase C scope; out of this PR). 7. ✅ Canonical op_kind table in arch.md §15.3a — `op_kind.rs::tests` asserts no byte collisions + all variants roundtrip. ## Tests * 17 unit tests in `agentkeys-core::audit` — envelope encode/decode, envelope hash determinism, unknown-op_kind tolerance, version refusal, typed body decode, op_kind byte uniqueness, commit_intent parity with `clear_signing::commit_intent`. * 7 integration tests in `agentkeys-worker-audit::tests::envelope_v2`: - append → 200 + envelope_hash with correct shape - GET → 200 application/cbor with canonical bytes - GET unknown hash → 404 envelope_not_found - reject envelope version 99 - reject malformed actor_omni - accept unknown op_kind (non-break invariant #1 + #4) - envelope_hash deterministic across appends - ts_unix=0 gets server-assigned * `cargo test --workspace` — 600+ tests, **0 failures, 1 ignored** (network-dependent test; pre-existing). * `cargo clippy` — clean on all new code. ## What does NOT land in this PR Tracked in #97 as Phases C + F: * On-chain `CredentialAudit.appendV2` + `appendRootV2` + new events with indexed opKind topic — needs contract revision + Heima Mainnet redeploy. * Migration of credentials-service + memory-service + signer + broker emit sites from legacy `AuditEvent` to `AuditEnvelope`. Each new op_kind PR will append a row to the arch.md §15.3a table + add the worker emit-site call. * Persistent storage for envelopes (S3 `audit/envelopes/<hash>.cbor`). In-memory v0 is sufficient for the worker's lifecycle; if the worker restarts before chain commitment lands, callers re-emit. * Subscan-essentials indexer decoder + UI renderer (subscan-essentials#12). * issue #97 phase B: AuditClient — convenience HTTP client for the V2 endpoints Future emit sites (credentials-service, memory-service, signer, broker, payment-service, email-service, SidecarRegistry, K3EpochCounter) all need the same `POST /v1/audit/append/v2` + `GET /v1/audit/envelope/<hash>` wire shape. Putting the client in agentkeys-core means each emitter consumes the contract from one place — and the wire-level test surface is centralized. ## What ships * `agentkeys_core::audit::AuditClient`: - `new(base_url)` / `from_env()` (reads `$AGENTKEYS_AUDIT_WORKER_URL`, defaults to `https://audit.litentry.org`). - `append(envelope)` → returns `{ok, envelope_hash}` from the worker. - `get_envelope(hash)` → `Option<Vec<u8>>` (None on 404). * `envelope_for(actor, operator, op_kind, op_body, result, intent_text, intent_commitment)` convenience builder — constructs an envelope from a typed body (any `serde::Serialize`), wires the canonical CBOR. ## Emit-and-forget semantics Per arch.md §15.3a, chain commitment is the durability mechanism — the worker's in-memory envelope map is best-effort cache. Emitters that need guaranteed delivery either retry on transient failure or fall back to direct on-chain `CredentialAudit.append`. ## Tests Two unit tests added in `audit::client::tests`: * `envelope_for_builds_typed_body` — round-trip through the typed body decoder: `SignEip712Body` → envelope → `typed_body()` returns the same body. * `envelope_for_emits_canonical_cbor` — same inputs produce same `envelope_hash` regardless of build path (cross-encoder stability). Total audit-module tests now 19. Full workspace `cargo test --workspace` clean (600+ tests, 0 failures). * issue #97 phase C: CredentialAudit.appendV2 + appendRootV2 (contract code only) Adds the V2 surface to the CredentialAudit contract per arch.md §15.3a. V1 (`append` + `appendRoot`) is retained unchanged so existing indexers + the live tier-A worker keep working through the migration cycle. ## What ships * `appendV2(operatorOmni, actorOmni, opKind, envelopeHash)` — emits `AuditAppendedV2(operatorOmni indexed, actorOmni indexed, opKind indexed, envelopeHash)`. **Event-only — no on-chain storage.** The full envelope lives off-chain at the audit-service worker, addressed by `envelopeHash = keccak256(canonical_cbor(AuditEnvelope))`. The `opKind` indexed topic lets explorers filter `eth_getLogs` by op_kind without scanning every row. * `appendRootV2(operatorOmni, merkleRoot, opKindBitmap, batchEntryCount)` — emits `AuditRootAppendedV2`. `opKindBitmap` is `bytes32` where bit N = op_kind N is present in the batch. Lets explorers filter batches by op_kind without fetching every leaf from the worker. Gated to the operator's master wallet (same as V1 `appendRoot`, codex M1). * No on-chain decode of `op_body` — the contract stays op-kind-agnostic (non-break invariant #6 per arch.md §15.3a). New op_kinds need ZERO contract redeploys. ## Forge tests 5 new tests in `AgentKeysV1.t.sol` (alongside 4 existing CredentialAudit tests): * `test_CredentialAudit_AppendV2_EmitsEvent` — confirms the event topics carry operator + actor + opKind for `eth_getLogs` filtering. * `test_CredentialAudit_AppendV2_AcceptsAnyOpKind` — invariant #1 + invariant #6: op_kind=250 (reserved future byte) accepted without revert. * `test_CredentialAudit_AppendV2_OpenToAnyCaller` — `appendV2` is open to any caller (chain ordering + gas is the safety; indexer filters out attacker-emitted noise via canonical envelope hashes). * `test_CredentialAudit_AppendRootV2_EmitsEvent` — Merkle-batch path with multi-op_kind bitmap (bits 0 + 21 + 40 = CredStore + SignEip712 + ScopeGrant set). * `test_CredentialAudit_AppendRootV2_RejectsNonMaster` — gated to operator's master wallet per codex M1. * `test_CredentialAudit_V1_And_V2_Coexist` — V1 `append` + V2 `appendV2` write to disjoint paths; V2 emits don't touch V1's `entries` storage. Forge: 9/9 CredentialAudit tests pass; full forge suite 39/39 tests pass. Workspace cargo test still clean. ## Redeploy: operator action This commit ships the contract code + tests. The actual Heima Mainnet redeploy via `scripts/heima-bring-up.sh --upgrade` is operator action gated on PR review — left for a follow-up operator step. Until redeployed, the live `CredentialAudit` on Heima still has only V1 methods, so callers of `agentkeys-worker-audit::handlers::append_v2` can store envelopes off-chain but can't commit `envelopeHash` to chain until redeploy lands. Migration sequence per arch.md §15.3a Phase C: 1. Operator reviews this PR. 2. Operator runs `bash scripts/heima-bring-up.sh --upgrade` (idempotent — redeploys CredentialAudit if address bytecode hash changed). 3. Operator captures new address into `scripts/operator-workstation.env` + `docs/spec/deployed-contracts.md`. 4. Run `AGENTKEYS_CHAIN=heima bash scripts/verify-heima-contracts.sh`. 5. Run harness/v2-stage1-demo.sh through 3 to confirm no regression (V1 path still works on the redeployed contract). * issue #97: recursive op_body canonicalization + arch.md event sig fix Address two architect-review findings against earlier commits in this PR (reviewer: oh-my-claudecode:architect on PR #95). ## Fix 1 — recursive op_body canonicalization (cross-language hash determinism) Architect finding (section 4): the canonical CBOR encoder sorted only envelope-level keys, not `op_body` map keys recursively. The Rust ecosystem happened to produce stable hashes because `serde_json::Value:: Object` is `BTreeMap`-backed, but a Go or TypeScript encoder building `op_body` with unsorted keys would have produced different CBOR bytes and a different `envelope_hash` — silently breaking the chain-commitment property for cross-language clients. `audit::cbor::canonicalize()` now walks `op_body` recursively: every nested map's keys are sorted by their canonical CBOR-encoded bytes (RFC 8949 §4.2.3). Arrays preserve order (semantic ordering). Two new tests prove the property: * `op_body_key_order_does_not_affect_hash` — flat map, alphabetical vs reverse-alphabetical insertion order → identical envelope_hash. * `op_body_nested_map_key_order_does_not_affect_hash` — nested map recursion check. Total audit-module tests now 21. Workspace cargo test clean. ## Fix 2 — arch.md event signatures match the actual contract Architect finding (section 3): arch.md §15.3a `AuditAppendedV2` / `AuditRootAppendedV2` declarations included `entryIndex` / `rootIndex` fields that the actual `CredentialAudit.sol` events do NOT emit. Explorer implementers reading arch.md would have expected fields that aren't there. Doc updated to match the live contract surface. Added a sentence explaining V2's event-only design: position within the operator's stream is derivable from `(block_number, log_index)` so the contract doesn't need to carry `entryIndex` explicitly. ## What this PR ships (cumulative across all commits) Phase A — arch.md §15.3a (canonical schema + table + non-break invariants + migration phases) ✅ Phase B — agentkeys-core::audit module + worker V2 endpoints + AuditClient ✅ Phase C — CredentialAudit.appendV2 + appendRootV2 (code + 5 forge tests; redeploy is operator action) ✅ Phase D / E (subscan-essentials decoder + UI) tracked at subscan-essentials#12. Phase F (extend emit coverage to sign/scope/device/payment/email/K3) tracked at agentkeys#97. * docs+ops: add-op-kind ritual + setup-heima orchestrator + idempotency rule Three related changes addressing user request after the #97 op-kind work: ## 1. How-to-add-a-new-op-kind documentation ### arch.md §15.3b — the 5-step ritual Brief operator-facing ritual: (1) pick the byte from the appropriate family range, (2) append a row to §15.3a canonical table, (3) add the Rust variant in `audit::{op_kind,bodies,mod}`, (4) wire the emit site via `envelope_for` + `AuditClient::append`, (5) ship 3 tests (CBOR roundtrip + explorer Unknown(byte) fallback + arch.md row uniqueness). Critical invariant called out: never bump ENVELOPE_VERSION for a new op_kind. The version is reserved for envelope-level breakage; open-enum op_kinds are the whole point. ### wiki/audit-envelope-add-op-kind.md — detailed worked example Walks through adding `PaymentRefund` (byte 32) end-to-end: - Step-by-step code for op_kind.rs / bodies.rs / mod.rs. - Sample emit-site wiring in a worker handler. - Complete PR checklist + the explicit "what you DON'T need to do" list (no contract redeploy, no version bump, no migration, no synchronous rollout). Lives under `./wiki/` per CLAUDE.md "Wiki-location policy" — auto- publishes to the GitHub wiki on every push to main. ## 2. scripts/setup-heima.sh — single idempotent entry point Mirrors the `scripts/setup-broker-host.sh` pattern: one operator-facing orchestrator that runs the entire Heima chain bring-up + binding flow end-to-end in 15 idempotent steps. Delegates to the existing per-action helpers (`heima-bring-up.sh`, `heima-device-register.sh`, `heima-agent-create.sh`, `heima-scope-set.sh`, `heima-credential-audit.sh`, `heima-worker-smoke.sh`, `verify-heima-contracts.sh`) so: - Each helper's existing idempotency check (`cast call <view-fn>`, `cast code <addr>`, `cast balance ≥ amount`, file-exists guards) is preserved. - Per-action helpers stay callable directly for surgical re-runs (e.g. `bash scripts/heima-scope-set.sh ...` for just the scope work). - The orchestrator is THE entry point operators run — same posture as setup-broker-host.sh. Flag surface mirrors the harness orchestrators: `--chain`, `--session-id`, `--agent-label`, `--service`, `--webauthn`, `--yes`, `--from-step N`, `--to-step N`, `--only-step N`, `--help`. Two append-only steps (13 audit append + 14 tier-A relay) are explicitly called out in the header per the CLAUDE.md rule: "If a remote-setup script you're writing CAN'T be made idempotent (...append-only audit event), explicitly call it out." `bash -n` clean; `--help` renders correctly. ## 3. CLAUDE.md — idempotent remote-setup rule New section "Idempotent remote-setup rule (CLOUD / BLOCKCHAIN / CI / VM)" makes the existing implicit pattern an explicit project policy: - Every remote-mutation script (AWS / Heima / CI / VM / Cloudflare / Tencent / IAM / DNS) MUST be idempotent. Re-runs MUST exit 0 without re-applying. - Three reasons: operators retry, CI re-runs, the harness re-runs as a regression gate. - Concrete pre-check / short-circuit table for 9 mutation types (contract deploy, chain tx, fund EVM account, AWS resource, systemd unit, env file, nginx vhost, DNS A record, key gen). - Output convention: `ok proceeding` / `skip <reason>` / `fail <reason>` so the harness can read state per step. - Exception clause: if truly non-idempotent (one-shot CAS-burn cap, append-only audit event), explicitly call it out in script header AND runbook. Also adds "Heima chain (single entry point)" section pointing at the new `setup-heima.sh`. * wiki(add-op-kind): detail the explorer-side update (indexer + UI) The previous version of this guide stopped at the agentKeys-side ritual and left explorer work as a one-line bullet ('explorer-side PR'). Per follow-up request — flesh out what 'update the explorer' actually means across the two separate repos (subscan-essentials + subscan-essentials- ui-react) so an operator working through the guide doesn't have to reverse-engineer the seam. ## New section structure The page now has three parallel tracks: 1. **agentKeys-side PR** — the original 5-step ritual (unchanged). 2. **Indexer-side PR** ([litentry/subscan-essentials](https://github.com/litentry/subscan-essentials)): Go decoder registration, typed XxxDecoder impl, REST shape, three tests (canonical-fixture decode + unknown-byte non-break + cross-language hash match). 3. **UI-side PR** ([litentry/subscan-essentials-ui-react](https://github.com/litentry/subscan-essentials-ui-react)): React renderer component, registration in OP_KIND_RENDERERS map, Storybook story + fallback story. ## What the new explorer section adds - **§A1-A4**: Concrete Go code samples for the new PaymentRefund (byte 32) example — decoder table entry, typed body struct with CBOR tags, REST shape function, generic event-handler dispatch that stays op-kind-agnostic, and the three required tests. - **§B1-B3**: React renderer component with Field/Card layout, registry entry, Storybook expectation. - **§C**: Shared cross-language test vectors as the load-bearing cross-encoder determinism guard. Tracked as a follow-up alongside the next new op_kind. - **Phasing table**: Visual confirmation of the non-break trade-off at each column (operator emit-site → chain event → worker → indexer → UI), showing that at every step the system is functional and the only visible degradation between phases is 'uglier UI temporarily for old explorers.' ## PR checklist split The checklist is now three sub-checklists — one per repo — so a PR author can see exactly what lands in each of the three independent PRs. The agentKeys-side PR is fully self-contained; the other two land on their own cadence per the non-break design. * K11 WebAuthn: render operator-readable intent on the confirmation page ## The gap (what the user asked) Before this commit, the K11 WebAuthn ceremony's localhost confirmation page showed the operator ONLY: Operator 0xb3224706… RP ID localhost Challenge 0xdead…beef ← 32 bytes — what's actually signed The operator had no way to tell WHAT they were authorizing — just the opaque 32-byte challenge hex. WebAuthn's OS-level Touch ID prompt is fixed by the platform; it can't show application text either. So the operator was blind-signing — exact same failure mode arch.md §15.3a called out for typed-data signs, but at the K11 binding site. ## What this commit changes `crates/agentkeys-cli/src/k11_webauthn.rs`: * **New public type** — `K11IntentContext { text: Option<String>, fields: Vec<(String, String)> }`. Display-only operator-readable intent description + per-field rows. * **New public entry points**: - `assert_webauthn_with_intent(operator_omni, message, rp_id, intent)` — assert with operator intent rendered. - `assert_webauthn_for_chain_with_intent(operator_omni, expected_challenge, rp_id, intent)` — chain-ready variant. * **Legacy entry points unchanged**: `assert_webauthn`, `assert_webauthn_with_rp`, `assert_webauthn_for_chain` still work — they pass `K11IntentContext::empty()` internally, so existing call sites + existing tests are bit-identical to before. * **Confirmation page HTML** now renders a bordered intent block above the raw challenge dump when intent is supplied: YOU ARE ABOUT TO AUTHORIZE: Grant agent demo-agent access to openrouter Agent omni 0xb3224706…cc999E02 Service openrouter Max calls / hour 100 K3 epoch 1 Expires 2026-06-20T22:13:20Z Review the above BEFORE pressing Sign. The Touch ID prompt itself cannot show this text — your eyes are the last line of defense between the daemon's claim and the signature. * **New `html_escape` helper** + 3 tests proving malicious daemon-supplied intent strings cannot inject `<script>` into the page. The daemon controls the intent payload but the page's safety properties (operator sees real intent, localhost-only origin, OS prompt fires) hold regardless. * **Challenge label updated** to `Challenge (raw)` + meta-text `"32-byte commitment — what WebAuthn actually signs"` so the operator understands the relationship between the intent text + the challenge bytes. ## Cryptographic binding (unchanged) The intent parameter is DISPLAY-ONLY. The signed payload is still: challenge_bytes = sha256(message) # or pre-computed for chain submission clientDataJSON = {"type":"webauthn.get","challenge":b64url(challenge_bytes),"origin":"..."} authData = rpIdHash || flags || signCount signature = ECDSA-P256(sha256(authData || sha256(clientDataJSON))) Adding the intent does NOT change any existing signature consumer (broker / on-chain K11Verifier / audit-row verifier). ## Audit binding — intent_commitment The same intent string fed to the WebAuthn page SHOULD populate `AuditEnvelope.intent_text` + `AuditEnvelope.intent_commitment`. The audit commitment is `keccak256(intent_text || 0x7c || op_payload_digest)` — so auditors later can verify the operator saw text T AND the audit row commits to T. Closes the "what did the operator actually see?" forensics gap end-to-end (page-render → operator-eyes → audit-row → chain-commitment). ## Documentation * `wiki/k11-webauthn-intent-rendering.md` (NEW, 200+ lines): - The OS-level constraint (why custom Touch ID prompts are impossible). - Where AgentKeys closes the gap (localhost confirmation page). - The intent block design (header / headline / fields / caveat). - Public API + worked example for scope-grant. - Cryptographic-binding-unchanged guarantee. - Audit-binding mapping to AuditEnvelope.intent_text + intent_commitment. - When-to-provide-an-intent table per call site. - Tests reference. * `wiki/audit-envelope-add-op-kind.md`: cross-link added — every new master-mutation op_kind PR also wires `assert_webauthn_*_with_intent`. * `docs/spec/architecture.md` §10.1: cross-link added pointing at the new wiki page; explains the page is where intent rendering happens and binds to the audit row. ## Tests `cargo test -p agentkeys-cli --lib k11_webauthn`: 9 tests pass (5 new): * html_escape_neutralizes_script_injection — load-bearing safety check. * html_escape_handles_quote_chars. * html_escape_passes_safe_text_through. * k11_intent_context_empty_is_default. * k11_intent_context_with_text_is_not_empty. Full workspace `cargo test --workspace` clean. End-to-end visual verification (manual): open the confirmation page during `harness/v2-stage1-demo.sh --webauthn` — intent block renders above the challenge hex. * heima-device-add: idempotency check — skip if companion already on-chain ## Symptom (the user's report) \`bash harness/v2-stage2-demo.sh --webauthn\` step 6 failed with: fail cast send failed: Error: Failed to estimate gas: server returned an error response: error code -32603: VM Exception while processing transaction: revert, data: \"0xa98bbce05f0fa99105175d11f8a6f7e5f60…\" ## Diagnosis Selector \`0xa98bbce0\` decodes to \`SidecarRegistry.DeviceAlreadyRegistered(bytes32)\`. The 32-byte arg \`0x5f0fa991…\` is the companion's device_key_hash — the device was ALREADY registered on chain (from a prior \`--webauthn\` run that ran through). The script blindly re-submitted the registerAdditionalMaster tx instead of pre-checking + skipping. Idempotency hole. ## Fix \`harness/scripts/heima-device-add.sh\` Step 1 now pre-reads \`SidecarRegistry.getDevice(deviceKeyHash)\` and short-circuits when \`registeredAt > 0\` (the canonical pre-check shape from CLAUDE.md \"Idempotent remote-setup rule\" — \"Chain tx → cast call <view-fn> returning canonical state → skip already-registered\"). Three paths: * \`registeredAt = 0\` (not on chain yet) → log \"proceeding\" + continue the existing flow (K11 ceremony + cast send). * \`registeredAt > 0\` + \`revoked = false\` → log \`skip already-registered\` with JSON output \`{\"ok\":true,\"skipped\":\"already-registered\", \"device_key_hash\":\"…\",\"registered_at\":<ts>}\` and exit 0 — no K11 ceremony, no tx, the harness step records green. * \`registeredAt > 0\` + \`revoked = true\` → die with clear operator message: \"re-registering a revoked device requires a new device hash; generate a fresh companion device + re-enroll.\" (the contract would revert anyway; failing loud + clear here saves the operator one round-trip + one Touch ID tap.) Sibling scripts (\`heima-register-first-master.sh\`, \`heima-register-spare-master.sh\`, \`heima-agent-create.sh\`, \`heima-device-register.sh\`) already had this check — verified via \`grep -c\`. \`heima-device-add.sh\` was the only outlier. ## Why this is the CLAUDE.md \"runbook-fix-fold-back\" pattern This is the second iteration of CLAUDE.md \"Idempotent remote-setup rule\" enforcement. The rule listed \"Chain tx (register / scope / audit append) → cast call <view-fn> returning canonical state\" as the canonical pre-check shape. Every script that mutates chain state needs that check; the one without it broke the harness on re-run. The fix lives where the bug is (the device-add helper); no runbook revision needed because \`v2-stage2-demo.sh\` already calls the helper by name + would now skip cleanly on re-runs. ## Test \`bash -n harness/scripts/heima-device-add.sh\` clean. Live: operator re-runs \`bash harness/v2-stage2-demo.sh --webauthn\` — step 6 should now log \`skip device 0x…5f0fa991… already registered\` and advance to step 7 instead of reverting. * codex review fixes (PR #95): 3 P1 + 3 P2 findings addressed Independent diff review via \`codex review --base main\`. Six findings, all real; all six fixed in this commit with regression tests for the testable ones (5 tests added). Workspace cargo test clean (47 suites, 0 failures). ## P1 (blocking) findings ### P1-1: Canonical CBOR top-level map order was lexicographic-by-text, not RFC 8949 §4.2.3 \`crates/agentkeys-core/src/audit/cbor.rs\` — the encoder hard-coded the top-level map in alphabetical-by-text order, but canonical CBOR sorts by the encoded BYTES (length-prefix first, then bytes). For our 9 envelope- level keys this means shorter keys like \`result\` (6 chars) MUST sort before longer keys like \`actor_omni\` (10 chars). The bug would have silently desynchronized \`envelope_hash\` between the Rust encoder and any RFC-8949-correct Go or TypeScript encoder — exactly the cross-language determinism property the doc + the tests claim. The existing recursive \`canonicalize()\` helper already had the correct sort logic for \`op_body\` inner maps; the top-level map was simply bypassing it. **Fix:** route the top-level map through the same \`canonicalize()\` helper. Single source of truth for byte ordering — top-level + nested can never drift again. **Regression test:** \`top_level_map_keys_emitted_in_canonical_cbor_order\` decodes the output bytes and asserts the key order is the exact canonical sequence: \`result, op_body, op_kind, ts_unix, version, actor_omni, intent_text, operator_omni, intent_commitment\`. ### P1-2 + P1-3: setup-heima.sh called non-existent flags on helper scripts \`scripts/setup-heima.sh\` step 4 called \`heima-bring-up.sh --only-step gen-key\` and step 5 called \`heima-fund-account.sh --target deployer\`. Neither flag exists. \`heima-bring-up.sh\` has no \`--only-step\` parser so extra args were silently ignored and the FULL bring-up ran from step 1 (funding + deploying contracts when the operator only wanted key generation). \`heima-fund-account.sh\` rejects unknown flags so step 5 would hard-fail with \"--to is required\". **Fix:** delegate the entire \"make-chain-ready\" flow (key gen → fund → deploy → persist addresses) to a SINGLE call to \`heima-bring-up.sh\` in step 4 — that script is the canonical idempotent owner of the flow and pre-checks every mutation itself. Step 5 now derives the deployer address from the persisted key (\`cast wallet address\`) and calls \`heima-fund-account.sh --to <addr>\` with the flag the helper actually accepts. Steps 6 + 7 become explicit no-ops with comments pointing at step 4. \`bash -n scripts/setup-heima.sh\` clean. ## P2 (quality) findings ### P2-4: U256::shl returned ZERO at 64-bit boundaries \`crates/agentkeys-core/src/clear_signing/eip712.rs\` — \`U256::ONE.shl(64)\` produced \`0\` because the prior off-by-one impl copied \`self.limbs[3 - src]\` where \`src = i + limb_shift\`. When \`bit_shift == 0\` (i.e. \`bits\` is a multiple of 64), \`hi\` reduced to a plain limb copy from the wrong slot — for \`Self::ONE.shl(64)\` this copied \`self.limbs[2]\` (zero) into \`out[3]\` instead of \`self.limbs[3]\` (the value 1) into \`out[2]\`. Practical effect: every \`uint64: N\`, \`uint128: N\`, \`uint192: N\` (and the matching int sizes) in a typed-data field hit the range check \`big >= U256::ONE.shl(bits)\` with the right side spuriously zero, so the EIP-712 signer rejected valid values like \`uint64: 1\` as out-of-range — making the new typed-data sign path unusable for common fixed-width integer fields outside the existing \`uint8\`/\`uint256\` test coverage. **Fix:** re-implement \`shl\` to iterate INPUT limbs LSB-first; each non-zero limb's bits land in its primary output slot (shifted up by \`bit_shift\`) plus a secondary slot when \`bit_shift > 0\`. No off-by-one possible. **Regression tests:** - \`u256_shl_at_64_bit_boundary_does_not_drop_to_zero\`: asserts \`U256::ONE.shl(64) == 2^64\`, same for 128 + 192. - \`uint64_accepts_value_one\`: end-to-end at the encoder layer. - \`uint128_accepts_mid_range_value\`: confirms 2^127 round-trips. ### P2-5: int256 range check was skipped entirely \`encode_int\` guarded the range check behind \`if bits < 256\` so for \`int256\` fields no check ran. Values >= 2^255 (which should be rejected — they wrap into negative two's-complement under signed-256) were accepted silently. An attacker could craft a typed-data payload whose declared int256 value lies outside the signed range and get a signature anyway. **Fix:** drop the \`if bits < 256\` guard. The boundary \`pos_max = U256::ONE.shl(bits - 1)\` fits in U256 for every supported N from 8 to 256 (for N=256, pos_max = 2^255 — exactly representable). **Regression tests:** - \`int256_rejects_value_at_or_above_2_pow_255\`: 2^255 → rejected. - \`int256_accepts_max_positive\`: 2^255 - 1 → accepted. - \`int256_accepts_min_negative\`: -2^255 → accepted. ### P2-6: clap-derived flag name was --seven-thirty-file, docs said --7730-file \`crates/agentkeys-cli/src/main.rs\` — clap derives the long-flag name from the Rust field ident. \`seven_thirty_file\` becomes \`--seven-thirty-file\`. But the command's \`long_about\` text + every example advertised \`--7730-file\`. Users following the doc would hit \"unrecognized argument: --7730-file\". **Fix:** explicit \`#[arg(long = \"7730-file\", ...)]\` override. \`agentkeys signer preview-7730 --help\` now shows the \`--7730-file <SEVEN_THIRTY_FILE>\` flag matching the docs. ## Test summary - \`cargo test -p agentkeys-core --lib audit\`: 22 tests pass. - \`cargo test -p agentkeys-core --lib clear_signing\`: 37 tests pass. - \`cargo test --workspace\`: 47 test suites, 0 failures. - \`bash -n scripts/setup-heima.sh\`: clean. - \`target/debug/agentkeys signer preview-7730 --help\`: shows \`--7730-file\`. * K11 WebAuthn: wire intent text through CLI + harness call sites ## Answer to the user's question > in local webauthn signing process with touchID, I see challenge is a > encoded raw data, is there a readable original text? YES — the library API for it shipped in PR #95 (\`assert_webauthn_with_intent\`, \`assert_webauthn_for_chain_with_intent\`, the \`K11IntentContext\` type, the HTML intent block above the raw challenge dump on the confirmation page). But the CLI subcommand \`agentkeys k11 assert --webauthn\` and the harness helper scripts still used the LEGACY non-intent entry points — so when the user ran the harness with \`--webauthn\`, the confirmation page rendered only the 32-byte challenge hex. The plumbing was incomplete at the seam between the harness scripts and the library. This commit completes the plumbing end-to-end. ## What changed ### CLI: \`agentkeys k11 assert --webauthn\` accepts intent flags \`crates/agentkeys-cli/src/main.rs\` — \`K11Action::Assert\` gains two new flags: - \`--intent-text <STRING>\` — the headline rendered prominently on the WebAuthn confirmation page. Example: \`--intent-text \"Grant agent demo-agent access to openrouter\"\`. - \`--intent-field <Label=Value>\` (repeatable) — per-field detail rows below the headline. Example: \`--intent-field \"Service=openrouter\" --intent-field \"K3 epoch=1\"\`. Both flags are ignored in stub mode (\`--webauthn\` not passed). The dispatch builds a \`K11IntentContext\` and calls the corresponding \`*_with_intent\` library entry point. \`Label=Value\` parsing splits on the FIRST \`=\` (so values may contain \`=\` themselves); empty labels + rows without \`=\` are rejected with a clear operator-facing error. ### Harness scripts: 5 call sites now pass op-specific intents | Script | Op | Intent text | |---|---|---| | \`harness/scripts/heima-device-add.sh\` | \`registerAdditionalMasterDevice\` | \"Register companion device as 2nd master\" + new device hash, role bitfield, companion RP ID, chain ID, nonce | | \`harness/scripts/heima-recovery.sh\` | \`revokeMasterDevice\` (M-of-N) | \"Revoke master device via M-of-N recovery quorum\" + target hash, threshold, asserting role, chain ID | | \`scripts/heima-device-revoke.sh\` | \`revokeDevice\` (master) | \"⚠ REVOKE MASTER device — this disables the operator's master entirely\" + master hash, wallet, recovery note | | \`scripts/heima-scope-set.sh\` | \`setScopeWithWebauthn\` | \"Grant agent '<label>' access to: <services>\" + agent omni, services list, read-only flag, max-per-call, max-per-period, max-total, period, chain ID, scope nonce | | \`scripts/heima-scope-revoke.sh\` | \`revokeScope\` | \"Revoke all scope grants for agent '<label>'\" + agent omni, effect note, chain ID, scope nonce | Each intent is hand-tailored to the op's actual semantics — the \`device-revoke\` master path gets a ⚠-prefixed warning because the operator is one Touch ID tap away from disabling their own master entirely; the others get straightforward descriptive text. ## What the operator sees now Before: \`\`\` 🔑 PRIMARY MASTER K11 assertion Operator 0xb3224706… RP ID localhost Challenge 0xdead…beef ← 32 bytes — only what they saw \`\`\` After (scope-set example): \`\`\` 🔑 PRIMARY MASTER K11 assertion YOU ARE ABOUT TO AUTHORIZE: Grant agent 'demo-agent' access to: openrouter,brave-search Agent label demo-agent Agent omni 0xb3224706… Services openrouter,brave-search Read-only false Max amount per call 1000000000000000000 (0 = unlimited) Max amount per period 10000000000000000000 over 86400s (0 = unlimited) Max total amount 0 (0 = unlimited) Chain ID 212013 Scope nonce 5 Review the above BEFORE pressing Sign. The Touch ID prompt itself cannot show this text — your eyes are the last line of defense between the daemon's claim and the signature. Operator 0xb3224706… RP ID localhost Challenge (raw) 0xdead…beef ← 32-byte commitment — what WebAuthn actually signs [ Sign as PRIMARY MASTER ] \`\`\` The intent rendering is display-only (cryptographic binding is still \`challenge = sha256(message)\`, unchanged). It exists because WebAuthn's OS-level Touch ID prompt is fixed by the platform — no application can inject custom text. The localhost confirmation page is the only surface where AgentKeys can render what's being authorized. ## Tests - \`cargo build -p agentkeys-cli\` clean. - \`cargo test -p agentkeys-cli --lib k11_webauthn\` — 9 tests pass (including the html_escape regression tests proving malicious daemon- supplied intent strings cannot inject \`<script>\` into the page). - \`bash -n\` clean on all 5 updated scripts. End-to-end visual verification (manual): re-run \`harness/v2-stage2-demo.sh --webauthn\` — the Touch ID confirmation page for each master mutation now shows the headline + per-field rows above the challenge hex. * aws: surface STS error source chain so 'dispatch failure' reveals WHY ## Symptom (operator-reported) \`bash harness/v2-stage1-demo.sh\` step 8 (Smoke-test S3 envelope) fails: read failed: internal error: assume_role_with_web_identity(arn:aws:iam::…:role/agentkeys-vault-role): dispatch failure \"dispatch failure\" alone is unactionable — could be DNS, TCP, TLS, proxy, or 'no connector available' (a config bug). The operator can't tell which without re-running the SDK with debug logs. ## Root cause \`aws_sdk_sts::Error\`'s \`Display\` impl renders ONLY the top-level \`SdkError\` variant. For \`DispatchFailure\` that's the literal string \"dispatch failure\" with no causal info. The real reason lives in the \`source()\` chain — which both AgentKeys call sites swallowed: * [crates/agentkeys-provisioner/src/aws_creds.rs](crates/agentkeys-provisioner/src/aws_creds.rs) — operator-side STS for cred reads * [crates/agentkeys-broker-server/src/sts.rs](crates/agentkeys-broker-server/src/sts.rs) — broker-side \`/v1/mint-aws-creds\` Both did \`format!(\"…: {}\", e)\` which loses the chain. ## Fix Walk \`std::error::Error::source()\` recursively at the catch site, flatten into a one-line message: msg = \"assume_role_with_web_identity(…): dispatch failure | caused by: dns error: failed to lookup address information: nodename nor servname provided, or not known\" (...or whichever layer actually failed.) After this lands, the operator's next retry surfaces the actual error: DNS, TCP, TLS, proxy, or no-connector-configured. From there the fix is one-line (\"export HTTPS_PROXY=…\" / \"check corporate VPN\" / \"update CA bundle\") or, if it turns out to be no-connector, a separate in-repo fix (add hyper-rustls feature). ## Why both call sites Symmetry: the same diagnostic gap exists on broker-side (when the broker mints creds via \`/v1/mint-aws-creds\`). Fixing only the operator side would leave the broker emitting the same useless message later. ## Test plan - \`cargo build -p agentkeys-provisioner -p agentkeys-broker-server --release\` clean. - Operator retries: \`bash harness/v2-stage1-demo.sh --only-step 8\` Expect: \"dispatch failure | caused by: <real reason>\" replacing the bare \"dispatch failure\". * heima k11 wrappers: stop swallowing \`agentkeys k11 assert\` stderr ## Symptom (operator) \`bash harness/v2-stage1-demo.sh\` step 13 fails with the unactionable: fail primary K11 ceremony failed fail heima-scope-set.sh failed No hint why — Touch ID was cancelled? challenge mismatch? signature parse error? WebAuthn ceremony timeout? Operator has to manually re-run \`agentkeys k11 assert\` outside the harness to see the real error, reconstructing every CLI flag by hand. ## Root cause Four helper scripts redirected \`agentkeys k11 assert\`'s stderr to \`/dev/null\`: ASSERTION_JSON=\$("\$AGENTKEYS_BIN" k11 assert ... 2>/dev/null) \\ || die "primary K11 ceremony failed" Same diagnostic-swallow pattern that hid the STS \`dispatch failure\` root cause two commits ago (\`238d8ff\`). The shipped error message was the lowest-information form possible: a generic phrase with zero indication of which layer (browser / Touch ID / k11 binary / CLI flag parser) actually failed. ## Fix All four call sites now capture stderr to a tmpfile, print it on failure, clean up on success: K11_ERR=\$(mktemp -t heima-<name>-k11.XXXXXX) || die "mktemp failed" ASSERTION_JSON=\$("\$AGENTKEYS_BIN" k11 assert ... 2>"\$K11_ERR") \\ || { echo "==> K11 assert stderr ↓ ↓ ↓" >&2 cat "\$K11_ERR" >&2 echo "==> K11 assert stderr ↑ ↑ ↑" >&2 rm -f "\$K11_ERR" die "primary K11 ceremony failed (see stderr above for root cause)" } rm -f "\$K11_ERR" Sites fixed: * \`scripts/heima-scope-set.sh\` (line 197) → step 13 * \`scripts/heima-scope-revoke.sh\` (line 122) * \`harness/scripts/heima-register-spare-master.sh\` (line 144) → stage 2 step 8 * \`harness/scripts/heima-device-add.sh\` (line 181) → stage 2 step 6 \`grep -rn "k11 assert.*2>/dev/null"\` returns empty after this commit — no remaining swallows in the harness or scripts/ dirs. ## Why land everywhere at once (per CLAUDE.md Land-the-fix policy) The bug is structural: every heima-*.sh that drives k11 has the same shape. Fixing only \`heima-scope-set.sh\` would leave the operator guessing again when they hit step 6 or step 8 of stage 2. \`grep\` proves the four sites above are the complete set; fixing all four in one commit closes the diagnostic gap for the whole harness. ## Test - \`bash -n\` clean on all 4 scripts. - Operator retries: \`bash harness/v2-stage1-demo.sh --only-step 13\` Expect: instead of just "primary K11 ceremony failed", the new output includes the K11 binary's full stderr — Touch ID error code, CLI parse error, challenge-mismatch detail, etc. From there the next fix is one-line (operator-side action, or in-repo edit per the diagnosis). Same diagnostic-pattern as commit 238d8ff (STS dispatch failure source-chain unrolling). Both close the same class of bug: catch sites that throw away the real reason their dependency failed. Follow-up: heima-register-spare-master.sh also doesn't yet pass \`--intent-text\` to the k11 ceremony so the operator can't see what they're authorizing on the Touch ID confirmation page. Tracked as inline TODO comment; per-script intent wiring lands separately. * k11: uniform intent on every Touch ID prompt (stage-2 step 7/8/9 fix) ## Symptom (operator) In stage-2 demo with --webauthn: step 7 (set recovery threshold): K11 prompt had NO signing info step 8 (register synthetic spare): K11 prompt had NO signing info step 9 PRIMARY (revoke quorum): K11 prompt HAD signing info step 9 COMPANION (revoke quorum): K11 prompt had NO signing info Inconsistent across prompts. Operators learn to ignore the page when some ceremonies show intent + others don't — exactly the failure mode the K11 binding is supposed to prevent (tap-to-approve). ## Root cause Three sites still called \`agentkeys k11 assert\` (or its daemon equivalent) WITHOUT the \`--intent-text\` + \`--intent-field\` flags shipped in commit 69540f2: * \`harness/scripts/heima-set-recovery-threshold.sh\` → step 7 prompt * \`harness/scripts/heima-register-spare-master.sh\` → step 8 prompt * \`crates/agentkeys-daemon/src/companion.rs::approve\` → step 9 COMPANION prompt (rendering side; the API endpoint had no field for the caller to pass intent through) Step 9 PRIMARY worked because heima-recovery.sh had already wired intent on the PRIMARY side. The asymmetry inside one ceremony was the worst case — the operator saw intent on one tap + nothing on the next tap of the same operation. ## Fix Four sites updated to the uniform K11-intent shape (documented in wiki/k11-intent-conventions.md): ### 1. heima-set-recovery-threshold.sh Adds the full intent envelope: --intent-text \"Set recovery threshold to ${THRESHOLD} (M-of-N master quorum)\" --intent-field \"Operator omni=0x${OPERATOR_OMNI}\" --intent-field \"Asserting role=PRIMARY (key hash ${PRIMARY_DEVICE_KEY_HASH})\" --intent-field \"New recovery threshold=${THRESHOLD}\" --intent-field \"Effect=future master-device revokes will require this many active master signatures\" --intent-field \"Chain ID=${LIVE_CHAIN_ID}\" --intent-field \"Operator nonce=${NONCE}\" ### 2. heima-register-spare-master.sh Same envelope, operation-specific headline: --intent-text \"Register synthetic 3rd master (spare) device\" + standard rows + per-op rows (new device hash, role bitfield, effect) ### 3. crates/agentkeys-daemon/src/companion.rs \`ApproveRequest\` extended: pub intent_text: Option<String> pub intent_fields: Vec<String> // each \"Label=Value\" Handler: - Builds K11IntentContext from request fields (splits each \"Label=Value\" on the first \`=\`) - Calls \`assert_webauthn_for_chain_with_intent\` instead of the no-intent variant - Logs intent_text + field count for diagnostics This is the ONLY API change in this commit — the field is optional + serde-defaulted to None/empty so existing callers that don't pass it stay bit-compatible. ### 4. heima-recovery.sh - Both PRIMARY + COMPANION K11 ceremonies now render the SAME headline + same per-op rows + same Effect; only \`Asserting role\` differs per master. - Builds the COMPANION POST body via \`jq -n\` so multi-word labels, equals signs in values, and special characters round-trip safely to the daemon (no shell-quoting traps). - Same uniform envelope: Operator omni / Asserting role / Target device hash / Recovery threshold / Effect / Chain ID / Operator nonce. - stderr capture (per d58aab1 diagnostic pattern) also applied to the PRIMARY k11 assert call so future failures surface the real error. ## Documentation New wiki page \`wiki/k11-intent-conventions.md\`: - Why uniform (load-bearing operator safety property). - The required envelope shape (Operator omni + Asserting role + Chain ID + Nonce + operation rows + Effect). - Canonical headline + Effect text table for every operation (one row per op_kind that needs K11). - Multi-party ceremony rule — both prompts MUST be uniform; only Asserting role differs. - Conformant K11 emit sites table (all 7 sites listed) — checked in by this commit. - \"What doesn't count\" anti-pattern list — caught on every PR review. - Warning-prefix convention (\`⚠ \`) for catastrophic operations (master-device revoke) — used sparingly. \`wiki/k11-webauthn-intent-rendering.md\` (the rendering-mechanism page) cross-links to the new conventions page. ## Test - \`cargo build --release -p agentkeys-daemon\` clean. - \`bash -n\` clean on all 3 modified scripts. - Operator retries: bash harness/v2-stage2-demo.sh --webauthn Expect: every K11 Touch ID prompt across steps 6-9 renders the uniform intent envelope. Step 9 PRIMARY + COMPANION look identical apart from the \`Asserting role\` row. ## Why all four in one commit (per CLAUDE.md Land-the-fix policy) The bug is the asymmetry. Fixing only step 7 + step 8 would still leave step 9 with PRIMARY-shows-intent + COMPANION-doesn't, which is the WORST case the user actually reported. Same root cause + same fix shape across all 4 sites — land together so the convention is enforceable from this commit forward. Follow-up: integration test that asserts every K11 confirmation page contains the required rows, so the convention is mechanically enforced not convention-only. Stub for the test in \`wiki/k11-intent-conventions.md\` § Verification. * k11: typed K11OpIntent enum — concise, decoded, single source of truth ## Symptom (operator feedback) The previous K11 intent rendering was correct but VERBOSE + drifted: Role bitfield = 3 (bit0=CAP_MINT, bit1=RECOVERY, bit2=SCOPE_MGMT) …instead of just: Permissions: CAP_MINT | RECOVERY (raw 3) Operator: "are they hard coded? I want messages to be typed." Diagnosis from `grep -rho 'intent-field "[^=]*='`: - 45 \`--intent-field\` calls across 7 bash scripts - 24 unique label variants (Chain ID vs Chain, etc.) — drift - Role bitfield postfix duplicated in 2 scripts verbatim - Max amounts: every script appended \`(0 = unlimited)\` manually - Hash truncation: every prompt showed the full 66-char omni ## Fix Replace the free-form \`--intent-field "Label=Value"\` flag-spam with a **typed K11 operation intent** carried as a single JSON payload. One enum variant per master-mutation operation; the Rust renderer in \`crates/agentkeys-cli/src/k11_intent.rs\` owns ALL formatting concerns: Raw input → Rendered output ---------- -------------- roles: 3 → "CAP_MINT | RECOVERY (raw 3)" roles: 7 → "CAP_MINT | RECOVERY | SCOPE_MGMT (raw 7)" roles: 0b1000 → "bit3(unknown) (raw 8)" (future-bit surfaces) max_per_call: "0" → "unlimited" three zero amounts → single "Spending limits: unlimited" row 0x941c…64-chars → "0x941cb1…6bef2" (truncated) chain_id: 212013 → "Heima Mainnet (212013)" period_seconds: 3700 → "1h 1m 40s" read_only: true → "Access mode: read-only" ### What landed - \`crates/agentkeys-cli/src/k11_intent.rs\` (NEW, 700+ lines): * \`K11OpIntent\` enum, 8 variants covering every wired master-mutation: SetScopeGrant, SetScopeRevoke, RegisterCompanionAs2ndMaster, RegisterSpareMaster, SetRecoveryThreshold, RecoveryDeviceRevoke, RevokeMasterDevice, RevokeAgentDevice. * \`AssertingRole\` sub-enum (Primary / Companion + key hash). * \`render() -> K11IntentContext\` per variant. Single source of truth for headlines + field labels + format rules. * Formatting helpers: \`format_roles\`, \`truncate_hash\`, \`format_amount\`, \`format_duration\`, \`format_chain_id\`. * 12 unit tests covering: role decode + future-bit surfacing, hash truncation, unlimited-amount rendering, duration units, chain-id labels, scope-grant concise rendering when amounts are zero, role-bitfield-3 end-to-end, multi-party uniformity (recovery PRIMARY vs COMPANION produce identical fields except Asserting role). * Optional fields on RevokeMasterDevice + RevokeAgentDevice (recovery_threshold_remaining, operator_nonce) because the EOA-signed revoke paths don't have a K11Verifier chain nonce — renderer skips the row when None. - \`crates/agentkeys-cli/src/lib.rs\`: module declared. - \`crates/agentkeys-cli/src/main.rs\`: new \`--intent-op-json\` flag on \`k11 assert\`. When set, parses to K11OpIntent + renders via the shared formatter. Takes precedence over the raw \`--intent-text\`/\`--intent-field\` flags (which remain as ad-hoc escape hatches for unwired operations). - \`crates/agentkeys-daemon/src/companion.rs\`: \`ApproveRequest\` gains an \`intent_op: Option<K11OpIntent>\` field. The handler picks it over the legacy raw flags when present + calls \`assert_webauthn_for_chain_with_intent\` with the rendered context. PRIMARY-side caller passes the SAME K11OpIntent (except \`asserting\` differs) → PRIMARY + COMPANION prompts are uniform by construction, not by convention. - Scripts migrated to construct typed JSON via \`jq -n\` + pass \`--intent-op-json\`: * harness/scripts/heima-set-recovery-threshold.sh * harness/scripts/heima-register-spare-master.sh * harness/scripts/heima-device-add.sh * harness/scripts/heima-recovery.sh (both PRIMARY local + COMPANION via POST body's intent_op) * scripts/heima-scope-set.sh * scripts/heima-scope-revoke.sh * scripts/heima-device-revoke.sh \`grep -rln intent-field scripts/ harness/scripts/\` returns empty. - \`wiki/k11-intent-conventions.md\`: rewritten to lead with the typed contract. New "The typed contract" section documents the wire-format JSON, the 8 variants + their required fields, and the formatting-rules table above. The "What does NOT count" + "Verification" sections updated to point at typed tests. ## Test summary - \`cargo test -p agentkeys-cli --lib k11_intent\`: 12 tests pass. - \`cargo test --workspace\`: 0 failures. - \`bash -n\` clean on all 7 migrated scripts. - \`grep -c FAILED\` after workspace test: 0. - \`grep -rln intent-field scripts/ harness/scripts/\`: empty. ## Single-commit reason (CLAUDE.md Land-the-fix policy) The bug is the asymmetry across 7 scripts. Fixing only some leaves operators with mixed-form prompts — the worst case for an attention-as-safety-mechanism. The typed enum, renderer, CLI flag, daemon field, all 7 scripts, and the wiki land together so the contract is enforceable from this commit forward. ## Next step for operators Rebuild + retry stage-2 demo to see the typed prompts: cargo build --release -p agentkeys-cli -p agentkeys-daemon && \\ bash harness/v2-stage2-demo.sh --webauthn Step 6 (companion as 2nd master) should now show "Permissions: CAP_MINT | RECOVERY (raw 3)" instead of the verbose \`Role bitfield=3 (bit0=CAP_MINT, bit1=RECOVERY, bit2=SCOPE_MGMT)\`. Same uniform envelope on every prompt; only \`Asserting role\` and operation-specific rows differ per ceremony. Follow-up tracked in wiki: integration test that crawls the localhost confirmation server + asserts the rendered DOM per op matches expected fixtures, so the convention is mechanically enforced rather than convention-only. * k11 page: drop duplicate Operator/RP-ID rows, unify with intent style ## Symptom (operator screenshot, post-typed-intent refactor) The K11 confirmation page rendered the intent block at the top + then a separate "Operator / RP ID / Challenge (raw)" section below in a DIFFERENT visual layout. Operator omni appeared TWICE — once in the intent block, once in the bottom section. RP ID appeared THREE times: in the rp-callout, in the intent block's "Asserting role" row, and in the bottom section. ## Root cause Two HTML sections rendered on every K11 page: 1. \`<section class="intent">\` — the new typed-intent block (added in commit 8cd6ab9). Already shows Operator omni + Asserting role. 2. \`<section class="kv">\` — the original legacy block. Always rendered Operator + RP ID + Challenge (raw) unconditionally. The legacy block was unconditional, so once the intent block also landed those rows it triplicated the omni + duplicated the RP ID without anyone noticing during the typed-intent work. ## Fix Rebuilt the second section as a typed \`crypto_block\` with two shapes: * **Intent present (the common case)**: shows ONLY the unique cryptographic fact — \`Challenge (raw)\`. Operator omni + RP ID + role are already surfaced above. Same dl-grid layout as the intent block; neutral gray accent so it's clearly the secondary "cryptographic primitives" section, not a parallel call-to-action. * **No intent (legacy callers)**: falls back to the original Operator + RP ID + Challenge layout so any future caller that hasn't migrated to the typed-intent path still sees every fact. CSS \`.crypto / .crypto-h / .crypto-fields\` matches the intent block's border-radius / padding / grid template, so the two sections look like a coordinated pair rather than two different design eras stacked. ## Test - \`cargo build --release -p agentkeys-cli -p agentkeys-daemon\` clean. - \`cargo test -p agentkeys-cli --lib k11\` → 24 tests pass. - Manual verification on next harness run: the second confirmation page section now shows only the raw challenge hex with the same grid layout as the intent block above. --------- Co-authored-by: wildmeta-agent <agent@wildmeta.ai>
Move docs/spec/architecture.md to docs/arch.md, hoist wiki/ to docs/wiki/, and relocate aiosandbox/ from spec/ to research/. Update every cross-link across 60+ files (markdown, Rust comments, GitHub workflows) and rewrite the publish-wiki.yml path to mirror docs/wiki/ instead of wiki/. Five-folder layout, each one audience: spec/ (developers + coordinating colleagues), plan/ (agent-authored pre-implementation plans), research/ (third-party context), wiki/ (end users + hardware integrators, mirrored to GitHub Wiki), archived/ (superseded files; never linked from arch.md). CLAUDE.md gets a 99-word "Docs layout (lean)" section so future doc creation lands in the right place precisely. Wiki-location and arch-source-of-truth policies updated to the new paths. The agentkeys-docs skill (global) enforces this layout going forward: audits cross-links, moves stale files to archived/, surfaces arch.md drift, and keeps each folder's audience separation honest. cargo check on agentkeys-mock-server passes. Co-authored-by: wildmeta-agent <agent@wildmeta.ai>
The claude-code-review.yml workflow previously ran on every push to a PR branch (synchronize event), which burned Claude usage tokens on every iteration of a long-lived PR. Trim the trigger to submission-only events: opened — first PR submission ready_for_review — draft promoted to ready reopened — closed PR resubmitted One auto-review per submission; subsequent commits skip. Re-trigger manually by `@claude` mention (claude.yml) or by closing + reopening the PR. Updates REVIEW_GUIDELINES.md to document the new cadence.
…oker tier-2) (#98) * issue #82: ERC-7730 clear-signing + EIP-712 typed-data sign (v2-aligned) Refresh of issue #82 against v2 architecture (#87/#92). Original issue targeted v1 (mock-server-as-signer, daemon-side metadata, broker SQLite audit); plan was rewritten to the v2 surfaces (signer typed RPC, worker audit rows with intent commitments, ERC-7730 catalog as a §22 pluggable surface). Plan: docs/spec/plans/issue-82-erc7730-v2-aligned.md. ## What ships in this PR ### Phase 1 — EIP-712 typed-data signing at the signer * New endpoint `POST /dev/sign-typed-data` on the mock-server signer: accepts canonical EIP-712 v4 JSON (matches MetaMask `eth_signTypedData_v4`), parses + hashes internally (never trusts a caller-supplied prehash), returns the 65-byte canonical signature + every intermediate digest (`primary_type_hash`, `domain_separator`, final `digest`). * `DevKeyService::sign_eip712` + `Eip712SignResult` envelope. * New `SignerError::InvalidTypedData` (400) + propagation through `SignerClientError`. * `SignerClient::sign_eip712` trait method + `HttpSignerClient` impl. * Wire signer-only + full routers in agentkeys-mock-server. ### Phase 2 — clear_signing module in agentkeys-core New crate module at `crates/agentkeys-core/src/clear_signing/`: * `eip712.rs` — EIP-712 v4 encoder (no external dep). Supports string/bytes/bool/address, uint{8..256}, int{8..256}, bytes{1..32}, static/dynamic arrays, nested struct types. Cycle detection on type graph. Spec reference vector (`Mail` example) matches exactly. * `parser.rs` — ERC-7730 v2 JSON parser (subset for v0). * `format.rs` — per-field formatters (tokenAmount with decimals+ticker, address with truncation, integer, date as ISO-8601 UTC, bool, raw) + `{name}` intent interpolator. * `binding.rs` — domain-{name,version,chainId,verifyingContract} → 7730-file lookup; case-insensitive on address; refuses wildcard matches. * `catalog.rs` — bundled set (USDC permit fixture) + filesystem dir loading via `extend_from_dir` (operators ship custom files via `$AGENTKEYS_7730_DIR`). * `mod.rs::build_preview` — top-level "render this typed-data against this catalog" returning `intent_text` + `intent_commitment` = `keccak256(intent_text || 0x7c || digest)`. ### Phase 3 — CLI preview surfaces Two new subcommands under `agentkeys signer`: * `sign-typed-data` — call `/dev/sign-typed-data`. With `--preview-7730`, renders + prints operator intent + per-field review before signing. * `preview-7730` — render WITHOUT signing. Dry-run for new 7730 files before plumbing them into automated agent signing. Both pick up `$AGENTKEYS_7730_DIR` for operator-custom 7730 files; both support `--json` for machine-readable output. ### Phase 4 — audit-row intent-commitment schema (arch.md only) `arch.md §15.3` extended with two optional audit-row fields (`signed_intent_text`, `signed_intent_hash`). Schema is backwards- compatible — pre-#82 rows have the fields absent; worker reads/writes land in a follow-up PR (broker cap-mint propagation + on-chain `CredentialAudit` event extension also follow-up). ### Docs * `docs/spec/signer-protocol.md` — full `/dev/sign-typed-data` wire contract documented (request, response, supported type-string subset, errors). * `docs/spec/architecture.md` §14.2 + §15.3 + §22 — typed-data RPC in the signer surface, audit-row intent-commitment fields, clear-signing metadata as a pluggable surface (bundled → registry → on-chain progression). * `docs/spec/plans/issue-82-erc7730-v2-aligned.md` — full refreshed plan, including the K11-binding-on-high-value-signs follow-up (Phase 5 — out of scope here, tracked as separate issue since it needs a ScopeContract extension). ## Test plan * `cargo test --workspace` — 600+ tests across the workspace, all pass. * New tests added in this PR: - 30 unit tests under `agentkeys-core::clear_signing` (EIP-712 spec reference vector, cyclic type detection, integer range checks, array length validation, U256 dec/hex roundtrip, two's-complement negation, parser, formatter, binding, catalog). - 2 sign_eip712 unit tests in `dev_key_service.rs` (recovers-to-derived-address, malformed-typed-data rejection). - 6 route tests in `dev_key_service_routes.rs` (200 / 400-unknown- primary / 400-out-of-range-uint / 503-signer-disabled / address- matches-derive / full-sig-recovery-roundtrip). * `cargo clippy` — clean on all new code; pre-existing warnings unchanged. * Signature roundtrip verified: HKDF-derived secp256k1 key signs the EIP-712 digest, `ecrecover` returns the same address that `derive_address` produces for the same `omni_account`. ## What did NOT land in this PR Tracked as follow-ups so this PR stays scoped: * **Broker cap-mint policy gate** — the broker cap-mint endpoint doesn't yet require an `intent_commitment` for typed-data signs. Today the daemon goes direct to the signer via `signer_client`. When broker mediation lands, the cap-token carries the commitment. * **Worker audit-row wiring** — `agentkeys-worker-audit` doesn't read the new schema fields yet (forward-compatible; unknown fields are silently ignored). Schema is documented in arch.md §15.3 so the follow-up PR has a fixed target. * **On-chain `CredentialAudit` event extension** — needs a contract revision + redeploy; out of scope for a signer + worker change. * **Registry fetch (v1 source)** — `github.com/ethereum/clear-signing- erc7730-registry` integration is the v1 catalog source per arch.md §22 (the bundled set is the v0 default that ships in this PR). * **EIP-4337 UserOp clear signing** — out of scope per original #82. * **K11 binding on high-value signs** — Phase 5 in the plan; needs a ScopeContract extension to express "agent A may sign EIP-712 binding to chainId=1 verifyingContract=$X with tokenAmount ≤ Y". Plan-completion summary: * **What landed**: Plan refresh, signer-protocol.md update, arch.md §14.2/§15.3/§22 updates, `/dev/sign-typed-data` endpoint, signer-side EIP-712 hashing (no external dep), `clear_signing` module (parser + formatter + binding + catalog + EIP-712), bundled USDC permit fixture, CLI `sign-typed-data` + `preview-7730` subcommands, audit-row intent- commitment schema doc, full sig-recovery roundtrip test. * **What did NOT land**: Broker cap-mint policy gate, worker audit-row wiring, on-chain `CredentialAudit` event extension, registry-fetch catalog source, K11-on-high-value-signs (Phase 5). All tracked explicitly in the plan doc as follow-ups. * issue #97: arch.md §15.3a — AuditEnvelope v1 canonical schema Defines the unified abstract audit message format that every audit-producing surface (creds, memory, signer, broker, payment-service, email-service, SidecarRegistry, K3EpochCounter) MUST emit going forward, and that the chain + explorer + indexer consume. ## What this section adds * **Envelope schema** — version, ts_unix, actor_omni, operator_omni, op_kind (u8), op_body (CBOR), result, intent_text + intent_commitment (PR #95). Canonical CBOR per RFC 8949 §4.2.1. * **Wire shape** — `POST /v1/audit/append` accepts the envelope; `GET /v1/audit/envelope/<hash>` returns the full envelope on demand (used by explorers). * **On-chain shape** — `CredentialAudit.appendV2(operatorOmni, actorOmni, opKind, envelopeHash)` + `appendRootV2(... opKindBitmap)` lands additively alongside the v1 `append`/`appendRoot`. New events `AuditAppendedV2` + `AuditRootAppendedV2` with `indexed opKind` topic so explorers can filter via `eth_getLogs`. * **Canonical op_kind table** — 17 op_kinds across 8 families (creds=0..2, memory=10..12, signs=20..21, payments=30..31, scope=40..41, device=50..52, email=60..61, K3=70). Grouped by 10s leaves room for related ops. PRs adding new op_kinds MUST append a row; numbers never reused, never reordered. * **Eight non-break invariants** — the cost of adding a new op_kind is "uglier UI temporarily for old explorers" — never "broken explorer / dropped event." Open enum, stable envelope-level fields, version gating, fallback renderer, opaque body pass-through, op-kind-agnostic contract, canonical table, 3-test contract per new op_kind. * **5-phase migration** — A (this doc) → B (worker + core migration) → C (contract revision) → D (subscan-essentials decoder) → E (subscan-essentials-ui-react renderer) → F (extend op_kind coverage). Phases B / C / F tracked at agentkeys#97; phases D / E tracked at subscan-essentials#12. ## Why this matters Today's audit surface only has 3 op_kinds (STORE / READ / TEARDOWN) and those are credential-CRUD-only. A typed-data sign event, a scope mutation, a device add, a payment, a memory put, an email send, a K3 epoch advance — none of these have a row to render in the explorer. With this section in place, the explorer can render a uniform timeline across all of them, and adding a new op_kind doesn't require the explorer to ship a release before AgentKeys can ship the feature. ## What does NOT land in this PR This is the schema lock-in (Phase A). The implementation phases (worker migration, contract redeploy, explorer decoder, UI renderer) ship as follow-ups in their respective repos. agentkeys#97 + subscan-essentials#12 are the tracking issues. * issue #97 phase B: AuditEnvelope v1 struct + worker V2 endpoints Lands the canonical AuditEnvelope shape as live code, not just a doc. Documented in arch.md §15.3a; this commit ships the worker side. Contract revision (Phase C) + emit-site migration across signer/scope/device/payment/ memory/email/K3 (Phase F) remain follow-ups in #97. ## What ships ### `agentkeys-core::audit` — canonical envelope (new module) * `AuditEnvelope` struct — version + ts_unix + actor_omni + operator_omni + op_kind (u8 open enum) + op_body (ciborium::Value) + result + intent_text + intent_commitment. Envelope-level fields are stable across all op_kinds. * `AuditOpKind` repr-u8 enum — 18 variants matching arch.md §15.3a canonical table (creds=0..2, memory=10..12, signs=20..21, payments=30..31, scope=40..41, device=50..52, email=60..61, K3=70). Open enum: `from_u8` returns Option, never panics. * `AuditResult` repr-u8 enum (Success=0, Failure=1, NotPermitted=2). * Per-op_kind typed body schemas in `audit::bodies` — 18 structs with serde derives matching the canonical table field-for-field. * Canonical CBOR codec in `audit::cbor` — deterministic per RFC 8949 §4.2.1. Encoder builds the envelope as an ordered CBOR map with keys sorted by canonical CBOR ordering. Decoder ignores unknown envelope-level keys (forward-compat) and rejects unsupported envelope versions. * `envelope_hash()` = keccak256(canonical_cbor). The 32-byte commitment that lands on chain as the second arg to the future `CredentialAudit.appendV2(operatorOmni, actorOmni, opKind, hash)`. * `commit_intent()` helper — same scheme as `clear_signing::commit_intent` (PR #95); verified by a test that asserts byte-for-byte equality between the two. ### `agentkeys-worker-audit` — V2 endpoints * `POST /v1/audit/append/v2` — accept envelope (as JSON), convert op_body to CBOR, compute envelope_hash, store CBOR by hash. Returns `{envelope_hash}`. * `GET /v1/audit/envelope/:hash` — return canonical CBOR bytes for the envelope (200 application/cbor) or 404 envelope_not_found. Explorers fetch via this endpoint after seeing the on-chain hash. * V1 endpoints (`/v1/audit/append`, `/v1/audit/flush/:op`, etc.) retained so existing callers keep working through the migration cycle. * `state.rs` extended with `envelopes: Mutex<HashMap<String, Vec<u8>>>` — in-memory v0; persistent S3 storage is a separate concern tracked alongside Phase C. ### Non-break invariants enforced by code Per arch.md §15.3a: 1. ✅ `op_kind` is `u8`, never a sealed enum (open enum design; `AuditOpKind::from_u8` returns Option). 2. ✅ Envelope-level fields decode for ANY op_kind, even op_kind=250 (test: `unknown_op_kind_still_decodes_envelope_level_fields`). 3. ✅ `version` bumped only on envelope-level breakage; new op_kinds stay at v1. 4. ✅ Worker accepts unknown op_kinds + stores the opaque body for explorers to fetch (test: `append_v2_accepts_unknown_op_kind`). 5. ✅ Decoder ignores unknown envelope-level keys (forward-compat for future versions; test: `decoder_ignores_unknown_envelope_keys`). 6. ✅ No contract-side decode of op_body — only `(opKind, envelopeHash)` would land on chain (Phase C scope; out of this PR). 7. ✅ Canonical op_kind table in arch.md §15.3a — `op_kind.rs::tests` asserts no byte collisions + all variants roundtrip. ## Tests * 17 unit tests in `agentkeys-core::audit` — envelope encode/decode, envelope hash determinism, unknown-op_kind tolerance, version refusal, typed body decode, op_kind byte uniqueness, commit_intent parity with `clear_signing::commit_intent`. * 7 integration tests in `agentkeys-worker-audit::tests::envelope_v2`: - append → 200 + envelope_hash with correct shape - GET → 200 application/cbor with canonical bytes - GET unknown hash → 404 envelope_not_found - reject envelope version 99 - reject malformed actor_omni - accept unknown op_kind (non-break invariant #1 + #4) - envelope_hash deterministic across appends - ts_unix=0 gets server-assigned * `cargo test --workspace` — 600+ tests, **0 failures, 1 ignored** (network-dependent test; pre-existing). * `cargo clippy` — clean on all new code. ## What does NOT land in this PR Tracked in #97 as Phases C + F: * On-chain `CredentialAudit.appendV2` + `appendRootV2` + new events with indexed opKind topic — needs contract revision + Heima Mainnet redeploy. * Migration of credentials-service + memory-service + signer + broker emit sites from legacy `AuditEvent` to `AuditEnvelope`. Each new op_kind PR will append a row to the arch.md §15.3a table + add the worker emit-site call. * Persistent storage for envelopes (S3 `audit/envelopes/<hash>.cbor`). In-memory v0 is sufficient for the worker's lifecycle; if the worker restarts before chain commitment lands, callers re-emit. * Subscan-essentials indexer decoder + UI renderer (subscan-essentials#12). * issue #97 phase B: AuditClient — convenience HTTP client for the V2 endpoints Future emit sites (credentials-service, memory-service, signer, broker, payment-service, email-service, SidecarRegistry, K3EpochCounter) all need the same `POST /v1/audit/append/v2` + `GET /v1/audit/envelope/<hash>` wire shape. Putting the client in agentkeys-core means each emitter consumes the contract from one place — and the wire-level test surface is centralized. ## What ships * `agentkeys_core::audit::AuditClient`: - `new(base_url)` / `from_env()` (reads `$AGENTKEYS_AUDIT_WORKER_URL`, defaults to `https://audit.litentry.org`). - `append(envelope)` → returns `{ok, envelope_hash}` from the worker. - `get_envelope(hash)` → `Option<Vec<u8>>` (None on 404). * `envelope_for(actor, operator, op_kind, op_body, result, intent_text, intent_commitment)` convenience builder — constructs an envelope from a typed body (any `serde::Serialize`), wires the canonical CBOR. ## Emit-and-forget semantics Per arch.md §15.3a, chain commitment is the durability mechanism — the worker's in-memory envelope map is best-effort cache. Emitters that need guaranteed delivery either retry on transient failure or fall back to direct on-chain `CredentialAudit.append`. ## Tests Two unit tests added in `audit::client::tests`: * `envelope_for_builds_typed_body` — round-trip through the typed body decoder: `SignEip712Body` → envelope → `typed_body()` returns the same body. * `envelope_for_emits_canonical_cbor` — same inputs produce same `envelope_hash` regardless of build path (cross-encoder stability). Total audit-module tests now 19. Full workspace `cargo test --workspace` clean (600+ tests, 0 failures). * issue #97 phase C: CredentialAudit.appendV2 + appendRootV2 (contract code only) Adds the V2 surface to the CredentialAudit contract per arch.md §15.3a. V1 (`append` + `appendRoot`) is retained unchanged so existing indexers + the live tier-A worker keep working through the migration cycle. ## What ships * `appendV2(operatorOmni, actorOmni, opKind, envelopeHash)` — emits `AuditAppendedV2(operatorOmni indexed, actorOmni indexed, opKind indexed, envelopeHash)`. **Event-only — no on-chain storage.** The full envelope lives off-chain at the audit-service worker, addressed by `envelopeHash = keccak256(canonical_cbor(AuditEnvelope))`. The `opKind` indexed topic lets explorers filter `eth_getLogs` by op_kind without scanning every row. * `appendRootV2(operatorOmni, merkleRoot, opKindBitmap, batchEntryCount)` — emits `AuditRootAppendedV2`. `opKindBitmap` is `bytes32` where bit N = op_kind N is present in the batch. Lets explorers filter batches by op_kind without fetching every leaf from the worker. Gated to the operator's master wallet (same as V1 `appendRoot`, codex M1). * No on-chain decode of `op_body` — the contract stays op-kind-agnostic (non-break invariant #6 per arch.md §15.3a). New op_kinds need ZERO contract redeploys. ## Forge tests 5 new tests in `AgentKeysV1.t.sol` (alongside 4 existing CredentialAudit tests): * `test_CredentialAudit_AppendV2_EmitsEvent` — confirms the event topics carry operator + actor + opKind for `eth_getLogs` filtering. * `test_CredentialAudit_AppendV2_AcceptsAnyOpKind` — invariant #1 + invariant #6: op_kind=250 (reserved future byte) accepted without revert. * `test_CredentialAudit_AppendV2_OpenToAnyCaller` — `appendV2` is open to any caller (chain ordering + gas is the safety; indexer filters out attacker-emitted noise via canonical envelope hashes). * `test_CredentialAudit_AppendRootV2_EmitsEvent` — Merkle-batch path with multi-op_kind bitmap (bits 0 + 21 + 40 = CredStore + SignEip712 + ScopeGrant set). * `test_CredentialAudit_AppendRootV2_RejectsNonMaster` — gated to operator's master wallet per codex M1. * `test_CredentialAudit_V1_And_V2_Coexist` — V1 `append` + V2 `appendV2` write to disjoint paths; V2 emits don't touch V1's `entries` storage. Forge: 9/9 CredentialAudit tests pass; full forge suite 39/39 tests pass. Workspace cargo test still clean. ## Redeploy: operator action This commit ships the contract code + tests. The actual Heima Mainnet redeploy via `scripts/heima-bring-up.sh --upgrade` is operator action gated on PR review — left for a follow-up operator step. Until redeployed, the live `CredentialAudit` on Heima still has only V1 methods, so callers of `agentkeys-worker-audit::handlers::append_v2` can store envelopes off-chain but can't commit `envelopeHash` to chain until redeploy lands. Migration sequence per arch.md §15.3a Phase C: 1. Operator reviews this PR. 2. Operator runs `bash scripts/heima-bring-up.sh --upgrade` (idempotent — redeploys CredentialAudit if address bytecode hash changed). 3. Operator captures new address into `scripts/operator-workstation.env` + `docs/spec/deployed-contracts.md`. 4. Run `AGENTKEYS_CHAIN=heima bash scripts/verify-heima-contracts.sh`. 5. Run harness/v2-stage1-demo.sh through 3 to confirm no regression (V1 path still works on the redeployed contract). * issue #97: recursive op_body canonicalization + arch.md event sig fix Address two architect-review findings against earlier commits in this PR (reviewer: oh-my-claudecode:architect on PR #95). ## Fix 1 — recursive op_body canonicalization (cross-language hash determinism) Architect finding (section 4): the canonical CBOR encoder sorted only envelope-level keys, not `op_body` map keys recursively. The Rust ecosystem happened to produce stable hashes because `serde_json::Value:: Object` is `BTreeMap`-backed, but a Go or TypeScript encoder building `op_body` with unsorted keys would have produced different CBOR bytes and a different `envelope_hash` — silently breaking the chain-commitment property for cross-language clients. `audit::cbor::canonicalize()` now walks `op_body` recursively: every nested map's keys are sorted by their canonical CBOR-encoded bytes (RFC 8949 §4.2.3). Arrays preserve order (semantic ordering). Two new tests prove the property: * `op_body_key_order_does_not_affect_hash` — flat map, alphabetical vs reverse-alphabetical insertion order → identical envelope_hash. * `op_body_nested_map_key_order_does_not_affect_hash` — nested map recursion check. Total audit-module tests now 21. Workspace cargo test clean. ## Fix 2 — arch.md event signatures match the actual contract Architect finding (section 3): arch.md §15.3a `AuditAppendedV2` / `AuditRootAppendedV2` declarations included `entryIndex` / `rootIndex` fields that the actual `CredentialAudit.sol` events do NOT emit. Explorer implementers reading arch.md would have expected fields that aren't there. Doc updated to match the live contract surface. Added a sentence explaining V2's event-only design: position within the operator's stream is derivable from `(block_number, log_index)` so the contract doesn't need to carry `entryIndex` explicitly. ## What this PR ships (cumulative across all commits) Phase A — arch.md §15.3a (canonical schema + table + non-break invariants + migration phases) ✅ Phase B — agentkeys-core::audit module + worker V2 endpoints + AuditClient ✅ Phase C — CredentialAudit.appendV2 + appendRootV2 (code + 5 forge tests; redeploy is operator action) ✅ Phase D / E (subscan-essentials decoder + UI) tracked at subscan-essentials#12. Phase F (extend emit coverage to sign/scope/device/payment/email/K3) tracked at agentkeys#97. * docs+ops: add-op-kind ritual + setup-heima orchestrator + idempotency rule Three related changes addressing user request after the #97 op-kind work: ## 1. How-to-add-a-new-op-kind documentation ### arch.md §15.3b — the 5-step ritual Brief operator-facing ritual: (1) pick the byte from the appropriate family range, (2) append a row to §15.3a canonical table, (3) add the Rust variant in `audit::{op_kind,bodies,mod}`, (4) wire the emit site via `envelope_for` + `AuditClient::append`, (5) ship 3 tests (CBOR roundtrip + explorer Unknown(byte) fallback + arch.md row uniqueness). Critical invariant called out: never bump ENVELOPE_VERSION for a new op_kind. The version is reserved for envelope-level breakage; open-enum op_kinds are the whole point. ### wiki/audit-envelope-add-op-kind.md — detailed worked example Walks through adding `PaymentRefund` (byte 32) end-to-end: - Step-by-step code for op_kind.rs / bodies.rs / mod.rs. - Sample emit-site wiring in a worker handler. - Complete PR checklist + the explicit "what you DON'T need to do" list (no contract redeploy, no version bump, no migration, no synchronous rollout). Lives under `./wiki/` per CLAUDE.md "Wiki-location policy" — auto- publishes to the GitHub wiki on every push to main. ## 2. scripts/setup-heima.sh — single idempotent entry point Mirrors the `scripts/setup-broker-host.sh` pattern: one operator-facing orchestrator that runs the entire Heima chain bring-up + binding flow end-to-end in 15 idempotent steps. Delegates to the existing per-action helpers (`heima-bring-up.sh`, `heima-device-register.sh`, `heima-agent-create.sh`, `heima-scope-set.sh`, `heima-credential-audit.sh`, `heima-worker-smoke.sh`, `verify-heima-contracts.sh`) so: - Each helper's existing idempotency check (`cast call <view-fn>`, `cast code <addr>`, `cast balance ≥ amount`, file-exists guards) is preserved. - Per-action helpers stay callable directly for surgical re-runs (e.g. `bash scripts/heima-scope-set.sh ...` for just the scope work). - The orchestrator is THE entry point operators run — same posture as setup-broker-host.sh. Flag surface mirrors the harness orchestrators: `--chain`, `--session-id`, `--agent-label`, `--service`, `--webauthn`, `--yes`, `--from-step N`, `--to-step N`, `--only-step N`, `--help`. Two append-only steps (13 audit append + 14 tier-A relay) are explicitly called out in the header per the CLAUDE.md rule: "If a remote-setup script you're writing CAN'T be made idempotent (...append-only audit event), explicitly call it out." `bash -n` clean; `--help` renders correctly. ## 3. CLAUDE.md — idempotent remote-setup rule New section "Idempotent remote-setup rule (CLOUD / BLOCKCHAIN / CI / VM)" makes the existing implicit pattern an explicit project policy: - Every remote-mutation script (AWS / Heima / CI / VM / Cloudflare / Tencent / IAM / DNS) MUST be idempotent. Re-runs MUST exit 0 without re-applying. - Three reasons: operators retry, CI re-runs, the harness re-runs as a regression gate. - Concrete pre-check / short-circuit table for 9 mutation types (contract deploy, chain tx, fund EVM account, AWS resource, systemd unit, env file, nginx vhost, DNS A record, key gen). - Output convention: `ok proceeding` / `skip <reason>` / `fail <reason>` so the harness can read state per step. - Exception clause: if truly non-idempotent (one-shot CAS-burn cap, append-only audit event), explicitly call it out in script header AND runbook. Also adds "Heima chain (single entry point)" section pointing at the new `setup-heima.sh`. * issue #66: add no-LLM CI — ephemeral anvil + scaffolded test-broker E2E Two-tier CI matching issue #66's "shared test broker for CI + dev" vision: Tier 1 — ephemeral (every push/PR, fully self-contained, ~10–15 min): * .github/workflows/harness-ci.yml — cargo fmt + clippy + test + harness/ci-ephemeral-stack.sh. No LLM, no @claude invocation. * harness/ci-ephemeral-stack.sh — spins up anvil (new chain), runs forge build + test, deploys fresh v2 stage-1 contracts via DeployAgentKeysV1.s.sol (new contracts, new anvil-prefunded deployer), verifies via scripts/verify-heima-contracts.sh, then stands up mock-server + agentkeys-broker-server with --skip-startup-check (StubSts path) and probes OIDC discovery surface. EXIT trap tears everything down. Tier 2 — long-lived test broker (nightly + workflow_dispatch, scaffolded here, operator-activated via TEST_OIDC_AWS_ROLE_ARN secret): * .github/workflows/harness-e2e.yml — gated workflow that targets test-broker.litentry.org with real test AWS resources, runs all three stage demos against the long-lived parallel infra. Includes nightly cleanup of stale ci/ S3 prefixes. Uses GitHub Actions OIDC (id-token: write) for AWS auth, never long-lived secrets. * scripts/provision-test-environment.sh — operator-run one-shot provisioner that walks the 7 steps to stand up test-broker (separate OIDC provider, separate IAM roles, separate buckets, separate deployer wallet, fresh contracts on Heima-Paseo). * scripts/test-environment.env.example — committed env template mirroring operator-workstation.env with -test suffixes. * docs/test-environment.md — bring-up runbook, secret list, rotation, cleanup, and the two-tier design rationale. WebAuthn: harness scripts default to WEBAUTHN_MODE=0 (stage-1 line 131, stage-2 --stub) so no Touch ID prompt is ever needed; --webauthn is opt-in and never passed by either workflow. Validated locally: bash harness/ci-ephemeral-stack.sh --skip-broker passes all 8 steps (anvil up, 33 forge tests, 6 contracts deployed + verified, clean teardown). YAML + shell syntax checked. * issue #66: collapse to one CI file; mirror prod env on Heima mainnet Per operator feedback: 1. "do not create new files, only add the test file" — drop the ephemeral-stack helper, provisioner, env template, e2e workflow, and docs. Single deliverable: .github/workflows/harness-ci.yml. 2. "onchain solution should test on Heima mainnet with a new smart contract address" — confirmed possible: Solidity compiles deterministically and EVM contract addresses derive from (deployer, nonce). Identical crates/agentkeys-chain/src/*.sol + identical DeployAgentKeysV1.s.sol + a different deployer key on Heima mainnet = isolated parallel contract set at new addresses on the production chain. 3. "CI mirrors the production env" — the workflow now invokes the PRODUCTION harness scripts (harness/v2-stage{1,2,3}-demo.sh) unchanged. The only thing CI does differently from a prod operator is materialize scripts/operator-workstation.env with TEST_* resource names from GitHub secrets: - TEST_OIDC_AWS_ROLE_ARN (gate; until set, harness job skips) - TEST_ACCOUNT_ID / TEST_AWS_REGION / TEST_BROKER_HOST - TEST_VAULT_BUCKET / TEST_MEMORY_BUCKET - TEST_{VAULT,MEMORY,DATA}_ROLE_ARN - TEST_HEIMA_DEPLOYER_KEY (raw 0x-prefixed mainnet key — test wallet, distinct from prod deployer) - TEST_{SCOPE,SIDECAR_REGISTRY,K3_EPOCH_COUNTER, CREDENTIAL_AUDIT,P256_VERIFIER,K11_VERIFIER}_CONTRACT_ADDRESS_HEIMA (pre-deployed once per test-env refresh; harness skips deploy via --skip-deploy so CI doesn't burn HEI on every push) AWS auth via GitHub Actions OIDC (id-token: write), no long-lived secrets. Per-run S3 prefix isolation. The workflow gates itself on TEST_OIDC_AWS_ROLE_ARN being set so it's inert until the operator activates the test infra. WebAuthn: never invoked — harness scripts default to WEBAUTHN_MODE=0 (stage-1 line 131) and stage-2's --stub flag is passed explicitly. LLM: zero. Plain cargo/forge/aws-cli/curl orchestration. Distinct from claude.yml + claude-code-review.yml which intentionally do call @claude. * docs: concise setup guides aligned with scripts/setup-{broker-host,heima}.sh Per operator request: pivot cloud-setup.md from a verbose manual-bash runbook to a concise prereq/script-pointer split, add new heima-setup.md + ci-setup.md for the chain + CI flows, and move troubleshooting into the ./wiki/ folder. What changed: docs/cloud-setup.md — UPDATE, 970 → 314 lines Add a TL;DR with the three-command operator flow (manual §1-§4 prereqs, then setup-broker-host.sh, then setup-heima.sh). Slim §1-§4 to invariants + helper-script pointers + brief command blocks (DKIM bulk-record / receipt rule / per-data-class role provisioning all delegate to the existing scripts/*.sh). Replace the verbose §5/§6/§7 (EC2 broker / signer / workers, each with 100+ lines of inline bash) with one §5 "Run setup-broker-host.sh" section that names what the script does (build, systemd, nginx, certbot, keypairs, env files) + what it doesn't (DNS, IAM, OIDC provider — those stay in §1-§4). Keep §0 (identities table) and §6 (cleanup recipe). docs/heima-setup.md — NEW, 106 lines The 15-step pipeline in scripts/setup-heima.sh, with idempotency check + helper-script pointer per step. Mainnet vs Paseo vs Anvil tradeoff table. Per-step re-run examples. Heima London EVM pin explanation. docs/ci-setup.md — NEW, 184 lines The 7-step operator bring-up for the no-LLM .github/workflows/harness-ci.yml workflow: provision test broker via setup-broker-host.sh with -test suffix, provision parallel AWS resources, register the test OIDC provider, generate + fund the test deployer wallet, deploy fresh test contracts on Heima mainnet with the same .sol source (different deployer → different addresses → isolated parallel contract set), register the GitHub Actions OIDC role, set the repo secrets. Includes the full TEST_* secret list, manual-dispatch instructions, and a secret-hygiene reminder. wiki/cloud-setup-faq.md — NEW, 94 lines wiki/heima-setup-faq.md — NEW, 111 lines wiki/ci-setup-faq.md — NEW, 96 lines Troubleshooting + edge cases for each setup doc. Lives under ./wiki/ per CLAUDE.md "Wiki-location policy" — auto-published to the GitHub wiki on every push to main. Constraints applied: - Concise: every doc fits in a few screens. - Idempotent: every flow reuses the existing idempotent helper scripts (setup-broker-host.sh, setup-heima.sh, provision-*-role.sh, apply-*-bucket-policy.sh). - No project credentials exposed: account IDs, role ARNs, bucket names, deployer keys, contract addresses all referenced via ${ACCOUNT_ID} / ${BROKER_HOST} / ${REGION} placeholders or via "read from operator-workstation.env" / "from step N" pointers. Real values live only in the operator's local env file + the GitHub repo secrets store. All internal links verified via a python url-walker (every relative link resolves to an existing file). * docs: extract first-time cloud bootstrap into separate doc Per operator request: the very-beginning cloud-account provisioning (IAM users + role, DNS, SES, S3 buckets, instance profile) needs to live in a separate doc so it stays reachable when: - Adding a second AWS account (test instance, regional shard) - Migrating to AliCloud / GCP / Tencent Cloud - Re-bootstrapping after a teardown - Auditing the identity surface The previous condense pass collapsed those sections into cloud-setup.md's slim §1-§3 — convenient for day-to-day operators but stripped the depth needed for the migration / second-account use cases. What changed: docs/cloud-bootstrap.md — NEW, 365 lines First-time, per-account, cloud-provider-portable bootstrap doc: §1 Identities — four IAM principals, cloud-agnostic §2 Domain + DNS — subdomain map, parent-zone confirm §3 Email backend — SES domain verify + receipt rule + inbound S3 bucket creation §4 IAM users + roles — agentkeys-daemon + agentkeys-data-role + per-data-class vault/memory roles §5 Initial bucket policy — static-IAM variant (pre-OIDC) §6 Instance profile — agentkeys-broker-host (EC2 optional) §7 Security audit — strip legacy over-broad attached policies (`AmazonS3FullAccess` checklist from the pre-condense §3.4a) §8 Cloud-provider port — AWS / AliCloud / GCP / Tencent Cloud 1:1 mapping table + migration playbook Restores the operational depth (DKIM bulk-record bash, daemon user create, role trust shape, broker-host instance profile, security audit) that the previous condense pass removed. Adds the portability framing (concept first, AWS-specific commands as ONE implementation) so the doc is the durable reference for non-AWS deployments. docs/cloud-setup.md — UPDATE, 314 → 202 lines Refocus on what comes AFTER bootstrap: OIDC federation activation (§1, was §4) + the setup-broker-host.sh runtime entry point (§2, was §5) + cleanup (§3, was §6). Drop the duplicate §1-§3 prereqs; add a clear cross-ref to cloud-bootstrap.md at the top. Section numbers renumbered. wiki/cloud-setup-faq.md — minor header tweak The FAQ now covers both cloud-bootstrap.md and cloud-setup.md (operators hit the same gotchas across both phases). Constraints applied: - Concise: every doc still fits in a few screens (bootstrap is longest at 365 lines because it carries the actual provisioning commands; cloud-setup.md is now 202 lines, down from 970 originally). - Idempotent: every flow uses the existing idempotent helper scripts. - No project credentials exposed: same placeholder convention as the prior pass (${ACCOUNT_ID}, ${ZONE}, etc.). Verified via grep. All internal links verified (python url-walker). * ops: setup-cloud.sh — idempotent cloud-account bootstrap orchestrator Closes the gap operators hit on a fresh EC2: ci-setup.md / cloud-bootstrap.md referenced \$ZONE, \$PARENT_ZONE_ID, and a dozen other identifiers without saying where they come from, and the cloud-side first-time provisioning was scattered across half a dozen helper scripts with no orchestrator. What changed: scripts/setup-cloud.sh — NEW, 523 lines, 14 idempotent steps: 1. tool sanity-check (aws/jq/curl/openssl/awk/sed) 2. source operator-workstation.env + validate required keys (ACCOUNT_ID, REGION, ZONE, PARENT_ZONE_ID, BROKER_HOST, MAIL_DOMAIN, BUCKET) — dies with precise pointer if missing 3. AWS caller + parent zone validation (case-insensitive admin match) 4. allocate/reuse Elastic IP (tag: agentkeys-broker-eip) + persist to env file; attach to INSTANCE_ID if provided 5. SES domain identity (create-email-identity, idempotent) 6. bulk DNS UPSERT: 3 DKIM CNAMEs + SPF TXT + DMARC TXT + MX + 6 A records (broker + signer + audit + email + cred + memory → EIP) in one change-batch 7. mail bucket + public-access-block + 30-day inbound/ lifecycle 8. SES receipt rule (create + activate, both pre-checked) 9. SES sender verification (delegates to ses-verify-sender.sh) 10. IAM user agentkeys-daemon + access key (minted ONCE only; printed for operator to save to secret manager) 11. IAM role agentkeys-data-role (static-IAM trust variant; federated swap is in cloud-setup.md §1 once broker is reachable) 12. per-data-class buckets + roles (delegates to existing provision-{vault,memory}-{bucket,role}.sh + apply-*.sh) 13. initial mail bucket policy (SES write + daemon read) 14. summary + next steps Idempotency claims per CLAUDE.md "Idempotent remote-setup rule": every mutation pre-checks state; each step outputs one of `ok proceeding` / `skip <reason>` / `fail <reason>`. Flags: --yes / --dry-run / --from-step N / --to-step N / --only-step N. AGENTKEYS_TEST=1 adds "-test" suffix to identifiers (for the CI parallel test environment). scripts/operator-workstation.env — add ZONE + PARENT_ZONE_ID with inline discovery instructions. The two vars were referenced everywhere but defined nowhere. docs/cloud-bootstrap.md — front-load the one-shot setup-cloud.sh invocation in TL;DR; add a required-env table so operators see exactly which keys to fill into operator-workstation.env before running the script. docs/ci-setup.md — add "Where things run" section answering the common "does the GH runner host the services?" question (no — runner is operator only; broker EC2 hosts everything, same pattern as local dev). Collapse the 35-line "provision parallel AWS resources" block in step 1 down to a 3-line `AGENTKEYS_TEST=1 bash setup-cloud.sh` invocation since the orchestrator handles it. Verification (live): $ AWS_PROFILE=agentkeys-admin bash scripts/setup-cloud.sh --from-step 1 --to-step 3 ok tools present ok env sourced — ACCOUNT_ID=… REGION=us-east-1 ZONE=litentry.org ok caller: arn:aws:iam::…:user/agentKeys-admin ok parent zone: litentry.org. All internal doc links verified via python url-walker. * ops: restructure setup/env/docs along the 4×2 prod/test matrix Per operator request, collapse the cloud + chain + CI setup artifacts into a single coherent matrix: ENV FILES (4): - scripts/operator-workstation.env prod operator (existing) - scripts/operator-workstation.test.env test operator (NEW; -test names) - scripts/broker.env prod broker (existing) - scripts/broker.test.env test broker (NEW; -test names) BOOTSTRAP SCRIPTS (2 — local-operator + CI fold into one): - scripts/setup-cloud.sh bootstrap broker cloud-side resources (SES + S3 + IAM + DNS + EIP). Accepts --env-file to point at the test env file; AGENTKEYS_TEST=1 (or *test* in env-file path) auto-suffixes IAM identifiers with -test so prod + test never share trust policies. - scripts/setup-dev-env.sh bootstrap operator workstation tooling (rustup + node + jq + aws + jj + build). Same script works on local laptop AND on the GH Actions runner. BROKER-SETUP (1): - scripts/setup-broker-host.sh single idempotent broker bring-up, already parameterized for prod vs test via --issuer-url / --account-id / --signer-host / etc. HARNESS (1): - harness/run.sh NEW unified runner wrapping v2-stage{1,2,3}-demo.sh. Accepts --env-file, --stage {1,2,3,all}, --chain {heima,heima-paseo,anvil}, --webauthn. Auto-detects test mode from env-file path. Per-stage scripts stay callable directly for surgical re-runs. DOCS (4 operator-facing): - docs/cloud-bootstrap.md cloud-side bootstrap for both prod + test. Now covers the FULL cloud lifecycle: §§1-8 first-time bootstrap (existing), §9 OIDC federation activation (folded from cloud-setup.md), §10 broker host bring-up via setup-broker-host.sh (folded), §11 teardown (folded). - docs/dev-setup.md operator workstation setup (existing). - docs/ci-setup.md CI activation (existing; refs updated). - docs/chain-setup.md NEW (renamed + generalized from docs/heima-setup.md). Works for all EVM chains, not just Heima — adds Anvil / Ethereum / Base / Sepolia matrix row, documents the (deployer, nonce) trick for parallel test contracts on mainnet, keeps Heima as the primary example. DELETED: - docs/cloud-setup.md content folded into docs/cloud-bootstrap.md §§9-11 - docs/heima-setup.md renamed to docs/chain-setup.md (generalized) CROSS-REF SWEEP: Repo-wide sed: heima-setup.md → chain-setup.md, cloud-setup.md → cloud-bootstrap.md across docs/ + wiki/. Self-references in cloud-bootstrap.md that the blanket rewrite made circular were individually fixed to point at the right §N anchors of the merged doc. VERIFICATION: - setup-cloud.sh syntax OK; live test step 1-3 against prod env: ok tools present ok env sourced — ACCOUNT_ID=… REGION=us-east-1 ZONE=litentry.org ok caller: arn:aws:iam::…:user/agentKeys-admin ok parent zone: litentry.org. - setup-cloud.sh with test env file: ok env sourced from scripts/operator-workstation.test.env - harness/run.sh --help renders cleanly - No project credentials in any new file (grep clean for AKIA-prefix, sk-prefix, BEGIN PRIVATE KEY) - All internal doc links resolve (python url-walker) — one pre-existing broken link to ./stage7-wip.md in dev-setup.md is unrelated rot, not introduced here. * docs(cloud-bootstrap): env-files reference for all 4 files + CI; explicit IAM isolation matrix Addresses two operator gaps in the prior pass: 1. ENV FILES REFERENCE — was only "Required env (operator-workstation.env)". Now covers all 4 env files + the CI runner pattern: - scripts/operator-workstation.env (prod operator laptop) - scripts/operator-workstation.test.env (test operator laptop) - scripts/broker.env (prod broker /etc/agentkeys/) - scripts/broker.test.env (test broker /etc/agentkeys/) - GH Actions runner (no checked-in file — materializes operator env inline at job start from TEST_* secrets; full mapping table) Side-by-side prod-vs-test tables for operator env + broker env so operators can spot the exact identifier deltas at a glance. 2. IAM ISOLATION MATRIX — new §0.1 makes test-vs-prod isolation explicit. Per-resource mapping (IAM user / data role / vault role / memory role / OIDC provider / EIP / 3 buckets / SES sender / 6 contract addresses) showing prod name vs test name vs which script creates it. Documents the cross-trust enforcement chain: - OIDC provider URL is the trust scope (byte-for-byte distinct ARNs for broker.${ZONE} vs test-broker.${ZONE}) - PrincipalTag scoping (§9.4) is the secondary defense - Per-data-class bucket separation is the tertiary defense VERIFICATION re wiki/ci-setup-faq.md: The file IS on origin (blob b8af0d3). The link `[wiki/ci-setup-faq.md](../wiki/ci-setup-faq.md)` from docs/ci-setup.md resolves both locally and on the remote. Confirmed via `git ls-tree -r origin/claude/romantic-ardinghelli-34d7a7 -- wiki/`. No stale refs to cloud-setup.md or heima-setup.md anywhere — both the python url-walker and `grep -rn` are clean. The one remaining stale link (dev-setup.md → ./stage7-wip.md) is pre-existing rot from before this PR, unrelated. * docs(cloud-bootstrap): §0.1 manual prereqs — Route 53 zone + EC2 + EIP workflows Two operator actions weren't documented anywhere even though they gate every downstream step: 1. GETTING THE IP FOR THE TEST MACHINE Two workflows now spelled out: - Workflow A (recommended): EC2-first, then `INSTANCE_ID=<id> setup-cloud.sh` allocates EIP + attaches + persists to env in one shot. - Workflow B: EIP-first via `setup-cloud.sh`, then launch EC2, then `aws ec2 associate-address` manually. Both work for prod and test (test uses `--env-file scripts/operator-workstation.test.env` so the EIP gets tagged `agentkeys-broker-eip-test`). 2. BINDING THE DOMAIN WITH ROUTE 53 Was only validated (step 3 calls `route53 get-hosted-zone`), never created. New §0.1 covers: - aws route53 create-hosted-zone for $ZONE - Looking up PARENT_ZONE_ID via list-hosted-zones (with the /hostedzone/ prefix strip) - Copying the 4 NS records into the registrar's DNS settings - dig verification of delegation propagation - Non-Route 53 DNS providers — explicit "skip step 6, replicate 12 records manually" path so the doc isn't AWS-locked PLUS the implicit third prereq: - agentkeys-admin AWS profile — long-lived IAM user with full IAM/S3/SES/Route53 access. Pre-existing per CLAUDE.md "AWS local-profile ↔ remote-IAM mapping". Bootstrap doesn't auto- create it (root creds on disk = bad). Section placement: - Pre-existing §0.1 (IAM isolation matrix) → renumbered §0.2 - New §0.1 lives where it's read top-to-bottom: after the env-file reference (so operator knows what they're filling in) but before §1 (Identities — the actual bootstrap content). Verification: - python url-walker clean (only pre-existing dev-setup.md → ./stage7-wip.md is broken; not introduced here). - Section anchors §0.1/§0.2/§1..§11 unique + sequential. - +97/-1 lines (cloud-bootstrap.md → 711 lines). * ops(setup-cloud): --test flag (was env var); env file is source of truth for EIP + INSTANCE_ID Per operator feedback — the EIP / INSTANCE_ID / AGENTKEYS_TEST settings were documented as shell env vars, which is muddy: operator has to re-export per shell, and test-vs-prod selection isn't a CLI affordance. CHANGES: 1. `--test` CLI flag (new) Replaces the AGENTKEYS_TEST=1 env-var pattern. Explicit > magic. Auto-detection from env-file path containing "test" stays as an ergonomic shortcut for the conventional naming (scripts/operator-workstation.test.env) — explicit --test wins when both apply, and works with non-standard env-file names. 2. EIP + INSTANCE_ID move from "optional shell env" → env file Both are per-deployment identifiers — they belong in the env file next to ACCOUNT_ID, BROKER_HOST, etc. The script writes EIP back to the env file after allocation (step 4), and reads INSTANCE_ID from the env file to decide whether to attach. Placeholder lines (commented out) added to both operator-workstation.env and operator-workstation.test.env so operators see exactly where to paste: # INSTANCE_ID=i-0123456789abcdef0 # EIP= 3. ZONE_SUFFIX removed from docs — was never referenced in script body, dead doc. 4. The "INSTANCE_ID unset" warn message now tells the operator the exact one-liner to re-run after the env file edit: "Paste 'INSTANCE_ID=i-…' into the env file once EC2 exists, then re-run: bash setup-cloud.sh --env-file <path> --only-step 4" 5. cloud-bootstrap.md §0.1 Workflow A/B updated to the env-file-driven pattern. Workflow A now reads: 1. aws ec2 run-instances → note INSTANCE_ID 2. echo 'INSTANCE_ID=<id>' >> scripts/operator-workstation.env 3. bash scripts/setup-cloud.sh --yes 4. SSH using $(grep ^EIP= ...) Test stack: same pattern with --env-file scripts/operator-workstation.test.env --test. 6. ci-setup.md updated to invoke setup-cloud.sh with --test + --env-file (was AGENTKEYS_TEST=1 env var). VERIFIED (live): Case A: --test + prod env file → SUFFIX="-test" (flag overrides path) Case B: no flag + test env file → SUFFIX="-test" (auto-detect) Case C: no flag + prod env file → SUFFIX="" (neutral) All three smoke-tested through step 2 (read-only env source). bash -n scripts/setup-cloud.sh → clean. grep -rn AGENTKEYS_TEST docs/ scripts/ → empty (no leftover refs). python url-walker on all 7 operator-facing docs → clean (only pre-existing dev-setup.md → ./stage7-wip.md rot, not introduced here). DIFF: +71 / -34 across 5 files (script + 2 env files + 2 docs). * ops(setup-cloud + cloud-bootstrap): fix step-14 summary + TL;DR for env-file pattern Three concrete updates to land the --test/--env-file refactor end-to-end in the docs and the script's own output: 1. cloud-bootstrap.md TL;DR rewritten for the env-file-driven workflow - was: launch EC2 → setup-cloud → aws ec2 associate-address by hand - now: launch EC2 → paste 'INSTANCE_ID=i-…' into the env file → setup-cloud allocates EIP + attaches automatically - test stack: explicit "swap in --env-file scripts/operator-workstation.test.env --test" example so operators don't have to figure it out 2. setup-cloud.sh step 14 summary rewritten - was: hardcoded "agentkeys-data-role" string (wrong for --test) - now: prints $DAEMON_USER + $DATA_ROLE (suffix-aware) - was: stale ref to docs/cloud-setup.md §1 (DELETED doc!) - now: docs/cloud-bootstrap.md §9 (correct target — OIDC federation section in the folded-together doc) - was: generic "aws ec2 associate-address" instruction - now: precise "paste INSTANCE_ID into $ENV_FILE, re-run with --env-file $ENV_FILE [--test] --only-step 4" - new "Env file" and "Test mode" lines at the top of the summary so operator sees at a glance which mode they ran in 3. setup-cloud.sh — source $ENV_FILE unconditionally before main() - regression caught by live smoke: --only-step 14 was skipping step 2 (env source), then crashing at line 506 with "$ZONE: unbound variable" because set -u catches the unset. - fix: one line at the top after CLI parse: [ -f "$ENV_FILE" ] && { set -a; . "$ENV_FILE"; set +a; } Step 2's do_step_2 still runs when in scope (validates + prints "env sourced —..."); the unconditional source just makes --only-step N ergonomic for any N > 2. VERIFIED (live, three smoke runs in parallel): $ bash setup-cloud.sh --env-file …operator-workstation.env --only-step 14 Env file: scripts/operator-workstation.env Test mode: no (prod) Daemon user: agentkeys-daemon Data role: arn:aws:iam::…:role/agentkeys-data-role → next-steps point at scripts/operator-workstation.env $ bash setup-cloud.sh --env-file …operator-workstation.test.env --test --only-step 14 Env file: scripts/operator-workstation.test.env Test mode: yes (-test suffix on IAM identifiers) Daemon user: agentkeys-daemon-test Data role: arn:aws:iam::…:role/agentkeys-data-role-test → next-steps point at …test.env + include --test flag $ bash setup-cloud.sh --env-file …operator-workstation.env --from-step 1 --to-step 3 All three steps still pass (no double-source regression). grep -rn "cloud-setup.md\|heima-setup.md\|AGENTKEYS_TEST\|ZONE_SUFFIX" docs/ scripts/ wiki/ → empty. All stale refs to deleted docs + the old env-var pattern are gone. * ops(setup-cloud step 4): idempotent EIP adoption for "I already have EC2 + EIP" — path A Closes the operator-flagged hole: the prior step-4 logic checked for a tagged EIP and an EIP= env var, but if neither matched (e.g. EC2 was provisioned manually via Console with an EIP that the script never tagged), step 4 would silently allocate a FRESH EIP — wasted resource + the wrong public IP propagating into DNS in step 6. NEW PRECEDENCE LADDER (step 4, first-match wins): A. INSTANCE_ID has an EIP attached → adopt it (no allocate, no re-associate; retroactively tag for future idempotency) B. Tagged EIP exists in account → reuse (existing logic) C. EIP= set in env file → use it (existing logic) D. Allocate fresh → allocate-address + tag Path A is the new branch. It runs FIRST, so the operator's pre-existing EC2+EIP setup short-circuits the entire allocate-and-attach flow. The retroactive tag means re-runs without INSTANCE_ID set also resolve via path B. Path A's logic (`scripts/setup-cloud.sh:do_step_4`): 1. `aws ec2 describe-instances --instance-ids $INSTANCE_ID` → public IP. 2. Confirm it's a static EIP (has AllocationId) — auto-assigned public IPs that disappear on stop/start are skipped, fall through to B/D. 3. `EIP=<that-ip>`, `env_set EIP "$EIP"`, retroactive `aws ec2 create-tags` (best-effort; warn if tag write fails, operator-runnable by hand). 4. `return` from step 4. No allocate. No associate (already attached). Doc — cloud-bootstrap.md §0.1 Manual prereqs "Getting the IP — three workflows" (was two): added Workflow 0 covering this exact path, with the precise commands: 1. Discover INSTANCE_ID via `aws ec2 describe-instances --filters ip-address` 2. `echo 'INSTANCE_ID=i-…' >> scripts/operator-workstation.env` 3. `bash scripts/setup-cloud.sh --yes` The expected output ("skip EIP <ip> already attached... tagged existing EIP as agentkeys-broker-eip") is shown verbatim so the operator recognizes the no-op path. VERIFIED (parallel smoke): - bash -n scripts/setup-cloud.sh → SYNTAX OK - path B/C regression (no INSTANCE_ID, prod env): "skip EIP 54.x.x.x provided via env file; not allocating" + the "INSTANCE_ID unset" warn pointing operator at the env file edit + re-run command - path A simulation (fake INSTANCE_ID + --dry-run): falls through to B/C cleanly (no error, no allocate fires) - python url-walker on cloud-bootstrap.md: clean DIFF: +71 / −5 across 3 files (script + env file mtime + doc). * ops: EIP+INSTANCE_ID live in broker.env (not operator-workstation.env); add ssh-broker.sh helper Per operator feedback: EIP and INSTANCE_ID identify the BROKER MACHINE, not operator-account identifiers, so they belong in the broker-machine env file (scripts/broker.env / broker.test.env) — same place as BROKER_OIDC_ISSUER, BROKER_DATA_ROLE_ARN, etc. ENV FILE REFACTOR - scripts/broker.env — operator pasted INSTANCE_ID + EIP at the top (prod broker machine identifiers) - scripts/broker.test.env — operator pasted test EC2 INSTANCE_ID + EIP - operator-workstation.env — removed the EIP=… line + the INSTANCE_ID placeholder comment block (those values live in broker.env now) - operator-workstation.test.env — same cleanup; brief comment pointing readers at broker.test.env SCRIPT — scripts/setup-cloud.sh 1. New CLI flag: --broker-env-file <path> Default: scripts/broker.env (prod) or scripts/broker.test.env (test). Resolved post-CLI based on TEST_MODE. 2. Source order: operator env first, then broker env (so step 4 reads INSTANCE_ID / EIP from broker.env after operator vars are bound). 3. env_set() now takes an optional 3rd arg = target file path. Default stays $ENV_FILE for backwards-compat. Step 4's EIP write uses env_set EIP "$EIP" "$BROKER_ENV_FILE" so the EIP persists in the broker file where it conceptually belongs. 4. Step 14 summary prints BOTH env file paths up top + the new INSTANCE_ID warn message points operator at the broker file. NEW HELPER — scripts/ssh-broker.sh (83 lines) Single SSH entry point replacing per-operator shell aliases. Reads INSTANCE_ID + EIP from the right broker env file dynamically — when the EC2 is replaced and broker.env is updated, the script picks it up with no shell-config edit. Usage: bash scripts/ssh-broker.sh # prod via EC2 Instance Connect bash scripts/ssh-broker.sh test # test via EC2 Instance Connect bash scripts/ssh-broker.sh prod --fallback # raw SSH + .pem bash scripts/ssh-broker.sh test --fallback Default AWS profiles per stack (least-privilege; per CLAUDE.md "AWS local-profile ↔ remote-IAM mapping"): prod → agentkeys-broker test → agentkeys-broker-test DOC — docs/cloud-bootstrap.md §0.1 New "#### 2a. SSH into the broker host" subsection covers: - The ssh-broker.sh entry point + its 4 invocation modes - Default profile table (prod vs test) - One-shot create-user recipe for agentkeys-broker-test (not in setup-cloud.sh because it's an operator-facing SSH credential, not a data-plane principal) - Shell wrapper aliases (alias ssh-prod=… / ssh-test=…) Explicit note re: agentkeys-daemon-test — already auto-created by setup-cloud.sh step 10 when --test is passed; used for the pre-OIDC-federation bootstrap window + as a fallback. VERIFIED (live): $ bash scripts/setup-cloud.sh --env-file …/operator-workstation.env --only-step 14 Operator env file : scripts/operator-workstation.env Broker env file : scripts/broker.env EIP : 54.164.117.252 EIP attached to : i-0c0b739bd35643fd3 ← sourced from broker.env Next steps: SSH into 54.164.117.252 … ← no "unattached" warn $ bash scripts/ssh-broker.sh --help → renders cleanly $ bash -n scripts/setup-cloud.sh scripts/ssh-broker.sh → SYNTAX OK $ python url-walker docs/cloud-bootstrap.md → LINK CHECK OK DIFF: +132 / −61 across 7 files. * ops(setup-cloud): step 12 — idempotent SSH-user (agentkeys-broker[-test]) provisioning Per operator feedback: the agentkeys-broker / agentkeys-broker-test IAM user creation belongs in the idempotent orchestrator, not in ~/.zshrc or in a copy-paste recipe in the doc. NEW STEP 12: IAM user $SSH_USER (operator SSH via EC2 Instance Connect) Created users: prod → agentkeys-broker (suffix="" when no --test) test → agentkeys-broker-test (suffix="-test" via --test) Idempotent shape (matches step 10's daemon-user pattern): 1. INSTANCE_ID precheck — must be set in $BROKER_ENV_FILE; otherwise step skips with a pointer to paste it + re-run --only-step 12. 2. `aws iam get-user` — if exists, skip create-user. 3. `aws iam put-user-policy` — idempotent overwrite of the inline grant: ec2-instance-connect:SendSSHPublicKey scoped to the broker's INSTANCE_ID ARN, with Condition ec2:osuser=agentkey; plus ec2:DescribeInstances + ec2:DescribeInstanceConnectEndpoints for the AWS CLI to resolve instance metadata. 4. `aws iam list-access-keys` — if any active key exists, skip; otherwise mint ONCE + print the secret with paste-ready ~/.aws/credentials block. Operator NEVER needs to hand-edit IAM, shell config, or runbook. Re-runs are no-ops once user + policy + access key exist. When the EC2 is replaced (different INSTANCE_ID): operator pastes the new INSTANCE_ID into broker.env / broker.test.env, re-runs `--only-step 12` → put-user-policy overwrites the inline grant with the new resource ARN. Old grant gone, new grant active. RENUMBER (steps 12→15 shifted by +1): Step 12 was per-data-class → now SSH user (NEW) Step 13 was bucket policy → now per-data-class Step 14 was summary → now bucket policy Step 15 (new) → summary STEP_TOTAL: 14 → 15. TO_STEP default: 14 → 15. Header docstring idempotency claims block updated. main() dispatch updated. Summary's "re-run any step surgically" hint lists step 12 (re-create SSH user, e.g. after EC2 replace) + step 13 (re-run per-data-class) — the previously-broken "only-step 12 # per-data-class" hint is now correct. DOC — docs/cloud-bootstrap.md §0.1 #### 2a Replaced the 20-line manual `aws iam create-user` recipe with the one-line script invocation: AWS_PROFILE=agentkeys-admin bash scripts/setup-cloud.sh \ --env-file scripts/operator-workstation.test.env --test --only-step 12 Same recipe for prod (no --env-file, no --test). Script prints the access key once → operator pastes into ~/.aws/credentials by hand. Shell-config edits (~/.zshrc / ~/.zshenv) stay operator-owned — the script never touches those. VERIFIED (parallel): $ bash -n scripts/setup-cloud.sh → SYNTAX OK $ grep -cE 'do_step_1[2-5]\(\)' … → 4 (correct count) $ grep -cE 'in_scope (12|13|14|15) && …' … → 4 (correct dispatch) $ … --only-step 15 → "[step 15/15] Summary" "EIP attached to: i-0c0b…" Re-run hints reference step 12 + 13 correctly. $ … --only-step 12 --dry-run → "[step 12/15] IAM user agentkeys-broker … DRY: would create-user" $ … --only-step 12 --test --dry-run → "agentkeys-broker-test" $ python url-walker docs/ → LINK CHECK OK (only the pre-existing dev-setup.md → ./stage7-wip.md rot remains, unrelated) * fix(setup-cloud): --test auto-switches env-file + ANSI-C quote color vars Two operator-flagged bugs after running setup-cloud.sh --test: BUG 1 — Half-test trap `--test` alone (no --env-file) suffixed IAM identifiers with -test but kept BROKER_HOST=broker.litentry.org and MAIL_DOMAIN=bots.litentry.org from the prod env file. Summary showed prod hostnames + test IAM names → operator saw a half-test config. Fix: when --test is passed AND --env-file is still the prod default, auto-switch ENV_FILE to scripts/operator-workstation.test.env. A bare `bash setup-cloud.sh --test` now produces an end-to-end test invocation (hostnames + buckets + IAM all -test). Verified live: $ bash setup-cloud.sh --test --only-step 15 Operator env file : scripts/operator-workstation.test.env Broker env file : scripts/broker.test.env Test mode : yes (-test suffix on IAM identifiers) Mail domain : bots-test.litentry.org Broker host : test-broker.litentry.org Mail bucket : s3://agentkeys-mail-test-429071895007/ Daemon user : agentkeys-daemon-test Data role : arn:aws:iam::…:role/agentkeys-data-role-test Next: ssh into 3.214.219.209 (test EIP) + broker-host with --issuer-url https://test-broker.litentry.org. BUG 2 — Literal "\033[1m" in step 10 + step 12 access-key prompts The color vars were single-quoted ('\033[1m') so they held the literal six-char escape string, not the ESC byte. `printf "%s"` substitution printed the literal — operator saw "\033[1m" instead of bold. Format-string interpolation ("${COLOR_HEAD}...") worked because printf interprets backslash escapes in the format string. But the access-key prompts in step 10 (daemon user) + step 12 (SSH user) use "%s" for the color arg, so the literal leaked. Fix: ANSI-C quote at definition ($'\033[1m'). Now the var contains the actual ESC byte; both format-string and %s interpolation render bold. No printf-call changes needed. * fix(setup-cloud): step 6 + 8 — prod/test DNS + receipt-rule collisions Two critical bugs discovered when --test step 9 (SES…
…104) * docs+comments: fold back /v1/mint-aws-creds retirement (closes #72) The route + handler + tests were deleted in PR #96, but four downstream spots still described it as a live endpoint with current behavior. Land the doc + comment fixes so the next operator running the runbooks does not curl a 404. - docs/dev-setup.md:110 — describe the actual mint flow (OIDC JWT + client-side STS) instead of the deleted server-side aggregator. - crates/agentkeys-broker-server/src/env.rs — drop the stale "(broker-internal, used by /v1/mint-aws-creds)" parenthetical on the SessionJwt env group; name the current consumers (email-link / OAuth2 mint paths + /v1/mint-oidc-jwt). - crates/agentkeys-broker-server/src/main.rs — drop the stale comment about mint_v2 mirroring rows into the audit log (mint_v2 was deleted in PR #96); name /v1/mint-oidc-jwt as the current writer. - docs/operator-runbook-stage7.md — collapse the "two endpoints" § framing into the single surviving path. The old request label in the ASCII trust-relationship diagram now reads /v1/mint-oidc-jwt; the whole "POST /v1/mint-aws-creds — server-side gated" subsection is replaced with a one-paragraph retirement callout so operators searching for the old path find a clear "this is gone" note. - docs/stage7-demo-and-verification.md — drop "two paths" framing in §5, delete §5.2 (the server-side aggregator deep-dive that pointed into a deleted handlers/mint.rs and a deleted tests/mint_v2_flow.rs), rewrite §12.2 Idempotency-Key to explain the dedup layer is gone with the route (JWT TTL + daemon JWT cache cover the same use case), update the future-work bullet from "Retire /v1/mint-aws-creds entirely" to "✅ Done in PR #96", and rewrite §16's audit-trail note to say the broker-side row of the actual mint no longer exists (AWS CloudTrail is the STS-side trail). Per CLAUDE.md Runbook-fix-fold-back + Land-the-fix policies: PR #96 shipped the code work but did not touch these descriptive doc sections, so the operator-facing runbooks would still send the next person reading them to a 404. This patch closes that gap and explicitly closes #72 (GitHub did not auto-close from PR #96's body). cargo build -p agentkeys-broker-server clean (34s, exit 0). * docs+comments: address codex challenge findings on PR #104 Codex adversarial review (via /codex challenge) caught 5 categories of real defects the first commit on this branch missed. All P1s + P2s addressed here: [P1 #1] operator-runbook-stage7.md still described deleted endpoint behavior as live in two sections the first audit missed: - §L540-557 "Migration window — implicit-grant fallback" pointed operators at src/handlers/mint.rs::mint_v2 (deleted) and described a Phase E flip (BROKER_REQUIRE_EXPLICIT_GRANT=true) that no longer has a consumption point. Replaced with a "Grant enforcement retired" callout noting grant CRUD endpoints remain (so masters can still manage grants for audit / future re-introduction) but the mint-time try_consume path is gone. - §L698-707 "Idempotency-Key" claimed the mint endpoint accepts the header and dedups bodies within a 5min window. /v1/mint-oidc-jwt does not honor Idempotency-Key — replaced with a retirement note pointing at BROKER_OIDC_JWT_TTL_SECONDS=300 as the only re-mint cost knob. [P1 #2] My §12.2 rewrite in stage7-demo-and-verification.md invented a "daemon caches the JWT in-process" claim that does not match crates/agentkeys-provisioner/src/aws_creds.rs::fetch_via_broker, which fetches a fresh OIDC JWT and assumes a fresh role every call (no cache layer). Replaced with the truth: clients must implement batching / dedup / rate-limiting themselves, with a code reference for verification. [P1 #3] docs/spec/plans/issue-64/PLAN.md (and the prd.json next to it) still describe /v1/mint-aws-creds + Phase B grant try_consume + Idempotency-Key as live with passing acceptance criteria. Added a retirement preamble at the top of PLAN.md flagging that the route + gates were deleted in PR #96 and pointing readers at arch.md §17.2 for the current isolation contract. The prd.json acceptance entries are left as-is to preserve the audit record — the preamble + this commit message are the durable "this no longer matches reality" signal. [P2 #4] Code-comment cleanup: - state.rs:42-46 (grant_store) — re-cast from "the mint endpoint consults this" to "backs the /v1/grant/* CRUD endpoints; mint-time try_consume gone with mint_v2" - state.rs:52-55 (idempotency_store) — note that the only consumer is gone and the field is slated for removal (follow-up task spawned via mcp__ccd_session__spawn_task — see "Remove dead IdempotencyStore code"). - aws_creds.rs:32 — clarify the AwsTempCreds field shape matched the legacy /v1/mint-aws-creds response, which is now deleted. - aws_creds.rs::build_session_name doc comment + matches_broker_format test comment — drop references to handlers/mint.rs::build_session_name (deleted); reframe as daemon-side-only. - tests/grant_flow.rs module doc — drop the "covered in mint_v2_flow separately" claim (mint_v2_flow.rs is deleted); note CRUD-only surface today. - tests/oidc_flow.rs:181 — drop the "(parity with /v1/mint-aws-creds)" parenthetical. [P2 #5] PrincipalTag terminology drift between operator runbook and arch.md §17.2. Runbook said creds are tagged with `agentkeys_user_wallet`, but §17.2's per-actor isolation invariant is `agentkeys_actor_omni`. Code (oidc.rs:181) emits both for v0.1 bucket-policy back-compat. Rewrote runbook to lead with the §17.2-canonical tag and note the legacy tag stays for back-compat — per the "Terminology-source-of-truth rule" in CLAUDE.md. Also: a second pass of repo-wide grep found 3 more stale references outside the first commit's blast radius: - aws_creds.rs:32 field-shape comment (fixed) - tests/grant_flow.rs + tests/oidc_flow.rs comments (fixed) - docs/spec/plans/issue-74-dev-key-service-plan.md ASCII diagram showing /v1/mint-aws-creds as a live arrow (fixed inline + small retirement note) What stays (intentionally): - docs/spec/plans/development-stages.md:23 — historical "Stage 7 phase 1 (2026-04)" table entry. Date-anchored historical record, accurate at that stage. Not rewritten. - docs/archived/**, docs/research/**, docs/spec/plans/issue-64/*.md (other than PLAN.md preamble), progress.txt — pre-existing historical / scratch content, not operator-facing. Build still clean (cargo build -p agentkeys-broker-server -p agentkeys-provisioner, exit 0). Test suite unchanged in behavior — all edits to test files are comments only.
Issue #72 / PR #96 deleted POST /v1/mint-aws-creds and the crates/agentkeys-broker-server/src/handlers/mint.rs handler, which was the only production consumer of IdempotencyStore. The store remained wired through boot.rs -> AppState but no live code path read or wrote through it. Removed: - crates/agentkeys-broker-server/src/storage/idempotency.rs (the store + tests). - pub mod / pub use lines in storage/mod.rs. - idempotency_store field on AppState (state.rs). - idempotency_store field on BootArtifacts + open() block + idempotency_path() helper in boot.rs. - Assignment in main.rs AppState constructor. - agentkeys_broker_idempotency_hits / _conflicts AtomicU64 counters and their /metrics array entries (no live path bumped them); test assertion for help/type-line count updated from 10 to 8. - IdempotencyStore::open_in_memory() boilerplate in six integration tests. - Idempotency-Key sub-section + bullet in docs/operator-runbook-stage7.md and docs/stage7-demo-and-verification.md (only the parts that documented the removed metric counters + dedup feature; other /v1/mint-aws-creds doc residue from PR #96 stays for a separate doc-cleanup PR). cargo build + cargo test -p agentkeys-broker-server + cargo clippy -p agentkeys-broker-server -- -D warnings all exit 0.
* issue #101: path-conditional auto-deploy of test broker via SSM Adds two new harness-ci.yml jobs that re-deploy the test broker EC2 when a PR touches broker-affecting paths, so harness-e2e validates the PR's actual broker code instead of whatever stale binary the EC2 happens to be running. - detect-changes (dorny/paths-filter@v3) computes broker_changed - deploy-test-broker assumes a new OIDC role and drives setup-broker-host.sh --test --yes on the EC2 via aws ssm send-command - scripts/provision-ci-deploy-role.sh provisions the IAM role with a trust policy scoped to repo:litentry/agentKeys:* and an inline policy scoped to one EC2 instance ARN (separation of duties from the existing TEST_OIDC_AWS_ROLE_ARN e2e role) - harness-e2e now runs AFTER deploy-test-broker (deviation from the issue's `needs: harness-e2e` spec, documented inline) so broker bugs introduced by a PR fail that PR's harness — not the next one's Auto-deploy is fully opt-in: skipped silently unless both OIDC_AWS_ROLE_ARN_DEPLOY and TEST_BROKER_INSTANCE_ID secrets are set. A workflow_dispatch input force_deploy_broker enables dry-run validation without a broker-path change. Out of scope for this PR (rollout plan step 7 in issue #101): auto-deploy of the test Heima EVM contracts. Defers to a follow-up because it needs the SECRETS_REWRITE_PAT token to update six TEST_*_ADDRESS_HEIMA secrets after each redeploy. Prod broker auto-deploy stays explicitly out of scope per CLAUDE.md "Remote broker host (single entry point)" — manual via bash scripts/setup-broker-host.sh --upgrade only. Docs: docs/ci-setup.md gains §7 with the provisioning recipe, secret list, dry-run procedure, and disarm path. * fix(provision-ci-deploy-role): strip non-ASCII from --description IAM CreateRole rejects descriptions outside [\t\n\r\x20-\x7e\xa1-\xff] with 'Value at description failed to satisfy constraint'. The em-dash in the original description string tripped this regex at provisioning time. Replace with an ASCII hyphen and add an inline warning comment so a future editor doesn't reintroduce Unicode here. Reported by operator running docs/ci-setup.md §7.1. * fix(provision-ci-deploy-role): --fix-ssm auto-attaches SSM policy + folds into runbook Operator hit the second failure mode in docs/ci-setup.md §7.1: the test broker EC2 was not registered with SSM (PingStatus=None), so the script exited before SendCommand could ever work. The fix had to be one round- trip per CLAUDE.md runbook-fix-fold-back policy: a sanity check upgrade that catches the same case for the next operator AND a manual override. Script changes: - New --fix-ssm flag. When passed AND PingStatus != Online, the script: 1. Looks up the EC2's IamInstanceProfile via DescribeInstances. 2. Walks profile -> role via iam:GetInstanceProfile. 3. Attaches arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore (idempotent — aws iam attach-role-policy no-ops on re-attach). 4. Polls describe-instance-information up to 18x (~3 min) waiting for the agent to refresh creds. 5. If still offline after 3 min: prints both manual escape hatches (ssh + systemctl restart amazon-ssm-agent, OR aws ec2 reboot-instances). - Without --fix-ssm: same diagnostic message as before, plus a one-line hint pointing at --fix-ssm. No IAM mutation; safe default. - Handles the edge case of an instance with NO instance profile at all: prints associate-iam-instance-profile command, exits 1. Docs (docs/ci-setup.md §7.1): - Standard invocation now includes --fix-ssm on the first run. - New SSM remediation table maps each failure mode to what --fix-ssm covers vs what the operator must still do by hand (agent restart, reboot, install agent, VPC endpoint). Reported by operator after re-running the em-dash-fixed script; PingStatus=None on i-0135a8b2c53d14941. * fix(provision-ci-deploy-role): unbound $sub_pattern in idempotent log line set -u tripped on the role-already-exists branch because the log line referenced $sub_pattern as a shell variable, but it only exists as a jq --arg inside the trust-policy heredoc. Replace with ${REPO_SLUG} which is a real shell var. Latent since the first commit; surfaced now that the previous em-dash fix let the operator reach this branch on re-run. * fix(provision-ci-deploy-role): --fix-ssm auto-creates instance profile when EC2 has none Operator re-ran with --fix-ssm; auto-remediation hit the third failure mode: the test broker EC2 has NO IAM instance profile attached at all. A common state on test brokers spun up by setup-cloud.sh --test — the broker process authenticates to AWS via static creds in /etc/agentkeys/broker.env, so an instance profile was never wired up. Script changes: - New create_and_associate_ssm_profile() called when DescribeInstances reports no IamInstanceProfile.Arn. Idempotent end-to-end: 1. iam get-role agentkeys-test-broker-ssm → create if missing (EC2 service trust policy, AmazonSSMManagedInstanceCore attached). 2. iam get-instance-profile agentkeys-test-broker-ssm → create if missing. 3. iam get-instance-profile (.Roles[0]) → add-role-to-instance-profile if empty; refuse to swap if the profile already holds a different role (operator must reconcile manually). 4. 15s sleep for IAM eventual consistency (per AWS docs). 5. ec2 describe-iam-instance-profile-associations → associate-iam-instance-profile if no existing association. - attach_ssm_managed_policy_if_missing() now dispatches to create_and_associate_ssm_profile() when no profile is present, instead of exiting 1 with manual instructions. Why this is safe to add to a running broker: - The broker app reads AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY from broker.env explicitly; static creds always win over IMDS-served creds. - Adding an IMDS instance profile cannot reduce capability — only the SSM agent (and not the broker app) will read from IMDS. Runbook fold-back (CLAUDE.md policy): docs/ci-setup.md §7.1 SSM remediation table now reads '(handled)' for the no-profile row, describing the dedicated role/profile that gets created. * fix(setup-broker-host): install amazon-ssm-agent at bootstrap (issue #101 root cause) Operator hit the SSM-agent-not-installed failure mode after --fix-ssm created + associated the instance profile: 'Unit amazon-ssm-agent.service not found.' Some Ubuntu AMIs downstream of the AWS Marketplace base ship without amazon-ssm-agent. Without the agent, no IAM policy on earth lets the EC2 register with SSM, and the CI auto-deploy (issue #101) hangs. Per CLAUDE.md "Runbook-fix-fold-back policy": the cure for an operator-encountered failure is to upgrade the script that owns the broken step, not the script that surfaces the symptom. setup-broker-host.sh is the canonical entry point for the broker EC2 — the SSM agent install belongs there. Script changes (scripts/setup-broker-host.sh): - Idempotent SSM-agent install block right after the ec2-instance-connect block (same shape: ssm_unit_active() pre-check, install only on miss). - Two install paths in priority order: 1. snap install amazon-ssm-agent --classic (AWS-blessed on Ubuntu 22.04+; unit: snap.amazon-ssm-agent.amazon-ssm-agent.service) 2. .deb fallback from https://s3.$REGION.amazonaws.com/amazon-ssm-$REGION/latest/ (older / non-snap images; unit: amazon-ssm-agent.service) - Both paths converge on ssm_unit_active() returning true; subsequent --upgrade re-runs skip after that. Runbook fold-back (docs/ci-setup.md §7.1): - 'SSM Agent not installed' row of the remediation table now points operators at setup-broker-host.sh --test --yes for the structural fix, with a snap one-liner for one-shot manual recovery. Reported by operator after re-running provision-ci-deploy-role.sh --fix-ssm: the script created the profile + associated it, but the 3-min poll timed out because no SSM agent was running on the EC2. * fix(provision-ci-deploy-role): distinguish AccessDenied from instance-not-registered The SSM verify block has been masking caller-permission gaps as 'instance not registered with SSM' (state=None) because of the 2>/dev/null || echo None silent fallback. Result: 4 rounds of phantom remediation against the EC2 (em-dash fix, --fix-ssm flag, auto-create instance profile, install amazon-ssm-agent on the EC2) — none of which were addressing the actual cause, which was that the operator's admin group lacks ssm:DescribeInstanceInformation. Fix: - Capture stderr into a tmpfile. - Grep for 'AccessDenied' specifically; on hit, die() with the exact one-liner the operator needs to attach AmazonSSMReadOnlyAccess to the AgentKeyAdmin group. - Empty stdout (no AccessDenied in stderr) = genuinely not registered; proceeds to the existing remediation paths. Diagnosed by running aws ssm describe-instance-information directly against i-0135a8b2c53d14941 as agentkeys-admin and seeing the AccessDenied that the script had been swallowing all along. Lesson (CLAUDE.md fold-back): when a sanity check uses 2>/dev/null, make sure the discarded stderr can't be the answer to the question the check is asking. * docs(ci-setup §7.3): require --ref on pre-merge gh workflow run dispatch Operator hit 'HTTP 422: Unexpected inputs provided: [force_deploy_broker]' on first dry-run dispatch. Root cause is GHA's 'workflows are registered from the default branch' rule — same trap already documented in §6 ('Common first-run failure modes'), but I didn't repeat it in §7.3, so the operator hit it again. Fix: - §7.3 dispatch command now includes --ref <pr-branch>. - Distinguish pre-merge (--ref required, input lives on PR branch) from post-merge (--ref optional, input is on main). - Show the git rev-parse trick to look up the local branch name. Per CLAUDE.md runbook-fix-fold-back: every operator-encountered failure makes the runbook strictly more robust. * fix(ci): grant ssm:DescribeInstanceInformation to deploy role + distinguish AccessDenied in workflow Deploy-test-broker's sanity-check step failed in the first dry-run with 'i-XXX is not SSM-managed'. Root cause: same swallowed-stderr trap as the local script, now in the workflow. The deploy role's inline policy granted SendCommand + GetCommandInvocation + ListCommandInvocations, but NOT DescribeInstanceInformation. AccessDenied was silently mapped to 'None', which the workflow interpreted as 'not SSM-managed'. Three fixes: 1. provision-ci-deploy-role.sh: PollCommandStatus statement now includes ssm:DescribeInstanceInformation. put-role-policy is idempotent so re-running the script refreshes the existing role's inline policy in place. 2. harness-ci.yml sanity-check: captures stderr separately, greps for AccessDenied, prints actionable remediation. Empty state (no AccessDenied) still means genuinely-not-registered. 3. docs/ci-setup.md §7.1: lists DescribeInstanceInformation in the inline-policy bullet + notes 'already provisioned? re-run; idempotent'. Per CLAUDE.md runbook-fix-fold-back: every operator-encountered failure makes the runbook + scripts strictly more robust. The defensive workflow step catches this in the future if the policy template ever drifts. * fix(deploy-test-broker): auto-discover agentKeys repo path on EC2 Deploy job's SSM script failed with 'cd: can't cd to /home/ubuntu/agentKeys' on the operator's test broker. The hardcoded path assumed the ubuntu-user clone layout, but the operator's box has the repo at a different location (the broker EC2 may have been bootstrapped from a non-default user or path). Fix: - Auto-discover loop tries TEST_BROKER_REPO_DIR override (new optional secret), then 7 common candidates (/home/ubuntu/agentKeys, /opt/agentkeys, /srv/agentkeys, /root/agentKeys, etc.). First candidate containing scripts/setup-broker-host.sh wins. - stat -c '%U' to discover the actual tree owner instead of hardcoding 'ubuntu' — covers the agentkeys / root / custom-user cases. - Fail loud with the override secret name if no candidate matches. Docs (docs/ci-setup.md §7.2): TEST_BROKER_REPO_DIR added to the secrets table with a note that it's optional + only needed when auto-discover prints 'could not locate'. Diagnosed via SSM command stderr after the upstream AccessDenied + perm gaps were resolved earlier in this PR. * fix(deploy-test-broker): add /home/agentkey paths + safe-default REPO_DIR_OVERRIDE under set -u Two regressions caught by the second dispatch on the operator's box: 1. Auto-discover didn't find the repo. Operator confirmed the checkout lives at /home/agentkey/agentKeys — not in my original 8 candidates. Added /home/agentkey/agentKeys, /home/agentkey/agentkeys, and /home/agentkeys/agentKeys (covering the variations of the broker app user name). 2. Diagnostic echo referenced \$REPO_DIR_OVERRIDE under remote-shell set -u, which fires 'parameter not set' when the secret is unset. Fixed with a one-line default at the top of the remote script: REPO_DIR_OVERRIDE="${REPO_DIR_OVERRIDE:-...}" That makes subsequent references safe under set -u while still honoring an operator-set override. * fix(setup-broker-host): default HOME so SSM-driven invocations work under set -u After the auto-discover + repo path fix landed, the SSM-driven deploy got past clone, fetch, summary, apt deps, and rustup install — then hit 'HOME: unbound variable' at the rustup-env source line. SSM-driven remote shells (AWS-RunShellScript document) don't export HOME for the default user; setup-broker-host.sh uses 'set -euo pipefail', so the unset reference aborts. Fix: 'export HOME=${HOME:-$(getent passwd $(id -u) | cut -d: -f6)}' right after 'set -euo pipefail'. Resolves the running user's home dir from /etc/passwd when the env var is missing — portable across interactive ssh sessions (HOME already set) and SSM SendCommand (HOME unset). Same root cause family as the earlier IamInstanceProfile + agent-install fixes: bootstrap paths assume an interactive operator shell, but the CI auto-deploy path is the structural test for those assumptions. * fix(harness): heima-test-deployer nonce contention (codex adversarial findings) Codex adversarial review (PR #102) confirmed the harness-e2e failure 'replacement transaction underpriced' is NOT caused by this PR. The broker/workers have no chain-write code paths reachable from the shipped feature set: audit-evm is feature-gated (Phase C, unshipped), the worker-audit's auto-flush only LOGS 'ready for on-chain appendRoot' without submitting txs, and setup-broker-host.sh has zero deployer-key access. The actual mechanism is harness-side nonce contention: concurrent harness-e2e runs (PR branch + workflow_dispatch + re-triggers) share ONE Heima test deployer wallet, and 'cast send' without --nonce defaults to 'latest' nonce derivation — which collides with pending mempool txs from a prior run. Two-layer fix: 1. .github/workflows/harness-ci.yml — second concurrency group on harness-e2e, scoped to 'heima-test-deployer-nonce' (not the ref), with cancel-in-progress: false so queued runs wait rather than cancel. The outer 'harness-ci-${{ github.ref }}' lock only serializes per-branch; this one serializes globally for the shared deployer wallet. 2. scripts/heima-fund-account.sh + scripts/heima-agent-create.sh — pass '--nonce $(cast nonce ADDR --block pending)' so cast computes the nonce against the PENDING block, not the latest confirmed. This defends against a stuck mempool tx that survives the previous run's exit (concurrency lock alone doesn't help — the tx is in the mempool, not in another job). Both layers also add a specific error message when the underpriced case fires, telling the operator to wait ~1min for the stuck tx to confirm or drop. Codex investigation log (1.4M tokens): scanned setup-broker-host.sh, broker-server, all 4 workers, env files, harness scripts, and workflow YAML. Found zero chain-write paths reachable from the deployed broker binary. Specific evidence cited in codex's response (crates/agentkeys- broker-server/src/handlers/cap.rs uses eth_call reads only; worker-audit main.rs:71 logs intent but doesn't submit; broker.env has no deployer key). * fix(ci): add pull-requests:read for dorny/paths-filter on PR events The detect-changes job fails on pull_request triggers with 'Error: Bad credentials' from dorny/paths-filter@v3. Root cause: the workflow's explicit 'permissions:' block grants only id-token + contents, which sets every other scope (including pull-requests) to 'none'. paths-filter on PR events always queries the REST API (/repos/.../pulls/N/files) — without pull-requests:read, the token is rejected. Earlier workflow_dispatch + push triggers passed because dispatch + push don't take the PR-API code path (paths-filter does local git diff against the previous push). * docs: add broker + local operator dev guide New docs/spec/broker-and-operator-dev-guide.md focused on the inner edit-build-test loop: - The 7-process local stack (mock-server :8090, broker :8091, signer :8092, 4 workers :9092-:9095) with the exact ports + crates + env vars each one reads. - First-time keypair generation (one-shot keygen for the broker's ES256 OIDC + session keypairs). - Inner loop A — edit broker code: scripts/broker.dev.env template, the --features auth-email-link footgun, three-terminal foreground flow, hot-reload pattern. - Inner loop B — edit operator scripts: scripts/operator-workstation.dev.env template, the --from-step/--to-step/--only-step primitive, anvil for fully-local chain dev. - Inner loop C — CI auto-deploy (issue #101 / PR #102): which paths trigger the auto-deploy + how to dry-run via workflow_dispatch. - Config-file map distinguishing broker.env vs operator-workstation.env vs broker.test.env so the most common 'I sourced the wrong file' bug is debuggable from the guide. - Debugging cheatsheet — RUST_LOG, port collisions, the 5 most common broker-boot-fail shapes with their fixes. - Chain profile selection (anvil vs heima-paseo vs heima). Distinct from docs/dev-setup.md (environment bootstrap) and docs/operator-runbook-stage7.md (deploy-to-real-host) — those are the 'first machine' / 'first broker' docs. This is the 'I'm iterating on the broker right now' doc. Linked from README.md Development section. * docs(readme): split into 'For humans' + 'For AI coding agents' sections Top: project name, one-line description, status, arch.md link (shared). For humans: - What it does (4 component bullets) - Workspace layout - Build & test commands - First-machine setup (link to dev-setup.md) - Inner-loop dev (link to broker-and-operator-dev-guide.md) - License For AI coding agents: - Mandatory reading table (CLAUDE.md, arch.md, development-stages, execution-plan, dev guide) - Hard rules condensed from CLAUDE.md (jj usage, branch push policy, diagnose-before-edit, land-the-fix, runbook-fix-fold-back, no-hardcoded-values, idempotent-remote-setup, plan-completion, terminology-source-of-truth) - Per-session protocol (4 steps) - Single entry points (setup-broker-host.sh, setup-heima.sh) The split makes the README usable as the AI agent's session-start briefing AND as the human's project intro, without either side wading through content meant for the other. All 7 link targets verified present in the repo.
Two new docs slotted into the canonical docs/ layout established by PR #99: - docs/research/ai-memory-systems-survey.md (287 lines) Survey of 10 systems: Mem0, Letta/MemGPT, Zep/Graphiti, A-MEM, Cognee, MemMachine, LangMem, Claude memory tool, ChatGPT memory, OpenMemory MCP. Covers four-type memory taxonomy (episodic/semantic/procedural/profile), three-stage pipeline (extract/consolidate/retrieve), storage substrates (vector/graph/ JSON/files), retrieval mechanics (tool-call vs pre-call RAG vs full-context injection), portability formats (Letta .af, JSON Agents PAM, JSONL), privacy patterns, LoCoMo/LongMemEval benchmarks. - docs/plan/agentkeys-memory-design.md (796 lines) Design plan for evolving crates/agentkeys-worker-memory from a blob-storage primitive into a structured-memory service. Headline invariants: worker NEVER calls an LLM (embeddings come from caller); LLM never sees the whole memory (top-K snippets only); LLM is replaceable without re-keying. Storage: one S3 object per JSONL line at bots/<actor>/memory/episodic/<date>/ <ulid>.enc with atomic per-key PUT, HEAD-for-dedup, clean K3 rotation. Brute-force cosine over packed-binary index file is the v0 default (vector DB deferred as operator-elected cache). Prerequisite M-1: envelope v3 lands in agentkeys-worker-creds as a separate PR before any memory-worker code change. Plan went through /plan-eng-review (18 findings folded, 8 new test files spec'd, 4 critical failure modes covered, parallelization lanes documented). The two files are pre-implementation research and design. No code, no API changes, no migration. They inform the next batch of issues filed against crates/agentkeys-worker-memory.
* pm: declarative milestones + labels + issue automation + dashboard guide New pm/ subfolder for GitHub project management automation. Treats milestones / labels / issue categorization as code under version control with idempotent shell scripts that reconcile GitHub state to declarative JSON. Files: - pm/README.md — folder purpose + how to use - pm/milestones.json — 7 roadmap milestones (M1-M7) source of truth - pm/labels.json — 40-label taxonomy: area/ kind/ phase/ status/ priority/ + extras (needs-arch-review, vendor-blocker) - pm/issue-assignments.json — categorization of all 23 pre-existing open issues with milestone + labels + notes - pm/new-issues.json — 20 new Phase 1-7 issues to create - pm/arch-md-verification-report.md — #5/#6/#9/#37 verification - pm/PROJECT-DASHBOARD-GUIDE.md — how to use projects/19 board + CI integration patterns - pm/scripts/sync-milestones.sh — idempotent: creates/updates from milestones.json - pm/scripts/sync-labels.sh — idempotent: creates/updates from labels.json - pm/scripts/sync-issues.sh — idempotent: assigns milestone+labels to each issue in issue-assignments.json - pm/scripts/create-issues.sh — idempotent: creates new issues from new-issues.json, skips if title already exists - pm/scripts/audit.sh — read-only: groups open issues by milestone, flags uncategorized + missing area/* labels - pm/scripts/add-to-project.sh — adds issues to litentry/projects/19 (requires gh auth refresh -s project,read:project) Executed in this session: - Created 7 milestones (M1: First MCP demo + Volcano Ark PoC, M2: First vendor wedge, M3: Runtime neutrality, M4: Capability + revocation depth, M5: Native mobile + biometric, M6: TEE integration + security, M7: Standards + ecosystem) - Created 40 labels across 5 namespaces (area, kind, phase, status, priority) + extras (needs-arch-review, vendor-blocker) - Categorized 23 pre-existing open issues with milestones + labels - Created 20 new issues (#107-#126) for Phase 1-7 work per the agent-iam-strategy.md roadmap - Verified #5, #6, #9, #37 against arch.md — verdicts: #5 partially aligned (closed; lives as tier A in §15.3), #6 needs design refresh against current K11+SidecarRegistry, #9 already implemented as K3 HDKD per §6.2 (recommend close), #37 superseded by K11 WebAuthn per §K11 (recommend close) Final state: 43 open issues, 100% categorized to milestones, 100% labeled with area/*. No uncategorized issues. Per user direction: did NOT merge / close #5/#6/#9/#37 even though recommendations are clear. User to make final close decisions. * pm: fix bash 3.2 portability + add setup-project-fields.sh + labels-vs-fields strategy Three fixes responding to user feedback: 1. add-to-project.sh: replace mapfile (bash 4+) with while-read loop for macOS bash 3.2 portability per CLAUDE.md project standard. Verified working: 'bash pm/scripts/add-to-project.sh 103' now successfully adds the issue to litentry/projects/19. 2. NEW pm/scripts/setup-project-fields.sh: creates the canonical project-level fields (Priority, Phase, Estimate, Iteration, Risk, Notes) via gh project field-create. Solves the 'cluttered Labels column' UX pain by letting the user split single-value PM concerns (priority, phase, status) out of the multi-value labels pile into typed field columns. 3. PROJECT-DASHBOARD-GUIDE.md: added 'Labels vs Fields — when to use which' section explaining the split: - Labels (repo-level, multi-value): area/*, kind/*, semantic flags like needs-arch-review, vendor-blocker - Fields (project-level, single-value): Priority, Phase, Status, Estimate, Risk Plus step-by-step instructions to migrate the cluttered Labels column to clean field-based grouping. These don't change the strategic plan; they just fix the operational PM-board ergonomics the user surfaced from running the script live. * pm: workflow-first PM guidance + mark add-to-project.sh as backfill User pointed out the project board has 10 built-in workflows that replace much of what the scripts do. Updated guidance to prefer workflows; scripts are fallback/batch tools. PROJECT-DASHBOARD-GUIDE.md updates: - Replaced the brief 'Recommended workflows' section with a full table of the 10 built-in workflows + their default state + what to configure - New 'Script ↔ workflow split' table making clear which jobs use workflows vs scripts (workflows for runtime project events; scripts for repo-level state, batch creation, field definitions) - One-time workflow configuration checklist (3 steps to get the Auto-add filter set, verify other green workflows, optionally enable Auto-archive) add-to-project.sh updates: - Header now flags this as PRIMARILY A BACKFILL / FALLBACK TOOL - Lists three legit use cases: backfilling pre-existing issues, fallback when Auto-add workflow is misconfigured, adding from a different repo via PM_REPO override - Pointer to PROJECT-DASHBOARD-GUIDE.md for workflow setup No script behavior changes; only documentation tightens to match the workflow-first reality. * pm: programmatic workflow audit (names + enabled state; filter/action stay manual) User asked if workflows can be programmatically checked. Partial yes: GitHub's public GraphQL ProjectV2Workflow type exposes only: id, name, number, enabled, createdAt, updatedAt, project, fullDatabaseId NOT the filter expression or action configuration (UI-only, not in the public API). So we get: ✅ 'is the workflow enabled' check ❌ 'does the workflow do the right thing' check (filter/action body) New files: - pm/expected-workflows.json: declarative source of truth for what workflows should be enabled + what each one's filter/action should do (free-text 'verify_in_ui' field that engineers cross-check against the UI) - pm/scripts/check-workflows.sh: audits live workflows on litentry/projects/19 vs expected-workflows.json - Confirms enabled state matches - Flags unexpected workflows that exist but aren't in our list - Prints all per-workflow expected filter/action notes for manual UI verification - Exits 0 when all expectations match, 1 on mismatch (CI-friendly) Live audit result (verified on litentry/projects/19): 7 expected workflows enabled (Auto-add to project, Auto-add sub-issues to project, Item added/closed, Auto-close issue, PR linked/merged), 4 optional workflows correctly disabled (Auto-archive, Code review approved, Code changes requested, Item reopened). 11/11 match. This script can be wired into a future CI workflow to alert on drift if anyone disables Auto-add to project or similar. * pm: automate project field sync + workflow drift audit via GH Actions Adds two GitHub Actions and one supporting script to push project automation to its API ceiling. After this change, label-to-field sync and workflow drift detection both run on every event / daily schedule instead of as manual scripts. What landed: - .github/workflows/pm-sync-fields-from-labels.yml: triggers on issues labeled/unlabeled/opened/transferred. Calls sync-fields-from-labels.sh to mirror priority/p* + phase/v* labels into the project's Priority + Phase single-select fields. workflow_dispatch variant for backfill. - .github/workflows/pm-workflow-audit.yml: daily cron + push trigger. Runs check-workflows.sh against expected-workflows.json and opens (or comments on) a tracking issue when drift is detected. - pm/scripts/sync-fields-from-labels.sh: backing script for the sync workflow. Forgiving mode (warns + skips when a field is missing rather than aborting), bash 3.2 portable, uses -f for option-ID strings to avoid gh api numeric coercion. - pm/scripts/setup-project-fields.sh: now detects + rebuilds empty-placeholder single-select fields (GitHub's built-in Priority/Size ship with zero options) and cleans up "Project <Name>" zombie fields left behind when deleteProjectV2Field renames instead of deleting system-reserved names. Fully idempotent. - pm/PROJECT-DASHBOARD-GUIDE.md: new "What's automated vs UI-only" verdict table (built-in workflow filter/action contents + custom views are 100% UI-only — no API mutation exists for either). New "Known gotcha" section on Priority-field zombies. Script-vs-workflow split rewritten as three-tier matrix (built-in / our GH Action / bash script). Verification: tested live against litentry/projects/19. Backfilled 40+ issues onto board, synced Priority + Phase from labels on every one, zero zombie fields remain. setup-project-fields.sh second-run shows all skips. API ceiling discovered via GraphQL introspection: ProjectV2Workflow has no create/update mutation (only delete). ProjectV2View has no create/update mutation at all. Both are read-only via API, UI-only to configure. Required repo secret for CI: PM_PROJECT_TOKEN (fine-grained PAT with Projects=read+write, Issues=read+write). Documented in dashboard guide. * pm: strip refs to strategy doc not yet on main Three links in pm/README.md and pm/PROJECT-DASHBOARD-GUIDE.md pointed at docs/research/agent-iam-strategy.md, which is still on a feature branch. Replace with pointers to pm/milestones.json (the data that's actually on this PR) so the rendered markdown doesn't 404 once merged. The strategy doc + research folder land in a separate PR.
Two issues surfaced from the first pm-workflow-audit.yml run on main: 1. The audit reported real drift: Auto-archive items workflow is enabled on litentry/projects/19, but expected-workflows.json marked it as should_be_enabled=false. The operator enabled it via the UI (which is the recommended state per the original note). Flip the expected state to match reality. 2. The drift-issue-creation step failed: "could not add label: 'kind/automation' not found". The repo doesn't have a kind/automation label — only the 7 in pm/labels.json. Switch to kind/devx since the automation health belongs to dev experience. After this, the audit should report 11/11 match and the issue-create step won't fire (but is defensively label-correct for future drifts).
…#130) * docs(research): AI hardware companion wedge + office-hours design doc Add two business research artifacts under docs/research/: - ai-hardware-companion-wedge.md (round 1+2): market sizing, competitive landscape, direct competitors, business model critique, 12 critical comments, naming, Stripe ACP / Alipay+ AMP integration path, WeChat feasibility, security-first demo storyboard. - ai-hardware-companion-office-hours.md: YC-style office-hours diagnostic on the same wedge. Six forcing questions surfaced zero vendor conversations + no named buyer. P2 narrowed mid-session to memory portability + isolation + privacy. Approach D chosen: AgentKeys-native hosted sandbox (aiosandbox) with OpenClaw/Hermes agent runtime + per-actor isolation (issue #90) + cross-vendor memory consent model. Pricing pivoted to AWS-style elastic per-user (Free / Basic vendor-paid $2-3/active-device / Pro $10 user-paid with 30% lifetime acquirer revshare / future Compute usage-based). 8/10 quality after 2 spec-review iterations. Both index entries added to docs/research/README.md. * docs(plan): issue #102 — aiosandbox + Hermes + AgentKeys ESP32 demo plan End-to-end demo plan for the AgentKeys hardware-vendor wedge: ESP32 device + simple URL config → agent-infra/sandbox running Hermes (AgentKeys-native runtime) + agentkeys-daemon with mock memory injected from S3 MD blob at agent boot. 12-step implementation order. Reuses arch.md canonical primitives (sandbox runtime, supervisord lifecycle, memory bucket layout bots/<actor_omni>/memory/, agentkeys-daemon). v0 scope: single ESP32, single sandbox, single mock memory blob, text-mode chat. Voice mode, multi-tenancy, cap-token enforcement, cross-vendor portability, and payment rails are deferred to follow-up issues. 3-week effort estimate. Acceptance: reviewer can flash board + run setup script + see personalized response within 15 minutes. * issue #103: ESP32-S3 firmware foundation + plan rename Pivot canonical demo target from generic ESP32 to ESP32-S3-DevKitC-1: - Native USB-OTG (single USB-C, no separate UART chip) - PSRAM (8MB octal) for voice follow-up audio buffers - Xtensa LX7 with AI vector instructions for on-device wake-word - Still MCU-class authenticity (~$10-15 dev board, <$5 chip in BOM volume) Stack: PlatformIO + ESP-IDF (not Arduino) — production AI-toy vendors use ESP-IDF and S3-specific features (native USB CDC, PSRAM, ESP-DSP, secure boot, OTA) need IDF. Scaffolded firmware foundation under firmware/esp32s3-agentkeys/: - platformio.ini, CMakeLists.txt, sdkconfig.defaults, partitions.csv - main.c spawns 4 FreeRTOS tasks (wifi/button/chat/led) coordinated via event group + queue - wifi_sta.c: working STA mode + auto-reconnect - button.c: working GPIO interrupt + 200ms debounce on BOOT (GPIO 0) - led_status.c: stub blinker (real WS2812 RGB state machine is TODO) - https_chat.c: stub echoing user input (real esp_http_client POST is TODO) - config.h: NVS → secrets.h → hardcoded defaults priority order - README.md: flash quickstart + troubleshooting Foundation builds + flashes + boots into FreeRTOS loop today; chat returns mock '[mock] you said: ...' echo. Real HTTPS POST is the clear next step (esp_http_client + cJSON parse, ~100 lines). Renamed plan file issue-102 → issue-103 to match actual issue number. * research(xiaozhi): identify hardware as MagicLick 2.5 + pivot to Option 1 Hardware on hand confirmed via the device display showing 'magiclink 2p5/1.9.4': MagicLick 2.5 running xiaozhi-esp32 v1.9.4 firmware. xiaozhi-esp32 (github.com/78/xiaozhi-esp32, MIT, 26K stars) is the dominant Chinese open-source AI voice firmware for ESP32. Supports 70+ boards including ours. Full streaming voice pipeline already shipping: offline wake-word (ESP-SR) → ASR → LLM → TTS → OPUS over WebSocket or MQTT+UDP. MCP-based device + cloud control. MagicLick 2.5 hardware specs reconstructed from boards/magiclick-2p5/config.h + board.cc: - ESP32-S3 chip - ES8311 audio codec (full-duplex I2S, 24kHz) - 128x128 GC9107 SPI LCD with emoji rendering - 3 buttons (main GPIO 21, left GPIO 0, right GPIO 47) - 2 WS2812 LEDs on GPIO 38 - DualNetworkBoard: WiFi primary + ML307 Cat.1 4G fallback - Battery + power manager with tickless idle 'Hermes agent' clarified to mean NousResearch/hermes-agent (MIT, Python, self-improving learning loop, multi-interface gateway, LLM-agnostic). NOT an internal AgentKeys runtime as the original plan §C4 mistakenly stated. Strong recommendation: Option 1 — keep xiaozhi firmware unchanged, build cloud-side xiaozhi-hermes-bridge that speaks the xiaozhi WebSocket protocol while routing the agent loop to Hermes-agent (which pulls memory from agentkeys-daemon per §C3). Reduces v0 effort from ~3 months (custom firmware) to ~2-3 weeks (server-side adapter only). Forks from one of four existing reference server implementations (Python xinnan-tech, Go hackers365 with openclaw, Java joey-zhou, Go AnimeAIChat). Hardware verification: 5 paths documented (visual / ROM bootloader via boot button hold / WiFi captive portal / vendor app / disassembly). USB doesn't enumerate by default because device is in normal firmware mode; hold LEFT button while connecting USB to drop into ESP32-S3 ROM bootloader for esptool access. Added PIVOT banner at top of issue-103 plan flagging that C4/C5/C6 are superseded. Full new direction in docs/research/xiaozhi-esp32-magiclink.md. firmware/esp32s3-agentkeys/ stays in tree as reference scaffolding for future custom hardware (new product lines that need first-party firmware), not the path for the MagicLick demo. * research(xiaozhi-hermes): architecture diagrams + risk verification Two new research docs supporting the issue #103 Option 1 direction: docs/research/xiaozhi-hermes-architecture.md Permanent architecture reference with three ASCII diagrams: - Diagram A: baseline xiaozhi flow (device → cloud → LLM) - Diagram B: our pivoted flow with changed layers highlighted (UNCHANGED firmware, NEW URL only on device side, fork + one-module-rewrite on cloud side, new memory layer) - Diagram C: per-turn sequence with latency budget breakdown (~2.0-2.5s first-audio; ~+250-500ms delta vs baseline) Precise diff table: 13 layers compared, only 4 actually change, 3 of those are NEW additions (not modifications). The actual code change is concentrated in ONE module of the bridge fork. docs/research/xiaozhi-hermes-risks.md Risk verification grounded in actual Hermes-agent + xinnan-tech/xiaozhi-esp32-server source code, NOT assumptions. Specific file paths + line numbers cited throughout. R1 (Hermes HTTP gateway stateless-vs-session): REAL but mitigation is built-in. Gateway exposes /v1/chat/completions with three session modes (stateless per-call default, explicit continuation via X-Hermes-Session-Id, long-term memory scoping via X-Hermes-Session-Key). Bridge sets per-device session keys. Effort: 2-4 hours. R2 (Latency stack): mostly NOT real. agent/conversation_loop.py line 4152 confirms learning loop runs as background task AFTER response delivery, OFF the turn path. With enabled_toolsets=[] + max_iterations=1 + streaming SSE, overhead is ~50-200ms. xiaozhi-performance-research baselines: - ASR: 0.795s Xunfei / 0.85s Doubao - LLM first-token: 0.434s Qwen-Flash / 0.774s Kimi-K2 - TTS: 0.488s CosyVoice / 0.667s Edge-TTS / 0.103s PaddleSpeech Pipelined: 1.4-2.4s first-audio, within 2.0-2.5s target. Effort: 1 day (tune + measure). R3 (Concurrent device handling): less bad than feared. Hermes gateway IS multi-tenant by design (serves Telegram + Discord + Slack + WhatsApp + Signal + CLI from one process). Per-request memory ~20-80MB; 100 devices ~2-8GB on one VPS. xiaozhi-esp32- server's documented '100+ devices per process' claim is unverified in repo — only 6-concurrent demo documented. For v0: 0 hours. For production scale: 1-2 weeks sticky-LB. R4 (newly discovered during research): cold agent construction per request adds 50-300ms on every turn. _create_agent() called inside _handle_chat_completions for EVERY request, no pooling. Most impactful for voice UX (compounds turn-by-turn). Mitigation: fork-local agent pool (1 day) or upstream patch (2-4 days). Net effect: v0 timeline revised from ~3 weeks to ~1-2 weeks. Updated docs/research/README.md to index both new docs. * research(tuya) + revise v0 timeline ~3w → ~1-2w + fix unverified claim Three updates following the risk-verification research: 1. docs/research/tuya-vs-xiaozhi.md (new) Answers 'is Tuya the same role as xiaozhi?': DIFFERENT role, partial firmware overlap. Tuya = closed PaaS for brand-owners (NYSE: TUYA, $80.9M Q1 2026 revenue, 306 premium customers, 1.97M developers, 100+ countries). xiaozhi = open firmware for makers (MIT, 26.7K stars). TuyaOpen is a 1.6K-star defensive ESP32 SDK from Jan 2026 — 17x adoption gap. AgentKeys posture: complement both, never compete. - Phase 1 (now): xiaozhi cloud-side bridge (issue #103) - Phase 2 (3-6 mo): Tuya Cloud Development connector - Sit above both rails (same pattern as Alipay+ AMP / Stripe ACP) 2. v0 demo timeline revised from ~3 weeks to ~1-2 weeks in issue-103-aiosandbox-hermes-esp32-demo.md: - PIVOT banner at top of plan - Effort estimate section (line 441) The basis is xiaozhi-hermes-risks.md showing all four risks are smaller than originally feared (R1 built-in mitigation, R2 background loop, R3 multi-tenant by design, R4 cheap fork-local hack). 3. Fixed false cross-reference in xiaozhi-hermes-risks.md The 'unverified 100+ devices' claim was incorrectly attributed to the office-hours doc. It actually circulated in earlier informal discussion — not in any committed doc. Reworded to remove the false attribution. 4. Added implementation update banner to office-hours doc pointing readers at the four xiaozhi research docs + the revised v0 timeline. The §Recommended Approach / Pricing / Cross-Vendor Memory Model below stay unchanged — only the firmware-and-runtime layer shifted. * research(tuya): verify Phase 3 IoT cloud adapter feasibility per-platform Earlier version of tuya-vs-xiaozhi.md claimed Phase 3 would add adapters for Xiaomi MIoT, Alibaba Smart Home, and Volcano AI Hub without verifying each platform's third-party developer surface. Research findings per platform: Volcano Ark (ByteDance) — VERIFIED FEASIBLE - Open international developer signup, no PRC entity / ICP needed - MCP-server marketplace launched 2026 (mcp.so/server/mcp-server/volcengine) - AgentKeys publishes an MCP tool any Doubao-powered AI hardware can call - Genuinely Tuya-equivalent for the AI-side rather than IoT-side - ~1 week effort AliGenie / Tmall Genie (Alibaba) — FEASIBLE WITH PARTNERSHIP - International Alibaba Cloud account works for sandbox + custom-skill webhook - Production distribution onto Tmall Genie hardware requires Alibaba's skill review + de-facto PRC-domiciled brand - ~1 week dev + partnership lead time Xiaomi MIoT / XiaoAI — WEAKEST - Brand-tier integration requires Mi Ecosystem partnership admission - Publishable XiaoAI skills require PRC real-name verification - Consumer-OAuth path (Home-Assistant-style) works today for foreign servers but is a narrower wedge than brand-tier - Defer until partnership or scope to consumer-OAuth only Rewrote Phase 3 section to split into 3a (Volcano open), 3b (AliGenie with partner), 3c (Xiaomi deferred). Added explicit 'Honest note on Phase 3 verification' acknowledging the original claim was hand-wavy. Added 15 source URLs to the Sources block. * research(volcano-ark): MCP-server integration architecture + diagrams New research doc with three ASCII diagrams showing how AgentKeys integrates with Volcano Ark (ByteDance's enterprise AI cloud hosting Doubao LLM) as a Phase 3a hosted MCP server registered in their 2026 MCP marketplace. Pattern B (hosted by us, marketplace is discovery only): - AgentKeys MCP server at mcp.agentkeys.io exposes 5-7 tools (memory get/put, cred fetch, cap mint, audit append, whoami, permission check) mapped to existing Stage 7+ backend RPCs - Vendor Doubao agents call our MCP tools via HTTPS/SSE with per-vendor Bearer token + per-actor X-AgentKeys-Actor header - No vendor firmware changes; no Doubao runtime changes — just marketplace registration + one-checkbox vendor opt-in Diagram A: high-level architecture (device → RTC → Doubao → MCP → AgentKeys MCP server → backend) Diagram B: per-call MCP tool sequence with ~200-400ms per-call latency budget (concern noted: multiple tool calls per turn can stack — mitigation via batched 'context.bootstrap' tool) Diagram C: cross-vendor composition showing same user (O_kevin) with FoloToy (Doubao + MCP adapter) AND MagicLick (xiaozhi + Hermes bridge) both terminating at one AgentKeys backend with one memory namespace + one identity tree + one audit ledger. This is the cross-vendor portability moat materializing automatically per office-hours doc §Cross-Vendor Memory Model. Effort: ~1-1.5 weeks (sibling to xiaozhi-hermes-bridge). 6 open risks called out + mitigations sketched: - MCP latency stacking per turn - Marketplace approval SLA - Per-tenant auth model TBD - Actor omni resolution pattern (vendor-side vs whoami call) - MCP protocol version compat with Doubao runtime - Cross-vendor cap-token consent (resolved: same office-hours consent ceremony applies) Updated docs/research/README.md to index the new doc. * strategy: Agent IAM positioning + 4 architecture corrections New strategic anchor doc at docs/research/agent-iam-strategy.md captures the revised direction from multi-round discussion (original Agent IAM proposal → independent analysis → ChatGPT critique → synthesis). Three-layer positioning, three audiences: - AI Device Account (consumer/vendor BD pitch) - Agent IAM (B2B/investor/CTO category) - Trust Substrate (compliance/regulator/Web3 partner) Five accepted strategic moves: - Task Host vs Authority Host distinction (we are Authority) - Agent IAM as the technical category (not key management / not memory MCP) - MCP as integration surface, not product identity - Zero orchestration in v1 — hard line - Deploy → grow → standardize sequencing Four architecture corrections that tighten commitments: 1. Revocation: 'immediate online, bounded TTL/cache offline' (NOT 'no propagation delay'). High-risk actions always online; low-risk reads use short-lived cached caps; offline mode denies sensitive actions by default. 2. Audit (two-tier): real-time off-chain feed in parent-control UI + 10-min batched Merkle root anchored to Heima. NOT real-time on-chain. Heima explorer is tamper-evidence proof, not the UX surface. 3. Delegation: agentkeys.delegation.grant is schema-documented but not active in v1. Returns not_implemented_in_v1. Active delegation lands in Phase 4. 4. Dual narrative — don't lead with 'Agent IAM' in consumer contexts; don't lead with 'memory portability' anywhere. Authority is the category; privacy/memory are benefits. Phase 1 revised to three-act IAM demo (per office-hours doc §9.6 storyboard, now elevated to authoritative spec): - Act 1 Permissioned Memory (scoped read, not 'smart') - Act 2 Deterministic Denial (policy decides, no LLM) - Act 3 Online Revocation (parent UI → device denies) Implementation note: cap-token machinery is already shipped via Stage 7+ (broker, signer, K3/K10 HDKD, memory/cred/audit workers, per-actor isolation per issue #90). New Phase 1 work is the MCP server wrapper (~1 week), parent-control web UI (~3-4 days), two-tier audit wiring (~1 day), runbook (~half day). Total ~2 weeks. 12-month roadmap revised: - Phase 0: shipped (Stage 7+) - Phase 1 (0-2 wk): Agent IAM v0 demo - Phase 2 (1-2 mo): vendor pilot + multi-rail (Volcano Ark, Tuya) - Phase 3 (3-4 mo): runtime neutrality (Hermes/OpenClaw as MCP tools) - Phase 4 (6 mo): delegation + approval + ACL depth - Phase 5 (post-12mo): standards engagement (contingent on traction) Updates to existing docs: - docs/research/README.md: indexed new strategy doc as 'Strategic anchor' - ai-hardware-companion-office-hours.md: positioning note pivoted from 'implementation update' to 'strategic update' pointing at strategy doc - issue-103 plan: PIVOT banner expanded with three-act demo + four corrections; old §C4/C5/C6 marked superseded; cap-token shipped context made explicit; no implementation re-spec per user direction * strategy(nits): chain-agnostic positioning + 2-min batch + memory namespace model Three nits from review: 1. Generic chain instead of Heima-specific positioning The strategy doc shouldn't be Heima-locked — chain is a deployment config (arch.md describes 'Litentry parachain (or EVM L2 fallback)' so the design is already chain-agnostic at the contract layer). Updated all positioning text to 'audit chain' / 'on-chain' / 'chain explorer' instead of Heima-specific. Kept arch.md and runbook refs to Heima where they describe actual deployed infra (the 'currently Heima per arch.md, swappable' note in §Phase 0 captures the reality without committing the strategy to Heima). 2. 2-min batch instead of 10-min Modern fast-finality chains with cheap gas make sub-block-time batching viable. 10 min was too conservative — set 2 min as the default cadence. Faster batch = better UX for parents watching audit feed; the cost per anchor is sub-cent at typical batch sizes. 3. Memory namespace model (new §3.5) Read the memory research/design doc from main (commit 53ccc9f 'docs: AI memory worker design plan + agent-memory research survey'). It defines four STRUCTURAL types (profile / procedural / semantic / episodic) with specific S3 key derivation per type. For Agent IAM, namespaces are an ORTHOGONAL semantic dimension that composes with the 4 structural types. Memory item has BOTH a structural type AND a semantic namespace. Cap-tokens scope namespace access (namespaces_allowed claim, deterministic string-set membership check). v0 defaults: personal / family / work / travel (4 namespaces). kids/device/temp deferred to Phase 3-4. Composition is non-conflicting: namespaces live in wire-format metadata, NOT in the S3 key derivation. Memory worker filters at retrieval. The 4-type S3 layout from memory-design §3.2a is preserved exactly. Future evolution path documented (path-prefixed layout if scale demands). arch.md compatibility check: zero contradictions found. - Memory data_class binding (§17.5) unchanged - Per-actor PrincipalTag isolation (§17) unchanged - Cap-token format extensible (namespaces_allowed is additive) - Memory worker never calls LLM invariant preserved - K3 epoch rotation unchanged - Architecture-as-source-of-truth: future arch.md §17 + memory- design §3 get additive paragraphs when v0 ships, no canonical- name conflicts introduced. Files updated: - docs/research/agent-iam-strategy.md: §3.2 audit (2-min + chain- agnostic), §3.5 NEW memory namespace model with arch.md compat check, Phase 0 line (Heima → 'currently Heima per arch.md, swappable') - docs/research/README.md: strategy doc summary updated with 2-min + namespace model - docs/research/ai-hardware-companion-office-hours.md: implementation update banner reflects 2-min on-chain anchor - docs/research/volcano-ark-mcp-integration.md: diagram boxes generic ('AWS S3, audit chain', 'off-chain + chain') - docs/spec/plans/issue-103-aiosandbox-hermes-esp32-demo.md: PIVOT banner reflects 2-min chain-agnostic anchor; NOT-in-scope list generic 'on-chain audit anchoring' * pm: declarative milestones + labels + issue automation + dashboard guide New pm/ subfolder for GitHub project management automation. Treats milestones / labels / issue categorization as code under version control with idempotent shell scripts that reconcile GitHub state to declarative JSON. Files: - pm/README.md — folder purpose + how to use - pm/milestones.json — 7 roadmap milestones (M1-M7) source of truth - pm/labels.json — 40-label taxonomy: area/ kind/ phase/ status/ priority/ + extras (needs-arch-review, vendor-blocker) - pm/issue-assignments.json — categorization of all 23 pre-existing open issues with milestone + labels + notes - pm/new-issues.json — 20 new Phase 1-7 issues to create - pm/arch-md-verification-report.md — #5/#6/#9/#37 verification - pm/PROJECT-DASHBOARD-GUIDE.md — how to use projects/19 board + CI integration patterns - pm/scripts/sync-milestones.sh — idempotent: creates/updates from milestones.json - pm/scripts/sync-labels.sh — idempotent: creates/updates from labels.json - pm/scripts/sync-issues.sh — idempotent: assigns milestone+labels to each issue in issue-assignments.json - pm/scripts/create-issues.sh — idempotent: creates new issues from new-issues.json, skips if title already exists - pm/scripts/audit.sh — read-only: groups open issues by milestone, flags uncategorized + missing area/* labels - pm/scripts/add-to-project.sh — adds issues to litentry/projects/19 (requires gh auth refresh -s project,read:project) Executed in this session: - Created 7 milestones (M1: First MCP demo + Volcano Ark PoC, M2: First vendor wedge, M3: Runtime neutrality, M4: Capability + revocation depth, M5: Native mobile + biometric, M6: TEE integration + security, M7: Standards + ecosystem) - Created 40 labels across 5 namespaces (area, kind, phase, status, priority) + extras (needs-arch-review, vendor-blocker) - Categorized 23 pre-existing open issues with milestones + labels - Created 20 new issues (#107-#126) for Phase 1-7 work per the agent-iam-strategy.md roadmap - Verified #5, #6, #9, #37 against arch.md — verdicts: #5 partially aligned (closed; lives as tier A in §15.3), #6 needs design refresh against current K11+SidecarRegistry, #9 already implemented as K3 HDKD per §6.2 (recommend close), #37 superseded by K11 WebAuthn per §K11 (recommend close) Final state: 43 open issues, 100% categorized to milestones, 100% labeled with area/*. No uncategorized issues. Per user direction: did NOT merge / close #5/#6/#9/#37 even though recommendations are clear. User to make final close decisions. * pm: fix bash 3.2 portability + add setup-project-fields.sh + labels-vs-fields strategy Three fixes responding to user feedback: 1. add-to-project.sh: replace mapfile (bash 4+) with while-read loop for macOS bash 3.2 portability per CLAUDE.md project standard. Verified working: 'bash pm/scripts/add-to-project.sh 103' now successfully adds the issue to litentry/projects/19. 2. NEW pm/scripts/setup-project-fields.sh: creates the canonical project-level fields (Priority, Phase, Estimate, Iteration, Risk, Notes) via gh project field-create. Solves the 'cluttered Labels column' UX pain by letting the user split single-value PM concerns (priority, phase, status) out of the multi-value labels pile into typed field columns. 3. PROJECT-DASHBOARD-GUIDE.md: added 'Labels vs Fields — when to use which' section explaining the split: - Labels (repo-level, multi-value): area/*, kind/*, semantic flags like needs-arch-review, vendor-blocker - Fields (project-level, single-value): Priority, Phase, Status, Estimate, Risk Plus step-by-step instructions to migrate the cluttered Labels column to clean field-based grouping. These don't change the strategic plan; they just fix the operational PM-board ergonomics the user surfaced from running the script live. * pm: workflow-first PM guidance + mark add-to-project.sh as backfill User pointed out the project board has 10 built-in workflows that replace much of what the scripts do. Updated guidance to prefer workflows; scripts are fallback/batch tools. PROJECT-DASHBOARD-GUIDE.md updates: - Replaced the brief 'Recommended workflows' section with a full table of the 10 built-in workflows + their default state + what to configure - New 'Script ↔ workflow split' table making clear which jobs use workflows vs scripts (workflows for runtime project events; scripts for repo-level state, batch creation, field definitions) - One-time workflow configuration checklist (3 steps to get the Auto-add filter set, verify other green workflows, optionally enable Auto-archive) add-to-project.sh updates: - Header now flags this as PRIMARILY A BACKFILL / FALLBACK TOOL - Lists three legit use cases: backfilling pre-existing issues, fallback when Auto-add workflow is misconfigured, adding from a different repo via PM_REPO override - Pointer to PROJECT-DASHBOARD-GUIDE.md for workflow setup No script behavior changes; only documentation tightens to match the workflow-first reality. * pm: programmatic workflow audit (names + enabled state; filter/action stay manual) User asked if workflows can be programmatically checked. Partial yes: GitHub's public GraphQL ProjectV2Workflow type exposes only: id, name, number, enabled, createdAt, updatedAt, project, fullDatabaseId NOT the filter expression or action configuration (UI-only, not in the public API). So we get: ✅ 'is the workflow enabled' check ❌ 'does the workflow do the right thing' check (filter/action body) New files: - pm/expected-workflows.json: declarative source of truth for what workflows should be enabled + what each one's filter/action should do (free-text 'verify_in_ui' field that engineers cross-check against the UI) - pm/scripts/check-workflows.sh: audits live workflows on litentry/projects/19 vs expected-workflows.json - Confirms enabled state matches - Flags unexpected workflows that exist but aren't in our list - Prints all per-workflow expected filter/action notes for manual UI verification - Exits 0 when all expectations match, 1 on mismatch (CI-friendly) Live audit result (verified on litentry/projects/19): 7 expected workflows enabled (Auto-add to project, Auto-add sub-issues to project, Item added/closed, Auto-close issue, PR linked/merged), 4 optional workflows correctly disabled (Auto-archive, Code review approved, Code changes requested, Item reopened). 11/11 match. This script can be wired into a future CI workflow to alert on drift if anyone disables Auto-add to project or similar. * pm: automate project field sync + workflow drift audit via GH Actions Adds two GitHub Actions and one supporting script to push project automation to its API ceiling. After this change, label-to-field sync and workflow drift detection both run on every event / daily schedule instead of as manual scripts. What landed: - .github/workflows/pm-sync-fields-from-labels.yml: triggers on issues labeled/unlabeled/opened/transferred. Calls sync-fields-from-labels.sh to mirror priority/p* + phase/v* labels into the project's Priority + Phase single-select fields. workflow_dispatch variant for backfill. - .github/workflows/pm-workflow-audit.yml: daily cron + push trigger. Runs check-workflows.sh against expected-workflows.json and opens (or comments on) a tracking issue when drift is detected. - pm/scripts/sync-fields-from-labels.sh: backing script for the sync workflow. Forgiving mode (warns + skips when a field is missing rather than aborting), bash 3.2 portable, uses -f for option-ID strings to avoid gh api numeric coercion. - pm/scripts/setup-project-fields.sh: now detects + rebuilds empty-placeholder single-select fields (GitHub's built-in Priority/Size ship with zero options) and cleans up "Project <Name>" zombie fields left behind when deleteProjectV2Field renames instead of deleting system-reserved names. Fully idempotent. - pm/PROJECT-DASHBOARD-GUIDE.md: new "What's automated vs UI-only" verdict table (built-in workflow filter/action contents + custom views are 100% UI-only — no API mutation exists for either). New "Known gotcha" section on Priority-field zombies. Script-vs-workflow split rewritten as three-tier matrix (built-in / our GH Action / bash script). Verification: tested live against litentry/projects/19. Backfilled 40+ issues onto board, synced Priority + Phase from labels on every one, zero zombie fields remain. setup-project-fields.sh second-run shows all skips. API ceiling discovered via GraphQL introspection: ProjectV2Workflow has no create/update mutation (only delete). ProjectV2View has no create/update mutation at all. Both are read-only via API, UI-only to configure. Required repo secret for CI: PM_PROJECT_TOKEN (fine-grained PAT with Projects=read+write, Issues=read+write). Documented in dashboard guide. * pm: simplify automation — drop audit + label-sync workflows, use GitHub native User feedback after live use of the migration: - The label→field sync workflow is no longer needed (labels were deleted in PR #129; fields are now the source of truth, set via the issue-create skill or manually in UI). - The workflow-drift audit workflow added noise without value (built-in workflows rarely drift, and the operator manages them in UI anyway). - The Blocked-by TEXT project field duplicates GitHub's native issue relationships ("Mark as blocked by" / "Mark as blocking" in the UI side panel, keyboard `B B` / `B X`). Use the native feature. ## Removed - .github/workflows/pm-workflow-audit.yml (drift detection — operator handles in UI) - .github/workflows/pm-sync-fields-from-labels.yml (labels-to-fields sync — labels are gone) - pm/expected-workflows.json (declarative expectation for the audit) - pm/scripts/check-workflows.sh (called by the audit) - pm/scripts/sync-fields-from-labels.sh (called by the sync workflow) - "Blocked by" project field (deleted via API; setup-project-fields.sh no longer creates it) ## Kept / added - .github/workflows/pm-auto-archive-closed-pr.yml — auto-archives PRs from the board on close (built-in Auto-archive only fires after 30 days) - pm/scripts/sync-size-from-effort.sh (NEW) — one-shot bulk-populate of the Size project field by parsing each issue's "## Effort" body section. Idempotent (skips already-sized items). Defaults to M when no parseable effort line found. - ~/.claude/skills/agentkeys-issue-create — updated to: - Set Kind/Priority/Size project fields directly via API (replaces deleted label-sync workflow) - Use GitHub native relationships for blocked-by (replaces removed field) ## Live state after this change 39 open issues all have complete Kind + Priority + Size field values (36 mapped from explicit "## Effort" bodies; 3 defaulted to M for issues without parseable effort). ## What stays UI-only - The deprecated "Phase" project field still exists with v0..v4 data on issues — operator can delete in UI when ready. - The deprecated "Estimate" project field (duplicate of GitHub's built-in Size) still exists — same UI-cleanup-later. * docs: archive v1/v2 staging docs + add M1-M7 milestone roadmap The v1/v2 staged plan framing retires after v2-stage3 ships green. Going forward, milestone-level work (M1-M7) is tracked against the new docs/spec/plans/milestones-roadmap.md — the operational companion to agent-iam-strategy.md. ## Archived (moved to docs/archived/ with _2026-04 suffix) - docs/stage7-demo-and-verification.md (123KB, the big stage-7 end-to-end demo doc) - docs/operator-runbook-stage7.md (39KB, supplanted by scripts/setup-broker-host.sh) - docs/stage8-wip.md (15KB, off-chain vault design now in arch.md + threat-model) - docs/spec/plans/development-stages.md (the 8-stage v2 plan, replaced by milestones-roadmap.md) Per CLAUDE.md docs policy: archive, never delete; archived files are never read in normal dev. ## Added - docs/spec/plans/milestones-roadmap.md — M1-M7 detail + post-M7 horizons + strategic risks table + how-to-use-this-doc. Cross-references arch.md for invariants and agent-iam-strategy.md for positioning. This becomes the authoritative milestone plan from M1 onward. ## Cross-refs updated (active docs only) - docs/arch.md: §24 + §25 cross-refs now point at scripts/setup-broker-host.sh (canonical idempotent runbook) + archived stage-7 commentary for history - docs/dev-setup.md: 5 stage7/dev-stages refs → setup-broker-host.sh + milestones-roadmap.md - docs/v2-stage1-migration-and-demo.md: 4 stage7 refs → archive locations + status banner noting v1/v2 retirement after v2-stage3 - CLAUDE.md: 3 refs (build plan, runbook policy, harness workflow) → milestones-roadmap.md - docs/spec/{threat-model-key-custody,ses-email-architecture,credential-backend-interface}.md: stage8-wip refs → archive - docs/spec/heima-gaps-vs-desired-architecture.md: stage7 demo §4 → archive - docs/wiki/upstream-backend-classes-exercise-vs-distribution.md: stage7 demo refs → archive (wiki auto-publishes to GitHub Wiki via publish-wiki.yml) ## What's NOT updated (intentional) Issue-specific plan files under docs/spec/plans/issue-64/ + issue-74-* + issue-credential-storage-* still reference the archived docs by name. These are themselves historical issue-deliverable records; the references are timestamped artifacts of when those issues were planned, not active operational links. They stay as-is.
Extends crates/agentkeys-mcp/ additively with 10 new tools (7 active +
3 schema-only stubs returning not_implemented_in_v1) plus a JSON-RPC
dispatcher hook. Legacy stage-7 tools (get_credential, list_credentials,
provision) are preserved unchanged.
Tools shipped:
- agentkeys.identity.whoami (synthesized locally; broker endpoint = M4)
- agentkeys.permission.check (deterministic policy engine; payment-daily-cap)
- agentkeys.cap.mint (adapter onto broker /v1/cap/{cred,memory}-{store,fetch})
- agentkeys.cap.revoke (graceful M1 stub when broker endpoint absent)
- agentkeys.audit.append (AuditEnvelope v1 onto worker-audit /v1/audit/append/v2)
- agentkeys.memory.put / get (worker-memory adapter; namespace on request body)
- agentkeys.delegation.{grant,revoke}, agentkeys.approval.request — schema-only
#109 partial: audit worker default flush interval bumped 300s -> 120s to
match the issue's =<2min on-chain anchor SLA. Actual chain submission
(CredentialAudit.appendRootV2) still operator-driven; deferred to a
follow-up.
#108 partial: memory namespace passes at the request body level only.
Adding it as a SIGNED FIELD in CapPayload (broker + worker-creds mirror
+ verify::check_namespace) is the proper plumbing per arch.md S17;
deferred to a follow-up with the Namespace enum.
Tests: 30 passed in agentkeys-mcp (23 new M1 + 7 legacy); 14 passed in
agentkeys-worker-audit; agentkeys-daemon builds clean; harness/mcp/
smoke-test.sh acts replaced with real JSON-RPC drivers over the
daemon's stdio transport (graceful degradation when backend URLs unset).
#110 (parent UI) and #112 (Volcano Ark marketplace) explicitly deferred
to follow-up PRs per user direction. Full landed/deferred table in
docs/spec/plans/m1-mcp-server-phase1.md S8.
Co-author note: omitted intentionally per CLAUDE.md /create-pr policy
for Claude Code worktrees (correct author is the running agent identity).
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
crates/agentkeys-mcp/additively with 10 new tools (7 active + 3 schema-onlynot_implemented_in_v1stubs). Legacyget_credential/list_credentials/provisionpreserved unchanged.agentkeys-mcp(23 new M1 + 7 legacy); 14 passed inagentkeys-worker-audit; daemon builds clean.docs/spec/plans/m1-mcp-server-phase1.md§8 — the PR-summary section is intentionally there (not duplicated in the PR body) so future operators see the truth at the source.Tools shipped
agentkeys.identity.whoami/v1/identity/whoamideferred to M4agentkeys.permission.checkagentkeys.cap.mint/v1/cap/{cred,memory}-{store,fetch}; cross-actor pre-checkagentkeys.cap.revokeagentkeys.audit.appendAuditEnvelope v1adapter ontoworker-audit /v1/audit/append/v2agentkeys.memory.put/getworker-memory; namespace at request body (see deferral notes)agentkeys.delegation.{grant,revoke}+agentkeys.approval.request{"error":"not_implemented_in_v1","scheduled_for":"M4",...}#109 cadence
crates/agentkeys-worker-audit/src/main.rsdefault flush interval bumped 300s → 120s to match the issue's ≤2-min on-chain anchor SLA. ActualCredentialAudit.appendRootV2submission still operator-driven; tracked as a follow-up in the plan §8.2.#108 namespace (partial)
M1 passes
namespaceat the memory-worker request body level only. The proper plumbing — addingnamespaces_allowed: Vec<Namespace>as a SIGNED FIELD in both broker + worker-credsCapPayloadstructs — is documented as a follow-up in plan §8.2 with the exact files + functions to touch.Smoke harness
harness/mcp/smoke-test.shTODO(M1)stubs in acts 1/2/3 replaced with real JSON-RPC drivers over the daemon's stdio transport. Acts gracefully degrade (verify error-surface shapes) when backend URLs are unset and round-trip when wired.Test plan
cargo test -p agentkeys-mcp— 30/30 greencargo test -p agentkeys-worker-audit— 14/14 greencargo build -p agentkeys-daemon— cleanbash -n harness/mcp/smoke-test.sh— syntax greenSESSION_ID=alice bash harness/mcp/smoke-test.sh) — operator-driven; the full layer-4 acceptance gate peragent-iam-strategy.md§4.3Deferred to follow-up PRs (full table in plan §8.2)
CapPayloadfield — follow-up issue/v1/identity/whoami+/v1/revoke/cap/:id— M4 (paired with vendor portal Phase 2: Tuya Cloud Development connector #114)appendRootV2— follow-up issue🤖 Generated with Claude Code