Add optional Codex adapter

[SiyaoZheng]

Description | 描述

This PR proposes an optional Codex adapter for TencentDB Agent Memory, supporting both Codex CLI and Codex App. It is more than a Codex manifest: it wires Codex hooks, MCP tools, JSONL history, and context injection into the shared Gateway/Core pipeline, keeps Codex-specific state under codex-plugin/, and leaves the existing OpenClaw and Hermes loading paths unchanged.

The main contribution is a complete Codex memory path around the existing four-layer memory system:

• Codex lifecycle coverage: SessionStart, UserPromptSubmit, PreToolUse, PostToolUse, PermissionRequest, PreCompact, PostCompact, and Stop hook handlers for recall, capture, tool/permission activity, compaction safety, and periodic flushes.
• Codex CLI and App packaging: a Codex plugin manifest, bundled hook config, bundled MCP config, CLI-compatible user-level hook fallback, and documented local-development registration path.
• Codex App adaptation and validation: App-session and archived JSONL import, plugin-cache loading behavior, App-observed hook execution, and real Codex App / local Codex history validation.
• Active recall and capture resilience: bounded additionalContext injection, injected-memory tag cleanup before persistence, pending-turn preservation when capture fails, project-scoped session-key prefixes, and local L0 JSONL fallback when the Gateway is unavailable or returns no useful context.
• Short-term context offload: large PostToolUse results are redacted, stored under local JSONL/ref/Mermaid artifacts, replaced with compact hook feedback, and retrievable through tdai_offload_lookup or the CLI.
• Historical onboarding: a Codex JSONL importer converts active and archived ~/.codex sessions into Gateway /seed input, supports dry-run/cwd/time filters, and can wait for the full L0 -> L1 -> L2 -> L3 pipeline.
• Model-facing retrieval tools: MCP tools for scoped L1 memory search, L0 conversation search, and offload lookup, with cross-project search and exact offload content disabled unless explicitly enabled outside the model context.
• Host-neutral Gateway/seed support: tdai-memory-gateway package bin, token-file auth, root/health endpoint support, optional started_at metadata, session-prefix search filtering, opt-in full-pipeline /seed waiting, and a prepack-safe optional script build path.
• Security and operability hardening: loopback-first Gateway policy, bearer auth, private token/state/offload files, atomic generated-token creation under concurrent autostart, redacted hook.log diagnostics, non-loopback request refusal by default, blocked credential-bearing /seed config_override keys, symlink-safe standalone file tools, and secure TLS defaults.

I am happy to split the Gateway/seed/security support from the Codex adapter files if that would make review easier.

Related Issue | 关联 Issue

Related to #5 and #6.

Change Type | 修改类型

• Bug fix | Bug 修复
• New feature | 新功能
• Documentation update | 文档更新
• Code optimization | 代码优化

Self-test Checklist | 自测清单

• Verified locally | 本地验证通过
• No existing features affected | 无影响现有功能
• npm test -> 12 files / 67 tests passed
• find codex-plugin/scripts -name '*.mjs' -print0 | xargs -0 -n1 node --check
• git diff --check
• npm run build
• npm pack --dry-run runs prepack, includes dist/src/gateway/cli.mjs, codex-plugin/.codex-plugin/plugin.json, codex-plugin/.mcp.json, codex-plugin/hooks/hooks.codex.json, and runtime Codex scripts
• Codex CLI smoke: codex exec --json in this repo returned the expected response and created a captured Codex session with l0_recorded: 2
• Gateway CLI smoke: dist/src/gateway/cli.mjs starts with token-file auth and /health returns 200
• Gateway CLI tilde token-path smoke: TDAI_TOKEN_PATH=~/... expands consistently for adapter and spawned Gateway env
• Concurrent token smoke: generated Gateway token file is created atomically and reused across concurrent autostarts
• Gateway security smoke: unauthenticated tokenized request -> 401, authenticated -> 200, disallowed CORS origin -> 403, credential-bearing /seed config_override.llm.baseUrl -> 400, tokenless loopback POST -> 401 by default, explicit loopback dev opt-in reaches route validation
• MCP smoke: default model-facing schema exposes scoped tools only; cross-project and exact offload content require explicit env opt-in
• Scoped search regression: current-project results still return without expanding scoped searches to the full store
• Codex App prompt-capture regression: hook payload message/content arrays and transcript JSONL fallback preserve the real user prompt; assistant and synthetic interruption messages are ignored
• Diagnostics regression: hook failures append redacted entries to private hook.log and query status reports log paths
• Hook/CLI smoke tests cover hook entry envelopes, query status, and Codex JSONL importer dry-run
• Adapter doctor: TDAI_CODEX_AUTOSTART=false TDAI_CODEX_GATEWAY_URL=http://127.0.0.1:9 node codex-plugin/scripts/doctor.mjs --json -> ok: true
• Importer dry-run smoke prepared 1 session, 1 round, and 2 messages
• Privacy scan over the working tree found no sensitive private wording
• npm view @tencentdb-agent-memory/memory-tencentdb@0.3.4 bin checked; default package-bin launch no longer pins to a published version without tdai-memory-gateway

Additional Notes | 其他说明

Maintainer Review Checklist | 维护者评审确认

• External scan for latest head 3ff37c353b724c35f53ca024a76603358cb74807 is in progress. Previous GITHUB_OPENSOURCE_SCAN_PIPELINE passed on head commit 8b62ade9414547c2a20d2fe4a887683759c273f5.
• GitHub Actions CI is fork-gated and currently awaits maintainer approval; the equivalent local checks are listed above to avoid asking maintainers to approve repeated exploratory runs.
• DCO is complete: every commit in this PR includes Signed-off-by: Siyao Zheng <zhengsy24@sjtu.edu.cn>.
• Codex host testing is documented: the checklist above includes Codex CLI smoke, Codex App/local history validation, bundled MCP packaging, Gateway smokes, security smokes, and package dry-run verification.

Relation to PR #7 | 与 PR #7 的关系

Thank you to the author of #7 for pushing the host-adapter direction forward. It may be useful to review this PR alongside #7 because the two PRs intentionally overlap in Gateway, daemon, auth, and packaging concerns while targeting different host surfaces.

#7 is valuable Claude Code + Codex CLI plugin work: it adds dual manifests, a shared hook path, daemon/token handling, manual memory skills, and a useful test baseline. This PR is complementary rather than a replacement: it contributes a fuller Codex adapter that covers Codex CLI and Codex App, with additional Codex App adaptation and validation.

The practical difference is the Codex feature surface covered here. This PR implements Codex lifecycle and retrieval paths that are not covered by #7's current scope, including tool and permission hooks, compact hooks, bundled MCP tools, large tool-output offload/lookup, local L0 fallback, Codex JSONL importer, scoped offload lookup, injected-memory cleanup, and defensive Gateway/security/packaging checks.

To reduce duplicate review effort, this PR also incorporates compatible host-neutral pieces in the same spirit: the tdai-memory-gateway package bin, bearer token-file handling, Gateway auth/CORS hardening, packaging fixes, and targeted tests. I am happy to rebase on #7, split this into smaller PRs, or fold the Codex-specific pieces into a combined host-plugin structure if maintainers prefer that path.

Security posture | 安全边界

The intent is balanced local-agent security rather than maximum lock-down:

• tokenless Gateway access exposes loopback GET probes only by default; loopback tokenless POST requires explicit development opt-in;
• non-loopback Gateway access requires explicit opt-in and bearer auth;
• Codex adapter requests refuse non-loopback Gateway URLs unless explicitly allowed;
• token files are private, owner-checked, and fail closed if configured but unreadable or empty;
• generated Gateway token files are atomically created and reused under concurrent autostart;
• hook diagnostics are written to a private hook.log after redaction;
• MCP defaults are project-scoped and do not expose cross-project or exact offload content unless enabled outside the model context;
• adapter state/offload files are private on POSIX filesystems;
• file-tool sandboxing rejects symlink escapes, including existing file symlinks;
• offload/backend TLS verification is secure by default.

DCO

All commits are signed off as:

Signed-off-by: Siyao Zheng <zhengsy24@sjtu.edu.cn>

[YOMXXX]

Hi @SiyaoZheng — thanks for the very detailed PR description and the explicit "complementary, not replacement" framing on #7. Replying here to mirror what I just posted on #7 and offer a concrete sequence. Pinging @Maxwell-Code07 for triage awareness.

Overlap I noticed reading your PR description

Your "Security and operability hardening" bullets describe the same surface as four small follow-up PRs I opened off #7, each isolated to one concern:

#46 description bullet	Matching small PR
"loopback-first Gateway policy, non-loopback request refusal by default, atomic generated-token creation"	#39 — CORS opt-in + Host allowlist + shared startup-safety helpers
"secure TLS defaults"	#42 — offload backend TLS verification (closes #8)
OOM-guard surface around parseJsonBody	#51 — request body size cap → 413
"injected-memory tag cleanup before persistence" (adjacent angle)	#52 — before_message_write cache-friendly default (closes #11)

Each of these is ≤200 LOC, has its own isolated test file under src/gateway/__tests__/ (or src/__tests__/ for #52), and is independently mergeable against current main.

You explicitly offered to "split the Gateway/seed/security support from the Codex adapter files if that would make review easier" — yes, that would help me a lot too. Concretely, if reviewers land #39 / #42 / #51 / #52 first, your src/gateway/* diff in #46 collapses to mostly the Codex-specific additions, and the review surface for #46 gets much more tractable.

On the Codex side

Your codex-plugin/ directory with 8 lifecycle hooks, MCP tools server, and the Codex JSONL importer goes well beyond the manifest stub in #7 — that's the right place for the full Codex story. If #46 lands, I'm happy to drop the .codex-plugin/plugin.json stub from #7 in a follow-up commit and have #7 stay strictly cc-only.

Proposed sequence (mirroring what I wrote on #7)

1. Land the four small Gateway PRs first (fix(gateway): harden CORS / Host header / startup host binding #39, fix(offload): enable TLS verification by default; add env opt-out + CA path #42, fix(gateway): cap request body size at 8 MiB; return 413 instead of OOM #51, fix(plugin): make before_message_write hook a no-op by default (cache-friendly) #52). Each is small, focused, independently mergeable.
2. Rebase Add optional Codex adapter #46 on the new main — its Gateway hardening either becomes no-ops or merges cleanly against the smaller hand-rolled diffs.
3. Merge Adds Claude Code (cc) as a first-class memory host alongside OpenClaw and Hermes #7 for cc-only.
4. Merge Add optional Codex adapter #46 for the full Codex adapter.

Deferring to reviewers on the actual order. Wanted to surface this so we don't end up with three rounds of rebase-conflict ping-pong on the shared Gateway files.

[Maxwell-Code07]

Hi @SiyaoZheng,

We've received your PR for the Codex adapter integration. Thanks for the comprehensive work! Also thanks @YOMXXX for the detailed review and overlap analysis — community coordination like this is very helpful 👍. We'll review internally and work out the merge sequence.

[SiyaoZheng]

Thanks @YOMXXX and @Maxwell-Code07 — this sequencing discussion is helpful.

I have marked #46 ready for review now so maintainers can evaluate both the Codex adapter and the merge order explicitly. I agree the PR is large, so I am happy to follow the maintainer-preferred sequence:

• If fix(gateway): harden CORS / Host header / startup host binding #39 / fix(offload): enable TLS verification by default; add env opt-out + CA path #42 / fix(gateway): cap request body size at 8 MiB; return 413 instead of OOM #51 / fix(plugin): make before_message_write hook a no-op by default (cache-friendly) #52 land first, I can rebase Add optional Codex adapter #46 afterward and remove or narrow any duplicated Gateway/security hardening.
• I would prefer to keep the Codex-specific surface together in Add optional Codex adapter #46 (codex-plugin/, lifecycle hooks, MCP tools, JSONL importer, offload lookup, scoped recall/capture), because those pieces are coupled and easier to validate as one end-to-end adapter.

Latest local validation is reflected in the PR body: npm test passes 12 files / 67 tests, build passes, Codex hook/CLI smoke tests pass, importer dry-run smoke passes, adapter doctor is OK, and CodeCC is green on the current head.