Phase 1: Storage foundation for org-workspace messaging

[plur9]

Context

An audit of datacore-messaging v0.1.0 identified structural issues: duplicated code across hooks, no shared configuration, no input validation, and tight coupling that made the system hard to extend. The audit findings were reviewed by evaluator agents (CTO, Dijkstra, Feynman, User) until consensus, then used to create a detailed implementation plan.

This PR implements that plan — the storage foundation that all future messaging features will build on.

Full implementation plan: docs/superpowers/plans/2026-03-11-phase1-foundation.md

Core idea

Instead of hooks directly manipulating org files, this PR introduces a library layer (lib/) that provides:

• MessageStore — create, query, thread, and archive messages in org-mode files using org-workspace for concurrent-safe reads/writes
• AgentInbox — per-agent task queues with a full state machine (WAITING → QUEUED → WORKING → DONE → ARCHIVED, with retry and revision cycles)
• Governor — trust tiers and compute budgets that control who can send tasks and how much they cost
• Config — single source of truth for settings, identity, relay URL, and trust resolution

Hooks are rewired to import from lib/ instead of containing their own logic.

What this does NOT do

• No ActivityPub — the contacts.yaml template uses ActivityPub-style actor IDs (@user@instance) as a forward-compatible addressing format, but no federation protocol is implemented
• No fds-id integration — cryptographic identity is planned for Phase 2
• No CLI commands — datacore-msg.py is deprecated with a warning; proper commands come in Phase 2
• No GUI rewrite — the Tkinter GUI is untouched

What changed

• New: lib/config.py, lib/message_store.py, lib/agent_inbox.py, lib/governor.py, lib/relay.py, lib/_org_utils.py
• Rewired: all 4 hooks now import from lib/
• Consolidated: 3 duplicate relay scripts → 1 (lib/relay.py)
• Updated: module.yaml, CLAUDE.md, README.md, templates, settings example
• Added: UPGRADING.md migration guide, 68 tests

Test plan

cd /tmp/datacore-messaging
pip install -e ".[dev]"
pytest -v  # 68 tests

[Copilot]

Pull request overview

Introduces a new lib/ storage foundation for org-workspace–backed messaging and agent task governance, and rewires hooks/docs to use this shared library layer instead of duplicating file-parsing logic.

Changes:

• Added core library modules for config, message storage, agent task inbox lifecycle, governance, and a consolidated relay server.
• Rewired hooks to use the new library APIs; removed legacy relay/window implementations.
• Added a new test suite covering the new relay/store/inbox/governor behaviors and updated templates/docs for the v0.2.0 layout.

Reviewed changes

Copilot reviewed 42 out of 44 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
lib/config.py	Centralized settings + identity + trust tier/compute config helpers.
lib/message_store.py	org-workspace backed universal inbox storage + threading + file delivery nodes.
lib/agent_inbox.py	Agent task inbox state machine (WAITING→QUEUED→WORKING→DONE→…).
lib/governor.py	Trust tiers, budgets, and rate-limit enforcement with on-disk state.
lib/relay.py	Consolidated aiohttp WebSocket relay implementation + CLI parsing.
lib/_org_utils.py	Shared helper to refresh NodeView references across generations.
lib/__init__.py	Ensures repo root is importable when hooks run standalone.
hooks/inbox-watcher.py	Hook rewritten to use AgentInbox (claim queued tasks).
hooks/send-reply.py	Hook rewritten to use MessageStore + optionally complete tasks.
hooks/mark-message.py	Hook rewritten to mark messages read/archive via MessageStore.
hooks/task-queue.py	Hook rewritten to approve/reject/cancel tasks via AgentInbox.
tests/conftest.py	Test fixtures for temp org/messaging workspace + StateConfig.
tests/test_message_store.py	Tests for message creation, threading, querying, transitions, validation.
tests/test_agent_inbox.py	Tests for task lifecycle, retries, validation, queries, counts.
tests/test_governor.py	Tests for tier resolution, budgeting, persistence, rate limiting, effort limits.
tests/test_relay.py	Tests for relay /status privacy, empty secret refusal, arg parsing.
tests/test_config.py	Tests for settings cache behavior, defaults, overrides.
tests/test_hooks.py	Smoke tests that lib-based hook behaviors work via stores.
tests/test_integration.py	End-to-end integration test across store/inbox/governor.
templates/contacts.yaml	New contacts registry template (actors list).
templates/USERS.yaml	Removed legacy user registry template.
settings.local.yaml.example	Updated example settings (trust tiers/overrides, relay, compute).
requirements.txt	Updated runtime dependencies (includes org-workspace).
relay/datacore-msg-relay.py	Removed legacy relay implementation.
relay/Dockerfile	Updated relay container to run python -m lib.relay.
Procfile	Updated to run consolidated relay module.
pyproject.toml	Added project metadata + deps + pytest config.
module.yaml	Bumped version/schema and updated created paths + hook definitions.
datacore-msg.py	Added deprecation warning re: v0.2.0 incompatibility.
UPGRADING.md	Added migration guide for v0.1.0 → v0.2.0 layout changes.
README.md	Updated docs for new org/messaging layout, trust tiers, relay usage.
CLAUDE.md	Updated module context docs to match new storage + governance design.
commands/msg.md	Spec doc updated for new universal inbox + contacts.yaml.
commands/reply.md	Spec doc updated for new inbox path.
commands/my-messages.md	Spec doc updated for scanning universal inbox files.
commands/msg-add-user.md	Spec doc updated for contacts.yaml + messaging dir.
commands/broadcast.md	Spec doc updated for contacts.yaml + universal inbox.
agents/message-task-intake.md	New spec doc describing task intake/governance/task state machine.
agents/message-digest.md	Spec doc updated to use MessageStore and universal inbox.
agents/claude-inbox.md	Removed legacy agent spec tied to v0.1.0 layout.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

RelayServer.add_user() overwrites any existing connection for the same username without closing the old websocket or guarding removal. If a client reconnects, the old connection’s finally block will call remove_user(username) and can inadvertently delete the new session, causing flapping/disconnects. Consider tracking the ws per user and either (a) close and replace the old ws on re-auth, and/or (b) make remove_user conditional on the ws instance that’s disconnecting (e.g., remove_user(username, ws) only if current ws matches).

Suggested change

	self.users.pop(username, None)
	user = self.users.get(username)
	# Only remove the user entry if the currently-registered websocket
	# is already closed. This prevents an older connection's cleanup
	# from removing a newer active session for the same username.
	if user is not None and user.ws.closed:
	self.users.pop(username, None)

[Copilot]

check_and_record() acquires FileLock on the budget file, then calls check_task(), which calls _check_date_rollover() and may call _load_state() (which acquires FileLock again). This introduces nested locking on the same file and can deadlock or hang at midnight rollover depending on FileLock’s reentrancy. Refactor so date rollover/state loading inside locked sections uses an “_unlocked” variant (no internal FileLock) or perform rollover before entering the FileLock and then lock the correct file.

Suggested change

	with FileLock(self._budget_file):
	self._load_state_unlocked()
	result = self.check_task(actor_id, estimated_tokens=estimated_tokens, effort=effort)
	if result.allowed:
	sender = self._get_sender(actor_id)
	sender.tasks_this_hour.append(time.time())
	sender.tasks_today += 1
	sender.active_tasks += 1
	self._write_state_unlocked()
	# Perform the governance check outside of any FileLock to avoid nested locking
	result = self.check_task(actor_id, estimated_tokens=estimated_tokens, effort=effort)
	if not result.allowed:
	return result
	# Record the task under a FileLock while using *_unlocked state helpers
	with FileLock(self._budget_file):
	self._load_state_unlocked()
	sender = self._get_sender(actor_id)
	sender.tasks_this_hour.append(time.time())
	sender.tasks_today += 1
	sender.active_tasks += 1
	self._write_state_unlocked()

[Copilot]

The hook examples here don’t match the current hook CLIs:

• hooks/send-reply.py uses --complete-task, but the docs show --complete.
• hooks/task-queue.py requires --task-id (not a positional task id), but the docs show approve msg-... etc.
• “Cancel a running task” doesn’t align with AgentInbox.cancel() which only allows cancelling QUEUED tasks.
  Please update the examples or add compatible aliases/positional args in the scripts.

[Copilot]

Argument parsing expects --task-id for approve/reject/cancel, but current docs/examples in README use a positional task id. Consider supporting an optional positional task_id (and/or making --task-id required for those actions) and returning a non-zero exit with a clear message when the task id is missing or not found.

[Copilot]

The documented contacts.yaml structure here uses a top-level contacts: mapping, but the new template/migration guide uses a top-level actors: list with id, name, and trust_tier. This mismatch will confuse users and implementers; update the example/schema to match templates/contacts.yaml and UPGRADING.md.

[Copilot]

The comment says write methods enforce transitions via “_assert_state preconditions”, but MessageStore doesn’t define or use an _assert_state helper. Either add explicit state precondition checks (similar to AgentInbox) or update this comment/docstring so it reflects the actual behavior.

Suggested change

	# Edge enforcement is handled exclusively via _assert_state preconditions
	# in write methods. This allows non-linear workflows (e.g. DONE -> QUEUED).
	# MessageStore itself does not enforce transition edges; any policy checks
	# are implemented by callers (e.g. AgentInbox), allowing non-linear workflows.

[Copilot]

MessageStore tracks every returned NodeView in _live_nodes and never removes entries. In long-running processes (e.g., GUI/daemon), repeated create_message/create_file_delivery calls will cause unbounded growth and extra work in _refresh_live_nodes(). Consider pruning old nodes, using weakrefs, or tracking only nodes that callers explicitly “pin”.

[Copilot]

The README instructs using python3 datacore-msg.py send|inbox|gui, but datacore-msg.py is explicitly marked “not yet compatible with v0.2.0” and (currently) doesn’t implement these subcommands (it launches the legacy GUI instead). Either update the README usage examples to point to the supported hooks/ scripts / lib/ API, or implement an argparse-based CLI in datacore-msg.py that provides these commands.

[Copilot]

The CLI flag for completing a task is --complete-task, but existing docs/examples in README (and likely prior usage) refer to --complete. To preserve compatibility and avoid confusion, consider adding --complete as an alias or renaming the argument to --complete (keeping --complete-task as a deprecated alias if desired).

Suggested change

	parser.add_argument("--complete-task", help="Task ID to mark complete")
	parser.add_argument("--complete-task", "--complete", help="Task ID to mark complete")

[Copilot]

parse_relay_args() defines a --host flag and several callers pass it (Procfile/Dockerfile), but run_relay() doesn’t branch on args.host (it always starts the server). Either remove --host entirely or use it to control behavior (e.g., require --host to start serving, otherwise print help / run a client). Keeping an unused flag makes the CLI misleading.