Skip to content

Pivot orchestrator to continuous-control VLA loop#8

Open
9LogM wants to merge 1 commit into
mainfrom
feat/vla-prep
Open

Pivot orchestrator to continuous-control VLA loop#8
9LogM wants to merge 1 commit into
mainfrom
feat/vla-prep

Conversation

@9LogM
Copy link
Copy Markdown
Owner

@9LogM 9LogM commented May 4, 2026

Summary

  • Replaces the symbolic VLM-per-command path (Gemini + mission state machine + re-prompt loop) with a continuous-control VLA loop. Each tick the orchestrator hands the latest RGB frame + held instruction + telemetry to a VLA backend at orchestrator.control_hz and dispatches at most one tool call.
  • Slims the tool schema to the action space the VLA actually emits: arm, disarm, set_velocity_ned, land, abort, kill. takeoff, goto_ned, return_to_launch, hold are gone — continuous velocity setpoints subsume them.
  • Default VLA backend is a no-op so the stack boots and the control loop runs without a model wired in. Drop a model into python/services/orchestrator/vla.py (OpenVLA, π0, Octo, custom) to make it act.

What changed

  • vlm_base.py / vlm_vla.py / vlm_gemini.pyvla_base.py + vla.py (Gemini deleted)
  • OrchestratorMission, _TERMINAL_TOOLS, stuck-detection — removed; nothing terminal under continuous control
  • Voice-arm latch is now an orchestrator-side flag set by the command thread; a VLA-emitted arm without a prior spoken arm is rejected
  • Frame staleness gate (orchestrator.frame_max_age_s) idles the loop when RGB is missing/old instead of letting the model extrapolate
  • Config: perception.vlm.*perception.vla.{model_path,device}; orchestrator.{mission_max_steps,telemetry_history_seconds}orchestrator.{control_hz,frame_max_age_s}
  • GEMINI_API_KEY / gemini extra removed from .env.example, pyproject.toml, Dockerfile, scripts/preflight.sh
  • TUI alarm row vlm_errorvla_error
  • Deleted docs/phase-2.md (mission-state design doc, superseded by this pivot)
  • Swept VLM/Gemini out of every comment, docstring, and README — git grep vlm gemini returns nothing

Diff stat: 23 files changed, +397 / -851 (orchestrator simplified significantly).

Tests

  • 43/43 green excluding the pre-existing replay slow-joiner flake (test_phase2_replay.py — unrelated)
  • Removed the mission-state-machine integration tests that no longer have a referent
  • Added 9 continuous-control tests in test_orch_vla.py covering: setpoint dispatch, idle on no-tool-call, voice instruction propagation, abort short-circuit, arm-authorization latch, VLA-emitted-arm without voice rejection, stale-frame idle, bridge-offline survival, VLA exception survival

Test plan

  • docker compose build succeeds; both images come up
  • lexaire-python test suite green (pytest python/tests --ignore=python/tests/integration/test_phase2_replay.py)
  • Stack boots end-to-end with perception.vla.model_path: null (no-op backend, control loop ticks idle)
  • Wire a real VLA in vla.py and verify set_velocity_ned setpoints flow through to the bridge

🤖 Generated with Claude Code

Drops the symbolic VLM + mission state machine in favor of a continuous
control loop: each tick the orchestrator hands the latest RGB frame +
held instruction + telemetry to a VLA backend and dispatches at most
one tool call. Tool schema slims to the action-space the VLA emits
(arm/disarm/set_velocity_ned/land/abort/kill).

- Replace vlm_base / vlm_vla / vlm_gemini with vla_base + vla (no-op
  default backend; drop a model into vla.py to wire one in).
- Remove OrchestratorMission, mission re-prompt loop, terminal-tool
  set, and stuck-detection — none apply under continuous control.
- Replace mission_max_steps / telemetry_history_seconds config with
  control_hz / frame_max_age_s; drop perception.vlm.* in favor of
  perception.vla.{model_path,device}.
- Voice-arm latch is now an orchestrator-side flag set by the command
  thread, not a per-command field — VLA-emitted arm without a prior
  spoken arm is rejected.
- Frame staleness gate (frame_max_age_s) idles the loop when RGB is
  missing/old instead of letting the model extrapolate.
- Drop Gemini from Dockerfile/pyproject and GEMINI_API_KEY from
  .env.example and preflight.
- TUI alarm row vlm_error -> vla_error.
- Delete docs/phase-2.md (mission-state design doc, superseded).
- Sweep all VLM/Gemini references out of comments, docstrings, and
  README; project is now genuinely VLM-free.

43/43 tests green excluding the pre-existing replay slow-joiner flake.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant