Pivot orchestrator to continuous-control VLA loop#8
Open
9LogM wants to merge 1 commit into
Open
Conversation
Drops the symbolic VLM + mission state machine in favor of a continuous
control loop: each tick the orchestrator hands the latest RGB frame +
held instruction + telemetry to a VLA backend and dispatches at most
one tool call. Tool schema slims to the action-space the VLA emits
(arm/disarm/set_velocity_ned/land/abort/kill).
- Replace vlm_base / vlm_vla / vlm_gemini with vla_base + vla (no-op
default backend; drop a model into vla.py to wire one in).
- Remove OrchestratorMission, mission re-prompt loop, terminal-tool
set, and stuck-detection — none apply under continuous control.
- Replace mission_max_steps / telemetry_history_seconds config with
control_hz / frame_max_age_s; drop perception.vlm.* in favor of
perception.vla.{model_path,device}.
- Voice-arm latch is now an orchestrator-side flag set by the command
thread, not a per-command field — VLA-emitted arm without a prior
spoken arm is rejected.
- Frame staleness gate (frame_max_age_s) idles the loop when RGB is
missing/old instead of letting the model extrapolate.
- Drop Gemini from Dockerfile/pyproject and GEMINI_API_KEY from
.env.example and preflight.
- TUI alarm row vlm_error -> vla_error.
- Delete docs/phase-2.md (mission-state design doc, superseded).
- Sweep all VLM/Gemini references out of comments, docstrings, and
README; project is now genuinely VLM-free.
43/43 tests green excluding the pre-existing replay slow-joiner flake.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
orchestrator.control_hzand dispatches at most one tool call.arm,disarm,set_velocity_ned,land,abort,kill.takeoff,goto_ned,return_to_launch,holdare gone — continuous velocity setpoints subsume them.python/services/orchestrator/vla.py(OpenVLA, π0, Octo, custom) to make it act.What changed
vlm_base.py/vlm_vla.py/vlm_gemini.py→vla_base.py+vla.py(Gemini deleted)OrchestratorMission,_TERMINAL_TOOLS, stuck-detection — removed; nothing terminal under continuous controlarmwithout a prior spoken arm is rejectedorchestrator.frame_max_age_s) idles the loop when RGB is missing/old instead of letting the model extrapolateperception.vlm.*→perception.vla.{model_path,device};orchestrator.{mission_max_steps,telemetry_history_seconds}→orchestrator.{control_hz,frame_max_age_s}GEMINI_API_KEY/geminiextra removed from.env.example,pyproject.toml,Dockerfile,scripts/preflight.shvlm_error→vla_errordocs/phase-2.md(mission-state design doc, superseded by this pivot)git grep vlm geminireturns nothingDiff stat: 23 files changed, +397 / -851 (orchestrator simplified significantly).
Tests
test_phase2_replay.py— unrelated)test_orch_vla.pycovering: setpoint dispatch, idle on no-tool-call, voice instruction propagation, abort short-circuit, arm-authorization latch, VLA-emitted-arm without voice rejection, stale-frame idle, bridge-offline survival, VLA exception survivalTest plan
docker compose buildsucceeds; both images come uplexaire-pythontest suite green (pytest python/tests --ignore=python/tests/integration/test_phase2_replay.py)perception.vla.model_path: null(no-op backend, control loop ticks idle)vla.pyand verifyset_velocity_nedsetpoints flow through to the bridge🤖 Generated with Claude Code