add web UI#29
Merged
tridge merged 10 commits intoArduPilot:mainfrom May 8, 2026
Merged
Conversation
tridge
force-pushed
the
pr-web-UI
branch
2 times, most recently
from
28037e6 to
0831d8b
Compare
Schema becomes append-only and forward-compatible so future fields can
be added without flag-day migrations. Readers accept any record of size
>= KEYENTRY_MIN_SIZE (96 bytes, the pre-flags layout) and zero-extend
short records; writers preserve any trailing bytes the on-disk record
has beyond the writer's sizeof(KeyEntry) so older code never truncates
fields added by newer code. db_save_key now does read-modify-write to
keep that tail.
Add a uint32_t flags field at the end of KeyEntry. Two bits for now:
KEY_FLAG_ADMIN - the entry's owner can manage every entry via
the web admin UI
KEY_FLAG_BIDI_SIGN - the proxy also requires signed MAVLink on the
user side (enforcement in udpproxy.cpp)
Split keydb.py into keydb_lib.py (importable, no top-level
side effects) and a thin CLI shim. The library is what the upcoming
webadmin app uses; the CLI keeps the same arguments/exit codes the
existing test harness and CI scripts depend on, plus three new
subcommands: setflag / clearflag / flags.
Makefile now declares mavlink.o's keydb.h dependency (mavlink.h
includes keydb.h). Without it, a struct-size change would silently
desync between compilation units, which previously corrupted memory
because mavlink.o held the old 96-byte sizeof while keydb.o held the
new 104-byte one.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two changes that work together to stop the persisted timestamp from racing ahead of real time across short-lived sessions: load_signing_key clamps signing.timestamp to max(saved+15s, now). The +15s buffer over the saved value is the existing replay guard for the window between key saves (saves are throttled to once per 10s, so a crash can lose up to that window of advancement). Taking max() with current wall clock keeps signing.timestamp aligned with real time when the saved value was written long enough ago that real time has caught up. Without this, a session whose first event is an INCOMING packet would see signing.timestamp stuck at saved+15s — update_signing_timestamp is only called from send_message and so doesn't run before the helper checks the first incoming packet's tstamp. save_signing_timestamp caps the persisted value at current wall clock. The +15s buffer added in load_signing_key() is a per-load replay guard; allowing it to round-trip through this save would let it compound across sequential short-lived sessions until signing.timestamp parks far enough in the future that incoming packets fall below signing.timestamp - MAVLINK_SIGNING_TIMESTAMP_LIMIT seconds and the helper rejects them as MAVLINK_SIGNING_STATUS_OLD_TIMESTAMP. With the cap, the stored timestamp tracks real time and the +15s buffer is reapplied fresh on each load instead of accumulating. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
handle_record now accepts records of size >= KEYENTRY_MIN_SIZE rather than requiring exact sizeof(KeyEntry), so it tolerates legacy 96-byte records (zero-extended) and future records with trailing fields it doesn't understand (truncated to known size). Wrap the tdb_traverse calls in reload_ports and main startup in a TDB transaction so they see a consistent snapshot of all entries even when keydb.py / the web admin UI is mutating in parallel. Single-record fetches (e.g. on connection close) already use transactions; this closes the consistency hole for the multi-record reads. Plumb the on-disk flags word through handle_record / add_port into struct listen_port. add_port now also refreshes flags on existing entries so a flag flipped via keydb.py / the web admin UI is picked up on the next reload (the running per-port-pair child still uses the flags it was forked with, so the change takes effect when that child idles out and the parent re-forks). When KEY_FLAG_BIDI_SIGN is set on an entry, mav1 (the user-side MAVLink instance) is initialized with signing required and key_id = port2 — so it loads the same secret_key keys.tdb already stores for the engineer side. mav1.receive_message then drops unsigned and wrong-key user packets, so they are never forwarded to the engineer. Removed the now-unused have_port2 helper (add_port iterates the list itself to update existing entries). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The keydb.py CLI shim now imports keydb_lib at the top, so the runtime stage needs both files. Without keydb_lib.py the docker-test job fails at the first 'docker run ... keydb.py initialise' with ModuleNotFoundError. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two roles using the same auth mechanism (port + passphrase): owners
edit only their own entry, anyone with KEY_FLAG_ADMIN set on their
entry can edit any entry. Sessions store the resolved port2 plus an
is_admin boolean so role checks don't reload the entry on every
request. Login accepts either port1 or port2 — find_by_port tries
port2 first then scans port1 — and uses hmac.compare_digest for the
SHA-256 comparison.
Routes:
/login, /logout — auth
/me/ — owner self-edit (name / passphrase /
bidi_sign / reset signing timestamp)
/admin/ — admin: list all entries, add new
/admin/<port2> — admin: edit any entry, including
grant/revoke FLAG_ADMIN and
FLAG_BIDI_SIGN
/admin/<port2>/delete — admin: delete with last-admin guard
Last-admin guard: revoking FLAG_ADMIN or deleting the entry is refused
when there is exactly one admin and that's the entry being changed; the
check counts admins inside the same TDB transaction as the mutation so
a concurrent grant/revoke can't slip through.
All writes (and all multi-record reads — /admin/ listing and the login
lookup) run inside a TDB transaction via the tdb_transaction /
tdb_readonly context managers in webadmin/db.py. CSRF on every form
via Flask-WTF.
Deployment: gunicorn -w 2 -b 127.0.0.1:8080 webadmin.wsgi:application,
either standalone or behind Apache via ProxyPass. ProxyFix middleware
honours X-Forwarded-Prefix so URL building works when mounted under a
path prefix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Connection tests (tests/test_connections.py + tests/conftest.py +
tests/test_config.py):
* conftest seeds a second port pair (TEST_PORTS_BIDI) with
KEY_FLAG_BIDI_SIGN already set and a per-worker self-signed cert
so udpproxy can serve WSS clients.
* BaseConnectionTest.create_connection adds 'ws' and 'wss' branches
using pymavlink's mavwebsocket_client. ssl.create_default_context
is patched session-wide so pymavlink's ws client (which builds a
fresh context on every reconnect) accepts the local cert.
* TestUDPConnections / TestTCPConnections split their three signing
scenarios into separate top-level tests so xdist can distribute
them — each scenario pays the proxy's 10s conn1-idle wait, so
keeping them sequential pinned the wall clock to the sum.
* TestBidiSigning is parametrized across all 4x4 = 16 transport
combinations of (user, engineer) for udp / tcp / ws / wss, each
against three signing scenarios (unsigned, wrong key, correct
key). 48 connection tests total.
* run_test_scenario gains 'ports' and 'user_signing_key' so the
same harness drives both regular and bidi flows. Drains every
available message per second and tallies HEARTBEAT and SYSTEM_TIME
against the full stream so a burst of buffered HEARTBEATs from
before SYSTEM_TIME forwarding starts can't mask SYSTEM_TIMEs that
arrive afterwards. Sender threads catch per-send exceptions so a
transient transport drop doesn't crash via pytest's
thread-exception plugin.
* wait_for_connection_close now waits for the parent's "Child N
exited" log, not just the child's "Closed connection", so the
next test doesn't race against parent socket reopen and see
ECONNRESET on a fresh wss: connect.
* assert_with_proxy_log helper attaches the proxy's recent
stdout/stderr to assertion failures so CI logs surface what the
proxy was doing.
Webadmin tests (tests/webadmin/):
Each test gets a fresh keys.tdb in a tmpdir seeded with a non-admin
(alice) and an admin (bob_admin). 32 tests across five files:
test_auth.py — login by port1 or port2, wrong passphrase,
unknown port, role assignment, redirects
test_owner.py — passphrase change, name change, bidi_sign
toggle, signing-timestamp reset, mismatch
rejected, admin routes 403'd
test_admin.py — list, edit, grant/revoke admin, last-admin
guard (revoke + delete), bidi_sign toggle,
port1 collision rejection, non-admin POST
blocked
test_csrf.py — POST without CSRF token rejected
test_concurrent.py — UI rename and a thread bumping counters via
the keydb_lib RMW path interleave under TDB
locking; verifies neither write clobbers
the other
Test files import constants and helpers from _test_helpers.py
rather than from .conftest because the directory is intentionally
not a Python package — the name 'webadmin' would collide with the
real webadmin/ package at the repo root.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
run_tests.py grows three modes:
-j N - N pytest-xdist workers (omit for sequential)
-j 0 - one worker per test (collect-only counts first,
then -n <count>)
--list - enumerate tests across all phases without running
--timing - capture per-test durations (setup+call+teardown)
and print sorted ascending so the slowest test
ends up at the bottom of the output. Forces
pytest --color=yes when our stdout is a TTY so
the captured output keeps PASSED/FAILED color.
Positional selectors. Tokens with /, ::, or .py keep their existing
meaning as paths/NodeIDs. Bare words become a case-insensitive
substring filter via pytest -k (multiple bare words OR'd together).
With selectors the runner does one pytest invocation instead of the
three default phases. Adds a third pytest invocation in default mode
for tests/webadmin/, kept separate from the connection / authentication
phases so each can have isolated cwd / keys.tdb / process expectations.
run_webui.sh launcher for local web UI testing — run from any
directory containing keys.tdb (typically test/), passes
WEBADMIN_KEYDB_PATH=$(pwd)/keys.tdb plus a per-cwd .webadmin_secret so
sessions survive restarts. Uses gunicorn when available, falls back
to Flask's dev server otherwise. WEBADMIN_TLS=1 picks up
fullchain.pem/privkey.pem in cwd for HTTPS dev.
setup_ci.sh installs Flask, Flask-WTF, and wsproto for the webadmin
app and ws/wss test transports.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pull_request trigger fires on the receiving repo (typically ArduPilot/UDPProxy), so a tridge fork pushing to a feature branch gets no pre-PR CI runs. Add 'pr-*' to the push trigger's branches filter so feature-branch pushes on either fork run CI before the PR is opened. GitHub-hosted ubuntu-latest runners have plenty of headroom for an IO-bound test suite where the dominant cost is the proxy's 10s conn1-idle timeout. -j 16 distributes the connection-phase tests close to one-per-worker, dropping the wall clock substantially. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a Web Admin UI section covering install, first-admin bootstrap, standalone gunicorn, and Apache ProxyPass deployment, plus an env-var configuration table and a pointer to scripts/run_webui.sh for local testing. Document the new keydb.py setflag / clearflag / flags subcommands (used by the web UI for the admin and bidi_sign role flags), and note the existing resettimestamp command. Replace the Testing section with the current run_tests.py usage: -j N parallelism, -j 0 for one-worker-per-test, --list, --timing, and pytest selector pass-through for targeting specific tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One-shot deploy script for the standard layout: a checkout in
~/UDPProxy and the data directory in ~/proxy. Pass it one or more
[user@]host arguments and it:
1. rsync's the local working tree (including the mavlink submodule
and any uncommitted changes) to ~/UDPProxy on each host with
--delete, excluding build artifacts and runtime data so it
can't clobber keys.tdb, proxy.log, the live cert, etc.
2. ssh's in and rebuilds from clean (make distclean && make).
3. kills any running udpproxy bound to ~/UDPProxy/udpproxy and
re-runs ~/UDPProxy/start_proxy.sh so the new binary takes over
immediately (the cron-based respawn would do it within a minute
anyway, but this avoids the gap).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
added web admin interface for end-users and admins
example: