group: add MutableAutoSelect outbound

[garmr-ulfr]

Summary

New mutableautoselect outbound group that complements mutableurltest by distinguishing probe-URL health from real user-traffic health. Targets the connection-oscillation case (engineering#3484) where a tunnel handshakes successfully, probes pass on every cycle, but bytes stop flowing — and the URL-test layer keeps re-picking the same dead server because the probe URL says it's healthy.

Why

MutableURLTest treats a passing probe URL as full health. In hostile networks where DPI lets the probe through but throttles user traffic — or where a tunnel handshakes successfully but stops carrying data — a failing outbound looks healthy on every cycle. There is no signal that distinguishes "this server probes fine" from "this server is actually carrying bytes."

What's in this PR

Laundering-resistant user-failure window

Each member carries a sliding window of user_failures timestamps, separate from consecutive_failures. Appended on dial errors, listen errors, and data-plane stalls. Probe successes never touch it — a passing probe URL cannot launder a failing outbound. Failures age out naturally over userFailureWindow (default 5 min) without depending on traffic being routed through the demoted member. A userFailureDedupeWindow (default 30 s) collapses bursts on the same outbound into one entry, so a broken outbound with N orphaned keep-alive conns hitting their stall timer in sequence doesn't inflate one event into many.

Data-plane stall watchdog

Every dialed conn and listened packet conn is wrapped (dataPlaneStream / dataPlanePacket) with a per-conn time.AfterFunc idle timer (default 60 s). Two gates have to clear before a stall counts:

• proven — the conn has delivered ≥ provedReadBytes (default 4096) of cumulative non-empty Read payload, distinguishing a stalled tunnel from a handshake-only or keepalive-only conn that never carried traffic.
• last-was-write — the most recent non-empty IO was a Write, i.e. we sent bytes and got nothing back. A proven conn whose last activity was a Read is the user going idle on a healthy keep-alive, not a broken tunnel.

When both gates clear and the timer expires, the wrapper appends one user-failure timestamp and triggers the reconnection ladder. CAS-guarded one-shot per conn so a late read returning bytes just before Close can't deliver a phantom stall.

Three-tier demote ranking, per network

Candidates rank by (demoteLevel, candidateKind, delayMs):

• clean — fewer than softFailLimit user-failures in window, and consecutive-failure threshold not tripped.
• soft — userFailsInWindow >= softFailLimit (default 2). Loses to every clean peer but beats hard. A single transient stall is not enough; two within userFailureWindow is corroboration.
• hard — consec >= consecutiveFailLimit or userFailsInWindow >= consecutiveFailLimit (default 3).

splitHealthyFor filters by network first, then returns clean → soft → filtered (hard as last resort). Per-network filtering matters because a clean TCP-only candidate must not starve a soft UDP-only peer.

Switch-penalty boost

The hard-demote threshold doubles when the candidate has a real-seeded delay and the best real-seeded alternative is more than 3× slower (switchPenaltyAltFactor). Hard-demoting a 100 ms member onto a 990 ms alternative on three transient failures is itself a cost; the boost requires more evidence when switching is meaningfully worse. Synthetic and unknown delays are excluded from the bestAlt comparison so the rescue is anchored on real measurements only.

Switch-tolerance hysteresis

applyStickiness keeps the previous sticky tag unless a new winner beats it by SwitchToleranceMs (default 50 ms). Dampens churn from tiny probe-time deltas. TCP and UDP have independent sticky tags.

Dial-site fast failover

Before the ladder runs, a DialContext / ListenPacket error retries the dial once against the next-best member from the current rank, excluding the failing tag. The active 3-minute probe cadence keeps history fresh enough to pick a working alternative without re-probing. Only one alternate is attempted; the dial caller already retries on its own schedule.

Reconnection ladder

Triggered from dial/listen errors (after the fast-failover step) and from the stall watchdog. Full parallel re-probe across the candidate pool inside a ladderTotalBudgetSeconds (default 10 s) deadline. The ladder kick is gated on the user-failure dedupe: if the triggering event was deduped, the ladder skips so orphan-conn cascades don't trigger N redundant probe sweeps. If no candidate succeeds within the budget, the group emits one value on the buffered exhaustion channel. Concurrent ladder invocations collapse via a CAS guard.

Adaptive background probe cadence

A low-priority background loop keeps alternative members warm, registered with sing-box's pause manager. Faster active interval (default 180 s) when traffic has flowed recently through a wrapped conn; slower idle interval (default 900 s) otherwise. Brand-new groups with no observed traffic are treated as idle so we don't burn the fast cadence on a tunnel nobody is using.

Persistent history

A new AutoSelectHistoryStorage service interface lets hosts plug in durable storage. The group writes a full TagHistory snapshot (probe scalars + user_failures window) after every state-changing mutation. The included in-memory implementation is the default; hosts wiring durable storage register their own and wire SetHook to flush snapshots out-of-band. On Add, history is lazy-hydrated; persisted entries older than maxPersistedAgeSeconds (default 15 min) are dropped on hydrate so a stale snapshot from a previous poll window doesn't start the member innocent. On Remove or URL-override change, the affected entries are dropped.

Exhaustion signal

ExhaustionSignaler interface. The group sends a value on a buffered channel when the ladder's full budget elapses with no successful candidate, letting the host refetch /config-new instead of endlessly re-probing the same dead set.

Per-protocol probe behavior

Each protocol gets a probe timeout suited to its handshake cost: longer for multi-step handshakes like algeneva/ssh/shadowtls/reflex/trojan, longest for self-probing protocols like outline (StrategyFinder discovers a winning strategy on first use), and shorter for UDP-based protocols where blocked ports time out silently rather than failing fast. Peer/network protocols (tor, unbounded) are excluded from the candidate pool entirely — they run alongside the group with their own connection managers.

Candidate kind (tie-break under demote tier)

• kindRealSeeded — measured delay from a real probe.
• kindUnknown — no in-memory data, no persisted seed.

Configurable knobs

option.MutableAutoSelectOutboundOptions exposes every threshold (demote limits, user-failure window + dedupe window, max persisted age, cadence intervals, idle threshold, switch tolerance, ladder budget, data-plane idle + proved-read bytes) as a config field with documented defaults.

[Copilot]

Pull request overview

This PR introduces a new outbound group type, mutableautoselect, intended to improve server selection robustness versus mutableurltest by separating probe-URL health from real user-traffic health and by demoting/avoiding candidates that “probe fine” but stall under actual traffic.

Changes:

• Add MutableAutoSelect outbound group with parallel probing, user-failure tracking, stall watchdog wrappers, reconnection ladder, stickiness hysteresis, and an exhaustion signal.
• Add persistent history plumbing via AutoSelectHistoryStorage (with an in-memory default implementation) to carry health signals across restarts.
• Extend tests and mocks to cover per-network candidate selection and the new group’s ranking/demotion behavior.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
protocol/register.go	Registers the new mutableautoselect group type with the outbound registry.
protocol/group/urltest_test.go	Updates the shared test mock to support per-outbound network lists.
protocol/group/mutableautoselect.go	Core MutableAutoSelect implementation: selection, probing lifecycle, ladder, stickiness, exhaustion signaling, background loop.
protocol/group/mutableautoselect_test.go	Comprehensive unit tests covering ranking tiers, stickiness, ladder behavior, history persistence, stall wrappers, and pause-manager integration.
protocol/group/mutableautoselect_protocols.go	Defines per-protocol probe behavior (timeouts, pool exclusions, substituted delays).
protocol/group/mutableautoselect_probe.go	Implements per-member HTTP probe execution and bounded-concurrency probe fan-out.
protocol/group/mutableautoselect_history.go	Implements in-memory history ring, demotion logic, and hydration/snapshot conversions to persisted form.
protocol/group/mutableautoselect_dataplane.go	Adds data-plane stream/packet wrappers with idle-stall watchdog and activity hooks.
protocol/group/fallback_test.go	Extends the shared mock outbound struct used across group tests.
option/group.go	Adds MutableAutoSelectOutboundOptions and documents default behaviors/knobs.
constant/proxy.go	Adds TypeMutableAutoSelect constant.
adapter/group.go	Adds ExhaustionSignaler interface.
adapter/autoselect_history.go	Adds AutoSelectHistoryStorage interface + in-memory implementation and TagHistory/Outcome types.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.