Retry transient network errors in background token monitor

[reyortiz3]

Summary

• OAuth token refresh fails when WARP VPN disconnects overnight — the refresh endpoint becomes temporarily unreachable from the home ISP IP, causing the workload to be immediately marked as unauthenticated and requiring manual intervention.
• Adds isTransientNetworkError to distinguish DNS failures, TCP-level errors, and timeouts (transient) from OAuth 4xx / invalid_grant responses (non-transient). The background monitor now retries transient failures with exponential backoff (10 s initial → 2 min max, bounded by 5 attempts and 5 min elapsed time). Context cancellation exits the loop cleanly without marking the workload as unauthenticated. Non-transient errors still fail immediately as before.

Type of change

• Bug fix

Test plan

• Unit tests (task test)

Changes

File	Change
pkg/auth/monitored_token_source.go	Add isTransientNetworkError, exponential backoff retry loop in Token, injectable newRetryBackOff for tests
pkg/auth/monitored_token_source_test.go	Tests for isTransientNetworkError and the three retry scenarios: recovery, context cancellation, transient→non-transient escalation
go.mod / go.sum	Add github.com/cenkalti/backoff/v5 (direct dep), bump otel exporters and grpc-gateway (incidental go mod tidy updates)

Does this introduce a user-facing change?

No — the workload stays authenticated through VPN reconnects instead of going unauthenticated overnight. No CLI or config changes.

Special notes for reviewers

• Retries are bounded by both attempt count (WithMaxTries, default 5) and wall-clock time (WithMaxElapsedTime, default 5 min). Both limits are configurable via environment variables (TOOLHIVE_TOKEN_REFRESH_MAX_TRIES, TOOLHIVE_TOKEN_REFRESH_MAX_ELAPSED_TIME).
• The newRetryBackOff field is a test seam; it is nil in production and replaced with a fast backoff in tests to avoid multi-second sleeps.
• The code scanning alert on resolveTokenRefreshMaxTries is a false positive — ParseUint with strconv.IntSize already constrains the value to fit in uint.

Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 70.83333% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.23%. Comparing base (04ae197) to head (5a59030).
⚠️ Report is 21 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/auth/monitored_token_source.go	70.83%	26 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4281      +/-   ##
==========================================
+ Coverage   68.77%   69.23%   +0.45%     
==========================================
  Files         473      478       +5     
  Lines       47917    48415     +498     
==========================================
+ Hits        32956    33521     +565     
- Misses      12300    12310      +10     
+ Partials     2661     2584      -77     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[JAORMX]

Code looks good! One thing before we merge though...

The "Special notes for reviewers" section says MaxElapsedTime is intentionally left at zero with no deadline, and that the retry loop runs until the context is cancelled or the network recovers. But retryTransientRefresh() actually passes backoff.WithMaxTries(5) and backoff.WithMaxElapsedTime(5*time.Minute), so retries are bounded. The comment in getRetryBackOff() ("No MaxElapsedTime — context cancellation is the stop signal") is stale for the same reason.

Can you update the PR description and that code comment to match what the code actually does? The bounded retry is the right call, the docs just need to catch up :)

Note that the code scanning alert on resolveTokenRefreshMaxTries is a false positive... ParseUint with strconv.IntSize already constrains the value to fit in uint.

Concerns have been addressed