HTTPProxy reconcile backs off 3-4 min after 409 conflict burst at tunnel creation

[drewr]

This is part of a four-issue tunnel creation story

A newly created tunnel takes up to ~14 minutes before it reliably routes traffic. There are two distinct delays, each with an operator-side and a client-side component:

• Delay 1 (~3-4 min): creation → toggle turns green
  • network-services-operator#166 — NSO reconcile loop sleeps after 409 conflict burst ← this issue
  • network-services-operator#167 — NSO waits for connector heartbeat instead of pushing addressing proactively
• Delay 2 (~0-10 min): toggle green → traffic flows
  • network-services-operator#167 — NSO waits for connector heartbeat instead of pushing addressing proactively
  • app#159 — heartbeat interval can reach ~10 min
• UX consequence
  • app#160 — green toggle shown before tunnel is usable

---

Summary

When a new tunnel (HTTPProxy) is created, the NSO fires a burst of writes to the child Gateway and HTTPRoute. Multiple goroutines race and produce several 409 Conflict errors, after which the controller waits for its next periodic requeue tick (~3-4 min) rather than re-queuing immediately on conflict resolution.

Observed

Traced via audit log on tunnel tunnel-xnhnb (project drewr-y4nd1b), 2026-05-22:

Tunnel	Created	Accepted	Programmed	Gap
tunnel-xnhnb	19:33:47Z	19:33:50Z (+3s)	19:37:37Z	3m47s
tunnel-79nsr	18:19:05Z	18:19:08Z (+3s)	18:22:09Z	3m01s
tunnel-hz2v8	18:21:47Z	18:21:49Z (+2s)	18:24:06Z	2m17s

The operator writes to default/gateways/tunnel-xnhnb and default/httproutes/tunnel-xnhnb from 19:33:47Z–19:33:52Z (several 409s), then goes completely silent until 19:37:37Z — exactly the periodic requeue window — when it sets Programmed=True.

Root Cause

After the 409 conflict burst resolves, no event-driven trigger wakes the reconcile loop early. The controller relies on its periodic requeue interval, which is empirically 2-4 minutes.

Expected

After a 409 conflict, the controller should requeue with a short explicit duration (e.g. 5s) so the next reconcile attempt fires quickly once the write conflict resolves. The total creation → Programmed=True time should be well under 30s.

Impact

Every tunnel creation imposes a mandatory ~3-4 minute wait before the UI toggle turns green and the user can interact with the tunnel.