Long Running AWS Connections Failing

[gsexton]

I'm running into an issue with long lived connections on AWS ECS containers failing and was hoping for some help.

I'm using:

github.com/go-openapi/runtime v0.28.0
go: 1.23.6

The application has swagger generated clients that are talking to other services. I frequently see context deadline exceed messages in my error logs. I opened a ticket and the VPC support engineer said this was a keepalive/timeout issue. The NAT gateway has an idle connection timeout of 350 seconds.

https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-troubleshooting.html#nat-gateway-troubleshooting-timeout

According to the SE, when the connection is closed by the NAT gateway, the client doesn't receive a notification, and it hangs until the context deadline expires. The solution is to either use keepalive packets, or to close the connections and re-open them before the idle connection timeout expires.

Does this diagnosis sound correct?

I looked through the docs and found Runtime.EnableConnectionReuse(), but from reading the docs and looking at the code, it doesn't look like it addresses my problem. I tried it anyhow, and it makes no difference.

I have not tried re-generating the clients using swagger. Would that help?

Any ideas would be really appreciated.

[fredbi]

The diagnosis is correct — AWS NAT's 350s idle conntrack timeout is the culprit. Runtime.EnableConnectionReuse() doesn't help here despite the suggestive name: it controls HTTP-level body draining for idle-pool reuse, not TCP-level keepalive packets. You should probably not use EnableConnectionReuse (which naming is a bit unfortunate).

The fix is one or both of:

1. Make sure your Runtime.Transport is either http.DefaultTransport or a custom transport that inherits its Dialer.KeepAlive=30s and IdleConnTimeout=90s. If you're constructing your own transport (for TLS config, proxy, etc.), you must set both explicitly.
2. On Go 1.23+, set an explicit net.Dialer.KeepAliveConfig{Enable: true, Idle: 60time.Second, Interval: 30time.Second, Count: 4} — the bare KeepAlive=30s field only sets the interval, not the initial idle delay, and the OS default for idle on Linux is often 7200s, so probes never fire before NAT drops the conntrack. With an explicit Idle under 350s, the kernel sends keepalive probes that NAT sees as live traffic.

Regenerating with newer swagger won't change this — the issue is in how Runtime.Transport is built, not in the generated code.

[fredbi]

Better late than never... Apologies for having slipped that one through.
I believe the explanation closes the issue. Feel free to reopen it if this is not the case.