Connection terminated unexpectedly with postgresql since v2.5.0

[brunob]

Hi, sorry to mention an old release, but we are facing this bug since months at infini.fr and we are stuck on 2.4.2.

We use etherpad with postgresql for years and it works well for us. Our setup is a patroni 3 cluster running pgsql 17 accessible on a virtual IP provided by haproxy.

Since version 2.5.0 of etherpad we observe that the service is restarting every 5 minutes with the following error :

[2026-04-28T10:24:21.780] [ERROR] server - Error: Connection terminated unexpectedly
    at Connection.<anonymous> (/opt/etherpad/master/node_modules/.pnpm/pg@8.20.0/node_modules/pg/lib/client.js:180:73)

We've tried to update etherpad to v2.7.2 and the error is the same. We've also cloned the vm, updated nodejs to v24 and tested with latest etherpad 3.2.0, same bug.

In order to narrow the bug, we've tried to bypass haproxy and plugged etherpad directly on the pgsql node which was leader on the cluster. This eliminate the bug. So, we played timeout server & timeout client parameters on haproxy for pgsql backend/fronted and observed that etherpad was restarted every 5 minutes if we apply 5 minutes for theses parameters, and every 30 minutes if we apply 30 minutes for theses parameters.

So, it seems a lib used by etherpad (ueberdb2 or node-postgres) change the way it handles closing of idle connections to pgsql since version 2.5.0. I've seen that this version introduce a bump to ueberdb2 from 5.0.14 to 5.0.15 with f44ee8b which introduce a bump to pg from 8.13.3 to 8.15.1 with ether/ueberDB@90f3ce9 but i can't find any mention to this change of behavior related to this in the issues of these repos.

I've read this porsager/postgres#310 (comment) about haproxy and tcp keep alives but this seems to be enabled by default in node-postgres lib.

Does anyone have faced this kind of bug ?

[JohnMcLear]

Thanks for the exceptionally clear report @brunob — the haproxy timeout server/timeout client correlation and the "direct to leader = no bug" test made this straightforward to pin down. You were right that it's in the ueberdb2/pg layer.

What's happening

Two things in ueberDB's postgres driver combine:

1. The pool keeps connections warm but silent. It sets min: 4 and never sets keepAlive (so it defaults to false). In pg-pool, idleTimeoutMillis only reaps connections above min, so 4 connections sit idle indefinitely with zero traffic. haproxy's timeout server/timeout client then closes those idle TCP sockets — hence the restart interval tracking your timeout exactly, and bypassing haproxy making it vanish.

2. There was no pool error handler. pg re-emits idle-connection errors on the pool object; with no listener, Node turns the 'error' event into an uncaught exception and kills the process → your supervisor restarts Etherpad. That uncaught Connection terminated unexpectedly is the crash.

On the "why since 2.5.0" question: honoring min as a real minimum-pool-size landed in pg 8.16.0. Since Etherpad floats pg: ^8, current installs resolve ≥8.16, which is when the 4 warm idle connections started persisting long enough for haproxy to reap them. That lines up with your bisection to the ueberdb2/pg bump, and with fresh 2.5.x and 3.2.0 both reproducing.

Workaround you can apply right now (no upgrade needed)

Add TCP keep-alive to your postgres dbSettings in settings.json so the idle sockets aren't silent. The initial delay must be shorter than your haproxy timeout:

"dbType": "postgres",
"dbSettings": {
  "host": "...",
  "port": 5432,
  "database": "...",
  "user": "...",
  "password": "...",
  "keepAlive": true,
  "keepAliveInitialDelayMillis": 60000
}

(Note: setting "min": 0 does not work as a workaround — the driver does min || 4, so 0 is clobbered back to 4. keepAlive is the reliable lever.) Alternatively/additionally, raise haproxy's timeout server/timeout client for the pgsql backend, or enable TCP keepalive on the haproxy side.

Proper fix

ueberDB PR: ether/ueberDB#998 — attaches a pool error handler (so a dropped idle connection is logged and discarded instead of crashing the process) and defaults keepAlive: true. Once that's released and Etherpad bumps ueberdb2, this should be fixed out of the box.

Could you try the keepAlive workaround and confirm the 5-minute restarts stop? That would also confirm the diagnosis end-to-end before the fix ships.

[brunob]

Hi @JohnMcLear and thx for this kind and detailled answer :)

I've tested 3.2.0 with these additions in dbSettings part of setting.json :

    "keepAlive": true,
    "keepAliveInitialDelayMillis": 50000,

Still getting the same error, also with 20000 despite the fact that the default timeout server & client of haproxy is set to 1m (60s) :\

[JohnMcLear]

Thanks for testing so quickly — and that result is actually really useful, because it corrects part of what I told you. My apologies: keepAlive does not defeat HAProxy's timeout server/timeout client.

I checked two things after your report:

1. The setting is reaching pg. Etherpad passes dbSettings verbatim into new pg.Pool(...), so keepAlive/keepAliveInitialDelayMillis were applied. The problem isn't that it was ignored.
2. TCP keep-alive doesn't reset HAProxy's idle timer. timeout server/timeout client measure data inactivity. TCP keep-alive probes are empty, kernel-level segments — HAProxy (in userspace) never sees them as activity, so it still closes the connection right on schedule. Keep-alive only helps against kernel/NAT/firewall/conntrack idle-state expiry, not an application-layer proxy. That's exactly why 20s/50s made no difference against your 60s timeout.

So there are really two separate facts here:

• HAProxy closing an idle pooled connection is expected and benign — the pool just needs to drop that connection and open a fresh one on next use.
• The crash is a separate bug: ueberDB never attached an error listener to the pg pool, so that routine idle-disconnect became an uncaught EventEmitter 'error' and took the whole process down. That is what PR fix(postgres): handle pool errors + enable TCP keep-alive (fixes ether/etherpad#7878) ueberDB#998 fixes. Once it ships, HAProxy can keep closing idle connections all it likes — Etherpad will log it and reconnect instead of restarting.

What to do right now (stock 3.2.0, before #998 ships)

There's no settings-only fix that fully prevents this on stock 3.2.0 (the pool keeps min: 4 warm connections that HAProxy will reap, and the min default can't be lowered to 0 via settings due to a min || 4 clobber). So the effective interim options are:

1. Raise the HAProxy idle timeout for the postgres backend. In mode tcp the right knob for long-lived connections is timeout tunnel (it governs the established bidirectional phase and overrides timeout client/server there). Something like:

   backend pgsql
       mode tcp
       timeout tunnel 1h        # or longer
       timeout server 1h
       timeout client 1h
   

   This makes the reaping rare rather than every 60s. (It doesn't make Etherpad immune — a Patroni failover or HAProxy reload still drops connections, and stock 3.2.0 will still crash on those. Bump API version to v1.1 #998 is what makes all of these harmless.)
2. Patch in Bump API version to v1.1 #998 locally if you'd rather fix the root cause now — it's a one-liner (pool.on('error', …)) in databases/postgres_db.ts.

Could you try timeout tunnel/raised timeouts on the pgsql backend and confirm the every-60s restarts stop? And I've corrected the PR docs so they don't repeat my keep-alive mistake. Thanks again for the careful testing — it materially improved the fix.

Also I don't think pg is the only db that can fail here, I this mysql and redis can fail in the same way

[brunob]

Thx again for answering !

i've already tried to raise timeout server & timeout client on backend pgsql haproxy config as i stated in the first post :

So, we played timeout server & timeout client parameters on haproxy for pgsql backend/fronted and observed that etherpad was restarted every 5 minutes if we apply 5 minutes for theses parameters, and every 30 minutes if we apply 30 minutes for theses parameters.

So it already "works" without playing with timeout tunnel.

I'm trying to find the way to apply ether/ueberDB@0ea0206 without luck for now, i'll give feedback asap :)

[JohnMcLear]

You're completely right — raising timeout server/timeout client just moves the restart interval (5m→5m, 30m→30m); it never fixes it, because the warm idle connections still get reaped, they just get reaped less often. And timeout tunnel would only push that out further (e.g. hourly). The only real fix is the pool error handler — so let's get #998 onto your install so you can confirm it.

Applying #998 to a stock 3.2.0 install (no rebuild needed)

ueberDB ships compiled-but-readable JS, so you can add the one-liner directly. From your Etherpad root (e.g. /opt/etherpad/master):

# 1) find the compiled postgres driver inside the installed ueberdb2
grep -rl 'new pg.Pool(this.settings)' \
  node_modules/.pnpm/ueberdb2@*/node_modules/ueberdb2/dist/ 2>/dev/null

That prints one file (a dist/postgres_db-XXXXXXXX.js). Open it and find this line:

this.db = new pg.Pool(this.settings);

Add the handler immediately after it:

this.db = new pg.Pool(this.settings);
this.db.on("error", (err) => {
  console.error("ueberdb postgres idle client error (connection discarded): " + (err && err.stack || err));
});

Then restart Etherpad. After that, when HAProxy closes an idle connection you'll see that line logged (instead of a crash), and the pool quietly opens a fresh connection on the next query — no more 60s restart loop.

Two notes:

• This edit lives in node_modules, so it's wiped by the next pnpm install/upgrade. It's for confirming the fix now; the durable version is Bump API version to v1.1 #998 landing in an ueberDB release that Etherpad then picks up.
• If grep finds the file under a different path on your setup, this also works: find . -path '*ueberdb2/dist/*' -name '*.js' | xargs grep -l 'new pg.Pool(this.settings)'.

If you can confirm this stops the restarts on your Patroni + HAProxy cluster, that's the end-to-end validation of #998 in a real environment — which would be hugely useful. Thanks again!

(And re: your earlier point — yes, I audited the other drivers: redis has the identical bug, fixed in ether/ueberDB#999. mysql turned out to be safe — mysql2's pooled connections self-register an error listener — so no change needed there. rethink is also affected: ether/ueberDB#1000.)

[brunob]

Thx for explaining me how to apply the patch !

Tested it and it seems to do the job :

[2026-06-02T12:57:09.106] [INFO] server - Etherpad is running
...
[2026-06-02T12:58:19.024] [ERROR] settings - ueberdb postgres idle client error (connection discarded): Error: Connection terminated unexpectedly
[2026-06-02T12:58:19.025] [ERROR] settings - ueberdb postgres idle client error (connection discarded): Error: Connection terminated unexpectedly
[2026-06-02T12:58:19.028] [ERROR] settings - ueberdb postgres idle client error (connection discarded): Error: Connection terminated unexpectedly
...
[2026-06-02T13:01:16.521] [ERROR] settings - ueberdb postgres idle client error (connection discarded): Error: Connection terminated unexpectedly
[2026-06-02T13:01:16.522] [ERROR] settings - ueberdb postgres idle client error (connection discarded): Error: Connection terminated unexpectedly
[2026-06-02T13:01:16.522] [ERROR] settings - ueberdb postgres idle client error (connection discarded): Error: Connection terminated unexpectedly

And no more crash of the service \o/

[JohnMcLear]

huzzzaaaa - okies I bumped Sam to review so hopefully we will get some stuff merged this week and released :) 🎉 Thanks!