Physical Bridge Backlog not reflected in ServerCounters #2997
Description
Our service creates N ConnectionMultiplexer instances to N isolated redis servers, with Default BacklogPolicy. We then use ServerCounters.TotalOutstanding to load balance to least loaded server connections.
However, we noticed error messages like below, where qu (PhsyicalBridge backlog count) is `M+
StackExchange.Redis.RedisTimeoutException: The message timed out in the backlog attempting to send because no connection became available, command=ARGGALAXYUPSERTVERTEXTXN2, timeout: 10000, outbound: 0KiB, inbound: 0KiB, inst: 0, qu: 1144678, qs: 0, aw: False, bw: Activating, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 406, cur-in: 0, sync-ops: 751280241, async-ops: 46587, serverEndpoint: 127.0.0.1:3278, conn-sec: 3059281.44, aoc: 0, mc: 1/1/0, clientName: 47..3, IOCP: (Busy=101,Free=3995,Min=1024,Max=4096), WORKER: (Busy=639,Free=3457,Min=1024,Max=4096), v: 2.7.17.27058 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
Further investigation showed that this ConnectionMultiplexer, despite have >1M messages in the queue, was reporting TotalOutstanding=0, and thus being preferentially chosen by our load-balancing algorithm.
The library currently exposes no public members for accessing the value of PhysicalBridge._backlogCurrentEnqueued directly. Thus, I propose the following small change to make ConnectionCounters.PendingUnsentItems more semantically meaningful, and allow end-users to more precisely differentiate between clients with varying backlog counts: #2996