How we fixed non-uniform load distribution

Problem Statement

We encountered a distributed load balancing issue between a stateless API layer (~50 nodes) and a 5-node CockroachDB cluster. Each API node randomly sampled a subset of 3 database instances to establish its connection pool.

Although this configuration assumed that uniform randomization at the client level would translate to an even distribution across the storage tier, it introduced a significant systemic imbalance. The aggregate traffic pattern created localized hotspots, causing disproportionate resource consumption on specific database nodes while underutilizing available cluster capacity.

Initial Investigation

When I was first exposed to this problem, I checked if the API load was uneven across the servers, either in volume or API type. But that wasn’t the case. The DB nodes were of the same hardware class too.

Statistics

Eventually, I did figure out why we were facing the issue and the solution lies in statistics.

If we look from the point of view of the server, it is equally distributing load amongst the 3 DB nodes, that the server has picked. But, this does not translate to uniform load distribution across the DB nodes.

From the point of view of the DB node, the probability that the DB node is chosen out of 5 nodes(k) by an server is:

       (k - 1) C 2      3    
 p =  ------------   =  -- 
        k C 3           k  

There are n(50) api servers.

The number of API Servers that pick a DB Node, let’s call it X, follows a Binomial Distribution.

 X ~ Binomial (n, 3 / k)

Probability of a DB Node picked by exactly x API Servers:

P(X = x) = ( n C x) * ( 3 / k) power x *  (1 - 3/k) power (n-x)

Expected number of API Servers per DB Node: n * p = 3n / k

Binomial Distribution
n = 50 , p = 0.6

P 0.11 |                              |
r 0.10 +                             | | |
o 0.09 |                          | | | | |
b 0.08 +                        | | | | | | |
a 0.07 |                        | | | | | | |
b 0.06 +                      | | | | | | | | |
i 0.05 |                      | | | | | | | | |
l 0.04 +                    | | | | | | | | | | |
i 0.03 |                    | | | | | | | | | | |
t 0.02 +                  | | | | | | | | | | | | |
y 0.01 |              | | | | | | | | | | | | | | | | |
    0.00 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
         15        20        25        30        35        40        45

                         Number of API Servers per DB Node

Conclusion

It’s easy to look at a cluster and assume that standard randomization will handle the heavy lifting of load balancing. But as the binomial distribution chart shows, giving your servers a limited view of the world naturally creates systemic hotspots over time.

By switching our backend to pool connections across all available DB nodes uniformly, we eliminated the mathematical bias causing the uneven distribution. The load flattened out, the erratic latency spikes vanished, and the cluster finally started performing with lower P99s.