Is round robin the default best choice?

Round robin is often a reasonable default when requests are similar in cost and backends have similar capacity. It optimises for even request counts, not even resource use. If some requests are slow, stateful, streaming, or much heavier than others, equal counts can still overload one replica while others look underused.

What is the difference between hashing and consistent hashing?

Plain hashing maps a key to a backend using the current backend count, so changing the replica set can move many keys. Consistent hashing arranges keys and backends so that adding or removing a backend moves only a smaller portion of keys, preserving cache locality and session affinity better during scaling.

When should I use least connections?

Use least connections when active connections are a decent proxy for work, such as long-lived sessions or uneven request duration. Be careful with HTTP/2 and gRPC, where a single connection may carry many concurrent streams. In those systems, request-aware or latency-aware balancing may represent load more accurately than connection count.

Are sticky sessions a scaling axis?

Yes. Sticky sessions are an affinity choice, usually implemented by hashing a client, cookie, or user key to a backend. They can improve cache hits and keep in-memory state reachable, but they also make failover and deploys harder unless session state lives in shared storage such as Redis, Memcached, or a database.

Scaling Axes

Scaling axes are the choices a system makes about how to spread traffic across replicas, usually through a load-balancing algorithm. The axis you choose decides what is treated as fair: request count, active work, observed latency, backend capacity, cache affinity, geographic closeness, or resilience when replicas appear and disappear.

Replicas do not automatically mean balanced capacity. If every request costs roughly the same, spreading request counts evenly may work. If some requests run for much longer, hold open streams, or depend on warm caches, the same policy can create hotspots. Scaling axes exist because “send it to any healthy instance” hides a choice about what kind of imbalance you are willing to tolerate.

The mechanism is usually simple routing state at a proxy, service mesh, ingress, or kernel datapath. Round robin walks through backends in order. Weighted round robin repeats larger backends more often. Least connections sends new work to the backend with fewer open connections. Least response time combines recent latency with load. Hashing maps a client, user, or object key to a backend, while consistent hashing reduces key movement when the backend set changes.

Every axis trades one failure mode for another. Round robin is cheap but blind to request cost. Least connections helps with long-lived work, but can misread multiplexed protocols where one connection carries many streams. Hashing preserves locality, but can create skew if keys are uneven. Consistent hashing protects caches during scaling, but adds complexity and still needs a plan for failed or drained nodes.

Engineers meet these choices in Kubernetes Services, Ingress controllers, Envoy, NGINX, service meshes, and edge load balancers. The honest answer is rarely that one algorithm is best. It depends on whether your bottleneck is CPU, connections, latency, cache misses, regional distance, or backend heterogeneity. The practical skill is naming the bias of the policy before production traffic discovers it for you.

Go deeper

DeploymentScaling AxesCompare the common load balancing algorithms - round robin, least connections, hashing and the rest - and say what each one optimises for.Open the lesson

Common questions

Is round robin the default best choice?: Round robin is often a reasonable default when requests are similar in cost and backends have similar capacity. It optimises for even request counts, not even resource use. If some requests are slow, stateful, streaming, or much heavier than others, equal counts can still overload one replica while others look underused.
What is the difference between hashing and consistent hashing?: Plain hashing maps a key to a backend using the current backend count, so changing the replica set can move many keys. Consistent hashing arranges keys and backends so that adding or removing a backend moves only a smaller portion of keys, preserving cache locality and session affinity better during scaling.
When should I use least connections?: Use least connections when active connections are a decent proxy for work, such as long-lived sessions or uneven request duration. Be careful with HTTP/2 and gRPC, where a single connection may carry many concurrent streams. In those systems, request-aware or latency-aware balancing may represent load more accurately than connection count.
Are sticky sessions a scaling axis?: Yes. Sticky sessions are an affinity choice, usually implemented by hashing a client, cookie, or user key to a backend. They can improve cache hits and keep in-memory state reachable, but they also make failover and deploys harder unless session state lives in shared storage such as Redis, Memcached, or a database.

Every term