Networking Internals: The Load Balancer Is Making Decisions You Did Not Configure
Most teams configure a load balancer once. Pick the listener, attach the targets, point the health check at /health, confirm traffic flows, move on. From that day forward it’s treated as plumbing: a pipe that spreads requests evenly across whatever sits behind it.
It isn’t a pipe. It’s a decision engine running continuously in the middle of every request, and most of the decisions it makes come from defaults nobody on your team chose. Which servers count as alive, how long a quiet connection survives, whether a failed request gets sent again, how traffic splits across zones. All of it was decided, just not by you.
Those defaults are sensible in general and wrong in particular, and you find out which ones during an incident.
Who Is Alive Is the Load Balancer’s Opinion
Your service doesn’t decide whether it’s healthy. The load balancer does, using a check you probably wrote in five minutes.
A health check has an interval, a timeout, and thresholds for how many failures mark a target unhealthy and how many successes bring it back. With common defaults, a dead instance can keep receiving traffic for twenty or thirty seconds before the load balancer gives up on it. During that window, a fixed share of your users get errors and your dashboards show a service that is “mostly fine.”
The check itself is usually too shallow. An endpoint that returns 200 without touching the database tells the load balancer the process is running, not that it can do any work. Make it too deep and you get the opposite failure: one slow dependency fails every instance’s check at once.
What happens then is the default almost nobody knows. When every target is unhealthy, many load balancers, AWS’s Application Load Balancer among them, fail open and route to all of them anyway, on the theory that some chance of success beats none. It’s a defensible choice. It also means “all targets unhealthy” and “traffic still flowing” can both be true, which confuses everyone reading the graphs mid-incident.
It Balances Connections, Not Work
Round robin distributes requests evenly, which is only fair if requests cost the same. One instance can draw ten cheap lookups while its neighbour draws ten report exports, and the load balancer considers that balanced.
The load balancer is balancing what it can count. That is rarely the same as what your servers feel.
The gap widens with long-lived connections. A layer 4 load balancer picks a backend once per connection, not once per request. With HTTP/2 and gRPC, a client opens one connection and multiplexes thousands of requests over it, so every one of those requests lands on the same backend. You scale from four instances to eight and the new four sit idle, because the existing connections never moved. The fix is balancing at layer 7, client-side balancing, or periodically recycling connections, but first you have to notice that “we added capacity and nothing changed” is a load balancing symptom.
New instances have the reverse problem. A freshly started target with cold caches and an unwarmed runtime gets its full share of traffic immediately, unless slow start is enabled. It usually isn’t.
The Timeouts You Never Set
Every load balancer has an idle timeout, the period after which it closes a quiet connection. On an ALB the default is 60 seconds.
Your backend has its own keep-alive timeout, and if it’s shorter than the load balancer’s, you have a race. The backend closes an idle connection at the same moment the load balancer reuses it for a new request. The request goes into a closed socket and the client gets a 502. Node.js made this famous, with a default keep-alive of five seconds sitting behind a sixty-second load balancer.
The signature is intermittent 502s at low volume that vanish under load, because busy connections never sit idle long enough to hit the race. That pattern looks like a flaky network and sends people hunting in the wrong place for days. The rule is simple once you know it: the backend’s keep-alive must outlast the load balancer’s idle timeout.
Deploys have their own version. When a target deregisters, the load balancer keeps in-flight requests alive for a draining period, 300 seconds by default on AWS. Leave it alone and deploys crawl. Cut it below your longest request and every deploy drops a few.
Zones, Retries and Other Quiet Choices
Cross-zone balancing decides whether traffic spreads across all targets or only within each zone. The defaults differ by product: on for AWS’s Application Load Balancer, off for its Network Load Balancer. With it off, each zone gets an equal share of traffic regardless of how many targets it holds. Two instances in one zone and eight in another means the two each carry four times the load of the eight.
Retries are the dangerous one. Some proxies and load balancers resend a request to another backend when the first attempt fails. Without that, a single bad instance surfaces as user-facing errors. With it, a request that timed out afterthe backend did the work gets executed twice.
A retry you did not configure is a duplicate you will not be able to explain.
And buffering: many proxies read the entire request or response before forwarding it. That’s invisible for ordinary APIs and fatal for streaming, where the client waits on a response the proxy is politely holding.
Read the Defaults Once
The fix is one afternoon. Open your load balancer’s configuration and write down what it actually does: check interval and thresholds, behaviour when all targets fail, algorithm, idle timeout, draining period, cross-zone setting, retry policy, buffering.
Then compare each against the service behind it. Is the backend keep-alive longer than the idle timeout? Does the draining period cover your slowest request? Does the health check fail when the service can’t do its job, and only then? Are your gRPC clients pinned to backends that existed an hour ago?
Most teams find at least one mismatch, usually the timeout pair, and usually one that explains a class of intermittent errors written off long ago as “the network.”
The load balancer was never neutral. It has been making these calls on every request since the day you set it up, and it’s worth knowing what they are.
Enjoyed this post?
Stay in the loop
New posts + weekly digest, straight to your inbox.
Create a free account
- Save posts to your vault
- Like posts & build history
- New-post alerts
No comments yet. Be the first to comment!