System Realities: The Hard Truth About Real-Time Systems

Introduction:

Real-time systems are one of the most misused terms in software engineering. Marketing teams describe analytics dashboards as real-time when they refresh every thirty seconds. Product managers request real-time features without understanding what real-time actually requires to implement correctly. Engineers build systems they call real-time that would not satisfy any rigorous definition of the term.

The misuse matters because real-time systems are genuinely difficult and genuinely expensive. Building a system that actually delivers data or responses within strict time bounds — not just quickly on average but reliably within a defined latency ceiling — requires architectural decisions, infrastructure investments, and operational practices that most teams are not prepared for when they commit to real-time requirements.

Understanding what real-time systems actually demand is the first step toward building them correctly — or toward making an informed decision that eventual consistency or near-real-time is what you actually need.

Real-Time Does Not Mean Fast — It Means Predictably Bounded:

The defining characteristic of a real-time system is not speed but determinism. A real-time system guarantees that operations complete within a defined time bound — not on average, not most of the time, but reliably and predictably. Missing that bound is a system failure, regardless of how quickly the system performs under normal conditions.

Hard real-time systems — the kind used in aircraft control systems, medical devices, and industrial automation — treat a missed deadline as a critical failure. Soft real-time systems tolerate occasional deadline misses but degrade meaningfully when misses become frequent. Most software systems that use the term real-time are neither — they are low-latency systems that perform well under normal conditions but have no formal guarantees about worst-case behaviour.

This distinction matters because the engineering practices required for genuinely bounded latency are fundamentally different from the practices required for low average latency. Optimising for average case performance and optimising for worst case performance are different problems that require different solutions.

Tail Latency Is the Real Challenge:

In any system that handles requests, the distribution of response times has a tail — a small percentage of requests that take significantly longer than the median. For most systems, tail latency is an inconvenience. For real-time systems, it is the primary engineering challenge.

A system with a median latency of five milliseconds and a 99th percentile latency of two seconds is not a real-time system. The two-second tail is indistinguishable from a failure in any application where users or downstream systems are waiting on a response within a strict time bound.

Tail latency is caused by garbage collection pauses, CPU scheduling delays, network jitter, lock contention, and resource exhaustion — all of which are difficult to eliminate entirely and require careful engineering to bound. Languages with stop-the-world garbage collection like Java and Go require explicit tuning to prevent garbage collection pauses from producing unacceptable tail latency. Systems that share hardware with other workloads are subject to noisy neighbour effects that introduce unpredictable latency spikes.

The Network Is Not Your Friend:

Network latency is one of the most significant sources of unpredictability in distributed real-time systems. Packets are delayed, reordered, and occasionally dropped. Network switches experience congestion. Virtual network overlays in cloud environments introduce additional latency variability that is outside your control.

TCP — the protocol underlying most networked systems — introduces its own latency sources. The three-way handshake adds a round trip to connection establishment. Retransmission of lost packets introduces variable delays. Nagle's algorithm, which buffers small packets to reduce network overhead, adds latency in interactive applications and must be disabled explicitly with TCP_NODELAY in latency-sensitive systems.

Building real-time systems on top of infrastructure you do not fully control — shared cloud networks, virtualised hardware, managed services with their own latency characteristics — means accepting latency variability that you cannot eliminate and can only partially mitigate.

Backpressure Is Not Optional:

Real-time systems that receive more data than they can process within their latency bounds need a mechanism to signal that they are overwhelmed and slow down the rate at which data is sent to them. This is backpressure — and in real-time systems, it is not an optimisation but a correctness requirement.

A system without backpressure accumulates a processing backlog when input rates exceed processing capacity. The backlog grows, processing latency increases, and the system falls further and further behind real time. Eventually it is processing data that is minutes or hours old while new data continues to arrive — a system that looks operational but is not delivering on its real-time promise.

Implementing backpressure requires coordination between producers and consumers, explicit queue depth monitoring, and the ability to shed load gracefully when capacity is exceeded. Systems built without backpressure in mind are difficult to retrofit — the assumption that inputs can always be accepted propagates through the architecture in ways that are hard to unwind.

State Management Becomes Significantly Harder:

Stateless systems are easier to scale and easier to reason about, but real-time systems frequently require state — maintaining running aggregations, tracking session context, storing the history needed to compute incremental updates. Managing that state within real-time latency bounds introduces constraints that stateless systems do not face.

State that lives in memory is fast but lost on restart. State that is persisted to disk or a database introduces I/O latency that can push operations outside their time bounds. State that is distributed across multiple nodes requires coordination that adds latency and introduces failure modes.

Real-time systems that manage significant state need explicit strategies for state partitioning, state recovery after failures, and state consistency across replicas — all within latency bounds that leave little margin for the coordination overhead these strategies introduce.

Operational Complexity Is Higher Than Expected:

Real-time systems require more sophisticated operational practices than conventional systems. Latency must be monitored at fine granularity — not just average and p95 but p99, p99.9, and maximum. Anomalies that would be noise in a conventional system are signal in a real-time one.

Deployments require more care because rolling restarts introduce latency spikes as connections are redistributed. Capacity planning requires understanding not just average load but peak load with enough headroom that the system remains within its latency bounds during traffic spikes. Debugging latency regressions requires tooling that can identify the source of increased tail latency — whether it is garbage collection, network, lock contention, or something else entirely.

Teams that build real-time systems without investing in the operational infrastructure to run them reliably discover that the system works in testing and fails in production in ways that are difficult to diagnose without the right observability in place.

Conclusion:

Real-time systems are harder than they appear because the requirements that define them — bounded latency, predictable behaviour under load, graceful handling of backpressure — conflict with the design patterns and infrastructure choices that make most systems easy to build and operate.

The teams that build real-time systems successfully are the ones that understand these requirements before committing to them, choose infrastructure that gives them control over the latency sources that matter, and invest in the operational practices needed to run latency-sensitive systems reliably. The teams that struggle are the ones that commit to real-time without understanding what it actually demands — and discover the gap between their assumptions and reality only after the system is in production.