Real-time analytics
« Back to Glossary Index
Real-time analytics processes and analyzes data as it is generated, so insight is available within seconds or milliseconds rather than after an overnight batch. It supports decisions that lose value quickly — blocking a fraudulent transaction, rerouting a delayed shipment, reacting to a system fault while it is still contained.
It is also the capability most often built where it is not needed. The question is never whether faster data would be nice, but whether any decision actually changes as a result of having it sooner.
What Is Real-Time Analytics?
Traditional analytics works in batches: data is collected through the day and processed overnight, so reporting describes a position that is already hours or days old. That is entirely adequate for most management reporting, where the decisions being informed operate on weekly or monthly cycles.
Real-time analytics processes each event as it arrives. A transaction, sensor reading or click is ingested, evaluated and acted on immediately — often automatically, because the decision window is shorter than a human response time.
The defining characteristic is therefore not speed for its own sake but decision latency: the time between something happening and someone or something being able to respond. Real-time architectures exist to compress that window, and their value is exactly proportional to how quickly the opportunity to act disappears.
How Real-Time Is Real-Time?
The term covers a wide range, and being precise about the requirement prevents substantial overspend.
True real-time means single-digit milliseconds — algorithmic trading, industrial control. Rare in business analytics and expensive to achieve.
Near real-time means seconds to a minute. Fraud scoring, operational alerting and live dashboards sit here, and this covers the overwhelming majority of genuine business requirements.
Micro-batch processes every few minutes. Often indistinguishable from near real-time to the user while being considerably simpler and cheaper to build and operate.
Most requests for real-time analytics are satisfied by micro-batch. Asking what happens if the data is five minutes old rather than five seconds old usually reveals that nothing does — and that question is worth asking before any architecture is chosen.
Streaming Architecture
A streaming pipeline has four parts. Ingestion captures events from sources — applications, devices, databases via change data capture. The event stream, typically Kafka, Kinesis or Pub/Sub, holds events durably and lets multiple consumers read independently. Stream processing — Flink, Spark Structured Streaming or ksqlDB — filters, aggregates, joins and applies models to events in motion. Serving makes results available to dashboards, applications or automated actions.
Two design problems dominate and have no equivalent in batch work. Late and out-of-order events are normal in distributed systems, so the pipeline must decide how long to wait for stragglers before finalizing a window — a direct trade-off between completeness and latency. Exactly-once processing matters wherever duplicates would corrupt a result, and achieving it costs throughput.
Most estates run streaming alongside batch rather than replacing it, with the stream serving immediate decisions and batch producing the authoritative historical record. Building that coherently is data engineering and integration work.
Where Real-Time Genuinely Pays
Fraud and risk scoring. The decision must be made before a transaction completes. There is no batch alternative.
Industrial and equipment monitoring. Sensor data flagging deviation while intervention can still prevent failure or scrap.
Logistics and field operations. Rerouting, reallocating and reprioritizing while the day is still in progress.
Digital experience. Personalization and recommendation within a session, where the visitor has left long before a batch job would run.
Infrastructure and platform monitoring. Detecting failures and anomalies as they emerge rather than in a morning report.
The pattern across all of these is a short, closing window of opportunity and, usually, an automated response. Where a human reads the output and acts within hours, batch or micro-batch is almost always sufficient.
What It Costs
Real-time is materially more expensive than batch across every dimension, and the compute bill is the smallest part of it.
Streaming infrastructure runs continuously rather than for a scheduled window. Engineering is harder, because distributed streaming introduces failure modes — ordering, duplication, backpressure, state management — that batch does not have. Operations are harder, since a failed batch job can be rerun while a failed stream loses events unless replay is designed in. Testing is harder, because reproducing a specific sequence of timed events is considerably more involved than rerunning a dataset. And the skills are scarcer and more expensive.
None of this argues against real-time where it is warranted. It argues for being honest about the requirement, because an organization that builds streaming for dashboards people check each morning has taken on all of this cost for no change in outcome.
Deciding Whether You Need It
Four questions settle it in most cases.
What decision does this enable? If the answer is monitoring rather than a specific action, real-time is unlikely to be justified.
How quickly does the opportunity to act disappear? This defines the actual latency requirement, and it is usually far more forgiving than the initial request implies.
Who or what responds? Automated responses can exploit sub-second latency. Human responses cannot — a person notified in 200 milliseconds still acts in minutes.
What does the delay currently cost? If nobody can quantify it, the case for the investment does not yet exist.
Where the answers point to genuine need, the sensible path is one high-value use case built properly rather than a platform-wide streaming programme. Where they do not, improving batch frequency usually delivers most of the perceived benefit at a fraction of the cost — a judgement that belongs in business analytics design rather than in platform selection.
Frequently Asked Questions
Real-time analytics processes and analyzes data as it is generated so insight is available within seconds or milliseconds rather than after an overnight batch, supporting decisions that lose value quickly.
Batch collects data and processes it on a schedule, so reporting describes a position hours or days old. Real-time processes each event as it arrives. Batch is adequate wherever the decisions being informed operate on daily or weekly cycles.
True real-time means single-digit milliseconds and is rare in business analytics. Near real-time means seconds to a minute and covers most genuine requirements. Micro-batch runs every few minutes and is often indistinguishable to users while being far cheaper.
Ingestion capturing events from applications, devices or change data capture; a durable event stream such as Kafka, Kinesis or Pub/Sub; stream processing with Flink or Spark Structured Streaming; and a serving layer feeding dashboards, applications or automated actions.
Handling late and out-of-order events, which forces a trade-off between completeness and latency; achieving exactly-once processing where duplicates would corrupt results; and designing replay, since a failed stream loses events where a failed batch job can simply be rerun.
Fraud and risk scoring before a transaction completes, equipment monitoring where intervention prevents failure, logistics rerouting during the operating day, in-session personalization, and infrastructure anomaly detection.
Infrastructure runs continuously rather than on a schedule, distributed streaming introduces failure modes batch does not have, operations and testing are harder because events cannot simply be rerun, and the required skills are scarcer.
Ask what decision it enables, how quickly the opportunity to act disappears, whether the response is automated or human, and what the current delay costs. Where a person acts within hours, batch or micro-batch is almost always sufficient.
