Spark Streaming Windowing, Watermarking & Stateful Processing
How Spark Manages Time and Memory in Long-Running Systems
Streaming systems don’t fail because they cannot process data fast enough. They fail because time and state are misunderstood.
When a Spark Structured Streaming job runs for weeks or months, the real engineering challenge is not transformations. It is how the engine manages event time, late data, and memory pressure while keeping results correct.
Windowing defines how we group time.
Watermarking defines how long we wait.
State defines what we remember.
If you understand how these three interact, you understand how Spark survives long-running production workloads.
Windowing in Streaming: Grouping an Infinite Timeline
In batch systems, time is just a column. In streaming systems, time defines lifecycle.
When you group streaming data by time, Spark must maintain state for each open window. The choice of window type directly affects how much state accumulates in memory.
Tumbling Windows
A tumbling window divides time into fixed, non-overlapping chunks.
Every event belongs to exactly one window.
State grows predictably and closes cleanly once the window is finalized.
This is the most memory-stable pattern and usually the safest default for production systems.
Sliding Windows
Sliding windows overlap.
An event may belong to multiple windows depending on the slide interval. For example, a 10-minute window sliding every 1 minute creates 10 overlapping windows.
That means:
• The same event contributes to multiple aggregates
• State multiplies
• Memory grows faster than expected
Sliding windows are powerful, but they are also the most common source of state explosion in streaming jobs.
Session Windows
Session windows are dynamic.
Instead of fixed time boundaries, sessions close when there is inactivity for a defined gap duration.
Spark must keep session state open until it is confident no more events will extend that session. That makes session windows more complex internally because the end boundary is not known upfront.
Session windows are useful for user activity tracking, but they require careful watermark configuration to prevent long-lived state.
Watermarking: The Control Plane for Time
Watermarking is not about performance tuning. It is about correctness boundaries.
When you define:
withWatermark("event_time", "10 minutes")
you are telling Spark:
“I accept data that arrives up to 10 minutes late. Anything later can be dropped.”
Internally, Spark tracks the maximum event time seen so far.
The watermark is calculated as:
maximum_event_time_seen minus allowed_lateness
Once the watermark passes the end of a window, Spark considers that window safe to finalize and eligible for state cleanup.
Without watermarking:
• Windows never close
• State never gets cleaned
• Memory keeps growing
Watermark is the mechanism that allows long-running systems to remain bounded in memory.
Stateful Processing: What Spark Actually Stores
Every aggregation, join, or deduplication in streaming creates state.
Spark stores this state in a state store backed by checkpointing. Depending on configuration, this may be HDFS, cloud object storage, or local disk with replication.
For each key and window combination, Spark maintains:
• Aggregated values
• Window boundaries
• Metadata for cleanup
State is versioned per micro-batch. That is how Spark guarantees fault tolerance and exactly-once semantics when supported by the sink.
State is not abstract. It is physical data written and compacted continuously.
How State Grows with Sliding Windows
Sliding windows multiply keys.
If you have:
• 1 million unique keys
• A 10-minute window
• A 1-minute slide
You effectively maintain state for roughly 10 million window-key combinations at peak.
Now add late data tolerance and joins, and the memory footprint increases further.
This is why sliding windows combined with high-cardinality keys can quietly exhaust executors over time.
Production failures often appear after days of smooth execution because state accumulation is gradual.
Stream–Stream Joins: Coordinating Two Infinite Inputs
Joining two streams is fundamentally different from joining static datasets.
Spark must hold state for both sides of the join until it is confident no matching record will arrive.
Watermarks must be defined on both streams.
Join conditions must include time constraints.
If not:
• State grows indefinitely
• Old records remain in memory
• Executors eventually fail
Stream-stream joins are powerful, but they require disciplined time boundaries. Without them, the system has no signal to evict old data.
State Lifecycle and Cleanup
The lifecycle of state follows a predictable pattern:
Data arrives
State updates
Watermark advances
Window becomes eligible for eviction
State is cleaned
Cleanup does not happen instantly. It occurs during micro-batch execution when Spark evaluates watermark progress.
If event time stalls or watermark does not advance, cleanup also stalls.
That is why skewed or delayed partitions can indirectly cause memory pressure across the cluster.
Time progression drives memory release.
Common Production Mistakes
One frequent mistake is using sliding windows without estimating key cardinality. Memory usage grows faster than anticipated, especially under high throughput.
Another mistake is defining watermark too conservatively. Large lateness tolerance keeps windows open longer and increases retained state.
Sometimes teams forget that event time must actually progress. If a source system emits stale timestamps, watermark never advances, and cleanup never happens.
Stream-stream joins without proper time bounds are another recurring issue. They appear correct in testing but fail after extended runtime.
Finally, ignoring state store metrics is dangerous. Long-running jobs must be observed continuously. State size trends matter more than CPU utilization.
The Core Principle
Spark Structured Streaming is not just about processing data continuously. It is about managing time as a first-class dimension and keeping memory bounded in an unbounded system.
Windowing defines grouping.
Watermarking defines tolerance.
State defines memory footprint.
When these three are aligned, streaming systems remain stable for months.
When they are misunderstood, failures are inevitable — not immediately, but eventually.
Time and memory are the real architecture decisions in streaming.
Everything else is syntax.