Spark Structured Streaming Fundamentals: Continuous Batch Processing

Structured Streaming is often treated as a separate processing model inside Spark — something fundamentally different from batch. That assumption creates unnecessary confusion.

In reality, Structured Streaming is incremental execution over an unbounded table, powered by the same Spark SQL engine, the same Catalyst optimizer, and the same physical execution layer used for batch workloads.

Nothing magical. No separate runtime.

When you understand that streaming is simply continuous batch computation, concepts like watermarks, triggers, state management, and output modes stop feeling abstract — they become predictable engineering decisions rather than mysterious configuration rules..

---

1. Batch vs Streaming: Fix the Mental Model

In Spark:

• Batch = bounded table

• Streaming = unbounded table

That’s it.

In batch:

df = spark.read.parquet("/sales")
result = df.groupBy("region").sum("amount")

In streaming:

df = spark.readStream.format("kafka")...
result = df.groupBy("region").sum("amount")

Same transformations. Same API.

The only difference:

• Batch runs once on finite data

• Streaming runs repeatedly on new data

Structured Streaming treats input as a continuously growing table and computes incremental results instead of recomputing everything.

This is why it is called Structured Streaming — it works on structured datasets using the same DataFrame abstraction.

---

2. Micro-Batch Execution Explained

Structured Streaming is not record-by-record processing (by default).

It uses micro-batch execution.

Under the hood:

1. Spark checks for new data

2. Creates a small batch (micro-batch)

3. Executes the logical plan

4. Writes output

5. Stores offsets + state

6. Repeats

Every trigger interval produces a new mini job.

Think of it as:

While(true):
    Read new data
    Run batch plan
    Commit offsets

Key implications:

• Same Catalyst optimizer is used

• Same DAG planning

• Fault tolerance via checkpointing

• Exactly-once semantics (when supported sink is used)

It is not a different engine. It is incremental execution.

---

3. Event Time vs Processing Time

This is where most confusion starts.

Processing Time

Time when Spark processes the record.

• Depends on system clock

• Affected by delays

• Not reliable for late data

Example:
If a Kafka message arrives late, processing time reflects when it was read — not when the event happened.

---

Event Time

Time embedded in the record itself.

Example:

{
  "order_id": 123,
  "event_time": "2026-03-01T10:05:00"
}

This represents when the event actually occurred.

For aggregations like:

df.groupBy(window("event_time", "10 minutes")).count()

Spark groups based on event time, not arrival time.

Event time is essential for:

• Correct window aggregations

• Handling out-of-order data

• Watermarking

---

4. Trigger Types

Triggers define when micro-batches run.

a.processingTime

Runs every fixed interval.

.trigger(processingTime="10 seconds")

Spark checks for new data every 10 seconds.

Use when:

• Continuous ingestion

• Low latency required

---

b. availableNow

Processes all available data and then stops.

.trigger(availableNow=True)

Use when:

• Backfill scenarios

• Incremental batch jobs

• Scheduled streaming pipelines

This bridges batch and streaming beautifully.

---

5. Output Modes

Output mode defines what gets written to the sink.

Append Mode

Writes only new rows.

.outputMode("append")

Use when:

• No updates to old rows

• Window is finalized

• Watermark is defined

---

Update Mode

Writes only changed rows.

.outputMode("update")

Use when:

• Aggregations are evolving

• Sink supports upserts

---

Complete Mode

Writes entire result table every trigger.

.outputMode("complete")

Use when:

• Small aggregations

• In-memory sinks

• Debugging

Not scalable for large datasets.

---

6. Why Append Mode Requires Watermark

This is the most misunderstood rule.

If you perform windowed aggregation:

df.groupBy(window("event_time", "10 minutes")).count()

Spark cannot know when a window is complete.

Late data might still arrive.

If Spark writes in append mode without watermark:

• It may output a window

• Later receive late data

• But append mode cannot update old rows

• Result becomes incorrect

So Spark requires watermark:

df.withWatermark("event_time", "5 minutes")

Meaning:

“I am okay dropping data later than 5 minutes.”

Now Spark knows when a window is safe to finalize and append.

No watermark → no guarantee window is final → append mode disallowed.

This is about correctness, not syntax.

---

7. Common Misconceptions

Misconception 1: Streaming is real-time row-by-row
Reality: It is micro-batch by default.

Misconception 2: Streaming API is different
Reality: Same DataFrame transformations.

Misconception 3: Watermark improves performance
Reality: It defines correctness boundary and state cleanup.

Misconception 4: Streaming replaces batch
Reality: Streaming is incremental batch.

---

8. Final Mental Model

Structured Streaming =

• Unbounded table

• Incremental execution

• Micro-batch engine

• Stateful computation

• Event-time aware processing

Once you stop thinking of it as “real-time magic” and start seeing it as continuous batch computation, everything becomes predictable.

And predictable systems scale.

---

If you are building data platforms on Databricks or Spark clusters, clarity on these fundamentals prevents costly design mistakes later — especially around windowing, watermarking, and sink guarantees.

Streaming is not a different API.

It is disciplined incremental execution.