GitHub user Ma77Ball edited a discussion: Columnar (Apache Arrow) execution in 
Amber: the opt-in PR series and why

## Columnar (Apache Arrow) execution in Amber: the opt-in PR series and why

**Status, updated 2026-11-19.** This revives the thread with what actually 
shipped. The work now lives on a reviewable, stacked PR series tracked by 
parent issue #9542, and the whole feature is off by default 
(`ApplicationConfig.enableColumnarWire`), so the row path is unchanged until a 
user opts in.

### Why
Amber moves and processes data one row `Tuple` at a time at every operator and 
across every edge. For exchange- and scan-heavy workflows that costs two 
things: serializing an object graph per batch on a cross-worker edge, and 
building a JVM object per row at every stage. Columnar (Apache Arrow) execution 
addresses both: Arrow batches serialize at near memory-copy speed, and a 
columnar-capable operator reads the columns directly instead of materializing a 
tuple per row. This is the same mechanism Spark/Photon, Velox, Gluten, and 
DataFusion Comet use.

### The key design choice: decide per edge, not globally
Columnar is not always a win. A same-JVM edge already passes data by reference, 
so wrapping it in Arrow bytes would only add overhead. So each edge picks its 
own format from two questions: can the receiver read columns, and is it local 
or remote. Remote and columnar-capable goes Arrow; local and columnar-capable 
gets a live root by reference (zero copy); everything else stays rows. This is 
what keeps the feature from ever regressing a workflow, and it is why it can 
eventually default on.

### The reviewable parts (parent #9542)
- Part 1, Foundation (#9547): the `ColumnarOperatorExecutor` / `ColumnarResult` 
contract, the opt-in `ColumnarFrame` wire, a faster Arrow-to-tuple decode, and 
vectorized receive for filter and aggregate.
- Part 2, Output and per-edge planner (#9548): keep columns on output without 
the row round-trip, and decide the wire format per edge from the downstream 
operator.
- Part 3, Shuffle and projection (#9544): partition an Arrow batch across 
workers without decoding, plus a native-Arrow projection operator.
- Part 4, Locality-aware edges (#9545): keep same-JVM edges on rows, send 
columns only to the remote receivers on a mixed-locality edge.
- Part 5, Local zero-copy (#9546): the `processColumnarRoot` primitive and 
handing a live Arrow root to a co-located receiver by reference, with a 
checkpoint-safe serializer.

### Measured so far
Single-JVM serialization simulation, median of 9 trials, across a matrix of 
workflow shapes with a correctness parity gate (the columnar and row paths must 
return identical results per cell):
- Cross-node (serialized edges): 3x to 50x faster at 100k rows on exchange- and 
projection-heavy workflows; small batches (about 1k rows) can lose, since 
Arrow's fixed per-batch overhead is not amortized.
- Local (no serialization): the win concentrates where columnar avoids 
per-tuple tuple construction (projection); pure aggregates and mostly-passing 
filters are near par.
A real multi-node cluster benchmark is the remaining measurement gap and is 
tracked as follow-up. Note that a large part of the projection speedup reflects 
how expensive Amber's per-tuple row path is, which is also worth attacking 
independently.

### Supersedes
This replaces the earlier four-PR outline in this thread; the old sub-issues 
(#8558 through #8561) are closed. Feedback on the split, the per-edge planner, 
or the measurement method is welcome here.

GitHub link: https://github.com/apache/texera/discussions/8571

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to