GitHub user Ma77Ball edited a discussion: Columnar (Apache Arrow) execution in Amber: the opt-in PR series and why
## Columnar (Apache Arrow) execution in Amber: the opt-in PR series and why **Status, updated 2026-11-19.** This revives the thread with what actually shipped. The work now lives on a reviewable, stacked PR series tracked by parent issue #9542, and the whole feature is off by default (`ApplicationConfig.enableColumnarWire`), so the row path is unchanged until a user opts in. ### Why Amber moves and processes data one row `Tuple` at a time at every operator and across every edge. For exchange- and scan-heavy workflows that costs two things: serializing an object graph per batch on a cross-worker edge, and building a JVM object per row at every stage. Columnar (Apache Arrow) execution addresses both: Arrow batches serialize at near memory-copy speed, and a columnar-capable operator reads the columns directly instead of materializing a tuple per row. This is the same mechanism Spark/Photon, Velox, Gluten, and DataFusion Comet use. ### The key design choice: decide per edge, not globally Columnar is not always a win. A same-JVM edge already passes data by reference, so wrapping it in Arrow bytes would only add overhead. So each edge picks its own format from two questions: can the receiver read columns, and is it local or remote. Remote and columnar-capable goes Arrow; local and columnar-capable gets a live root by reference (zero copy); everything else stays rows. This is what keeps the feature from ever regressing a workflow, and it is why it can eventually default on. ### The reviewable parts (parent #9542) - Part 1, Foundation (#9547): the `ColumnarOperatorExecutor` / `ColumnarResult` contract, the opt-in `ColumnarFrame` wire, a faster Arrow-to-tuple decode, and vectorized receive for filter and aggregate. - Part 2, Output and per-edge planner (#9548): keep columns on output without the row round-trip, and decide the wire format per edge from the downstream operator. - Part 3, Shuffle and projection (#9544): partition an Arrow batch across workers without decoding, plus a native-Arrow projection operator. - Part 4, Locality-aware edges (#9545): keep same-JVM edges on rows, send columns only to the remote receivers on a mixed-locality edge. - Part 5, Local zero-copy (#9546): the `processColumnarRoot` primitive and handing a live Arrow root to a co-located receiver by reference, with a checkpoint-safe serializer. ### Measured so far Single-JVM serialization simulation, median of 9 trials, across a matrix of workflow shapes with a correctness parity gate (the columnar and row paths must return identical results per cell): - Cross-node (serialized edges): 3x to 50x faster at 100k rows on exchange- and projection-heavy workflows; small batches (about 1k rows) can lose, since Arrow's fixed per-batch overhead is not amortized. - Local (no serialization): the win concentrates where columnar avoids per-tuple tuple construction (projection); pure aggregates and mostly-passing filters are near par. A real multi-node cluster benchmark is the remaining measurement gap and is tracked as follow-up. Note that a large part of the projection speedup reflects how expensive Amber's per-tuple row path is, which is also worth attacking independently. ### Supersedes This replaces the earlier four-PR outline in this thread; the old sub-issues (#8558 through #8561) are closed. Feedback on the split, the per-edge planner, or the measurement method is welcome here. GitHub link: https://github.com/apache/texera/discussions/8571 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
