GitHub user Ma77Ball edited a discussion: Columnar (Apache Arrow) execution in Amber: the opt-in PR series and why
## Columnar (Apache Arrow) execution in Amber: the opt-in PR series and why **Status, updated 2026-11-21.** This revives the thread with what actually shipped. The work lives on a reviewable, stacked PR series tracked by parent issue #9542, and the whole feature is off by default (`ApplicationConfig.enableColumnarWire`), so the row path is unchanged until a user opts in. Update since 2026-11-19: columnar now runs end to end on a single node, and sources emit Arrow directly (not just operators). ### Why Amber moves and processes data one row `Tuple` at a time at every operator and across every edge. For exchange- and scan-heavy workflows that costs two things: serializing an object graph per batch on a cross-worker edge, and building a JVM object per row at every stage. Columnar (Apache Arrow) execution addresses both: Arrow batches serialize at near memory-copy speed, and a columnar-capable operator reads the columns directly instead of materializing a tuple per row. This is the same mechanism Spark/Photon, Velox, Gluten, and DataFusion Comet use. ### The key design choice: decide per edge, not globally Columnar is not always a win. A same-JVM edge already passes data by reference, so wrapping it in Arrow bytes would only add overhead. So each edge picks its own format from two questions: can the receiver read columns, and is it local or remote. Remote and columnar-capable goes Arrow bytes (`VectorFrame`); local and columnar-capable gets a live Arrow root by reference (`LocalColumnarBatch`, zero copy, no serialization); everything else stays rows. Locality selects the delivery mechanism, it does not disable columnar. This is what keeps the feature from regressing a workflow, lets it benefit a single node as well as a cluster, and is why it can eventually default on. ### Columnar sources (newest work) The parts below made operators and edges columnar, but every source still emitted row `Tuple`s, so a scan-heavy columnar run still paid the per-row cost at the source. That gap is now closed: every source is columnar-capable. A generic default on `SourceOperatorExecutor` batches any source's output into Arrow (so SQL, JSON, text, and the rest work), with native no-`Tuple` fast paths for the bulk scans: Arrow (zero copy), Parquet (decode columns straight into Arrow), CSV (parse straight into vectors). A columnar source caps each Arrow batch at `columnarBatchRows` (default 10000, override with `-Dtexera.columnar.batch.rows`), so a large Parquet row group or Arrow record batch streams in bounded chunks instead of materializing whole, which fixes an OOM on big single-row-group files. Enabler: `Tuple.inMemSize` is now computed lazily, so the columnar path never pays the `deepSizeOf` object-graph walk that dominates the row path. This is on the feature branch and not yet split into the #95 42 series. ### The reviewable parts (parent #9542) - Part 1, Foundation (#9547): the `ColumnarOperatorExecutor` / `ColumnarResult` contract, the opt-in Arrow wire (`VectorFrame`), a faster Arrow-to-tuple decode, and vectorized receive for filter and aggregate. - Part 2, Output and per-edge planner (#9548): keep columns on output without the row round-trip, and decide the wire format per edge from the downstream operator. - Part 3, Shuffle and projection (#9544): partition an Arrow batch across workers without decoding, plus a native-Arrow projection operator. - Part 4, Locality-aware edges (#9545): keep same-JVM edges on rows, send columns only to the remote receivers on a mixed-locality edge. - Part 5, Local zero-copy (#9546): the `processColumnarRoot` primitive and handing a live Arrow root to a co-located receiver by reference (including the local multi-receiver shuffle, where each co-located worker gets its own gathered bucket by reference), with a checkpoint-safe serializer. ### Measured Operator-level, a source plus a vectorized filter over 800k rows by 22 columns, isolated from actor messaging, the wire, and result storage, with a correctness parity gate (columnar and row return identical rows): - Arrow source: row 33.0s to columnar 0.47s (about 70x; Arrow is read zero copy). - Parquet source: row 34.5s to columnar 2.9s (about 12x). - CSV source: row 35.2s to columnar 2.7s (about 13x). The dominant row-path cost is confirmed to be `Tuple.inMemSize`'s per-tuple `deepSizeOf` walk, which the columnar path skips; for CSV the text parse itself is only about 3s. Earlier single-JVM serialization simulation, median of 9 trials, across a matrix of workflow shapes with the same parity gate: - Cross-node (serialized edges): 3x to 50x faster at 100k rows on exchange- and projection-heavy workflows; small batches (about 1k rows) can lose, since Arrow's fixed per-batch overhead is not amortized. - Local (no serialization): the win concentrates where columnar avoids per-tuple tuple construction (projection); pure aggregates and mostly-passing filters are near par. Live end to end on a single node (ParquetScan to a selective Filter, 800k rows): row about 35s vs columnar about 8s (about 4.5x; actor, wire, and storage overhead is paid by both paths). A real multi-node cluster benchmark is the remaining measurement gap and is tracked as follow-up. ### Supersedes This replaces the earlier four-PR outline in this thread; the old sub-issues (#8558 through #8561) are closed. Feedback on the split, the per-edge planner, or the measurement method is welcome here. GitHub link: https://github.com/apache/texera/discussions/8571 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
