matanper commented on issue #68890: URL: https://github.com/apache/doris/issues/68890#issuecomment-6105764562
Update after implementing and load-testing proposal (2), the binary input, on a 4.1.4-based build: 4 BEs (192 vCPU, arm64), cloud mode, VARIANT V2, sustained Arrow Stream Load of semi-structured events. **1. The reader should also accept a dictionary-encoded `metadata` child.** The cheapest form for a producer is one shared metadata dictionary per record batch. As a plain `struct<metadata: binary, value: binary>`, that shared blob is copied into every row's `metadata` cell. In our test, a batch mixing about 10 event sources carried about 1 KB of metadata per row, and a batch mixing all sources about 45 KB per row, against about 1.3–1.4 KB of value bytes per row. With `metadata` as `dictionary<int32, binary>`, holding one entry per batch with every index 0, metadata drops to about 4–10 bytes per row. Each IPC row is then smaller than the same event as JSON text: about 2.5–3.0 KB vs 3.2–3.7 KB. On the BE this maps directly onto `ColumnVariantV2::insert_encoded_rows`: - one entry (or every row using the same index): the shared-metadata form, with empty `meta_ids` and no per-row metadata copy or hashing; - several entries: dictionary values become `metadata_bytes`/`offsets`, and indices become `meta_ids`. Our reader accepts int8–int64 and uint8–uint64 indices, binary or large_binary values, and rejects out-of-range or null indices on non-null rows. **2. Measured effect** A/B on the same build and load, comparing JSON text input with binary input using dictionary-encoded metadata: | | JSON text | binary + dictionary metadata | |---|---|---| | BE cores per 1M rows/s | 111 | 88 (−21%) | | Throughput (BE-bound in the JSON arm) | 6.46M rows/s | 7.44M rows/s (+15%) | The BE saving comes from skipping the JSON parse: the Arrow read path's CPU per row fell by about half. The remaining cost is validating and copying the encoded rows. The parse work moves to the producer, so the gain is in BE capacity, not total CPU. In our producer, which re-parses JSON it had just serialized, combined producer+BE CPU per row came out about equal. A producer that builds Variant directly from its in-memory values should come out ahead. So the updated proposal is that `read_column_from_arrow` for VARIANT V2 accepts: 1. JSON text (string/binary); 2. `struct<metadata, value>` with binary metadata; 3. the same struct with dictionary-encoded metadata. All three should be accepted with or without the `arrow.parquet.variant` extension name. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
