matanper opened a new issue, #68890: URL: https://github.com/apache/doris/issues/68890
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description Stream Load with `format=arrow` cannot load into VARIANT columns. `DataTypeVariantV2SerDe::read_column_from_arrow` returns `NOT_IMPLEMENTED_ERROR` on both 4.1.4 and master (`be/src/core/data_type_serde/data_type_variant_v2_serde.cpp`), so an Arrow load into any table with a VARIANT column fails. The same missing function is behind #61883 (a Doris catalog over Arrow Flight fails to read VARIANT). Arrow output for VARIANT is already defined: #68667 (branch-4.1) and #68883 (master) return Variant V2 as the canonical `arrow.parquet.variant` extension with `struct<metadata: binary not null, value: binary not null>` storage. ### Use case High-throughput ingestion of semi-structured events through Stream Load `format=arrow` (record batches with ZSTD/LZ4 compression) into tables with VARIANT columns. With Arrow, the client can send VARIANT data either as JSON text or already encoded in the Parquet Variant binary form, which saves the BE from parsing JSON on every row. ### Proposal Implement `read_column_from_arrow` for VARIANT V2, accepting: 1. `utf8` / `large_utf8` / `binary` / `large_binary` holding one JSON document per row (parsed like JSON loads); 2. the binary layout Doris already writes for Arrow: `struct<metadata: binary, value: binary>`, with or without the `arrow.parquet.variant` extension name, validated and appended through `ColumnVariantV2::insert_encoded_rows`, so no JSON is parsed on the BE. Rows in a batch may share one metadata blob. Plus the Arrow stream reader's schema checks accepting these Arrow types for a VARIANT column. We run (1) on a 4.1.4-based build in load tests at several million rows/s, and are implementing (2). ### Related issues #61883, #68667, #68883 ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
