Gabriel39 commented on code in PR #68667:
URL: https://github.com/apache/doris/pull/68667#discussion_r4141474242
##########
fe/fe-core/src/main/java/org/apache/doris/planner/ResultSink.java:
##########
@@ -34,15 +35,20 @@ public class ResultSink extends DataSink {
// Two phase fetch option
private TFetchOption fetchOption;
+ private final boolean nativeVariant;
private TResultSinkType resultSinkType = TResultSinkType.MYSQL_PROTOCOL;
public ResultSink(PlanNodeId exchNodeId) {
- this.exchNodeId = exchNodeId;
+ this(exchNodeId, TResultSinkType.MYSQL_PROTOCOL);
}
public ResultSink(PlanNodeId exchNodeId, TResultSinkType resultSinkType) {
this.exchNodeId = exchNodeId;
this.resultSinkType = resultSinkType;
+ ConnectContext context = ConnectContext.get();
+ // The session may change before deferred result fetching; pin the
format during planning.
+ nativeVariant = resultSinkType ==
TResultSinkType.ARROW_FLIGHT_PROTOCOL && context != null
Review Comment:
Fixed in ff92c38643d. Added an optional BE heartbeat capability, persisted
through BackendHbResponse replay and Backend state. Both result-sink planning
and GetTables require every registered BE to advertise support, so
unknown/older result or proxy BEs keep UTF8 output. A missing capability on a
later heartbeat also clears previous support after a downgrade. FE tests cover
mixed-version gating, sink option capture, and heartbeat replay/downgrade.
##########
be/src/core/data_type_serde/data_type_variant_serde.cpp:
##########
@@ -157,6 +163,64 @@ Status DataTypeVariantSerDe::write_column_to_arrow(const
IColumn& column, const
int64_t start, int64_t end,
const cctz::time_zone& ctz)
const {
const auto* var = check_and_get_column<ColumnVariant>(column);
+ if (array_builder->type()->id() == arrow::Type::STRUCT) {
+ // Legacy documents need JSON conversion; typed scalar roots can keep
their type.
+ // The outer null map must remain SQL NULL on the wire.
+ if (start < 0 || end < start || end > column.size() ||
+ (null_map != nullptr && end > null_map->size())) {
+ return Status::InvalidArgument("Invalid Variant Arrow row range
[{}, {})", start, end);
+ }
+ if (var->is_scalar_variant()) {
+ auto scalar_type = remove_nullable(var->get_root_type());
+ if (scalar_type->get_primitive_type() == TYPE_DECIMAL256) {
+ return Status::NotSupported(
+ "Native Arrow Variant does not support Decimal256
roots");
+ }
+ if
(is_supported_variant_typed_identity(scalar_type->get_primitive_type())) {
+ // Avoid a JSON round trip that would turn exact decimal roots
into doubles.
+ auto typed =
+
ColumnVariantV2::create_typed(make_nullable(var->get_root()), scalar_type);
+ return DataTypeVariantV2SerDe().write_column_to_arrow(
+ *typed, null_map, array_builder, start, end, ctz);
+ }
+ }
+ JsonToVariantOptions parse_options;
+ parse_options.throw_on_invalid_json = true;
+ // Stored keys were already accepted at ingestion; mutable parse
limits must not reject reads.
+ parse_options.max_json_key_length =
std::numeric_limits<uint32_t>::max();
+ parse_options.check_duplicate_json_path = false;
+ JsonStringToVariantEncoder encoder(parse_options);
+ FormatOptions options;
+ options.timezone = &ctz;
+ NullMap selected_nulls;
+ if (null_map != nullptr) {
+ selected_nulls.assign(null_map->begin() + start, null_map->begin()
+ end);
+ }
+ for (int64_t row = start; row < end; ++row) {
+ std::string json;
+ if (null_map != nullptr && (*null_map)[row]) {
+ json = "null";
+ } else {
+ var->serialize_one_row_to_string(row, &json, options);
+ if (var->get_root_type()->get_primitive_type() == TYPE_STRING
&&
+ !var->get_root()->is_null_at(row)) {
+ // The legacy root string serializer emits raw text, not a
JSON string literal.
+ auto quoted = ColumnString::create();
+ VectorBufferWriter writer(*quoted);
+ writer.write_json_string(json);
+ writer.commit();
+ json = quoted->get_data_at(0).to_string();
+ }
+ }
+ encoder.add_json({json.data(), json.size()});
Review Comment:
Fixed in ff92c38643d. Visible legacy roots now use the existing typed
Variant V2 CAST encoder, including arrays and roots in mixed-path batches,
instead of reparsing their JSON text. Nullable roots and sliced row ranges
retain their masks. Added BE tests for DECIMAL(20,2) 9007199254740993.01 inside
an array, NaN/Infinity arrays, and a DATE root mixed with object rows
(including sliced output). All 37 related BE tests pass.
##########
be/src/core/data_type_serde/data_type_variant_serde.cpp:
##########
@@ -157,6 +163,64 @@ Status DataTypeVariantSerDe::write_column_to_arrow(const
IColumn& column, const
int64_t start, int64_t end,
const cctz::time_zone& ctz)
const {
const auto* var = check_and_get_column<ColumnVariant>(column);
+ if (array_builder->type()->id() == arrow::Type::STRUCT) {
+ // Legacy documents need JSON conversion; typed scalar roots can keep
their type.
+ // The outer null map must remain SQL NULL on the wire.
+ if (start < 0 || end < start || end > column.size() ||
+ (null_map != nullptr && end > null_map->size())) {
+ return Status::InvalidArgument("Invalid Variant Arrow row range
[{}, {})", start, end);
+ }
+ if (var->is_scalar_variant()) {
+ auto scalar_type = remove_nullable(var->get_root_type());
+ if (scalar_type->get_primitive_type() == TYPE_DECIMAL256) {
+ return Status::NotSupported(
+ "Native Arrow Variant does not support Decimal256
roots");
+ }
+ if
(is_supported_variant_typed_identity(scalar_type->get_primitive_type())) {
+ // Avoid a JSON round trip that would turn exact decimal roots
into doubles.
+ auto typed =
+
ColumnVariantV2::create_typed(make_nullable(var->get_root()), scalar_type);
+ return DataTypeVariantV2SerDe().write_column_to_arrow(
+ *typed, null_map, array_builder, start, end, ctz);
+ }
+ }
+ JsonToVariantOptions parse_options;
+ parse_options.throw_on_invalid_json = true;
+ // Stored keys were already accepted at ingestion; mutable parse
limits must not reject reads.
+ parse_options.max_json_key_length =
std::numeric_limits<uint32_t>::max();
+ parse_options.check_duplicate_json_path = false;
+ JsonStringToVariantEncoder encoder(parse_options);
Review Comment:
Addressed using the explicit compatibility-limit option in ff92c38643d. The
README documents the 128-level native encoding limit, and conversion errors
direct users to enable_arrow_flight_sql_native_variant=false for UTF8 output. A
129-level legacy document test verifies that UTF8 remains readable and native
output reports this actionable limitation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
