This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new 4a3fce2a8fc [refactor](load) Deserialize simdjson values through type 
SerDes (#70470)
4a3fce2a8fc is described below

commit 4a3fce2a8fc1000b90f14793241aeeb574402e7e
Author: Mryange <[email protected]>
AuthorDate: Tue Nov 17 10:45:37 2026 +0800

    [refactor](load) Deserialize simdjson values through type SerDes (#70470)
    
    ### Background
    
    The JSON reader currently converts simdjson values to columns through a
    type-dispatch chain. JSONB objects and arrays are serialized to JSON
    text and parsed again before insertion.
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: None
    PR Type: Refactor
    
    Problem Summary:
    Move simdjson-to-column conversion into
    `DataTypeSerDe::deserialize_one_cell_from_simdjson`. Complex types
    recursively use their child SerDes, and JSONB objects and arrays are
    encoded directly from the parsed subtree. Scalar types retain the
    existing text conversion, including boolean coercion.
    
    ### Details
    
    - **Change map:** 511 changed production lines across the SerDe
    interface/implementations, reader integration, read context, and JSONB
    parser helpers; 381 added test/fixture/result lines; no generated source
    or documentation changes.
    - **Moves vs. logic changes:** Move Array/Map/Struct/Nullable dispatch
    into SerDes. Add a default text fallback and a direct JSONB subtree
    encoder. Keep the string cache and writer in reader-owned context for
    overlapping JSONPaths.
    - **Behavior change:** No intended user-visible semantic change; legacy
    scalar conversions and error propagation are retained.
    - **Compatibility:** No storage format, protocol, or configuration
    changes. Independent of parser-reuse changes.
    - **Performance impact:** Removes the JSON text/reparse round trip for
    JSONB objects and arrays. No benchmark claim; the common SerDe header
    now includes simdjson.
    - **Rollback:** Revert this PR.
    
    ### Testing
    
    Latest validation at `d702ec80284` on the development machine:
    - Added two Reader UTs for ignored malformed nested Array/Map followed
    by an empty collection, each exercised with field-name and JSONPath
    projection. Both tests failed before the fix: the next empty collection
    offset was 1 for Array and 2 for Map instead of 0.
    - Restored legacy `previous offset + current entry count` for Array and
    Map. No new rollback semantics were added.
    - ASAN `run-be-ut.sh --run -j48
    
--filter="SimdJsonSerDeTest.*:JsonReader*Test.*:JsonbParserTest.*:DataType*SerDeTest.*"`:
    all 179 tests from 16 suites passed after the fix.
    - Repository clang-format 16, build hygiene, `git diff --check`, and
    clang-tidy on all three changed files passed.
    
    Earlier validation:
    - ASAN BE/FE builds passed at `c44d740953a`.
    - Regression suites `test_simdjson_column_deserialization` and
    `test_load_json_with_jsonpath` passed on the earlier build. The new
    regression output was generated with the regression runner.
    
    Final code is compiled and tested by the ASAN UT build. Full BE/FE
    builds and regression tests were not rerun for these final adjustments.
    No greater-than-1-GiB reproduction was run; the direct JSONB subtree
    path preserves the legacy size check and InternalError.
    
    ### Release note
    
    None
    
    ### Check List (For Author)
    
    - Behavior changed: No intended user-visible change
    - Does this need documentation: No
    - Pre-submission status: Partially verified: final HEAD passed ASAN UT
    compilation and 179 tests; full BE/FE builds and regression tests were
    not rerun.
    
    ### AI Disclosure
    
    - Tool: Codex
    - Generated: Implementation, tests, and PR description.
    - Human review: The author reviewed the main implementation and
    requested interface and compatibility adjustments; BE unit tests were
    not manually reviewed.
---
 .../core/data_type_serde/data_type_array_serde.cpp |  22 +++
 .../core/data_type_serde/data_type_array_serde.h   |   5 +
 .../core/data_type_serde/data_type_jsonb_serde.cpp |  23 +++
 .../core/data_type_serde/data_type_jsonb_serde.h   |   5 +
 .../core/data_type_serde/data_type_map_serde.cpp   |  41 +++++
 be/src/core/data_type_serde/data_type_map_serde.h  |   5 +
 .../data_type_serde/data_type_nullable_serde.cpp   |  16 ++
 .../data_type_serde/data_type_nullable_serde.h     |   5 +
 be/src/core/data_type_serde/data_type_serde.cpp    |  27 +++
 be/src/core/data_type_serde/data_type_serde.h      |  11 ++
 .../data_type_serde/data_type_struct_serde.cpp     |  54 ++++++
 .../core/data_type_serde/data_type_struct_serde.h  |   6 +-
 .../core/data_type_serde/simdjson_read_context.h   |  43 +++++
 be/src/format_v2/json/json_reader.cpp              | 183 ++-----------------
 be/src/format_v2/json/json_reader.h                |   3 +-
 be/src/util/jsonb_parser_simd.h                    |  62 ++++++-
 .../core/data_type_serde/simdjson_serde_test.cpp   | 201 +++++++++++++++++++++
 be/test/format_v2/json/json_reader_test.cpp        |  91 ++++++++++
 .../jsonb_p0/simdjson_column_deserialization.json  |   4 +
 .../test_simdjson_column_deserialization.out       |  11 ++
 .../test_simdjson_column_deserialization.groovy    |  74 ++++++++
 21 files changed, 712 insertions(+), 180 deletions(-)

diff --git a/be/src/core/data_type_serde/data_type_array_serde.cpp 
b/be/src/core/data_type_serde/data_type_array_serde.cpp
index 8c1c2f644f6..e51230fc417 100644
--- a/be/src/core/data_type_serde/data_type_array_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_array_serde.cpp
@@ -36,6 +36,7 @@
 #include "core/data_type_serde/bson_value_view.h"
 #include "core/data_type_serde/complex_type_deserialize_util.h"
 #include "core/data_type_serde/orc_serde_utils.h"
+#include "core/data_type_serde/simdjson_read_context.h"
 #include "core/string_ref.h"
 #include "exprs/function/function_helpers.h"
 #include "storage/field_type.h"
@@ -847,4 +848,25 @@ Status DataTypeArraySerDe::decode_bson_value(IColumn& 
column, const BsonValueVie
     return Status::OK();
 }
 
+Status DataTypeArraySerDe::deserialize_one_cell_from_simdjson(
+        IColumn& column, simdjson::ondemand::value& input, const 
FormatOptions& options,
+        SimdJsonReadContext& context, const std::string& column_name) const {
+    if (input.type().value() != simdjson::ondemand::json_type::array) {
+        return Status::DataQualityError("Json value isn't array, but the 
column `{}` is array.",
+                                        column_name);
+    }
+    auto& target = assert_cast<ColumnArray&>(column);
+    auto& data = target.get_data();
+
+    size_t field_count = 0;
+    for (simdjson::ondemand::value element : input.get_array()) {
+        RETURN_IF_ERROR(nested_serde->deserialize_one_cell_from_simdjson(
+                data, element, options, context, column_name + ".element"));
+        ++field_count;
+    }
+    auto& offsets = target.get_offsets();
+    offsets.push_back(offsets.back() + field_count);
+    return Status::OK();
+}
+
 } // namespace doris
diff --git a/be/src/core/data_type_serde/data_type_array_serde.h 
b/be/src/core/data_type_serde/data_type_array_serde.h
index 3d55eccc856..b491ef879ca 100644
--- a/be/src/core/data_type_serde/data_type_array_serde.h
+++ b/be/src/core/data_type_serde/data_type_array_serde.h
@@ -63,6 +63,11 @@ public:
     Status serialize_column_to_json(const IColumn& column, int64_t start_idx, 
int64_t end_idx,
                                     BufferWritable& bw, FormatOptions& 
options) const override;
 
+    Status deserialize_one_cell_from_simdjson(IColumn& column, 
simdjson::ondemand::value& value,
+                                              const FormatOptions& options,
+                                              SimdJsonReadContext& context,
+                                              const std::string& column_name) 
const override;
+
     Status deserialize_one_cell_from_json(IColumn& column, Slice& slice,
                                           const FormatOptions& options) const 
override;
 
diff --git a/be/src/core/data_type_serde/data_type_jsonb_serde.cpp 
b/be/src/core/data_type_serde/data_type_jsonb_serde.cpp
index b7e79184477..889dde9f82e 100644
--- a/be/src/core/data_type_serde/data_type_jsonb_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_jsonb_serde.cpp
@@ -27,9 +27,11 @@
 #include "common/status.h"
 #include "core/data_type_serde/arrow_validation.h"
 #include "core/data_type_serde/bson_value_view.h"
+#include "core/data_type_serde/simdjson_read_context.h"
 #include "core/value/jsonb_value.h"
 #include "exprs/json_functions.h"
 #include "util/jsonb_parser_simd.h"
+
 namespace doris {
 
 Status DataTypeJsonbSerDe::write_column_to_mysql_binary(const IColumn& column,
@@ -390,4 +392,25 @@ Status DataTypeJsonbSerDe::decode_bson_value(IColumn& 
column, const BsonValueVie
                        : Status::InvalidArgument("BSON Extended JSON exceeds 
JSONB limits");
 }
 
+Status DataTypeJsonbSerDe::deserialize_one_cell_from_simdjson(
+        IColumn& column, simdjson::ondemand::value& input, const 
FormatOptions& options,
+        SimdJsonReadContext& context, const std::string& column_name) const {
+    const auto type = input.type().value();
+    if (type != simdjson::ondemand::json_type::object &&
+        type != simdjson::ondemand::json_type::array) {
+        return DataTypeSerDe::deserialize_one_cell_from_simdjson(column, 
input, options, context,
+                                                                 column_name);
+    }
+    auto& writer = context.jsonb_writer;
+    RETURN_IF_ERROR(JsonbParser::parse_value(input, writer, context.cache()));
+    const auto size = writer.getOutput()->getSize();
+    if (size > JsonBinaryValue::MAX_LENGTH) {
+        return Status::InternalError(
+                "Jsonb value length {} exceeds maximum allowed length of {} 
bytes", size,
+                JsonBinaryValue::MAX_LENGTH);
+    }
+    column.insert_data(writer.getOutput()->getBuffer(), size);
+    return Status::OK();
+}
+
 } // namespace doris
diff --git a/be/src/core/data_type_serde/data_type_jsonb_serde.h 
b/be/src/core/data_type_serde/data_type_jsonb_serde.h
index 8c4d3c55de2..e8a377d261f 100644
--- a/be/src/core/data_type_serde/data_type_jsonb_serde.h
+++ b/be/src/core/data_type_serde/data_type_jsonb_serde.h
@@ -63,6 +63,11 @@ public:
     Status serialize_column_to_json(const IColumn& column, int64_t start_idx, 
int64_t end_idx,
                                     BufferWritable& bw, FormatOptions& 
options) const override;
 
+    Status deserialize_one_cell_from_simdjson(IColumn& column, 
simdjson::ondemand::value& value,
+                                              const FormatOptions& options,
+                                              SimdJsonReadContext& context,
+                                              const std::string& column_name) 
const override;
+
     Status deserialize_one_cell_from_json(IColumn& column, Slice& slice,
                                           const FormatOptions& options) const 
override;
 
diff --git a/be/src/core/data_type_serde/data_type_map_serde.cpp 
b/be/src/core/data_type_serde/data_type_map_serde.cpp
index 53cf76936cf..545b1bb9e68 100644
--- a/be/src/core/data_type_serde/data_type_map_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_map_serde.cpp
@@ -33,7 +33,9 @@
 #include "core/data_type_serde/arrow_validation.h"
 #include "core/data_type_serde/complex_type_deserialize_util.h"
 #include "core/data_type_serde/orc_serde_utils.h"
+#include "core/data_type_serde/simdjson_read_context.h"
 #include "core/string_ref.h"
+#include "util/defer_op.h"
 #include "util/jsonb_document.h"
 #include "util/jsonb_writer.h"
 #include "util/simd/bits.h"
@@ -912,4 +914,43 @@ Status DataTypeMapSerDe::read_column_from_orc(IColumn& 
column,
     return decode_map_orc_values(key_serde, value_serde, column, view);
 }
 
+Status DataTypeMapSerDe::deserialize_one_cell_from_simdjson(IColumn& column,
+                                                            
simdjson::ondemand::value& input,
+                                                            const 
FormatOptions& options,
+                                                            
SimdJsonReadContext& context,
+                                                            const std::string& 
column_name) const {
+    if (input.type().value() != simdjson::ondemand::json_type::object) {
+        return Status::DataQualityError("Json value isn't object, but the 
column `{}` is map.",
+                                        column_name);
+    }
+    auto& target = assert_cast<ColumnMap&>(column);
+    auto keys_owner = target.get_keys_ptr()->assert_mutable();
+    auto values_owner = target.get_values_ptr()->assert_mutable();
+    auto& keys = *keys_owner;
+    auto& values = *values_owner;
+
+    size_t field_count = 0;
+    for (simdjson::ondemand::field field : input.get_object()) {
+        const auto key = context.decoded_key(field);
+        Slice slice {key.data(), key.size()};
+        // Object keys are decoded strings, not JSON tokens; preserve the 
reader's text conversion.
+        if (keys.is_nullable()) {
+            auto& nullable = assert_cast<ColumnNullable&>(keys);
+            auto& nested_key = nullable.get_nested_column();
+            nullable.get_null_map_data().push_back(0);
+            
RETURN_IF_ERROR(key_serde->get_nested_serdes()[0]->deserialize_one_cell_from_json(
+                    nested_key, slice, options));
+        } else {
+            RETURN_IF_ERROR(key_serde->deserialize_one_cell_from_json(keys, 
slice, options));
+        }
+        auto value = field.value();
+        RETURN_IF_ERROR(value_serde->deserialize_one_cell_from_simdjson(
+                values, value, options, context, column_name + ".value"));
+        ++field_count;
+    }
+    auto& offsets = target.get_offsets();
+    offsets.push_back(offsets.back() + field_count);
+    return Status::OK();
+}
+
 } // namespace doris
diff --git a/be/src/core/data_type_serde/data_type_map_serde.h 
b/be/src/core/data_type_serde/data_type_map_serde.h
index bd6560c053c..1d04928b017 100644
--- a/be/src/core/data_type_serde/data_type_map_serde.h
+++ b/be/src/core/data_type_serde/data_type_map_serde.h
@@ -51,6 +51,11 @@ public:
                                       FormatOptions& options) const override;
     Status serialize_column_to_json(const IColumn& column, int64_t start_idx, 
int64_t end_idx,
                                     BufferWritable& bw, FormatOptions& 
options) const override;
+    Status deserialize_one_cell_from_simdjson(IColumn& column, 
simdjson::ondemand::value& value,
+                                              const FormatOptions& options,
+                                              SimdJsonReadContext& context,
+                                              const std::string& column_name) 
const override;
+
     Status deserialize_one_cell_from_json(IColumn& column, Slice& slice,
                                           const FormatOptions& options) const 
override;
 
diff --git a/be/src/core/data_type_serde/data_type_nullable_serde.cpp 
b/be/src/core/data_type_serde/data_type_nullable_serde.cpp
index 90fe0af9fb3..941a31ae25f 100644
--- a/be/src/core/data_type_serde/data_type_nullable_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_nullable_serde.cpp
@@ -38,6 +38,7 @@
 #include "core/data_type_serde/data_type_string_serde.h"
 #include "core/data_type_serde/decoded_column_view.h"
 #include "core/data_type_serde/orc_serde_utils.h"
+#include "core/data_type_serde/simdjson_read_context.h"
 #include "exprs/function/cast/cast_base.h"
 #include "format/transformer/vcsv_transformer.h"
 #include "util/defer_op.h"
@@ -733,4 +734,19 @@ Status DataTypeNullableSerDe::decode_bson_value(IColumn& 
column, const BsonValue
     return Status::OK();
 }
 
+Status DataTypeNullableSerDe::deserialize_one_cell_from_simdjson(
+        IColumn& column, simdjson::ondemand::value& input, const 
FormatOptions& options,
+        SimdJsonReadContext& context, const std::string& column_name) const {
+    auto& target = assert_cast<ColumnNullable&>(column);
+    if (input.type().value() == simdjson::ondemand::json_type::null) {
+        target.insert_default();
+        return Status::OK();
+    }
+
+    RETURN_IF_ERROR(nested_serde->deserialize_one_cell_from_simdjson(
+            target.get_nested_column(), input, options, context, column_name));
+    target.get_null_map_data().push_back(0);
+    return Status::OK();
+}
+
 } // namespace doris
diff --git a/be/src/core/data_type_serde/data_type_nullable_serde.h 
b/be/src/core/data_type_serde/data_type_nullable_serde.h
index 119e68f4fd9..5a51f83de83 100644
--- a/be/src/core/data_type_serde/data_type_nullable_serde.h
+++ b/be/src/core/data_type_serde/data_type_nullable_serde.h
@@ -54,6 +54,11 @@ public:
                                       FormatOptions& options) const override;
     Status serialize_column_to_json(const IColumn& column, int64_t start_idx, 
int64_t end_idx,
                                     BufferWritable& bw, FormatOptions& 
options) const override;
+    Status deserialize_one_cell_from_simdjson(IColumn& column, 
simdjson::ondemand::value& value,
+                                              const FormatOptions& options,
+                                              SimdJsonReadContext& context,
+                                              const std::string& column_name) 
const override;
+
     Status deserialize_one_cell_from_json(IColumn& column, Slice& slice,
                                           const FormatOptions& options) const 
override;
     Status deserialize_one_cell_from_csv(IColumn& column, Slice& slice,
diff --git a/be/src/core/data_type_serde/data_type_serde.cpp 
b/be/src/core/data_type_serde/data_type_serde.cpp
index 40f0c857864..42e7dee9910 100644
--- a/be/src/core/data_type_serde/data_type_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_serde.cpp
@@ -34,11 +34,13 @@
 #include "core/data_type_serde/data_type_number_serde.h"
 #include "core/data_type_serde/data_type_string_serde.h"
 #include "core/data_type_serde/parquet_decode_source.h"
+#include "core/data_type_serde/simdjson_read_context.h"
 #include "core/field.h"
 #include "exprs/function/cast/cast_base.h"
 #include "runtime/descriptors.h"
 #include "util/jsonb_document.h"
 #include "util/jsonb_writer.h"
+
 namespace doris {
 DataTypeSerDe::~DataTypeSerDe() = default;
 
@@ -423,4 +425,29 @@ Status DataTypeSerDe::decode_bson_value(IColumn&, const 
BsonValueView&,
     return Status::NotSupported("BSON decoding is not supported for this Doris 
type");
 }
 
+Status DataTypeSerDe::deserialize_one_cell_from_simdjson(IColumn& column,
+                                                         
simdjson::ondemand::value& value,
+                                                         const FormatOptions& 
options,
+                                                         SimdJsonReadContext& 
context,
+                                                         const std::string& 
column_name) const {
+    std::string_view text;
+    switch (value.type().value()) {
+    case simdjson::ondemand::json_type::null:
+        return Status::DataQualityError("Json value is null, but the column 
`{}` is not nullable.",
+                                        column_name);
+    case simdjson::ondemand::json_type::string:
+        text = JsonbParser::decoded_string(value, context.cache());
+        break;
+    case simdjson::ondemand::json_type::boolean:
+        // Preserve legacy true -> "1" coercion for text-based 
String/Decimal/JSONB targets.
+        text = value.get_bool() ? "1" : "0";
+        break;
+    default:
+        text = simdjson::to_json_string(value);
+        break;
+    }
+    Slice slice {text.data(), text.size()};
+    return deserialize_one_cell_from_json(column, slice, options);
+}
+
 } // namespace doris
diff --git a/be/src/core/data_type_serde/data_type_serde.h 
b/be/src/core/data_type_serde/data_type_serde.h
index 0693d68b731..91372205b8a 100644
--- a/be/src/core/data_type_serde/data_type_serde.h
+++ b/be/src/core/data_type_serde/data_type_serde.h
@@ -17,6 +17,8 @@
 
 #pragma once
 
+#include <simdjson.h>
+
 #include <cstddef>
 #include <cstdint>
 #include <memory>
@@ -93,6 +95,7 @@ struct CastParameters;
 
 struct BsonValueView;
 struct BsonDecodeContext;
+struct SimdJsonReadContext;
 class DataTypeSerDe;
 using DataTypeSerDeSPtr = std::shared_ptr<DataTypeSerDe>;
 using DataTypeSerDeSPtrs = std::vector<DataTypeSerDeSPtr>;
@@ -392,6 +395,14 @@ public:
                                             int64_t end_idx, BufferWritable& 
bw,
                                             FormatOptions& options) const = 0;
 
+    // Consumes one borrowed value and appends one cell. Errors propagate to 
the caller;
+    // the reader retains responsibility for handling invalid rows.
+    virtual Status deserialize_one_cell_from_simdjson(IColumn& column,
+                                                      
simdjson::ondemand::value& value,
+                                                      const FormatOptions& 
options,
+                                                      SimdJsonReadContext& 
context,
+                                                      const std::string& 
column_name) const;
+
     virtual Status deserialize_one_cell_from_json(IColumn& column, Slice& 
slice,
                                                   const FormatOptions& 
options) const = 0;
 
diff --git a/be/src/core/data_type_serde/data_type_struct_serde.cpp 
b/be/src/core/data_type_serde/data_type_struct_serde.cpp
index 28d5b351160..abf4eb6273e 100644
--- a/be/src/core/data_type_serde/data_type_struct_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_struct_serde.cpp
@@ -20,6 +20,9 @@
 #include <gen_cpp/types.pb.h>
 
 #include <algorithm>
+#include <cctype>
+#include <ranges>
+#include <unordered_map>
 
 #include "arrow/array/builder_nested.h"
 #include "common/cast_set.h"
@@ -35,6 +38,7 @@
 #include "core/data_type_serde/complex_type_deserialize_util.h"
 #include "core/data_type_serde/data_type_serde.h"
 #include "core/data_type_serde/orc_serde_utils.h"
+#include "core/data_type_serde/simdjson_read_context.h"
 #include "core/string_ref.h"
 #include "util/defer_op.h"
 #include "util/jsonb_document.h"
@@ -896,4 +900,54 @@ Status DataTypeStructSerDe::decode_bson_value(IColumn& 
column, const BsonValueVi
     return Status::OK();
 }
 
+Status DataTypeStructSerDe::deserialize_one_cell_from_simdjson(
+        IColumn& column, simdjson::ondemand::value& input, const 
FormatOptions& options,
+        SimdJsonReadContext& context, const std::string& column_name) const {
+    if (input.type().value() != simdjson::ondemand::json_type::object) {
+        return Status::DataQualityError("Json value isn't object, but the 
column `{}` is struct.",
+                                        column_name);
+    }
+    auto& target = assert_cast<ColumnStruct&>(column);
+
+    std::unordered_map<std::string, size_t> field_positions;
+    for (size_t i = 0; i < elem_names.size(); ++i) {
+        auto name = elem_names[i];
+        std::ranges::transform(name, name.begin(), [](unsigned char c) { 
return std::tolower(c); });
+        field_positions.emplace(std::move(name), i);
+    }
+    std::vector<bool> seen(elem_names.size(), false);
+    for (simdjson::ondemand::field member : input.get_object()) {
+        std::string name(context.decoded_key(member));
+        std::ranges::transform(name, name.begin(), [](unsigned char c) { 
return std::tolower(c); });
+        const auto it = field_positions.find(name);
+        if (it == field_positions.end()) {
+            continue;
+        }
+        const auto index = it->second;
+        auto& child = target.get_column(index);
+        // Preserve nested STRUCT's last-wins behavior, e.g. {"a":1,"A":2}.
+        if (seen[index]) {
+            child.pop_back(1);
+        }
+        auto value = member.value();
+        
RETURN_IF_ERROR(elem_serdes_ptrs[index]->deserialize_one_cell_from_simdjson(
+                child, value, options, context, column_name + "." + name));
+        seen[index] = true;
+    }
+    for (size_t i = 0; i < elem_names.size(); ++i) {
+        if (seen[i]) {
+            continue;
+        }
+        auto& child = target.get_column(i);
+        if (!child.is_nullable()) {
+            return Status::DataQualityError(
+                    "Json file structColumn miss field {} and this column 
isn't nullable.",
+                    column_name + "." + elem_names[i]);
+        }
+        child.insert_default();
+    }
+
+    return Status::OK();
+}
+
 } // namespace doris
diff --git a/be/src/core/data_type_serde/data_type_struct_serde.h 
b/be/src/core/data_type_serde/data_type_struct_serde.h
index 90e3cc3e7a0..8d4634556d9 100644
--- a/be/src/core/data_type_serde/data_type_struct_serde.h
+++ b/be/src/core/data_type_serde/data_type_struct_serde.h
@@ -16,7 +16,6 @@
 // under the License.
 
 #pragma once
-
 #include <glog/logging.h>
 
 #include <cstdint>
@@ -56,6 +55,11 @@ public:
     Status serialize_column_to_json(const IColumn& column, int64_t start_idx, 
int64_t end_idx,
                                     BufferWritable& bw, FormatOptions& 
options) const override;
 
+    Status deserialize_one_cell_from_simdjson(IColumn& column, 
simdjson::ondemand::value& value,
+                                              const FormatOptions& options,
+                                              SimdJsonReadContext& context,
+                                              const std::string& column_name) 
const override;
+
     Status deserialize_one_cell_from_json(IColumn& column, Slice& slice,
                                           const FormatOptions& options) const 
override;
 
diff --git a/be/src/core/data_type_serde/simdjson_read_context.h 
b/be/src/core/data_type_serde/simdjson_read_context.h
new file mode 100644
index 00000000000..a16b4844e95
--- /dev/null
+++ b/be/src/core/data_type_serde/simdjson_read_context.h
@@ -0,0 +1,43 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <string>
+#include <string_view>
+#include <unordered_map>
+
+#include "core/data_type_serde/data_type_serde.h"
+#include "util/jsonb_parser_simd.h"
+#include "util/jsonb_writer.h"
+
+namespace doris {
+// Reader-owned state, never stored in shared SerDes. Clear the cache before 
the next document:
+// its keys and values borrow the input and parser buffers.
+struct SimdJsonReadContext {
+    bool use_string_cache = false;
+    std::unordered_map<std::string_view, std::string_view> string_values;
+    JsonbWriter jsonb_writer;
+
+    JsonbParser::StringCache* cache() { return use_string_cache ? 
&string_values : nullptr; }
+    std::string_view decoded_key(simdjson::ondemand::field& field) {
+        return JsonbParser::decoded_key(field, cache());
+    }
+};
+} // namespace doris
diff --git a/be/src/format_v2/json/json_reader.cpp 
b/be/src/format_v2/json/json_reader.cpp
index db8a8e93c8a..3a3e2fe3615 100644
--- a/be/src/format_v2/json/json_reader.cpp
+++ b/be/src/format_v2/json/json_reader.cpp
@@ -23,18 +23,14 @@
 #include <algorithm>
 #include <cstring>
 #include <limits>
-#include <map>
 #include <string_view>
 #include <utility>
 
 #include "common/cast_set.h"
 #include "core/assert_cast.h"
 #include "core/block/block.h"
-#include "core/column/column_array.h"
-#include "core/column/column_map.h"
 #include "core/column/column_nullable.h"
 #include "core/column/column_string.h"
-#include "core/column/column_struct.h"
 #include "core/data_type/data_type_array.h"
 #include "core/data_type/data_type_map.h"
 #include "core/data_type/data_type_nullable.h"
@@ -151,12 +147,6 @@ ColumnDefinition synthetic_file_child(const std::string& 
name, DataTypePtr type,
     return child;
 }
 
-std::string lower_key(std::string_view key) {
-    std::string lowered(key.data(), key.size());
-    std::transform(lowered.begin(), lowered.end(), lowered.begin(), ::tolower);
-    return lowered;
-}
-
 } // namespace
 
 JsonReader::JsonReader(std::shared_ptr<io::FileSystemProperties>& 
system_properties,
@@ -469,7 +459,7 @@ Status JsonReader::close() {
     _requested_columns.clear();
     _slot_name_to_index.clear();
     _previous_positions.clear();
-    _cached_string_values.clear();
+    _simdjson_context.string_values.clear();
     return Status::OK();
 }
 
@@ -994,7 +984,8 @@ Status 
JsonReader::_write_columns_by_jsonpath(simdjson::ondemand::object* object
     DORIS_CHECK(valid != nullptr);
     bool has_valid_value = false;
     const size_t cur_row_count = _logical_rows;
-    _cached_string_values.clear();
+    _simdjson_context.string_values.clear();
+    _simdjson_context.use_string_cache = true;
 
     // Even without projected fields, jsonpaths still decides whether the 
object contributes a row.
     if (_requested_columns.empty()) {
@@ -1066,165 +1057,21 @@ Status 
JsonReader::_write_data_to_column(simdjson::ondemand::value& value,
                                          const DataTypePtr& type_desc, 
IColumn* column_ptr,
                                          const std::string& column_name,
                                          const DataTypeSerDeSPtr& serde, bool* 
valid) {
-    ColumnNullable* nullable_column = nullptr;
-    IColumn* data_column_ptr = column_ptr;
-    DataTypeSerDeSPtr data_serde = serde;
-    const auto value_type = value.type().value();
-
-    if (is_column_nullable(*column_ptr)) {
-        nullable_column = assert_cast<ColumnNullable*>(column_ptr);
-        data_column_ptr = &nullable_column->get_nested_column();
-        if (type_desc->is_nullable()) {
-            data_serde = serde->get_nested_serdes()[0];
-        }
-        if (value_type == simdjson::ondemand::json_type::null) {
-            nullable_column->insert_default();
-            *valid = true;
-            return Status::OK();
-        }
-    } else if (value_type == simdjson::ondemand::json_type::null) {
-        return Status::DataQualityError("Json value is null, but the column 
`{}` is not nullable.",
-                                        column_name);
-    }
-
-    const auto primitive_type = type_desc->get_primitive_type();
-    if (!is_complex_type(primitive_type)) {
-        if (value_type == simdjson::ondemand::json_type::string) {
-            std::string_view value_string;
-            if constexpr (use_string_cache) {
-                const auto cache_key = value.raw_json().value();
-                if (_cached_string_values.contains(cache_key)) {
-                    value_string = _cached_string_values[cache_key];
-                } else {
-                    value_string = value.get_string();
-                    _cached_string_values.emplace(cache_key, value_string);
-                }
-            } else {
-                value_string = value.get_string();
-            }
-            Slice slice {value_string.data(), value_string.size()};
-            
RETURN_IF_ERROR(data_serde->deserialize_one_cell_from_json(*data_column_ptr, 
slice,
-                                                                       
_serde_options));
-        } else if (value_type == simdjson::ondemand::json_type::boolean) {
-            const char* str_value = value.get_bool() ? "1" : "0";
-            Slice slice {str_value, 1};
-            
RETURN_IF_ERROR(data_serde->deserialize_one_cell_from_json(*data_column_ptr, 
slice,
-                                                                       
_serde_options));
+    _simdjson_context.use_string_cache = use_string_cache;
+    // File slots can be logically non-nullable while their materialized 
columns are nullable.
+    if (column_ptr->is_nullable() && !type_desc->is_nullable()) {
+        auto& nullable = assert_cast<ColumnNullable&>(*column_ptr);
+        if (value.type().value() == simdjson::ondemand::json_type::null) {
+            nullable.insert_default();
         } else {
-            std::string_view json_str = simdjson::to_json_string(value);
-            Slice slice {json_str.data(), json_str.size()};
-            
RETURN_IF_ERROR(data_serde->deserialize_one_cell_from_json(*data_column_ptr, 
slice,
-                                                                       
_serde_options));
-        }
-    } else if (primitive_type == TYPE_STRUCT) {
-        if (value_type != simdjson::ondemand::json_type::object) {
-            return Status::DataQualityError(
-                    "Json value isn't object, but the column `{}` is struct.", 
column_name);
-        }
-        const auto* type_struct =
-                assert_cast<const 
DataTypeStruct*>(remove_nullable(type_desc).get());
-        auto* struct_column_ptr = assert_cast<ColumnStruct*>(data_column_ptr);
-        const auto sub_serdes = data_serde->get_nested_serdes();
-        std::map<std::string, size_t> sub_col_name_to_idx;
-        for (size_t sub_col_idx = 0; sub_col_idx < 
type_struct->get_elements().size();
-             ++sub_col_idx) {
-            
sub_col_name_to_idx.emplace(lower_key(type_struct->get_element_name(sub_col_idx)),
-                                        sub_col_idx);
-        }
-        std::vector<bool> has_value(type_struct->get_elements().size(), false);
-        simdjson::ondemand::object struct_value = value.get_object();
-        for (auto sub : struct_value) {
-            const auto sub_key = lower_key(sub.unescaped_key().value());
-            const auto it = sub_col_name_to_idx.find(sub_key);
-            if (it == sub_col_name_to_idx.end()) {
-                continue;
-            }
-            const auto sub_column_idx = it->second;
-            auto sub_column_ptr = 
struct_column_ptr->get_column(sub_column_idx).get_ptr();
-            if (has_value[sub_column_idx]) {
-                // Struct fields follow Hive-style duplicate handling: the 
last matching nested key
-                // wins. Remove the earlier nested value before appending the 
new one.
-                sub_column_ptr->pop_back(1);
-            }
-            has_value[sub_column_idx] = true;
-            auto sub_value = sub.value().value();
-            RETURN_IF_ERROR(_write_data_to_column<use_string_cache>(
-                    sub_value, type_struct->get_element(sub_column_idx), 
sub_column_ptr.get(),
-                    column_name + "." + sub_key, sub_serdes[sub_column_idx], 
valid));
-        }
-        for (size_t sub_col_idx = 0; sub_col_idx < 
type_struct->get_elements().size();
-             ++sub_col_idx) {
-            if (has_value[sub_col_idx]) {
-                continue;
-            }
-            auto sub_column_ptr = 
struct_column_ptr->get_column(sub_col_idx).get_ptr();
-            if (!is_column_nullable(*sub_column_ptr)) {
-                return Status::DataQualityError(
-                        "Json file structColumn miss field {} and this column 
isn't nullable.",
-                        column_name + "." + 
type_struct->get_element_name(sub_col_idx));
-            }
-            sub_column_ptr->insert_default();
-        }
-    } else if (primitive_type == TYPE_MAP) {
-        if (value_type != simdjson::ondemand::json_type::object) {
-            return Status::DataQualityError("Json value isn't object, but the 
column `{}` is map.",
-                                            column_name);
+            auto& nested = nullable.get_nested_column();
+            RETURN_IF_ERROR(serde->deserialize_one_cell_from_simdjson(
+                    nested, value, _serde_options, _simdjson_context, 
column_name));
+            nullable.get_null_map_data().push_back(0);
         }
-        const auto* map_type = assert_cast<const 
DataTypeMap*>(remove_nullable(type_desc).get());
-        auto* map_column_ptr = assert_cast<ColumnMap*>(data_column_ptr);
-        const auto sub_serdes = data_serde->get_nested_serdes();
-        size_t field_count = 0;
-        simdjson::ondemand::object object_value = value.get_object();
-        for (auto member_value : object_value) {
-            auto mutable_keys = 
map_column_ptr->get_keys_ptr()->assert_mutable();
-            auto* key_column = mutable_keys.get();
-            auto key_serde = sub_serdes[0];
-            if (is_column_nullable(*key_column)) {
-                auto* nullable_key = assert_cast<ColumnNullable*>(key_column);
-                nullable_key->get_null_map_data().push_back(0);
-                key_column = &nullable_key->get_nested_column();
-                if (map_type->get_key_type()->is_nullable()) {
-                    key_serde = key_serde->get_nested_serdes()[0];
-                }
-            }
-            std::string_view key_view = member_value.unescaped_key().value();
-            Slice key_slice(key_view.data(), key_view.size());
-            
RETURN_IF_ERROR(key_serde->deserialize_one_cell_from_json(*key_column, 
key_slice,
-                                                                      
_serde_options));
-            simdjson::ondemand::value field_value = 
member_value.value().value();
-            auto mutable_values = 
map_column_ptr->get_values_ptr()->assert_mutable();
-            RETURN_IF_ERROR(_write_data_to_column<use_string_cache>(
-                    field_value, map_type->get_value_type(), 
mutable_values.get(),
-                    column_name + ".value", sub_serdes[1], valid));
-            ++field_count;
-        }
-        auto& offsets = map_column_ptr->get_offsets();
-        offsets.emplace_back(offsets.back() + field_count);
-    } else if (primitive_type == TYPE_ARRAY) {
-        if (value_type != simdjson::ondemand::json_type::array) {
-            return Status::DataQualityError("Json value isn't array, but the 
column `{}` is array.",
-                                            column_name);
-        }
-        const auto* array_type =
-                assert_cast<const 
DataTypeArray*>(remove_nullable(type_desc).get());
-        auto* array_column_ptr = assert_cast<ColumnArray*>(data_column_ptr);
-        const auto sub_serdes = data_serde->get_nested_serdes();
-        size_t field_count = 0;
-        simdjson::ondemand::array array_value = value.get_array();
-        for (simdjson::ondemand::value sub_value : array_value) {
-            RETURN_IF_ERROR(_write_data_to_column<use_string_cache>(
-                    sub_value, array_type->get_nested_type(), 
&array_column_ptr->get_data(),
-                    column_name + ".element", sub_serdes[0], valid));
-            ++field_count;
-        }
-        auto& offsets = array_column_ptr->get_offsets();
-        offsets.emplace_back(offsets.back() + field_count);
     } else {
-        return Status::InternalError("Not support JSON value to complex 
column");
-    }
-
-    if (nullable_column && value_type != simdjson::ondemand::json_type::null) {
-        nullable_column->get_null_map_data().push_back(0);
+        RETURN_IF_ERROR(serde->deserialize_one_cell_from_simdjson(
+                *column_ptr, value, _serde_options, _simdjson_context, 
column_name));
     }
     *valid = true;
     return Status::OK();
diff --git a/be/src/format_v2/json/json_reader.h 
b/be/src/format_v2/json/json_reader.h
index b1c413da0e7..2ac77c2242c 100644
--- a/be/src/format_v2/json/json_reader.h
+++ b/be/src/format_v2/json/json_reader.h
@@ -28,6 +28,7 @@
 
 #include "core/custom_allocator.h"
 #include "core/data_type_serde/data_type_serde.h"
+#include "core/data_type_serde/simdjson_read_context.h"
 #include "exprs/json_functions.h"
 #include "format_v2/file_reader.h"
 #include "gen_cpp/PlanNodes_types.h"
@@ -227,7 +228,7 @@ private:
     std::string _padding_buffer;
     size_t _original_doc_size = 0;
     size_t _padded_size = 1024 * 1024 * 8 + simdjson::SIMDJSON_PADDING;
-    std::unordered_map<std::string_view, std::string_view> 
_cached_string_values;
+    SimdJsonReadContext _simdjson_context;
 };
 
 } // namespace doris::format::json
diff --git a/be/src/util/jsonb_parser_simd.h b/be/src/util/jsonb_parser_simd.h
index 48e2a0cd1af..6e94a1b67b9 100644
--- a/be/src/util/jsonb_parser_simd.h
+++ b/be/src/util/jsonb_parser_simd.h
@@ -61,6 +61,7 @@
 
 #include <limits>
 #include <string_view>
+#include <unordered_map>
 
 #include "common/status.h"
 #include "util/jsonb_document.h"
@@ -70,6 +71,38 @@
 namespace doris {
 using int128_t = __int128;
 struct JsonbParser {
+    using StringCache = std::unordered_map<std::string_view, std::string_view>;
+
+    // Rewinding an On-Demand object does not rewind its unescape buffer. 
Cache decoded values
+    // and keys when jsonpaths revisit a subtree, e.g. $.payload followed by 
$.payload.text.
+    static std::string_view decoded_string(simdjson::ondemand::value& value, 
StringCache* cache) {
+        if (cache == nullptr) {
+            return value.get_string().value();
+        }
+        const auto token = value.raw_json().value();
+        const auto it = cache->find(token);
+        if (it != cache->end()) {
+            return it->second;
+        }
+        const auto text = value.get_string().value();
+        cache->emplace(token, text);
+        return text;
+    }
+
+    static std::string_view decoded_key(simdjson::ondemand::field& field, 
StringCache* cache) {
+        if (cache == nullptr) {
+            return field.unescaped_key().value();
+        }
+        const auto token = field.key_raw_json_token();
+        const auto it = cache->find(token);
+        if (it != cache->end()) {
+            return it->second;
+        }
+        const auto key = field.unescaped_key().value();
+        cache->emplace(token, key);
+        return key;
+    }
+
 private:
     static bool is_json_whitespace(char c) {
         return c == ' ' || c == '\t' || c == '\n' || c == '\r';
@@ -149,6 +182,18 @@ private:
     }
 
 public:
+    // Consume a subtree without copying/re-indexing its text. No value 
escapes: encoding
+    // row.payload leaves the reader ready to advance to row.id.
+    static Status parse_value(simdjson::ondemand::value value, JsonbWriter& 
writer,
+                              StringCache* cache = nullptr) {
+        writer.reset();
+        try {
+            return parse(value, writer, cache);
+        } catch (simdjson::simdjson_error& e) {
+            return Status::InvalidArgument("simdjson parse exception: {}", 
e.what());
+        }
+    }
+
     // parse a UTF-8 JSON string with length
     // will reset writer before parse
     static Status parse(const char* pch, size_t len, JsonbWriter& writer) {
@@ -238,7 +283,8 @@ private:
 
     // parse json, recursively if necessary, by simdjson
     //  and serialize to binary format by writer
-    static Status parse(simdjson::ondemand::value value, JsonbWriter& writer) {
+    static Status parse(simdjson::ondemand::value value, JsonbWriter& writer,
+                        StringCache* cache = nullptr) {
         switch (value.type()) {
         case simdjson::ondemand::json_type::null: {
             return write_null(value, writer);
@@ -250,7 +296,7 @@ private:
             break;
         }
         case simdjson::ondemand::json_type::string: {
-            RETURN_IF_ERROR(write_string(value.get_string(), writer));
+            RETURN_IF_ERROR(write_string(decoded_string(value, cache), 
writer));
             break;
         }
         case simdjson::ondemand::json_type::number: {
@@ -261,12 +307,8 @@ private:
                 return Status::InvalidArgument("writeStartObject failed");
             }
 
-            for (auto kv : value.get_object()) {
-                std::string_view key;
-                simdjson::error_code e = kv.unescaped_key().get(key);
-                if (e != simdjson::SUCCESS) {
-                    return Status::InvalidArgument(fmt::format("simdjson get 
key failed: {}", e));
-                }
+            for (simdjson::ondemand::field field : value.get_object()) {
+                const auto key = decoded_key(field, cache);
 
                 // write key
                 if (key.size() > std::numeric_limits<uint8_t>::max()) {
@@ -278,7 +320,7 @@ private:
                 }
 
                 // parse object value
-                RETURN_IF_ERROR(parse(kv.value(), writer));
+                RETURN_IF_ERROR(parse(field.value(), writer, cache));
             }
 
             if (!writer.writeEndObject()) {
@@ -294,7 +336,7 @@ private:
 
             for (auto elem : value.get_array()) {
                 // parse array element
-                RETURN_IF_ERROR(parse(elem.value(), writer));
+                RETURN_IF_ERROR(parse(elem.value(), writer, cache));
             }
 
             if (!writer.writeEndArray()) {
diff --git a/be/test/core/data_type_serde/simdjson_serde_test.cpp 
b/be/test/core/data_type_serde/simdjson_serde_test.cpp
new file mode 100644
index 00000000000..2f03977aeeb
--- /dev/null
+++ b/be/test/core/data_type_serde/simdjson_serde_test.cpp
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "core/column/column_map.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_decimal.h"
+#include "core/data_type/data_type_jsonb.h"
+#include "core/data_type/data_type_map.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_string.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type_serde/simdjson_read_context.h"
+#include "core/value/jsonb_value.h"
+
+namespace doris {
+namespace {
+Status append_simdjson(const DataTypeSerDe& serde, IColumn& column, const 
std::string& json,
+                       SimdJsonReadContext& context) {
+    context.string_values.clear();
+    simdjson::ondemand::parser parser;
+    simdjson::padded_string text("{\"v\":" + json + ",\"tail\":42}");
+    try {
+        auto doc = parser.iterate(text);
+        auto object = doc.get_object().value();
+        auto value = object.find_field_unordered("v").value();
+        auto status = serde.deserialize_one_cell_from_simdjson(
+                column, value, DataTypeSerDe::get_default_format_options(), 
context, "v");
+        if (!status.ok()) {
+            return status;
+        }
+        EXPECT_EQ(object.find_field_unordered("tail").get_int64().value(), 42);
+        return Status::OK();
+    } catch (const simdjson::simdjson_error& e) {
+        return Status::DataQualityError("{}", e.what());
+    }
+}
+
+Status append_legacy(const DataTypeSerDe& serde, IColumn& column, const 
std::string& json) {
+    simdjson::ondemand::parser parser;
+    simdjson::padded_string text("{\"v\":" + json + "}");
+    auto doc = parser.iterate(text);
+    auto object = doc.get_object().value();
+    auto value = object.find_field_unordered("v").value();
+    std::string_view token;
+    if (value.type().value() == simdjson::ondemand::json_type::string) {
+        token = value.get_string();
+    } else if (value.type().value() == simdjson::ondemand::json_type::boolean) 
{
+        token = value.get_bool() ? "1" : "0";
+    } else {
+        token = simdjson::to_json_string(value);
+    }
+    Slice slice {token.data(), token.size()};
+    return serde.deserialize_one_cell_from_json(column, slice,
+                                                
DataTypeSerDe::get_default_format_options());
+}
+} // namespace
+
+TEST(SimdJsonSerDeTest, ScalarConversionsMatchReader) {
+    const DataTypes types {std::make_shared<DataTypeInt8>(),
+                           std::make_shared<DataTypeInt16>(),
+                           std::make_shared<DataTypeInt32>(),
+                           std::make_shared<DataTypeInt64>(),
+                           std::make_shared<DataTypeInt128>(),
+                           std::make_shared<DataTypeBool>(),
+                           std::make_shared<DataTypeFloat64>(),
+                           std::make_shared<DataTypeString>(),
+                           
std::make_shared<DataTypeDecimal<TYPE_DECIMAL128I>>(30, 5)};
+    const std::vector<std::string> values {"0",
+                                           "-1",
+                                           "127",
+                                           "128",
+                                           "-129",
+                                           "32768",
+                                           "2147483648",
+                                           "9223372036854775807",
+                                           "-9223372036854775808",
+                                           "9223372036854775808",
+                                           "18446744073709551616",
+                                           "1.5",
+                                           "1e3",
+                                           "true",
+                                           "false",
+                                           R"("123")",
+                                           R"("bad")",
+                                           R"("a\nb\u4e2d")"};
+    SimdJsonReadContext context;
+    for (const auto& type : types) {
+        const auto serde = type->get_serde();
+        for (const auto& json : values) {
+            SCOPED_TRACE(type->get_name() + ": " + json);
+            auto actual = type->create_column();
+            auto expected = type->create_column();
+            auto old_status = append_legacy(*serde, *expected, json);
+            auto status = append_simdjson(*serde, *actual, json, context);
+            ASSERT_EQ(status.ok(), old_status.ok()) << status << " / " << 
old_status;
+            ASSERT_EQ(actual->size(), expected->size());
+            if (status.ok()) {
+                EXPECT_EQ(actual->compare_at(0, 0, *expected, 1), 0);
+            }
+        }
+    }
+}
+
+TEST(SimdJsonSerDeTest, JsonbBytesAndLegacyScalarSemantics) {
+    DataTypeJsonb type;
+    auto serde = type.get_serde();
+    auto actual = type.create_column();
+    auto expected = type.create_column();
+    SimdJsonReadContext context;
+    const std::vector<std::string> values {
+            R"({"x":[1,"a\nb",true,null]})",
+            "[]",
+            "{}",
+            
R"([18446744073709551616,-170141183460469231731687303715884105728,1.25])",
+            R"("{\"x\":1}")",
+            "true",
+            "false",
+            "123",
+            "\"" + std::string(8192, 'x') + "\""};
+    for (const auto& json : values) {
+        const auto old_status = append_legacy(*serde, *expected, json);
+        const auto status = append_simdjson(*serde, *actual, json, context);
+        ASSERT_EQ(status.ok(), old_status.ok()) << json << ": " << status;
+    }
+    ASSERT_EQ(actual->size(), expected->size());
+    for (size_t i = 0; i < actual->size(); ++i) {
+        EXPECT_EQ(actual->get_data_at(i).to_string(), 
expected->get_data_at(i).to_string());
+    }
+}
+
+TEST(SimdJsonSerDeTest, NestedConversionsAndErrors) {
+    auto integer = make_nullable(std::make_shared<DataTypeInt8>());
+    auto structure =
+            std::make_shared<DataTypeStruct>(DataTypes {integer, integer}, 
Strings {"a", "b"});
+    auto array = std::make_shared<DataTypeArray>(make_nullable(structure));
+    auto map = 
std::make_shared<DataTypeMap>(make_nullable(std::make_shared<DataTypeString>()),
+                                             make_nullable(array));
+    SimdJsonReadContext context;
+    auto column = map->create_column();
+    auto serde = map->get_serde();
+    ASSERT_TRUE(append_simdjson(*serde, *column, 
R"({"ok":[{"a":1,"A":2,"ignored":5}]})", context)
+                        .ok());
+    for (const auto& json :
+         {R"({"first":[{"a":3}],"bad":[{"a":4,"b":128}]})", 
R"({"bad":[{"a":1},{"a":"oops"}]})"}) {
+        auto invalid_column = map->create_column();
+        EXPECT_FALSE(append_simdjson(*serde, *invalid_column, json, 
context).ok());
+    }
+    EXPECT_TRUE(append_simdjson(*serde, *column, R"({"next":[null,{"b":7}]})", 
context).ok());
+    EXPECT_EQ(column->size(), 2);
+}
+
+TEST(SimdJsonSerDeTest, RepeatedJsonpathStringAndSubtree) {
+    simdjson::ondemand::parser parser;
+    simdjson::padded_string 
text(std::string_view(R"({"payload":{"s":"a\nb","n":7},"tail":42})"));
+    auto doc = parser.iterate(text);
+    auto object = doc.get_object().value();
+    SimdJsonReadContext context;
+    context.use_string_cache = true;
+    DataTypeString string_type;
+    auto strings = string_type.create_column();
+    auto string_serde = string_type.get_serde();
+    DataTypeJsonb jsonb_type;
+    auto jsonbs = jsonb_type.create_column();
+    auto jsonb_serde = jsonb_type.get_serde();
+    for (int i = 0; i < 32; ++i) {
+        auto value = 
object.find_field_unordered("payload").find_field_unordered("s").value();
+        ASSERT_TRUE(string_serde
+                            ->deserialize_one_cell_from_simdjson(
+                                    *strings, value, 
DataTypeSerDe::get_default_format_options(),
+                                    context, "s")
+                            .ok());
+        auto payload = object.find_field_unordered("payload").value();
+        ASSERT_TRUE(jsonb_serde
+                            ->deserialize_one_cell_from_simdjson(
+                                    *jsonbs, payload, 
DataTypeSerDe::get_default_format_options(),
+                                    context, "payload")
+                            .ok());
+    }
+    EXPECT_EQ(strings->get_data_at(0).to_string(), "a\nb");
+    EXPECT_EQ(strings->get_data_at(0).to_string(), 
strings->get_data_at(1).to_string());
+    EXPECT_EQ(jsonbs->get_data_at(0).to_string(), 
jsonbs->get_data_at(1).to_string());
+    EXPECT_EQ(object.find_field_unordered("tail").get_int64().value(), 42);
+}
+} // namespace doris
diff --git a/be/test/format_v2/json/json_reader_test.cpp 
b/be/test/format_v2/json/json_reader_test.cpp
index bf82fbcf01d..3e52e1d2a37 100644
--- a/be/test/format_v2/json/json_reader_test.cpp
+++ b/be/test/format_v2/json/json_reader_test.cpp
@@ -33,15 +33,19 @@
 #include "common/object_pool.h"
 #include "core/assert_cast.h"
 #include "core/block/block.h"
+#include "core/column/column_array.h"
+#include "core/column/column_map.h"
 #include "core/column/column_nullable.h"
 #include "core/column/column_string.h"
 #include "core/column/column_vector.h"
 #include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_jsonb.h"
 #include "core/data_type/data_type_map.h"
 #include "core/data_type/data_type_nullable.h"
 #include "core/data_type/data_type_number.h"
 #include "core/data_type/data_type_string.h"
 #include "core/data_type/data_type_struct.h"
+#include "core/value/jsonb_value.h"
 #include "exec/scan/scanner.h"
 #include "exprs/vexpr.h"
 #include "exprs/vexpr_context.h"
@@ -971,6 +975,61 @@ TEST(JsonReaderTest, 
IgnoresMalformedJsonAsNullRowsWhenConfigured) {
     EXPECT_EQ(nullable_string_at(*result.block.get_by_position(1).column, 1), 
"judy");
 }
 
+TEST(JsonReaderTest, 
IgnoredMalformedNestedArrayDoesNotPopulateFollowingEmptyArray) {
+    for (const auto& paths : {std::string(), std::string(R"(["$.a"])")}) {
+        SCOPED_TRACE(paths);
+        ObjectPool pool;
+        auto inner = make_nullable(
+                
std::make_shared<DataTypeArray>(make_nullable(std::make_shared<DataTypeInt32>())));
+        auto type = make_nullable(std::make_shared<DataTypeArray>(inner));
+        std::vector<SlotDescriptor*> slots {make_test_slot(&pool, 0, 0, type, 
"a")};
+        auto result =
+                read_once("ignored_nested_array.jsonl",
+                          R"({"a":[[1],2]})"
+                          "\n"
+                          R"({"a":[]})"
+                          "\n",
+                          json_scan_params(true, false, paths, "", true), 
slots, {0}, false, true);
+        ASSERT_TRUE(result.status.ok()) << result.status;
+        ASSERT_EQ(result.rows, 2);
+        const auto& nullable =
+                assert_cast<const 
ColumnNullable&>(*result.block.get_by_position(0).column);
+        EXPECT_TRUE(nullable.is_null_at(0));
+        EXPECT_FALSE(nullable.is_null_at(1));
+        const auto& array = assert_cast<const 
ColumnArray&>(nullable.get_nested_column());
+        EXPECT_EQ(array.get_offsets()[0], 0);
+        EXPECT_EQ(array.get_offsets()[1], 0);
+    }
+}
+
+TEST(JsonReaderTest, IgnoredMalformedMapDoesNotPopulateFollowingEmptyMap) {
+    for (const auto& paths : {std::string(), std::string(R"(["$.m"])")}) {
+        SCOPED_TRACE(paths);
+        ObjectPool pool;
+        auto values = make_nullable(
+                
std::make_shared<DataTypeArray>(make_nullable(std::make_shared<DataTypeInt32>())));
+        auto type = make_nullable(std::make_shared<DataTypeMap>(
+                make_nullable(std::make_shared<DataTypeString>()), values));
+        std::vector<SlotDescriptor*> slots {make_test_slot(&pool, 0, 0, type, 
"m")};
+        auto result =
+                read_once("ignored_map.jsonl",
+                          R"({"m":{"good":[1],"bad":2}})"
+                          "\n"
+                          R"({"m":{}})"
+                          "\n",
+                          json_scan_params(true, false, paths, "", true), 
slots, {0}, false, true);
+        ASSERT_TRUE(result.status.ok()) << result.status;
+        ASSERT_EQ(result.rows, 2);
+        const auto& nullable =
+                assert_cast<const 
ColumnNullable&>(*result.block.get_by_position(0).column);
+        EXPECT_TRUE(nullable.is_null_at(0));
+        EXPECT_FALSE(nullable.is_null_at(1));
+        const auto& map = assert_cast<const 
ColumnMap&>(nullable.get_nested_column());
+        EXPECT_EQ(map.get_offsets()[0], 0);
+        EXPECT_EQ(map.get_offsets()[1], 0);
+    }
+}
+
 TEST(JsonReaderTest, SkipsEmptyJsonLine) {
     ObjectPool pool;
     auto slots = build_slots(&pool);
@@ -1193,4 +1252,36 @@ TEST(JsonReaderTest, 
EmptyFileProjectionMaterializesPartitionRows) {
     EXPECT_TRUE(reader.close().ok());
 }
 
+TEST(JsonReaderTest, SimdjsonJsonbSubtreesAndOverlappingPaths) {
+    ObjectPool pool;
+    auto jsonb = make_nullable(std::make_shared<DataTypeJsonb>());
+    auto string = make_nullable(std::make_shared<DataTypeString>());
+    std::vector<SlotDescriptor*> slots {make_test_slot(&pool, 0, 0, jsonb, 
"payload"),
+                                        make_test_slot(&pool, 1, 1, string, 
"first"),
+                                        make_test_slot(&pool, 2, 2, string, 
"second"),
+                                        make_test_slot(&pool, 3, 3, jsonb, 
"copy")};
+    auto result = read_once(
+            "simdjson_jsonb_paths.jsonl",
+            R"({"payload":{"text":"a\nb","large":18446744073709551616}})"
+            "\n"
+            R"({"payload":{"text":"next","array":[true,null,{}]}})"
+            "\n",
+            json_scan_params(true, false,
+                             
R"(["$.payload","$.payload.text","$.payload.text","$.payload"])"),
+            slots, {0, 1, 2, 3});
+    ASSERT_TRUE(result.status.ok()) << result.status;
+    ASSERT_EQ(result.rows, 2);
+    for (size_t i = 0; i < result.rows; ++i) {
+        EXPECT_EQ(nullable_string_at(*result.block.get_by_position(0).column, 
i),
+                  nullable_string_at(*result.block.get_by_position(3).column, 
i));
+        EXPECT_EQ(nullable_string_at(*result.block.get_by_position(1).column, 
i),
+                  nullable_string_at(*result.block.get_by_position(2).column, 
i));
+    }
+    EXPECT_EQ(nullable_string_at(*result.block.get_by_position(1).column, 0), 
"a\nb");
+    JsonBinaryValue expected;
+    
ASSERT_TRUE(expected.from_json_string(R"({"text":"a\nb","large":18446744073709551616})").ok());
+    EXPECT_EQ(nullable_string_at(*result.block.get_by_position(0).column, 0),
+              std::string(expected.value(), expected.size()));
+}
+
 } // namespace doris::format::json
diff --git a/regression-test/data/jsonb_p0/simdjson_column_deserialization.json 
b/regression-test/data/jsonb_p0/simdjson_column_deserialization.json
new file mode 100644
index 00000000000..b0c9bfaf7ce
--- /dev/null
+++ b/regression-test/data/jsonb_p0/simdjson_column_deserialization.json
@@ -0,0 +1,4 @@
+{"id": 1, "small": 127, "flag": true, "amount": 123.5678, "payload": {"n": 
18446744073709551616, "s": "a\nb"}, "items": [{"x": 1, "X": 2, "y": "a\nb"}, 
null, {"y": "missing"}], "attrs": {"k": [1, null, 3]}, "first_text": "a\nb", 
"second_text": "other"}
+{"id": 2, "small": -128, "flag": false, "amount": "-0.1250", "payload": [1, 
{"a": [true, null, "中"]}], "items": [], "attrs": {}, "first_text": 
"quote\"slash\\", "second_text": "two"}
+{"id": 3, "small": 128, "flag": true, "amount": 1, "payload": {"bad": true}, 
"items": [{"x": 7}], "attrs": {"bad": [1]}, "first_text": "filtered"}
+{"id": 4, "small": "12", "flag": "true", "amount": 0, "payload": 
"{\"embedded\":1}", "items": [{"x": 4}], "attrs": {"next": []}, "first_text": 
"after", "second_text": "last"}
diff --git 
a/regression-test/data/jsonb_p0/test_simdjson_column_deserialization.out 
b/regression-test/data/jsonb_p0/test_simdjson_column_deserialization.out
new file mode 100644
index 00000000000..5a0e07c895d
--- /dev/null
+++ b/regression-test/data/jsonb_p0/test_simdjson_column_deserialization.out
@@ -0,0 +1,11 @@
+-- This file is automatically generated. You should know what you did if you 
want to edit this
+-- !by_name --
+1      127     true    123.5678        {"n":18446744073709551616,"s":"a\\nb"}  
[{"x":2, "y":"anb"}, null, {"x":null, "y":"missing"}]   {"k":[1, null, 3]}      
a\nb    other
+2      -128    false   -0.1250 [1,{"a":[true,null,"中"]}]       []      {}      
quote"slash\\   two
+4      12      true    0.0000  {"embedded":1}  [{"x":4, "y":null}]     
{"next":[]}     after   last
+
+-- !by_path --
+1      127     true    123.5678        {"n":18446744073709551616,"s":"a\\nb"}  
[{"x":2, "y":"anb"}, null, {"x":null, "y":"missing"}]   {"k":[1, null, 3]}      
a\nb    a\nb
+2      -128    false   -0.1250 [1,{"a":[true,null,"中"]}]       []      {}      
quote"slash\\   quote"slash\\
+4      12      true    0.0000  {"embedded":1}  [{"x":4, "y":null}]     
{"next":[]}     after   after
+
diff --git 
a/regression-test/suites/jsonb_p0/test_simdjson_column_deserialization.groovy 
b/regression-test/suites/jsonb_p0/test_simdjson_column_deserialization.groovy
new file mode 100644
index 00000000000..904a430e867
--- /dev/null
+++ 
b/regression-test/suites/jsonb_p0/test_simdjson_column_deserialization.groovy
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_simdjson_column_deserialization", "p0") {
+    sql "DROP TABLE IF EXISTS test_simdjson_column_deserialization"
+    sql """CREATE TABLE test_simdjson_column_deserialization (
+        id INT, small TINYINT, flag BOOLEAN, amount DECIMAL(20,4), payload 
JSONB,
+        items ARRAY<STRUCT<x:INT,y:STRING>>, attrs MAP<STRING,ARRAY<INT>>,
+        first_text STRING, second_text STRING
+    ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
+    PROPERTIES("replication_num"="1")"""
+    streamLoad {
+        table "test_simdjson_column_deserialization"
+        set "format", "json"
+        set "read_json_by_line", "true"
+        set "strict_mode", "true"
+        set "max_filter_ratio", "0.5"
+        file "simdjson_column_deserialization.json"
+        time 10000
+        check { result, exception, startTime, endTime ->
+            if (exception != null) {
+                throw exception
+            }
+            def load = parseJson(result)
+            assertEquals("Success", load.Status)
+            assertEquals(4, load.NumberTotalRows)
+            assertEquals(3, load.NumberLoadedRows)
+            assertEquals(1, load.NumberFilteredRows)
+        }
+    }
+    sql "sync"
+    order_qt_by_name """SELECT id,small,flag,amount,CAST(payload AS 
STRING),items,attrs,
+                              first_text,second_text
+                       FROM test_simdjson_column_deserialization"""
+    sql "TRUNCATE TABLE test_simdjson_column_deserialization"
+    streamLoad {
+        table "test_simdjson_column_deserialization"
+        set "format", "json"
+        set "read_json_by_line", "true"
+        set "strict_mode", "true"
+        set "max_filter_ratio", "0.5"
+        set "jsonpaths", 
'["$.id","$.small","$.flag","$.amount","$.payload","$.items","$.attrs","$.first_text","$.first_text"]'
+        file "simdjson_column_deserialization.json"
+        time 10000
+        check { result, exception, startTime, endTime ->
+            if (exception != null) {
+                throw exception
+            }
+            def load = parseJson(result)
+            assertEquals("Success", load.Status)
+            assertEquals(4, load.NumberTotalRows)
+            assertEquals(3, load.NumberLoadedRows)
+            assertEquals(1, load.NumberFilteredRows)
+        }
+    }
+    sql "sync"
+    order_qt_by_path """SELECT id,small,flag,amount,CAST(payload AS 
STRING),items,attrs,
+                              first_text,second_text
+                       FROM test_simdjson_column_deserialization"""
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to