This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 4a3fce2a8fc [refactor](load) Deserialize simdjson values through type
SerDes (#70470)
4a3fce2a8fc is described below
commit 4a3fce2a8fc1000b90f14793241aeeb574402e7e
Author: Mryange <[email protected]>
AuthorDate: Tue Nov 17 10:45:37 2026 +0800
[refactor](load) Deserialize simdjson values through type SerDes (#70470)
### Background
The JSON reader currently converts simdjson values to columns through a
type-dispatch chain. JSONB objects and arrays are serialized to JSON
text and parsed again before insertion.
### What problem does this PR solve?
Issue Number: None
Related PR: None
PR Type: Refactor
Problem Summary:
Move simdjson-to-column conversion into
`DataTypeSerDe::deserialize_one_cell_from_simdjson`. Complex types
recursively use their child SerDes, and JSONB objects and arrays are
encoded directly from the parsed subtree. Scalar types retain the
existing text conversion, including boolean coercion.
### Details
- **Change map:** 511 changed production lines across the SerDe
interface/implementations, reader integration, read context, and JSONB
parser helpers; 381 added test/fixture/result lines; no generated source
or documentation changes.
- **Moves vs. logic changes:** Move Array/Map/Struct/Nullable dispatch
into SerDes. Add a default text fallback and a direct JSONB subtree
encoder. Keep the string cache and writer in reader-owned context for
overlapping JSONPaths.
- **Behavior change:** No intended user-visible semantic change; legacy
scalar conversions and error propagation are retained.
- **Compatibility:** No storage format, protocol, or configuration
changes. Independent of parser-reuse changes.
- **Performance impact:** Removes the JSON text/reparse round trip for
JSONB objects and arrays. No benchmark claim; the common SerDe header
now includes simdjson.
- **Rollback:** Revert this PR.
### Testing
Latest validation at `d702ec80284` on the development machine:
- Added two Reader UTs for ignored malformed nested Array/Map followed
by an empty collection, each exercised with field-name and JSONPath
projection. Both tests failed before the fix: the next empty collection
offset was 1 for Array and 2 for Map instead of 0.
- Restored legacy `previous offset + current entry count` for Array and
Map. No new rollback semantics were added.
- ASAN `run-be-ut.sh --run -j48
--filter="SimdJsonSerDeTest.*:JsonReader*Test.*:JsonbParserTest.*:DataType*SerDeTest.*"`:
all 179 tests from 16 suites passed after the fix.
- Repository clang-format 16, build hygiene, `git diff --check`, and
clang-tidy on all three changed files passed.
Earlier validation:
- ASAN BE/FE builds passed at `c44d740953a`.
- Regression suites `test_simdjson_column_deserialization` and
`test_load_json_with_jsonpath` passed on the earlier build. The new
regression output was generated with the regression runner.
Final code is compiled and tested by the ASAN UT build. Full BE/FE
builds and regression tests were not rerun for these final adjustments.
No greater-than-1-GiB reproduction was run; the direct JSONB subtree
path preserves the legacy size check and InternalError.
### Release note
None
### Check List (For Author)
- Behavior changed: No intended user-visible change
- Does this need documentation: No
- Pre-submission status: Partially verified: final HEAD passed ASAN UT
compilation and 179 tests; full BE/FE builds and regression tests were
not rerun.
### AI Disclosure
- Tool: Codex
- Generated: Implementation, tests, and PR description.
- Human review: The author reviewed the main implementation and
requested interface and compatibility adjustments; BE unit tests were
not manually reviewed.
---
.../core/data_type_serde/data_type_array_serde.cpp | 22 +++
.../core/data_type_serde/data_type_array_serde.h | 5 +
.../core/data_type_serde/data_type_jsonb_serde.cpp | 23 +++
.../core/data_type_serde/data_type_jsonb_serde.h | 5 +
.../core/data_type_serde/data_type_map_serde.cpp | 41 +++++
be/src/core/data_type_serde/data_type_map_serde.h | 5 +
.../data_type_serde/data_type_nullable_serde.cpp | 16 ++
.../data_type_serde/data_type_nullable_serde.h | 5 +
be/src/core/data_type_serde/data_type_serde.cpp | 27 +++
be/src/core/data_type_serde/data_type_serde.h | 11 ++
.../data_type_serde/data_type_struct_serde.cpp | 54 ++++++
.../core/data_type_serde/data_type_struct_serde.h | 6 +-
.../core/data_type_serde/simdjson_read_context.h | 43 +++++
be/src/format_v2/json/json_reader.cpp | 183 ++-----------------
be/src/format_v2/json/json_reader.h | 3 +-
be/src/util/jsonb_parser_simd.h | 62 ++++++-
.../core/data_type_serde/simdjson_serde_test.cpp | 201 +++++++++++++++++++++
be/test/format_v2/json/json_reader_test.cpp | 91 ++++++++++
.../jsonb_p0/simdjson_column_deserialization.json | 4 +
.../test_simdjson_column_deserialization.out | 11 ++
.../test_simdjson_column_deserialization.groovy | 74 ++++++++
21 files changed, 712 insertions(+), 180 deletions(-)
diff --git a/be/src/core/data_type_serde/data_type_array_serde.cpp
b/be/src/core/data_type_serde/data_type_array_serde.cpp
index 8c1c2f644f6..e51230fc417 100644
--- a/be/src/core/data_type_serde/data_type_array_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_array_serde.cpp
@@ -36,6 +36,7 @@
#include "core/data_type_serde/bson_value_view.h"
#include "core/data_type_serde/complex_type_deserialize_util.h"
#include "core/data_type_serde/orc_serde_utils.h"
+#include "core/data_type_serde/simdjson_read_context.h"
#include "core/string_ref.h"
#include "exprs/function/function_helpers.h"
#include "storage/field_type.h"
@@ -847,4 +848,25 @@ Status DataTypeArraySerDe::decode_bson_value(IColumn&
column, const BsonValueVie
return Status::OK();
}
+Status DataTypeArraySerDe::deserialize_one_cell_from_simdjson(
+ IColumn& column, simdjson::ondemand::value& input, const
FormatOptions& options,
+ SimdJsonReadContext& context, const std::string& column_name) const {
+ if (input.type().value() != simdjson::ondemand::json_type::array) {
+ return Status::DataQualityError("Json value isn't array, but the
column `{}` is array.",
+ column_name);
+ }
+ auto& target = assert_cast<ColumnArray&>(column);
+ auto& data = target.get_data();
+
+ size_t field_count = 0;
+ for (simdjson::ondemand::value element : input.get_array()) {
+ RETURN_IF_ERROR(nested_serde->deserialize_one_cell_from_simdjson(
+ data, element, options, context, column_name + ".element"));
+ ++field_count;
+ }
+ auto& offsets = target.get_offsets();
+ offsets.push_back(offsets.back() + field_count);
+ return Status::OK();
+}
+
} // namespace doris
diff --git a/be/src/core/data_type_serde/data_type_array_serde.h
b/be/src/core/data_type_serde/data_type_array_serde.h
index 3d55eccc856..b491ef879ca 100644
--- a/be/src/core/data_type_serde/data_type_array_serde.h
+++ b/be/src/core/data_type_serde/data_type_array_serde.h
@@ -63,6 +63,11 @@ public:
Status serialize_column_to_json(const IColumn& column, int64_t start_idx,
int64_t end_idx,
BufferWritable& bw, FormatOptions&
options) const override;
+ Status deserialize_one_cell_from_simdjson(IColumn& column,
simdjson::ondemand::value& value,
+ const FormatOptions& options,
+ SimdJsonReadContext& context,
+ const std::string& column_name)
const override;
+
Status deserialize_one_cell_from_json(IColumn& column, Slice& slice,
const FormatOptions& options) const
override;
diff --git a/be/src/core/data_type_serde/data_type_jsonb_serde.cpp
b/be/src/core/data_type_serde/data_type_jsonb_serde.cpp
index b7e79184477..889dde9f82e 100644
--- a/be/src/core/data_type_serde/data_type_jsonb_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_jsonb_serde.cpp
@@ -27,9 +27,11 @@
#include "common/status.h"
#include "core/data_type_serde/arrow_validation.h"
#include "core/data_type_serde/bson_value_view.h"
+#include "core/data_type_serde/simdjson_read_context.h"
#include "core/value/jsonb_value.h"
#include "exprs/json_functions.h"
#include "util/jsonb_parser_simd.h"
+
namespace doris {
Status DataTypeJsonbSerDe::write_column_to_mysql_binary(const IColumn& column,
@@ -390,4 +392,25 @@ Status DataTypeJsonbSerDe::decode_bson_value(IColumn&
column, const BsonValueVie
: Status::InvalidArgument("BSON Extended JSON exceeds
JSONB limits");
}
+Status DataTypeJsonbSerDe::deserialize_one_cell_from_simdjson(
+ IColumn& column, simdjson::ondemand::value& input, const
FormatOptions& options,
+ SimdJsonReadContext& context, const std::string& column_name) const {
+ const auto type = input.type().value();
+ if (type != simdjson::ondemand::json_type::object &&
+ type != simdjson::ondemand::json_type::array) {
+ return DataTypeSerDe::deserialize_one_cell_from_simdjson(column,
input, options, context,
+ column_name);
+ }
+ auto& writer = context.jsonb_writer;
+ RETURN_IF_ERROR(JsonbParser::parse_value(input, writer, context.cache()));
+ const auto size = writer.getOutput()->getSize();
+ if (size > JsonBinaryValue::MAX_LENGTH) {
+ return Status::InternalError(
+ "Jsonb value length {} exceeds maximum allowed length of {}
bytes", size,
+ JsonBinaryValue::MAX_LENGTH);
+ }
+ column.insert_data(writer.getOutput()->getBuffer(), size);
+ return Status::OK();
+}
+
} // namespace doris
diff --git a/be/src/core/data_type_serde/data_type_jsonb_serde.h
b/be/src/core/data_type_serde/data_type_jsonb_serde.h
index 8c4d3c55de2..e8a377d261f 100644
--- a/be/src/core/data_type_serde/data_type_jsonb_serde.h
+++ b/be/src/core/data_type_serde/data_type_jsonb_serde.h
@@ -63,6 +63,11 @@ public:
Status serialize_column_to_json(const IColumn& column, int64_t start_idx,
int64_t end_idx,
BufferWritable& bw, FormatOptions&
options) const override;
+ Status deserialize_one_cell_from_simdjson(IColumn& column,
simdjson::ondemand::value& value,
+ const FormatOptions& options,
+ SimdJsonReadContext& context,
+ const std::string& column_name)
const override;
+
Status deserialize_one_cell_from_json(IColumn& column, Slice& slice,
const FormatOptions& options) const
override;
diff --git a/be/src/core/data_type_serde/data_type_map_serde.cpp
b/be/src/core/data_type_serde/data_type_map_serde.cpp
index 53cf76936cf..545b1bb9e68 100644
--- a/be/src/core/data_type_serde/data_type_map_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_map_serde.cpp
@@ -33,7 +33,9 @@
#include "core/data_type_serde/arrow_validation.h"
#include "core/data_type_serde/complex_type_deserialize_util.h"
#include "core/data_type_serde/orc_serde_utils.h"
+#include "core/data_type_serde/simdjson_read_context.h"
#include "core/string_ref.h"
+#include "util/defer_op.h"
#include "util/jsonb_document.h"
#include "util/jsonb_writer.h"
#include "util/simd/bits.h"
@@ -912,4 +914,43 @@ Status DataTypeMapSerDe::read_column_from_orc(IColumn&
column,
return decode_map_orc_values(key_serde, value_serde, column, view);
}
+Status DataTypeMapSerDe::deserialize_one_cell_from_simdjson(IColumn& column,
+
simdjson::ondemand::value& input,
+ const
FormatOptions& options,
+
SimdJsonReadContext& context,
+ const std::string&
column_name) const {
+ if (input.type().value() != simdjson::ondemand::json_type::object) {
+ return Status::DataQualityError("Json value isn't object, but the
column `{}` is map.",
+ column_name);
+ }
+ auto& target = assert_cast<ColumnMap&>(column);
+ auto keys_owner = target.get_keys_ptr()->assert_mutable();
+ auto values_owner = target.get_values_ptr()->assert_mutable();
+ auto& keys = *keys_owner;
+ auto& values = *values_owner;
+
+ size_t field_count = 0;
+ for (simdjson::ondemand::field field : input.get_object()) {
+ const auto key = context.decoded_key(field);
+ Slice slice {key.data(), key.size()};
+ // Object keys are decoded strings, not JSON tokens; preserve the
reader's text conversion.
+ if (keys.is_nullable()) {
+ auto& nullable = assert_cast<ColumnNullable&>(keys);
+ auto& nested_key = nullable.get_nested_column();
+ nullable.get_null_map_data().push_back(0);
+
RETURN_IF_ERROR(key_serde->get_nested_serdes()[0]->deserialize_one_cell_from_json(
+ nested_key, slice, options));
+ } else {
+ RETURN_IF_ERROR(key_serde->deserialize_one_cell_from_json(keys,
slice, options));
+ }
+ auto value = field.value();
+ RETURN_IF_ERROR(value_serde->deserialize_one_cell_from_simdjson(
+ values, value, options, context, column_name + ".value"));
+ ++field_count;
+ }
+ auto& offsets = target.get_offsets();
+ offsets.push_back(offsets.back() + field_count);
+ return Status::OK();
+}
+
} // namespace doris
diff --git a/be/src/core/data_type_serde/data_type_map_serde.h
b/be/src/core/data_type_serde/data_type_map_serde.h
index bd6560c053c..1d04928b017 100644
--- a/be/src/core/data_type_serde/data_type_map_serde.h
+++ b/be/src/core/data_type_serde/data_type_map_serde.h
@@ -51,6 +51,11 @@ public:
FormatOptions& options) const override;
Status serialize_column_to_json(const IColumn& column, int64_t start_idx,
int64_t end_idx,
BufferWritable& bw, FormatOptions&
options) const override;
+ Status deserialize_one_cell_from_simdjson(IColumn& column,
simdjson::ondemand::value& value,
+ const FormatOptions& options,
+ SimdJsonReadContext& context,
+ const std::string& column_name)
const override;
+
Status deserialize_one_cell_from_json(IColumn& column, Slice& slice,
const FormatOptions& options) const
override;
diff --git a/be/src/core/data_type_serde/data_type_nullable_serde.cpp
b/be/src/core/data_type_serde/data_type_nullable_serde.cpp
index 90fe0af9fb3..941a31ae25f 100644
--- a/be/src/core/data_type_serde/data_type_nullable_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_nullable_serde.cpp
@@ -38,6 +38,7 @@
#include "core/data_type_serde/data_type_string_serde.h"
#include "core/data_type_serde/decoded_column_view.h"
#include "core/data_type_serde/orc_serde_utils.h"
+#include "core/data_type_serde/simdjson_read_context.h"
#include "exprs/function/cast/cast_base.h"
#include "format/transformer/vcsv_transformer.h"
#include "util/defer_op.h"
@@ -733,4 +734,19 @@ Status DataTypeNullableSerDe::decode_bson_value(IColumn&
column, const BsonValue
return Status::OK();
}
+Status DataTypeNullableSerDe::deserialize_one_cell_from_simdjson(
+ IColumn& column, simdjson::ondemand::value& input, const
FormatOptions& options,
+ SimdJsonReadContext& context, const std::string& column_name) const {
+ auto& target = assert_cast<ColumnNullable&>(column);
+ if (input.type().value() == simdjson::ondemand::json_type::null) {
+ target.insert_default();
+ return Status::OK();
+ }
+
+ RETURN_IF_ERROR(nested_serde->deserialize_one_cell_from_simdjson(
+ target.get_nested_column(), input, options, context, column_name));
+ target.get_null_map_data().push_back(0);
+ return Status::OK();
+}
+
} // namespace doris
diff --git a/be/src/core/data_type_serde/data_type_nullable_serde.h
b/be/src/core/data_type_serde/data_type_nullable_serde.h
index 119e68f4fd9..5a51f83de83 100644
--- a/be/src/core/data_type_serde/data_type_nullable_serde.h
+++ b/be/src/core/data_type_serde/data_type_nullable_serde.h
@@ -54,6 +54,11 @@ public:
FormatOptions& options) const override;
Status serialize_column_to_json(const IColumn& column, int64_t start_idx,
int64_t end_idx,
BufferWritable& bw, FormatOptions&
options) const override;
+ Status deserialize_one_cell_from_simdjson(IColumn& column,
simdjson::ondemand::value& value,
+ const FormatOptions& options,
+ SimdJsonReadContext& context,
+ const std::string& column_name)
const override;
+
Status deserialize_one_cell_from_json(IColumn& column, Slice& slice,
const FormatOptions& options) const
override;
Status deserialize_one_cell_from_csv(IColumn& column, Slice& slice,
diff --git a/be/src/core/data_type_serde/data_type_serde.cpp
b/be/src/core/data_type_serde/data_type_serde.cpp
index 40f0c857864..42e7dee9910 100644
--- a/be/src/core/data_type_serde/data_type_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_serde.cpp
@@ -34,11 +34,13 @@
#include "core/data_type_serde/data_type_number_serde.h"
#include "core/data_type_serde/data_type_string_serde.h"
#include "core/data_type_serde/parquet_decode_source.h"
+#include "core/data_type_serde/simdjson_read_context.h"
#include "core/field.h"
#include "exprs/function/cast/cast_base.h"
#include "runtime/descriptors.h"
#include "util/jsonb_document.h"
#include "util/jsonb_writer.h"
+
namespace doris {
DataTypeSerDe::~DataTypeSerDe() = default;
@@ -423,4 +425,29 @@ Status DataTypeSerDe::decode_bson_value(IColumn&, const
BsonValueView&,
return Status::NotSupported("BSON decoding is not supported for this Doris
type");
}
+Status DataTypeSerDe::deserialize_one_cell_from_simdjson(IColumn& column,
+
simdjson::ondemand::value& value,
+ const FormatOptions&
options,
+ SimdJsonReadContext&
context,
+ const std::string&
column_name) const {
+ std::string_view text;
+ switch (value.type().value()) {
+ case simdjson::ondemand::json_type::null:
+ return Status::DataQualityError("Json value is null, but the column
`{}` is not nullable.",
+ column_name);
+ case simdjson::ondemand::json_type::string:
+ text = JsonbParser::decoded_string(value, context.cache());
+ break;
+ case simdjson::ondemand::json_type::boolean:
+ // Preserve legacy true -> "1" coercion for text-based
String/Decimal/JSONB targets.
+ text = value.get_bool() ? "1" : "0";
+ break;
+ default:
+ text = simdjson::to_json_string(value);
+ break;
+ }
+ Slice slice {text.data(), text.size()};
+ return deserialize_one_cell_from_json(column, slice, options);
+}
+
} // namespace doris
diff --git a/be/src/core/data_type_serde/data_type_serde.h
b/be/src/core/data_type_serde/data_type_serde.h
index 0693d68b731..91372205b8a 100644
--- a/be/src/core/data_type_serde/data_type_serde.h
+++ b/be/src/core/data_type_serde/data_type_serde.h
@@ -17,6 +17,8 @@
#pragma once
+#include <simdjson.h>
+
#include <cstddef>
#include <cstdint>
#include <memory>
@@ -93,6 +95,7 @@ struct CastParameters;
struct BsonValueView;
struct BsonDecodeContext;
+struct SimdJsonReadContext;
class DataTypeSerDe;
using DataTypeSerDeSPtr = std::shared_ptr<DataTypeSerDe>;
using DataTypeSerDeSPtrs = std::vector<DataTypeSerDeSPtr>;
@@ -392,6 +395,14 @@ public:
int64_t end_idx, BufferWritable&
bw,
FormatOptions& options) const = 0;
+ // Consumes one borrowed value and appends one cell. Errors propagate to
the caller;
+ // the reader retains responsibility for handling invalid rows.
+ virtual Status deserialize_one_cell_from_simdjson(IColumn& column,
+
simdjson::ondemand::value& value,
+ const FormatOptions&
options,
+ SimdJsonReadContext&
context,
+ const std::string&
column_name) const;
+
virtual Status deserialize_one_cell_from_json(IColumn& column, Slice&
slice,
const FormatOptions&
options) const = 0;
diff --git a/be/src/core/data_type_serde/data_type_struct_serde.cpp
b/be/src/core/data_type_serde/data_type_struct_serde.cpp
index 28d5b351160..abf4eb6273e 100644
--- a/be/src/core/data_type_serde/data_type_struct_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_struct_serde.cpp
@@ -20,6 +20,9 @@
#include <gen_cpp/types.pb.h>
#include <algorithm>
+#include <cctype>
+#include <ranges>
+#include <unordered_map>
#include "arrow/array/builder_nested.h"
#include "common/cast_set.h"
@@ -35,6 +38,7 @@
#include "core/data_type_serde/complex_type_deserialize_util.h"
#include "core/data_type_serde/data_type_serde.h"
#include "core/data_type_serde/orc_serde_utils.h"
+#include "core/data_type_serde/simdjson_read_context.h"
#include "core/string_ref.h"
#include "util/defer_op.h"
#include "util/jsonb_document.h"
@@ -896,4 +900,54 @@ Status DataTypeStructSerDe::decode_bson_value(IColumn&
column, const BsonValueVi
return Status::OK();
}
+Status DataTypeStructSerDe::deserialize_one_cell_from_simdjson(
+ IColumn& column, simdjson::ondemand::value& input, const
FormatOptions& options,
+ SimdJsonReadContext& context, const std::string& column_name) const {
+ if (input.type().value() != simdjson::ondemand::json_type::object) {
+ return Status::DataQualityError("Json value isn't object, but the
column `{}` is struct.",
+ column_name);
+ }
+ auto& target = assert_cast<ColumnStruct&>(column);
+
+ std::unordered_map<std::string, size_t> field_positions;
+ for (size_t i = 0; i < elem_names.size(); ++i) {
+ auto name = elem_names[i];
+ std::ranges::transform(name, name.begin(), [](unsigned char c) {
return std::tolower(c); });
+ field_positions.emplace(std::move(name), i);
+ }
+ std::vector<bool> seen(elem_names.size(), false);
+ for (simdjson::ondemand::field member : input.get_object()) {
+ std::string name(context.decoded_key(member));
+ std::ranges::transform(name, name.begin(), [](unsigned char c) {
return std::tolower(c); });
+ const auto it = field_positions.find(name);
+ if (it == field_positions.end()) {
+ continue;
+ }
+ const auto index = it->second;
+ auto& child = target.get_column(index);
+ // Preserve nested STRUCT's last-wins behavior, e.g. {"a":1,"A":2}.
+ if (seen[index]) {
+ child.pop_back(1);
+ }
+ auto value = member.value();
+
RETURN_IF_ERROR(elem_serdes_ptrs[index]->deserialize_one_cell_from_simdjson(
+ child, value, options, context, column_name + "." + name));
+ seen[index] = true;
+ }
+ for (size_t i = 0; i < elem_names.size(); ++i) {
+ if (seen[i]) {
+ continue;
+ }
+ auto& child = target.get_column(i);
+ if (!child.is_nullable()) {
+ return Status::DataQualityError(
+ "Json file structColumn miss field {} and this column
isn't nullable.",
+ column_name + "." + elem_names[i]);
+ }
+ child.insert_default();
+ }
+
+ return Status::OK();
+}
+
} // namespace doris
diff --git a/be/src/core/data_type_serde/data_type_struct_serde.h
b/be/src/core/data_type_serde/data_type_struct_serde.h
index 90e3cc3e7a0..8d4634556d9 100644
--- a/be/src/core/data_type_serde/data_type_struct_serde.h
+++ b/be/src/core/data_type_serde/data_type_struct_serde.h
@@ -16,7 +16,6 @@
// under the License.
#pragma once
-
#include <glog/logging.h>
#include <cstdint>
@@ -56,6 +55,11 @@ public:
Status serialize_column_to_json(const IColumn& column, int64_t start_idx,
int64_t end_idx,
BufferWritable& bw, FormatOptions&
options) const override;
+ Status deserialize_one_cell_from_simdjson(IColumn& column,
simdjson::ondemand::value& value,
+ const FormatOptions& options,
+ SimdJsonReadContext& context,
+ const std::string& column_name)
const override;
+
Status deserialize_one_cell_from_json(IColumn& column, Slice& slice,
const FormatOptions& options) const
override;
diff --git a/be/src/core/data_type_serde/simdjson_read_context.h
b/be/src/core/data_type_serde/simdjson_read_context.h
new file mode 100644
index 00000000000..a16b4844e95
--- /dev/null
+++ b/be/src/core/data_type_serde/simdjson_read_context.h
@@ -0,0 +1,43 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <string>
+#include <string_view>
+#include <unordered_map>
+
+#include "core/data_type_serde/data_type_serde.h"
+#include "util/jsonb_parser_simd.h"
+#include "util/jsonb_writer.h"
+
+namespace doris {
+// Reader-owned state, never stored in shared SerDes. Clear the cache before
the next document:
+// its keys and values borrow the input and parser buffers.
+struct SimdJsonReadContext {
+ bool use_string_cache = false;
+ std::unordered_map<std::string_view, std::string_view> string_values;
+ JsonbWriter jsonb_writer;
+
+ JsonbParser::StringCache* cache() { return use_string_cache ?
&string_values : nullptr; }
+ std::string_view decoded_key(simdjson::ondemand::field& field) {
+ return JsonbParser::decoded_key(field, cache());
+ }
+};
+} // namespace doris
diff --git a/be/src/format_v2/json/json_reader.cpp
b/be/src/format_v2/json/json_reader.cpp
index db8a8e93c8a..3a3e2fe3615 100644
--- a/be/src/format_v2/json/json_reader.cpp
+++ b/be/src/format_v2/json/json_reader.cpp
@@ -23,18 +23,14 @@
#include <algorithm>
#include <cstring>
#include <limits>
-#include <map>
#include <string_view>
#include <utility>
#include "common/cast_set.h"
#include "core/assert_cast.h"
#include "core/block/block.h"
-#include "core/column/column_array.h"
-#include "core/column/column_map.h"
#include "core/column/column_nullable.h"
#include "core/column/column_string.h"
-#include "core/column/column_struct.h"
#include "core/data_type/data_type_array.h"
#include "core/data_type/data_type_map.h"
#include "core/data_type/data_type_nullable.h"
@@ -151,12 +147,6 @@ ColumnDefinition synthetic_file_child(const std::string&
name, DataTypePtr type,
return child;
}
-std::string lower_key(std::string_view key) {
- std::string lowered(key.data(), key.size());
- std::transform(lowered.begin(), lowered.end(), lowered.begin(), ::tolower);
- return lowered;
-}
-
} // namespace
JsonReader::JsonReader(std::shared_ptr<io::FileSystemProperties>&
system_properties,
@@ -469,7 +459,7 @@ Status JsonReader::close() {
_requested_columns.clear();
_slot_name_to_index.clear();
_previous_positions.clear();
- _cached_string_values.clear();
+ _simdjson_context.string_values.clear();
return Status::OK();
}
@@ -994,7 +984,8 @@ Status
JsonReader::_write_columns_by_jsonpath(simdjson::ondemand::object* object
DORIS_CHECK(valid != nullptr);
bool has_valid_value = false;
const size_t cur_row_count = _logical_rows;
- _cached_string_values.clear();
+ _simdjson_context.string_values.clear();
+ _simdjson_context.use_string_cache = true;
// Even without projected fields, jsonpaths still decides whether the
object contributes a row.
if (_requested_columns.empty()) {
@@ -1066,165 +1057,21 @@ Status
JsonReader::_write_data_to_column(simdjson::ondemand::value& value,
const DataTypePtr& type_desc,
IColumn* column_ptr,
const std::string& column_name,
const DataTypeSerDeSPtr& serde, bool*
valid) {
- ColumnNullable* nullable_column = nullptr;
- IColumn* data_column_ptr = column_ptr;
- DataTypeSerDeSPtr data_serde = serde;
- const auto value_type = value.type().value();
-
- if (is_column_nullable(*column_ptr)) {
- nullable_column = assert_cast<ColumnNullable*>(column_ptr);
- data_column_ptr = &nullable_column->get_nested_column();
- if (type_desc->is_nullable()) {
- data_serde = serde->get_nested_serdes()[0];
- }
- if (value_type == simdjson::ondemand::json_type::null) {
- nullable_column->insert_default();
- *valid = true;
- return Status::OK();
- }
- } else if (value_type == simdjson::ondemand::json_type::null) {
- return Status::DataQualityError("Json value is null, but the column
`{}` is not nullable.",
- column_name);
- }
-
- const auto primitive_type = type_desc->get_primitive_type();
- if (!is_complex_type(primitive_type)) {
- if (value_type == simdjson::ondemand::json_type::string) {
- std::string_view value_string;
- if constexpr (use_string_cache) {
- const auto cache_key = value.raw_json().value();
- if (_cached_string_values.contains(cache_key)) {
- value_string = _cached_string_values[cache_key];
- } else {
- value_string = value.get_string();
- _cached_string_values.emplace(cache_key, value_string);
- }
- } else {
- value_string = value.get_string();
- }
- Slice slice {value_string.data(), value_string.size()};
-
RETURN_IF_ERROR(data_serde->deserialize_one_cell_from_json(*data_column_ptr,
slice,
-
_serde_options));
- } else if (value_type == simdjson::ondemand::json_type::boolean) {
- const char* str_value = value.get_bool() ? "1" : "0";
- Slice slice {str_value, 1};
-
RETURN_IF_ERROR(data_serde->deserialize_one_cell_from_json(*data_column_ptr,
slice,
-
_serde_options));
+ _simdjson_context.use_string_cache = use_string_cache;
+ // File slots can be logically non-nullable while their materialized
columns are nullable.
+ if (column_ptr->is_nullable() && !type_desc->is_nullable()) {
+ auto& nullable = assert_cast<ColumnNullable&>(*column_ptr);
+ if (value.type().value() == simdjson::ondemand::json_type::null) {
+ nullable.insert_default();
} else {
- std::string_view json_str = simdjson::to_json_string(value);
- Slice slice {json_str.data(), json_str.size()};
-
RETURN_IF_ERROR(data_serde->deserialize_one_cell_from_json(*data_column_ptr,
slice,
-
_serde_options));
- }
- } else if (primitive_type == TYPE_STRUCT) {
- if (value_type != simdjson::ondemand::json_type::object) {
- return Status::DataQualityError(
- "Json value isn't object, but the column `{}` is struct.",
column_name);
- }
- const auto* type_struct =
- assert_cast<const
DataTypeStruct*>(remove_nullable(type_desc).get());
- auto* struct_column_ptr = assert_cast<ColumnStruct*>(data_column_ptr);
- const auto sub_serdes = data_serde->get_nested_serdes();
- std::map<std::string, size_t> sub_col_name_to_idx;
- for (size_t sub_col_idx = 0; sub_col_idx <
type_struct->get_elements().size();
- ++sub_col_idx) {
-
sub_col_name_to_idx.emplace(lower_key(type_struct->get_element_name(sub_col_idx)),
- sub_col_idx);
- }
- std::vector<bool> has_value(type_struct->get_elements().size(), false);
- simdjson::ondemand::object struct_value = value.get_object();
- for (auto sub : struct_value) {
- const auto sub_key = lower_key(sub.unescaped_key().value());
- const auto it = sub_col_name_to_idx.find(sub_key);
- if (it == sub_col_name_to_idx.end()) {
- continue;
- }
- const auto sub_column_idx = it->second;
- auto sub_column_ptr =
struct_column_ptr->get_column(sub_column_idx).get_ptr();
- if (has_value[sub_column_idx]) {
- // Struct fields follow Hive-style duplicate handling: the
last matching nested key
- // wins. Remove the earlier nested value before appending the
new one.
- sub_column_ptr->pop_back(1);
- }
- has_value[sub_column_idx] = true;
- auto sub_value = sub.value().value();
- RETURN_IF_ERROR(_write_data_to_column<use_string_cache>(
- sub_value, type_struct->get_element(sub_column_idx),
sub_column_ptr.get(),
- column_name + "." + sub_key, sub_serdes[sub_column_idx],
valid));
- }
- for (size_t sub_col_idx = 0; sub_col_idx <
type_struct->get_elements().size();
- ++sub_col_idx) {
- if (has_value[sub_col_idx]) {
- continue;
- }
- auto sub_column_ptr =
struct_column_ptr->get_column(sub_col_idx).get_ptr();
- if (!is_column_nullable(*sub_column_ptr)) {
- return Status::DataQualityError(
- "Json file structColumn miss field {} and this column
isn't nullable.",
- column_name + "." +
type_struct->get_element_name(sub_col_idx));
- }
- sub_column_ptr->insert_default();
- }
- } else if (primitive_type == TYPE_MAP) {
- if (value_type != simdjson::ondemand::json_type::object) {
- return Status::DataQualityError("Json value isn't object, but the
column `{}` is map.",
- column_name);
+ auto& nested = nullable.get_nested_column();
+ RETURN_IF_ERROR(serde->deserialize_one_cell_from_simdjson(
+ nested, value, _serde_options, _simdjson_context,
column_name));
+ nullable.get_null_map_data().push_back(0);
}
- const auto* map_type = assert_cast<const
DataTypeMap*>(remove_nullable(type_desc).get());
- auto* map_column_ptr = assert_cast<ColumnMap*>(data_column_ptr);
- const auto sub_serdes = data_serde->get_nested_serdes();
- size_t field_count = 0;
- simdjson::ondemand::object object_value = value.get_object();
- for (auto member_value : object_value) {
- auto mutable_keys =
map_column_ptr->get_keys_ptr()->assert_mutable();
- auto* key_column = mutable_keys.get();
- auto key_serde = sub_serdes[0];
- if (is_column_nullable(*key_column)) {
- auto* nullable_key = assert_cast<ColumnNullable*>(key_column);
- nullable_key->get_null_map_data().push_back(0);
- key_column = &nullable_key->get_nested_column();
- if (map_type->get_key_type()->is_nullable()) {
- key_serde = key_serde->get_nested_serdes()[0];
- }
- }
- std::string_view key_view = member_value.unescaped_key().value();
- Slice key_slice(key_view.data(), key_view.size());
-
RETURN_IF_ERROR(key_serde->deserialize_one_cell_from_json(*key_column,
key_slice,
-
_serde_options));
- simdjson::ondemand::value field_value =
member_value.value().value();
- auto mutable_values =
map_column_ptr->get_values_ptr()->assert_mutable();
- RETURN_IF_ERROR(_write_data_to_column<use_string_cache>(
- field_value, map_type->get_value_type(),
mutable_values.get(),
- column_name + ".value", sub_serdes[1], valid));
- ++field_count;
- }
- auto& offsets = map_column_ptr->get_offsets();
- offsets.emplace_back(offsets.back() + field_count);
- } else if (primitive_type == TYPE_ARRAY) {
- if (value_type != simdjson::ondemand::json_type::array) {
- return Status::DataQualityError("Json value isn't array, but the
column `{}` is array.",
- column_name);
- }
- const auto* array_type =
- assert_cast<const
DataTypeArray*>(remove_nullable(type_desc).get());
- auto* array_column_ptr = assert_cast<ColumnArray*>(data_column_ptr);
- const auto sub_serdes = data_serde->get_nested_serdes();
- size_t field_count = 0;
- simdjson::ondemand::array array_value = value.get_array();
- for (simdjson::ondemand::value sub_value : array_value) {
- RETURN_IF_ERROR(_write_data_to_column<use_string_cache>(
- sub_value, array_type->get_nested_type(),
&array_column_ptr->get_data(),
- column_name + ".element", sub_serdes[0], valid));
- ++field_count;
- }
- auto& offsets = array_column_ptr->get_offsets();
- offsets.emplace_back(offsets.back() + field_count);
} else {
- return Status::InternalError("Not support JSON value to complex
column");
- }
-
- if (nullable_column && value_type != simdjson::ondemand::json_type::null) {
- nullable_column->get_null_map_data().push_back(0);
+ RETURN_IF_ERROR(serde->deserialize_one_cell_from_simdjson(
+ *column_ptr, value, _serde_options, _simdjson_context,
column_name));
}
*valid = true;
return Status::OK();
diff --git a/be/src/format_v2/json/json_reader.h
b/be/src/format_v2/json/json_reader.h
index b1c413da0e7..2ac77c2242c 100644
--- a/be/src/format_v2/json/json_reader.h
+++ b/be/src/format_v2/json/json_reader.h
@@ -28,6 +28,7 @@
#include "core/custom_allocator.h"
#include "core/data_type_serde/data_type_serde.h"
+#include "core/data_type_serde/simdjson_read_context.h"
#include "exprs/json_functions.h"
#include "format_v2/file_reader.h"
#include "gen_cpp/PlanNodes_types.h"
@@ -227,7 +228,7 @@ private:
std::string _padding_buffer;
size_t _original_doc_size = 0;
size_t _padded_size = 1024 * 1024 * 8 + simdjson::SIMDJSON_PADDING;
- std::unordered_map<std::string_view, std::string_view>
_cached_string_values;
+ SimdJsonReadContext _simdjson_context;
};
} // namespace doris::format::json
diff --git a/be/src/util/jsonb_parser_simd.h b/be/src/util/jsonb_parser_simd.h
index 48e2a0cd1af..6e94a1b67b9 100644
--- a/be/src/util/jsonb_parser_simd.h
+++ b/be/src/util/jsonb_parser_simd.h
@@ -61,6 +61,7 @@
#include <limits>
#include <string_view>
+#include <unordered_map>
#include "common/status.h"
#include "util/jsonb_document.h"
@@ -70,6 +71,38 @@
namespace doris {
using int128_t = __int128;
struct JsonbParser {
+ using StringCache = std::unordered_map<std::string_view, std::string_view>;
+
+ // Rewinding an On-Demand object does not rewind its unescape buffer.
Cache decoded values
+ // and keys when jsonpaths revisit a subtree, e.g. $.payload followed by
$.payload.text.
+ static std::string_view decoded_string(simdjson::ondemand::value& value,
StringCache* cache) {
+ if (cache == nullptr) {
+ return value.get_string().value();
+ }
+ const auto token = value.raw_json().value();
+ const auto it = cache->find(token);
+ if (it != cache->end()) {
+ return it->second;
+ }
+ const auto text = value.get_string().value();
+ cache->emplace(token, text);
+ return text;
+ }
+
+ static std::string_view decoded_key(simdjson::ondemand::field& field,
StringCache* cache) {
+ if (cache == nullptr) {
+ return field.unescaped_key().value();
+ }
+ const auto token = field.key_raw_json_token();
+ const auto it = cache->find(token);
+ if (it != cache->end()) {
+ return it->second;
+ }
+ const auto key = field.unescaped_key().value();
+ cache->emplace(token, key);
+ return key;
+ }
+
private:
static bool is_json_whitespace(char c) {
return c == ' ' || c == '\t' || c == '\n' || c == '\r';
@@ -149,6 +182,18 @@ private:
}
public:
+ // Consume a subtree without copying/re-indexing its text. No value
escapes: encoding
+ // row.payload leaves the reader ready to advance to row.id.
+ static Status parse_value(simdjson::ondemand::value value, JsonbWriter&
writer,
+ StringCache* cache = nullptr) {
+ writer.reset();
+ try {
+ return parse(value, writer, cache);
+ } catch (simdjson::simdjson_error& e) {
+ return Status::InvalidArgument("simdjson parse exception: {}",
e.what());
+ }
+ }
+
// parse a UTF-8 JSON string with length
// will reset writer before parse
static Status parse(const char* pch, size_t len, JsonbWriter& writer) {
@@ -238,7 +283,8 @@ private:
// parse json, recursively if necessary, by simdjson
// and serialize to binary format by writer
- static Status parse(simdjson::ondemand::value value, JsonbWriter& writer) {
+ static Status parse(simdjson::ondemand::value value, JsonbWriter& writer,
+ StringCache* cache = nullptr) {
switch (value.type()) {
case simdjson::ondemand::json_type::null: {
return write_null(value, writer);
@@ -250,7 +296,7 @@ private:
break;
}
case simdjson::ondemand::json_type::string: {
- RETURN_IF_ERROR(write_string(value.get_string(), writer));
+ RETURN_IF_ERROR(write_string(decoded_string(value, cache),
writer));
break;
}
case simdjson::ondemand::json_type::number: {
@@ -261,12 +307,8 @@ private:
return Status::InvalidArgument("writeStartObject failed");
}
- for (auto kv : value.get_object()) {
- std::string_view key;
- simdjson::error_code e = kv.unescaped_key().get(key);
- if (e != simdjson::SUCCESS) {
- return Status::InvalidArgument(fmt::format("simdjson get
key failed: {}", e));
- }
+ for (simdjson::ondemand::field field : value.get_object()) {
+ const auto key = decoded_key(field, cache);
// write key
if (key.size() > std::numeric_limits<uint8_t>::max()) {
@@ -278,7 +320,7 @@ private:
}
// parse object value
- RETURN_IF_ERROR(parse(kv.value(), writer));
+ RETURN_IF_ERROR(parse(field.value(), writer, cache));
}
if (!writer.writeEndObject()) {
@@ -294,7 +336,7 @@ private:
for (auto elem : value.get_array()) {
// parse array element
- RETURN_IF_ERROR(parse(elem.value(), writer));
+ RETURN_IF_ERROR(parse(elem.value(), writer, cache));
}
if (!writer.writeEndArray()) {
diff --git a/be/test/core/data_type_serde/simdjson_serde_test.cpp
b/be/test/core/data_type_serde/simdjson_serde_test.cpp
new file mode 100644
index 00000000000..2f03977aeeb
--- /dev/null
+++ b/be/test/core/data_type_serde/simdjson_serde_test.cpp
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "core/column/column_map.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_decimal.h"
+#include "core/data_type/data_type_jsonb.h"
+#include "core/data_type/data_type_map.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_string.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type_serde/simdjson_read_context.h"
+#include "core/value/jsonb_value.h"
+
+namespace doris {
+namespace {
+Status append_simdjson(const DataTypeSerDe& serde, IColumn& column, const
std::string& json,
+ SimdJsonReadContext& context) {
+ context.string_values.clear();
+ simdjson::ondemand::parser parser;
+ simdjson::padded_string text("{\"v\":" + json + ",\"tail\":42}");
+ try {
+ auto doc = parser.iterate(text);
+ auto object = doc.get_object().value();
+ auto value = object.find_field_unordered("v").value();
+ auto status = serde.deserialize_one_cell_from_simdjson(
+ column, value, DataTypeSerDe::get_default_format_options(),
context, "v");
+ if (!status.ok()) {
+ return status;
+ }
+ EXPECT_EQ(object.find_field_unordered("tail").get_int64().value(), 42);
+ return Status::OK();
+ } catch (const simdjson::simdjson_error& e) {
+ return Status::DataQualityError("{}", e.what());
+ }
+}
+
+Status append_legacy(const DataTypeSerDe& serde, IColumn& column, const
std::string& json) {
+ simdjson::ondemand::parser parser;
+ simdjson::padded_string text("{\"v\":" + json + "}");
+ auto doc = parser.iterate(text);
+ auto object = doc.get_object().value();
+ auto value = object.find_field_unordered("v").value();
+ std::string_view token;
+ if (value.type().value() == simdjson::ondemand::json_type::string) {
+ token = value.get_string();
+ } else if (value.type().value() == simdjson::ondemand::json_type::boolean)
{
+ token = value.get_bool() ? "1" : "0";
+ } else {
+ token = simdjson::to_json_string(value);
+ }
+ Slice slice {token.data(), token.size()};
+ return serde.deserialize_one_cell_from_json(column, slice,
+
DataTypeSerDe::get_default_format_options());
+}
+} // namespace
+
+TEST(SimdJsonSerDeTest, ScalarConversionsMatchReader) {
+ const DataTypes types {std::make_shared<DataTypeInt8>(),
+ std::make_shared<DataTypeInt16>(),
+ std::make_shared<DataTypeInt32>(),
+ std::make_shared<DataTypeInt64>(),
+ std::make_shared<DataTypeInt128>(),
+ std::make_shared<DataTypeBool>(),
+ std::make_shared<DataTypeFloat64>(),
+ std::make_shared<DataTypeString>(),
+
std::make_shared<DataTypeDecimal<TYPE_DECIMAL128I>>(30, 5)};
+ const std::vector<std::string> values {"0",
+ "-1",
+ "127",
+ "128",
+ "-129",
+ "32768",
+ "2147483648",
+ "9223372036854775807",
+ "-9223372036854775808",
+ "9223372036854775808",
+ "18446744073709551616",
+ "1.5",
+ "1e3",
+ "true",
+ "false",
+ R"("123")",
+ R"("bad")",
+ R"("a\nb\u4e2d")"};
+ SimdJsonReadContext context;
+ for (const auto& type : types) {
+ const auto serde = type->get_serde();
+ for (const auto& json : values) {
+ SCOPED_TRACE(type->get_name() + ": " + json);
+ auto actual = type->create_column();
+ auto expected = type->create_column();
+ auto old_status = append_legacy(*serde, *expected, json);
+ auto status = append_simdjson(*serde, *actual, json, context);
+ ASSERT_EQ(status.ok(), old_status.ok()) << status << " / " <<
old_status;
+ ASSERT_EQ(actual->size(), expected->size());
+ if (status.ok()) {
+ EXPECT_EQ(actual->compare_at(0, 0, *expected, 1), 0);
+ }
+ }
+ }
+}
+
+TEST(SimdJsonSerDeTest, JsonbBytesAndLegacyScalarSemantics) {
+ DataTypeJsonb type;
+ auto serde = type.get_serde();
+ auto actual = type.create_column();
+ auto expected = type.create_column();
+ SimdJsonReadContext context;
+ const std::vector<std::string> values {
+ R"({"x":[1,"a\nb",true,null]})",
+ "[]",
+ "{}",
+
R"([18446744073709551616,-170141183460469231731687303715884105728,1.25])",
+ R"("{\"x\":1}")",
+ "true",
+ "false",
+ "123",
+ "\"" + std::string(8192, 'x') + "\""};
+ for (const auto& json : values) {
+ const auto old_status = append_legacy(*serde, *expected, json);
+ const auto status = append_simdjson(*serde, *actual, json, context);
+ ASSERT_EQ(status.ok(), old_status.ok()) << json << ": " << status;
+ }
+ ASSERT_EQ(actual->size(), expected->size());
+ for (size_t i = 0; i < actual->size(); ++i) {
+ EXPECT_EQ(actual->get_data_at(i).to_string(),
expected->get_data_at(i).to_string());
+ }
+}
+
+TEST(SimdJsonSerDeTest, NestedConversionsAndErrors) {
+ auto integer = make_nullable(std::make_shared<DataTypeInt8>());
+ auto structure =
+ std::make_shared<DataTypeStruct>(DataTypes {integer, integer},
Strings {"a", "b"});
+ auto array = std::make_shared<DataTypeArray>(make_nullable(structure));
+ auto map =
std::make_shared<DataTypeMap>(make_nullable(std::make_shared<DataTypeString>()),
+ make_nullable(array));
+ SimdJsonReadContext context;
+ auto column = map->create_column();
+ auto serde = map->get_serde();
+ ASSERT_TRUE(append_simdjson(*serde, *column,
R"({"ok":[{"a":1,"A":2,"ignored":5}]})", context)
+ .ok());
+ for (const auto& json :
+ {R"({"first":[{"a":3}],"bad":[{"a":4,"b":128}]})",
R"({"bad":[{"a":1},{"a":"oops"}]})"}) {
+ auto invalid_column = map->create_column();
+ EXPECT_FALSE(append_simdjson(*serde, *invalid_column, json,
context).ok());
+ }
+ EXPECT_TRUE(append_simdjson(*serde, *column, R"({"next":[null,{"b":7}]})",
context).ok());
+ EXPECT_EQ(column->size(), 2);
+}
+
+TEST(SimdJsonSerDeTest, RepeatedJsonpathStringAndSubtree) {
+ simdjson::ondemand::parser parser;
+ simdjson::padded_string
text(std::string_view(R"({"payload":{"s":"a\nb","n":7},"tail":42})"));
+ auto doc = parser.iterate(text);
+ auto object = doc.get_object().value();
+ SimdJsonReadContext context;
+ context.use_string_cache = true;
+ DataTypeString string_type;
+ auto strings = string_type.create_column();
+ auto string_serde = string_type.get_serde();
+ DataTypeJsonb jsonb_type;
+ auto jsonbs = jsonb_type.create_column();
+ auto jsonb_serde = jsonb_type.get_serde();
+ for (int i = 0; i < 32; ++i) {
+ auto value =
object.find_field_unordered("payload").find_field_unordered("s").value();
+ ASSERT_TRUE(string_serde
+ ->deserialize_one_cell_from_simdjson(
+ *strings, value,
DataTypeSerDe::get_default_format_options(),
+ context, "s")
+ .ok());
+ auto payload = object.find_field_unordered("payload").value();
+ ASSERT_TRUE(jsonb_serde
+ ->deserialize_one_cell_from_simdjson(
+ *jsonbs, payload,
DataTypeSerDe::get_default_format_options(),
+ context, "payload")
+ .ok());
+ }
+ EXPECT_EQ(strings->get_data_at(0).to_string(), "a\nb");
+ EXPECT_EQ(strings->get_data_at(0).to_string(),
strings->get_data_at(1).to_string());
+ EXPECT_EQ(jsonbs->get_data_at(0).to_string(),
jsonbs->get_data_at(1).to_string());
+ EXPECT_EQ(object.find_field_unordered("tail").get_int64().value(), 42);
+}
+} // namespace doris
diff --git a/be/test/format_v2/json/json_reader_test.cpp
b/be/test/format_v2/json/json_reader_test.cpp
index bf82fbcf01d..3e52e1d2a37 100644
--- a/be/test/format_v2/json/json_reader_test.cpp
+++ b/be/test/format_v2/json/json_reader_test.cpp
@@ -33,15 +33,19 @@
#include "common/object_pool.h"
#include "core/assert_cast.h"
#include "core/block/block.h"
+#include "core/column/column_array.h"
+#include "core/column/column_map.h"
#include "core/column/column_nullable.h"
#include "core/column/column_string.h"
#include "core/column/column_vector.h"
#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_jsonb.h"
#include "core/data_type/data_type_map.h"
#include "core/data_type/data_type_nullable.h"
#include "core/data_type/data_type_number.h"
#include "core/data_type/data_type_string.h"
#include "core/data_type/data_type_struct.h"
+#include "core/value/jsonb_value.h"
#include "exec/scan/scanner.h"
#include "exprs/vexpr.h"
#include "exprs/vexpr_context.h"
@@ -971,6 +975,61 @@ TEST(JsonReaderTest,
IgnoresMalformedJsonAsNullRowsWhenConfigured) {
EXPECT_EQ(nullable_string_at(*result.block.get_by_position(1).column, 1),
"judy");
}
+TEST(JsonReaderTest,
IgnoredMalformedNestedArrayDoesNotPopulateFollowingEmptyArray) {
+ for (const auto& paths : {std::string(), std::string(R"(["$.a"])")}) {
+ SCOPED_TRACE(paths);
+ ObjectPool pool;
+ auto inner = make_nullable(
+
std::make_shared<DataTypeArray>(make_nullable(std::make_shared<DataTypeInt32>())));
+ auto type = make_nullable(std::make_shared<DataTypeArray>(inner));
+ std::vector<SlotDescriptor*> slots {make_test_slot(&pool, 0, 0, type,
"a")};
+ auto result =
+ read_once("ignored_nested_array.jsonl",
+ R"({"a":[[1],2]})"
+ "\n"
+ R"({"a":[]})"
+ "\n",
+ json_scan_params(true, false, paths, "", true),
slots, {0}, false, true);
+ ASSERT_TRUE(result.status.ok()) << result.status;
+ ASSERT_EQ(result.rows, 2);
+ const auto& nullable =
+ assert_cast<const
ColumnNullable&>(*result.block.get_by_position(0).column);
+ EXPECT_TRUE(nullable.is_null_at(0));
+ EXPECT_FALSE(nullable.is_null_at(1));
+ const auto& array = assert_cast<const
ColumnArray&>(nullable.get_nested_column());
+ EXPECT_EQ(array.get_offsets()[0], 0);
+ EXPECT_EQ(array.get_offsets()[1], 0);
+ }
+}
+
+TEST(JsonReaderTest, IgnoredMalformedMapDoesNotPopulateFollowingEmptyMap) {
+ for (const auto& paths : {std::string(), std::string(R"(["$.m"])")}) {
+ SCOPED_TRACE(paths);
+ ObjectPool pool;
+ auto values = make_nullable(
+
std::make_shared<DataTypeArray>(make_nullable(std::make_shared<DataTypeInt32>())));
+ auto type = make_nullable(std::make_shared<DataTypeMap>(
+ make_nullable(std::make_shared<DataTypeString>()), values));
+ std::vector<SlotDescriptor*> slots {make_test_slot(&pool, 0, 0, type,
"m")};
+ auto result =
+ read_once("ignored_map.jsonl",
+ R"({"m":{"good":[1],"bad":2}})"
+ "\n"
+ R"({"m":{}})"
+ "\n",
+ json_scan_params(true, false, paths, "", true),
slots, {0}, false, true);
+ ASSERT_TRUE(result.status.ok()) << result.status;
+ ASSERT_EQ(result.rows, 2);
+ const auto& nullable =
+ assert_cast<const
ColumnNullable&>(*result.block.get_by_position(0).column);
+ EXPECT_TRUE(nullable.is_null_at(0));
+ EXPECT_FALSE(nullable.is_null_at(1));
+ const auto& map = assert_cast<const
ColumnMap&>(nullable.get_nested_column());
+ EXPECT_EQ(map.get_offsets()[0], 0);
+ EXPECT_EQ(map.get_offsets()[1], 0);
+ }
+}
+
TEST(JsonReaderTest, SkipsEmptyJsonLine) {
ObjectPool pool;
auto slots = build_slots(&pool);
@@ -1193,4 +1252,36 @@ TEST(JsonReaderTest,
EmptyFileProjectionMaterializesPartitionRows) {
EXPECT_TRUE(reader.close().ok());
}
+TEST(JsonReaderTest, SimdjsonJsonbSubtreesAndOverlappingPaths) {
+ ObjectPool pool;
+ auto jsonb = make_nullable(std::make_shared<DataTypeJsonb>());
+ auto string = make_nullable(std::make_shared<DataTypeString>());
+ std::vector<SlotDescriptor*> slots {make_test_slot(&pool, 0, 0, jsonb,
"payload"),
+ make_test_slot(&pool, 1, 1, string,
"first"),
+ make_test_slot(&pool, 2, 2, string,
"second"),
+ make_test_slot(&pool, 3, 3, jsonb,
"copy")};
+ auto result = read_once(
+ "simdjson_jsonb_paths.jsonl",
+ R"({"payload":{"text":"a\nb","large":18446744073709551616}})"
+ "\n"
+ R"({"payload":{"text":"next","array":[true,null,{}]}})"
+ "\n",
+ json_scan_params(true, false,
+
R"(["$.payload","$.payload.text","$.payload.text","$.payload"])"),
+ slots, {0, 1, 2, 3});
+ ASSERT_TRUE(result.status.ok()) << result.status;
+ ASSERT_EQ(result.rows, 2);
+ for (size_t i = 0; i < result.rows; ++i) {
+ EXPECT_EQ(nullable_string_at(*result.block.get_by_position(0).column,
i),
+ nullable_string_at(*result.block.get_by_position(3).column,
i));
+ EXPECT_EQ(nullable_string_at(*result.block.get_by_position(1).column,
i),
+ nullable_string_at(*result.block.get_by_position(2).column,
i));
+ }
+ EXPECT_EQ(nullable_string_at(*result.block.get_by_position(1).column, 0),
"a\nb");
+ JsonBinaryValue expected;
+
ASSERT_TRUE(expected.from_json_string(R"({"text":"a\nb","large":18446744073709551616})").ok());
+ EXPECT_EQ(nullable_string_at(*result.block.get_by_position(0).column, 0),
+ std::string(expected.value(), expected.size()));
+}
+
} // namespace doris::format::json
diff --git a/regression-test/data/jsonb_p0/simdjson_column_deserialization.json
b/regression-test/data/jsonb_p0/simdjson_column_deserialization.json
new file mode 100644
index 00000000000..b0c9bfaf7ce
--- /dev/null
+++ b/regression-test/data/jsonb_p0/simdjson_column_deserialization.json
@@ -0,0 +1,4 @@
+{"id": 1, "small": 127, "flag": true, "amount": 123.5678, "payload": {"n":
18446744073709551616, "s": "a\nb"}, "items": [{"x": 1, "X": 2, "y": "a\nb"},
null, {"y": "missing"}], "attrs": {"k": [1, null, 3]}, "first_text": "a\nb",
"second_text": "other"}
+{"id": 2, "small": -128, "flag": false, "amount": "-0.1250", "payload": [1,
{"a": [true, null, "中"]}], "items": [], "attrs": {}, "first_text":
"quote\"slash\\", "second_text": "two"}
+{"id": 3, "small": 128, "flag": true, "amount": 1, "payload": {"bad": true},
"items": [{"x": 7}], "attrs": {"bad": [1]}, "first_text": "filtered"}
+{"id": 4, "small": "12", "flag": "true", "amount": 0, "payload":
"{\"embedded\":1}", "items": [{"x": 4}], "attrs": {"next": []}, "first_text":
"after", "second_text": "last"}
diff --git
a/regression-test/data/jsonb_p0/test_simdjson_column_deserialization.out
b/regression-test/data/jsonb_p0/test_simdjson_column_deserialization.out
new file mode 100644
index 00000000000..5a0e07c895d
--- /dev/null
+++ b/regression-test/data/jsonb_p0/test_simdjson_column_deserialization.out
@@ -0,0 +1,11 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !by_name --
+1 127 true 123.5678 {"n":18446744073709551616,"s":"a\\nb"}
[{"x":2, "y":"anb"}, null, {"x":null, "y":"missing"}] {"k":[1, null, 3]}
a\nb other
+2 -128 false -0.1250 [1,{"a":[true,null,"中"]}] [] {}
quote"slash\\ two
+4 12 true 0.0000 {"embedded":1} [{"x":4, "y":null}]
{"next":[]} after last
+
+-- !by_path --
+1 127 true 123.5678 {"n":18446744073709551616,"s":"a\\nb"}
[{"x":2, "y":"anb"}, null, {"x":null, "y":"missing"}] {"k":[1, null, 3]}
a\nb a\nb
+2 -128 false -0.1250 [1,{"a":[true,null,"中"]}] [] {}
quote"slash\\ quote"slash\\
+4 12 true 0.0000 {"embedded":1} [{"x":4, "y":null}]
{"next":[]} after after
+
diff --git
a/regression-test/suites/jsonb_p0/test_simdjson_column_deserialization.groovy
b/regression-test/suites/jsonb_p0/test_simdjson_column_deserialization.groovy
new file mode 100644
index 00000000000..904a430e867
--- /dev/null
+++
b/regression-test/suites/jsonb_p0/test_simdjson_column_deserialization.groovy
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_simdjson_column_deserialization", "p0") {
+ sql "DROP TABLE IF EXISTS test_simdjson_column_deserialization"
+ sql """CREATE TABLE test_simdjson_column_deserialization (
+ id INT, small TINYINT, flag BOOLEAN, amount DECIMAL(20,4), payload
JSONB,
+ items ARRAY<STRUCT<x:INT,y:STRING>>, attrs MAP<STRING,ARRAY<INT>>,
+ first_text STRING, second_text STRING
+ ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES("replication_num"="1")"""
+ streamLoad {
+ table "test_simdjson_column_deserialization"
+ set "format", "json"
+ set "read_json_by_line", "true"
+ set "strict_mode", "true"
+ set "max_filter_ratio", "0.5"
+ file "simdjson_column_deserialization.json"
+ time 10000
+ check { result, exception, startTime, endTime ->
+ if (exception != null) {
+ throw exception
+ }
+ def load = parseJson(result)
+ assertEquals("Success", load.Status)
+ assertEquals(4, load.NumberTotalRows)
+ assertEquals(3, load.NumberLoadedRows)
+ assertEquals(1, load.NumberFilteredRows)
+ }
+ }
+ sql "sync"
+ order_qt_by_name """SELECT id,small,flag,amount,CAST(payload AS
STRING),items,attrs,
+ first_text,second_text
+ FROM test_simdjson_column_deserialization"""
+ sql "TRUNCATE TABLE test_simdjson_column_deserialization"
+ streamLoad {
+ table "test_simdjson_column_deserialization"
+ set "format", "json"
+ set "read_json_by_line", "true"
+ set "strict_mode", "true"
+ set "max_filter_ratio", "0.5"
+ set "jsonpaths",
'["$.id","$.small","$.flag","$.amount","$.payload","$.items","$.attrs","$.first_text","$.first_text"]'
+ file "simdjson_column_deserialization.json"
+ time 10000
+ check { result, exception, startTime, endTime ->
+ if (exception != null) {
+ throw exception
+ }
+ def load = parseJson(result)
+ assertEquals("Success", load.Status)
+ assertEquals(4, load.NumberTotalRows)
+ assertEquals(3, load.NumberLoadedRows)
+ assertEquals(1, load.NumberFilteredRows)
+ }
+ }
+ sql "sync"
+ order_qt_by_path """SELECT id,small,flag,amount,CAST(payload AS
STRING),items,attrs,
+ first_text,second_text
+ FROM test_simdjson_column_deserialization"""
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]