This is an automated email from the ASF dual-hosted git repository.
blaginin pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-sandbox.git
from c4011e61f Merge branch 'main' into sandbox-main
add 2f10d453f Expose Spilling Progress Interface in DataFusion (#19708)
add 1d19c5285 perf: improve performance of `spark hex` function (#19738)
add 6bf5e98b5 dev: Add a script to auto fix all lint violations (#19560)
add 567ba7584 doc: Add an auto-generated dependency graph for internal
crates (#19280)
add 429f5a792 refactor: Optimize `required_columns` from `BTreeSet` to
`Vec` in struct `PushdownChecker` (#19678)
add 472a729af Revert Workround for Empty FixedSizeBinary Values Buffer
After arrow-rs Upgrade (#19801)
add 79869a753 feat: support Set Comparison Subquery (#19109)
add c4f039f53 chore(deps): bump taiki-e/install-action from 2.66.2 to
2.66.3 (#19802)
add f73da4da3 Add Reproducer for Issues with LEFT joins on Fixed Size
Binary Columns (#19800)
add 154ab7004 feat(spark): implement array slice function (#19811)
add 0808f3a8d Improvements to `list_files_cache` table function (#19703)
add eadbed51b perf: Optimize initcap scalar performance (#19776)
add a414ea078 Issue 19781 : Internal error: Assertion failed:
!self.finished: LimitedBatchCoalescer (#19785)
add c173de1fc fix: expose `ListFilesEntry` (#19804)
add 094e7ee90 physical plan: add `reset_plan_states `, plan re-use
benchmark (#19806)
add 6f92ea600 chore(deps): bump actions/setup-node from 6.1.0 to 6.2.0
(#19825)
add de40f0c9f Docs: Fix some links in docs (#19834)
add 54b848c07 feat(spark): implement substring function (#19805)
add 7c9a76acf Use correct setting for click bench queries in sql_planner
benchmark (#19835)
add 5d68b751c fix: trunc function with precision uses round instead of
trunc semantics (#19794)
add 14d919dda chore(deps): bump taiki-e/install-action from 2.66.3 to
2.66.5 (#19824)
add 144126942 chore: refactor scalarvalue/encoding using available
upstream arrow-rs methods (#19797)
add 7b0ed2d61 Refactor Spark `date_add`/`date_sub`/`bitwise_not` to remove
unnecessary scalar arg check (#19473)
add 444ddf2e1 Add BatchAdapter to simplify using PhysicalExprAdapter /
Projector to map RecordBatch between schemas (#19716)
add 5edda9b30 fix: calculate total seconds from interval fields for
`extract(epoch)` (#19807)
add ca904b308 Row group limit pruning for row groups that entirely match
predicates (#18868)
add 9208f1287 [Minor] Reuse indices buffer in RepartitionExec (#19775)
add 5e893ab92 fix: predicate cache stats calculation (#19561)
add f3f6dec7c feat: Add support for 'isoyear' in date_part function
(#19821)
add c91dcfdb6 Docs: add additional links to blog posts (#19833)
add ab81d3bd2 Fix(optimizer): Make `EnsureCooperative` optimizer
idempotent under multiple runs (#19757)
add ac67ae4af Ensure null inputs to array setop functions return null
output (#19683)
add 8afccc951 Allow dropping qualified columns (#19549)
add 0c959d6d9 Doc: Add more blog links to doc comments (#19837)
add 3d90d4b60 datafusion/common: Add support for hashing ListView arrays
(#19814)
add e353eb08f Project sort expressions in StreamingTable (#19719)
add 1ab7e4184 Fix grouping set subset satisfaction (#19853)
add 3ea21aa33 perf: Optimize trunc scalar performance (#19788)
add 0aab6a3d2 feat: support `SELECT DISTINCT id FROM t ORDER BY id LIMIT
n` query use GroupedTopKAggregateStream (#19653)
add 8b179d93e perf: optimize `spark_hex` dictionary path by avoiding
dictionary expansion (#19832)
add fc7d090fb Spark date part (#19823)
add 936f959dd Add FilterExecBuilder to avoid recomputing properties
multiple times (#19854)
add 4ed808a83 feat(spark): add trunc, date_trunc and time_trunc functions
(#19829)
add 05451da3c feat(spark): implement Spark `date_diff` function (#19845)
add d90d0746d perf: Optimize round scalar performance (#19831)
add 3f77ee4f0 chore(deps): bump wasm-bindgen-test from 0.3.56 to 0.3.58
(#19898)
add b8fcd0135 chore(deps): bump tokio-postgres from 0.7.15 to 0.7.16
(#19899)
add c35b897ea chore(deps): bump postgres-types from 0.2.11 to 0.2.12
(#19902)
add 4dfcd7b31 chore(deps): bump insta from 1.46.0 to 1.46.1 (#19901)
add 1a5c100c7 chore(deps): bump taiki-e/install-action from 2.66.5 to
2.66.7 (#19883)
add 5b122cc0d Consolidate cte_quoted_reference.slt into cte.slt (#19862)
add 84e1ccc0d chore(deps): bump sphinx from 8.2.3 to 9.1.0 in /docs
(#19647)
add dededf7c7 Disable failing `array_union` edge-case with nested null
array (#19904)
add 35ebe7955 feat(spark): implement add_months function (#19711)
add 913d1c0a7 perf: Optimize signum scalar performance with fast path
(#19871)
add 6b1a1cb81 fix: preserve state in DistinctMedianAccumulator::evaluate()
for window frame queries (#19887)
add 9626776b4 fix: null in array_agg with DISTINCT and IGNORE (#19736)
add 774a7b536 chore(deps): bump the proto group across 1 directory with 5
updates (#19745)
add 406e5aa1e test(wasmtest): enable compression feature for wasm build
(#19860)
add 4d8d48c0c perf: Optimize scalar performance for cot (#19888)
add e82dc21d1 Feat : added truncate table support (#19633)
add 10a1d4ea1 Remove UDAF manual Debug impls and simplify signatures
(#19727)
add db64be984 chore(deps): bump thiserror from 2.0.17 to 2.0.18 (#19900)
add b2c29ac8d Include license and notice files in more crates (#19913)
add 3b44bbf60 feat: support pushdown alias on dynamic filter with
`ProjectionExec` (#19404)
add 6b7ddd803 chore(deps): bump actions/setup-python from 6.1.0 to 6.2.0
(#19935)
add bfe7d18ab perf: Optimize scalar fast path for iszero (#19919)
add 9f27e933a Coerce expressions to udtf (#19915)
add efccfb1e4 Fix trailing whitespace in CROSS JOIN logical plan
formatting (#19936)
add f0e38df39 chore(deps): bump chrono from 0.4.42 to 0.4.43 (#19897)
add 7a09e2776 fix: union should retrun error instead of panic when input
schema's len different (#19922)
add cf76352d9 Improve error message when string functions receive Binary
types (#19819)
add 7cf63f1b0 Refactor ListArray hashing to consider only sliced values
(#19500)
add 736fa7cce feat(datafusion-spark): implement spark compatible `unhex`
function (#19909)
add c2f3d6541 Support API for "pre-image" for pruning predicate evaluation
(#19722)
add 0c82adec7 Fix struct casts to align fields by name (prevent positional
mis-casts) (#19674)
add 45c8ae43e Support LargeUtf8 as partition column (#19942)
add b463a9f9e chore(deps): bump actions/checkout from 6.0.1 to 6.0.2
(#19953)
add 35e99b97b preserve FilterExec batch size during ser/de (#19960)
add 23f5003ca Add struct pushdown query benchmark and projection pushdown
tests (#19962)
add 17cbff06e Misc hash / hash aggregation performance improvements
(#19910)
add c5e99e595 perf: Optimize scalar path for ascii function (#19951)
add 1897924b7 fix: change token consumption to pick to test on EOF in
parser (#19927)
add 05802e205 perf: Optimize factorial scalar path (#19949)
add c1352360b Improve error messages with nicer formatting of Date and
Time types (#19954)
add e5e76366a export `SessionState::register_catalog_list(...)` (#19925)
add 62f8237f7 Merge branch 'upstream-main' into sandbox-main
add b3c3f6bcf Merge remote-tracking branch 'sandbox/main' into sandbox-main
No new revisions were added by this update.
Summary of changes:
.github/workflows/audit.yml | 4 +-
.github/workflows/dependencies.yml | 6 +-
.github/workflows/dev.yml | 8 +-
.github/workflows/docs.yaml | 12 +-
.github/workflows/docs_pr.yaml | 11 +-
.github/workflows/extended.yml | 8 +-
.github/workflows/labeler.yml | 2 +-
.github/workflows/large_files.yml | 2 +-
.github/workflows/rust.yml | 60 +-
Cargo.lock | 166 +--
Cargo.toml | 8 +-
ci/scripts/doc_prettier_check.sh | 85 +-
ci/scripts/license_header.sh | 62 +-
ci/scripts/rust_clippy.sh | 59 +-
ci/scripts/rust_docs.sh | 1 +
ci/scripts/rust_fmt.sh | 51 +-
ci/scripts/rust_toml_fmt.sh | 55 +-
ci/scripts/typos_check.sh | 55 +-
ci/scripts/{rust_docs.sh => utils/git.sh} | 11 +-
datafusion-cli/src/functions.rs | 15 +-
datafusion/catalog-listing/src/helpers.rs | 1 +
datafusion/catalog/src/streaming.rs | 26 +-
datafusion/catalog/src/table.rs | 8 +
datafusion/common/Cargo.toml | 1 +
datafusion/common/src/config.rs | 2 +-
datafusion/common/src/hash_utils.rs | 202 +++-
datafusion/common/src/metadata.rs | 4 +
datafusion/common/src/nested_struct.rs | 402 +++++++-
datafusion/common/src/scalar/mod.rs | 62 +-
datafusion/common/src/types/native.rs | 52 +-
datafusion/common/src/utils/proxy.rs | 34 +-
datafusion/core/Cargo.toml | 11 +
datafusion/core/benches/parquet_struct_query.rs | 312 ++++++
datafusion/core/benches/reset_plan_states.rs | 198 ++++
datafusion/core/benches/sql_planner.rs | 5 +
datafusion/core/benches/topk_aggregate.rs | 183 ++++
datafusion/core/src/dataframe/mod.rs | 70 +-
datafusion/core/src/datasource/file_format/avro.rs | 4 +-
.../core/src/datasource/file_format/parquet.rs | 6 +-
.../core/src/datasource/listing_table_factory.rs | 4 +-
datafusion/core/src/execution/session_state.rs | 14 +-
datafusion/core/src/physical_planner.rs | 45 +-
.../tests/custom_sources_cases/dml_planning.rs | 102 +-
datafusion/core/tests/dataframe/mod.rs | 155 ++-
datafusion/core/tests/execution/coop.rs | 5 +-
datafusion/core/tests/parquet/filter_pushdown.rs | 16 +
datafusion/core/tests/parquet/mod.rs | 111 ++-
datafusion/core/tests/parquet/row_group_pruning.rs | 339 ++++++-
.../combine_partial_final_agg.rs | 4 +-
.../tests/physical_optimizer/enforce_sorting.rs | 50 +-
.../physical_optimizer/filter_pushdown/mod.rs | 249 ++++-
datafusion/core/tests/set_comparison.rs | 193 ++++
datafusion/core/tests/sql/unparser.rs | 4 +
.../user_defined/user_defined_table_functions.rs | 25 +
datafusion/datasource-parquet/src/access_plan.rs | 4 +
datafusion/datasource-parquet/src/metrics.rs | 27 +-
datafusion/datasource-parquet/src/opener.rs | 27 +-
datafusion/datasource-parquet/src/row_filter.rs | 33 +-
.../datasource-parquet/src/row_group_filter.rs | 219 +++-
datafusion/datasource-parquet/src/source.rs | 1 +
datafusion/datasource/src/file_scan_config.rs | 34 +
datafusion/datasource/src/schema_adapter.rs | 24 +-
datafusion/datasource/src/sink.rs | 2 +-
datafusion/datasource/src/source.rs | 17 +
datafusion/datasource/src/url.rs | 12 +-
datafusion/datasource/src/write/demux.rs | 10 +-
datafusion/execution/src/cache/cache_manager.rs | 1 +
datafusion/execution/src/cache/list_files_cache.rs | 5 +
datafusion/execution/src/cache/mod.rs | 1 +
datafusion/execution/src/disk_manager.rs | 31 +-
datafusion/execution/src/runtime_env.rs | 7 +-
datafusion/expr-common/src/columnar_value.rs | 153 ++-
datafusion/expr-common/src/groups_accumulator.rs | 3 +
datafusion/expr-common/src/type_coercion/binary.rs | 118 ++-
datafusion/expr/src/expr.rs | 97 ++
datafusion/expr/src/expr_schema.rs | 14 +-
datafusion/expr/src/lib.rs | 1 +
datafusion/expr/src/logical_plan/builder.rs | 4 +-
datafusion/expr/src/logical_plan/display.rs | 8 +-
datafusion/expr/src/logical_plan/dml.rs | 3 +
datafusion/expr/src/logical_plan/invariants.rs | 20 +-
datafusion/expr/src/logical_plan/plan.rs | 17 +-
datafusion/expr/src/logical_plan/tree_node.rs | 19 +-
datafusion/expr/src/partition_evaluator.rs | 4 +
datafusion/expr/src/planner.rs | 17 +-
.../arrow_writer.rs => expr/src/preimage.rs} | 22 +-
datafusion/expr/src/tree_node.rs | 20 +-
datafusion/expr/src/type_coercion/functions.rs | 11 +-
datafusion/expr/src/udf.rs | 48 +
datafusion/expr/src/utils.rs | 1 +
.../src/approx_percentile_cont_with_weight.rs | 12 +-
datafusion/functions-aggregate/src/bool_and_or.rs | 12 +-
datafusion/functions-aggregate/src/correlation.rs | 2 +-
datafusion/functions-aggregate/src/count.rs | 11 +-
datafusion/functions-aggregate/src/covariance.rs | 122 +--
datafusion/functions-aggregate/src/first_last.rs | 24 +-
datafusion/functions-aggregate/src/grouping.rs | 12 +-
datafusion/functions-aggregate/src/median.rs | 17 +-
datafusion/functions-aggregate/src/regr.rs | 104 +-
datafusion/functions-aggregate/src/stddev.rs | 45 +-
datafusion/functions-aggregate/src/variance.rs | 68 +-
datafusion/functions-nested/src/array_has.rs | 2 +-
datafusion/functions-nested/src/except.rs | 51 +-
datafusion/functions-nested/src/flatten.rs | 2 +-
datafusion/functions-nested/src/map.rs | 2 +-
datafusion/functions-nested/src/planner.rs | 16 +-
datafusion/functions-nested/src/set_ops.rs | 109 +-
datafusion/functions-nested/src/utils.rs | 2 +-
datafusion/functions/Cargo.toml | 5 +
datafusion/functions/benches/ascii.rs | 31 +-
datafusion/functions/benches/cot.rs | 47 +-
datafusion/functions/benches/initcap.rs | 125 ++-
datafusion/functions/benches/iszero.rs | 45 +-
.../functions/benches/{floor_ceil.rs => round.rs} | 67 +-
datafusion/functions/benches/signum.rs | 46 +
datafusion/functions/benches/trunc.rs | 48 +-
datafusion/functions/src/core/union_tag.rs | 2 +-
datafusion/functions/src/datetime/date_part.rs | 37 +-
datafusion/functions/src/datetime/date_trunc.rs | 20 +-
datafusion/functions/src/datetime/to_char.rs | 4 +-
datafusion/functions/src/encoding/inner.rs | 93 +-
datafusion/functions/src/math/cot.rs | 301 ++++--
datafusion/functions/src/math/factorial.rs | 94 +-
datafusion/functions/src/math/iszero.rs | 119 +--
datafusion/functions/src/math/round.rs | 64 +-
datafusion/functions/src/math/signum.rs | 82 +-
datafusion/functions/src/math/trunc.rs | 59 +-
datafusion/functions/src/string/ascii.rs | 30 +-
datafusion/functions/src/unicode/initcap.rs | 35 +-
datafusion/functions/src/unicode/substr.rs | 4 +-
datafusion/optimizer/src/analyzer/type_coercion.rs | 41 +-
.../src/decorrelate_predicate_subquery.rs | 4 +-
datafusion/optimizer/src/eliminate_cross_join.rs | 14 +-
datafusion/optimizer/src/lib.rs | 1 +
.../optimizer/src/optimize_projections/mod.rs | 13 +-
datafusion/optimizer/src/optimizer.rs | 2 +
datafusion/optimizer/src/push_down_filter.rs | 5 +-
datafusion/optimizer/src/push_down_limit.rs | 4 +-
datafusion/optimizer/src/rewrite_set_comparison.rs | 171 ++++
.../src/simplify_expressions/expr_simplifier.rs | 271 ++++-
.../optimizer/src/simplify_expressions/mod.rs | 1 +
.../src/simplify_expressions/udf_preimage.rs | 364 +++++++
.../optimizer/src/single_distinct_to_groupby.rs | 18 +-
.../LICENSE.txt | 0
.../NOTICE.txt | 0
datafusion/physical-expr-adapter/src/lib.rs | 5 +-
.../physical-expr-adapter/src/schema_rewriter.rs | 348 +++++++
datafusion/physical-expr-common/src/binary_map.rs | 4 +-
.../physical-expr-common/src/binary_view_map.rs | 5 +-
.../physical-expr-common/src/metrics/value.rs | 40 +-
.../src/equivalence/properties/mod.rs | 2 +-
datafusion/physical-expr/src/expressions/cast.rs | 35 +-
.../physical-expr/src/expressions/cast_column.rs | 2 +-
.../src/expressions/dynamic_filters.rs | 4 +
.../physical-expr/src/expressions/try_cast.rs | 6 +-
.../src/combine_partial_final_agg.rs | 4 +-
.../physical-optimizer/src/enforce_distribution.rs | 19 +-
.../physical-optimizer/src/enforce_sorting/mod.rs | 10 +-
datafusion/physical-optimizer/src/ensure_coop.rs | 328 +++++-
.../physical-optimizer/src/limit_pushdown.rs | 27 +-
.../src/limited_distinct_aggregation.rs | 4 +-
.../physical-optimizer/src/topk_aggregation.rs | 44 +-
.../group_values/single_group_by/primitive.rs | 4 +-
datafusion/physical-plan/src/aggregates/mod.rs | 85 +-
.../physical-plan/src/aggregates/row_hash.rs | 2 +-
.../physical-plan/src/aggregates/topk_stream.rs | 53 +-
datafusion/physical-plan/src/coalesce/mod.rs | 4 +
.../physical-plan/src/coalesce_partitions.rs | 13 +
datafusion/physical-plan/src/column_rewriter.rs | 383 +++++++
datafusion/physical-plan/src/coop.rs | 5 +
datafusion/physical-plan/src/execution_plan.rs | 38 +
datafusion/physical-plan/src/filter.rs | 642 +++++++++---
datafusion/physical-plan/src/filter_pushdown.rs | 76 +-
datafusion/physical-plan/src/lib.rs | 1 +
datafusion/physical-plan/src/limit.rs | 29 +
datafusion/physical-plan/src/projection.rs | 506 +++++++++-
datafusion/physical-plan/src/recursive_query.rs | 16 +-
datafusion/physical-plan/src/repartition/mod.rs | 53 +-
.../src/sorts/sort_preserving_merge.rs | 13 +
.../src/spill/in_progress_spill_file.rs | 29 +-
datafusion/physical-plan/src/spill/mod.rs | 74 +-
datafusion/physical-plan/src/spill/spill_pool.rs | 10 +-
datafusion/physical-plan/src/test.rs | 2 +-
datafusion/physical-plan/src/topk/mod.rs | 3 +
datafusion/physical-plan/src/union.rs | 45 +-
datafusion/proto-common/gen/Cargo.toml | 4 +-
datafusion/proto-common/src/generated/pbjson.rs | 132 +--
datafusion/proto/gen/Cargo.toml | 4 +-
datafusion/proto/proto/datafusion.proto | 4 +
datafusion/proto/src/generated/pbjson.rs | 414 ++++----
datafusion/proto/src/generated/prost.rs | 8 +
datafusion/proto/src/logical_plan/from_proto.rs | 1 +
datafusion/proto/src/logical_plan/to_proto.rs | 2 +
datafusion/proto/src/physical_plan/mod.rs | 32 +-
.../proto/tests/cases/roundtrip_logical_plan.rs | 1 +
.../proto/tests/cases/roundtrip_physical_plan.rs | 13 +-
.../{catalog-listing => pruning}/LICENSE.txt | 0
datafusion/{catalog-listing => pruning}/NOTICE.txt | 0
datafusion/pruning/src/pruning_predicate.rs | 1 -
datafusion/spark/Cargo.toml | 17 +
datafusion/spark/benches/hex.rs | 152 +++
.../array_slice.rs => spark/benches/slice.rs} | 59 +-
.../substr.rs => spark/benches/substring.rs} | 4 +-
datafusion/spark/benches/unhex.rs | 148 +++
datafusion/spark/src/function/array/mod.rs | 9 +-
datafusion/spark/src/function/array/slice.rs | 172 ++++
.../spark/src/function/bitwise/bitwise_not.rs | 52 +-
.../spark/src/function/datetime/add_months.rs | 95 ++
datafusion/spark/src/function/datetime/date_add.rs | 29 +-
.../spark/src/function/datetime/date_diff.rs | 119 +++
.../spark/src/function/datetime/date_part.rs | 139 +++
datafusion/spark/src/function/datetime/date_sub.rs | 26 +-
.../spark/src/function/datetime/date_trunc.rs | 172 ++++
datafusion/spark/src/function/datetime/mod.rs | 52 +-
.../spark/src/function/datetime/time_trunc.rs | 122 +++
datafusion/spark/src/function/datetime/trunc.rs | 143 +++
datafusion/spark/src/function/math/hex.rs | 337 ++++---
datafusion/spark/src/function/math/mod.rs | 4 +
datafusion/spark/src/function/math/unhex.rs | 214 ++++
.../spark/src/function/string/format_string.rs | 2 +-
datafusion/spark/src/function/string/mod.rs | 8 +
datafusion/spark/src/function/string/substring.rs | 258 +++++
datafusion/spark/src/function/url/parse_url.rs | 13 +-
datafusion/spark/src/lib.rs | 19 +
.../src/unicode => spark/src}/planner.rs | 12 +-
datafusion/sql/Cargo.toml | 1 +
datafusion/sql/src/expr/mod.rs | 62 +-
datafusion/sql/src/expr/subquery.rs | 51 +-
datafusion/sql/src/parser.rs | 46 +-
datafusion/sql/src/statement.rs | 50 +
datafusion/sql/src/unparser/expr.rs | 29 +-
datafusion/sql/tests/cases/plan_to_sql.rs | 2 +-
datafusion/sql/tests/sql_integration.rs | 12 +-
datafusion/sqllogictest/Cargo.toml | 6 +-
datafusion/sqllogictest/src/test_context.rs | 16 +-
datafusion/sqllogictest/test_files/aggregate.slt | 89 +-
.../sqllogictest/test_files/aggregates_topk.slt | 118 +++
datafusion/sqllogictest/test_files/array.slt | 81 +-
.../sqllogictest/test_files/arrow_typeof.slt | 18 +-
datafusion/sqllogictest/test_files/binary.slt | 10 +
datafusion/sqllogictest/test_files/case.slt | 12 +-
datafusion/sqllogictest/test_files/cte.slt | 63 +-
.../test_files/cte_quoted_reference.slt | 70 --
.../sqllogictest/test_files/datetime/date_part.slt | 132 ++-
.../test_files/datetime/timestamps.slt | 90 +-
datafusion/sqllogictest/test_files/dictionary.slt | 4 +-
.../test_files/dynamic_filter_pushdown_config.slt | 31 +
datafusion/sqllogictest/test_files/encoding.slt | 6 +-
datafusion/sqllogictest/test_files/explain.slt | 16 +-
datafusion/sqllogictest/test_files/expr.slt | 2 +-
datafusion/sqllogictest/test_files/group_by.slt | 6 +-
.../test_files/grouping_set_repartition.slt | 246 +++++
.../sqllogictest/test_files/information_schema.slt | 24 +-
datafusion/sqllogictest/test_files/joins.slt | 78 +-
datafusion/sqllogictest/test_files/limit.slt | 2 +-
.../sqllogictest/test_files/limit_pruning.slt | 81 ++
.../test_files/limit_single_row_batches.slt | 22 +
.../test_files/projection_pushdown.slt | 1054 ++++++++++++++++++++
datafusion/sqllogictest/test_files/scalar.slt | 8 +-
.../sqllogictest/test_files/spark/array/slice.slt | 101 ++
.../test_files/spark/datetime/add_months.slt | 52 +-
.../test_files/spark/datetime/date_add.slt | 11 +-
.../test_files/spark/datetime/date_diff.slt | 150 ++-
.../test_files/spark/datetime/date_part.slt | 304 +++++-
.../test_files/spark/datetime/date_sub.slt | 10 +
.../test_files/spark/datetime/date_trunc.slt | 177 +++-
.../test_files/spark/datetime/datediff.slt | 32 -
.../test_files/spark/datetime/datepart.slt | 62 --
.../test_files/spark/datetime/time_trunc.slt | 74 ++
.../test_files/spark/datetime/trunc.slt | 101 +-
.../sqllogictest/test_files/spark/math/hex.slt | 20 +
.../sqllogictest/test_files/spark/math/unhex.slt | 98 ++
.../test_files/spark/string/format_string.slt | 32 +-
.../test_files/spark/string/substr.slt | 37 -
.../test_files/spark/string/substring.slt | 206 +++-
datafusion/sqllogictest/test_files/struct.slt | 904 ++++++++++++++++-
datafusion/sqllogictest/test_files/subquery.slt | 61 +-
datafusion/sqllogictest/test_files/topk.slt | 2 +-
datafusion/sqllogictest/test_files/truncate.slt | 85 ++
datafusion/substrait/Cargo.toml | 3 +-
.../src/logical_plan/consumer/expr/literal.rs | 2 +
.../src/logical_plan/consumer/expr/mod.rs | 1 +
.../src/logical_plan/consumer/expr/subquery.rs | 56 +-
.../src/logical_plan/consumer/rel/fetch_rel.rs | 2 +
.../substrait/src/logical_plan/consumer/types.rs | 1 +
.../src/logical_plan/producer/expr/mod.rs | 1 +
.../src/logical_plan/producer/expr/subquery.rs | 56 +-
.../logical_plan/producer/substrait_producer.rs | 19 +-
.../substrait/tests/cases/consumer_integration.rs | 62 +-
.../tests/cases/roundtrip_logical_plan.rs | 80 +-
datafusion/substrait/tests/utils.rs | 1 +
datafusion/wasmtest/Cargo.toml | 6 +-
datafusion/wasmtest/src/lib.rs | 59 +-
dev/rust_lint.sh | 127 ++-
docs/.gitignore | 4 +
docs/README.md | 5 +
docs/build.sh | 9 +-
docs/requirements.txt | 4 +-
docs/scripts/generate_dependency_graph.sh | 97 ++
.../architecture/dependency-graph.md | 180 ++++
docs/source/index.rst | 1 +
docs/source/library-user-guide/extending-sql.md | 5 +
.../library-user-guide/functions/adding-udfs.md | 4 +
docs/source/library-user-guide/query-optimizer.md | 16 +-
docs/source/library-user-guide/upgrading.md | 36 +
docs/source/user-guide/concepts-readings-events.md | 28 +
docs/source/user-guide/explain-usage.md | 1 +
docs/source/user-guide/sql/data_types.md | 5 +
docs/source/user-guide/sql/index.rst | 1 +
docs/source/user-guide/sql/scalar_functions.md | 5 +-
docs/source/user-guide/sql/struct_coercion.md | 354 +++++++
311 files changed, 16848 insertions(+), 2901 deletions(-)
copy ci/scripts/{rust_docs.sh => utils/git.sh} (73%)
mode change 100755 => 100644
create mode 100644 datafusion/core/benches/parquet_struct_query.rs
create mode 100644 datafusion/core/benches/reset_plan_states.rs
create mode 100644 datafusion/core/tests/set_comparison.rs
copy datafusion/{common/src/file_options/arrow_writer.rs =>
expr/src/preimage.rs} (67%)
copy datafusion/functions/benches/{floor_ceil.rs => round.rs} (67%)
create mode 100644 datafusion/optimizer/src/rewrite_set_comparison.rs
create mode 100644
datafusion/optimizer/src/simplify_expressions/udf_preimage.rs
copy datafusion/{catalog-listing => physical-expr-adapter}/LICENSE.txt (100%)
copy datafusion/{catalog-listing => physical-expr-adapter}/NOTICE.txt (100%)
create mode 100644 datafusion/physical-plan/src/column_rewriter.rs
copy datafusion/{catalog-listing => pruning}/LICENSE.txt (100%)
copy datafusion/{catalog-listing => pruning}/NOTICE.txt (100%)
create mode 100644 datafusion/spark/benches/hex.rs
copy datafusion/{functions-nested/benches/array_slice.rs =>
spark/benches/slice.rs} (73%)
copy datafusion/{functions/benches/substr.rs => spark/benches/substring.rs}
(98%)
create mode 100644 datafusion/spark/benches/unhex.rs
create mode 100644 datafusion/spark/src/function/array/slice.rs
create mode 100644 datafusion/spark/src/function/datetime/add_months.rs
create mode 100644 datafusion/spark/src/function/datetime/date_diff.rs
create mode 100644 datafusion/spark/src/function/datetime/date_part.rs
create mode 100644 datafusion/spark/src/function/datetime/date_trunc.rs
create mode 100644 datafusion/spark/src/function/datetime/time_trunc.rs
create mode 100644 datafusion/spark/src/function/datetime/trunc.rs
create mode 100644 datafusion/spark/src/function/math/unhex.rs
create mode 100644 datafusion/spark/src/function/string/substring.rs
copy datafusion/{functions/src/unicode => spark/src}/planner.rs (81%)
delete mode 100644 datafusion/sqllogictest/test_files/cte_quoted_reference.slt
create mode 100644
datafusion/sqllogictest/test_files/grouping_set_repartition.slt
create mode 100644 datafusion/sqllogictest/test_files/limit_pruning.slt
create mode 100644
datafusion/sqllogictest/test_files/limit_single_row_batches.slt
create mode 100644 datafusion/sqllogictest/test_files/projection_pushdown.slt
create mode 100644 datafusion/sqllogictest/test_files/spark/array/slice.slt
delete mode 100644
datafusion/sqllogictest/test_files/spark/datetime/datediff.slt
delete mode 100644
datafusion/sqllogictest/test_files/spark/datetime/datepart.slt
create mode 100644
datafusion/sqllogictest/test_files/spark/datetime/time_trunc.slt
create mode 100644 datafusion/sqllogictest/test_files/spark/math/unhex.slt
delete mode 100644 datafusion/sqllogictest/test_files/spark/string/substr.slt
create mode 100644 datafusion/sqllogictest/test_files/truncate.slt
create mode 100755 docs/scripts/generate_dependency_graph.sh
create mode 100644
docs/source/contributor-guide/architecture/dependency-graph.md
create mode 100644 docs/source/user-guide/sql/struct_coercion.md
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
