This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/github_actions/taiki-e/install-action-2.67.16
in repository https://gitbox.apache.org/repos/asf/datafusion-sandbox.git
discard a6a354a78 chore(deps): bump taiki-e/install-action from 2.66.7 to
2.67.16
add 412768873 Change GitHub actions dependabot schedule to weekly (#19981)
add 0b60c588b chore(deps): bump setuptools from 80.9.0 to 80.10.1 in /docs
(#19988)
add 664758329 chore(deps): bump taiki-e/install-action from 2.66.7 to
2.67.9 (#19987)
add 853a61716 chore(deps): bump quote from 1.0.43 to 1.0.44 (#19992)
add 355cae870 chore(deps): bump nix from 0.30.1 to 0.31.1 (#19991)
add d590824ed chore(deps): bump sysinfo from 0.37.2 to 0.38.0 (#19990)
add 03c2751f1 chore(deps): bump uuid from 1.19.0 to 1.20.0 (#19993)
add e062ae505 minor: Fix doc about `write_batch_size` (#19979)
add d1eea0755 Fix broken links in the documentation (#19964)
add 8023947fa fix: maintain inner list nullability for `array_sort`
(#19948)
add 4d63f8c92 minor: Add favicon (#20000)
add b6f752175 Do not require mut in memory reservation methods (#19759)
add f8a22a51b docs: Fix some broken / missing links in the DataFusion
documentation (#19958)
add 50a3e138e Speedup statistics_from_parquet_metadata (#20004)
add 8efc2b605 feat(spark): add `base64` and `unbase64` functions (#19968)
add 27abe5a62 perf: improve performance of `array_remove`,
`array_remove_n` and `array_remove_all` functions (#19996)
add 6524d9193 minor: pull `uuid` into workspace dependencies (#19997)
add cda642528 perf: Optimize ArrowBytesViewMap with direct view access
(#19975)
add 5c47d6d13 Fix ClickBench EventDate handling by casting UInt16
days-since-epoch to DATE via `hits` view (#19881)
add adddd4c32 fix: Make `generate_series` return an empty set with invalid
ranges (#19999)
add 073174b03 feat: Show the number of matched Parquet pages in
`DataSourceExec` (#19977)
add 8653851f5 feat(spark): Add `SessionStateBuilderSpark` to
datafusion-spark (#19865)
add 58fb6e19d feat(spark): implement `from/to_utc_timestamp` functions
(#19880)
add c560beeb1 perf: Optimize repeat function for scalar and array fast
(#19976)
add f5709e748 feat(spark): implement `StringView` for `SparkConcat`
(#19984)
add 52deee513 chore(deps): bump setuptools from 80.10.1 to 80.10.2 in
/docs (#20022)
add 82f27e9af refactor: extract pushdown test utilities to shared module
(#20010)
add af771970b chore(deps): bump taiki-e/install-action from 2.67.9 to
2.67.13 (#20020)
add aca716901 feat(spark): add unix date and timestamp functions (#19892)
add 1a48d580c add more projection pushdown slt tests (#20015)
add adb8c8a71 minor: Move metric `page_index_rows_pruned` to verbose level
in `EXPLAIN ANALYZE` (#20026)
add aae3e0f4a refactor: make PhysicalExprAdatperFactory::create fallible
(#20017)
add 41d48b3f6 feat: implement protobuf converter trait to allow control
over serialization and deserialization processes (#19437)
add f81906183 Tweak `adapter serialization` example (#20035)
add bef1368c3 Simplify wait_complete function (#19937)
add 0a68b81ad [main] Update version to `52.1.0` (#19878) (#20028)
add 81e972328 Fix/parquet opener page index policy (#19890)
add 47bf17859 fix: return correct length array for scalar null input to
`calculate_binary_math` (#19861)
add 16368983b minor: add tests for coercible signature considering
nulls/dicts/ree (#19459)
add 5f9bacddc Enforce `clippy::allow_attributes` globally across workspace
(#19576)
add 4b31aaa14 Fix constant value from stats (#20042)
add 568f19fa3 Simplify Spark `sha2` implementation (#19475)
add a77e5a5c5 Further refactoring of type coercion function code (#19603)
add 20d5f06a5 perf: Push down join key filters for LEFT/RIGHT/ANTI joins
(#19918)
add 36c0cda20 fix: respect DataFrameWriteOptions::with_single_file_output
for paths without extensions (#19931)
add 92f60ad51 docs: Automatically update DataFusion version in docs
(#20001)
add 8152b4443 feat: optimise copying in `left` for Utf8 and LargeUtf8
(#19980)
add 41bfb794c feat: support Spark-compatible abs math function part 2 -
ANSI mode (#18828)
add ed0a06018 replace private is_volatile_expression_tree with equivalent
public is_volatile (#20056)
add f997169b2 Improve documentation for ScalarUDFImpl::preimage (#20008)
add 7c3ea0540 feat: add AggregateMode::PartialReduce for tree-reduce
aggregation (#20019)
add f0de02fd6 Use BooleanBufferBuilder rather than Vec<bool> in
ArrowBytesViewMap (#20064)
add cad38654c fix: correct weight handling in
approx_percentile_cont_with_weight (#19941)
add 2860adaf4 fix: The limit_pushdown physical optimization rule removes
limits in some cases leading to incorrect results (#20048)
add b50e1f755 chore: Add microbenchmark (compared to ExprOrExpr) (#20076)
add 4a6365979 Minor: update tests in limit_pushdown.rs to insta (#20066)
add a02e6836e Reduce number of traversals per node in
`PhysicalExprSimplifier` (#20082)
add bc4c24503 Automatically generate examples documentation adv (#19294)
(#19750)
add 9de192af2 docs: update data_types.md to reflect current Arrow type
mappings (#20072)
add f779e9668 perf: Optimize scalar path for chr function (#20073)
add 379799596 perf: improve performance of `array_repeat` function (#20049)
add 51c04754e Implement preimage for floor function to enable predicate
pushdown (#20059)
add c84937472 Refactor `iszero()` and `isnan()` to accept all numeric
types (#20093)
add 1a0c2e05c Use return_field_from_args in information schema and
date_trunc (#20079)
add 545c37fbc perf: optimise right for byte access and StringView (#20069)
add 6b83141a3 Merge branch 'main' into sanbox-main
add 1ccb78625 chore(deps): bump taiki-e/install-action from 2.66.7 to
2.67.16
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a6a354a78)
\
N -- N -- N
refs/heads/dependabot/github_actions/taiki-e/install-action-2.67.16 (1ccb78625)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/dependabot.yml | 2 +-
.github/workflows/audit.yml | 2 +-
.github/workflows/rust.yml | 19 +-
Cargo.lock | 176 ++--
Cargo.toml | 80 +-
benchmarks/src/clickbench.rs | 36 +-
ci/scripts/check_examples_docs.sh | 85 +-
datafusion-cli/src/exec.rs | 2 +-
datafusion-examples/Cargo.toml | 7 +-
datafusion-examples/README.md | 28 +-
.../examples/builtin_functions/main.rs | 12 +-
.../custom_data_source/adapter_serialization.rs | 519 +++++++++
.../custom_data_source/custom_file_casts.rs | 8 +-
.../custom_data_source/default_column_values.rs | 8 +-
.../custom_data_source/file_stream_provider.rs | 3 +-
.../examples/custom_data_source/main.rs | 36 +-
datafusion-examples/examples/data_io/catalog.rs | 8 -
.../examples/data_io/json_shredding.rs | 8 +-
datafusion-examples/examples/data_io/main.rs | 40 +-
.../examples/data_io/parquet_encrypted.rs | 2 +-
datafusion-examples/examples/dataframe/main.rs | 11 +-
.../examples/execution_monitoring/main.rs | 12 +-
.../examples/external_dependency/main.rs | 8 +-
datafusion-examples/examples/flight/main.rs | 12 +-
datafusion-examples/examples/flight/sql_server.rs | 5 -
.../examples/proto/expression_deduplication.rs | 275 +++++
datafusion-examples/examples/proto/main.rs | 14 +-
.../examples/query_planning/main.rs | 32 +-
.../examples/relation_planner/main.rs | 12 +-
datafusion-examples/examples/sql_ops/main.rs | 16 +-
datafusion-examples/examples/udf/main.rs | 32 +-
datafusion-examples/src/bin/examples-docs.rs | 45 +
datafusion-examples/src/utils/examples_docs.rs | 684 ++++++++++++
datafusion-examples/src/utils/mod.rs | 1 +
datafusion/catalog-listing/src/mod.rs | 1 -
datafusion/catalog-listing/src/table.rs | 3 +-
datafusion/catalog/src/information_schema.rs | 63 +-
datafusion/catalog/src/lib.rs | 1 -
datafusion/common-runtime/src/lib.rs | 1 -
datafusion/common/src/config.rs | 2 +-
datafusion/common/src/lib.rs | 1 -
datafusion/core/Cargo.toml | 6 +-
datafusion/core/benches/data_utils/mod.rs | 2 +
.../core/benches/preserve_file_partitioning.rs | 2 +-
datafusion/core/src/bin/print_functions_docs.rs | 27 +-
datafusion/core/src/dataframe/mod.rs | 35 +-
datafusion/core/src/dataframe/parquet.rs | 156 ++-
.../core/src/datasource/file_format/parquet.rs | 7 +-
datafusion/core/src/datasource/mod.rs | 6 +-
.../core/src/datasource/physical_plan/parquet.rs | 96 +-
datafusion/core/src/execution/context/mod.rs | 2 +-
datafusion/core/src/lib.rs | 1 -
datafusion/core/src/physical_planner.rs | 27 +-
.../tests/custom_sources_cases/dml_planning.rs | 2 +-
datafusion/core/tests/execution/coop.rs | 2 +-
datafusion/core/tests/fifo/mod.rs | 4 +-
.../aggregation_fuzzer/context_generator.rs | 2 +-
.../fuzz_cases/aggregation_fuzzer/query_builder.rs | 6 +-
datafusion/core/tests/fuzz_cases/join_fuzz.rs | 2 +-
.../fuzz_cases/sort_preserving_repartition_fuzz.rs | 2 +-
datafusion/core/tests/macro_hygiene/mod.rs | 4 +-
datafusion/core/tests/parquet/expr_adapter.rs | 6 +-
datafusion/core/tests/parquet/filter_pushdown.rs | 1 -
datafusion/core/tests/parquet/page_pruning.rs | 60 +-
.../{filter_pushdown/mod.rs => filter_pushdown.rs} | 8 +-
.../tests/physical_optimizer/limit_pushdown.rs | 398 +++++--
datafusion/core/tests/physical_optimizer/mod.rs | 3 +-
.../{filter_pushdown/util.rs => pushdown_utils.rs} | 37 +-
datafusion/core/tests/sql/explain_analyze.rs | 5 +-
.../user_defined/user_defined_window_functions.rs | 2 +-
datafusion/datasource-arrow/src/mod.rs | 1 -
datafusion/datasource-avro/src/mod.rs | 1 -
datafusion/datasource-csv/src/mod.rs | 1 -
datafusion/datasource-json/src/mod.rs | 1 -
datafusion/datasource-parquet/src/file_format.rs | 10 +-
datafusion/datasource-parquet/src/metadata.rs | 53 +-
datafusion/datasource-parquet/src/metrics.rs | 11 +-
datafusion/datasource-parquet/src/mod.rs | 1 -
datafusion/datasource-parquet/src/opener.rs | 10 +-
datafusion/datasource-parquet/src/page_filter.rs | 25 +-
datafusion/datasource-parquet/src/row_filter.rs | 2 +
datafusion/datasource/src/file_sink_config.rs | 48 +
datafusion/datasource/src/mod.rs | 1 -
datafusion/datasource/src/write/demux.rs | 5 +-
datafusion/doc/src/lib.rs | 1 -
datafusion/execution/src/config.rs | 6 +
datafusion/execution/src/lib.rs | 1 -
datafusion/execution/src/memory_pool/mod.rs | 92 +-
datafusion/execution/src/memory_pool/pool.rs | 38 +-
datafusion/expr-common/src/lib.rs | 1 -
datafusion/expr/src/expr.rs | 2 +-
datafusion/expr/src/expr_schema.rs | 254 ++---
datafusion/expr/src/lib.rs | 1 -
datafusion/expr/src/type_coercion/functions.rs | 282 +++--
datafusion/expr/src/udf.rs | 101 +-
datafusion/expr/src/utils.rs | 27 +-
datafusion/ffi/src/lib.rs | 1 -
datafusion/functions-aggregate-common/src/lib.rs | 2 -
.../functions-aggregate-common/src/tdigest.rs | 54 +-
.../functions-aggregate/benches/array_agg.rs | 2 +-
datafusion/functions-aggregate/benches/count.rs | 2 +-
.../functions-aggregate/src/approx_median.rs | 2 +-
.../src/approx_percentile_cont.rs | 8 +-
datafusion/functions-aggregate/src/lib.rs | 2 -
datafusion/functions-nested/Cargo.toml | 8 +
.../functions-nested/benches/array_remove.rs | 573 ++++++++++
.../functions-nested/benches/array_repeat.rs | 477 +++++++++
datafusion/functions-nested/src/lib.rs | 2 -
datafusion/functions-nested/src/make_array.rs | 12 +-
datafusion/functions-nested/src/remove.rs | 127 ++-
datafusion/functions-nested/src/repeat.rs | 175 ++--
datafusion/functions-nested/src/sort.rs | 36 +-
datafusion/functions-table/src/generate_series.rs | 37 +-
datafusion/functions-table/src/lib.rs | 2 -
datafusion/functions-window-common/src/lib.rs | 2 -
datafusion/functions-window/src/lib.rs | 1 -
datafusion/functions-window/src/nth_value.rs | 2 +-
datafusion/functions/Cargo.toml | 9 +-
datafusion/functions/benches/chr.rs | 31 +-
datafusion/functions/benches/date_trunc.rs | 11 +-
datafusion/functions/benches/encoding.rs | 20 +-
datafusion/functions/benches/left.rs | 141 ++-
datafusion/functions/benches/repeat.rs | 39 +
datafusion/functions/benches/right.rs | 150 +++
datafusion/functions/benches/trim.rs | 4 +-
datafusion/functions/src/datetime/date_trunc.rs | 26 +-
datafusion/functions/src/datetime/to_local_time.rs | 2 +-
datafusion/functions/src/datetime/to_timestamp.rs | 15 +-
datafusion/functions/src/datetime/to_unixtime.rs | 7 +-
datafusion/functions/src/encoding/inner.rs | 37 +-
datafusion/functions/src/lib.rs | 2 -
datafusion/functions/src/math/abs.rs | 4 +-
datafusion/functions/src/math/floor.rs | 264 ++++-
datafusion/functions/src/math/iszero.rs | 124 ++-
datafusion/functions/src/math/nans.rs | 173 ++-
datafusion/functions/src/string/chr.rs | 126 ++-
datafusion/functions/src/string/repeat.rs | 164 ++-
datafusion/functions/src/unicode/left.rs | 249 ++++-
datafusion/functions/src/unicode/right.rs | 245 ++++-
datafusion/functions/src/utils.rs | 22 +-
datafusion/macros/Cargo.toml | 2 +-
datafusion/macros/src/user_doc.rs | 1 -
datafusion/optimizer/src/analyzer/type_coercion.rs | 66 +-
datafusion/optimizer/src/lib.rs | 1 -
datafusion/optimizer/src/push_down_filter.rs | 22 +-
.../src/simplify_expressions/udf_preimage.rs | 1 -
datafusion/physical-expr-adapter/src/lib.rs | 2 -
.../physical-expr-adapter/src/schema_rewriter.rs | 65 +-
.../physical-expr-common/src/binary_view_map.rs | 195 +++-
datafusion/physical-expr-common/src/lib.rs | 2 -
.../physical-expr-common/src/metrics/value.rs | 27 +-
datafusion/physical-expr/benches/case_when.rs | 109 ++
.../src/expressions/dynamic_filters.rs | 17 +-
datafusion/physical-expr/src/lib.rs | 2 -
.../src/simplifier/const_evaluator.rs | 19 +-
datafusion/physical-expr/src/simplifier/mod.rs | 4 +-
.../physical-optimizer/src/aggregate_statistics.rs | 6 +-
datafusion/physical-optimizer/src/lib.rs | 2 -
.../physical-optimizer/src/limit_pushdown.rs | 3 +-
.../physical-optimizer/src/projection_pushdown.rs | 17 +-
.../physical-optimizer/src/update_aggr_exprs.rs | 6 +-
datafusion/physical-plan/src/aggregates/mod.rs | 294 +++++-
.../physical-plan/src/aggregates/no_grouping.rs | 25 +-
.../physical-plan/src/aggregates/row_hash.rs | 72 +-
datafusion/physical-plan/src/joins/cross_join.rs | 2 +-
.../physical-plan/src/joins/hash_join/exec.rs | 10 -
.../physical-plan/src/joins/nested_loop_join.rs | 4 +-
.../src/joins/piecewise_merge_join/exec.rs | 2 +-
datafusion/physical-plan/src/lib.rs | 2 -
datafusion/physical-plan/src/sorts/sort.rs | 4 +-
datafusion/physical-plan/src/sorts/stream.rs | 4 +-
datafusion/physical-plan/src/stream.rs | 4 +-
datafusion/physical-plan/src/work_table.rs | 2 +-
datafusion/proto-common/src/generated/mod.rs | 1 +
datafusion/proto-common/src/lib.rs | 1 -
datafusion/proto/proto/datafusion.proto | 13 +
datafusion/proto/src/bytes/mod.rs | 56 +-
datafusion/proto/src/generated/mod.rs | 4 +-
datafusion/proto/src/generated/pbjson.rs | 97 ++
datafusion/proto/src/generated/prost.rs | 39 +
datafusion/proto/src/lib.rs | 1 -
datafusion/proto/src/physical_plan/from_proto.rs | 169 ++-
datafusion/proto/src/physical_plan/mod.rs | 1105 ++++++++++++--------
datafusion/proto/src/physical_plan/to_proto.rs | 174 ++-
.../proto/tests/cases/roundtrip_physical_plan.rs | 237 ++++-
datafusion/pruning/src/lib.rs | 1 -
datafusion/session/src/lib.rs | 1 -
datafusion/spark/Cargo.toml | 9 +
datafusion/spark/src/function/array/spark_array.rs | 17 +-
.../src/function/datetime/from_utc_timestamp.rs | 195 ++++
datafusion/spark/src/function/datetime/mod.rs | 60 ++
.../src/function/datetime/to_utc_timestamp.rs | 225 ++++
datafusion/spark/src/function/datetime/unix.rs | 174 +++
datafusion/spark/src/function/hash/sha2.rs | 249 ++---
datafusion/spark/src/function/math/abs.rs | 356 +++++--
datafusion/spark/src/function/string/base64.rs | 183 ++++
datafusion/spark/src/function/string/concat.rs | 46 +-
datafusion/spark/src/function/string/mod.rs | 15 +
datafusion/spark/src/lib.rs | 22 +-
datafusion/spark/src/session_state.rs | 111 ++
datafusion/sql/src/lib.rs | 1 -
datafusion/sqllogictest/Cargo.toml | 2 +-
datafusion/sqllogictest/src/test_context.rs | 16 +-
datafusion/sqllogictest/src/util.rs | 2 +-
datafusion/sqllogictest/test_files/aggregate.slt | 21 +-
datafusion/sqllogictest/test_files/array.slt | 43 +
datafusion/sqllogictest/test_files/clickbench.slt | 34 +-
.../test_files/dynamic_filter_pushdown_config.slt | 2 +-
datafusion/sqllogictest/test_files/encoding.slt | 77 +-
datafusion/sqllogictest/test_files/expr.slt | 10 +
.../sqllogictest/test_files/information_schema.slt | 2 +-
.../test_files/join_disable_repartition_joins.slt | 20 +-
datafusion/sqllogictest/test_files/joins.slt | 7 +-
datafusion/sqllogictest/test_files/limit.slt | 4 +-
.../sqllogictest/test_files/limit_pruning.slt | 4 +-
datafusion/sqllogictest/test_files/math.slt | 32 +
.../test_files/projection_pushdown.slt | 331 +++++-
.../sqllogictest/test_files/push_down_filter.slt | 254 +++++
datafusion/sqllogictest/test_files/scalar.slt | 4 +-
datafusion/sqllogictest/test_files/spark/README.md | 12 +
.../spark/datetime/from_utc_timestamp.slt | 156 +++
.../test_files/spark/datetime/to_utc_timestamp.slt | 150 ++-
.../test_files/spark/datetime/unix.slt | 134 +++
.../sqllogictest/test_files/spark/hash/sha2.slt | 55 +
.../sqllogictest/test_files/spark/math/abs.slt | 202 +++-
.../test_files/spark/string/base64.slt | 113 +-
.../test_files/spark/string/concat.slt | 24 +
.../test_files/spark/string/unbase64.slt | 27 -
.../sqllogictest/test_files/table_functions.slt | 30 +-
datafusion/sqllogictest/test_files/union.slt | 35 +-
datafusion/substrait/Cargo.toml | 2 +-
datafusion/substrait/src/lib.rs | 1 -
datafusion/wasmtest/src/lib.rs | 7 +-
dev/changelog/{49.0.2.md => 52.1.0.md} | 17 +-
dev/update_datafusion_versions.py | 8 +-
docs/requirements.txt | 2 +-
.../_static/favicon.svg} | 0
docs/source/conf.py | 2 +
docs/source/contributor-guide/howtos.md | 2 +-
docs/source/contributor-guide/testing.md | 1 +
docs/source/download.md | 2 +-
docs/source/library-user-guide/extending-sql.md | 2 +-
.../library-user-guide/functions/adding-udfs.md | 1 -
docs/source/library-user-guide/query-optimizer.md | 2 +
.../source/library-user-guide/table-constraints.md | 6 +-
docs/source/library-user-guide/upgrading.md | 133 ++-
docs/source/user-guide/arrow-introduction.md | 9 +-
docs/source/user-guide/concepts-readings-events.md | 8 +-
docs/source/user-guide/configs.md | 4 +-
docs/source/user-guide/crate-configuration.md | 3 +-
docs/source/user-guide/example-usage.md | 6 +-
docs/source/user-guide/explain-usage.md | 1 +
docs/source/user-guide/sql/data_types.md | 80 +-
docs/source/user-guide/sql/format_options.md | 2 +-
docs/source/user-guide/sql/scalar_functions.md | 23 +-
test-utils/src/data_gen.rs | 2 +-
256 files changed, 11361 insertions(+), 2915 deletions(-)
create mode 100644
datafusion-examples/examples/custom_data_source/adapter_serialization.rs
create mode 100644
datafusion-examples/examples/proto/expression_deduplication.rs
create mode 100644 datafusion-examples/src/bin/examples-docs.rs
create mode 100644 datafusion-examples/src/utils/examples_docs.rs
rename datafusion/core/tests/physical_optimizer/{filter_pushdown/mod.rs =>
filter_pushdown.rs} (99%)
rename datafusion/core/tests/physical_optimizer/{filter_pushdown/util.rs =>
pushdown_utils.rs} (92%)
create mode 100644 datafusion/functions-nested/benches/array_remove.rs
create mode 100644 datafusion/functions-nested/benches/array_repeat.rs
create mode 100644 datafusion/functions/benches/right.rs
create mode 100644 datafusion/spark/src/function/datetime/from_utc_timestamp.rs
create mode 100644 datafusion/spark/src/function/datetime/to_utc_timestamp.rs
create mode 100644 datafusion/spark/src/function/datetime/unix.rs
create mode 100644 datafusion/spark/src/function/string/base64.rs
create mode 100644 datafusion/spark/src/session_state.rs
create mode 100644
datafusion/sqllogictest/test_files/spark/datetime/from_utc_timestamp.slt
create mode 100644 datafusion/sqllogictest/test_files/spark/datetime/unix.slt
delete mode 100644 datafusion/sqllogictest/test_files/spark/string/unbase64.slt
copy dev/changelog/{49.0.2.md => 52.1.0.md} (67%)
copy docs/{logos/standalone_logo/logo_original.svg =>
source/_static/favicon.svg} (100%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
