ozankabak commented on code in PR #64:
URL: https://github.com/apache/datafusion-site/pull/64#discussion_r2012487934
##########
content/blog/2025-03-24-datafusion-46.0.0.md:
##########
@@ -0,0 +1,92 @@
+---
+layout: post
+title: Apache DataFusion 46.0.0 Released
+date: 2025-03-24
+author: Oznur Hanci and Berkay Sahin on behalf of the PMC
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+We’re excited to announce the release of **Apache DataFusion 46.0.0**! This
new version represents a significant milestone for the project, packing in a
wide range of improvements and fixes. You can find the complete details in the
full
[changelog](https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md).
We’ll highlight the most important changes below and guide you through
upgrading.
+
+## Breaking Changes
+
+DataFusion 46.0.0 brings a few **breaking changes** that may require
adjustments to your code:
+
+- [Unified `DataSourceExec` Execution
Plan](https://github.com/apache/datafusion/pull/14224#)**:** DataFusion 46.0.0
introduces a major refactor of scan operators. The separate
file-format-specific execution plan nodes (`ParquetExec`, `CsvExec`,
`JsonExec`, `AvroExec`, etc.) have been **deprecated and merged into a single
`DataSourceExec` plan**. Format-specific logic is now encapsulated in new
`DataSource` and `FileSource` traits. This change simplifies the execution
model, but if you have code that directly references the old plan nodes, you’ll
need to update it to use `DataSourceExec` (see the [Upgrade
Guide](https://datafusion.apache.org/library-user-guide/upgrading.html) for
examples of the new API).
+- [**Error Handling
Improvements](https://github.com/apache/arrow-datafusion/issues/7360#:~:text=2)
(`DataFusionError::Collection`):** We began overhauling DataFusion’s approach
to error handling. In this release, a new error variant
`DataFusionError::Collection` (and related mechanisms) has been introduced to
aggregate multiple errors into one. This is part of a broader effort to provide
richer error context and reduce internal panics. As a result, some error types
or messages have changed. Downstream code that matches on specific
`DataFusionError` variants might need adjustment.
+
+## Highlighted New Features
+
+### Improved Diagnostics
+
+DataFusion 46.0.0 introduces a new [**SQL Diagnostics
framework**](https://github.com/apache/datafusion/issues/14429) to make error
messages more understandable. This comes in the form of new `Diagnostic` and
`DiagnosticEntry` types, which allow the system to attach rich context (like
source query text spans) to error messages. In practical terms, certain planner
errors will now point to the exact location in your SQL query that caused the
issue.
+
+For example, if you reference an unknown table or miss a column in `GROUP BY`
the error message will include the query snippet causing the error. These
diagnostics are meant for end-users of applications built on DataFusion,
providing clearer messages instead of generic errors. Currently, diagnostics
cover unresolved table/column references, missing `GROUP BY`columns, ambiguous
references, wrong number of UNION columns, type mismatches, and a few others.
Future releases will extend this to more error types. This feature should
greatly ease debugging of complex SQL by pinpointing errors directly in the
query text. We thank [@eliaperantoni](https://github.com/eliaperantoni) for his
contributions in this project.
+
+### Unified `DataSourceExec` for Table Providers
+
+As mentioned, DataFusion now uses a unified `DataSourceExec` for reading
tables, which is both a breaking change and a feature. *Why is this important?*
The new approach simplifies how custom table providers are integrated and
optimized. Namely, the optimizer can treat file scans uniformly and push down
filters/limits more consistently when there is one execution plan that handles
all data sources. The new `DataSourceExec` is paired with a `DataSource` trait
that encapsulates format-specific behaviors (Parquet, CSV, JSON, Avro, etc.) in
a pluggable way.
+
+All built-in sources (Parquet, CSV, Avro, Arrow, JSON, etc.) have been
migrated to this framework. This unification makes the codebase cleaner and
sets the stage for future enhancements (like consistent metadata handling and
limit pushdown across all formats). Check out PR
[#14224](https://github.com/apache/datafusion/pull/14224) for design details.
We thank [@mertak-synnada](https://github.com/mertak-synnada) and
[@ozankabak](https://github.com/ozankabak) for their contributions.
+
+### FFI Support for Scalar UDFs
+
+DataFusion’s Foreign Function Interface (FFI) has been extended to support
[**user-defined scalar
functions**](https://github.com/apache/datafusion/pull/14579) defined in
external languages. In 46.0.0, you can now expose a custom scalar UDF through
the FFI layer and use it in DataFusion as if it were built-in. This is
particularly exciting for the **Python bindings** and other language
integrations – it means you could define a function in Python (or C, etc.) and
register it with DataFusion’s Rust core via the FFI crate. Thanks,
[@timsaucer](https://github.com/timsaucer)!
+
+### New Statistics/Distribution Framework
+
+This release, thanks mainly to [@Fly-Style](https://github.com/Fly-Style) with
contributions from [@ozankabak](https://github.com/ozankabak) and
[@berkaysynnada](https://github.com/berkaysynnada), includes the initial pieces
of a [**redesigned statistics
framework](https://github.com/apache/datafusion/pull/14699).** DataFusion’s
optimizer can now represent column data distributions using a new
`Distribution` enum, instead of the old precision or range estimations. The
supported distribution types currently include **Uniform, Gaussian (normal),
Exponential, Bernoulli**, and a **Generic** catch-all.
+
+For example, if a filter expression is applied to a column with a known
uniform distribution range, the optimizer can propagate that to estimate result
selectivity more accurately. Similarly, comparisons (`=`, `>`, etc.) on columns
yield Bernoulli distributions (with true/false probabilities) in this model.
+
+This is a foundational change with many follow-on PRs underway. Even though
the immediate user-visible effect is limited (the optimizer didn't magically
improve by an order of magnitude overnight), but it lays groundwork for more
advanced query planning in the future. Over time, as statistics information
encapsulated in `Distribution`s get integrated, DataFusion will be able to make
smarter decisions like more aggressive parquet pruning, better join orderings,
and so on based on data distribution information. The core framework is now in
place and is being hooked up to column and table level statistics.
+
+### Aggregate Monotonicity and Window Ordering
+
+DataFusion 46.0.0 adds a new concept of
[set](https://github.com/apache/datafusion/pull/14271#)[-monotonicity](https://github.com/apache/datafusion/blob/5210a2bac32e43dc7bf6e7e6000cdeaf2833c06e/datafusion/expr/src/udaf.rs#L1090)
for certain transformations, which helps avoid unnecessary sort operations. In
particular, the planner now understands when a **window function introduces new
orderings of data**. For example, DataFusion now recognizes that a
window-aggregate like `MAX` on a column can have an ordering even if the column
itself doesn't have an ordering (for certain window frames). PR
[#14271](https://github.com/apache/datafusion/pull/14271) introduced a
“set-monotonicity” property for window functions, and a follow-up PR
[#14813](https://github.com/apache/datafusion/pull/14813) refined the handling
of sort order in window frames. Huge thanks to
[@berkaysynnada](https://github.com/berkaysynnada) and
[@mertak-synnada](https://github.com/mertak-synnada) for this feature.
+
+## Performance Improvements
Review Comment:
Agreed, let's do it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
