Omega359 commented on code in PR #57:
URL: https://github.com/apache/datafusion-site/pull/57#discussion_r1966565413
##########
content/blog/2025-02-20-datafusion-45.0.0.md:
##########
@@ -0,0 +1,300 @@
+---
+layout: post
+title: Apache DataFusion 45.0.0 Released
+date: 2025-02-20
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!-- see https://github.com/apache/datafusion/issues/9602 for details -->
+
+## Introduction
+
+We are very proud to announce [DataFusion 45.0.0]. This blog highlights some
of the
+many major improvements since we released [DataFusion 40.0.0] and a preview of
+what the community is thinking about in the next 6 months. It has been an
exciting
+period of development for DataFusion!
+
+[DataFusion 40.0.0]:
https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/
+[DataFusion 45.0.0]: https://crates.io/crates/datafusion/45.0.0
+
+[Apache DataFusion] is an extensible query engine, written in [Rust], that
+uses [Apache Arrow] as its in-memory format. DataFusion is used by developers
to
+create new, fast data centric systems such as databases, dataframe libraries,
+machine learning and streaming applications. While [DataFusion’s primary design
+goal] is to accelerate the creation of other data centric systems, it has a
+reasonable experience directly out of the box as a [dataframe library],
+[python library] and [command line SQL tool].
+
+[apache datafusion]: https://datafusion.apache.org/
+[rust]: https://www.rust-lang.org/
+[apache arrow]: https://arrow.apache.org
+[DataFusion’s primary design goal]:
https://datafusion.apache.org/user-guide/introduction.html#project-goals
+[dataframe library]: https://datafusion.apache.org/user-guide/dataframe.html
+[python library]: https://datafusion.apache.org/python/
+[command line SQL tool]: https://datafusion.apache.org/user-guide/cli/
+
+DataFusion's core thesis is that as a community, together we can build much
more
+advanced technology than any of us as individuals or companies could do alone.
+Without DataFusion, highly performant vectorized query engines would remain
+the domain of a few large companies and world-class research institutions.
+With DataFusion, we can all build on top of a shared foundation, and focus on
+what makes our projects unique.
+
+
+## Community Growth 📈
+
+In the last 6 months, between `40.0.0` and `45.0.0`, our community continues to
+grow in new and exciting ways.
+
+1. We added several PMC members and new committers: [@jayzhan211] and
[@jonahgoa] joined the PMC,
+ [@2010YOUY01], [@rachelint], [@findpi], [@iffyio], [@goldmedal],
[@Weijun-H], [@Michael-J-Ward] and [@korowa]
+ joined as committers. See the [mailing list] for more details.
+2. In the [core DataFusion repo] alone we reviewed and accepted almost 1600
PRs from 206 different
+ committers, created over 1100 issues and closed 751 of them 🚀. All changes
are listed in the detailed
+ [changelogs].
+3. DataFusion focused meetups happened in multiple cities around the world:
[Hangzhou], [Belgrade], [New York],
+ [Seattle], [Chicago], [Boston] and [Amsterdam] as well as a Rust NYC meetup
in NYC focussed on DataFusion.
+
+[core DataFusion repo]: https://github.com/apache/arrow-datafusion
+[changelogs]: https://github.com/apache/datafusion/tree/main/dev/changelog
+[mailing list]: https://lists.apache.org/list.html?d...@datafusion.apache.org
+[Hangzhou]:
https://github.com/apache/datafusion/discussions/10341#discussioncomment-10110273
+[Belgrade]: https://github.com/apache/datafusion/discussions/11431
+[New York]: https://github.com/apache/datafusion/discussions/11213
+[Seattle]: https://github.com/apache/datafusion/discussions/10348
+[Chicago]: https://github.com/apache/datafusion/discussions/12894
+[Boston]: https://github.com/apache/datafusion/discussions/13165
+[Amsterdam]: https://github.com/apache/datafusion/discussions/12988
+
+<!--
+$ git log --pretty=oneline 40.0.0..45.0.0 . | wc -l
+ 1532 (up from 1453)
+
+$ git shortlog -sn 40.0.0..45.0.0 . | wc -l
+ 206 (up from 182)
+
+https://crates.io/crates/datafusion/45.0.0
+DataFusion 45 released Feb 7, 2025
+
+https://crates.io/crates/datafusion/40.0.0
+DataFusion 40 released July 12, 2024
+
+Issues created in this time: 375 open, 751 closed (from 321 open, 781 closed)
+https://github.com/apache/datafusion/issues?q=is%3Aissue+created%3A2024-07-12..2025-02-07
+
+Issues closed: 956 (up from 911)
+https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+closed%3A2024-07-12..2025-02-07
+
+PRs merged in this time 1597 (up from 1490)
+https://github.com/apache/arrow-datafusion/pulls?q=is%3Apr+merged%3A2024-07-12..2025-02-07
+
+-->
+
+DataFusion has put in an application to be part of [Google Summer of Code]
with a
+[number of ideas] for projects with mentors already selected. Additionally,
[some ideas] on
+how to make DataFusion an ideal selection for university database projects
such as the
+[CMU database classes] have been put forward.
+
+[Google Summer of Code]: https://summerofcode.withgoogle.com/
+[number of ideas]: https://github.com/apache/datafusion/issues/14478
+[some ideas]: https://github.com/apache/datafusion/issues/14373
+[CMU database classes]: https://15445.courses.cs.cmu.edu/spring2025/
+
+In addition, DataFusion has been appearing publicly more and more, both online
and offline. Here are some highlights:
+
+1. A [demonstration of how uwheel] is integrated into DataFusion
+2. Integrating StringView into DataFusion - [part 1] and [part 2]
+3. [Building streams] with DataFusion
+4. [Caching in DataFusion]: Don't read twice
+5. [Parquet pruning in DataFusion]: Read no more than you need
+6. DataFusion is one of [The 10 coolest open source software tools]
+7. [Building databases over a weekend]
+
+[demonstration of how uwheel]: https://uwheel.rs/post/datafusion_uwheel/
+[part 1]:
https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/
+[part 2]:
https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/
+[Building streams]: https://techontherocks.show/3
+[Caching in DataFusion]: https://blog.haoxp.xyz/posts/caching-datafusion
+[Parquet pruning in DataFusion]: https://blog.haoxp.xyz/posts/parquet-to-arrow/
+[The 10 coolest open source software tools]:
https://www.crn.com/news/software/2024/the-10-coolest-open-source-software-tools-of-2024?page=3
+[Building databases over a weekend]:
https://www.denormalized.io/blog/building-databases
+
+## Improved Performance 🚀
+
+DataFusion hit a milestone in its development by becoming [the fastest single
node engine]
+for querying Apache Parquet files in [clickbench] benchmark for the 43.0.0
release. A lot
+of work went into making this happen!
+
+<img
+src="/blog/images/datafusion-45.0.0/performance_over_time.png"
+width="100%"
+class="img-responsive"
+alt="ClickBench performance results over time for DataFusion"
+/>
+
+**Figure 1**: ClickBench performance improved over 33% between DataFusion 33
+(released Nov. 2023) and DataFusion 45 (released Feb. 2025).
+
+The task of [integrating] the new [Arrow StringView] which significantly
improves performance
+for workloads that scan, filter and group by variable length string and binary
data was completed
+and enabled by default in the past 6 months. The improvement is especially
pronounced for Parquet
+files due to [upstream work in the parquet reader]. Kudos to [@XiangpengHong],
[@AriesDevil],
+[@PsiACE], [@Weijun-H], [@a10y], and [@RinChanNOWWW] for driving this project.
+
+[the fastest single node engine]:
https://datafusion.apache.org/blog/2024/11/18/datafusion-fastest-single-node-parquet-clickbench/
+[clickbench]: https://benchmark.clickhouse.com/
+[integrating]: https://github.com/apache/datafusion/issues/10918
+[Arrow StringView]:
https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html
+[multiple variable length columns in the `GROUP BY` clause]:
https://github.com/apache/datafusion/issues/9403
+[upstream work in the parquet reader]:
https://github.com/apache/arrow-rs/issues/5530
+
+## Improved Quality 📋
+
+DataFusion continues to improve overall in quality. In addition to ongoing bug
+fixes, one of the most exciting improvements in the last 6 months was the
addition of the
+[SQLite sqllogictest suite] thanks to [@Omega359]. These tests run over 5
million
+sql statements on every push to the main branch.
+
+Support for [explicitly checking logical plan invariants] was added by
[@wiedld] which
+can help catch implicit changes that might cause problems during upgrades.
+
+[SQLite sqllogictest suite]: https://github.com/apache/datafusion/pull/13936
+[explicitly checking logical plan invariants]:
https://github.com/apache/datafusion/pull/13651
+
+## Improved Documentation 📚
+
+We continue to improve the documentation to make it easier to get started
using DataFusion.
+During the last 6 months two projects were initiated to migrate the function
documentation
+from strictly static markdown files. First, [@Omega359] [created a framework]
to allow function
+documentation to be generated from code and [@jonathanc-n] and others helped
with the migration,
+then [@comphead] lead a project to [create a doc macro] to allow for an even
easier way to write
+function documentation. A special thanks to [@Chen-Yuan-Lai] for migrating
many functions to
+the new syntax.
+
+[created a framework]: https://github.com/apache/datafusion/pull/12668
+[create a doc macro]: https://github.com/apache/datafusion/pull/12822
+
+Additionally, the [examples] were [refactored] and [cleaned up] to improve
their usefulness.
+
+[examples]: https://github.com/apache/datafusion/pull/13877
+[refactored]: https://github.com/apache/datafusion/pull/13905
+[cleaned up]: https://github.com/apache/datafusion/pull/13950
+
+## New Features ✨
+
+There are too many new features in the last 6 months to list them all, but here
+are some highlights:
+
+### Functions
+* Window functions were migrated from BuiltInWindowFunctions to UDFs
([@jcsherin])
+* Aggregate functions were migrated from builtin functions to UDFs
+* As mentioned above function documentation was extracted from the markdown
files
+* Some new functions and sql support were added including '[show functions]',
'[to_local_time]',
+ '[regexp_count]', '[map_extract]', '[array_distance]', '[array_any_value]',
'[greatest]',
+ '[least]', '[arrays_overlap]'
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
