andygrove commented on code in PR #210:
URL: https://github.com/apache/datafusion-site/pull/210#discussion_r4447656857
##########
content/blog/2026-11-03-datafusion-comet-1.2.0.md:
##########
@@ -0,0 +1,525 @@
+---
+layout: post
+title: Apache DataFusion Comet 1.2.0 Release
+date: 2026-11-03
+author: pmc
+categories: [subprojects]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+The Apache DataFusion PMC is pleased to announce version 1.2.0 of the
[Comet](https://datafusion.apache.org/comet/) subproject.
+
+Comet is an accelerator for Apache Spark that translates Spark physical plans
to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.
+
+This release covers roughly five weeks of development since 1.1.0: 399 commits
from 40 contributors. See the
+[change log] for the full list of changes.
Review Comment:
Applied, thanks. The intro now says where the figures come from. I dropped
the per-bullet attribution from the shuffle, broadcast, Python UDF, and scan
bullets, and kept it on the spill compression bullet, which cites the nested
TPC-H investigation.
##########
content/blog/2026-11-03-datafusion-comet-1.2.0.md:
##########
@@ -0,0 +1,525 @@
+---
+layout: post
+title: Apache DataFusion Comet 1.2.0 Release
+date: 2026-11-03
+author: pmc
+categories: [subprojects]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+The Apache DataFusion PMC is pleased to announce version 1.2.0 of the
[Comet](https://datafusion.apache.org/comet/) subproject.
+
+Comet is an accelerator for Apache Spark that translates Spark physical plans
to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.
+
+This release covers roughly five weeks of development since 1.1.0: 399 commits
from 40 contributors. See the
+[change log] for the full list of changes.
+
+[change log]:
https://github.com/apache/datafusion-comet/blob/branch-1.2/docs/source/changelog/1.2.0.md
+
+Comet has always aimed to return the same results as Spark. Much of this cycle
went into the edge cases that
+remained, working through whole categories at a time (timezones,
floating-point `-0.0` and `NaN`, collations)
+and adding tests that keep them fixed.
+
+The highlights:
+
+- **Results that match Spark**: native timestamp results carry the right
timezone label. `-0.0`, `0.0`, and
+ `NaN` follow Spark in hashing, comparison, sorting, aggregation, and array
functions. Decimal `AVG`, nested
+ types, broadcasts, and join order get fixes too.
Review Comment:
Applied, thanks.
##########
content/blog/2026-11-03-datafusion-comet-1.2.0.md:
##########
@@ -0,0 +1,525 @@
+---
+layout: post
+title: Apache DataFusion Comet 1.2.0 Release
+date: 2026-11-03
+author: pmc
+categories: [subprojects]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+The Apache DataFusion PMC is pleased to announce version 1.2.0 of the
[Comet](https://datafusion.apache.org/comet/) subproject.
+
+Comet is an accelerator for Apache Spark that translates Spark physical plans
to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.
+
+This release covers roughly five weeks of development since 1.1.0: 399 commits
from 40 contributors. See the
+[change log] for the full list of changes.
+
+[change log]:
https://github.com/apache/datafusion-comet/blob/branch-1.2/docs/source/changelog/1.2.0.md
+
+Comet has always aimed to return the same results as Spark. Much of this cycle
went into the edge cases that
+remained, working through whole categories at a time (timezones,
floating-point `-0.0` and `NaN`, collations)
+and adding tests that keep them fixed.
+
+The highlights:
+
+- **Results that match Spark**: native timestamp results carry the right
timezone label. `-0.0`, `0.0`, and
+ `NaN` follow Spark in hashing, comparison, sorting, aggregation, and array
functions. Decimal `AVG`, nested
+ types, broadcasts, and join order get fixes too.
+- **Memory, spill, and shuffle at scale**: memory returned to Spark sooner,
fewer memory-related task failures,
+ much smaller JVM shuffle memory use with many partitions, and smaller
shuffle files.
+- **Iceberg**: native writes are a documented, opt-in feature with support for
format-version 3 tables, and the
+ native scan reads fewer nested fields and more files at once.
+- **More of the plan runs natively**: existence joins, `SortAggregateExec`,
the `PIVOT` fast path, and
+ HyperLogLog sketch functions, among others.
+- **New places to run**: a native Delta Lake scan (experimental), Arrow Python
UDFs (experimental), and Spark 4.2
+ as a supported version.
+
+## Results That Match Spark
+
+Timestamps and floating point produced the most reports, so we started there,
followed by collations. Individual
+fixes in other areas come after that.
+
Review Comment:
Agreed, removed.
##########
content/blog/2026-11-03-datafusion-comet-1.2.0.md:
##########
@@ -0,0 +1,525 @@
+---
+layout: post
+title: Apache DataFusion Comet 1.2.0 Release
+date: 2026-11-03
+author: pmc
+categories: [subprojects]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+The Apache DataFusion PMC is pleased to announce version 1.2.0 of the
[Comet](https://datafusion.apache.org/comet/) subproject.
+
+Comet is an accelerator for Apache Spark that translates Spark physical plans
to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.
+
+This release covers roughly five weeks of development since 1.1.0: 399 commits
from 40 contributors. See the
+[change log] for the full list of changes.
+
+[change log]:
https://github.com/apache/datafusion-comet/blob/branch-1.2/docs/source/changelog/1.2.0.md
+
+Comet has always aimed to return the same results as Spark. Much of this cycle
went into the edge cases that
+remained, working through whole categories at a time (timezones,
floating-point `-0.0` and `NaN`, collations)
+and adding tests that keep them fixed.
+
+The highlights:
+
+- **Results that match Spark**: native timestamp results carry the right
timezone label. `-0.0`, `0.0`, and
+ `NaN` follow Spark in hashing, comparison, sorting, aggregation, and array
functions. Decimal `AVG`, nested
+ types, broadcasts, and join order get fixes too.
+- **Memory, spill, and shuffle at scale**: memory returned to Spark sooner,
fewer memory-related task failures,
+ much smaller JVM shuffle memory use with many partitions, and smaller
shuffle files.
+- **Iceberg**: native writes are a documented, opt-in feature with support for
format-version 3 tables, and the
+ native scan reads fewer nested fields and more files at once.
+- **More of the plan runs natively**: existence joins, `SortAggregateExec`,
the `PIVOT` fast path, and
+ HyperLogLog sketch functions, among others.
+- **New places to run**: a native Delta Lake scan (experimental), Arrow Python
UDFs (experimental), and Spark 4.2
+ as a supported version.
+
+## Results That Match Spark
+
+Timestamps and floating point produced the most reports, so we started there,
followed by collations. Individual
+fixes in other areas come after that.
+
+### Timestamps and timezones
+
+Spark labels a timestamp with the session timezone, and Comet's native code
has to produce the same label, not
+only the same instant. Several native expressions got this wrong. 1.2.0 fixes
them:
+
+- Native timestamp results carry the correct timezone label:
`timestamp_seconds` results are labeled as UTC
+ timestamps ([#6341]), `date_trunc` keeps its input's label ([#6345]), and
`CASE` and `COALESCE` relabel
+ timestamp branches instead of panicking ([#6347]).
Review Comment:
Applied, thanks.
##########
content/blog/2026-11-03-datafusion-comet-1.2.0.md:
##########
@@ -0,0 +1,525 @@
+---
+layout: post
+title: Apache DataFusion Comet 1.2.0 Release
+date: 2026-11-03
+author: pmc
+categories: [subprojects]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+The Apache DataFusion PMC is pleased to announce version 1.2.0 of the
[Comet](https://datafusion.apache.org/comet/) subproject.
+
+Comet is an accelerator for Apache Spark that translates Spark physical plans
to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.
+
+This release covers roughly five weeks of development since 1.1.0: 399 commits
from 40 contributors. See the
+[change log] for the full list of changes.
+
+[change log]:
https://github.com/apache/datafusion-comet/blob/branch-1.2/docs/source/changelog/1.2.0.md
+
+Comet has always aimed to return the same results as Spark. Much of this cycle
went into the edge cases that
+remained, working through whole categories at a time (timezones,
floating-point `-0.0` and `NaN`, collations)
+and adding tests that keep them fixed.
+
+The highlights:
+
+- **Results that match Spark**: native timestamp results carry the right
timezone label. `-0.0`, `0.0`, and
+ `NaN` follow Spark in hashing, comparison, sorting, aggregation, and array
functions. Decimal `AVG`, nested
+ types, broadcasts, and join order get fixes too.
+- **Memory, spill, and shuffle at scale**: memory returned to Spark sooner,
fewer memory-related task failures,
+ much smaller JVM shuffle memory use with many partitions, and smaller
shuffle files.
+- **Iceberg**: native writes are a documented, opt-in feature with support for
format-version 3 tables, and the
+ native scan reads fewer nested fields and more files at once.
+- **More of the plan runs natively**: existence joins, `SortAggregateExec`,
the `PIVOT` fast path, and
+ HyperLogLog sketch functions, among others.
+- **New places to run**: a native Delta Lake scan (experimental), Arrow Python
UDFs (experimental), and Spark 4.2
+ as a supported version.
+
+## Results That Match Spark
+
+Timestamps and floating point produced the most reports, so we started there,
followed by collations. Individual
+fixes in other areas come after that.
+
+### Timestamps and timezones
+
+Spark labels a timestamp with the session timezone, and Comet's native code
has to produce the same label, not
+only the same instant. Several native expressions got this wrong. 1.2.0 fixes
them:
+
+- Native timestamp results carry the correct timezone label:
`timestamp_seconds` results are labeled as UTC
+ timestamps ([#6341]), `date_trunc` keeps its input's label ([#6345]), and
`CASE` and `COALESCE` relabel
+ timestamp branches instead of panicking ([#6347]).
+- Session timezone IDs are normalized before they reach native code ([#6351]).
+- `date_trunc` with a format that differs from row to row uses the same kernel
as a literal format ([#6354]).
+- The driver warns when the native and JVM timezone databases differ
([#6353]), because Comet cannot correct
+ for that difference.
+- The test harness now fails a test when a native plan mislabels a timestamp,
so this class of bug can't come
+ back unnoticed ([#6868], [#6930]).
+
+Thanks to [@andygrove] for the timezone sweep and [@LinSimon-901101] for the
label checks. The new
+[timezone handling page] in the contributor guide explains the rules.
+
+[timezone handling page]:
https://datafusion.apache.org/comet/contributor-guide/timezones.html
+
+### Floating point: `-0.0`, `0.0`, and `NaN`
+
+In some operations Spark treats `-0.0` and `0.0` as equal and every `NaN` as
one value; in others it treats them
+as distinct. Which rule applies depends on the operation and the Spark
version, and DataFusion makes different
+choices. The differences are easy to miss in testing and show up as wrong
joins, groups, and sort orders. In
+1.2.0, these follow Spark:
Review Comment:
Applied, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
