andygrove opened a new pull request, #6783:
URL: https://github.com/apache/datafusion-comet/pull/6783
## Which issue does this PR close?
No issue. This follows up on #6778, which added the FAQ page with a single
entry.
## Rationale for this change
Some questions come up repeatedly in GitHub Discussions and in issues users
file, and the docs answer them only in passing or not at all. Examples:
- `ClassNotFoundException` for `CometShuffleManager` (#864, 13 comments)
- vendor Spark builds failing with `NoSuchMethodError` (#414, #1576)
- no speedup on TPC-H or TPC-DS (#1450, #588)
- the published jars not loading on older glibc (#2504, #1499)
- how Comet compares with Gluten and other accelerators (discussion #994)
The Gluten comparison page had also gone stale. Some of its claims about
Gluten no longer match Gluten's own docs and code.
## What changes are included in this PR?
**FAQ (`about/faq.md`).** Fourteen new entries in four sections. The
existing vectorized Java/Scala UDF entry moves into Feature Support unchanged.
- Choosing Comet:
- how Comet compares to other open-source Spark accelerators (Gluten,
Auron, the NVIDIA cuDF plugin)
- whether results match Spark
- supported Spark and Java versions
- vendor distributions of Spark
- platforms the published jars support (Linux amd64/arm64, glibc 2.31 or
newer, CPU baselines)
- Running Comet:
- the `CometShuffleManager` classpath error
- memory
- why a query isn't faster
- Feature Support:
- Delta Lake, Hudi and Paimon
- PySpark and Python UDFs
- Structured Streaming
- remote shuffle services
- Project and Community:
- the roadmap
- where to ask questions and report bugs
**Gluten comparison (`about/gluten_comparison.md`).** These claims were
checked against Gluten's v1.7.0 docs and code:
- Partial projection: Gluten has had this on by default since 1.3.0. It runs
only the unsupported expressions in a projection, including Scala and Hive
UDFs, in Spark. The codegen dispatch and UDF sections said Gluten had no
equivalent and fell back. They now describe partial projection and note that
Comet doesn't cover Hive UDFs.
- Regular expressions: Gluten falls back for patterns RE2 can't compile and
has a setting that sends all regex functions to Spark. The section no longer
says it "cannot fully match Spark's regex semantics regardless of
configuration".
- ANSI: mentions `spark.gluten.sql.ansiFallback.enabled` and Gluten's note
that results in that mode don't yet match Spark.
- Iceberg:
- Comet: now spec v3, deletion vectors, encrypted v3 tables, Iceberg 1.11
and experimental native writes. The page still said writes go through Spark.
- Gluten: what its Iceberg guide documents (unpartitioned reads offloaded,
partitioned tables mostly fall back, equality deletes and writes fall back).
- Versions:
- Delta: 2.3 through 4.1 depending on the Spark version, not 2.0 through
4.0.
- Spark and backends: pinned to Gluten 1.7.0, with Bolt mentioned as
unreleased.
- The "timestamp encodings" note is dropped, since Gluten removed it from
its limitations page.
- Performance: adds AWS Labs' separate Comet 1.0.0 (1.57×) and Gluten 1.6.0
(1.63×) runs, which use the same Spark baseline. The Comet 0.16.0 vs Gluten
1.6.0 head-to-head is still AWS Labs' latest comparison of the two.
**About index.** Updated the descriptions of both pages.
## How are these changes tested?
Documentation only.
- `prettier --check` passes on the changed files.
- A script checked that every relative link and `#anchor` resolves to an
existing page and heading.
- The github.io and apache.org links couldn't be fetched from my machine.
The two new AWS Labs URLs follow the routing that the AWS Labs site's own index
uses.
- The docs site builds only on pushes to main, so the pages haven't been
rendered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
