Hi all,
I'd like to bring the discussion around PR #22908[0] to the mailing list
and get broader input on how we should move Variant support forward in
DataFusion.
PR #22908 proposes bringing datafusion-variant into the DataFusion
repository as an extension crate, following a model similar to
datafusion-functions-spark. More recently, Andrew suggested another
option: keeping the crate external and making it easier to integrate
with DataFusion through cookbook examples and documentation.[1]
Either way, I think the goal is the same: make Variant straightforward
to use with DataFusion.
I'd appreciate input on a few areas:
- Integration and maintenance: Should we bring the crate into the main
DataFusion repository, or focus on making the external crate easier to
integrate? If we bring it in, are there concerns about the additional
maintenance commitment?
- SQL API and semantics: Spark compatibility was the original target for
datafusion-variant, with some additional functions influenced by
datafusion-functions-json[2] and DuckDB. Should Spark semantics and
naming be the compatibility baseline? How should we handle useful
functions that do not have Spark equivalents? I also think it would be
useful to distinguish the user-facing SQL API from functions that
primarily exist to enable optimizer rewrites.[3]
- Comet coordination: Comet now has an active native Variant support
epic[4]. There appears to be significant overlap around Variant
semantics, expression kernels, and representation boundaries. I’d like
to coordinate where possible rather than build parallel implementations.
Input from Comet contributors would be especially useful here.
If we decide to bring the crate into the DataFusion repository, I think
the incremental approach discussed on the PR makes sense: start with the
crate scaffolding and a basic JSON-to-Variant function, then add and
review the remaining functionality in smaller PRs. We should also make
sure we document both how to integrate Variant support and how users are
expected to query Variant values.
I’m happy to help move this forward whichever integration approach we
choose. Feedback on the overall direction, use cases we should
prioritize, and offers to help with implementation or review would be
very welcome.
Thanks,
Kosta
[0] https://github.com/apache/datafusion/pull/22908
[1] https://github.com/apache/datafusion/pull/22908#issuecomment-5603803800
[2] https://github.com/datafusion-contrib/datafusion-functions-json
[3] https://github.com/apache/datafusion/pull/22908#issuecomment-5469828372
[4] https://github.com/apache/datafusion-comet/issues/5438
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
