wirybeaver commented on PR #2388:
URL:
https://github.com/apache/datafusion-ballista/pull/2388#issuecomment-5648257437
Thank you for taking another look. I think I caused confusion by using Hudi
as the first example, so I want to clarify the boundary precisely.
**#2388 has no dependency on hudi-rs, my fork, or any Hudi commit SHA.** The
only new dependencies behind `contrib-hudi` are `datafusion-proto` and `prost`,
both already part of the Ballista/DataFusion ecosystem. The SHA shown on the
review is the commit SHA of this PR itself, not a dependency pin.
```text
#2388: Ballista build/release graph
+------------------------------------------------------+
| ballista crate |
| |
| default build |
| - no hudi-rs dependency |
| - no Hudi runtime path |
| |
| optional contrib-hudi build gate |
| - wire descriptor |
| - provider codec interface |
| - still no hudi-rs dependency |
+--------------------------+---------------------------+
|
| implemented externally
| only when explicitly built
v
+------------------------------------------------------+
| Separate format integration crate |
| - owns the format SDK dependency |
| - excluded from Ballista's workspace/release graph |
+------------------------------------------------------+
```
The broader proposal is a convention rather than special treatment for Hudi:
```text
Ballista main crate: stable gates/SPIs, no table-format SDKs
|
+-- Apache Iceberg integration crate -> Iceberg dependencies
+-- Apache Delta integration crate -> Delta dependencies
+-- Apache Lance integration crate -> Lance dependencies
+-- Apache Hudi integration crate -> Hudi dependencies
```
Each format would own a separate crate under the Apache project umbrella and
would not add its SDK to the main Ballista crate, root lockfile, or release
graph. This is intended to limit the release and dependency impact of
third-party open-table-format integrations.
I chose hudi-rs only as a small validation vehicle because Iceberg and Delta
Lake integrations are already being explored, but those efforts do not
currently demonstrate this isolation convention. #2388 itself does not require
accepting #2389. If it would make the design easier to evaluate independently
of Hudi, I would be happy to provide Lance as a second example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
