wirybeaver commented on issue #1241:
URL:
https://github.com/apache/datafusion-ballista/issues/1241#issuecomment-5648257640
@milenkovicm, thank you for clarifying the evaluation criteria. I fully
understand that the project has no obligation to merge this work, and I did not
intend to imply otherwise. My goal is to make the proposed boundary concrete
enough for maintainers to decide whether the convention is useful.
There is one factual distinction that I did not explain clearly enough:
**#2388 does not depend on hudi-rs, my fork, or any Hudi SHA.** The unreleased
hudi-rs pin exists only in the Draft follow-up #2389, inside a separate crate
excluded from Ballista's workspace and root lockfile. #2389 should not be
considered for merge until hudi-rs has an official compatible release.
```text
Ballista release boundary
+----------------------------------------------------------+
| Ballista main crates |
| |
| default build: no open-table-format SDK dependencies |
| optional gates: wire contracts + small provider SPIs |
+-----------------------------+----------------------------+
|
explicit, format-specific integration
|
+----------------------+----------------------+
| | |
v v v
+---------------+ +---------------+ +---------------+
| Iceberg crate | | Delta crate | | Lance crate |
| own SDK/deps | | own SDK/deps | | own SDK/deps |
+---------------+ +---------------+ +---------------+
|
v
+---------------+
| Hudi crate |
| own SDK/deps |
+---------------+
Each format crate: under the Apache project umbrella,
but outside Ballista's main crate, root lockfile, and release graph.
```
So #2388/#2389 are intended to evaluate a reusable integration policy, not
merely to add Hudi. The build gate is meant to prevent a third-party
table-format SDK from affecting Ballista's default build and main-crate
release. I used hudi-rs as the first proof because Iceberg and Delta Lake work
is already underway, but those implementations do not currently follow this
proposed crate and release isolation convention. If maintainers do not mind, I
can use Lance as a second example to demonstrate that the boundary is
format-independent in practice.
@andygrove, thank you as well for explaining the maintainer-bandwidth
constraint. I appreciate that this work is largely supported by maintainers'
personal time. The separate-crate/build-gate proposal is specifically trying to
minimize, not dismiss, that burden. I understand that dependency isolation does
not eliminate review and ownership cost, and I am happy to adjust, defer, or
keep the implementations outside Ballista if the maintainers decide that even
the small gate is not worth that cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
