iamoceans commented on issue #13171:
URL: https://github.com/apache/gravitino/issues/13171#issuecomment-5759801663
I'd like to work on this. I reproduced it on `main` with
`publishToMavenLocal`, and the
cause is in the publication setup rather than in the POM template:
- `spark-connector/v3.5/spark-runtime/build.gradle.kts:32-42` declares the
bundled modules
with `implementation(...)`, and `shadowJar` (lines 44-57) packs
`configurations.runtimeClasspath` into the jar together with the
relocations.
- the `publishing` block (lines 59-65) only sets `artifactId`, so nothing
prunes the
dependency list, and Gradle maps `implementation` to `runtime` scope in
the published POM.
At HEAD the generated `gravitino-spark-connector-runtime-3.5_2.12.pom`
declares the same
three runtime-scoped dependencies, and the jar it publishes already contains
`org/apache/gravitino/client/` (106 entries), `org/apache/gravitino/spark/`
(94),
`org/apache/gravitino/shaded/` (8746) and `org/apache/kyuubi/` (2113), so
the POM asks
consumers to download what the jar already carries.
Two things I found while checking how far this reaches, which affect how
wide the fix should be:
1. It is not only the POM. The `runtimeElements` variant of the Gradle
module metadata
lists the same dependencies, so consumers resolving the coordinate with
Gradle download
them as well. (The shadow plugin already exposes a dependency-free
`shadowRuntimeElements` variant, but it is not the default.)
2. It is not only the connector runtimes. Every shaded runtime artifact I
checked for 1.3.0
has it: spark 3.3/3.4/3.5 in both Scala versions, flink 1.18/1.19/1.20,
`gravitino-client-java-runtime` (1 dependency) and
`gravitino-filesystem-hadoop3-runtime` (7 dependencies).
I have a fix working locally on all five runtime modules in the tree:
declare the bundled
inputs on a configuration that only feeds `shadowJar` instead of
`implementation`. The
published POM and the module metadata then declare no dependencies, and the
packaged
content is unchanged — the entry list is identical and the only differing
entry is the
generated `gravitino-build-info.properties` (build timestamp and commit id),
because the
resolved input set does not change.
Before I open a PR, could you confirm the scope:
- the five runtime modules in the tree (spark 3.5/4.0, flink
1.18/1.19/1.20), or also
`client-java-runtime` and `filesystem-hadoop3-runtime`, which share the
cause but are
outside this issue's title?
- I chose the configuration over pruning the POM in `publishing`, because
that would fix
only the POM and leave the Gradle metadata as it is. Happy to switch to
the smaller diff
if you prefer it.
Could you assign this to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
