gaborgsomogyi opened a new pull request, #29042:
URL: https://github.com/apache/flink/pull/29042
## What is the purpose of the change
`flink-s3-fs-native`'s shade-plugin configuration uses an explicit
`artifactSet` allowlist of dependency groups to bundle into the plugin's shaded
jar, instead of including everything by default (as `flink-s3-fs-hadoop` does).
This means any dependency that isn't in the allowlist is silently excluded from
the packaged jar at build time, with no compile-time or test-time signal — the
module still compiles and unit tests still pass, because the classes are
present on the unshaded classpath.
This is not just a theoretical risk: `commons-codec`, a transitive runtime
dependency of `org.apache.httpcomponents:httpclient` (itself a dependency of
the bundled AWS SDK Apache HTTP client), is missing from the allowlist today
and is therefore absent from the packaged plugin jar, even though it's present
on the module's compile/runtime classpath. Any code path that ends up needing a
`commons-codec` class at runtime (e.g. certain HTTP client
authentication/encoding paths) would fail with a `NoClassDefFoundError` inside
the plugin's isolated classloader, which cannot fall back to the Flink lib or
user classpath to resolve it. This is hard to catch during review or testing,
since the allowlist and the module's actual dependency tree live in different
parts of the pom with nothing forcing them to stay in sync — every new
(transitive) dependency needs a matching allowlist entry or it silently
disappears from the shaded jar.
This pull request switches `flink-s3-fs-native`'s shade `artifactSet` to
include all dependencies (`*:*`), matching `flink-s3-fs-hadoop`'s existing
approach, so the shaded jar always reflects the module's actual dependency tree
and future dependency changes can't silently regress plugin functionality.
## Brief change log
- Changed the `maven-shade-plugin` `artifactSet` in
`flink-s3-fs-native/pom.xml` from an explicit list of included dependency
groups to `*:*`, so all compile/runtime-scope dependencies are bundled into the
shaded jar
## Verifying this change
This change is already covered by existing tests, such as the module's
existing unit test suite.
- Manually verified by inspecting the built shaded jar before and after
the change: `org/apache/commons/codec/**` (and other transitive dependency
classes not covered by the previous allowlist) were entirely absent from the
packaged jar before this change, despite being present on the module's
compile/runtime classpath; after the change, all of the module's declared
dependencies, including their transitive dependencies, are present in the
shaded jar
- Ran the full `flink-s3-fs-native` unit test suite; all tests pass
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: yes
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Claude code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
