Hi Archit, Thanks for proposing this. Overall +1. Proposal looks feasible. YARN supports PUBLIC ARCHIVE resources, Flink’s resource descriptor already preserves visibility and resource type when resources are propagated to TaskManagers, and the Java launcher expands wildcards in `CLASSPATH`.
Below are a few queries on the proposal 1. `yarn.provided.lib.dirs` is not merely an unordered collection of jars. The current deployment path treats several resources specially: - `flink-dist` is identified and propagated separately to dynamically allocated TaskManagers. - Plugin jars are deliberately excluded from the system classpath and remain under the plugin classloader hierarchy. - Non-jar resource directories can be added to the classpath. - Local `lib/` and `plugins/` shipping is suppressed when a complete provided distribution is used. - User jars remain governed by `yarn.classpath.include-user-jar`. A flat `<archive>/*` entry does not preserve these semantics. It would load every top-level jar through the system classloader, exclude loose resources, and provide no plugin isolation. Can you elaborate on what archive replacement will look like? 2. The JDK launcher specification explicitly states that the order of jars produced by directory/* is unspecified. This matters because the current Flink startup scripts sort jars and place flink-dist separately, and the YARN implementation similarly sorts system classpath entries and appends flink-dist last. Replacing this with a wildcard can resolve duplicate classes, service providers, logging bindings, and resources that depend on the NodeManager, filesystem, or JDK implementation. Including flink-dist in that wildcard is especially concerning because its current ordering would be lost. How will this be handled deterministically? 3. Every archive descriptor must reach both the AM and later TaskManagers. The existing TaskManager launch path requires _FLINK_DIST_JAR and reconstructs other resources from _CLIENT_SHIP_FILES. For archive-only deployments, which resource satisfies _FLINK_DIST_JAR? If a local distribution is retained, how are duplicate Flink jars prevented? 4. The localised resource name is part of both the YARN resource map and the classpath. Two paths such as /v1/flink-libs.tgz and /v2/flink-libs.tgz have the same basename. Will there be any silent override? 5. Thinking loud from the perspective of security, archives are executable. Are there any known gotchas? Can you shed some light on YARN Private and Public only in the flip? 6. Can you cover the test plan and alternatives in the flip with more detail, if there are any? Bests, Samrat On Mon, Aug 10, 2026 at 11:10 PM archit goyal <[email protected]> wrote: > Hi All, > > Gentle bump on the thread for FLIP-563 review to support > *yarn.provided.lib.archives* to point to pre-uploaded HDFS archives. > > On Tue, Aug 4, 2026 at 9:28 PM archit goyal <[email protected]> > wrote: > > > Hi All, > > > > Gentle bump on FLIP-563 review to support *yarn.provided.lib.archives* to > > point to pre-uploaded HDFS archives. > > > > Thanks, > > Archit Goyal > > > > On Mon, Feb 2, 2026 at 9:38 AM archit goyal <[email protected]> > > wrote: > > > >> Hi all, > >> > >> Gentle ping on FLIP-563 - would appreciate reviews/feedback from the > >> community. > >> > >> On Wed, Jan 21, 2026 at 3:29 PM archit goyal <[email protected]> > >> wrote: > >> > > >> > Hi All, > >> > > >> > I am opening this thread to discuss FLIP-563: Support for provided lib > >> archives in YARN Application Mode. > >> > > >> > Today yarn.provided.lib.dirs assumes pre-uploaded directories of jars. > >> In practice, many deployments package shared libs as .tgz/.tar.gz and/or > >> hit classpath “argument too long” issues when many jars are enumerated. > >> > > >> > Proposal: introduce yarn.provided.lib.archives to point to > pre-uploaded > >> HDFS archives. Flink would localize them as public archive resources and > >> add a single wildcard classpath entry per archive (e.g., > <archiveDir>/*). > >> Backward compatible and no change unless configured. > >> > > >> > Please feel free to reply to this email thread and share your > opinions. > >> > > >> > Thanks, > >> > Archit Goyal > >> > >> > >> > >> -- > >> Yours Sincerely > >> Archit Goyal > >> Email: [email protected] > >> Mobile: +1-347-471-9251 > >> > > > > > > -- > > Yours Sincerely > > Archit Goyal > > Email: [email protected] > > > > > -- > Yours Sincerely > Archit Goyal > Email: [email protected] >
