Hi Samrat, Thanks for the review comments. Please find my response below:
1. *yarn.provided.lib.dirs is not merely an unordered collection of jars...Can you elaborate on what archive replacement will look like? * Agree that a flat wildcard cannot replicate flink-dist/plugin/resource-dir handling. Rather than trying to extend archives to cover all of that, I am scoping yarn.provided.lib.archives narrowly as a wildcard classpath mechanism for jars only. Non-jar resources, flink-dist, and plugins remain the responsibility of yarn.provided.lib.dirs. Archives are additive alongside lib.dirs, not a replacement, so this is not a loss of capability. 2. *The JDK launcher specification explicitly states...How will this be handled deterministically? * Since flink-dist stays out of archives entirely, it keeps its current explicit, ordering-guaranteed handling via lib.dirs, unaffected by wildcard ordering. For jars inside an archive, ordering is genuinely unspecified per the JDK wildcard spec, and there is no way to fix that client-side since Flink never sees individual jar names before the NodeManager unpacks the archive. I have documented this as a hard constraint: jars in a provided archive must not conflict with each other or with lib.dirs jars (duplicate classes, service providers, logging bindings) which is the same responsibility users already have today avoiding collisions across multiple lib.dirs entries. 3. *Every archive descriptor must reach both the AM and later TaskManagers...If a local distribution is retained, how are duplicate Flink jars prevented? * flink-dist is unaffected by this proposal and it is never placed inside an archive's wildcard entry. I have added a validation step where submission fails fast if yarn.provided.lib.archives is set with no flink-dist source available anywhere (no lib.dirs, no local distribution). 4. *The localised resource name is part of both the YARN resource map and the classpath...Will there be any silent override?* This is a real gap regardless of scoping as of today, getAllFilesInProvidedLibDirs dedupes by filename across lib.dirs and throws on collision, but there is no equivalent for multiple archives resolving to the same localized resource name. I have added that as an explicit validation failure in the FLIP. 5. *Thinking loud from the perspective of security, archives are executable...Can you shed some light on YARN Private and Public only in the flip?* Archives use the same PUBLIC visibility model as lib.dirs today ( compared to PRIVATE / APPLICATION for regular per-application resources, which are not cached/reused across apps). World readability is documented but not enforced by Flink, consistent with existing behavior. Unpacking is delegated entirely to YARN NodeManager, not done by Flink. I have flagged in the FLIP that deeper YARN-internals security review might be needed and welcomed reviewers with that background to help review. 6. *Can you cover the test plan and alternatives in the flip with more detail? * Expanded the test plan to cover the new validation failures (missing dist source, cross-archive name collision), classpath/wildcard registration checks, and the jars-only exclusion for flink-dist/plugins/non-jar content, plus an integration test submitting a real YARN job with an archive. Also, added two Rejected Alternatives: client-side pre-extraction (defeats NodeManager caching, can reintroduce the long-classpath problem), and extending wildcard support directly to lib.dirs (doesn't address the motivation of shipping a single pre-packaged archive). FLIP is updated with all of the above. Happy to adjust further based on the feedback. Thanks, Archit On Wed, Aug 12, 2026 at 10:56 AM Samrat Deb <[email protected]> wrote: > Hi Archit, > > Thanks for proposing this. Overall +1. Proposal looks feasible. YARN > supports PUBLIC ARCHIVE resources, Flink’s resource descriptor already > preserves visibility and resource type when resources are propagated to > TaskManagers, and the Java launcher expands wildcards in `CLASSPATH`. > > Below are a few queries on the proposal > > 1. `yarn.provided.lib.dirs` is not merely an unordered collection of jars. > The current deployment path treats several resources specially: > - `flink-dist` is identified and propagated separately to dynamically > allocated TaskManagers. > - Plugin jars are deliberately excluded from the system classpath and > remain under the plugin classloader hierarchy. > - Non-jar resource directories can be added to the classpath. > - Local `lib/` and `plugins/` shipping is suppressed when a complete > provided distribution is used. > - User jars remain governed by `yarn.classpath.include-user-jar`. > A flat `<archive>/*` entry does not preserve these semantics. It would load > every top-level jar through the system classloader, exclude loose > resources, and provide no plugin isolation. > Can you elaborate on what archive replacement will look like? > > 2. The JDK launcher specification explicitly states that the order of jars > produced by directory/* is unspecified. This matters because the current > Flink startup scripts sort jars and place flink-dist separately, and the > YARN implementation similarly sorts system classpath entries and appends > flink-dist last. Replacing this with a wildcard can resolve duplicate > classes, service providers, logging bindings, and resources that depend on > the NodeManager, filesystem, or JDK implementation. Including flink-dist in > that wildcard is especially concerning because its current ordering would > be lost. How will this be handled deterministically? > > 3. Every archive descriptor must reach both the AM and later TaskManagers. > The existing TaskManager launch path requires _FLINK_DIST_JAR and > reconstructs other resources from _CLIENT_SHIP_FILES. For archive-only > deployments, which resource satisfies _FLINK_DIST_JAR? If a local > distribution is retained, how are duplicate Flink jars prevented? > > 4. The localised resource name is part of both the YARN resource map and > the classpath. Two paths such as /v1/flink-libs.tgz and /v2/flink-libs.tgz > have the same basename. Will there be any silent override? > > 5. Thinking loud from the perspective of security, archives are executable. > Are there any known gotchas? Can you shed some light on YARN Private and > Public only in the flip? > > 6. Can you cover the test plan and alternatives in the flip with more > detail, if there are any? > > Bests, > Samrat > > On Mon, Aug 10, 2026 at 11:10 PM archit goyal <[email protected]> > wrote: > > > Hi All, > > > > Gentle bump on the thread for FLIP-563 review to support > > *yarn.provided.lib.archives* to point to pre-uploaded HDFS archives. > > > > On Tue, Aug 4, 2026 at 9:28 PM archit goyal <[email protected]> > > wrote: > > > > > Hi All, > > > > > > Gentle bump on FLIP-563 review to support *yarn.provided.lib.archives* > to > > > point to pre-uploaded HDFS archives. > > > > > > Thanks, > > > Archit Goyal > > > > > > On Mon, Feb 2, 2026 at 9:38 AM archit goyal <[email protected]> > > > wrote: > > > > > >> Hi all, > > >> > > >> Gentle ping on FLIP-563 - would appreciate reviews/feedback from the > > >> community. > > >> > > >> On Wed, Jan 21, 2026 at 3:29 PM archit goyal <[email protected]> > > >> wrote: > > >> > > > >> > Hi All, > > >> > > > >> > I am opening this thread to discuss FLIP-563: Support for provided > lib > > >> archives in YARN Application Mode. > > >> > > > >> > Today yarn.provided.lib.dirs assumes pre-uploaded directories of > jars. > > >> In practice, many deployments package shared libs as .tgz/.tar.gz > and/or > > >> hit classpath “argument too long” issues when many jars are > enumerated. > > >> > > > >> > Proposal: introduce yarn.provided.lib.archives to point to > > pre-uploaded > > >> HDFS archives. Flink would localize them as public archive resources > and > > >> add a single wildcard classpath entry per archive (e.g., > > <archiveDir>/*). > > >> Backward compatible and no change unless configured. > > >> > > > >> > Please feel free to reply to this email thread and share your > > opinions. > > >> > > > >> > Thanks, > > >> > Archit Goyal > > >> > > >> > > >> > > >> -- > > >> Yours Sincerely > > >> Archit Goyal > > >> Email: [email protected] > > >> Mobile: +1-347-471-9251 > > >> > > > > > > > > > -- > > > Yours Sincerely > > > Archit Goyal > > > Email: [email protected] > > > > > > > > > -- > > Yours Sincerely > > Archit Goyal > > Email: [email protected] > > > -- Yours Sincerely Archit Goyal Email: [email protected]
