[
https://issues.apache.org/jira/browse/FLINK-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15636303#comment-15636303
]
ASF GitHub Bot commented on FLINK-4946:
---------------------------------------
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/2708#discussion_r86542605
--- Diff: flink-dist/src/main/flink-bin/yarn-bin/yarn-session.sh ---
@@ -35,14 +33,15 @@ JVM_ARGS="$JVM_ARGS -Xmx512m"
# Flink CLI client
constructCLIClientClassPath() {
- for jarfile in $FLINK_LIB_DIR/*.jar ; do
- if [[ $CC_CLASSPATH = "" ]]; then
- CC_CLASSPATH=$jarfile;
- else
- CC_CLASSPATH=$CC_CLASSPATH:$jarfile
- fi
- done
- echo $CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS
+ while read -d '' -r jarfile ; do
+ if [[ $FLINK_CLASSPATH = "" ]]; then
+ CC_CLASSPATH="$jarfile";
+ else
+ CC_CLASSPATH="$CC_CLASSPATH":"$jarfile"
+ fi
+ done < <(find "$FLINK_LIB_DIR" -name '*.jar' -print0)
+
+ echo $CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS
--- End diff --
I think this is not enough for the YARN side of things. We need to update
the logic in `AbstractYarnClusterDescriptor` (and maybe `FlinkYarnSessionCli`
as well).
In `AbstractYarnClusterDescriptor` there is `addLibFolderToShipFiles` which
adds the lib directory to the list of ship files. These are uploaded and the
classpath of the containers has it as `*` (see line 620 of the descriptor).
We need to update this to upload the sub directories as well and add these
to the class path. Maybe it's best to just iterate over the lib folder
recursively and add each JAR file. @mxm What do you think?
> Load jar files from subdirectories of lib
> -----------------------------------------
>
> Key: FLINK-4946
> URL: https://issues.apache.org/jira/browse/FLINK-4946
> Project: Flink
> Issue Type: Improvement
> Components: Startup Shell Scripts
> Affects Versions: 1.2.0
> Reporter: Greg Hogan
> Assignee: Greg Hogan
> Priority: Minor
> Fix For: 1.2.0
>
>
> Users can more easily track Flink jars with transitive dependencies when
> copied into subdirectories of {{lib}}. This is the arrangement of {{opt}} for
> FLINK-4861.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
