[
https://issues.apache.org/jira/browse/FLINK-30318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644255#comment-17644255
]
Samrat Deb edited comment on FLINK-30318 at 12/7/22 9:46 AM:
-------------------------------------------------------------
I tested as per
[flink-sql-client-doc|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sqlclient/#sql-client-startup-options]
, `-j ` and `-l` for `flink 1.16` and `master` , this is not working
FLINK-27659 missed to register the Jar Resources passed through CLIOptions into
the ResourceManager (Manager deals with all user defined resources) during
SessionContext creation.
This is registered only during AddJar or Create Function. This looks like a
simple fix to register the jar dependencies during SessionContext creation.
I can submit a patch for this.
was (Author: samrat007):
I tested as per
[flink-sql-client-doc|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sqlclient/#sql-client-startup-options]
, `-j ` and `-l` for `flink 1.16` and `master` , this is not working
FLINK-27659 missed to register the Jar Resources passed through CLIOptions into
the ResourceManager (Manager deals with all user defined resources) during
SessionContext creation.
This is registered only during AddJar or Create Function. This looks needs a
simple fix to register the jar dependencies during SessionContext creation.
I can submit a patch for this.
> sql-client failed to load jar passed with -j args in flink 1.16.0
> ------------------------------------------------------------------
>
> Key: FLINK-30318
> URL: https://issues.apache.org/jira/browse/FLINK-30318
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
> Affects Versions: 1.16.0
> Reporter: Samrat Deb
> Priority: Major
>
> In Flink 1.16.0 , /usr/lib/flink/bin/sql-client.sh embedded -j
> hudi-flink-bundle_2.12-0.10.1.jar failed to load jar passed through argument
>
> {code:java}
> /usr/lib/flink/bin/sql-client.sh embedded -j
> hudi-flink-bundle_2.12-0.10.1.jar
> Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set.
> Using the result of 'hadoop classpath' to augment the Hadoop classpath:
> /etc/hadoop/conf:/usr/lib/hadoop/lib/:/usr/lib/hadoop/.//:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/:/usr/lib/hadoop-hdfs/.//:/usr/lib/hadoop-mapreduce/.//:/usr/lib/hadoop-yarn/lib/:/usr/lib/hadoop-yarn/.//:/etc/tez/conf:/usr/lib/tez/hadoop-shim-0.10.2-amzn-0.jar:/usr/lib/tez/hadoop-shim-2.8-0.10.2-amzn-0.jar:/usr/lib/tez/lib:/usr/lib/tez/LICENSE:/usr/lib/tez/LICENSE-BSD-3clause:/usr/lib/tez/LICENSE-CDDLv1.1-GPLv2_withCPE:/usr/lib/tez/LICENSE-MIT:/usr/lib/tez/LICENSE-SIL_OpenFontLicense-v1.1:/usr/lib/tez/NOTICE:/usr/lib/tez/tez-api-0.10.2-amzn-0.jar:/usr/lib/tez/tez-aux-services-0.10.2-amzn-0.jar:/usr/lib/tez/tez-build-tools-0.10.2-amzn-0.jar:/usr/lib/tez/tez-common-0.10.2-amzn-0.jar:/usr/lib/tez/tez-dag-0.10.2-amzn-0.jar:/usr/lib/tez/tez-examples-0.10.2-amzn-0.jar:/usr/lib/tez/tez-history-parser-0.10.2-amzn-0.jar:/usr/lib/tez/tez-javadoc-tools-0.10.2-amzn-0.jar:/usr/lib/tez/tez-job-analyzer-0.10.2-amzn-0.jar:/usr/lib/tez/tez-mapreduce-0.10.2-amzn-0.jar:/usr/lib/tez/tez-protobuf-history-plugin-0.10.2-amzn-0.jar:/usr/lib/tez/tez-runtime-internals-0.10.2-amzn-0.jar:/usr/lib/tez/tez-runtime-library-0.10.2-amzn-0.jar:/usr/lib/tez/tez-tests-0.10.2-amzn-0.jar:/usr/lib/tez/tez-ui-0.10.2-amzn-0.war:/usr/lib/tez/tez-yarn-timeline-cache-plugin-0.10.2-amzn-0.jar:/usr/lib/tez/tez-yarn-timeline-history-0.10.2-amzn-0.jar:/usr/lib/tez/tez-yarn-timeline-history-with-acls-0.10.2-amzn-0.jar:/usr/lib/tez/tez-yarn-timeline-history-with-fs-0.10.2-amzn-0.jar:/usr/lib/tez/lib/async-http-client-2.12.3.jar:/usr/lib/tez/lib/commons-cli-1.2.jar:/usr/lib/tez/lib/commons-codec-1.11.jar:/usr/lib/tez/lib/commons-collections4-4.1.jar:/usr/lib/tez/lib/commons-io-2.8.0.jar:/usr/lib/tez/lib/commons-lang-2.6.jar:/usr/lib/tez/lib/guava-31.1-jre.jar:/usr/lib/tez/lib/hadoop-annotations.jar:/usr/lib/tez/lib/hadoop-auth.jar:/usr/lib/tez/lib/hadoop-hdfs-client-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-common-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-common.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-core-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-yarn-server-timeline-pluginstorage-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-yarn-server-web-proxy.jar:/usr/lib/tez/lib/hhadoop-mapreduce-client-core.jar:/usr/lib/tez/lib/javax.servlet-api-3.1.0.jar:/usr/lib/tez/lib/jersey-client-1.19.jar:/usr/lib/tez/lib/jersey-json-1.19.jar:/usr/lib/tez/lib/jettison-1.3.4.jar:/usr/lib/tez/lib/jsr305-3.0.0.jar:/usr/lib/tez/lib/metrics-core-3.1.0.jar:/usr/lib/tez/lib/netty-all-4.1.72.Final.jar:/usr/lib/tez/lib/protobuf-java-2.5.0.jar:/usr/lib/tez/lib/RoaringBitmap-0.7.45.jar:/usr/lib/tez/lib/slf4j-api-1.7.36.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/native:/usr/share/aws/aws-java-sdk/aws-java-sdk-bundle-1.12.331.jar:/usr/share/aws/aws-java-sdk/LICENSE.txt:/usr/share/aws/aws-java-sdk/NOTICE.txt:/usr/share/aws/aws-java-sdk/README.md:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/animal-sniffer-annotations-1.14.jar:/usr/share/aws/emr/emrfs/lib/annotations-16.0.2.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/bcprov-ext-jdk15on-1.66.jar:/usr/share/aws/emr/emrfs/lib/checker-qual-2.5.2.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.54.0.jar:/usr/share/aws/emr/emrfs/lib/error_prone_annotations-2.1.3.jar:/usr/share/aws/emr/emrfs/lib/findbugs-annotations-3.0.1.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/j2objc-annotations-1.1.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.12.331.jar:/usr/share/aws/emr/emrfs/lib/jsr305-3.0.2.jar:/usr/share/aws/emr/emrfs/auxlib/:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-2.3.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/lib/flink/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/lib/hadoop/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> 2022-12-07 03:17:05,554 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Found Yarn properties file under
> /var/lib/flink/yarn/.yarn-properties-hadoop.
> 2022-12-07 03:17:05,554 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Found Yarn properties file under
> /var/lib/flink/yarn/.yarn-properties-hadoop.
> ????????
> ????????????????
> ??????? ??????? ?
> ???? ????????? ?????
> ??? ??????? ?????
> ??? ??? ?????
> ?? ???????????????
> ?? ? ??? ?????? ?????
> ????? ???? ????? ?????
> ??????? ??? ??????? ???
> ????????? ?? ?? ??????????
> ???????? ?? ? ?? ???????
> ???? ??? ? ?? ???????? ?????
> ???? ? ?? ? ?? ???????? ???? ??
> ???? ???? ?????????? ??? ?? ????
> ???? ?? ??? ??????????? ???? ? ? ???
> ??? ?? ??? ????????? ???? ???
> ?? ? ??????? ???????? ??? ??
> ??? ??? ???????????????????? ???? ?
> ????? ??? ?????? ???????? ???? ??
> ???????? ??????????????? ??
> ?? ???? ??????? ??? ?????? ?? ???
> ??? ??? ??? ??????? ???? ?????????????
> ??? ????? ???? ?? ?? ???? ???
> ?? ??? ? ?? ?? ??
> ?? ?? ?? ?? ????????
> ?? ????? ?? ??????????? ??
> ?? ???? ? ??????? ??
> ??? ????? ?? ???????????
> ???? ???? ??????? ????????
> ????? ?? ???? ?????
> ????????????????????????????????? ?????
> ______ _ _ _ _____ ____ _ _____ _ _ _
> BETA
> | ___| () | | / ___|/ __ | | / ____| () | |
> | |__ | |_ _ __ | | __ | (___ | | | | | | | | |_ ___ _ __ | |_
> | _| | | | ' | |/ / ___ | | | | | | | | | |/ _ \ '_ | __|
> | | | | | | | | < ___) | || | |___ | |___| | | / | | | |
> || |||| |||_\ |____/ ______| ___|||_|| ||_|
> Welcome! Enter 'HELP;' to list all available commands. 'QUIT;' to
> exit.
> Command history file path: /home/hadoop/.flink-sql-history
> Flink SQL> show jars;
> Empty set
> {code}
> This works only when keeping the hudi-flink bundle jar in /flink/lib/ path
>
> But this works with flink 1.15.2
> {code:java}
> /usr/lib/flink/bin/sql-client.sh embedded -j hudi-flink-bundle_2.12-0.10.1.jar
> Setting HBASE_CONF_DIR=/etc/hbase/conf because no HBASE_CONF_DIR was set.
> Using the result of 'hadoop classpath' to augment the Hadoop classpath:
> /etc/hadoop/conf:/usr/lib/hadoop/lib/:/usr/lib/hadoop/.//:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/:/usr/lib/hadoop-hdfs/.//:/usr/lib/hadoop-mapreduce/.//:/usr/lib/hadoop-yarn/lib/:/usr/lib/hadoop-yarn/.//:/etc/tez/conf:/usr/lib/tez/hadoop-shim-0.10.2-amzn-0.jar:/usr/lib/tez/hadoop-shim-2.8-0.10.2-amzn-0.jar:/usr/lib/tez/lib:/usr/lib/tez/LICENSE:/usr/lib/tez/LICENSE-BSD-3clause:/usr/lib/tez/LICENSE-CDDLv1.1-GPLv2_withCPE:/usr/lib/tez/LICENSE-MIT:/usr/lib/tez/LICENSE-SIL_OpenFontLicense-v1.1:/usr/lib/tez/NOTICE:/usr/lib/tez/tez-api-0.10.2-amzn-0.jar:/usr/lib/tez/tez-aux-services-0.10.2-amzn-0.jar:/usr/lib/tez/tez-build-tools-0.10.2-amzn-0.jar:/usr/lib/tez/tez-common-0.10.2-amzn-0.jar:/usr/lib/tez/tez-dag-0.10.2-amzn-0.jar:/usr/lib/tez/tez-examples-0.10.2-amzn-0.jar:/usr/lib/tez/tez-history-parser-0.10.2-amzn-0.jar:/usr/lib/tez/tez-javadoc-tools-0.10.2-amzn-0.jar:/usr/lib/tez/tez-job-analyzer-0.10.2-amzn-0.jar:/usr/lib/tez/tez-mapreduce-0.10.2-amzn-0.jar:/usr/lib/tez/tez-protobuf-history-plugin-0.10.2-amzn-0.jar:/usr/lib/tez/tez-runtime-internals-0.10.2-amzn-0.jar:/usr/lib/tez/tez-runtime-library-0.10.2-amzn-0.jar:/usr/lib/tez/tez-tests-0.10.2-amzn-0.jar:/usr/lib/tez/tez-ui-0.10.2-amzn-0.war:/usr/lib/tez/tez-yarn-timeline-cache-plugin-0.10.2-amzn-0.jar:/usr/lib/tez/tez-yarn-timeline-history-0.10.2-amzn-0.jar:/usr/lib/tez/tez-yarn-timeline-history-with-acls-0.10.2-amzn-0.jar:/usr/lib/tez/tez-yarn-timeline-history-with-fs-0.10.2-amzn-0.jar:/usr/lib/tez/lib/async-http-client-2.12.3.jar:/usr/lib/tez/lib/commons-cli-1.2.jar:/usr/lib/tez/lib/commons-codec-1.11.jar:/usr/lib/tez/lib/commons-collections4-4.1.jar:/usr/lib/tez/lib/commons-io-2.8.0.jar:/usr/lib/tez/lib/commons-lang-2.6.jar:/usr/lib/tez/lib/guava-31.1-jre.jar:/usr/lib/tez/lib/hadoop-annotations.jar:/usr/lib/tez/lib/hadoop-auth.jar:/usr/lib/tez/lib/hadoop-hdfs-client-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-common-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-common.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-core-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-yarn-server-timeline-pluginstorage-3.3.3-amzn-0.jar:/usr/lib/tez/lib/hadoop-yarn-server-web-proxy.jar:/usr/lib/tez/lib/hhadoop-mapreduce-client-core.jar:/usr/lib/tez/lib/javax.servlet-api-3.1.0.jar:/usr/lib/tez/lib/jersey-client-1.19.jar:/usr/lib/tez/lib/jersey-json-1.19.jar:/usr/lib/tez/lib/jettison-1.3.4.jar:/usr/lib/tez/lib/jsr305-3.0.0.jar:/usr/lib/tez/lib/metrics-core-3.1.0.jar:/usr/lib/tez/lib/netty-all-4.1.72.Final.jar:/usr/lib/tez/lib/protobuf-java-2.5.0.jar:/usr/lib/tez/lib/RoaringBitmap-0.7.45.jar:/usr/lib/tez/lib/slf4j-api-1.7.36.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/native:/usr/share/aws/aws-java-sdk/aws-java-sdk-bundle-1.12.331.jar:/usr/share/aws/aws-java-sdk/LICENSE.txt:/usr/share/aws/aws-java-sdk/NOTICE.txt:/usr/share/aws/aws-java-sdk/README.md:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/animal-sniffer-annotations-1.14.jar:/usr/share/aws/emr/emrfs/lib/annotations-16.0.2.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/bcprov-ext-jdk15on-1.66.jar:/usr/share/aws/emr/emrfs/lib/checker-qual-2.5.2.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.54.0.jar:/usr/share/aws/emr/emrfs/lib/error_prone_annotations-2.1.3.jar:/usr/share/aws/emr/emrfs/lib/findbugs-annotations-3.0.1.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/j2objc-annotations-1.1.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.12.331.jar:/usr/share/aws/emr/emrfs/lib/jsr305-3.0.2.jar:/usr/share/aws/emr/emrfs/auxlib/:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-2.3.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/lib/flink/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/lib/hadoop/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> 2022-12-07 03:19:37,344 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Found Yarn properties file under
> /var/lib/flink/yarn/.yarn-properties-hadoop.
> 2022-12-07 03:19:37,344 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
> [] - Found Yarn properties file under
> /var/lib/flink/yarn/.yarn-properties-hadoop.
> ????????
> ????????????????
> ??????? ??????? ?
> ???? ????????? ?????
> ??? ??????? ?????
> ??? ??? ?????
> ?? ???????????????
> ?? ? ??? ?????? ?????
> ????? ???? ????? ?????
> ??????? ??? ??????? ???
> ????????? ?? ?? ??????????
> ???????? ?? ? ?? ???????
> ???? ??? ? ?? ???????? ?????
> ???? ? ?? ? ?? ???????? ???? ??
> ???? ???? ?????????? ??? ?? ????
> ???? ?? ??? ??????????? ???? ? ? ???
> ??? ?? ??? ????????? ???? ???
> ?? ? ??????? ???????? ??? ??
> ??? ??? ???????????????????? ???? ?
> ????? ??? ?????? ???????? ???? ??
> ???????? ??????????????? ??
> ?? ???? ??????? ??? ?????? ?? ???
> ??? ??? ??? ??????? ???? ?????????????
> ??? ????? ???? ?? ?? ???? ???
> ?? ??? ? ?? ?? ??
> ?? ?? ?? ?? ????????
> ?? ????? ?? ??????????? ??
> ?? ???? ? ??????? ??
> ??? ????? ?? ???????????
> ???? ???? ??????? ????????
> ????? ?? ???? ?????
> ????????????????????????????????? ?????
> ______ _ _ _ _____ ____ _ _____ _ _ _
> BETA
> | ___| () | | / ___|/ __ | | / ____| () | |
> | |__ | |_ _ __ | | __ | (___ | | | | | | | | |_ ___ _ __ | |_
> | _| | | | ' | |/ / ___ | | | | | | | | | |/ _ \ '_ | __|
> | | | | | | | | < ___) | || | |___ | |___| | | / | | | |
> || |||| |||_\ |____/ ______| ___|||_|| ||_|
> Welcome! Enter 'HELP;' to list all available commands. 'QUIT;' to
> exit.
> Command history file path: /home/hadoop/.flink-sql-history
> Flink SQL> show jars;
> /usr/lib/flink/opt/flink-sql-client-1.15.2.jar
> /usr/lib/flink/opt/flink-python_2.12-1.15.2.jar
> /home/hadoop/hudi-flink-bundle_2.12-0.10.1.jar
> {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
