Hi Georg, Thanks for the feedback. What made you think that Flink required any additional dependencies except for the ones that are mentioned to download?
Best regards, Martijn On Thu, 24 Mar 2022 at 10:12, Georg Heiler <georg.kf.hei...@gmail.com> wrote: > Hi, > > the solution is the following: > > wget > https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-kafka_2.12/1.14.4/flink-sql-connector-kafka_2.12-1.14.4.jar > -P lib/ > wget > https://repo1.maven.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/1.14.4/flink-sql-avro-confluent-registry-1.14.4.jar > -P lib/ > > instead of manually specifying some jars where transitive dependencies > need to be added additionally. Flink seems to publish flink-sql-** jars > which are fat jars including any required transitive dependencies. > > Perhaps this part of the documentation should/could be improved on the > website so this is clearer for new Flink users. > > Best, > Georg > > Am Do., 24. März 2022 um 07:27 Uhr schrieb Biao Geng <biaoge...@gmail.com > >: > >> Hi Georg, >> >> I recently made a demo using flink sql client + schema registry + kafka >> as well to test my own schema registry catalog. To help us locate the root >> cause, I think you can add "SET 'sql-client.verbose' = 'true';" in >> your launched sql client to enable the sql client output more information >> or go to check the sql client logs under "/log" dir. "Caused by: >> java.lang.ClassNotFoundException: org.apache.avro.SchemaParseException" may >> not be the root cause as you have already tried to use -j option to specify >> flink-avro jar. >> For me, the root cause is that I miss the jar dependencies of jackson >> jars. I attach my command for starting sql client here and hope it can give >> you some hints: >> >> sql-client.sh -j jackson-core-*.jar -j jackson-annotations-*.jar -j >> jackson-databind-*.jar -j ververica-connector-kafka-*.jar -j >> flink-avro-*-jar-with-dependencies.jar >> >> Note, you may need to substitute ververica-connector-kafka with open >> sourced flink-sql-connector-kafka jar. >> >> For your questions, to my best knowledge, '-j' and '-l' options are the >> only options for now. Maybe others in the community can provide more >> information. >> >> Best, >> Biao Geng >> >> >> Georg Heiler <georg.kf.hei...@gmail.com> 于2022年3月23日周三 23:59写道： >> >>> Hi, >>> >>> When trying to set up a demo for the kafka-sql-client reading an Avro >>> topic from Kafka I run into problems with regards to the additional >>> dependencies. >>> In the spark-shell there is a --packages option which automatically >>> resolves any additional required jars (transitively) using the provided >>> maven coordinates. So far, I could not find this function for flink. Am >>> I missing something? >>> >>> When now instead of trying to set this up manually I first get the >>> additional jars (for flink 1.14.1 scala 2.12) which are mentioned here: >>> >>> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/kafka/ >>> and >>> >>> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/formats/avro-confluent/ >>> >>> wget >>> https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka_2.12/1.14.4/flink-connector-kafka_2.12-1.14.4.jar >>> -P lib/ >>> wget >>> https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/3.0.0/kafka-clients-3.0.0.jar >>> -P lib/ >>> wget >>> https://repo1.maven.org/maven2/org/apache/flink/flink-avro-confluent-registry/1.14.4/flink-avro-confluent-registry-1.14.4.jar >>> -P lib/ >>> wget >>> https://repo1.maven.org/maven2/org/apache/flink/flink-avro/1.14.4/flink-avro-1.14.4.jar >>> -P lib/ >>> >>> I still fail to get them loaded (even though they are put into the >>> default lib path. >>> When starting a local cluster: >>> >>> ./bin/start-cluster.sh local >>> >>> and the SQL client: >>> >>> ./bin/sql-client.sh >>> >>> Any option: >>> ./bin/sql-client.sh -j or ./bin/sql-client.sh -l (with the path to the >>> lib folder or the additional jars wich were added before) all fails with >>> the same reason: >>> >>> Caused by: java.lang.ClassNotFoundException: >>> org.apache.avro.SchemaParseException >>> >>> when trying to execute: >>> CREATE TABLE foo (foo string) WITH ( >>> 'connector' = 'kafka', >>> 'topic' = 'foo', >>> 'scan.startup.mode' = 'earliest-offset', >>> 'format' = 'avro-confluent', >>> 'avro-confluent.schema-registry.url' = 'http://localhost:8081/', >>> 'properties.group.id' = 'flink-test-001', >>> 'properties.bootstrap.servers' = 'localhost:9092' >>> ); >>> SELECT * FROM foo; >>> >>> (irrespective of any dummy data loaded) Though I have some dummy data in >>> the following structure available using the Kafka Connect dummy data >>> generator for the following Avro schema (and serialized using Avro into >>> Kafkas topic): >>> >>> { >>> "type": "record", >>> "name": "commercialrating", >>> "fields": [ >>> { >>> "name": "brand", >>> "type": { >>> "type": "string", >>> "arg.properties": { >>> "options": ["Acme", "Globex"] >>> } >>> } >>> }, >>> { >>> "name": "duration", >>> "type": { >>> "type": "int", >>> "arg.properties": { >>> "options": [30, 45, 60] >>> } >>> } >>> }, >>> { >>> "name": "rating", >>> "type": { >>> "type": "int", >>> "arg.properties": { >>> "range": { "min": 1, "max": 5 } >>> } >>> } >>> } >>> ] >>> } >>> >>> >>> *Questions:* >>> >>> *1) can I somehow specify maven coordinates directly? (for the naive >>> method of using the SQL client like in the spark-shell) to simplify the >>> setup of the required jars?* >>> >>> >>> *2) given the fact that I manually have downloaded the jars into the lib >>> folder of the flink installation - why are they not loaded by default? What >>> needs to change so the additional (required) jars for Avro + >>> confluent-schema-registry + Kafka are loaded by the flink SQL client?* >>> >>> Best, >>> Georg >>> >>> >>> >>>
