Hi, the solution is the following:
wget https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-kafka_2.12/1.14.4/flink-sql-connector-kafka_2.12-1.14.4.jar -P lib/ wget https://repo1.maven.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/1.14.4/flink-sql-avro-confluent-registry-1.14.4.jar -P lib/ instead of manually specifying some jars where transitive dependencies need to be added additionally. Flink seems to publish flink-sql-** jars which are fat jars including any required transitive dependencies. Perhaps this part of the documentation should/could be improved on the website so this is clearer for new Flink users. Best, Georg Am Do., 24. März 2022 um 07:27 Uhr schrieb Biao Geng <biaoge...@gmail.com>: > Hi Georg, > > I recently made a demo using flink sql client + schema registry + kafka as > well to test my own schema registry catalog. To help us locate the root > cause, I think you can add "SET 'sql-client.verbose' = 'true';" in your > launched sql client to enable the sql client output more information or go > to check the sql client logs under "/log" dir. "Caused by: > java.lang.ClassNotFoundException: org.apache.avro.SchemaParseException" may > not be the root cause as you have already tried to use -j option to specify > flink-avro jar. > For me, the root cause is that I miss the jar dependencies of jackson > jars. I attach my command for starting sql client here and hope it can give > you some hints: > > sql-client.sh -j jackson-core-*.jar -j jackson-annotations-*.jar -j > jackson-databind-*.jar -j ververica-connector-kafka-*.jar -j > flink-avro-*-jar-with-dependencies.jar > > Note, you may need to substitute ververica-connector-kafka with open > sourced flink-sql-connector-kafka jar. > > For your questions, to my best knowledge, '-j' and '-l' options are the > only options for now. Maybe others in the community can provide more > information. > > Best, > Biao Geng > > > Georg Heiler <georg.kf.hei...@gmail.com> 于2022年3月23日周三 23:59写道： > >> Hi, >> >> When trying to set up a demo for the kafka-sql-client reading an Avro >> topic from Kafka I run into problems with regards to the additional >> dependencies. >> In the spark-shell there is a --packages option which automatically >> resolves any additional required jars (transitively) using the provided >> maven coordinates. So far, I could not find this function for flink. Am >> I missing something? >> >> When now instead of trying to set this up manually I first get the >> additional jars (for flink 1.14.1 scala 2.12) which are mentioned here: >> >> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/kafka/ >> and >> >> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/formats/avro-confluent/ >> >> wget >> https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka_2.12/1.14.4/flink-connector-kafka_2.12-1.14.4.jar >> -P lib/ >> wget >> https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/3.0.0/kafka-clients-3.0.0.jar >> -P lib/ >> wget >> https://repo1.maven.org/maven2/org/apache/flink/flink-avro-confluent-registry/1.14.4/flink-avro-confluent-registry-1.14.4.jar >> -P lib/ >> wget >> https://repo1.maven.org/maven2/org/apache/flink/flink-avro/1.14.4/flink-avro-1.14.4.jar >> -P lib/ >> >> I still fail to get them loaded (even though they are put into the >> default lib path. >> When starting a local cluster: >> >> ./bin/start-cluster.sh local >> >> and the SQL client: >> >> ./bin/sql-client.sh >> >> Any option: >> ./bin/sql-client.sh -j or ./bin/sql-client.sh -l (with the path to the >> lib folder or the additional jars wich were added before) all fails with >> the same reason: >> >> Caused by: java.lang.ClassNotFoundException: >> org.apache.avro.SchemaParseException >> >> when trying to execute: >> CREATE TABLE foo (foo string) WITH ( >> 'connector' = 'kafka', >> 'topic' = 'foo', >> 'scan.startup.mode' = 'earliest-offset', >> 'format' = 'avro-confluent', >> 'avro-confluent.schema-registry.url' = 'http://localhost:8081/', >> 'properties.group.id' = 'flink-test-001', >> 'properties.bootstrap.servers' = 'localhost:9092' >> ); >> SELECT * FROM foo; >> >> (irrespective of any dummy data loaded) Though I have some dummy data in >> the following structure available using the Kafka Connect dummy data >> generator for the following Avro schema (and serialized using Avro into >> Kafkas topic): >> >> { >> "type": "record", >> "name": "commercialrating", >> "fields": [ >> { >> "name": "brand", >> "type": { >> "type": "string", >> "arg.properties": { >> "options": ["Acme", "Globex"] >> } >> } >> }, >> { >> "name": "duration", >> "type": { >> "type": "int", >> "arg.properties": { >> "options": [30, 45, 60] >> } >> } >> }, >> { >> "name": "rating", >> "type": { >> "type": "int", >> "arg.properties": { >> "range": { "min": 1, "max": 5 } >> } >> } >> } >> ] >> } >> >> >> *Questions:* >> >> *1) can I somehow specify maven coordinates directly? (for the naive >> method of using the SQL client like in the spark-shell) to simplify the >> setup of the required jars?* >> >> >> *2) given the fact that I manually have downloaded the jars into the lib >> folder of the flink installation - why are they not loaded by default? What >> needs to change so the additional (required) jars for Avro + >> confluent-schema-registry + Kafka are loaded by the flink SQL client?* >> >> Best, >> Georg >> >> >> >>
