>From the latest exception message, it seems that the avro factory problem
has been resolved.
The new exception indicates that you don't have proper Apache Avro
dependencies (because flink-avro doesn't bundle Apache Avro),
so you have to add Apache Avro into your project dependency, or
export HADOOP_CLASSPATH if hadoop is installed in your environment.
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.8.2</version>
</dependency>
Best,
Jark
On Mon, 13 Jul 2020 at 03:04, Lian Jiang <jiangok2...@gmail.com> wrote:
> Thanks guys.
>
> I missed the runtime dependencies. After adding below into
> https://github.com/apache/flink-playgrounds/blob/master/table-walkthrough/Dockerfile.
> The original issue of "Could not find any factory for identifier" is gone.
>
> wget -P /opt/flink/lib/
> https://repo1.maven.org/maven2/org/apache/flink/flink-avro/1.11.0/flink-avro-1.11.0.jar;
> \
> wget -P /opt/flink/lib/
> https://repo1.maven.org/maven2/org/apache/avro/avro/1.8.2/avro-1.8.2.jar; \
> wget -P /opt/flink/lib/
> https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar;
> \
> wget -P /opt/flink/lib/
> https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar;
> \
> wget -P /opt/flink/lib/
> https://repo1.maven.org/maven2/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar;
>
>
> However, I got various NoSuchMethodException related to
> JsonNode/JsonNull/GenricRecord/... The most recent exception is:
>
> java.lang.RuntimeException: java.lang.NoSuchMethodException:
> org.apache.avro.generic.GenericRecord.<init>()
> jobmanager_1 | at
> org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:353)
> ~[avro-1.8.2.jar:1.8.2]
> jobmanager_1 | at
> org.apache.avro.specific.SpecificData.newRecord(SpecificData.java:369)
> ~[avro-1.8.2.jar:1.8.2]
> jobmanager_1 | at
> org.apache.avro.reflect.ReflectData.newRecord(ReflectData.java:901)
> ~[avro-1.8.2.jar:1.8.2]
> jobmanager_1 | at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:212)
> ~[avro-1.8.2.jar:1.8.2]
> jobmanager_1 | at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
> ~[avro-1.8.2.jar:1.8.2]
> jobmanager_1 | at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> ~[avro-1.8.2.jar:1.8.2]
> jobmanager_1 | at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
> ~[avro-1.8.2.jar:1.8.2]
> jobmanager_1 | at
> org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
> ~[avro-1.8.2.jar:1.8.2]
> jobmanager_1 | at
> org.apache.flink.formats.avro.AvroInputFormat.nextRecord(AvroInputFormat.java:165)
> ~[flink-avro-1.11.0.jar:1.11.0]
> jobmanager_1 | at
> org.apache.flink.formats.avro.AvroFileSystemFormatFactory$RowDataAvroInputFormat.nextRecord(AvroFileSystemFormatFactory.java:200)
> ~[flink-avro-1.11.0.jar:1.11.0]
> jobmanager_1 | at
> org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:91)
> ~[flink-dist_2.11-1.11.0.jar:1.11.0]
> jobmanager_1 | at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
> ~[flink-dist_2.11-1.11.0.jar:1.11.0]
> jobmanager_1 | at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
> ~[flink-dist_2.11-1.11.0.jar:1.11.0]
> jobmanager_1 | at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:201)
> ~[flink-dist_2.11-1.11.0.jar:1.11.0]
>
> Is there a uber jar or a list of runtime dependencies so that developers can
> easily make the above example of Flink SQL for avro work? Thanks.
>
>
>
>
>
> On Sat, Jul 11, 2020 at 11:39 PM Leonard Xu <xbjt...@gmail.com> wrote:
>
>> Hi, Jiang
>>
>>
>> jobmanager_1 | Available factory identifiers are:
>> jobmanager_1 |
>> jobmanager_1 | csv
>> jobmanager_1 | json
>> jobmanager_1 | parquet
>>
>>
>> After added the flink-avro dependency, did you restart your
>> cluster/sql-client？ It looks flink-avro dependency did not load properly
>> from the log.
>>
>>
>> Best,
>> Leonard Xu
>>
>
>
> --
>
> Create your own email signature
> <https://www.wisestamp.com/signature-in-email/?utm_source=promotion&utm_medium=signature&utm_campaign=create_your_own&srcid=5234462839406592>
>
