Hi Thomas,
thanks for your feedback.
The error that you are experiencing is definitely a bug in 1.13.3 and
the missing method should be reintroduced in the next patch version to
make code compiled against older patch versions run again.
Regarding the discussion points:
I agree that flink-table-blink uber jar should not contain a dependency
to flink-connector-base. Even filesystem connectors should be optional
and put in a dedicated module that is not in /lib by default.
With having the flink-table-blink uber jar in /lib we would like to
improve the SQL experience as this API is as important as the DataStream
API nowadays. But the depenencies should be minimal nevertheless.
Regards,
Timo
On 23.10.21 00:59, Thomas Weise wrote:
Hi,
As part of upgrading to Flink 1.13.3 from 1.13.2 we run into the
following problem with KafkaSource (Flink distribution is 1.13.2 and
the application was built with 1.13.3):
java.lang.NoSuchMethodError: 'void
org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager.<init>(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue,
java.util.function.Supplier, java.util.function.Consumer)'
at
org.apache.flink.connector.kafka.source.reader.fetcher.KafkaSourceFetcherManager.<init>(KafkaSourceFetcherManager.java:67)
at
org.apache.flink.connector.kafka.source.KafkaSource.createReader(KafkaSource.java:160)
at
org.apache.flink.connector.kafka.source.KafkaSource.createReader(KafkaSource.java:127)
It turns out that flink-table-blink_2.12-1.13.2.jar contains
flink-connector-base and because that jar is under lib the 1.13.2
connector base gets picked up instead of the one bundled in the
application jar.
(The constructor in
org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager
was added in 1.13.3.)
There are a few points I would like to discuss:
1) Version compatibility: A *patch* version should ideally not
introduce such a change, it should be forward and backward compatible.
Hopefully this will be the case after 1.14 with stable source API.
2) flink-table-blink - if it is meant to be self contained and usable
as a library - should not leak its shaded dependencies. It contains
FileSource and other deps from Flink, can those be relocated?
3) Do we need flink-table-blink under lib? Can it be bundled with the
application instead? It would be great if the dependencies under lib
are strictly Flink core.
Thanks,
Thomas
