Re: PyFlink SQL from Kafka to Iceberg issues

2023-07-17 Thread Martijn Visser
Hi Dani, Plugins need to be placed in a folder inside the plugins directory, I think that might be the problem. Best regards, Martijn On Sun, Jul 9, 2023 at 7:00 PM Dániel Pálma wrote: > Thanks for the tips Martijn! > > I've fixed the library versions to 1.16 everywhere and also decided to >

Re: PyFlink SQL from Kafka to Iceberg issues

2023-07-09 Thread Dániel Pálma
Thanks for the tips Martijn! I've fixed the library versions to 1.16 everywhere and also decided to scrap pyflink and go for the sql-client instead to keep things simpler for now. This is the Dockerfile I am using for both the *jobmanager* and the *sql-client* FROM flink:1.16.2-scala_2.12-java11

Re: PyFlink SQL from Kafka to Iceberg issues

2023-06-29 Thread Martijn Visser
Hi Dani, There are two things that I notice: 1. You're mixing different Flink versions (1.16 and 1.17): all Flink artifacts should be from the same Flink version 2. S3 plugins need to be added to the plugins folder of Flink, because they are loaded via the plugin mechanism. See https://nightlies.