Re: Using Parquet format in Flink hosted in k8s operator

2023-02-22 Thread Frank Lyaruu
Thanks for the help, guys. I can work with that. Maybe it makes sense to add something like that to the parquet doc file: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/formats/parquet/ This documentation does not mention hadoop at all, and it seemed just as strai

Re: Using Parquet format in Flink hosted in k8s operator

2023-02-22 Thread Martijn Visser
Hi Frank, Parquet always requires Hadoop. There is a Parquet ticket to make it possible to read/write Parquet without depending on Hadoop, but that's still open. So in order for Flink to be able to work with Hadoop, it requires the necessary Hadoop dependencies as outlined in https://nightlies.apa

Using Parquet format in Flink hosted in k8s operator

2023-02-09 Thread Frank Lyaruu
Hi all, I’m using the Flink k8s operator to run a SQL stream to/from various connectors, and just added a Parquet format. I customized the image a bit per the example (mostly by adding maven downloads of flink-connector* jars). If I do that for flink-parquet-1.16.1 it fails on missing org/apache/ha