Re: Flink plugin File System for GCS

2020-06-17 Thread Yang Wang
Hi Alex, Build a fat jar is a good practice for flink filesystem plugin, just like flink-s3-fs-hadoop, flink-s3-fs-presto, flink-azure-fs-hadoop and flink-oss-fs-hadoop. All the provided filesystem plugins are self-contained, which means you need to bundle the hadoop in your fat jar. The reason w

Flink plugin File System for GCS

2020-06-17 Thread Alexander Filipchik
Hello, I'm trying to implement a flink native FS for GCS which can be used with a streaming file sink. I used S3 one as a reference and made it work locally. however, it fails to load when I deploy it to the cluster. If I put hadoop in the fat jar I get: Caused by: java.lang.LinkageError: loader c