Re: Using Google Cloud Storage for checkpointing

Glad to hear it! On Sat, Jun 30, 2018 at 9:48 AM Rohil Surana wrote: > Thanks Till, > > On trying locally the setup seemed to have worked until I submitted jobs > to it, it started giving me ClassNotFoundExceptions for some other classes. > So I created a new shaded fat jar from the gcs-connecto

Re: Using Google Cloud Storage for checkpointing

Thanks Till, On trying locally the setup seemed to have worked until I submitted jobs to it, it started giving me ClassNotFoundExceptions for some other classes. So I created a new shaded fat jar from the gcs-connector source code, which worked locally and on K8s. Thanks for all the help. Rohil

Re: Using Google Cloud Storage for checkpointing

Hi Rohil, this sounds a little bit strange. If the GoogleHadoopFileSystem jar is on the classpath and the implementation is specified in core-site.xml, then the Hadoop Filesystem should be able to load the GCS filesystem. I just tried it out locally (without K8s though) and it seemed to work. Cou

Using Google Cloud Storage for checkpointing

Hi I was trying to setup checkpointing on Google Cloud Storage with Flink on Kubernetes, but was facing issues with Google Cloud Storage Connector classes not loading, even though in the logs I can see it being included in the classpath. Logs showing classpath - *https://pastebin.com/R1P7Eepz