Hi

I was trying to setup checkpointing on Google Cloud Storage with Flink on
Kubernetes, but was facing issues with Google Cloud Storage Connector
classes not loading, even though in the logs I can see it being included in
the classpath.

Logs showing classpath - *https://pastebin.com/R1P7Eepz
<https://pastebin.com/R1P7Eepz>*
Logs showing ClassNotFound Exception for class GoogleHadoopFileSystem -
https://pastebin.com/LGMPzVbp
Hadoop conf core-site.xml - https://pastebin.com/CfEmTk2t

What extra I have done -
1.) Create a new Flink image with Google Cloud Storage connector jar
<https://cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage>
in /etc/flink/lib folder.
2.) Add GCS service account credentials as Kubernetes secret
3.) Mount secret and hadoop, flink ConfigMaps on the taskmanager and
jobmanager deployments.

Flink version - 1.4.2
GCS Hadoop - gcs-connector-latest-hadoop2.jar

Any help is appreciated.
Thank you.

Rohil

Reply via email to