I made some more tests and the issue is still not resolved
Since the submitted job main method is executed before the execution graph
is submitted i added the aws sdk as an dependency and used it to upload
files to the bucket in the main method
Once with the default credentials provider, this wor
K8s is self managed on ec2 nodes
After submitting the job and getting an exception I checked:
1.ssh into the machine and verify using the cli the pod has access.
2. In the job main method I instantiate a s3 client from the sdk (once with
default credential chain and once with access key and secret)
Flink version - 1.12.2
Operation type - Session cluster, running on k8s with k8s HA and S3 used for
savepoints, checkpoints and state
We have this version installed for a couple of months now and it was working
perfectly. A couple of weeks ago when submitting any job the job will stay
in "initiali
Solved this.
After trying multiple things i just decided to delete the massive number of
old leader configMaps from my k8s cluster (200+)
After that the time to initialize + schedule is down to a couple of seconds
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com
