Shared Checkpoint Cleanup and S3 Lifecycle Policy

e the _entropy_ trick would help this, because the issue is ultimately that there's a single shared directory. Thank you! Trystan

Re: Shared Checkpoint Cleanup and S3 Lifecycle Policy

ould you suggest to use it? My understanding is that I don't control anything beyond the checkpoint directory, and since shared is in that directory I can't put entropy inside the shared directory itself (which is what I would need). Thanks, Trystan On Wed, May 6, 2020 at 7:31 PM Congxian Qiu

Re: Shared Checkpoint Cleanup and S3 Lifecycle Policy

s). So this shared state lives on forever and there is no way to ever clean it up, at all. I am surprised that this hasn't been a problem for anyone else. Maybe I should just file a feature request for this, at least to find some solution for ways to clean up these directories. I appreciate your

Can BroadcastProcessFunction invoke both methods concurrently?

ize that this is very close to playing with fire, and is exactly why we have a broadcast state where Flink can hide all the danger, so would be open to other ideas if this is cardinal sin! Trystan

RocksDB efficiency and keyby

nd 250 GB. Attempting to switch to heap state immediately ran into OOM (parallelism: 120, 8gb memory each). And perhaps the answer is just "scale out" :) but if there are any signals to know when we've reached the limit of current scale, it'd be great to know what signals to look for! Thanks! Trystan

Re: RocksDB efficiency and keyby

Thanks for the info! We're running EBS gp2 volumes... awhile back we tested local SSDs with a different job and didn't notice any gains, but that was likely due to an under-optimized job where the bottleneck was elsewhere On Wed, Apr 20, 2022, 11:08 AM Yaroslav Tkachenko wrote: >

Avoid dynamic classloading in native mode with Kubernetes Operator

Is it possible to avoid dynamic classloading when using the operator with a native kubernetes application deployment? If I put the job jar into /opt/flinklib, then there are two possible outcomes: 1. If I point jarURI to the jar, I get linkage errors (presumably: the class have already been

Re: Avoid dynamic classloading in native mode with Kubernetes Operator

wo-jar approach. Thanks, Trystan On Fri, Nov 17, 2023 at 12:19 PM Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hi Trystan, > > I imagine you can create 2 jars, one should only have a class with the > main method, and the other should be a fat jar with everything else for &

Re: Avoid dynamic classloading in native mode with Kubernetes Operator

Thanks! Yeah I am not sure why it's handled so different with non-native k8s mode. If it's possible I think this would be a huge improvement. On Mon, Nov 20, 2023, 12:55 PM Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hi Trystan, I'm actually not very fami

Re: Avoid dynamic classloading in native mode with Kubernetes Operator

27, 2023 at 5:50 PM Trystan wrote: > Thanks! Yeah I am not sure why it's handled so different with non-native > k8s mode. > > If it's possible I think this would be a huge improvement. > > On Mon, Nov 20, 2023, 12:55 PM Alexis Sarda-Espinosa < > sarda.espin...