Re: Avoid dynamic classloading in native mode with Kubernetes Operator

2023-12-15 Thread Trystan
27, 2023 at 5:50 PM Trystan wrote: > Thanks! Yeah I am not sure why it's handled so different with non-native > k8s mode. > > If it's possible I think this would be a huge improvement. > > On Mon, Nov 20, 2023, 12:55 PM Alexis Sarda-Espinosa < > sarda.espin...

Re: Avoid dynamic classloading in native mode with Kubernetes Operator

2023-11-27 Thread Trystan
Thanks! Yeah I am not sure why it's handled so different with non-native k8s mode. If it's possible I think this would be a huge improvement. On Mon, Nov 20, 2023, 12:55 PM Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hi Trystan, I'm actually not very fami

Re: Avoid dynamic classloading in native mode with Kubernetes Operator

2023-11-20 Thread Trystan
wo-jar approach. Thanks, Trystan On Fri, Nov 17, 2023 at 12:19 PM Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hi Trystan, > > I imagine you can create 2 jars, one should only have a class with the > main method, and the other should be a fat jar with everything else for &

Avoid dynamic classloading in native mode with Kubernetes Operator

2023-11-16 Thread Trystan
Is it possible to avoid dynamic classloading when using the operator with a native kubernetes application deployment? If I put the job jar into /opt/flinklib, then there are two possible outcomes: 1. If I point jarURI to the jar, I get linkage errors (presumably: the class have already been

Re: RocksDB efficiency and keyby

2022-04-20 Thread Trystan
Thanks for the info! We're running EBS gp2 volumes... awhile back we tested local SSDs with a different job and didn't notice any gains, but that was likely due to an under-optimized job where the bottleneck was elsewhere On Wed, Apr 20, 2022, 11:08 AM Yaroslav Tkachenko wrote: >

RocksDB efficiency and keyby

2022-04-20 Thread Trystan
nd 250 GB. Attempting to switch to heap state immediately ran into OOM (parallelism: 120, 8gb memory each). And perhaps the answer is just "scale out" :) but if there are any signals to know when we've reached the limit of current scale, it'd be great to know what signals to look for! Thanks! Trystan

Can BroadcastProcessFunction invoke both methods concurrently?

2021-10-06 Thread Trystan
ize that this is very close to playing with fire, and is exactly why we have a broadcast state where Flink can hide all the danger, so would be open to other ideas if this is cardinal sin! Trystan

Re: Shared Checkpoint Cleanup and S3 Lifecycle Policy

2020-05-07 Thread Trystan
s). So this shared state lives on forever and there is no way to ever clean it up, at all. I am surprised that this hasn't been a problem for anyone else. Maybe I should just file a feature request for this, at least to find some solution for ways to clean up these directories. I appreciate your

Re: Shared Checkpoint Cleanup and S3 Lifecycle Policy

2020-05-06 Thread Trystan
ould you suggest to use it? My understanding is that I don't control anything beyond the checkpoint directory, and since shared is in that directory I can't put entropy inside the shared directory itself (which is what I would need). Thanks, Trystan On Wed, May 6, 2020 at 7:31 PM Congxian Qiu

Shared Checkpoint Cleanup and S3 Lifecycle Policy

2020-05-06 Thread Trystan
e the _entropy_ trick would help this, because the issue is ultimately that there's a single shared directory. Thank you! Trystan