Re: How to read a savepoint fast without exploding the memory

2025-02-10 Thread Gabor Somogyi
Hi Jean-Marc, FYI, I've just opened this [1] PR to address the issue in a clean way. May I ask you to test it on your side? [1] https://github.com/apache/flink/pull/26134 BR, G On Fri, Feb 7, 2025 at 6:14 PM Gabor Somogyi wrote: > Just a little update on this. We've made our first POC with t

PyFlink StateTtlConfig and Thread Mode

2025-02-10 Thread Justin Ngai via user
Hello I've been trying to setup StateTtl for my PyFlink application using thread mode. I was following the examples in the docs (needed to use Time instead of Duration) -- https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/state/#state-in-the-scala-datastream-

File Sink add Header with forRowFormat

2025-02-10 Thread Kyle Nussdorfer
Hi all, we have a use case we are stuck on. We would like to take streaming input data and create files (s3) every XX minutes or when the size of records exceeds a size limit. the forRowFormate works great as the rolling policy supports both of these use cases. However one missing feature is the a

Re: ssl certificate reload capabilities

2025-02-10 Thread Gabor Somogyi
Hi Nicolas, This is only supported by the operator but not by Flink. It's on my list but not yet reached this to actually do it. BR, G On Mon, Feb 10, 2025 at 10:41 AM Nicolas Fraison via user < user@flink.apache.org> wrote: > Hi, > > We are looking into enabling SSL on our flink jobs. > Follo

ssl certificate reload capabilities

2025-02-10 Thread Nicolas Fraison via user
Hi, We are looking into enabling SSL on our flink jobs. Following the flink documentation https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/security/security-ssl/ is working fine but we have a specific constraint with ssl certificates having only a one day validity. Due to that

Re: Skip Restore Incompatibility of Key Serializer for Savepoint

2025-02-10 Thread Gabor Somogyi
Seems like I was not clear in my previous suggestions. `execution.savepoint.ignore-unclaimed-state=true` would help only if FLINK-20375 [1] would be implemented. Please has a look at the state processor API [2]. I would try something like: SavepointWriter.fromExistingSavepoint(env, oldPath, n