Re: Broadcast state and OutOfMemoryError: Direct buffer memory

Hi Dan, Usually broadcast state needs more network buffers, the network buffer used to exchange data records between tasks would request a portion of direct memory[1], I think it is possible to get the “Direct buffer memory” OOM errors in this scenarios. Maybe you can try to increase taskmanager.m

Difference between DataStream.broadcast() vs DataStream.broadcast(MapStateDescriptor)

Hi all, I am trying to figure out how Datastream.broadcast() and DataStream.broadcast(MapStateDescriptor) differ. My use case: I have 2 streams: Stream 1 contains updates, which collectively build up a state Stream 2 is keyed and every parallel instance need to connect with EVERY update from Str

Re: Modify savepoints in Flink

Thanks, I'll check it out. On Fri, Oct 21, 2022, 18:20 Piotr Nowojski wrote: > Hi, > > Yes and no. StateProcessor API can read any Flink state, but you have to > describe the state you want it to access. Take a look at the example in the > docs [1]. > > First you have an example of a theoretical

Re: Cannot run pyflink example using Flink CLI

Great, thanks! Kind regards, Levan Huyen On Fri, 21 Oct 2022 at 00:53, Biao Geng wrote: > You are right. > It contains the python package `pyflink` and some dependencies like py4j > and cloudpickle but does not contain all relevant dependencies(e.g. > `google.protobuf` as the error log shows, w

Re: Modify savepoints in Flink

Hi, Yes and no. StateProcessor API can read any Flink state, but you have to describe the state you want it to access. Take a look at the example in the docs [1]. First you have an example of a theoretical production function `StatefulFunctionWithTime`, which state you want to modify. Note the `V

Re: Modify savepoints in Flink

I have question on this. Different connector can have different serialisation and de-serlisation technique right?. Wouldn't that impact?. If I use StateProcessor API, would that be agnostic to all the sources and sinks?. On Fri, Oct 21, 2022, 18:00 Piotr Nowojski wrote: > ops > > > Alternatively

Re: Modify savepoints in Flink

ops > Alternatively, you can modify a code of your function/operator for which you want to modify the state. For example in the `org.apache.flink.streaming.api.checkpoint.CheckpointedFunction#initializeState` method you could add some code that would do a migration of your old state to a new one.

Re: Passing SSLContext to the ElasticSearchSink

Hi Nick, I would say open a PR so one of the maintainers can have a look :) Thanks, Martijn On Sun, Oct 9, 2022 at 3:16 AM Nick Levandoski wrote: > Hello, > > I've had an issue similar to the one reported in ticket FLINK-27054. I > would like to pass the SSLContext to the ElasticSearchSink vi

Re: Modify savepoints in Flink

Thanks !. Will try this. On Fri, Oct 21, 2022 at 2:22 PM Piotr Nowojski wrote: > Hi Sriram, > > You can read and modify savepoints using StateProcessor API [1]. > > Alternatively, you can modify a code of your function/operator for which > you want to modify the state. For example in the > `org.

Re: Modify savepoints in Flink

Hi Sriram, You can read and modify savepoints using StateProcessor API [1]. Alternatively, you can modify a code of your function/operator for which you want to modify the state. For example in the `org.apache.flink.streaming.api.checkpoint.CheckpointedFunction#initializeState` method you could a

Modify savepoints in Flink

Hi All, I am working on a scenario where I need to modify the existing savepoint operator state. Ex: Wanted to remove some offset of the savepoint. What is the better practice for these scenarios?. Could you please help me with any example as such? Thanks in advance. -- *Sriram G* *Tech*

Broadcast state and OutOfMemoryError: Direct buffer memory

Hi. My team recently added broadcast state to our Flink jobs. We've started hitting this OOM "Direct buffer memory" error. Is this a common problem with broadcast state? Or is it likely a different problem? Thanks! - Dan