Re: Editing job graph at runtime

Hi, Option 2 is to implement your own Source/Sink. Currently, we have the old, discouraged interfaces along with the new interfaces. For source, you want to read [1]. There is a KafkaSource already in 1.12 that we consider Beta, you can replace it with the 1.13 after the 1.13 release (should be c

Re: Editing job graph at runtime

Hi Arvid, Thanks for the reply. I am currently exploring the flink features and we have certain use cases where new producers will be added the system dynamically and we don't want to restart the application frequently. It will be helpful if you explain the option 2 in detail ? Thanks & Regards

Re: Editing job graph at runtime

Thanks Arvid for the reply. Can you please elaborate a little bit on option 2 , if possible ? We are looking for a similar option. Currently we are proceeding with option 1. Thanks Jessy for the question On Mon, Mar 22, 2021, 7:27 PM Arvid Heise wrote: > Hi Jessy, > > Can I add a new sink int

Re: Editing job graph at runtime

Thanks Arvid for the reply. Can you please elaborate a little bit on option 2 , if possible ? Thanks Jessy On Mon, Mar 22, 2021, 7:27 PM Arvid Heise wrote: > Hi Jessy, > > Can I add a new sink into the execution graph at runtime, for example : a >> new Kafka producer , without restarting the c

Re: Editing job graph at runtime

Hi Jessy, Can I add a new sink into the execution graph at runtime, for example : a > new Kafka producer , without restarting the current application or using > option1 ? > No, there is no way to add a sink without restart currently. Could you elaborate why a restart is not an option for you? Y

Re: Editing job graph at runtime

Hi Team, Can you provide your thoughts on this, it will be helpful .. Thanks Jessy On Tue, 16 Mar 2021 at 21:29, Jessy Ping wrote: > Hi Timo/Team, > Thanks for the reply. > > Just take the example from the following pseduo code, > Suppose , this is the current application logic. > > firstInput

Re: Editing job graph at runtime

Hi Timo/Team, Thanks for the reply. Just take the example from the following pseduo code, Suppose , this is the current application logic. firstInputStream = addSource(...)* //Kafka consumer C1* secondInputStream = addSource(...) *//Kafka consumer C2* outputStream = firstInputStream,keyBy(a ->

Re: Editing job graph at runtime

Hi Jessy, to be precise, the JobGraph is not used at runtime. It is translated into an ExecutionGraph. But nevertheless such patterns are possible but require a bit of manual implementation. Option 1) You stop the job with a savepoint and restart the application with slightly different par