Re: CoProcessFunction processElement1 and processElement2

They are designed to avoid race condition. Get Outlook for Android From: Tan, Min Sent: Sunday, November 1, 2020 2:15:54 AM To: user Subject: CoProcessFunction processElement1 and processElement2 Hi, Can someone confirm that the proce

Re: Connecting two streams and order of their processing

With Keyed dual stream processing, you make sure that events for same key to processElement 1 and 2 are received to same partition. However, when you receive an event in processElement1, you should store that in flinks state so that if an another event arrives on delay to processElement2, you ca

Re: Flink issue in emitting data to same sideoutput from onTimer and processElement

I am using KeyedCoProcessFunction Get Outlook for Android<https://aka.ms/ghei36> From: Jaswin Shah Sent: Friday, August 14, 2020 3:09:21 PM To: user@flink.apache.org ; Dawid Wysakowicz ; Yun Tang Subject: Flink issue in emitting data to same sideoutpu

Flink issue in emitting data to same sideoutput from onTimer and processElement

Hi, I have a coProcessFunction which emits data to same side output from processElement1 method and on timer method.But, data is not getting emitted to sideoutput from onTimer. Is it like to the same sideoutput, we can not emit data from onTimer and processElement methods? Get Outlook for And

[no subject]

Hi, I have a coProcessFunction which emits data to same side output from processElement1 method and on timer method.But, data is not getting emitted to sideoutput from onTimer. Is it like to the same sideoutput, we can not emit data from onTimer and processElement methods? Get Outlook for Andr

Custom service configs in flink

I have multiple flink jobs and have custom business configs which are shared between the job. Is it possible if one flink job loads configs in memory and all the flink jobs share the same configs? Basically, I am thinking to fetch configs in one flink job in memory via rest call which is one tim

Re: DROOLS rule engine with flink

Thats what I wanted to know I will I be able to achieve same only with flink if not use drool engine? From: Jörn Franke Sent: 24 June 2020 12:46 To: Jaswin Shah Cc: user@flink.apache.org Subject: Re: DROOLS rule engine with flink I would implement them

Re: DROOLS rule engine with flink

Bs aren't down. From: Georg Heiler Sent: 24 June 2020 00:59 To: Jaswin Shah Cc: user@flink.apache.org Subject: Re: DROOLS rule engine with flink Why not use flink CEP? https://flink.apache.org/news/2020/03/24/demo-fraud-detection-2.html has a nice intera

DROOLS rule engine with flink

Hi I am thinking of using some rule engine like DROOLS with flink to solve a problem described below: I have stream of events coming from kafka topic and I want to analyze those events based on some rules and give the results in results streams when rules are satisfied. Now, I am able to solve

Re: Shared state between two process functions

thinking of some shared state accessible between multiple keyedprocessfunctions. Get Outlook for Android<https://aka.ms/ghei36> From: Yun Gao Sent: Monday, June 15, 2020 8:27:38 AM To: Jaswin Shah ; user@flink.apache.org Subject: Re: Shared state between two p

Shared state between two process functions

Hi, Is it possible to create the shared state(MapState) between two different keyedProcessFunction? If it's possible, how can we do that in flink? Thanks, Jaswin

Re: MapState in flink

I need some representation like this: [cid:d11ee982-30b1-463f-b755-27047b743e8d] From: Jaswin Shah Sent: 14 June 2020 01:01 To: user@flink.apache.org Subject: MapState in flink Hi, Can anyone please help me on how can I create a MapState of ListState in flink

MapState in flink

Hi, Can anyone please help me on how can I create a MapState of ListState in flink, does flink support the same and if supports, how to declare the descriptor for same state data structure? If it is not supported, how may I create similar datastructure for state in flink? Thanks, Jaswin

KeyedStream and keyedProcessFunction

Hi All, I have a keyed data stream and calling a keyedProcessFunction after keyBy operation on datastream. Till now my understanding was, "For all different n- elements in keyed stream if their keys are same, same instance of keyedProcessFunction is called and for another elements with differen

Re: Getting Window information from coGroup functin

object you can fetch the information of window, start time, end time etc. From: Jaswin Shah Sent: 04 June 2020 13:45 To: Dawid Wysakowicz ; user@flink.apache.org Cc: Aljoscha Krettek Subject: Re: Getting Window information from coGroup functin I think here apply

Re: Getting Window information from coGroup functin

I think here apply function would receive only the events but not necessarily a complete window at same time. From: Dawid Wysakowicz Sent: Thursday, June 04, 2020 13:39 To: Sudan S; user@flink.apache.org Cc: Aljoscha Krettek Subject: Re: Getting Window informatio

Re: Flink TTL for MapStates and Sideoutputs implementations

Thanks for responding Alexander. We have solved the problem now with ValueState now. Basically, here we are implementing outer join logic with custom keyedCoprocessFunction implementations. From: Alexander Fedulov Sent: 28 May 2020 17:24 To: Jaswin Shah Cc

Re: Timeout Callbacks issue -Flink

Yeah yes, I got it what u tried to convey. From: Dawid Wysakowicz Sent: Monday, May 25, 2020 16:48 To: Jaswin Shah; user@flink.apache.org; ankit.sing...@paytm.com; isha.sing...@paytm.com Subject: Re: Timeout Callbacks issue -Flink Where did I say you cannot

Re: Timeout Callbacks issue -Flink

From: Jaswin Shah Sent: 25 May 2020 16:06 To: Dawid Wysakowicz ; user@flink.apache.org ; ankit.sing...@paytm.com ; isha.sing...@paytm.com Subject: Re: Timeout Callbacks issue -Flink OMG!!! If this is the case Dawid, I think I am solving the problem in an incorrect way. Here I would

Re: Timeout Callbacks issue -Flink

Dawid Wysakowicz Sent: Monday, May 25, 2020 15:57 To: Jaswin Shah; user@flink.apache.org; ankit.sing...@paytm.com; isha.sing...@paytm.com Subject: Re: Timeout Callbacks issue -Flink I don't necessarily know how can I better describe it. The MapState/ValueState is always implicitly scoped to t

Re: Timeout Callbacks issue -Flink

n't receive the events from other stream. I hope here my understanding is correct. Please correct me if I am wrong here. ____ From: Jaswin Shah Sent: 25 May 2020 15:19 To: Dawid Wysakowicz ; user@flink.apache.org ; ankit.sing...@paytm.com ; isha.sing...@paytm.com Sub

Re: Timeout Callbacks issue -Flink

eive the callbacks. I hope here my understanding is correct. Please correct me if I am wrong here. From: Dawid Wysakowicz Sent: Monday, May 25, 2020 15:14 To: Jaswin Shah; user@flink.apache.org; ankit.sing...@paytm.com; isha.sing...@paytm.com Subject: Re: Timeout C

Re: Timeout Callbacks issue -Flink

ValueState will get updated with most recent user key on every events arrival. From: Jaswin Shah Sent: 25 May 2020 14:47 To: Dawid Wysakowicz ; user@flink.apache.org ; ankit.sing...@paytm.com ; isha.sing...@paytm.com Subject: Re: Timeout Callbacks issue -Flink

Re: Timeout Callbacks issue -Flink

If I understand correctly, you are trying to tell that I should have valueState of Map? From: Jaswin Shah Sent: 25 May 2020 14:43 To: Dawid Wysakowicz ; user@flink.apache.org ; ankit.sing...@paytm.com ; isha.sing...@paytm.com Subject: Re: Timeout Callbacks

Re: Timeout Callbacks issue -Flink

stream. So, in that case how valueState could benefit me? Can you please explain me that, might be I have understood it incorrectly what you are trying to convey here. Thanks, Jaswin From: Dawid Wysakowicz Sent: Monday, May 25, 2020 14:23 To: Jaswin Shah; user

Re: Timeout Callbacks issue -Flink

registrations? Thanks, Jaswin From: Jaswin Shah Sent: 23 May 2020 17:18 To: user@flink.apache.org ; Arvid Heise ; Yun Tang Subject: Timeout Callbacks issue -Flink Hi, I am running flink job with following functionality: 1. I consume stream1 and stream2 from

Timeout Callbacks issue -Flink

Hi, I am running flink job with following functionality: 1. I consume stream1 and stream2 from two kafka topics and assign the watermarks to the events of two streams by extracting the timestamps from the events in streams. 2. Then, I am connecting two streams and calling KeyedCoProcessFun

onTimer method in CoProcessFunction in flink

How can I identify the type of element for which onTime is called in flink? I want to store the objects for which onTimer is called to sideOutputs and then streamout the sideoutput data to kafka topic. I am not understanding how to stream out the sideoutput data like where should I write that pro

Flink TTL for MapStates and Sideoutputs implementations

public class CartPGCoprocessFunction extends KeyedCoProcessFunction { private static final Logger logger = LoggerFactory.getLogger(CartPGCoprocessFunction.class); /** * Map state for cart messages, orderId+mid is key and cartMessage is value. */ private static MapState car

Re: Rocksdb implementation

Okay, so on checkpointing window's data would also be persisted. From: Arvid Heise Sent: 20 May 2020 01:05 To: Jaswin Shah Cc: Yun Tang ; isha.sing...@paytm.com ; ankit.sing...@paytm.com ; user@flink.apache.org Subject: Re: Rocksdb implementation I

Re: Rocksdb implementation

10 To: Jaswin Shah Cc: Yun Tang ; isha.sing...@paytm.com ; ankit.sing...@paytm.com ; user@flink.apache.org Subject: Re: Rocksdb implementation Hi Jaswin, you cannot run a DataSet program inside a DataStream. However, you can perform the same query on a windowed stream. So if you would execut

Re: Rocksdb implementation

same streaming job that I have. Basically, I need that missing message infos outside. From: Jaswin Shah Sent: 19 May 2020 13:29 To: Yun Tang ; Arvid Heise ; isha.sing...@paytm.com ; ankit.sing...@paytm.com Cc: user@flink.apache.org Subject: Re: Rocksdb

Re: Rocksdb implementation

++ From: Yun Tang Sent: 18 May 2020 23:47 To: Arvid Heise ; Jaswin Shah Cc: user@flink.apache.org Subject: Re: Rocksdb implementation Hi Jaswin As Arvid suggested, it's not encouraged to query the internal RocksDB directly. Apart from Arvid's s

Re: Rocksdb implementation

); } catch (KafkaException e) { producer.abortTransaction(); } catch (MissingConfigurationsException e) { e.printStackTrace(); } producer.close(); } } This is the snapshot of implementation I have done From: Jas

Rocksdb implementation

Hi, I have implemented the flink job with MapStates. The functionality is like, 1. I have two datastreams which I connect with connect operator and then call coprocessfunction with every pair of objects. 2. For element of first datastream, processElement1 method is called and for an elemen

Re: Not able to implement an usecase

If I go with table apis, can I write the streams to hive or it is only for batch processing as of now. Get Outlook for Android<https://aka.ms/ghei36> From: Khachatryan Roman Sent: Tuesday, May 12, 2020 1:49:10 AM To: Jaswin Shah Cc: user@flink.apac

Not able to implement an usecase

Hi, I want to implement the below use case in my application: I am doing an interval join between two data streams and then, in process function catching up the discrepant results on joining. Joining is done on key orderId. Now, I want to identify all the messages in both datastreams which are n