Using futures within co-process function

I have a co-process function which goes through a loop and performs the same task against the input event using different parameters in the state. So, on the one hand we have the basic input events and on the other the parameters. Would that be a good idea to parallelize this task by means of

Re: Table API in process function

i, Oct 13, 2023 at 9:31 PM Yashoda Krishna T wrote: > Is it possible to use table API inside a processAll window function . > Lets say, the use case is process function should enrich for each element > by querying some SQL queries over the entire elements in the window using > table

Table API in process function

Is it possible to use table API inside a processAll window function . Lets say, the use case is process function should enrich for each element by querying some SQL queries over the entire elements in the window using table API. Is this case supported in flink? If not what is the suggested way

Re: Access to collector in the process function

case. > > For that, we would like to call an interface from the use case that > effectively sends the event ultimately via out.collect > > The problem is that for instantiating the use case we need to inject the > collector as dependency and we dont have access to the collector

Access to collector in the process function

instantiating the use case we need to inject the collector as dependency and we dont have access to the collector at the process function class level, only at the processelement method level. Is there any way to access the collector from the process function class, in the open method ? Regards, Oscar

Re: How to trigger process function when no event in eventTimeWindow ?

erun situation. > 4、when using flink eventTimeWindow function, process function can not be > trigger if no event input at that window. > > For example, a product named A which have two data input with time > 9:25:12 and 9:28:23 ,how can I output zero between 9:25 and 9:28 with > EventTime window? > > Best regards, > Xlf >

How to trigger process function when no event in eventTimeWindow ?

. 4、when using flink eventTimeWindow function, process function can not be trigger if no event input at that window. For example, a product named A which have two data input with time 9:25:12 and 9:28:23 ,how can I output zero between 9:25 and 9:28 with EventTime window? Best regards, Xlf

Re: Implement watermark buffering with Process Function

ing. It seems that WindowOperator consists of some >> InternalStates, of which signature is where window is namespace or key, if >> I understand correctly. But internal states are not available for Flink >> users. >> >> So my question is: is there an efficient way to simulate watermark >> buffering using process function for Flink users? >> >> Thanks. >> >

Re: Implement watermark buffering with Process Function

; users. > > So my question is: is there an efficient way to simulate watermark > buffering using process function for Flink users? > > Thanks. >

Implement watermark buffering with Process Function

It seems that WindowOperator consists of some InternalStates, of which signature is where window is namespace or key, if I understand correctly. But internal states are not available for Flink users. So my question is: is there an efficient way to simulate watermark buffering using process function fo

Re: Passing msg and record to the process function

nit) type while the second one >> produces a (String, String) type, so the whole if expression produces >> (String, Any) type. However your parseJson should return Either[String, >> String], thus causing this issue. >> >> >> Siddhesh Kalgaonkar 于2022年1月5日周三 19:04写

Re: Passing msg and record to the process function

ype. However your parseJson should return Either[String, String], thus > causing this issue. > > > Siddhesh Kalgaonkar 于2022年1月5日周三 19:04写道： > >> I have written a process function where I am parsing the JSON and if it >> is not according to the expected format it passes as

Re: Passing msg and record to the process function

he second one produces a (String, String) type, so the whole if expression produces (String, Any) type. However your parseJson should return Either[String, String], thus causing this issue. Siddhesh Kalgaonkar 于2022年1月5日周三 19:04写道： > I have written a process function where I am parsing the JSON an

Passing msg and record to the process function

I have written a process function where I am parsing the JSON and if it is not according to the expected format it passes as Failure to the process function and I print the records which are working fine. Now, I was trying to print the message and the record in case of Success and Failure. I

Re: Flink 1.11 loses track of event timestamps and watermarks after process function

s to puzzle me. Here's a contrived > example to clarify: A process function receives data but never emits > anything (neither in the processElement or based on a timer).. i.e., the > processFunction is just a black hole for event records passing through it. > Do watermarks still make it

Re: Flink 1.11 loses track of event timestamps and watermarks after process function

*Watermark progression:* The code you pointed out in [1], seems to indicate that watermarks are a logical side-effect that travel alongside events in the stream but can also progress on their own? This continues to puzzle me. Here's a contrived example to clarify: A process function receives

Re: Flink 1.11 loses track of event timestamps and watermarks after process function

Hi Ahmad, The ProcessFunction is simply forwarding the Watermark [1]. So I don't have any explanation as to why it would not advance anymore as soon as you emit data. My assumption was that by emitting in the process function causes backpressure and thus halts the advancement of the wate

Re: Flink 1.11 loses track of event timestamps and watermarks after process function

I replaced the Aync IO with a simple process function and print statements in the body of the process function. The process function simply emits what it received. I also removed the custom sink (that has an external dependency) and replaced it with a simple lambda that occasionally prints ju

Re: Flink 1.11 loses track of event timestamps and watermarks after process function

n key's by a field and > uses a KeyedProcessFunciton. > > The keyed process function outputs events from with the `processElement` > method using `out.collect`. No timers are used to collect or output any > elements (or do anything for that matter). > > I also have a simple pri

Flink 1.11 loses track of event timestamps and watermarks after process function

Flink 1.11 I have a simple Flink application that reads from Kafka, uses event timestamps, assigns timestamps and watermarks and then key's by a field and uses a KeyedProcessFunciton. The keyed process function outputs events from with the `processElement` method using `out.collect`. No t

Re: callback by using process function

t modification time) match with the previous timestamp count. > > Is there refere about checking the previous count? am I understanding > correctly? help me to understand this part. > > (ii)  can the process function be used to look back the previous > key/count? > > [1] 

callback by using process function

? am I understanding correctly? help me to understand this part. (ii) can the process function be used to look back the previous key/count? [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/operators/process_function/ Thank you

Re: Window Process function is not getting trigger

e. Your timestamp assigner returns timestamp in >>> second resolution while Flink requires millisecond resolution. >>> >>> >>> Best, >>> Kezhu Wang >>> >>> On February 24, 2021 at 11:49:59, sagar (sagarban...@gmail.com) wrote: >>> >

Re: Window Process function is not getting trigger

gt;> >>> I saw one potential issue. Your timestamp assigner returns timestamp in >>> second resolution while Flink requires millisecond resolution. >>> >>> >>> Best, >>> Kezhu Wang >>> >>> On February 24, 2021 at 11:49:59, sagar

Re: Window Process function is not getting trigger

gt; On February 24, 2021 at 11:49:59, sagar (sagarban...@gmail.com) wrote: >> >> I have simple flink stream program, where I am using socket as my >> continuous source >> I have window size of 2 seconds. >> >> Somehow my window process function is not trigge

Re: Window Process function is not getting trigger

e I am using socket as my >> continuous source >> I have window size of 2 seconds. >> >> Somehow my window process function is not triggering and even if I pass >> events in any order, flink is not ignoring >> >> I can see the output only when I kill my sock

Re: Window Process function is not getting trigger

ream program, where I am using socket as my > continuous source > I have window size of 2 seconds. > > Somehow my window process function is not triggering and even if I pass > events in any order, flink is not ignoring > > I can see the output only when I kill my sock

Re: Window Process function is not getting trigger

continuous source I have window size of 2 seconds. Somehow my window process function is not triggering and even if I pass events in any order, flink is not ignoring I can see the output only when I kill my socket , please find the code snippet below final StreamExecutionEnvironment env

Window Process function is not getting trigger

I have simple flink stream program, where I am using socket as my continuous source I have window size of 2 seconds. Somehow my window process function is not triggering and even if I pass events in any order, flink is not ignoring I can see the output only when I kill my socket , please find

Re: How do I pass the aggregated results of an SQL TUMBLING Window as one set of data to a Process Function?

Hi Marco, sorry for the late reply. Have you looked into user-defined aggregate functions for SQL? I think your requirements can be easily implemented there. You can declare multiple aggregate functions per window. There is also the built-in function LISTAGG that might help for your use case.

Re: How do I pass the aggregated results of an SQL TUMBLING Window as one set of data to a Process Function?

Alright, maybe my example needs to be more concrete. How about this: In this example, I don't want to create to windows just to re-combine what was just aggregated in SQL. Is there a way to transform the aggregate results into one datastream object so that I don't have to aggregate again? // agg

How do I pass the aggregated results of an SQL TUMBLING Window as one set of data to a Process Function?

DOW --> aggregated table (the is results of table with 20 records in it ) --> PROCESS FUNCTION (aggregated table)

Re: Adding keyed state to test harness before calling process function.

t;> flink-state-backends/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksIncrementalCheckpointRescalingTest.java >> Best, >> Guowei >> >> >> On Fri, Nov 13, 2020 at 10:36 AM Marco Villalobos < >> mvillalo...@kineteq

Re: Adding keyed state to test harness before calling process function.

> > On Fri, Nov 13, 2020 at 10:36 AM Marco Villalobos < > mvillalo...@kineteque.com> wrote: > >> Hi, >> >> I would like to adding keyed state to test harness before calling process >> function. >> >> I am using the OneInputStreamOperatorTestHar

Re: Adding keyed state to test harness before calling process function.

; I would like to adding keyed state to test harness before calling process > function. > > I am using the OneInputStreamOperatorTestHarness. > > I can't find any examples online on how to do that, and I am struggling to > figure this out. > > Can somebody please provide g

Adding keyed state to test harness before calling process function.

Hi, I would like to adding keyed state to test harness before calling process function. I am using the OneInputStreamOperatorTestHarness. I can't find any examples online on how to do that, and I am struggling to figure this out. Can somebody please provide guidance? My test case has

Re: Parallelism of Keyed Process Function

-spread-out-slots configuration [1]. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/config.html#cluster-evenly-spread-out-slots On Mon, Sep 14, 2020 at 5:33 PM Arti Pande wrote: > Hi, > > Here is a question related to parallelism of keyed-process-function

Parallelism of Keyed Process Function

Hi, Here is a question related to parallelism of keyed-process-function that is applied to the KeyedStream. For some code that looks like this myStream.keyBy(...) .process(new MyKeyedProcessFunction()) .process().setParallelism(10) On a Flink cluster with 5 TM nodes each with 10 task

Re: flink: keyed process function, why are timestamp of register event timer different as "on timer" function timestamp

; > Hi User, > > We are using keyed process function with Event time for flink streaming > application. > We register event time on "processElement" function, and mentioned that > "onTimer" function had different "timestamp" as registered on &

flink: keyed process function, why are timestamp of register event timer different as "on timer" function timestamp

Hi User, We are using keyed process function with Event time for flink streaming application. We register event time on "processElement" function, and mentioned that "onTimer" function had different "timestamp" as registered on "processElement" functio

Re: How can I set configuration of process function from job's main?

Hi, the configuration parameter is just legacy API. You can simply pass any serializable object to the constructor of your process function. Regards, Timo Am 29.03.18 um 20:38 schrieb Main Frame: Hi guys! Iam newbie in flink and I have probably silly question about streaming api. So for

How can I set configuration of process function from job's main?

Hi guys! Iam newbie in flink and I have probably silly question about streaming api. So for the instance: I trying to apply SomeProcessFunction to stream1 … DataStream stream2 = stream1.process(new MyProcessFunction()).name("Ingest data»); … I have created package-private class with MyProcess

Re: SideOutput doesn't receive anything if filter is applied after the process function

Thanks Chesnay, So I think to support multi input and multiple output model like data flow paper indicates, Flink needs to get credit based scheduling as well as side input ready and doing a new set of data stream apis that doesn’t constrained with backwards compatibility issues. Only then can

Re: SideOutput doesn't receive anything if filter is applied after the process function

I've opened https://issues.apache.org/jira/browse/FLINK-8437 Unfortunately i doubt we can fix this properly. The proposed solution will not work if we ever allow arbitrary functions to use side-outputs. On 16.01.2018 08:59, Juho Autio wrote: Could someone with knowledge of the right terms crea

Re: SideOutput doesn't receive anything if filter is applied after the process function

Could someone with knowledge of the right terms create this in JIRA, please? I guess I could also create it if needed.. On Mon, Jan 15, 2018 at 3:15 PM, Chesnay Schepler wrote: > yes, i meant that process() returns the special operator. This would > definitely deserve a JIRA issue. > > > On 15.0

Re: SideOutput doesn't receive anything if filter is applied after the process function

yes, i meant that process() returns the special operator. This would definitely deserve a JIRA issue. On 15.01.2018 14:09, Juho Autio wrote: Thanks for the explanation. Did you meant that process() would return a SingleOutputWithSideOutputOperator? Any way, that should be enough to avoid the

Re: SideOutput doesn't receive anything if filter is applied after the process function

Thanks for the explanation. Did you meant that process() would return a SingleOutputWithSideOutputOperator? Any way, that should be enough to avoid the problem that I hit (and it also seems like the best & only way). Maybe the name should be something more generic though, like ProcessedSingleOutp

Re: SideOutput doesn't receive anything if filter is applied after the process function

It would mean that getSideOutput() would return a SingleOutputWithSideOutputOperator which extends SingleOutputOperator offering getSideOutput(). Other transformations would still return a SingleOutputOperator. With this the following code wouldn't compile. stream .process(...) .filte

Re: SideOutput doesn't receive anything if filter is applied after the process function

> sideoutput might deserve a seperate class which inherit form singleoutput. It might prevent lot of confusions Thanks, but how could that be done? Do you mean that if one calls .process(), then the stream would change to another class which would only allow calls like .getMainOutput() or .getSide

Re: SideOutput doesn't receive anything if filter is applied after the process function

Hi Juho, I think sideoutput might deserve a seperate class which inherit form singleoutput. It might prevent lot of confusions. A more generic question is whether datastream api can be mulitple ins and mulitple outs natively. It's more like scheduling problem when you come from single process syst

Re: SideOutput doesn't receive anything if filter is applied after the process function

Maybe I could express it in a slightly different way: if adding the .filter() after .process() causes the side output to be somehow totally "lost", then I believe the .getSideOutput() could be aware that there is not such side output to be listened to from upstream, and throw an exception. I mean,

Re: SideOutput doesn't receive anything if filter is applied after the process function

Hi Juho, Now that I think of it this seems like a bug to me: why does the call to getSideOutput succeed if it doesn't provide _any_ input? With the way side outputs work, I don’t think this is possible (or would make sense). An operator does not know whether or not it would ever emit some elem

SideOutput doesn't receive anything if filter is applied after the process function

When I run the code below (Flink 1.4.0 or 1.3.1), only "a" is printed. If I switch the position of .process() & .filter() (ie. filter first, then process), both "a" & "b" are printed, as expected. I guess it's a bit hard to say what the side output should include in this case: the stream before fi

Re: Process Function

ep >> a lastModified variable as a ValueState, >> then compare it to the timestamp provided by the timer to see if the >> current key should be evicted. >> Checkout the example on the ProcessFunction page. >> >> https://ci.apache.org/projects/flink/flin

Re: Process Function

Hi All, I have a streaming pipeline which is keyed by userid and then to a flatmap function. I need to clear the state after sometime and I was looking at process function for it. Inside the process element function if I register a timer wouldn't it create a timer f

Re: Process Function

org/projects/flink/flink-docs-release-1.2/dev/stream/process_function.html >> >> <https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/process_function.html> >> >> Best regards, >> Kien >> >> On 9/5/2017 11:49 AM, Navneeth K

Re: Process Function

> current key should be evicted. > Checkout the example on the ProcessFunction page. > > https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/ > process_function.html > > Best regards, > Kien > > On 9/5/2017 11:49 AM, Navneeth Krishnan wrote: > &g

Re: Process Function

ease-1.2/dev/stream/process_function.html > > <https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/process_function.html> > > Best regards, > Kien > > On 9/5/2017 11:49 AM, Navneeth Krishnan wrote: >> Hi All, >> >> I have a stre

Re: Process Function

/dev/stream/process_function.html Best regards, Kien On 9/5/2017 11:49 AM, Navneeth Krishnan wrote: Hi All, I have a streaming pipeline which is keyed by userid and then to a flatmap function. I need to clear the state after sometime and I was looking at process function for it. Inside the

Re: Process Function

Hi Navneeth, Currently, I don't think there is any built-in functionality to trigger onTimer periodically. As for the second part of your question, do you mean that you want to query on which key the fired timer was registered from? I think this also isn't possible right now. I'm looping in Aljos

Re: Process Function

How are you determining your data is stale? Also if you want to know the key, why don't you store the key in your state as well? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Process Function

Hi All, I have a streaming pipeline which is keyed by userid and then to a flatmap function. I need to clear the state after sometime and I was looking at process function for it. Inside the process element function if I register a timer wouldn't it create a timer for each incoming me