I think I got it
Glad you solved this tricky issue and thanks for sharing your solution :-)
Best, Fabian
2018-01-06 14:33 GMT+01:00 Vishal Santoshi :
> Yep, this though is suboptimal as you imagined. Two things
>
> * has a internally has a that is a ultra lite version of IN,
> only required
Yep, this though is suboptimal as you imagined. Two things
* has a internally has a that is a ultra lite version of IN,
only required for the path analysis.
* Sessionization being expensive, we piggy back multiple other aggregations
that do not depend on the path or order ( count etc ) . Essen
Hi,
you would not need the ListStateDescriptor. A WindowProcessFunction stores
all events that are assigned to a window (IN objects in your case) in an
internal ListState.
The Iterable parameter of the process() method iterates over the
internal list state.
So you would have a Trigger that fires
Hello Fabian, Thank you for your response.
I thought about it and may be am missing something
obvious here. The code below is what I think you suggest. The issue is that
the window now is a list of Session's ( or shall subsets of the Session).
What is required is that on a ne
Hi Vishal,
thanks for sharing your solution!
Looking at this issue again and your mail in which you shared your
SessionProcessWindow ProcessWindowFunction, I'm wondering why you need the
ValueState that prevents the ProcessWindowFunction to be used in a
mergeable window.
You could have created a
Dear Fabian,
I was able to create a pretty functional ProcessFunction and
here is the synopsis and please see if it makes sense.
Sessionization is unique as in it entails windows of dynamic length. The
way flink approaches is pretty simple. It will create a TimeWindow of size
"gap" rel
This makes sense. Thanks.
On Sat, Dec 23, 2017 at 10:58 AM, Fabian Hueske wrote:
> Hi,
>
> all calls to onElement() or onTimer() are syncronized for any keys. Think
> of a single thread calling these methods.
> Event-time timers are called when a watermark passes the timer. Watermarks
> are rec
Hi,
all calls to onElement() or onTimer() are syncronized for any keys. Think
of a single thread calling these methods.
Event-time timers are called when a watermark passes the timer. Watermarks
are received as special records, so the methods are called in the same
order as records (actual records
Thanks.
I have a few follow up questions regarding ProcessFunction. I think
that the core should take care of any synchronization issues between calls
to onElement and onTimer in case of a keyed stream but tests do not seem to
suggest that.
I have specifically 2 questions.
1. Are cal
That's correct. Removal of timers is not supported in ProcessFunction. Not
sure why this is supported for Triggers.
The common workaround for ProcessFunctions is to register multiple timers
and have a ValueState that stores the valid timestamp on which the onTimer
method should be executed.
When a
And that further begs the question.. how performant is Timer Service. I
tried to peruse through the architecture behind it but cold not find a
definite clue. Is it a Scheduled Service and if yes how many threads etc...
On Wed, Dec 20, 2017 at 4:25 PM, Vishal Santoshi
wrote:
> Makes sense. Did a
Makes sense. Did a first stab at Using ProcessFunction. The TimeService
exposed by the Context does not have remove timer. Is it primarily b'coz A
Priority Queue is the storage ad remove from a PriorityQueue is expensive
? Trigger Context does expose another version that has removal abilities
so w
Hi Vishal,
it is not guaranteed that add() and onElement() receive the same object,
and even if they do it is not guaranteed that a mutation of the object in
onElement() has an effect. The object might have been serialized and stored
in RocksDB.
Hence, elements should not be modified in onElement(
I guess
https://github.com/apache/flink/blob/7f99a0df669dc73c983913c505c7f72dab3c0a4d/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java#L362
is where We could fashion as to what is emitted. Again for us it seems
natural to use WM to mater
The Trigger in this case would be some CountBased Trigger Again the
motive is the keep the state lean as we desire to search for patterns,
sorted on even time, in the incoming sessionized ( and thus of un
deterministic length ) stream
On Mon, Dec 18, 2017 at 11:26 AM, Vishal Santoshi wr
For example, this would have worked perfect if it did not complain about
MergeableWindow and state. The Session class in this encapsulates the trim
up to watermark behavior ( reduce call after telling it the current WM )
we desire
public class SessionProcessWindow extends ProcessWindowFunction {
Hello Fabian, Thank you for the response.
I think that does not work, as it is the WM of the Window Operator is what
is desired to make deterministic decisions rather than off an operator the
precedes the Window ? This is doable using ProcessWindowFunction using
state but only in the case of non
Hi Vishal,
the Trigger is not designed to augment records but just to control when a
window is evaluated.
I would recommend to use a ProcessFunction to enrich records with the
current watermark before passing them into the window operator.
The context object of the processElement() method gives ac
An addendum
Is the element reference IN in onElement(IN element.. ) in Trigger,
the same as IN the one provided to add(IN value) in Accumulator. It
seems that any mutations to IN in the onElement() is not visible to the
Accumulator that is carrying it as a previous element reference albeit in
th
I want to augment a POJO in Trigger's onElement method, specifically
supply the POJO with the watermark from the TriggerContext. The sequence of
execution is this sequence
1. call to add() in the accumulator for the window and save the POJO
reference in the Accumulator.
2. call to onElement on T
20 matches
Mail list logo
