Thanks everyone, I learned a lot through this single thread!
On 2019/05/13 07:19:30, Fabian Hueske wrote:
> Hi,
>
> Am Fr., 10. Mai 2019 um 16:55 Uhr schrieb an0 :
>
> > > Q2: after a, map(A), and map(B) would work fine. Assign watermarks
> > > immediatedly afte
;t
make progress. Is this understand correct?
On 2019/05/10 10:38:35, Fabian Hueske wrote:
> Hi,
>
> Again answers below ;-)
>
> Am Do., 9. Mai 2019 um 17:12 Uhr schrieb an0 :
>
> > You are right, thanks. But something is still not totally clear to me.
> > I'll reu
the heap are both copied to a persistent data store (HDFS, S3,
> ...).
>
> I was trying to find the documentation that explains how operator state is
> managed, but couldn't find it.
> I'll create a Jira to fix that.
>
> Best, Fabian
>
> Am Do., 9. Mai 2019
n4.nabble.com/Regarding-implementation-of-aggregate-function-using-a-ProcessFunction-td23473.html#a23531
>
> On Thu, May 9, 2019 at 5:14 AM an0 wrote:
>
> > I want to use ProcessWindowFunction.Context#globalState in my window
> > function. But I don't want to apply ProcessWindo
:p. Is this understanding
correct?
On 2019/05/09 10:01:44, Fabian Hueske wrote:
> Hi,
>
> Please find my response below.
>
> Am Fr., 3. Mai 2019 um 16:16 Uhr schrieb an0 :
>
> > Thanks, but it does't seem covering this rule:
> > --- Quote
> > Watermarks
state is maintained on heap only
> (not in RocksDB).
>
> Best, Fabian
>
>
> Am Mi., 8. Mai 2019 um 20:44 Uhr schrieb an0 :
>
> > I switched to using operator list state. It is more clear. It is also
> > supported by RocksDBKeyedStateBackend, isn't it?
&g
I want to use ProcessWindowFunction.Context#globalState in my window function.
But I don't want to apply ProcessWindowFunction directly to my WindowedStream
because I don't want to buffer all the elements of each window. Currently I'm
using WindowedStream#aggregate(AggregateFunction, ProcessWind
r state size
> should grow very large.
>
> Cheers,
> Till
>
> On Tue, May 7, 2019 at 5:57 PM an0 wrote:
>
> > But I only have one stream, nothing to connect it to.
> >
> > On 2019/05/07 00:15:59, Averell wrote:
> > > From my understanding, havin
But I only have one stream, nothing to connect it to.
On 2019/05/07 00:15:59, Averell wrote:
> From my understanding, having a fake keyBy (stream.keyBy(r => "dummyString"))
> means there would be only one slot handling the data.
> Would a broadcast function [1] work for your case?
>
> Regards,
One way I can think of is:
1. Apply a fake keyBy to the stream and let all the data generate the same key.
2. Use MapState in KeyedProcessFunction on the result of the keyBy above.
But is it a good solution? What's the implication to parallelism? Are there
better ways?
ermarks-in-parallel-streams
>
> Best, Fabian
>
> Am Do., 2. Mai 2019 um 17:48 Uhr schrieb an0 :
>
> > This explanation is exactly what I'm looking for, thanks! Is such an
> > important rule documented anywhere in the official document?
> >
> > On 2019/04
ll propagate its watermarks to B.1 because only B.1 will receive its
> output events.
> However, B.1 will propagate its watermarks to C.1 and C.2 because the
> output of B.1 is partitioned and all C tasks might receive output events
> from B.1.
>
> Best, Fabian
>
> Am Mo
e is no progress beyond this point.
>
> I hope this clarifies it a bit.
>
> Best,
>
> Dawid
>
> On 25/04/2019 16:49, an0 wrote:
> > If my understanding is correct, then why `assignTimestampsAndWatermarks`
> > before `keyBy` works? The `
;
> [1]
> https://www.ververica.com/resources/webinar/webinar/debugging-flink-tutorial
>
> On 22/04/2019 22:55, an0 wrote:
> > Thanks, I feel I'm getting closer to the truth.
> >
> > So parallelism is the cause? Say my parallelism is 2. Does that mean I get
> &g
m
> task, for example only no. 3 BoundedOutOfOrdernessTimestampExtractor.
>
> For before Keyby:
> In your case, the Source and BoundedOutOfOrdernessTimestampExtractors would
> be chained together, which means every
> BoundedOutOfOrdernessTimestampExtractors will receive elements.
>
triggered.
> You could add a shuffle() before the assignTimestampsAndWatermarks in your
> second case and check if the window is triggered. If it could be triggered
> you could check the distribution of elements generated by the source.
>
> Best,
> Guowei
>
>
> an0...@gmail.com 于201
16 matches
Mail list logo
