Yes, I think that should work.

Best, Fabian

Am Mo., 4. Feb. 2019 um 18:35 Uhr schrieb morin.david....@gmail.com <
morin.david....@gmail.com>:

> Hello Fabian,
>
> Thanks !
> According to your answers on this post
> https://stackoverflow.com/questions/50340107/order-of-events-with-chained-keyby-calls-on-same-key,
> if I'm right I can use my sort function followed by a keyby and use a
> Window for aggregate these events. And the order will be preserved if I use
> the same Key and the same partionning. I'm right ?
>
>         SingleOutputStreamOperator<XXX> sortStream =
> conversionStreamKeyed.process(new
> SortEventFunction()).setParallelism(1).name("Sort events");
>
>        // use the same key
>         KeyedStream<XX, String> sortStreamKeyed = sortStream.keyBy((XXX
> event) -> event.getPartitionKey());
>
>
> sortStreamKeyed.window(TumblingProcessingTimeWindow....setParallelism(1).name("Aggregate
> events");
>
> Thanks
> David
>
> On 2019/02/04 13:54:14, Fabian Hueske <fhue...@gmail.com> wrote:
> > Hi,
> >
> > A WindowAll is executed in a single task. If you sort the data before the
> > window, the sorting must also happen in a single task, i.e., with
> > parallelism 1.
> > The reasons is that an operator somewhat randomly merges multiple input
> > partitions. So even if each input partition is sorted, the merging will
> > result in out-of-order data.
> >
> > Best,
> > Fabian
> >
> > Am Sa., 2. Feb. 2019 um 17:11 Uhr schrieb morin.david....@gmail.com <
> > morin.david....@gmail.com>:
> >
> > > Hello,
> > >
> > > I use Watermarks and a function to sort events at the end of my
> pipeline.
> > > I've used this tutorial to sort my data:
> > > https://training.da-platform.com/exercises/carSort.html
> > > SingleOutputStreamOperator<XXXX> sortStream = streamKeyed.process(new
> > > SortEventFunction())..
> > >
> > > Then I want to apply a Window and use AggregateFunction to obtain a
> group
> > > of data. Thus when a trigger is launched, I can push all these data to
> my
> > > backend at the same time (with puts method for Hbase for example)
> > > But the order here must be guaranteed.
> > > Can I use a windowAll on that stream ?
> > > sortStream.windowAll(...
> > >
> > > Thanks in advance
> > > David
> > >
> > >
> > >
> >
>

Reply via email to