ok great. Thanks !
On 2019/02/04 18:00:16, Fabian Hueske <fhue...@gmail.com> wrote: > Yes, I think that should work. > > Best, Fabian > > Am Mo., 4. Feb. 2019 um 18:35 Uhr schrieb morin.david....@gmail.com < > morin.david....@gmail.com>: > > > Hello Fabian, > > > > Thanks ! > > According to your answers on this post > > https://stackoverflow.com/questions/50340107/order-of-events-with-chained-keyby-calls-on-same-key, > > if I'm right I can use my sort function followed by a keyby and use a > > Window for aggregate these events. And the order will be preserved if I use > > the same Key and the same partionning. I'm right ? > > > > SingleOutputStreamOperator<XXX> sortStream = > > conversionStreamKeyed.process(new > > SortEventFunction()).setParallelism(1).name("Sort events"); > > > > // use the same key > > KeyedStream<XX, String> sortStreamKeyed = sortStream.keyBy((XXX > > event) -> event.getPartitionKey()); > > > > > > sortStreamKeyed.window(TumblingProcessingTimeWindow....setParallelism(1).name("Aggregate > > events"); > > > > Thanks > > David > > > > On 2019/02/04 13:54:14, Fabian Hueske <fhue...@gmail.com> wrote: > > > Hi, > > > > > > A WindowAll is executed in a single task. If you sort the data before the > > > window, the sorting must also happen in a single task, i.e., with > > > parallelism 1. > > > The reasons is that an operator somewhat randomly merges multiple input > > > partitions. So even if each input partition is sorted, the merging will > > > result in out-of-order data. > > > > > > Best, > > > Fabian > > > > > > Am Sa., 2. Feb. 2019 um 17:11 Uhr schrieb morin.david....@gmail.com < > > > morin.david....@gmail.com>: > > > > > > > Hello, > > > > > > > > I use Watermarks and a function to sort events at the end of my > > pipeline. > > > > I've used this tutorial to sort my data: > > > > https://training.da-platform.com/exercises/carSort.html > > > > SingleOutputStreamOperator<XXXX> sortStream = streamKeyed.process(new > > > > SortEventFunction()).. > > > > > > > > Then I want to apply a Window and use AggregateFunction to obtain a > > group > > > > of data. Thus when a trigger is launched, I can push all these data to > > my > > > > backend at the same time (with puts method for Hbase for example) > > > > But the order here must be guaranteed. > > > > Can I use a windowAll on that stream ? > > > > sortStream.windowAll(... > > > > > > > > Thanks in advance > > > > David > > > > > > > > > > > > > > > > > >
