Thanks. On Thu, Aug 3, 2017 at 2:11 PM Lukasz Cwik <[email protected]> wrote:
> But the windows can still be processed out of order. > > On Thu, Aug 3, 2017 at 2:10 PM, Lukasz Cwik <[email protected]> wrote: > >> Yes. >> >> On Thu, Aug 3, 2017 at 2:09 PM, Eric Fang <[email protected]> >> wrote: >> >>> Thanks Lukasz. If the stream is infinite, I am assuming you mean by >>> window the stream into pans and sort the bundle for each trigger to an >>> order? >>> >>> Eric >>> >>> On Thu, Aug 3, 2017 at 12:49 PM Lukasz Cwik <[email protected]> wrote: >>> >>>> There is currently no strict ordering which is supported within Apache >>>> Beam (timestamp or not) and any ordering which may be occurring is just a >>>> side effect and not guaranteed in any way. >>>> >>>> Since the smallest unit of work is a bundle containing 1 element, the >>>> only way to get ordering is to make one giant element containing all your >>>> data that needs to be ordered and perform the ordering yourself (e.g >>>> GroupByKey with single dummy key). >>>> >>>> On Thu, Aug 3, 2017 at 12:41 PM, Eric Fang <[email protected]> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> We have a stream of data that's ordered by a timestamp and our use >>>>> case requires us to process the data in order with respect to the previous >>>>> element. For example, we have a stream of true/false ingested from PubSub >>>>> and we want to make sure for each key, a true always follows by a false. >>>>> >>>>> I know from PubSub, the order is not guaranteed, but for the same >>>>> Dataflow job, does the ProcessContext.output guarantee order when >>>>> processElement is called based on event time or process time? From my >>>>> experiment, this assumption seems to hold up but I wonder if this is an >>>>> actual assumption of the system. >>>>> >>>>> In addition, if I key the stream with another key, does the assumption >>>>> still hold? If not, is there any way with Beam to ensure that >>>>> processElement is called in order of some time stamp. >>>>> >>>>> Thanks >>>>> Eric >>>>> >>>>> >>>>> -- >>>>> >>>>> Eric Fang >>>>> >>>>> Stack Labs | 10054 Pasadena Ave, Cupertino, CA 95014 >>>>> >>>>> >>>>> This electronic mail transmission may contain private, confidential >>>>> and privileged information that is for the sole use of the intended >>>>> recipient. If you are not the intended recipient, you are hereby >>>>> notified that any review, dissemination, distribution, archiving, or >>>>> copying of this communication is strictly prohibited. If you >>>>> received this communication in error, please reply to this message >>>>> immediately and delete the original message and the associated reply. >>>>> >>>> >>>> -- >>> >>> Eric Fang >>> >>> Stack Labs | 10054 Pasadena Ave, Cupertino, CA 95014 >>> >>> >>> This electronic mail transmission may contain private, confidential and >>> privileged information that is for the sole use of the intended >>> recipient. If you are not the intended recipient, you are hereby >>> notified that any review, dissemination, distribution, archiving, or >>> copying of this communication is strictly prohibited. If you received >>> this communication in error, please reply to this message immediately >>> and delete the original message and the associated reply. >>> >> >> > -- Eric Fang Stack Labs | 10054 Pasadena Ave, Cupertino, CA 95014 This electronic mail transmission may contain private, confidential and privileged information that is for the sole use of the intended recipient. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, archiving, or copying of this communication is strictly prohibited. If you received this communication in error, please reply to this message immediately and delete the original message and the associated reply.
