t; > > > >>> > > standing operators that gather and store data as the
> stream
> > > > > evolves
> > > > > > >>> > (unlike
> > > > > > >>> > > in the dataset api). Many programs, like windowing,
>
> (unlike
> > > > > >>> > > in the dataset api). Many programs, like windowing,
> sampling
> > > etc
> > > > > hold
> > > > > >>> the
> > > > > >>> > > state in the form of past data. And withou
time these programs will break or have unnecessary copies.
> > > > >>> > >
> > > > >>> > > This is why I think immutability should be the default so we
> > can
> > > > have
> > > > >>> a
> > > > >
gt; >>> > > clear dataflow model with immutable streams.
> > > >>> > >
> > > >>> > > I see absolutely no reason why we cant have the non-copy
> version
> > > as an
> > > >>> > > optiona
; a
>> > >>> > > clear dataflow model with immutable streams.
>> > >>> > >
>> > >>> > > I see absolutely no reason why we cant have the non-copy version
>> > as an
>> > >>> >
setting for the users.
> > >>> > >
> > >>> > >
> > >>> > > On Wed, May 20, 2015 at 2:21 PM, Paris Carbone
> > wrote:
> > >>> > >
> > >>> > > > @stephan I see your point. If we assum
>> > >
> >>> > > > @stephan I see your point. If we assume that operators do not
> hold
> >>> > > > references in their state to any transmitted records it works
> fine.
> >>> We
> >>> > > > therefore need to
t;> > > > @stephan I see your point. If we assume that operators do not hold
>>> > > > references in their state to any transmitted records it works fine.
>>> We
>>> > > > therefore need to make this clear to the users. I need to ch
r integrations as well that
>> > assume
>> > > > immutability. For example in SAMOA there are often local metric
>> objects
>> > > > that are being constantly mutated and simply forwarded periodically
>> to
>> > > > other (possibly
rs. I need to check if
> that
> > > > would break semantics in SAMOA or other integrations as well that
> > assume
> > > > immutability. For example in SAMOA there are often local metric
> objects
> > > > that are being constantly mutated and simply forwarded p
en local metric objects
> > > that are being constantly mutated and simply forwarded periodically to
> > > other (possibly chained) operators that need to evaluate them.
> > >
> > > ________
> > > From: Gyula Fóra
> > &g
periodically to
> > other (possibly chained) operators that need to evaluate them.
> >
> >
> > From: Gyula Fóra
> > Sent: Wednesday, May 20, 2015 2:06 PM
> > To: dev@flink.apache.org
> > Subject: Re: [DISCUSS] R
> From: Gyula Fóra
> Sent: Wednesday, May 20, 2015 2:06 PM
> To: dev@flink.apache.org
> Subject: Re: [DISCUSS] Re-add record copy to chained operator calls
>
> "Copy before putting it into a window buffer and any other group buffer."
>
> Exactl
: dev@flink.apache.org
Subject: Re: [DISCUSS] Re-add record copy to chained operator calls
"Copy before putting it into a window buffer and any other group buffer."
Exactly my point. Any stateful operator should be able to implement
something like this without having to worry about c
"Copy before putting it into a window buffer and any other group buffer."
Exactly my point. Any stateful operator should be able to implement
something like this without having to worry about copying the object (and
at this point the user would need to know whether it comes from the network
to avo
I am curious why the copying is actually needed.
In the batch API, we chain and do not copy and it is rather predictable.
The cornerpoints of that design is to follow these rules:
1) Objects read from the network or any buffer are always new objects.
That comes naturally when they are deseriali
Yes, in fact I anticipated this. There is one central place where we
can insert a copy step, in OperatorCollector in OutputHandler.
On Wed, May 20, 2015 at 11:17 AM, Paris Carbone wrote:
> I guess it was not intended ^^.
>
> Chaining should be transparent and not break the correct/expected behavi
I guess it was not intended ^^.
Chaining should be transparent and not break the correct/expected behaviour.
Paris?
On 20 May 2015, at 11:02, Márton Balassi wrote:
+1 for copying.
On May 20, 2015 10:50 AM, "Gyula Fóra" wrote:
Hey,
The latest streaming operator rework removed the copying of
+1 for copying.
On May 20, 2015 10:50 AM, "Gyula Fóra" wrote:
> Hey,
>
> The latest streaming operator rework removed the copying of the outputs
> before passing them to chained operators. This is a major break for the
> previous operator semantics which guaranteed immutability.
>
> I think this
19 matches
Mail list logo
