I created these FLIPs a while back, sorry for being late to this discussion but
I can try to elaborate.
The idea from FLIP-16 is proven to be correct [1] (see chapter 3) and I think
it is the only way to go but I have been in favour of providing channel
implementations with checkpoint behaviour
prototype please do PM me!
Paris
[1]
https://www.slideshare.net/FlinkForward/flink-forward-berlin-2018-paris-carbone-stream-loops-on-flink-reinventing-the-wheel-for-the-streaming-era
> On 29 Sep 2018, at 20:51, Christian Lehner
> wrote:
>
> Hi all,
>
>
> if you don'
Hey!
I would recommend against using iterations with windows for that problem at the
moment.
Alongside loop scoping and backpressure that will be addressed by FLIP-15 [1] I
think you also need the notion of stream supersteps, which is experimental work
in progress for now, from my side at least
Hey,
Indeed checkpointing iterations and dealing with closed sources are orthogonal
issues, that is why the latter is not part of FLIP-15. Though, you kinda need
both to have meaningful checkpoints for jobs with iterations.
One has to do with correctness (checkpointing strongly connected compone
Unless I got this wrong, if he meant relaxing FIFO processing per
channel/stream partition then Robert is absolutely right.
On 23 Mar 2017, at 12:28, Paris Carbone mailto:par...@kth.se>>
wrote:
I think what Theo meant is to allow for different: high/low priority on
different channels (o
I think what Theo meant is to allow for different: high/low priority on
different channels (or data streams per se) for n-ary operators such as
ConnectedStream binary maps, loops etc.. not to change the sequence of events
within channels I guess.
This does not violate the FIFO channel assumptio
Hi Nancy,
Flink’s vanilla rollback recovery mechanism restarts computation from a global
checkpoint thus sink duplicates (job output) can occur no matter how many sinks
are declared; the whole computation in the failed execution graph will roll
back.
cheers
Paris
> On 5 Jan 2017, at 14:24,
+1 for the references
Imho the most relevant scientific publication related to the current
state-of-the-art of Flink is the first one cited by Dominik (IEEE Bulletin). So
it makes sense to cite that one.
However, Hanna, if you are also interested about prior work at TU Berlin that
bootstrapped
Very interesting!
Thank you for sharing Andrew!
Paris
> On Oct 13, 2016, at 11:00 AM, Andrew Ge Wu wrote:
>
> Hi guys
>
> I just published my code to maven central, open source ofc.
> I try to make this as generic as possible.
> If you are interested, please try it out, and help me to improv
This is possibly related to the way the queue between StreamIterationTail and
Head is currently implemented.
I think this part is a bit prone to records loss when things get wacky and
backpressure kicks in (but at least it avoids deadlocks, right?).
I don’t have the time availability to look int
Hey Do,
I think that more sophisticated samplers could make a better fit in the ML
library and not in the core API but I am not very familiar with the milestones
there.
Maybe the maintainers of the batch ML library could check if sampling
techniques could be useful there I guess.
Paris
> On 1
:abhis...@tetrationanalytics.com>> wrote:
I was wondering how checkpoints can be async? Because your state is constantly
mutating. You probably need versioned state, or immutable data structs?
-Abhishek-
On May 19, 2016, at 11:14 AM, Paris Carbone
mailto:par...@kth.se>> wrote:
Hi Stavros,
Currently
e data structs?
-Abhishek-
On May 19, 2016, at 11:14 AM, Paris Carbone
mailto:par...@kth.se>> wrote:
Hi Stavros,
Currently, rollback failure recovery in Flink works in the pipeline level, not
in the task level (see Millwheel [1]). It further builds on repayable stream
logs (i.e. Kafka), th
passes through operators
you issue a warning or fail the pipeline and return an exception to the driver?
On Thu, May 19, 2016 at 9:30 PM, Paris Carbone
mailto:par...@kth.se>> wrote:
In that case, typically a timeout invalidates the whole snapshot (all states
for the same epoch) until e
You start a new
checkpoint for that one? or a global new checkpoint for the rest of N tasks as
well?
On Thu, May 19, 2016 at 9:21 PM, Paris Carbone
mailto:par...@kth.se>> wrote:
Regarding your last question,
If a checkpoint expires it just gets invalidated and a new complete checkpoint
. 3cp does not need assumptions i think, but engineering is more
practical (it should be) and a different story in general.
The [2] mentions also the assumptions...
Best,
Stavros
On Thu, May 19, 2016 at 9:14 PM, Paris Carbone
mailto:par...@kth.se>> wrote:
Hi Stavros,
Currently, roll
Regarding your last question,
If a checkpoint expires it just gets invalidated and a new complete checkpoint
will eventually occur that can be used for recovery. If I am wrong, or
something has changed please correct me.
Paris
On 19 May 2016, at 20:14, Paris Carbone mailto:par...@kth.se
Hi Stavros,
Currently, rollback failure recovery in Flink works in the pipeline level, not
in the task level (see Millwheel [1]). It further builds on repayable stream
logs (i.e. Kafka), thus, there is no need for 3pc or backup in the pipeline
sources. You can also check this presentation [2] w
Some people might find my slides on the FT fundamentals from last summer
interesting. If you like it feel free to include it.
http://www.slideshare.net/ParisCarbone/tech-talk-google-on-flink-fault-tolerance-and-ha
Paris
On 04 Apr 2016, at 11:33, Ufuk Celebi mailto:u...@apache.org>>
wrote:
Dea
This is not a bad take. It still makes a few assumptions
1) the output checkpoints the id of the last *known* ID that was *persisted* in
kafka (not just pushed)
2) we assume deterministic tuple order, as Stephan pointed out
On 05 Feb 2016, at 13:41, Niels Basjes
mailto:ni...@basjes.nl>> wrote:
you tell us more?
> On 05 Feb 2016, at 13:33, Paris Carbone wrote:
>
> That would be good indeed. I just learned about it from Stephan mentioned. It
> sounds correct to me along the lines but it would be nice to see the details.
>
>> On 05 Feb 2016, at 13:32, Ufuk Celebi wro
That would be good indeed. I just learned about it from Stephan mentioned. It
sounds correct to me along the lines but it would be nice to see the details.
> On 05 Feb 2016, at 13:32, Ufuk Celebi wrote:
>
>
>> On 05 Feb 2016, at 13:28, Paris Carbone wrote:
>>
>>
Hi Gabor,
The sinks should aware that the global checkpoint is indeed persisted before
emitting so they will have to wait until they are notified for its completion
before pushing to Kafka. The current view of the local state is not the actual
persisted view so checking against is like relying
Hi Wepngong,
This is an interesting proposal. There are indeed many streaming optimisations
out there but as Gyula said we should focus on a few and engineer them in a
nice way. Perhaps for the time being it makes sense to focus on a streaming job
graph optimiser that basically applies optimisa
24 matches
Mail list logo
