Done, I will merge it after travis passes.
Maximilian Michels ezt írta (időpont: 2015. jún. 10., Sze,
15:25):
> Let's mark the method of the environment as deprecated like Aljoscha
> suggested. Then I think we could merge it.
>
> On Wed, Jun 10, 2015 at 2:50 PM, Gyula Fóra wrote:
>
> > Then I s
Let's mark the method of the environment as deprecated like Aljoscha
suggested. Then I think we could merge it.
On Wed, Jun 10, 2015 at 2:50 PM, Gyula Fóra wrote:
> Then I suggest we leave it in the environment along with the other
> checkpointing methods.
>
> I updated my PR so it includes hint
Then I suggest we leave it in the environment along with the other
checkpointing methods.
I updated my PR so it includes hints how to force enable checkpoints (and
the reduced guarantees) when an error is thrown for iterative jobs.
On Wed, Jun 10, 2015 at 2:46 PM, Aljoscha Krettek
wrote:
> We c
We could add a method on the ExecutionConfig but mark it as deprecated
and explain that it will go away once the interplay of iterations,
state and so on is properly figured out.
On Wed, Jun 10, 2015 at 2:36 PM, Ufuk Celebi wrote:
> On 10 Jun 2015, at 14:29, Gyula Fóra wrote:
>
>> Max suggested
On 10 Jun 2015, at 14:29, Gyula Fóra wrote:
> Max suggested that I add this feature slightly hidden to the execution
> config instance.
>
> The problem then is that I either make a public field in the config or once
> again add a method.
>
> Any ideas?
I thought about this as well. The only wa
of the master
> node.
> >>> >> Which
> >>> >> >> is
> >>> >> >> > non existent at the moment.
> >>> >> >> >
> >>> >> >> > So I don't see a reason why a user should not be able
> >> >
>>> >> >> > So I don't see a reason why a user should not be able to choose
>>> >> whether
>>> >> >> he
>>> >> >> > wants state checkpoints for iterations as well.
>>> >> >> &
; >> > In any case this will be used by King for instance, so making it
>> part
>> >> of
>> >> >> > the release would save a lot of work for everyone.
>> >> >> >
>> >> >> > Paris Carbone ezt írta (időpont: 2
iterations as well.
> >> >> >
> >> >> > In any case this will be used by King for instance, so making it
> part
> >> of
> >> >> > the release would save a lot of work for everyone.
> >> >> >
> >> >>
ease would save a lot of work for everyone.
>> >> >
>> >> > Paris Carbone ezt írta (időpont: 2015. jún. 10., Sze,
>> >> > 10:29):
>> >> >
>> >> > >
>> >> > > To continue Gyula's point, for consistent snapshots w
oint, for consistent snapshots we need to
> persist
> >> > the
> >> > > records in transit within the loop and also slightly change the
> >> current
> >> > > protocol since it works only for DAGs. Before going into that
> direction
ent
>> > > protocol since it works only for DAGs. Before going into that direction
>> > > though I would propose we first see whether there is a nice way to make
>> > > iterations more structured.
>> > >
>> > > Paris
>> > > ___
see whether there is a nice way to make
> > > iterations more structured.
> > >
> > > Paris
> > >
> > > From: Gyula Fóra
> > > Sent: Wednesday, June 10, 2015 10:19 AM
> > > To: dev@flink.apache
a Fóra
> > Sent: Wednesday, June 10, 2015 10:19 AM
> > To: dev@flink.apache.org
> > Subject: Re: Force enabling checkpoints for iterative streaming jobs
> >
> > I disagree. Not having checkpointed operators inside the iteration still
> > breaks the guarantees.
>
__
> From: Gyula Fóra
> Sent: Wednesday, June 10, 2015 10:19 AM
> To: dev@flink.apache.org
> Subject: Re: Force enabling checkpoints for iterative streaming jobs
>
> I disagree. Not having checkpointed operators inside the iteration still
> breaks the guarantees.
>
&
make iterations
more structured.
Paris
From: Gyula Fóra
Sent: Wednesday, June 10, 2015 10:19 AM
To: dev@flink.apache.org
Subject: Re: Force enabling checkpoints for iterative streaming jobs
I disagree. Not having checkpointed operators inside the iteration
I disagree. Not having checkpointed operators inside the iteration still
breaks the guarantees.
It is not about the states it is about the loop itself.
On Wed, Jun 10, 2015 at 10:12 AM Aljoscha Krettek
wrote:
> This is the answer I gave on the PR (we should have one place for
> discussing this,
This is the answer I gave on the PR (we should have one place for
discussing this, though):
I would be against merging this in the current form. What I propose is
to analyse the topology to verify that there are no checkpointed
operators inside iterations. Operators before and after iterations can
I agree that for the sake of the above mentioned use cases it is reasonable
to add this to the release with the right documentation, for machine
learning potentially loosing one round of feedback data should not matter.
Let us not block prominent users until the next release on this.
On Wed, Jun
As for people currently suffering from it:
An application King is developing requires iterations, and they need
checkpoints. Practically all SAMOA programs would need this.
It is very likely that the state interfaces will be changed after the
release, so this is not something that we can just add
Hey Gyula,
I understand your reasoning, but I don't think its worth to rush this into the
release.
As you've said, we cannot give precise guarantees. But this is arguably one of
the key requirements for any fault tolerance mechanism. Therefore I disagree
that this is better than not having any
Hey all,
It is currently impossible to enable state checkpointing for iterative
jobs, because en exception is thrown when creating the jobgraph. This
behaviour is motivated by the lack of precise guarantees that we can give
with the current fault-tolerance implementations for cyclic graphs.
This
22 matches
Mail list logo
