re, I just think it would be a
good
idea to
keep
the two restore listeners aligned to the highest degree
possible
for
as
we can.
I was actually considering proposing a short KIP with a
new
RecyclingListener (or something) specifically for this
exact
kind of
thing,
since we
current
t;>>>>>>>>>> different costs to a rebalance, including the shuffling
> of
> >>>>>>>>>>>>>>>>>> standby
> >>>>>>>>>>>>>>>> tasks.
>
hare, I just think it would be a
good
idea to
keep
the two restore listeners aligned to the highest degree
possible
for
as
we can.
I was actually considering proposing a short KIP with a
new
RecyclingListener (or something) specifically for this
exact
kind of
thing,
since we
currently
;>>> KIP (:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 3a. I don't like the word "Restore" here, since
> >> Restoration
> >>>>>>>>>>&g
nt is
that
active
tasks can also be recycled into standbys through the same
mechanism.
This
way they can share the SuspendReason enum -- not that
it's
necessary for them to share, I just think it would be a
good
idea to
keep
the two restore listeners aligned to the highest degree
pos
;>>>>>> going
> > > >>>>>>>>>>>> from a situation with an up-to-date standby before the
> > > >>>>>>>>>>>> restart, and
> > > >>>>>>>>>> then
> > > >&
t;>>>>> If a call over the network is required to determine the
> > >>>>>>>> earliestOffset,
> > >>>>>>>>>>>> then this is a "hard no-go" for me, and we will remove it
> > (I'll
> > >&g
t;>>>>>> `startingOffset` is easy to remove as it can be determined
> >>>>>>>>>>>> from the
> >>>>>>>>>> first
> >>>>>>>>>>>> call to `onBatch{Restored/Updated/Processed/Loaded}()`.
> >>>>>>>>>&g
sary for them to share, I just think it would be a good
idea to
keep
the two restore listeners aligned to the highest degree
possible
for
as
we can.
I was actually considering proposing a short KIP with a new
RecyclingListener (or something) specifically for this exact
kind of
thing,
since we
ing proposing a short KIP with a new
RecyclingListener (or something) specifically for this exact
kind of
thing,
since we
currently have literally zero insight into the recycling
process.
It's
practically impossible to tell when a store has been converted
from
active
to
standby, or
are the SuspendReason enum -- not that it's
necessary for them to share, I just think it would be a good
idea to
keep
the two restore listeners aligned to the highest degree possible
for
as
we can.
I was actually considering proposing a short KIP with a new
RecyclingListener (or somet
teners aligned to the highest degree possible
for
as
we can.
I was actually considering proposing a short KIP with a new
RecyclingListener (or something) specifically for this exact
kind of
thing,
since we
currently have literally zero insight into the recycling process.
It's
practic
clear
> >>>>>> to me
> >>>>>>>> and convincing enough, but I'm not quite sure I understand the
> >>>>> example
> >>>>>> at
> >>>>>>>> the end of the Motivation section. How are standby tasks (an
KIP with a new
RecyclingListener (or something) specifically for this exact
kind of
thing,
since we
currently have literally zero insight into the recycling process.
It's
practically impossible to tell when a store has been converted
from
active
to
standby, or vice versa. So having acc
; this
> > > > > > > class should of course be reflected in the KafkaStreams#setXXX
> > API as
> > > > > well
> > > > > > >
> > > > > > > 3b. #onTaskCreated
> > > > > &
> > > > callback method names the same way we do for the class name,
> which in
> > > > this
> > > > > > case would give us #onUpdateStart. Personally I like this better,
> > > > > > but it's ultimately up to you. However, I would push back against
> > > >
store
> > > > > One last thing on this callback -- do we really need both the
> > > > > `earliestOffset` and `startingOffset`? I feel like this might be more
> > > > > confusing than it
> > > > > is helpful (tbh even I'm no
e
> > > > admin client. If so, the ROI on including this parameter seems
> > > > quite low (if not outright negative)
> > > >
> > > > 3c. #onBatchRestored
> > > > If we opt to use the term "update" in place of "restore&
> >
> > > 3d. #onTaskSuspended
> > > Along the same lines as 3b above, #onUpdateSuspended or just
> > > #onRestoreSuspended probably makes more sense for this callback. Also,
> > > I notice the StateRestoreListener passes in the total number of
> records
> > > rest
listener with this
> > SuspendReason enum as well. It would be really useful for both variants of
> > restore listener
> >
> > 4b. Assuming we do 4a, let's rename PROMOTED to RECYCLED -- for standby
> > tasks it means basically the same thing, the point is that ac
e can.
> I was actually considering proposing a short KIP with a new
> RecyclingListener (or something) specifically for this exact kind of thing,
> since we
> currently have literally zero insight into the recycling process. It's
> practically impossible to tell when a store h
teed to notify you when a
state store is recycled whether active or standby, would be amazing.
Thanks for the KIP!
-Sophie "otterStandbyTaskUpdateListener :P" Blee-Goldman
-- Forwarded message -
> From: Colt McNealy
> Date: Tue, Oct 3, 2023 at 12:48 PM
> Subjec
Hi all,
We would like to propose a small KIP to improve the ability of Streams apps
to monitor the progress of their standby tasks through a callback interface.
We have a nearly-working implementation on our fork and are curious for
feedback.
https://cwiki.apache.org/confluence/display/KAFKA/KIP
23 matches
Mail list logo
