Re: Skip replicating ledger after bookkeeper server starts up again

[zhangao]

Thanks for your explanation , I will go through this process, to check how to 
avoid replicating ledger even when copy process gets started.



发自我的iPhone


------------------ Original ------------------
From: Matteo Merli <mme...@apache.org&gt;
Date: Sat,Aug 28,2021 10:57 PM
To: Bookkeeper-Dev <dev@bookkeeper.apache.org&gt;
Subject: Re: Skip replicating ledger after bookkeeper server starts up again



This&nbsp;is&nbsp;already&nbsp;happening,&nbsp;to&nbsp;a&nbsp;certain&nbsp;extent.

When&nbsp;the&nbsp;auto-recovery&nbsp;workers&nbsp;are&nbsp;starting&nbsp;to&nbsp;replicate&nbsp;a&nbsp;ledger&nbsp;that
got&nbsp;marked&nbsp;as&nbsp;under-replicated,&nbsp;they&nbsp;will&nbsp;first&nbsp;check&nbsp;every&nbsp;fragment
of&nbsp;the&nbsp;ledger&nbsp;and&nbsp;will&nbsp;perform&nbsp;a&nbsp;test&nbsp;to&nbsp;decide&nbsp;whether&nbsp;it&nbsp;really
needs&nbsp;to&nbsp;get&nbsp;replicated.

There's&nbsp;a&nbsp;configurable&nbsp;threshold&nbsp;of&nbsp;a&nbsp;percentage&nbsp;of&nbsp;entries&nbsp;to&nbsp;try&nbsp;to
read&nbsp;from&nbsp;each&nbsp;bookie&nbsp;on&nbsp;the&nbsp;ensemble.&nbsp;If&nbsp;all&nbsp;the&nbsp;replicas&nbsp;can&nbsp;be
read,&nbsp;then&nbsp;the&nbsp;ledger&nbsp;is&nbsp;skipped.

By&nbsp;default,&nbsp;we&nbsp;only&nbsp;read&nbsp;the&nbsp;first&nbsp;and&nbsp;last&nbsp;entry&nbsp;on&nbsp;each&nbsp;fragment.&nbsp;If
these&nbsp;are&nbsp;good&nbsp;on&nbsp;all&nbsp;the&nbsp;bookies,&nbsp;then&nbsp;we&nbsp;skip.

The&nbsp;gotcha&nbsp;here&nbsp;is&nbsp;that&nbsp;this&nbsp;check&nbsp;is&nbsp;done&nbsp;before&nbsp;starting&nbsp;replicating
a&nbsp;ledger.&nbsp;If&nbsp;the&nbsp;copy&nbsp;process&nbsp;gets&nbsp;started,&nbsp;it&nbsp;will&nbsp;then&nbsp;continue
until&nbsp;done.&nbsp;A&nbsp;potential&nbsp;improvement&nbsp;here&nbsp;would&nbsp;be&nbsp;to&nbsp;even&nbsp;break&nbsp;the
copy&nbsp;of&nbsp;1&nbsp;ledger.

Matteo
--
Matteo&nbsp;Merli
<mme...@apache.org&gt;

On&nbsp;Sat,&nbsp;Aug&nbsp;28,&nbsp;2021&nbsp;at&nbsp;6:47&nbsp;AM&nbsp;zhangao&nbsp;<gaozhangmin...@qq.com.invalid&gt;&nbsp;wrote:
&gt;
&gt;&nbsp;Maybe&nbsp;the&nbsp;idea&nbsp;remove&nbsp;ledger&nbsp;from&nbsp;underreplicated&amp;nbsp;&nbsp;when&amp;nbsp;&nbsp;bookie&nbsp;starts&nbsp;up&nbsp;has&nbsp;some&nbsp;problem,&nbsp;What&nbsp;important&nbsp;is,&nbsp;replicate&nbsp;legder&nbsp;puts&nbsp;so&nbsp;much&nbsp;heavy&nbsp;loads&nbsp;on&nbsp;cluster.&nbsp;I&nbsp;want&nbsp;to&nbsp;reduce&nbsp;the&nbsp;affect.
&gt;
&gt;
&gt;
&gt;
&gt;&nbsp;suppose&nbsp;there&nbsp;are&nbsp;5&nbsp;copies&nbsp;and&nbsp;4&nbsp;bookies&nbsp;are&nbsp;lost,&nbsp;when&nbsp;one&nbsp;lost&nbsp;bookie&nbsp;is&nbsp;online&nbsp;again,&amp;nbsp;&nbsp;I&nbsp;don't&nbsp;simply&nbsp;mark&nbsp;the&nbsp;ledger&nbsp;replicated.&amp;nbsp;&amp;nbsp;When&nbsp;the&nbsp;replicateWork&nbsp;gets&nbsp;the&nbsp;failed&nbsp;bookies,&nbsp;it&nbsp;will&amp;nbsp;&nbsp;ignore&nbsp;the&nbsp;one&nbsp;which&nbsp;online&nbsp;again.&amp;nbsp;&nbsp;So,&nbsp;this&nbsp;will&nbsp;reduce&nbsp;replicating&nbsp;one&nbsp;copy&nbsp;of&nbsp;data.
&gt;
&gt;
&gt;
&gt;
&gt;
&gt;
&gt;&nbsp;------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
&gt;&nbsp;发件人:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"dev"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<eolive...@gmail.com&amp;gt;;
&gt;&nbsp;发送时间:&amp;nbsp;2021年8月28日(星期六)&nbsp;晚上6:07
&gt;&nbsp;收件人:&amp;nbsp;"Bookkeeper-Dev"<dev@bookkeeper.apache.org&amp;gt;;
&gt;
&gt;&nbsp;主题:&amp;nbsp;Re:&nbsp;Skip&nbsp;replicating&nbsp;ledger&nbsp;after&nbsp;bookkeeper&nbsp;server&nbsp;starts&nbsp;up&nbsp;again
&gt;
&gt;
&gt;
&gt;&nbsp;Candy&nbsp;Rain,
&gt;&nbsp;Thanks&nbsp;for&nbsp;sharing&nbsp;your&nbsp;proposal
&gt;
&gt;&nbsp;Il&nbsp;Ven&nbsp;27&nbsp;Ago&nbsp;2021,&nbsp;04:30&nbsp;Candy&nbsp;Rain&nbsp;<gaozhangmin...@qq.com.invalid&amp;gt;&nbsp;ha
&gt;&nbsp;scritto:
&gt;
&gt;&nbsp;&amp;gt;&nbsp;Describe
&gt;&nbsp;&amp;gt;
&gt;&nbsp;&amp;gt;&nbsp;When&amp;amp;nbsp;&nbsp;bookies&nbsp;in&nbsp;the&nbsp;cluster&nbsp;are&nbsp;down,&nbsp;the&nbsp;auto-recovery&nbsp;gets
&gt;&nbsp;&amp;gt;&nbsp;triggered,&nbsp;but&nbsp;as&nbsp;these&nbsp;come&nbsp;back&nbsp;online,&nbsp;the&nbsp;rereplication&nbsp;worker&nbsp;should
&gt;&nbsp;&amp;gt;&nbsp;Ideally&nbsp;skip&nbsp;the&nbsp;rereplication&nbsp;of&nbsp;the&nbsp;ledgers&nbsp;that&nbsp;are&nbsp;marked&nbsp;as
&gt;&nbsp;&amp;gt;&nbsp;underreplicated.
&gt;
&gt;
&gt;&nbsp;The&nbsp;replicator&nbsp;should&nbsp;be&nbsp;able&nbsp;to&nbsp;listen&nbsp;on&nbsp;zookeeper&nbsp;for&nbsp;changes&nbsp;I'm&nbsp;the
&gt;&nbsp;availability&nbsp;of&nbsp;a&nbsp;bookie.
&gt;&nbsp;Probably&nbsp;there&nbsp;is&nbsp;some&nbsp;space&nbsp;for&nbsp;improvements&nbsp;here.
&gt;
&gt;&nbsp;I&nbsp;don't&nbsp;think&nbsp;it&nbsp;is&nbsp;a&nbsp;good&nbsp;idea&nbsp;to&nbsp;eagerly&nbsp;set&nbsp;the&nbsp;ledger&nbsp;as&nbsp;no&nbsp;more
&gt;&nbsp;underreplicated&nbsp;as&nbsp;soon&nbsp;as&nbsp;one&nbsp;of&nbsp;the&nbsp;bookie&nbsp;in&nbsp;the&nbsp;ensemble&nbsp;comes&nbsp;back
&gt;&nbsp;only.
&gt;&nbsp;The&nbsp;fact&nbsp;that&nbsp;a&nbsp;ledger&nbsp;is&nbsp;underreplicated&nbsp;is&nbsp;to&nbsp;be&nbsp;verified&nbsp;against&nbsp;the
&gt;&nbsp;requested&nbsp;number&nbsp;of&nbsp;replicas,&nbsp;as&nbsp;it&nbsp;is&nbsp;not&nbsp;enough&nbsp;that&nbsp;one&nbsp;bookie&nbsp;is&nbsp;running
&gt;
&gt;&nbsp;Regards
&gt;&nbsp;Enrico
&gt;
&gt;
&gt;&nbsp;But&nbsp;the&nbsp;ledgers&nbsp;are&nbsp;rereplicated&nbsp;instead
&gt;&nbsp;&amp;gt;
&gt;&nbsp;&amp;gt;
&gt;&nbsp;&amp;gt;
&gt;&nbsp;&amp;gt;&nbsp;Expected&nbsp;behavior
&gt;&nbsp;&amp;gt;
&gt;&nbsp;&amp;gt;&nbsp;Ideally&nbsp;as&nbsp;and&nbsp;when&nbsp;the&nbsp;bookies&nbsp;come&nbsp;up&nbsp;the&nbsp;ledgers&nbsp;marked&nbsp;as
&gt;&nbsp;&amp;gt;&nbsp;underreplicated&nbsp;would&nbsp;be&nbsp;read&nbsp;by&nbsp;the&nbsp;rereplicaton&nbsp;worker&nbsp;and&nbsp;from&nbsp;the
&gt;&nbsp;&amp;gt;&nbsp;metadata&nbsp;the&nbsp;worked&nbsp;should&nbsp;skip&nbsp;these&nbsp;as&nbsp;the&nbsp;bookies&nbsp;are&nbsp;available.