Hey Jun, Thanks for this information. I am not aware of this difference between the purge and delete. Given this difference, I will prefer to the existing name of the purge.
Ismael, please let me if you are strong about using delete. Thanks, Dong On Thu, Mar 2, 2017 at 9:40 AM, Jun Rao <j...@confluent.io> wrote: > Hi, Dong, > > It seems that delete means removing everything while purge means removing a > portion. So, it seems that it's better to be able to distinguish the two? > > Thanks, > > Jun > > On Wed, Mar 1, 2017 at 1:57 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hi all, > > > > I have updated the KIP to include a script that allows user to purge data > > by providing a map from partition to offset. I think this script may be > > convenience and useful, e.g., if user simply wants to purge all data of > > given partitions from command line. I am wondering if anyone object this > > script or has suggestions on the interface. > > > > Besides, Ismael commented in the pull request that it may be better to > > rename PurgeDataBefore() to DeleteDataBefore() and rename PurgeRequest to > > DeleteRequest. I think it may be a good idea because kafka-topics.sh > > already use "delete" as an option. Personally I don't have strong > > preference between "purge" and "delete". I am wondering if anyone object > to > > this change. > > > > Thanks, > > Dong > > > > > > > > On Wed, Mar 1, 2017 at 9:46 AM, Dong Lin <lindon...@gmail.com> wrote: > > > > > Hi Ismael, > > > > > > I actually mean log_start_offset. I realized that it is a better name > > > after I start implementation because "logStartOffset" is already used > in > > > Log.scala and LogCleanerManager.scala. So I changed it from > > > log_begin_offset to log_start_offset in the patch. But I forgot to > update > > > the KIP and specify it in the mailing thread. > > > > > > Thanks for catching this. Let me update the KIP to reflect this change. > > > > > > Dong > > > > > > > > > On Wed, Mar 1, 2017 at 6:15 AM, Ismael Juma <ism...@juma.me.uk> wrote: > > > > > >> Hi Dong, > > >> > > >> When you say "logStartOffset", do you mean "log_begin_offset "? I > could > > >> only find the latter in the KIP. If so, would log_start_offset be a > > better > > >> name? > > >> > > >> Ismael > > >> > > >> On Tue, Feb 28, 2017 at 4:26 AM, Dong Lin <lindon...@gmail.com> > wrote: > > >> > > >> > Hi Jun and everyone, > > >> > > > >> > I would like to change the KIP in the following way. Currently, if > any > > >> > replica if offline, the purge result for a partition will > > >> > be NotEnoughReplicasException and its low_watermark will be 0. The > > >> > motivation for this approach is that we want to guarantee that the > > data > > >> > before purgedOffset has been deleted on all replicas of this > partition > > >> if > > >> > purge result indicates success. > > >> > > > >> > But this approach seems too conservative. It should be sufficient in > > >> most > > >> > cases to just tell user success and set low_watermark to minimum > > >> > logStartOffset of all live replicas in the PurgeResponse if > > >> logStartOffset > > >> > of all live replicas have reached purgedOffset. This is because for > an > > >> > offline replicas to become online and be elected leader, it should > > have > > >> > received one FetchReponse from the current leader which should tell > it > > >> to > > >> > purge beyond purgedOffset. The benefit of doing this change is that > we > > >> can > > >> > allow purge operation to succeed when some replica is offline. > > >> > > > >> > Are you OK with this change? If so, I will go ahead to update the > KIP > > >> and > > >> > implement this behavior. > > >> > > > >> > Thanks, > > >> > Dong > > >> > > > >> > > > >> > > > >> > On Tue, Jan 17, 2017 at 10:18 AM, Dong Lin <lindon...@gmail.com> > > wrote: > > >> > > > >> > > Hey Jun, > > >> > > > > >> > > Do you have time to review the KIP again or vote for it? > > >> > > > > >> > > Hey Ewen, > > >> > > > > >> > > Can you also review the KIP again or vote for it? I have discussed > > >> with > > >> > > Radai and Becket regarding your concern. We still think putting it > > in > > >> > Admin > > >> > > Client seems more intuitive because there is use-case where > > >> application > > >> > > which manages topic or produces data may also want to purge data. > It > > >> > seems > > >> > > weird if they need to create a consumer to do this. > > >> > > > > >> > > Thanks, > > >> > > Dong > > >> > > > > >> > > On Thu, Jan 12, 2017 at 9:34 AM, Mayuresh Gharat < > > >> > > gharatmayures...@gmail.com> wrote: > > >> > > > > >> > >> +1 (non-binding) > > >> > >> > > >> > >> Thanks, > > >> > >> > > >> > >> Mayuresh > > >> > >> > > >> > >> On Wed, Jan 11, 2017 at 1:03 PM, Dong Lin <lindon...@gmail.com> > > >> wrote: > > >> > >> > > >> > >> > Sorry for the duplicated email. It seems that gmail will put > the > > >> > voting > > >> > >> > email in this thread if I simply replace DISCUSS with VOTE in > the > > >> > >> subject. > > >> > >> > > > >> > >> > On Wed, Jan 11, 2017 at 12:57 PM, Dong Lin < > lindon...@gmail.com> > > >> > wrote: > > >> > >> > > > >> > >> > > Hi all, > > >> > >> > > > > >> > >> > > It seems that there is no further concern with the KIP-107. > At > > >> this > > >> > >> point > > >> > >> > > we would like to start the voting process. The KIP can be > found > > >> at > > >> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-107 > > >> > >> > > %3A+Add+purgeDataBefore%28%29+API+in+AdminClient. > > >> > >> > > > > >> > >> > > Thanks, > > >> > >> > > Dong > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > > >> > >> > > >> > >> -- > > >> > >> -Regards, > > >> > >> Mayuresh R. Gharat > > >> > >> (862) 250-7125 > > >> > >> > > >> > > > > >> > > > > >> > > > >> > > > > > > > > >
