Re: Possible implementation for KAFKA-560

2019-07-08 Thread Gwen Shapira
I love it and also have few use-cases. Agree with Harsha that we need a KIP. To cover edge-cases and also clearly define the expected behavior, whether it will be implemented in the admin client or in a tool, etc. Gwen On Mon, Jul 8, 2019 at 1:52 PM Harsha wrote: > > Hi Carlos, >Thi

Re: Possible implementation for KAFKA-560

2019-07-08 Thread Harsha
Hi Carlos, This is a really useful feature and we would like to have it as well. I think high_watermark == log_start_offset is a good starting point to consider but we may also have a case where the topic is empty and the clients producing it may be offline so we might end up garbage

Re: Possible implementation for KAFKA-560

2019-06-23 Thread Carlos Manuel Duclos-Vergara
Hi, Thanks for the answer. Looking at high water mark, then the logic would be to flag the partitions that have high_watermark == log_start_offset In addition, I'm thinking that having the leader fulfill that criteria is enough to flag a partition, maybe check the replicas only if requested by th

Re: Possible implementation for KAFKA-560

2019-06-21 Thread Colin McCabe
I don't think this requires a change in the protocol. It seems like you should be able to use the high water mark to figure something out here? best, Colin On Fri, Jun 21, 2019, at 04:56, Carlos Manuel Duclos-Vergara wrote: > Hi, > > This is an ancient task, but I feel it is still current tod

Possible implementation for KAFKA-560

2019-06-21 Thread Carlos Manuel Duclos-Vergara
Hi, This is an ancient task, but I feel it is still current today (specially since as somebody that deals with a Kafka cluster I know that this happens more often than not). The task is about garbage collection of topics in a sort of automated way. After some consideration I started a prototype i