The update looks good to me. Thanks Damian.
Guozhang
On Fri, Aug 19, 2016 at 11:05 AM, Damian Guy wrote:
> Thanks Jun.
>
> Everyone - i've updated the KIP to use a comma-separated list of
> cleanup.policies as suggested. I know we have already voted on this
> proposal, so are there any objectio
Thanks Jun.
Everyone - i've updated the KIP to use a comma-separated list of
cleanup.policies as suggested. I know we have already voted on this
proposal, so are there any objections to this change?
Thanks,
Damian
On Fri, 19 Aug 2016 at 18:38 Jun Rao wrote:
> Damian,
>
> Yes, using comma-separ
Damian,
Yes, using comma-separated policies does seem more extensible for the
future. If we want to adopt it, it's better to do it as part of this KIP.
Perhaps you can just update the KIP and ask this thread to see if there is
any objections with the change.
Thanks,
Jun
On Fri, Aug 19, 2016 at
Hi Grant,
I apologise - I seemed to have skipped over Joel's email.
It is not something we considered, but seems valid.
I'm not sure if we should do it as part of this KIP or revisit it if/when
we have more cleanup policies?
Thanks,
Damian
On Fri, 19 Aug 2016 at 15:57 Grant Henke wrote:
> Than
Thanks for this KIP Damien.
I know this vote has passed and I think its is okay as is, but I had
similar thoughts as Joel about combining compaction policies.
I'm just wondering if it makes sense to allow specifying multiple
> comma-separated policies "compact,delete" as opposed to
> "compact_and
Thanks for the proposal. I'm +1 overall with a thought somewhat related to
Jun's comments.
While there may not yet be a sensible use case for it, it should be (in
theory) legal to have compact_and_delete with size based retention as well.
I'm just wondering if it makes sense to allow specifying mu
Thanks Jason.
The log retention.ms will be set to a value that greater than the window
retention time. So as windows expire, they eventually get cleaned up by the
broker. It doesn't matter if old windows are around for sometime beyond
their usefulness, more that they do eventually get removed and t
Hey Damian,
That's true, but it would avoid the need to write tombstones for the
expired offsets I guess. I'm actually not sure it's a great idea anyway.
One thing we've talked about is not expiring any offsets as long as a group
is alive, which would require some custom expiration logic. There's
Thanks Jun
On Fri, 12 Aug 2016 at 16:41 Jun Rao wrote:
> Hi, Damian,
>
> I was just wondering if we should disable size-based retention in the
> compact_and_delete mode. So, it sounds like that there could be a use case
> for that. Since by default, the size-based retention is infinite, I think
Hi, Damian,
I was just wondering if we should disable size-based retention in the
compact_and_delete mode. So, it sounds like that there could be a use case
for that. Since by default, the size-based retention is infinite, I think
we can just leave the KIP as it is.
Thanks,
Jun
On Fri, Aug 12,
Hi Jason,
It is still not clear to me how adding a callback would help. You could
remove the periodic task, but you would still need to scan the entire cache
to remove the expired offsets. Am i missing something?
Thanks,
Damian
On 11 August 2016 at 10:00, Damian Guy wrote:
> Hi Jason,
>
> Tha
Hi,
The only concrete example i can think of is a case for limiting disk usage.
Say, i had something like Connect running that was tracking changes in a
database. Downstream i don't really care about every change, i just want
the latest values, so compaction could be enabled. However, the kafka
cl
Jun,
The motivation for this KIP is to handle joins and windows in Kafka streams
better and since Streams supports time-based windows, the KIP suggests
combining time-based deletion and compaction.
It might make sense to do the same for size-based windows, but can you
think of a concrete use case
Hi, Damian,
Thanks for the proposal. It makes sense to use time-based deletion
retention and compaction together, as you mentioned in the KStream.
Is there a use case where we want to combine size-based deletion retention
and compaction together?
Jun
On Thu, Aug 11, 2016 at 2:00 AM, Damian Guy
Hi Jason,
Thanks for your input - appreciated.
1. Would it make sense to use this KIP in the consumer coordinator to
> expire offsets based on the topic's retention time? Currently, we have a
> periodic task which scans the full cache to check which offsets can be
> expired, but we might be able
Hi Damian,
Thanks for the KIP. We have a number of use cases in which we maintain a
materialized cache of a compacted topic. The consumer coordinator, for
example, has a cache of consumer offsets which is populated from the
__consumer_offsets topic. Kafka Connect also uses this pattern for its own
Hi,
We have created KIP 71: Enable log compaction and deletion to co-exist`
https://cwiki.apache.org/confluence/display/KAFKA/KIP-71%3A+Enable+log+compaction+and+deletion+to+co-exist
Please take a look. Feedback is appreciated.
Thank you
17 matches
Mail list logo
