Just to clarify too, if the only use case for log-compaction we use is for the __consumer_offsets, we should be ok, correct? I assume compression is not used by default for consumer offsets?
Jason On Fri, Sep 25, 2015 at 12:15 AM, Todd Palino <tpal...@gmail.com> wrote: > For now, that's the way it is. Historically, we've only monitored the lag > for our infrastructure applications. Other users are responsible for their > own checking, typically using the maxlag mbean or some application specific > metric. Besides the core, we've probably got a dozen or so consumers moved > over to Kafka committed offsets at this point. > > Of course, just those apps do cover well over a hundred consumer groups :) > > -Todd > > On Thursday, September 24, 2015, James Cheng <jch...@tivo.com> wrote: > > > > > > On Sep 24, 2015, at 8:11 PM, Todd Palino <tpal...@gmail.com > > <javascript:;>> wrote: > > > > > > Well, in general you can't currently use compressed messages in any > topic > > > that has compaction turned on regardless of whether or not you are > using > > > Kafka-committed offsets. The log compaction thread will die either way. > > > There's only one compression thread for the broker that runs on all > > topics > > > that use compaction. > > > > > > Jason, to address your question, it's probably wise to wait for now. > > > Zookeeper offsets work, so unless it's broke, don't fix it for now. > We're > > > using Kafka-committed offsets at LinkedIn for our mirror makers and our > > > auditor application (both of which are considered infrastructure > > > applications for Kafka), but we're not encouraging other internal users > > to > > > switch over just yet. > > > > > > > Burrow depends on kafka-commited offsets, doesn’t it? I guess that means > > Burrow is only being used to monitor your mirror makers and auditor > > application, then? > > > > -James > > > > > -Todd > > > > > > > > > On Wed, Sep 23, 2015 at 3:21 PM, James Cheng <jch...@tivo.com > > <javascript:;>> wrote: > > > > > >> > > >> On Sep 18, 2015, at 10:25 AM, Todd Palino <tpal...@gmail.com > > <javascript:;>> wrote: > > >> > > >>> I think the last major issue with log compaction (that it couldn't > > handle > > >>> compressed messages) was committed as part of > > >>> https://issues.apache.org/jira/browse/KAFKA-1374 in August, but I'm > > not > > >>> certain what version this will end up in. It may be part of 0.8.2.2. > > >>> > > >>> Regardless, you'll probably be OK now. We've found that once we clean > > >> this > > >>> issue up once it doesn't appear to recur. As long as you're not > writing > > >> in > > >>> compressed messages to a log compacted topic (and that won't happen > > with > > >>> __consumer_offsets, as it's managed by the brokers themselves - it > > would > > >>> only be if you were using other log compacted topics), you're likely > in > > >> the > > >>> clear now. > > >>> > > >> > > >> Todd, > > >> > > >> If I understand your description of the problem, you are saying that > > >> enabling log compaction on a topic with compressed messages can > (will?) > > >> cause the log cleaner to crash when it encounters those compressed > > >> messages. And the death of the cleaner thread will prevent log > > compaction > > >> from running on other topics, even ones that don't have compressed > > messages. > > >> > > >> That means if we have a cluster where we want to use log compaction on > > >> *any* topic, we need to either: > > >> 1) apply https://issues.apache.org/jira/browse/KAFKA-1374 (or upgrade > > to > > >> some version it is applied) > > >> OR > > >> 2) make sure that we don't use compressed messages in *any* topic that > > has > > >> log compaction turned on. > > >> > > >> And, more specifically, if we want to make use of __consumer_offsets, > > then > > >> we cannot use compressed messages in any topic that has compaction > > turned > > >> on. > > >> > > >> Is that right? > > >> -James > > >> > > >>> -Todd > > >>> > > >>> > > >>> On Fri, Sep 18, 2015 at 9:54 AM, John Holland < > > >>> john.holl...@objectpartners.com <javascript:;>> wrote: > > >>> > > >>>> Thanks! > > >>>> > > >>>> I did what you suggested and it worked except it was necessary for > me > > to > > >>>> remove the cleaner-offset-checkpoint file from the data directory > and > > >>>> restart the servers. The log indicates all is well. > > >>>> > > >>>> Do you know what version the fix to this will be in? I'm not looking > > >>>> forward to dealing with this on a reoccurring basis. > > >>>> > > >>>> -John > > >>>> > > >>>> On Fri, Sep 18, 2015 at 8:48 AM Todd Palino <tpal...@gmail.com > > <javascript:;>> wrote: > > >>>> > > >>>>> Yes, this is a known concern, and it should be fixed with recent > > >> commits. > > >>>>> In the meantime, you'll have to do a little manual cleanup. > > >>>>> > > >>>>> The problem you're running into is a corrupt message in the offsets > > >>>> topic. > > >>>>> We've seen this a lot. What you need to do is set the topic > > >> configuration > > >>>>> to remove the cleanup.policy config, and set retention.ms and > > >> segment.ms > > >>>>> to > > >>>>> something reasonably low. I suggest using a value of 3 or 4 times > > your > > >>>>> commit interval for consumers. Then wait until the log segments are > > >>>> reaped > > >>>>> (wait twice as long as the retention.ms you chose, to be safe). > Once > > >>>> this > > >>>>> is done, you can set the topic configuration back the way it was > > >> (remove > > >>>>> segment.ms and retention.ms configs, and set > > cleanup.policy=compact). > > >>>>> Lastly, you'll need to do a rolling bounce of the cluster to > restart > > >> the > > >>>>> brokers (which restarts the log cleaner threads). Technically, you > > only > > >>>>> need to restart brokers where the threads have died, but it's > easier > > to > > >>>>> just restart all of them. > > >>>>> > > >>>>> Keep in mind that when you do this, you are deleting old offsets. > If > > >> your > > >>>>> consumers are all live and healthy, this shouldn't be a problem > > because > > >>>>> they will just continue to commit their offsets properly. But if > you > > >> have > > >>>>> an offline consumer, you'll lose the committed offsets by doing > this. > > >>>>> > > >>>>> -Todd > > >>>>> > > >>>>> > > >>>>> On Fri, Sep 18, 2015 at 5:31 AM, John Holland < > > >>>>> john.holl...@objectpartners.com <javascript:;>> wrote: > > >>>>> > > >>>>>> I've been experiencing this issue across several of our > environments > > >>>> ever > > >>>>>> since we enabled the log cleaner for the __consumer_offsets topic. > > >>>>>> > > >>>>>> We are on version 0.8.2.1 of kafka, using the new producer. All > of > > >> our > > >>>>>> consumers are set to commit to kafka only. > > >>>>>> > > >>>>>> Below is the stack trace in the log I've encountered across > several > > >>>>>> different clusters. A simple restart of kafka will allow > compaction > > >> to > > >>>>>> continue on all of the other partitions but the incorrect one will > > >>>> always > > >>>>>> fail. > > >>>>>> > > >>>>>> Here are the values for it from the kafka-topics --describe > command: > > >>>>>> > > >>>>>> Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 > > >>>>>> Configs:segment.bytes=104857600,cleanup.policy=compact > > >>>>>> > > >>>>>> Are there any recommendations on how to prevent this and the best > > way > > >>>> to > > >>>>>> recover from this exception? This is causing disk space to fill > up > > >>>>> quickly > > >>>>>> on the node. > > >>>>>> > > >>>>>> I did see an open issue that seems very similar to this > > >>>>>> https://issues.apache.org/jira/browse/KAFKA-1641 but this is the > > >>>>>> __consumer_offsets topic which I have not had any part in setting > up > > >>>> nor > > >>>>>> producing to. > > >>>>>> > > >>>>>> [2015-09-18 02:57:25,520] INFO Cleaner 0: Beginning cleaning of > log > > >>>>>> __consumer_offsets-17. (kafka.log.LogCleaner) > > >>>>>> [2015-09-18 02:57:25,520] INFO Cleaner 0: Building offset map for > > >>>>>> __consumer_offsets-17... (kafka.log.LogCleaner) > > >>>>>> [2015-09-18 02:57:25,609] INFO Cleaner 0: Building offset map for > > log > > >>>>>> __consumer_offsets-17 for 46 segments in offset range [468079184, > > >>>>>> 528707475). (kafka.log.LogCleaner) > > >>>>>> [2015-09-18 02:57:25,645] ERROR [kafka-log-cleaner-thread-0], > Error > > >> due > > >>>>> to > > >>>>>> (kafka.log.LogCleaner) > > >>>>>> java.lang.IllegalArgumentException: requirement failed: Last clean > > >>>> offset > > >>>>>> is 468079184 but segment base offset is 0 for log > > >>>> __consumer_offsets-17. > > >>>>>> at scala.Predef$.require(Predef.scala:233) > > >>>>>> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:509) > > >>>>>> at kafka.log.Cleaner.clean(LogCleaner.scala:307) > > >>>>>> at > > >>>>>> > > kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:221) > > >>>>>> at > > >>>>> kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:199) > > >>>>>> at > > >>>>> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) > > >>>>>> [2015-09-18 02:57:25,654] INFO [kafka-log-cleaner-thread-0], > Stopped > > >>>>>> (kafka.log.LogCleaner) > > >>>>>> > > >>>>>> -John > > >>>>>> > > >>>>> > > >>>> > > >> > > >> > > > > > > ________________________________ > > > > This email and any attachments may contain confidential and privileged > > material for the sole use of the intended recipient. Any review, copying, > > or distribution of this email (or any attachments) by others is > prohibited. > > If you are not the intended recipient, please contact the sender > > immediately and permanently delete this email and any attachments. No > > employee or agent of TiVo Inc. is authorized to conclude any binding > > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo > > Inc. may only be made by a signed written agreement. > > >
