Someone pointed out a particularly easy fix: don't reuse files after a restart. Done. I really like that. Simple. Any chance of this happening any time soon?
On Sun, Apr 28, 2013 at 2:04 AM, Swapnil Ghike <sgh...@linkedin.com> wrote: > @Dan: Upon restart of the broker, if a segment already has data, the > broker resets the firstAppendTime of the segment to the time when that > segment's file handles are being loaded into memory. Thus as you correctly > explained, every time you shut down a broker, the broker essentially > forgets the firstAppendTime. This behavior is present in both 0.7.2 and > 0.8. > > As Jun said, ideally we should set firstAppendTime to the file creation > time. Unfortunately Java nio can provide you provide you that information > only if the underlying filesystem implementation supports the notion of > file creation time. > > Thanks for filing the JIRA, these are good suggestions. > > @Jason: Thanks for pointing out that log.roll.hours is not documented on > the website. This config was added late in 0.7 and we probably forgot to > update the website. We have filed KAFKA-834/KAFKA-835 to update the > configs and other documentation on the website in general. Please let us > know if you see any other missing piece. > > Thanks, > Swapnil > > On 4/27/13 2:36 PM, "Dan Frankowski" <dfran...@gmail.com> wrote: > > >I believe there is a separate watcher thread. The only issue is upon > >restart the broker forgets when the file was created. The behavior I > >described (files can be appended to infinitely) is awkward for us. We have > >tried to work around it. > > > > > >On Fri, Apr 26, 2013 at 10:32 AM, Adam Talaat <atal...@extole.com> wrote: > > > >> I don't know how Kafka's rollover algorithm is implemented, but this is > >> common behavior for other logging frameworks. You would need a separate > >> watcher/scheduled thread to rollover a log file, even if no events were > >> coming in. Logback (and probably log4j, by the same author) dispenses > >>with > >> the watcher thread. Instead, it checks each message as it comes in and > >> decides whether the message should belong in a new file. If it should, a > >> rollover of the old file is triggered and the message is deposited in > >>the > >> new file. But no rollover will occur until a message that belongs in a > >>new > >> file arrives. > >> > >> Cheers, > >> Adam > >> > >> > >> > >> On Fri, Apr 26, 2013 at 9:52 AM, Jason Rosenberg <j...@squareup.com> > >>wrote: > >> > >> > By the way, is there a reason why 'log.roll.hours' is not documented > >>on > >> the > >> > apache configuration page: > >>http://kafka.apache.org/configuration.html ? > >> > > >> > It's possible to find this setting (and several other undocumented > >> > settings) by looking at the source code. I'm just not sure why the > >> > complete set of options is not documented on the site (is it meant to > >>be > >> > experimental?). > >> > > >> > Jason > >> > > >> > > >> > On Fri, Apr 26, 2013 at 8:19 AM, Dan Frankowski <dfran...@gmail.com> > >> > wrote: > >> > > >> > > https://issues.apache.org/jira/browse/KAFKA-881 > >> > > > >> > > Thanks. > >> > > > >> > > > >> > > On Fri, Apr 26, 2013 at 7:40 AM, Jun Rao <jun...@gmail.com> wrote: > >> > > > >> > > > Yes, for low volume topic, the time-based rolling can be > >>imprecise. > >> > Could > >> > > > you file a jira and describe your suggestions there? Ideally, we > >> should > >> > > set > >> > > > firstAppendTime to the file creation time. However, it doesn't > >>seem > >> you > >> > > can > >> > > > get the creation time in java. > >> > > > > >> > > > Thanks, > >> > > > > >> > > > Jun > >> > > > > >> > > > > >> > > > On Thu, Apr 25, 2013 at 11:12 PM, Dan Frankowski > >><dfran...@gmail.com > >> > > >> > > > wrote: > >> > > > > >> > > > > We have high-volume topics and low-volume topics. The problem > >> occurs > >> > > more > >> > > > > often for low-volume topics to be sure. > >> > > > > > >> > > > > But if my hypothesis is correct about why it is happening, here > >>is > >> a > >> > > case > >> > > > > where rolling is longer than an hour, even on a high volume > >>topic: > >> > > > > > >> > > > > - write to a topic for 20 minutes > >> > > > > - restart the broker > >> > > > > - wait for 5 days > >> > > > > - write to a topic for 20 minutes > >> > > > > - restart the broker > >> > > > > - write to a topic for an hour > >> > > > > > >> > > > > The rollover time was now 5 days, 1 hour, 40 minutes. You can > >>make > >> it > >> > > as > >> > > > > long as you want. Does this make sense? > >> > > > > > >> > > > > We would like the rollover time to be no more than an hour, > >>even if > >> > the > >> > > > > broker is restarted, or the topic is low-volume. > >> > > > > > >> > > > > The cleanest way to do that might be to roll over on the hour no > >> > matter > >> > > > > when the file was started. That would be too fast sometimes, but > >> > that's > >> > > > > fine. A second way would be to embed the first append time in > >>the > >> > file > >> > > > > name. A third way (not perfect, but an approximation at least) > >> would > >> > be > >> > > > to > >> > > > > not to write to a segment if firstAppendTime is not defined and > >>the > >> > > > > timestamp on the file is more than an hour old. There are > >>probably > >> > > other > >> > > > > ways. > >> > > > > > >> > > > > What say you? > >> > > > > > >> > > > > > >> > > > > On Thu, Apr 25, 2013 at 9:49 PM, Jun Rao <jun...@gmail.com> > >>wrote: > >> > > > > > >> > > > > > That logic in 0.7.2 seems correct. Basically, firstAppendTime > >>is > >> > set > >> > > on > >> > > > > > first append to a log segment. Then, later on, when a new > >>message > >> > is > >> > > > > > appended and the elapsed time since firstAppendTime is larger > >> than > >> > > the > >> > > > > roll > >> > > > > > time, a new segment is rolled. Is your data constantly being > >> > > produced? > >> > > > > > > >> > > > > > Thanks, > >> > > > > > > >> > > > > > Jun > >> > > > > > > >> > > > > > > >> > > > > > On Thu, Apr 25, 2013 at 12:44 PM, Dan Frankowski < > >> > dfran...@gmail.com > >> > > > > >> > > > > > wrote: > >> > > > > > > >> > > > > > > We are running Kafka 0.7.2. We set log.roll.hours=1. I hoped > >> that > >> > > > meant > >> > > > > > > logs would be rolled every hour, or more. Only, sometimes > >>logs > >> > that > >> > > > are > >> > > > > > > many hours (sometimes days) old have more data added to > >>them. > >> > This > >> > > > > > perturbs > >> > > > > > > our systems for reasons I won't get in to. > >> > > > > > > > >> > > > > > > Have others observed this? Is it a bug? Is there a planned > >>fix? > >> > > > > > > > >> > > > > > > I don't know Scala or Kafka well, but I have proposal for > >>why > >> > this > >> > > > > might > >> > > > > > > happen: upon restart, a broker forgets when its log files > >>have > >> > been > >> > > > > > > appended to ("firstAppendTime"). Then a potentially infinite > >> > amount > >> > > > of > >> > > > > > time > >> > > > > > > later, the restarted broker receives another message for the > >> > > > particular > >> > > > > > > (topic, partition), and starts the clock again. It will then > >> roll > >> > > > over > >> > > > > > that > >> > > > > > > log after an hour. > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > https://svn.apache.org/repos/asf/kafka/branches/0.7/core/src/main/scala/k > >>afka/server/KafkaConfig.scalasays > >> > > > > > > : > >> > > > > > > > >> > > > > > > /* the maximum time before a new log segment is rolled > >>out */ > >> > > > > > > val logRollHours = Utils.getIntInRange(props, > >> "log.roll.hours", > >> > > > 24*7, > >> > > > > > (1, > >> > > > > > > Int.MaxValue)) > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > https://svn.apache.org/repos/asf/kafka/branches/0.7/core/src/main/scala/k > >>afka/log/Log.scalahas > >> > > > > > > maybeRoll, which needs segment.firstAppendTime defined. It > >>also > >> > has > >> > > > > > > updateFirstAppendTime() which says if it's empty, then set > >>it. > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >
