Hi Vincent, As log.roll is every 7 days and retention is 30 days, what you suspect will only happen if that partition received the first message after 23 days. Which is highly unlikely, given there is no partition logic, messages should be uniformly distributed. When I said inflow is not high, I meant relatively. That partition still got 1.7 million messages in 30 days.
Anything else you might suspect? Thanks, Rajat On Tue, May 21, 2019 at 6:28 PM Vincent Maurin <vincent.maurin...@gmail.com> wrote: > Hi Rajat > > I am not 100% sure, but I think the roll logic is based on incoming > messages. > When a message is received, it will compare the timestamp of the first > message in the log vs the timestamp of the incoming message. If it is > greater than log.roll.ms, it will roll the segment > > In your scenario, it could be that your data is not well partioned, so this > given partition doesn't receive a lot of message, resulting in the behavior > you have observed > > Vincent > > On Tue, May 21, 2019 at 1:43 PM Rajat Gangwar <rajatgangwar...@gmail.com> > wrote: > > > Hi, > > > > We have 1 topic with 8 partitions. > > *log.roll* is set to the default value of 7 days. > > *log.segment.bytes* is also set to default : 1 GB. > > *log.retention.hours* is set to 30 days. > > > > Inflow is not much so every 7 days, logs are getting segmented into new > > file even though it never reaches 1 GB. > > But for 1 partition it didn't spawn new segments even after 30 days. And > > since retention policy is for 30 days, it deleted that only segment which > > had all the messages. > > > > We end up losing messages in the production. > > > > *Is this the intended behavior, that until the log file reaches segment > max > > size, it may or may not get split into new segments, no matter what value > > is set for log.roll?* > > > > Thanks, > > Rajat > > >
