Hi Sriram,
One sentence of code missed, will update code review board and KIP soon.
For LogSegment and FileMessageSet, must use different constructor
function for existing file and new file, then the code "
channel.position(math.min(channel.size().toInt, end)) " will make sure the
position at end of existing file.
Thanks, Honghai Chen
-----Original Message-----
From: Jay Kreps [mailto:jay.kr...@gmail.com]
Sent: Wednesday, April 22, 2015 5:22 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
My understanding of the patch is that clean shutdown truncates the file back to
it's true size (and reallocates it on startup). Hard crash is handled by the
normal recovery which should truncate off the empty portion of the file.
On Tue, Apr 21, 2015 at 10:52 AM, Sriram Subramanian <
srsubraman...@linkedin.com.invalid> wrote:
> Could you describe how recovery works in this mode? Say, we had a 250
> MB preallocated segment and we wrote till 50MB and crashed. Till what
> point do we recover? Also, on startup, how is the append end pointer
> set even on a clean shutdown? How does the FileChannel end position
> get set to 50 MB instead of 250 MB? The existing code might just work
> for it but explaining that would be useful.
>
> On 4/21/15 9:40 AM, "Neha Narkhede" <n...@confluent.io> wrote:
>
> >+1. I've tried this on Linux and it helps reduce the spikes in append
> >+(and
> >hence producer) latency for high throughput writes. I am not entirely
> >sure why but my suspicion is that in the absence of preallocation,
> >you see spikes writes need to happen faster than the time it takes
> >Linux to allocate the next block to the file.
> >
> >It will be great to see some performance test results too.
> >
> >On Tue, Apr 21, 2015 at 9:23 AM, Jay Kreps <jay.kr...@gmail.com> wrote:
> >
> >> I'm also +1 on this. The change is quite small and may actually
> >>help perf on Linux as well (we've never tried this).
> >>
> >> I have a lot of concerns on testing the various failure conditions
> >> but I think since it will be off by default the risk is not too high.
> >>
> >> -Jay
> >>
> >> On Mon, Apr 20, 2015 at 6:58 PM, Honghai Chen
> >><honghai.c...@microsoft.com>
> >> wrote:
> >>
> >> > I wrote a KIP for this after some discussion on KAFKA-1646.
> >> > https://issues.apache.org/jira/browse/KAFKA-1646
> >> >
> >> >
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-20+-+Enable+log+
> pre
> >>allocate+to+improve+consume+performance+under+windows+and+some+old+Linux+
> >>file+system
> >> > The RB is here: https://reviews.apache.org/r/33204/diff/
> >> >
> >> > Thanks, Honghai
> >> >
> >> >
> >>
> >
> >
> >
> >--
> >Thanks,
> >Neha
>
>
