My understanding of the patch is that clean shutdown truncates the file back to it's true size (and reallocates it on startup). Hard crash is handled by the normal recovery which should truncate off the empty portion of the file.
On Tue, Apr 21, 2015 at 10:52 AM, Sriram Subramanian < srsubraman...@linkedin.com.invalid> wrote: > Could you describe how recovery works in this mode? Say, we had a 250 MB > preallocated segment and we wrote till 50MB and crashed. Till what point > do we recover? Also, on startup, how is the append end pointer set even on > a clean shutdown? How does the FileChannel end position get set to 50 MB > instead of 250 MB? The existing code might just work for it but explaining > that would be useful. > > On 4/21/15 9:40 AM, "Neha Narkhede" <n...@confluent.io> wrote: > > >+1. I've tried this on Linux and it helps reduce the spikes in append (and > >hence producer) latency for high throughput writes. I am not entirely sure > >why but my suspicion is that in the absence of preallocation, you see > >spikes writes need to happen faster than the time it takes Linux to > >allocate the next block to the file. > > > >It will be great to see some performance test results too. > > > >On Tue, Apr 21, 2015 at 9:23 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > >> I'm also +1 on this. The change is quite small and may actually help > >>perf > >> on Linux as well (we've never tried this). > >> > >> I have a lot of concerns on testing the various failure conditions but I > >> think since it will be off by default the risk is not too high. > >> > >> -Jay > >> > >> On Mon, Apr 20, 2015 at 6:58 PM, Honghai Chen > >><honghai.c...@microsoft.com> > >> wrote: > >> > >> > I wrote a KIP for this after some discussion on KAFKA-1646. > >> > https://issues.apache.org/jira/browse/KAFKA-1646 > >> > > >> > > >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-20+-+Enable+log+pre > >>allocate+to+improve+consume+performance+under+windows+and+some+old+Linux+ > >>file+system > >> > The RB is here: https://reviews.apache.org/r/33204/diff/ > >> > > >> > Thanks, Honghai > >> > > >> > > >> > > > > > > > >-- > >Thanks, > >Neha > >
