Now I am completely disturbed. Which numbers come from which filesystem? Ext4 on HDD, tmpfs (which is a filesystem of its own), ext3 on HDD,...
Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik > Seeley > Sent: Tuesday, September 15, 2009 9:33 PM > To: java-user@lucene.apache.org > Subject: Re: lucene 2.9.0RC4 slower than 2.4.1? > > Remember to disable CPU frequency scaling when benchmarking... some > things with IO cause the freq to drop, and when it's CPU bound again > it takes a while for Linux to scale up the freq again. > > For example, on my ubuntu box, ChannelFile went from 100MB/sec to > 388MB/sec. This effect probably won't be uniform across > implementations either. > > -Yonik > http://www.lucidimagination.com > > On Tue, Sep 15, 2009 at 3:25 PM, Mark Miller <markrmil...@gmail.com> > wrote: > > I just really I hadn't sent this one. Here are results from the > harddrive: > > > > It looks like its closer to the same speed on the hardrive once > > everything is loaded in the system cache (as you'd expect). SeparateFile > > was 1200 vs almost 1700 on RAMDISK. ChannelPread looked a lot closer > though. > > > > > > - Mark > > > > Tests from harddisk (filesystem cache warmed): > > > > config: impl=SeparateFile serial=false nThreads=4 iterations=100 > > bufsize=1024 poolsize=2 filelen=730554368 > > answer=-282293977, ms=238230, MB/sec=1226.6370616630988 > > > > config: impl=ChannelPread serial=false nThreads=4 iterations=100 > > bufsize=1024 poolsize=2 filelen=730554368 > > answer=-282295361, ms=766340, MB/sec=381.3212767179059 > > > > > > Mark Miller wrote: > >> Michael McCandless wrote: > >> > >>> I don't like that the answer is different... but it's really really > >>> odd that it's different-yet-almost-the-same. > >>> > >>> Mark, were these 4 results on a normal (ext4) filesystem, or tmpfs? > >>> (Because the top 2 entries of your 4 results match the first set of 2 > >>> entries you sent... so I'm thinking these 4 were actually tmpfs not > >>> ext4). > >>> > >>> > >> Those 4 were tmpfs - I mention ext4 at the end because I had just given > >> a feel for the hardrive tests and wanted to note it was from ext4 - the > >> results are def ramdisk though. > >> > >>> What JRE/OS, linux, kernel versions, and hardware, are you running on? > >>> > >>> > >> These are on: > >> Ubuntu Karmic Koala 9.10, currently updated > >> java-1.5.0-sun-1.5.0.20 > >> 2.6.31-10-generic > >> > >> RAM: 3.9 Gig > >> 4 core Intel Core2 duo 2.0GHz > >> > >> Slow 5200 rpm laptop drives. > >> > >> > >>> The gains of SeparateFile over all else are stunning. And, quite > >>> different from the linux tests I had run under LUCENE-753. Maybe we > >>> need to revert FSDir.open to return SimpleFSDir again, on non-Windows > >>> hosts. But then we don't have good concurrency... > >>> > >>> Mike > >>> > >>> On Tue, Sep 15, 2009 at 2:59 PM, Yonik Seeley > >>> <yonik.see...@lucidimagination.com> wrote: > >>> > >>> > >>>> It's been a while since I wrote that benchmarker... is it OK that the > >>>> answer is different? Did you use the same test file? > >>>> > >>>> -Yonik > >>>> http://www.lucidimagination.com > >>>> > >>>> > >>>> > >>>> On Tue, Sep 15, 2009 at 2:18 PM, Mark Miller <markrmil...@gmail.com> > wrote: > >>>> > >>>> > >>>>> The results: > >>>>> > >>>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100 > >>>>> bufsize=1024 poolsize=2 filelen=730554368 > >>>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938 > >>>>> > >>>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100 > >>>>> bufsize=1024 poolsize=2 filelen=730554368 > >>>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383 > >>>>> > >>>>> config: impl=ChannelPread serial=false nThreads=4 iterations=100 > >>>>> bufsize=1024 poolsize=2 filelen=730554368 > >>>>> answer=-282295361, ms=632253, MB/sec=462.19115955163517 > >>>>> > >>>>> config: impl=PooledPread serial=false nThreads=4 iterations=100 > >>>>> bufsize=1024 poolsize=2 filelen=730554368 > >>>>> answer=-282295361, ms=774664, MB/sec=377.2238637654518 > >>>>> > >>>>> ClassicFile was heading for the same fate as ChannelFile. > >>>>> > >>>>> > >>>>> I'll have to check what its like on the file system - but it appears > >>>>> just ridiculously slower. Even with SeparateFile, All 4 cores are > bouncing > >>>>> from 0-12% independently and really favoring the low end of that. > >>>>> ChannelPread appears no better. > >>>>> > >>>>> There are results from other OS's/setups in the JIRA issue. > >>>>> > >>>>> I'm using ext4. > >>>>> > >>>>> Uwe Schindler wrote: > >>>>> > >>>>> > >>>>>> How does a conventional file system compare? > >>>>>> > >>>>>> ----- > >>>>>> Uwe Schindler > >>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen > >>>>>> http://www.thetaphi.de > >>>>>> eMail: u...@thetaphi.de > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: Mark Miller [mailto:markrmil...@gmail.com] > >>>>>>> Sent: Tuesday, September 15, 2009 7:15 PM > >>>>>>> To: java-user@lucene.apache.org > >>>>>>> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1? > >>>>>>> > >>>>>>> Mark Miller wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> Indeed - I just ran the FileReaderTest on a Linux tmpfs ramdisk - > with > >>>>>>>> SeparateFile all 4 of my cores are immediately pinned and remain > so. > >>>>>>>> With ChannelFile, all 4 cores hover 20-30%. > >>>>>>>> > >>>>>>>> It would appear it may not be a good idea to use NIOFSDirectory > on > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> ramdisks. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> Even still though - it looks like you have a further issue - your > Lucene > >>>>>>>> 2.9 old-api results don't use it, and are still not good. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> The quick results: > >>>>>>> > >>>>>>> ramdisk: sudo mount -t tmpfs tmpfs /tmp/space -o > >>>>>>> size=1G,nr_inodes=200k,mode=01777 > >>>>>>> > >>>>>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100 > >>>>>>> bufsize=1024 poolsize=2 filelen=730554368 > >>>>>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938 > >>>>>>> > >>>>>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100 > >>>>>>> bufsize=1024 poolsize=2 filelen=730554368 > >>>>>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383 > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> - Mark > >>>>>>> > >>>>>>> http://www.lucidimagination.com > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> ------------------------------------------------------------------ > --- > >>>>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >>>>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> ------------------------------------------------------------------- > -- > >>>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >>>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> -- > >>>>> - Mark > >>>>> > >>>>> http://www.lucidimagination.com > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> -------------------------------------------------------------------- > - > >>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org > >>>>> > >>>>> > >>>>> > >>>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >>>> For additional commands, e-mail: java-user-h...@lucene.apache.org > >>>> > >>>> > >>>> > >>>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >>> For additional commands, e-mail: java-user-h...@lucene.apache.org > >>> > >>> > >>> > >> > >> > >> > > > > > > -- > > - Mark > > > > http://www.lucidimagination.com > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
