Disturbed reminds me of the owl from sword in the stone ;) Thats a great one liner - now I am completely disturbed.
Sorry - I've been known to do that - The two results that I say specifically are from the harddisk - those are from the harddisk and are ext4. They are a tad slower than the ramdisk results. Everything else is the ramdisk - and I wasn't sure if thats ext4 or ext3 (and my mind hasn't wandered towards finding the answer yet). I didn't specify when creating it. - Mark Uwe Schindler wrote: > Now I am completely disturbed. Which numbers come from which filesystem? > Ext4 on HDD, tmpfs (which is a filesystem of its own), ext3 on HDD,... > > Uwe > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > >> -----Original Message----- >> From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik >> Seeley >> Sent: Tuesday, September 15, 2009 9:33 PM >> To: java-user@lucene.apache.org >> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1? >> >> Remember to disable CPU frequency scaling when benchmarking... some >> things with IO cause the freq to drop, and when it's CPU bound again >> it takes a while for Linux to scale up the freq again. >> >> For example, on my ubuntu box, ChannelFile went from 100MB/sec to >> 388MB/sec. This effect probably won't be uniform across >> implementations either. >> >> -Yonik >> http://www.lucidimagination.com >> >> On Tue, Sep 15, 2009 at 3:25 PM, Mark Miller <markrmil...@gmail.com> >> wrote: >> >>> I just really I hadn't sent this one. Here are results from the >>> >> harddrive: >> >>> It looks like its closer to the same speed on the hardrive once >>> everything is loaded in the system cache (as you'd expect). SeparateFile >>> was 1200 vs almost 1700 on RAMDISK. ChannelPread looked a lot closer >>> >> though. >> >>> - Mark >>> >>> Tests from harddisk (filesystem cache warmed): >>> >>> config: impl=SeparateFile serial=false nThreads=4 iterations=100 >>> bufsize=1024 poolsize=2 filelen=730554368 >>> answer=-282293977, ms=238230, MB/sec=1226.6370616630988 >>> >>> config: impl=ChannelPread serial=false nThreads=4 iterations=100 >>> bufsize=1024 poolsize=2 filelen=730554368 >>> answer=-282295361, ms=766340, MB/sec=381.3212767179059 >>> >>> >>> Mark Miller wrote: >>> >>>> Michael McCandless wrote: >>>> >>>> >>>>> I don't like that the answer is different... but it's really really >>>>> odd that it's different-yet-almost-the-same. >>>>> >>>>> Mark, were these 4 results on a normal (ext4) filesystem, or tmpfs? >>>>> (Because the top 2 entries of your 4 results match the first set of 2 >>>>> entries you sent... so I'm thinking these 4 were actually tmpfs not >>>>> ext4). >>>>> >>>>> >>>>> >>>> Those 4 were tmpfs - I mention ext4 at the end because I had just given >>>> a feel for the hardrive tests and wanted to note it was from ext4 - the >>>> results are def ramdisk though. >>>> >>>> >>>>> What JRE/OS, linux, kernel versions, and hardware, are you running on? >>>>> >>>>> >>>>> >>>> These are on: >>>> Ubuntu Karmic Koala 9.10, currently updated >>>> java-1.5.0-sun-1.5.0.20 >>>> 2.6.31-10-generic >>>> >>>> RAM: 3.9 Gig >>>> 4 core Intel Core2 duo 2.0GHz >>>> >>>> Slow 5200 rpm laptop drives. >>>> >>>> >>>> >>>>> The gains of SeparateFile over all else are stunning. And, quite >>>>> different from the linux tests I had run under LUCENE-753. Maybe we >>>>> need to revert FSDir.open to return SimpleFSDir again, on non-Windows >>>>> hosts. But then we don't have good concurrency... >>>>> >>>>> Mike >>>>> >>>>> On Tue, Sep 15, 2009 at 2:59 PM, Yonik Seeley >>>>> <yonik.see...@lucidimagination.com> wrote: >>>>> >>>>> >>>>> >>>>>> It's been a while since I wrote that benchmarker... is it OK that the >>>>>> answer is different? Did you use the same test file? >>>>>> >>>>>> -Yonik >>>>>> http://www.lucidimagination.com >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Sep 15, 2009 at 2:18 PM, Mark Miller <markrmil...@gmail.com> >>>>>> >> wrote: >> >>>>>> >>>>>>> The results: >>>>>>> >>>>>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100 >>>>>>> bufsize=1024 poolsize=2 filelen=730554368 >>>>>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938 >>>>>>> >>>>>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100 >>>>>>> bufsize=1024 poolsize=2 filelen=730554368 >>>>>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383 >>>>>>> >>>>>>> config: impl=ChannelPread serial=false nThreads=4 iterations=100 >>>>>>> bufsize=1024 poolsize=2 filelen=730554368 >>>>>>> answer=-282295361, ms=632253, MB/sec=462.19115955163517 >>>>>>> >>>>>>> config: impl=PooledPread serial=false nThreads=4 iterations=100 >>>>>>> bufsize=1024 poolsize=2 filelen=730554368 >>>>>>> answer=-282295361, ms=774664, MB/sec=377.2238637654518 >>>>>>> >>>>>>> ClassicFile was heading for the same fate as ChannelFile. >>>>>>> >>>>>>> >>>>>>> I'll have to check what its like on the file system - but it appears >>>>>>> just ridiculously slower. Even with SeparateFile, All 4 cores are >>>>>>> >> bouncing >> >>>>>>> from 0-12% independently and really favoring the low end of that. >>>>>>> ChannelPread appears no better. >>>>>>> >>>>>>> There are results from other OS's/setups in the JIRA issue. >>>>>>> >>>>>>> I'm using ext4. >>>>>>> >>>>>>> Uwe Schindler wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> How does a conventional file system compare? >>>>>>>> >>>>>>>> ----- >>>>>>>> Uwe Schindler >>>>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen >>>>>>>> http://www.thetaphi.de >>>>>>>> eMail: u...@thetaphi.de >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Mark Miller [mailto:markrmil...@gmail.com] >>>>>>>>> Sent: Tuesday, September 15, 2009 7:15 PM >>>>>>>>> To: java-user@lucene.apache.org >>>>>>>>> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1? >>>>>>>>> >>>>>>>>> Mark Miller wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Indeed - I just ran the FileReaderTest on a Linux tmpfs ramdisk - >>>>>>>>>> >> with >> >>>>>>>>>> SeparateFile all 4 of my cores are immediately pinned and remain >>>>>>>>>> >> so. >> >>>>>>>>>> With ChannelFile, all 4 cores hover 20-30%. >>>>>>>>>> >>>>>>>>>> It would appear it may not be a good idea to use NIOFSDirectory >>>>>>>>>> >> on >> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> ramdisks. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Even still though - it looks like you have a further issue - your >>>>>>>>>> >> Lucene >> >>>>>>>>>> 2.9 old-api results don't use it, and are still not good. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> The quick results: >>>>>>>>> >>>>>>>>> ramdisk: sudo mount -t tmpfs tmpfs /tmp/space -o >>>>>>>>> size=1G,nr_inodes=200k,mode=01777 >>>>>>>>> >>>>>>>>> config: impl=SeparateFile serial=false nThreads=4 iterations=100 >>>>>>>>> bufsize=1024 poolsize=2 filelen=730554368 >>>>>>>>> answer=-282295611, ms=173550, MB/sec=1683.7899579371938 >>>>>>>>> >>>>>>>>> config: impl=ChannelFile serial=false nThreads=4 iterations=100 >>>>>>>>> bufsize=1024 poolsize=2 filelen=730554368 >>>>>>>>> answer=-282295361, ms=1377768, MB/sec=212.09793463050383 >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> - Mark >>>>>>>>> >>>>>>>>> http://www.lucidimagination.com >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ------------------------------------------------------------------ >>>>>>>>> >> --- >> >>>>>>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>>>>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> ------------------------------------------------------------------- >>>>>>>> >> -- >> >>>>>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>>>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> - Mark >>>>>>> >>>>>>> http://www.lucidimagination.com >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -------------------------------------------------------------------- >>>>>>> >> - >> >>>>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> -- >>> - Mark >>> >>> http://www.lucidimagination.com >>> >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >>> >>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
