Sorting by score down to the second will use a lot of memory. Can you make it less granular? And I think that switching that field to a NumericField will give you some savings - this has come up before but I can't remember the details. I'm sure someone else will.
-- Ian. On Tue, Apr 27, 2010 at 2:49 PM, Samarendra Pratap <samarz...@gmail.com> wrote: > Hi Ian. Thanks for the points > > Here are my answers - > > 1. Our default option is sort by score, however almost 8% of searches use > sorting on a field (yyyymmddHHMMSS). This field is indexed as string (not as > NumericField or DateField). > > 2. We are opening readers at the time of starting the application, but > Searchers are opened for every request choosing the right readers. A few > days back I got an answer on this very forum that reopening a searcher is > not as big issue as reopening a reader. However we are *not closing Searcher > *s after request is served. Should that cause memory problem? Shouldn't GC > handle that all as the reference of Searcher becomes inaccessible? > > 3. I can't do a profiling on production servers but running "top" shows that > immediately starting search server resident memory (RES) starts from 500m > and Virtual memory (VIRT) is around 5400m. It grows to RES => 1g in 5 > minutes and 4g within half an hour while VIRT remains same. > > 4. Right! I'll be doing a memory profiling. I hope I'll get some hints from > there too. > > Do above points (specially # 2) indicate towards anything which I can do > besides profiling? > > > > On Tue, Apr 27, 2010 at 6:53 PM, Ian Lea <ian....@gmail.com> wrote: > >> There is no simple answer. However your app does sound to be using >> rather a lot of memory for what you describe as simple searches. >> >> Are you using lucene sorting? That can use lots of memory. How are >> you using/reusing searchers/readers? Having multiple ones open, or >> failing to close old ones, will use more memory. Does memory usage >> grow then stabilize or keep on growing? >> >> A memory profiler/heap dump could tell you what is really using all the >> space. >> >> >> -- >> Ian. >> >> On Tue, Apr 27, 2010 at 1:51 PM, Samarendra Pratap <samarz...@gmail.com> >> wrote: >> > Hi. >> > I am searching for some guidance on right memory options for my Search >> > Server application. How much memory a lucene based application should be >> > given? >> > >> > Till a few days back I was running my search server on java 1.4 with >> memory >> > options "-Xmx3600m" which was running quite fine. After upgrading the JVM >> to >> > *java 6* we noticed a few times that our application fell idle (perhaps >> > hung). It was not serving the requests although the port was open. I >> changed >> > the "-Xmx" from 3600m to 5000m. >> > Currently it is running OK but this created a curiosity in my mind that >> how >> > to find the right memory size for a lucene based application (or any java >> > application). I believe it heavily depends on index size but how do I >> > calculate it (at least approximately) without hit & trial? >> > >> > The details about my application and server are following >> > >> > *[ system configuration ]* >> > # uname -a >> > Linux xxx 2.6.17-13.2.xxx.finalsmp #1 SMP Wed May 9 17:27:56 IST 2007 >> x86_64 >> > x86_64 x86_64 GNU/Linux >> > >> > *[ java version ]* >> > # /usr/lib/jre1.6.0_20/bin/java -version >> > java version "1.6.0_20" >> > Java(TM) SE Runtime Environment (build 1.6.0_20-b02) >> > Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode) >> > >> > *[ application command line ]* >> > # java -Xmx5000m -server -classpath >> > lib/lucene-core-2.9.2.jar:lib/mysql-connector-j >> > ava-3.1.11-bin.jar:lib/search.jar com.xxx.xxx.SearchServer >> conf/search.conf >> > >> > *[ memory information ]* >> > * >> > # cat /proc/meminfo >> > MemTotal: 8156660 kB >> > MemFree: 46924 kB >> > Buffers: 53768 kB >> > Cached: 3194224 kB >> > SwapCached: 60 kB >> > Active: 6086272 kB >> > Inactive: 919276 kB >> > SwapTotal: 2096472 kB >> > SwapFree: 2095432 kB >> > Dirty: 1148 kB >> > Writeback: 0 kB >> > AnonPages: 3756496 kB >> > Mapped: 24800 kB >> > Slab: 1058124 kB >> > SReclaimable: 15144 kB >> > SUnreclaim: 1042980 kB >> > PageTables: 12232 kB >> > NFS_Unstable: 0 kB >> > Bounce: 0 kB >> > CommitLimit: 6174800 kB >> > Committed_AS: 4588900 kB >> > VmallocTotal: 34359738367 kB >> > VmallocUsed: 264904 kB >> > VmallocChunk: 34359473387 kB >> > >> > More info* >> > Search Server application is running on a few servers each of which >> contains >> > same of copy of index. >> > Total size of index: 23 GB >> > Total Documents in index: 18,000,000 >> > Maximum fields per index: 56 (all analyzed, 4 small fields stored, field >> > "contents" is the largest one, created from a text of as large as 5 - 8 >> KB) >> > Query frequency: 1 query per second per server (in peak hours) >> > Queries are taking around 800 - 1000 ms per query >> > >> > *Other memory related things in Search application* >> > There is really nothing else which will consume noticeable memory. No >> > database connection is made. The application simply returns the IDs from >> the >> > index based on the query. >> > >> > Any help on choosing right memory option is much appreciated. >> > >> > -- >> > Regards, >> > Samar >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> > > > -- > Regards, > Samar > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
