How about using MMapDirectory? The speed is similar to RAMDirectory and often even faster when GC goes crazy.
----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: suman.holani [mailto:suman.hol...@zapak.co.in] > Sent: Thursday, September 29, 2011 7:46 AM > To: java-user@lucene.apache.org > Subject: RE: growing segments issue in lucene > > Hi Ian, > Thanks for reply. > > Let me reframe the question > > My aim is to search fast in ever changing indexes. > So I decided onto use RAM directory index.but since it is not persistent .I need > to copy onto FSFirectory , which is nothing but the replica of it for failure > scenarios. > > RAMDirectory is used for adding/deleting and searching.I am doing > optimization on it and then copying .I am assuming that since RAMdirectory is > optimized so copy of it would also be optimized. But when I look at FSDirectory > it is showing me too many files. > > I am not getting why the FSdirectory is growing Also , How can I check if > proper optimsation is done in RAMdirectory . all is in memory or does it stores > segments somewhere in disk also. > > > merge factor is10 > > > > max merge docs is2147483647 > > My concerned is growing segments not getting merged on optimization ( as I > can see in my fsdirectory )irrespective of merge factor. > > > Regards, > suman > > > -----Original Message----- > From: Ian Lea [mailto:ian....@gmail.com] > Sent: Wednesday, September 28, 2011 8:55 PM > To: java-user@lucene.apache.org > Subject: Re: growing segments issue in lucene > > Seems a strange way of doing things. What do you gain by messing around > with RAMDirectory? > > Anyway, you appear to just add docs to the file based index, so surely it will > keep growing? You could try optimizing the file based index but that is unlikely > to gain you a huge amount unless it holds deleted docs. > > > -- > Ian. > > > On Wed, Sep 28, 2011 at 11:32 AM, suman.holani <suman.hol...@zapak.co.in> > wrote: > > Hi , > > > > > > > > I am using lucene 3.0.3. > > > > It seems like either the lucene indexes are not getting optimized > > properly as the files created in lucene index dir is growing with > > every doc added > > > > Please let me know the way to restrict the files and segments created . > > > > > > > > merge factor is10 > > > > max merge docs is2147483647 > > > > > > > > > > > > I am using RAM index to add doc ,optimising it and then copying it to > > FSDirectory. > > > > > > > > indexWriter.optimize(); > > > > indexWriter.commit(); > > > > Directory.copy(RamDir, FSDirectory.open(FILE_INDEX_DIR), > > false); //hoping that optimized indexes will b transferred since m > > optimizing ramdir before copying > > > > > > > > > > > > Files created in FSDirectory as shown as .and these are increasing > > with a index change. > > > > _0.cfs _16.frq _1e.fdt _1k.prx _1s.fnm _1y.tis _4.nrm _c.fdx > _i.tii > > _q.frq segments_l _u.tii > > > > _0.cfx _16.nrm _1e.fdx _1k.tii _1s.frq _20.fdt _4.prx _c.fnm > _i.tis > > _q.nrm segments_m _u.tis > > > > _10.fdt _16.prx _1e.fnm _1k.tis _1s.nrm _20.fdx _4.tii _c.frq > _k.fdt > > _q.prx segments_n _w.fdt > > > > _10.fdx _16.tii _1e.frq _1m.fdt _1s.prx _20.fnm _4.tis _c.nrm > _k.fdx > > _q.tii segments_o _w.fdx > > > > _10.fnm _16.tis _1e.nrm _1m.fdx _1s.tii _20.frq _6.fdt _c.prx > _k.fnm > > _q.tis segments_p _w.fnm > > > > _10.frq _18.fdt _1e.prx _1m.fnm _1s.tis _20.nrm _6.fdx _c.tii > _k.frq > > segments_10 segments_q _w.frq > > > > _10.nrm _18.fdx _1e.tii _1m.frq _1u.fdt _20.prx _6.fnm _c.tis > _k.nrm > > segments_11 segments_r _w.nrm > > > > _10.prx _18.fnm _1e.tis _1m.nrm _1u.fdx _20.tii _6.frq _e.fdt > _k.prx > > segments_12 segments_s _w.prx > > > > _10.tii _18.frq _1g.fdt _1m.prx _1u.fnm _20.tis _6.nrm _e.fdx > _k.tii > > segments_13 segments_t _w.tii > > > > _10.tis _18.nrm _1g.fdx _1m.tii _1u.frq _22.fdt _6.prx _e.fnm > _k.tis > > segments_2 segments_u _w.tis > > > > _12.fdt _18.prx _1g.fnm _1m.tis _1u.nrm _22.fdx _6.tii _e.frq > _m.fdt > > segments_3 segments_v _y.fdt > > > > _12.fdx _18.tii _1g.frq _1o.fdt _1u.prx _22.fnm _6.tis _e.nrm > _m.fdx > > segments_4 segments_w _y.fdx > > > > _12.fnm _18.tis _1g.nrm _1o.fdx _1u.tii _22.frq _8.fdt _e.prx > _m.fnm > > segments_5 segments_x _y.fnm > > > > _12.frq _1a.fdt _1g.prx _1o.fnm _1u.tis _22.nrm _8.fdx _e.tii > _m.frq > > segments_6 segments_y _y.frq > > > > _12.nrm _1a.fdx _1g.tii _1o.frq _1w.fdt _22.prx _8.fnm _e.tis > _m.nrm > > segments_7 segments_z _y.nrm > > > > _12.prx _1a.fnm _1g.tis _1o.nrm _1w.fdx _22.tii _8.frq _g.fdt > _m.prx > > segments_8 _s.fdt _y.prx > > > > _12.tii _1a.frq _1i.fdt _1o.prx _1w.fnm _22.tis _8.nrm _g.fdx > _m.tii > > segments_9 _s.fdx _y.tii > > > > _12.tis _1a.nrm _1i.fdx _1o.tii _1w.frq _2.fdt _8.prx _g.fnm > _m.tis > > segments_a _s.fnm _y.tis > > > > _14.fdt _1a.prx _1i.fnm _1o.tis _1w.nrm _2.fdx _8.tii _g.frq > _o.fdt > > segments_b _s.frq > > > > _14.fdx _1a.tii _1i.frq _1q.fdt _1w.prx _2.fnm _8.tis _g.nrm > _o.fdx > > segments_c _s.nrm > > > > _14.fnm _1a.tis _1i.nrm _1q.fdx _1w.tii _2.frq _a.fdt _g.prx > _o.fnm > > segments_d _s.prx > > > > _14.frq _1c.fdt _1i.prx _1q.fnm _1w.tis _2.nrm _a.fdx _g.tii > _o.frq > > segments_e _s.tii > > > > _14.nrm _1c.fdx _1i.tii _1q.frq _1y.fdt _2.prx _a.fnm _g.tis > _o.nrm > > segments_f _s.tis > > > > _14.prx _1c.fnm _1i.tis _1q.nrm _1y.fdx _2.tii _a.frq _i.fdt > _o.prx > > segments_g _u.fdt > > > > _14.tii _1c.frq _1k.fdt _1q.prx _1y.fnm _2.tis _a.nrm _i.fdx > _o.tii > > segments.gen _u.fdx > > > > _14.tis _1c.nrm _1k.fdx _1q.tii _1y.frq _4.fdt _a.prx _i.fnm > _o.tis > > segments_h _u.fnm > > > > _16.fdt _1c.prx _1k.fnm _1q.tis _1y.nrm _4.fdx _a.tii _i.frq > _q.fdt > > segments_i _u.frq > > > > _16.fdx _1c.tii _1k.frq _1s.fdt _1y.prx _4.fnm _a.tis _i.nrm > _q.fdx > > segments_j _u.nrm > > > > _16.fnm _1c.tis _1k.nrm _1s.fdx _1y.tii _4.frq _c.fdt _i.prx > _q.fnm > > segments_k _u.prx > > > > > > > > Regards > > > > suman > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
