Lucene is used in the following steps: 1) store interested data in Lucene indexes;
2) search key words against the indexes; 3) write new data into indexes and refresh the reader; 4) use the reader to search key words, and the 2-4 steps repeat. As you see, there are lots of read, update actions. I guess since MMapDir needs to synchronize to a local drive that causes it to be slower. The code is attached: public class YYTLucene { private static Logger logger = Logger.getLogger(YYTLuceneImpl.class); private static FSDirectory indexDir; private static RAMDirectory ramDir; // private static MMapDirectory ramDir; private static IndexWriter iw; private static IndexSearcher is; private static IndexReader ir; private static YYTLucene instance; public static YYTLucene getInstance(String type) { if (instance == null) { instance = new YYTLucene(type); } return instance; } private YYTLucene(String type) { try { indexDir = new NIOFSDirectory(new File(ERConstants.indexFolder1 + "/" + type)); ramDir = new RAMDirectory(indexDir); // ramDir = new MMapDirectory(new File(ERConstants.indexFolder1 + // "/" // + type)); IndexWriterConfig iwConfig = new IndexWriterConfig( ERConstants.version, new LimitTokenCountAnalyzer( ERConstants.analyzer, ERConstants.maxTokenNum)); // iwConfig.setMaxBufferedDocs(ERConstants.maxBufferedDocs); // // iwConfig.setRAMBufferSizeMB(ERConstants.RAMBufferSizeMB); iw = new IndexWriter(ramDir, iwConfig); iw.commit(); ir = IndexReader.open(iw, true); is = new IndexSearcher(ir); } catch (IOException e) { e.printStackTrace(); logger.info("Can't initiate YYTLuceneImpl..."); } } public IndexWriter getIndexWriter() { return iw; } public void setIndexWriter(IndexWriter iw) { YYTLucene.iw = iw; } public IndexSearcher getIndexSearcher() { return is; } public void setIndexSearcher(IndexSearcher is) { YYTLucene.is = is; } public IndexReader getIndexReader() { return ir; } public static void setIndexReader(IndexReader ir) { YYTLucene.ir = ir; } } On Mon, Jun 18, 2012 at 7:32 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > 9 fold improvement using RAMDir over MMapDir is much more than I've > seen (~30-40% maybe) in the past. > > Can you explain how you are using Lucene? > > You may also want to try the CachingRAMDirectory patch on > https://issues.apache.org/jira/browse/LUCENE-4123 > > Mike McCandless > > http://blog.mikemccandless.com > > On Sat, Jun 16, 2012 at 7:18 AM, Cheng <zhoucheng2...@gmail.com> wrote: > > After a number of test, the performance of MMapDirectory is not even > close > > to that of RAMDirectory, in terms of speed. > > > > My application w/ the former can only deal with 10 tasks per round while > it > > could handle over 90 w/ RAMDirectory. > > > > I use the application in Linux. > > > > What can be the reasons? > > > > Thanks. > > > > > > On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <u...@thetaphi.de> wrote: > > > >> This is managed by your operating system. In general OS kernels like > Linux > >> or Windows use all free memory to cache disk accesses. > >> > >> ----- > >> Uwe Schindler > >> H.-H.-Meier-Allee 63, D-28213 Bremen > >> http://www.thetaphi.de > >> eMail: u...@thetaphi.de > >> > >> > >> > -----Original Message----- > >> > From: Cheng [mailto:zhoucheng2...@gmail.com] > >> > Sent: Monday, June 04, 2012 6:10 PM > >> > To: java-user@lucene.apache.org > >> > Subject: Re: RAMDirectory unexpectedly slows > >> > > >> > Can I control the size of ram given to either MMapDirectory or > >> > ByteBufferDirectory? > >> > > >> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <u...@thetaphi.de> > wrote: > >> > > >> > > Hi, > >> > > > >> > > If you are using MMapDirectory or this ByteBufferDirectory (which is > >> > > similar to the first) the used RAM is outside JVM heap, it is in the > >> > > FS cache of the OS kernel. Giving too much memory to the JVM > penalizes > >> > > the OS cache, so give only as much as the App needs. Lucene and the > OS > >> > > kernel will then utilize the remaining memory for caching. > >> > > > >> > > Please read docs of MMapDirectory and inform yourself about mmap in > >> e.g. > >> > > Wikipedia. > >> > > > >> > > Uwe > >> > > -- > >> > > Uwe Schindler > >> > > H.-H.-Meier-Allee 63, 28213 Bremen > >> > > http://www.thetaphi.de > >> > > > >> > > > >> > > > >> > > Cheng <zhoucheng2...@gmail.com> schrieb: > >> > > > >> > > Please shed more insight into the difference between JVM heap size > and > >> > > the memory size used by Lucene. > >> > > > >> > > What I am getting at is that no matter however much ram I give my > >> > > apps, Lucene can't utilize it. Is that right? > >> > > > >> > > What about the ByteBufferDirectory? Can this specific directory > >> > > utilize the 2GB memory I grant to the app? > >> > > > >> > > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen < > >> > > jason.rutherg...@gmail.com> wrote: > >> > > > >> > > > If you want the index to be stored completely in RAM, there is the > >> > > > ByteBuffer directory [1]. Though I do not see the point in putting > >> > > > an index in RAM, it will be cached in RAM regardless in the OS > >> > > > system IO cache. > >> > > > > >> > > > 1. > >> > > > > >> > > > https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja > >> > > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java > >> > > > > >> > > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <zhoucheng2...@gmail.com> > >> > wrote: > >> > > > > My indexes are 500MB+. So it seems like that RAMDirectory is not > >> > > > > good > >> > > for > >> > > > > that big a size. > >> > > > > > >> > > > > My challenge, on the other side, is that I need to update the > >> > > > > indexes > >> > > > very > >> > > > > frequently. So, do you think MMapDirectory is the solution? > >> > > > > > >> > > > > Thanks. > >> > > > > > >> > > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky < > >> > > j...@basetechnology.com > >> > > > >wrote: > >> > > > > > >> > > > >> From the javadoc for RAMDirectory: > >> > > > >> > >> > > > >> "Warning: This class is not intended to work with huge indexes. > >> > > > Everything > >> > > > >> beyond several hundred megabytes will waste resources (GC > >> > > > >> cycles), > >> > > > because > >> > > > >> it uses an internal buffer size of 1024 bytes, producing > millions > >> > > > >> of byte[1024] arrays. This class is optimized for small > >> > > > >> memory-resident indexes. It also has bad concurrency on > >> multithreaded > >> > environments. > >> > > > >> > >> > > > >> It is recommended to materialize large indexes on disk and use > >> > > > >> MMapDirectory, which is a high-performance directory > >> > > > >> implementation > >> > > > working > >> > > > >> directly on the file system cache of the operating system, so > >> > > > >> copying > >> > > > data > >> > > > >> to Java heap space is not useful." > >> > > > >> > >> > > > >> -- Jack Krupansky > >> > > > >> > >> > > > >> -----Original Message----- From: Cheng > >> > > > >> Sent: Monday, June 04, 2012 10:08 AM > >> > > > >> To: java-user@lucene.apache.org > >> > > > >> Subject: RAMDirectory unexpectedly slows > >> > > > >> > >> > > > >> > >> > > > >> Hi, > >> > > > >> > >> > > > >> My apps need to read from and write to some big indexes > >> frequently. > >> > > So I > >> > > > >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB > >> > > > >> memory size. > >> > > > >> > >> > > > >> I notice that the speed of reading and writing unexpectedly > slows > >> > > > >> as > >> > > the > >> > > > >> size of the indexes increases. Since the usage of RAM is less > >> > > > >> than > >> > > 20%, > >> > > > I > >> > > > >> think by default the RAMDirectory doesn't take advantage of the > >> > > memory I > >> > > > >> assigned to JVM. > >> > > > >> > >> > > > >> What are the steps to improve the reading and writing speed of > >> > > > >> RAMDirectory? > >> > > > >> > >> > > > >> Thanks! > >> > > > >> Jeff > >> > > > >> > >> > > > >> > >> > > >_____________________________________________ > >> > > **_____________________________________________ > >> > > **--------- > >> > > > >> To unsubscribe, e-mail: > >> > > > >> java-user-unsubscribe@lucene.**apache.org< > >> > > > java-user-unsubscr...@lucene.apache.org> > >> > > > >> For additional commands, e-mail: > >> > > > >> java-user-help@lucene.apache.**org< > >> > > > java-user-h...@lucene.apache.org> > >> > > > >> > >> > > > >> > >> > > > > >> > > >_____________________________________________ > >> > > > >> > > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> > > > For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > > > > >> > > > > >> > > > >> > > > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >