RE: IndexWriter and memory usage

2010-05-06 Thread Woolf, Ross
p alot of memory. Ross can you try the patch I just attached on that issue (merge it w/ the other issues) and see if that fixes it? Thanks. Mike On Thu, Apr 29, 2010 at 11:58 AM, Woolf, Ross wrote: > I ported the patch to 2.9.2 dev but it did not seem to help. Attached is my > port of the p

RE: IndexWriter and memory usage

2010-05-19 Thread Woolf, Ross
en you see the memory growth? Mike On Tue, May 11, 2010 at 2:57 PM, Woolf, Ross wrote: > Still working on some of the things you ask, namely searching without > indexing.  I need to modify our code and the general indexing process takes 2 > hours, so I won't have a quick turn aroun

numDeletedDocs()

2010-06-03 Thread Woolf, Ross
There seems to be a mismatch between the IndexWriter().numDeletedDocs() method as stated in the javadocs supplied in the 2.9.2 download and what is actual. JavaDocs for 2.9.2 as came with the 2.9.2 download numDeletedDocs public int numDeletedDocs()Returns the number of deleted documents. (No

RE: numDeletedDocs()

2010-06-03 Thread Woolf, Ross
03, 2010 2:02 PM To: java-user@lucene.apache.org Subject: Re: numDeletedDocs() Hmm... I don't think IndexWriter has ever had a numDeletedDocs() (w/ no params)? (IndexReader does). Mike On Thu, Jun 3, 2010 at 3:50 PM, Woolf, Ross wrote: > There seems to be a mismatch between the Ind

Using field cache on real time index

2010-06-07 Thread Woolf, Ross
I'm looking for some clarification on the use of field cache in a real time index situation. We are using Lucene in a real time fashion, but we update our reader via IndexReader.reopen() rather than using the IndexWriter.getReader(); After opening a new reader the old reader is closed. In the

MultiPhraseQuery.toString() throws null pointer exception

2010-07-01 Thread Woolf, Ross
In Lucene 2.9.2 (have not checked 3.x) calling MultiPhraseQuery.toString() throws a null pointer exception. Below is very simple code to test this out. import org.apache.lucene.search.MultiPhraseQuery; public class testMPQ { public static void main(String[] args){ Multi

SmartChineseAnalyzer for Traditional Chinese?

2011-03-24 Thread Woolf, Ross
As I look at the api for SmartChineseAnalyzer it indicates it is for Simplified Chinese. Has anyone attempted modifying it for Traditional Chinese? Or does anyone know of any other "smart" analyzer that is geared towards traditional Chinese? Thanks, Ross

Problems with IndexWriter.commit()

2009-07-29 Thread Woolf, Ross
I'm experiencing a problem/unexpected behavior with IndexWriter.commit(). I have an open IndexWriter and I am writing a lot of documents to the index (addDocument). I call a commit() and the data is committed as expected, but now as I continue to write documents to the index I have lost all ca

RE: Problems with IndexWriter.commit()

2009-07-30 Thread Woolf, Ross
IndexWriter.commit() This is certainly not expected. Can you turn on IndexWriter's infoStream and post the result? Mike On Wed, Jul 29, 2009 at 7:03 PM, Woolf, Ross wrote: > I'm experiencing a problem/unexpected behavior with IndexWriter.commit().  I > have an open IndexWriter

UNC speed vs DOS path speed

2009-08-07 Thread Woolf, Ross
On a Windows machine I have noticed that using a UNC path instead of a DOS path when instantiating an index writer causes the performance to slow considerably, even when the UNC is to the same location as DOS path. Is anyone aware of this and why? Is there anything that can be done to improve

Exception invoking MultiPhraseQuery

2010-01-12 Thread Woolf, Ross
I can't invoke MultiPhraseQuery. It produces the error: com.sun.jdi.InvocationException occurred invoking method Here is the code: MultiPhraseQuery mpq = new MultiPhraseQuery(); In the eclipse debugger when I try to inspect mpq after instantiating it shows the error. I'm on Lucene 2.9.1 with J

RE: Exception invoking MultiPhraseQuery

2010-01-12 Thread Woolf, Ross
ects referenced to.. Alternately, you can look for all the Lucene jars on your machine and delete (or move) any old ones. And if none of this helps, can you post the entire stack trace? HTH Erick On Tue, Jan 12, 2010 at 1:28 PM, Woolf, Ross wrote: > I can't invoke MultiPhraseQ

Not getting any highlighter results when using wildcards

2010-02-24 Thread Woolf, Ross
When I use a WildcardQuery with the highlighter, I don't get any fragments back, I get null returned to strBetText. If I just use a term query then it works. TokenStream tokenStream = TokenSources.getTokenStream(indexReader, docId, strFieldName); QueryScorer scorer = new QueryScorer(query, s

UNC speed vs DOS path speed

2010-03-22 Thread Woolf, Ross
On a Windows machine I have noticed that using a UNC path instead of a DOS path when instantiating an index writer causes the performance to slow considerably, even when the UNC is to the same location as DOS path. Is anyone aware of this and know why? Is there anything that can be done to im

IndexWriter and memory usage

2010-04-01 Thread Woolf, Ross
We are seeing a situation where the IndexWriter is using up the Java Heap space and only releases memory for garbage collection upon a commit. We are using the default RAMBufferSize of 16 mb. We are using Lucene 2.9.1. We are set at heap size of 512 mb. We have a large number of documents th

RE: IndexWriter and memory usage

2010-04-02 Thread Woolf, Ross
ening a new one? Mike On Thu, Apr 1, 2010 at 6:58 PM, Woolf, Ross wrote: > We are seeing a situation where the IndexWriter is using up the Java Heap > space and only releases memory for garbage collection upon a commit.   We are > using the default RAMBufferSize of 16 mb.  We are us

RE: IndexWriter and memory usage

2010-04-13 Thread Woolf, Ross
t: Re: IndexWriter and memory usage There is some bugs where the writer data structures retain data after it is flushed. They are committed as of maybe the past week. If you can pull the trunk and try it with your use case, that would be great. On Mon, Apr 12, 2010 at 8:54 AM, Woolf, Ross wrote: >

RE: IndexWriter and memory usage

2010-04-13 Thread Woolf, Ross
as I state in the prior message, I'm hoping these fixes are in a 2.9x branch and hoping for someone to point me to where I can get those fixes to try out. Thanks -Original Message----- From: Woolf, Ross [mailto:ross_wo...@bmc.com] Sent: Tuesday, April 13, 2010 1:29 PM To: java-user@lucene.

RE: IndexWriter and memory usage

2010-04-14 Thread Woolf, Ross
t just a 2.9.x patch for resetting JFlex's zzBuffer? You could also try switching analyzers (eg to WhitespaceAnalyzer) to see if in fact LUCENE-2074 (which affects StandandAnalyzer, since it uses JFlex) is [part of] your problem. Mike On Tue, Apr 13, 2010 at 6:42 PM, Woolf, Ross wrote: >

RE: IndexWriter and memory usage

2010-04-26 Thread Woolf, Ross
Re: IndexWriter and memory usage Run this: svn co https://svn.apache.org/repos/asf/lucene/java/branches/lucene_2_9 lucene.29x Then apply the patch, then, run "ant jar-core", and in that should create the lucene-core-2.9.2-dev.jar. Mike On Wed, Apr 14, 2010 at 1:28 PM, Woolf, Ross wrote