On Apr 14, 2010, at 10:21, "Herbert Roitblat" <h...@orcatec.com> wrote:
Hi, folks.
I am using PyLucene and doing a lot of get tokens. lucene.py reports
version 2.4.0. It is rpath linux with 8GB of memory. Python is 2.4.
The system indexes 116,000 documents just fine.
Maxheap is '2048m', 64 bit environment.
Then I need to get the tokens from these documents and near the end,
I run
into:
java.lang.OutOfMemoryError: GC overhead limit exceeded
The heap is apparently filling up with each document retrieved and
never getting cleared. I was expecting that it would give me the
information for one document, then clear that and give me the info
for another, etc. I've looked at it with jhat.
I have tried deleting the Python objects that receive any
information from Lucene--no effect.
I have tried reusing the Python objects that receive any information
from Lucene--no effect.
I have tried running the Python garbage collector (it slowed the
program slightly, but generally no effect).
Is there anything else I can do to get the tokens for a document and
make sure that this does not fill up the heap? I need to be able to
run a million or more documents through this and get their tokens.
Could you try a newer version of PyLucene, such as 2.9.2 or 3.0.1 ?
I remember that some string leaks got fixed since 2.4.0 which is quite
old by now.
Andi..
Here is a code snippet.
reader = self.index.getReader()
lReader = reader.get()
searcher = self.index.getSearcher()
lSearcher = searcher.get()
query = lucene.TermQuery(lucene.Term(OTDocument.UID_FIELD_ID,
uid))
hits = list(lSearcher.search(query))
if hits:
hit = lucene.Hit.cast_(hits[0])
tfvs = lReader.getTermFreqVectors(hit.id)
if tfvs is not None: # this happens if the vector is not
stored
for tfv in tfvs: # There's one for each field that
has a TermFreqVector
tfvP = lucene.TermFreqVector.cast_(tfv)
if returnAllFields or tfvP.field in termFields: #
add only asked fields
tFields[tfvP.field] = dict([(t,f) for (t,f)
in zip(tfvP.getTerms(),tfvP.getTermFrequencies()) if f >=minFreq])
else:
# This shouldn't happen, but we just log the error and
march on
self.log.error("Unable to fetch doc %s from index"%(uid))
lReader.close()
lSearcher.close()
lReader is really:
lucene.IndexReader.open(self._store)
I've tried the Lucene list, but no one there has yet come up with a
solution. If filling the heap is a Lucene problem (is it a bug), I
need to look for a way to circumvent that bug.
Thanks,
Herb
