Re: How to avoid huge index files

2009-09-10 Thread Ted Stockwell
009 2:18:35 PM > Subject: Re: How to avoid huge index files > > > Is it possible to upload to GAE an already exist index? My index is data I'm > collecting for long time, and I prefer not to give it up. > >

Re: How to avoid huge index files

2009-09-10 Thread Dvora
Is it possible to upload to GAE an already exist index? My index is data I'm collecting for long time, and I prefer not to give it up. ted stockwell wrote: > > Another alternative is storing the indexes in the Google Datastore, I > think Compass already supports that (though I have not used it

Re: How to avoid huge index files

2009-09-10 Thread Ted Stockwell
Another alternative is storing the indexes in the Google Datastore, I think Compass already supports that (though I have not used it). Also, I have successfully run Lucene on GAE using GaeVFS (http://code.google.com/p/gaevfs/) to store the index in the Datastore. (I developed a Lucene Directory

RE: How to avoid huge index files

2009-09-10 Thread Dvora
28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > >> From: Dvora [mailto:barak.ya...@gmail.com] >> Sent: Thursday, September 10, 2009 1:23 PM >> To: java-user@lucene.apache.org >> Subject: Re: How to avoid huge index files >> >> >> Hi aga

RE: How to avoid huge index files

2009-09-10 Thread Uwe Schindler
MiB limit. Just "virtualize" the files. - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > From: Dvora [mailto:barak.ya...@gmail.com] > Sent: Thursday, September 10, 2009 1:23 PM > To: java-user@lucene.apache.org > Subject:

Re: How to avoid huge index files

2009-09-10 Thread Dvora
Hi again, Can you add some details and guidelines how to implement that? Different files types have different structure, is such spliting doable without knowing Lucene internals? Michael McCandless-2 wrote: > > You're welcome! > > Another, bottoms-up option would be to make a custom Directory

Re: How to avoid huge index files

2009-09-10 Thread Michael McCandless
You're welcome! Another, bottoms-up option would be to make a custom Directory impl that simply splits up files above a certain size. That'd be more generic and more reliable... Mike On Thu, Sep 10, 2009 at 5:26 AM, Dvora wrote: > > Hi, > > Thanks a lot for that, will peforms the experiments a

Re: How to avoid huge index files

2009-09-10 Thread Dvora
Hi, Thanks a lot for that, will peforms the experiments and publish the results. I'm aware to the risk of peformance degredation, but for the pilot I'm trying to run I think it's acceptable. Thanks again! Michael McCandless-2 wrote: > > First, you need to limit the size of segments initially

Re: How to avoid huge index files

2009-09-10 Thread Michael McCandless
First, you need to limit the size of segments initially created by IndexWriter due to newly added documents. Probably the simplest way is to call IndexWriter.commit() frequently enough. You might want to use IndexWriter.ramSizeInBytes() to gauge how much RAM is currently consumed by IndexWriter's

Re: How to avoid huge index files

2009-09-09 Thread Dvora
Hello again, Can someone please comment on that, whether what I'm looking is possible or not? Dvora wrote: > > Hello, > > I'm using Lucene2.4. I'm developing a web application that using Lucene > (via compass) to do the searches. > I'm intending to deploy the application in Google App Engine