Reg: Housekeeping Lucene index files

2021-11-10 Thread sundar saba
Hi All, I am facing the problem lucene index files occupy more memory space in my machine. So I want to purge lucene index files automatically based on the configured time period. Is there a way to do this through any of the lucene methods,class or purging policy configuration. You

Warming up index files via cat to make it in memory index

2021-03-25 Thread baris . kazar
Hi,-  This new thread is the continuation of previous thread back in Feb 2021: Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory) May i mention that i cat'ed *fdt files (largest index files among 98 index files generated) by directing to new files so that these

RE: Un-used index files are not getting released

2017-05-12 Thread Siraj Haider
-la : 236 lsof : 79 -- Regards -Siraj Haider (212) 306-0154 -Original Message- From: Chris Hostetter [mailto:hossman_luc...@fucit.org] Sent: Thursday, May 11, 2017 1:34 PM To: java-user@lucene.apache.org Cc: ian@gmail.com Subject: RE: Un-used index files are not getting released

RE: Un-used index files are not getting released

2017-05-11 Thread Chris Hostetter
: We do not open any IndexReader explicitly. We keep one instance on : IndexWriter open (and never close) and for searching we use : SearcherManager. I checked the lsof and did not find any files with : delete status. what exactly does your SearchManager usage look like? is every searcher =

Re: Un-used index files are not getting released

2017-05-09 Thread Ian Lea
index folder using java > (File.listFiles()) it lists 1761 files in that folder. This count goes down > to a double digit number when I restart the tomcat. > > Thanks for looking into it. > > -- > Regards > -Siraj Haider > (212) 306-0154 > > -Original Mess

RE: Un-used index files are not getting released

2017-05-08 Thread Siraj Haider
double digit number when I restart the tomcat. Thanks for looking into it. -- Regards -Siraj Haider (212) 306-0154 -Original Message- From: Ian Lea [mailto:ian@gmail.com] Sent: Friday, May 05, 2017 9:33 AM To: java-user@lucene.apache.org Subject: Re: Un-used index files are not

Re: Un-used index files are not getting released

2017-05-05 Thread Ian Lea
The most common cause is unclosed index readers. If you run lsof against the tomcat process id and see that some deleted files are still open, that's almost certainly the problem. Then all you have to do is track it down in your code. -- Ian. On Thu, May 4, 2017 at 10:09 PM, Siraj Haider wro

Un-used index files are not getting released

2017-05-04 Thread Siraj Haider
Hi all, We recently switched to Lucene 6.5 from 2.9 and we have an issue that the files in index directory are not getting released after the IndexWriter finishes up writing a batch of documents. We are using IndexFolder.listFiles().length to check the number of files in index folder. We have ev

Un-used index files are not getting released

2017-05-03 Thread Siraj Haider
Hi all, We recently switched to Lucene 6.5 from 2.9 and we have an issue that the files in index directory are not getting released after the IndexWriter finishes up writing a batch of documents. We are using IndexFolder.listFiles().length to check the number of files in index folder. We have ev

Re: how do lucene read large index files?

2016-11-29 Thread Kumaran Ramasubramanian
s? or any further pointers? > >> >> > NIOFSDirectory or SimpleFSDirectory, we have to pay another price: > > Our > >> >> code > >> >> >> has to do a lot of syscalls to the O/S kernel to copy blocks of data > >> >> >> be

Re: how do lucene read large index files?

2016-11-29 Thread Michael McCandless
to the O/S kernel to copy blocks of data >> >> >> between the disk or filesystem cache and our buffers residing in > Java >> >> heap. >> >> >> This needs to be done on every search request, over and over again. >> > >> > the blog post

Re: how do lucene read large index files?

2016-11-28 Thread Kumaran Ramasubramanian
> between the disk or filesystem cache and our buffers residing in Java > >> heap. > >> >> This needs to be done on every search request, over and over again. > > > > the blog post just says it simple: You should use MMapDirectory and avoid SimpleFSDir or MMapD

Re: how do lucene read large index files?

2016-11-24 Thread Erick Erickson
Java >> heap. >> >> This needs to be done on every search request, over and over again. > > the blog post just says it simple: You should use MMapDirectory and avoid > SimpleFSDir or MMapDirectory! The blog post explains why: SimpleFSDir and > NIOFSDir extend BufferedInde

RE: how do lucene read large index files?

2016-11-24 Thread Uwe Schindler
Directory and avoid SimpleFSDir or MMapDirectory! The blog post explains why: SimpleFSDir and NIOFSDir extend BufferedIndexInput. This class uses an on-heap buffer for reading index files (which is 16 KB). For some parts of the index (like doc values), this is not ideal. E.g. if you sort against a doc v

Re: how do lucene read large index files?

2016-11-24 Thread Erick Erickson
//blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html >> >> Short form: files are read into the OS's memory as needed. the whole >> file isn't read at once. >> >> Best, >> Erick >> >> On Wed, Nov 23, 2016 at 12:04 AM, Kumaran

Re: how do lucene read large index files?

2016-11-24 Thread Kumaran Ramasubramanian
/2012/07/use-lucenes-mmapdirectory-on-64bit.html > > Short form: files are read into the OS's memory as needed. the whole > file isn't read at once. > > Best, > Erick > > On Wed, Nov 23, 2016 at 12:04 AM, Kumaran Ramasubramanian > wrote: > > Hi All, > &

Re: Non-index files under the search directory

2016-11-24 Thread András Péteri
Correct, this data is associated with individual IndexCommits (you should be able to see the key-value pairs in the segment_xy files' raw contents in an index directory). To consolidate the entries, you'll have to retrieve user data from each sub-index, put all of them into a new map, then set this

Re: how do lucene read large index files?

2016-11-23 Thread Erick Erickson
> how do lucene read large index files? > for example, if one file (for eg: .dat file) is 4GB. > lucene read only part of file to RAM? or > is it different approach for different lucene file formats? > > > Related Link: > How do applications (and OS) handle ve

how do lucene read large index files?

2016-11-23 Thread Kumaran Ramasubramanian
Hi All, how do lucene read large index files? for example, if one file (for eg: .dat file) is 4GB. lucene read only part of file to RAM? or is it different approach for different lucene file formats? Related Link: How do applications (and OS) handle very big files? http://superuser.com/a/361201

Re: Non-index files under the search directory

2016-11-22 Thread Xiaolong Zheng
Hi András, Thanks, this is what I need! I also notice this user commit data does not carry over if I am consolidating several search database into a new one, I guess the solution should be explicitly use getCommitData for each sub-index, then set it into new consolidated search database, right?

Re: Non-index files under the search directory

2016-11-22 Thread amarnath cse
Can anyone tell me the procedure of text document indexing using Lucene. Thanks.. On Nov 22, 2016 10:40 PM, "András Péteri" wrote: > Hi Xiaolong, > > A Map of key-value pairs can be supplied to > IndexWriter#setCommitData(Map) and will be persisted > when committing changes (setting the commit d

Re: Non-index files under the search directory

2016-11-22 Thread András Péteri
Hi Xiaolong, A Map of key-value pairs can be supplied to IndexWriter#setCommitData(Map) and will be persisted when committing changes (setting the commit data counts as a change). It can be retrieved with IndexWriter#getCommitData() later. This may serve as good storage for metadata; as an exampl

Re: Non-index files under the search directory

2016-11-22 Thread Xiaolong Zheng
Thanks, StoredField seems still down to the per-document level, which means for every document they will contains this search field. What I really would like is a global level storage to hold this single value. Maybe this is impossible. Sincerely, --Xiaolong On Tue, Nov 22, 2016 at 5:13 AM, Mi

Re: Non-index files under the search directory

2016-11-22 Thread Michael McCandless
Lucene won't merge foreign files for you, and in general it's dangerous to put such files into Lucene's index directory because if they look like codec files Lucene may delete them. Can you just add a StoredField to each document to hold your information? Mike McCandless http://blog.mikemccandle

Non-index files under the search directory

2016-11-21 Thread Xiaolong Zheng
Hello, I am trying to adding some meta data into the search data base. Instead of adding a new search filed or adding a phony document, I am looking at the method org.apache.lucene.store.Directory#createOutpu, which is create new file in the search directory. I am wondering does indexwriter can

Re: how to backup index files with Replicator

2016-01-23 Thread will martin
ere is my code to backup index files with Lucene Replicator,but It doesn't > work well, No files were backuped. > Could you check my code and give me your advice? > > > public class IndexFiles { > > > private static Directory dir; > private sta

how to backup index files with Replicator

2016-01-23 Thread Dancer
Hi, here is my code to backup index files with Lucene Replicator,but It doesn't work well, No files were backuped. Could you check my code and give me your advice? public class IndexFiles { private static Directory dir; private static Path bakPath; private s

RE: mutability of lucene index files

2015-09-12 Thread Uwe Schindler
ache.org > Subject: Re: mutability of lucene index files > > Hi Erick, > > Thank you. > > Deleting old files is fine (and expected), so it sounds like the segment files > are immutable (prior to deletion) and the file that handles deletion is > renamed with every change, so

Re: mutability of lucene index files

2015-09-12 Thread Larry White
Hi Erick, Thank you. Deleting old files is fine (and expected), so it sounds like the segment files are immutable (prior to deletion) and the file that handles deletion is renamed with every change, so it's effectively immutable, too. That leaves the segments_* files and segments.gen, if I under

Re: mutability of lucene index files

2015-09-12 Thread Erick Erickson
The Lucene index segment files are immutable, once they're closed, they are never changed. These are things like _1.fdt, _1.tim, etc. All of the files with the same prefix (_1 in my example) comprise a single "segment". Segments _will_, however, disappear. During indexing, two or more segment are c

mutability of lucene index files

2015-09-12 Thread Larry White
Hi, I'm writing a backup routine for a system that includes Lucene for full-text search. The primary data store is based on immutable files, so it can be backed-up incrementally by copying any new files (and removing any files that have been deleted from earlier backups). It's my understanding fro

Re: How to configure lucene 4.x to read 3.x index files

2014-09-24 Thread Michael McCandless
chpoint.co.nz] >> Sent: Wednesday, 24 September 2014 12:24 p.m. >> To: 'java-user@lucene.apache.org' >> Subject: RE: How to configure lucene 4.x to read 3.x index files >> >> Hi Robert/Uwe, >> >> Thanks very much for the quick response. >> >&g

Re: How to configure lucene 4.x to read 3.x index files

2014-09-23 Thread Robert Muir
ber 2014 12:24 p.m. > To: 'java-user@lucene.apache.org' > Subject: RE: How to configure lucene 4.x to read 3.x index files > > Hi Robert/Uwe, > > Thanks very much for the quick response. > > I have tried again with a different set of index(28k documents) generated > f

RE: How to configure lucene 4.x to read 3.x index files

2014-09-23 Thread Patrick Mi
RE: How to configure lucene 4.x to read 3.x index files Hi Robert/Uwe, Thanks very much for the quick response. I have tried again with a different set of index(28k documents) generated from V3 too and that worked. But the one(30k documents) I tried indeed worked for the V3 but not V4.10. Maybe

RE: How to configure lucene 4.x to read 3.x index files

2014-09-23 Thread Patrick Mi
to configure lucene 4.x to read 3.x index files You should not have to configure anything. The exception should not happen: can I have this index to debug the issue? On Mon, Sep 22, 2014 at 11:07 PM, Patrick Mi wrote: > Hi there, > > I understood that Lucene V4 could read 3.x index

Re: How to configure lucene 4.x to read 3.x index files

2014-09-23 Thread Robert Muir
I opened an issue with a patch for this: https://issues.apache.org/jira/browse/LUCENE-5975 Thanks for reporting it! On Mon, Sep 22, 2014 at 11:07 PM, Patrick Mi wrote: > Hi there, > > I understood that Lucene V4 could read 3.x index files by configuring > Lucene3xCodec but what e

Re: How to configure lucene 4.x to read 3.x index files

2014-09-23 Thread Robert Muir
You should not have to configure anything. The exception should not happen: can I have this index to debug the issue? On Mon, Sep 22, 2014 at 11:07 PM, Patrick Mi wrote: > Hi there, > > I understood that Lucene V4 could read 3.x index files by configuring > Lucene3xCodec but what e

Re: How to configure lucene 4.x to read 3.x index files

2014-09-23 Thread Uwe Schindler
Yes it can read 3.x index files without extra configuaration. You cannot enforce that, it is automatically. Unfortunately, Lucene 4.10 has some problems, which will be fixed with a bugfix release soon. Those bugs can lead to index corruption. Maybe try 4.9.1 first. Are you sure the 3.x index

How to configure lucene 4.x to read 3.x index files

2014-09-22 Thread Patrick Mi
Hi there, I understood that Lucene V4 could read 3.x index files by configuring Lucene3xCodec but what exactly needs to be done here? I used DEMO code from V4.10.0 to generate v4 index files and could read them without problem. When I tried to read index files generated from V3 I got the

What happens when making many index files?

2014-06-25 Thread benglish
Hi pals, I have created an index out of about 1000 documents. When I searched through the index, I got an accuracy of around 37.7%, then I made another index file with same documents. So, the number of index files doubled and surprisingly, the accuracy enhanced to 55%!! Making the index for

Re: Bulk indexing and delete old index files

2013-12-03 Thread 장용석
ionizeit > Did you get this working? Are you able to share the solution? > > I am looking for a solution to similar problem. > > Thanks in advance. > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Bulk-indexing-and-delete-old-inde

Re: Bulk indexing and delete old index files

2013-12-01 Thread revolutionizeit
Did you get this working? Are you able to share the solution? I am looking for a solution to similar problem. Thanks in advance. -- View this message in context: http://lucene.472066.n3.nabble.com/Bulk-indexing-and-delete-old-index-files-tp4044746p4104297.html Sent from the Lucene - Java

Lucene index files are empty

2013-08-26 Thread Babak Bastan
I want to umplement full text search using Lucene and Hibernate search. As I understand, Lucene creats two files. These two files should be contain some text data, but my index files are empty ( I open these two files with Luke) [image: Inline image 1] here is my code: @Entity@Indexed@Table(name

Re: Lucene version naming of index files

2013-03-14 Thread Robert Muir
On Thu, Mar 14, 2013 at 7:22 AM, Bernd Fehling wrote: > Hi list, > > a stupid question about the naming of the index files. > While using lucene (and solr) 4.2 I still see files with "Lucene41" in the > name. > This is somewhat confusing if lucene 4.x produces files

Lucene version naming of index files

2013-03-14 Thread Bernd Fehling
Hi list, a stupid question about the naming of the index files. While using lucene (and solr) 4.2 I still see files with "Lucene41" in the name. This is somewhat confusing if lucene 4.x produces files with "Lucene4y". This also means indexes built with 4.2 or 4.3 are fully

Re: Bulk indexing and delete old index files

2013-03-05 Thread 장용석
Erick > > > On Tue, Mar 5, 2013 at 12:10 AM, 장용석 wrote: > >> Hi. >> >> I want indexing all documents once a day and after indexing delete old >> index files that indexed before a day. >> >> I think to do this, indexing all documents in n

Re: Bulk indexing and delete old index files

2013-03-05 Thread Erick Erickson
be already replaced if your new documents had the same value as the old ones, then the old ones would be automatically replaced. Best Erick On Tue, Mar 5, 2013 at 12:10 AM, 장용석 wrote: > Hi. > > I want indexing all documents once a day and after indexing delete old > index files tha

Re: Bulk indexing and delete old index files

2013-03-05 Thread Ian Lea
> Hi. > > I want indexing all documents once a day and after indexing delete old > index files that indexed before a day. > > I think to do this, indexing all documents in new directory and replace > IndexSearcher and IndexWriter with olds, and delete old index directory. >

Bulk indexing and delete old index files

2013-03-04 Thread 장용석
Hi. I want indexing all documents once a day and after indexing delete old index files that indexed before a day. I think to do this, indexing all documents in new directory and replace IndexSearcher and IndexWriter with olds, and delete old index directory. Is there more good indexing strategy

Re: IndexWriterConfig.OpenMode.CREATE vs OpenMode.APPEND (index files)

2013-02-01 Thread saisantoshi
cument separately, then it would produce more files in the index (with the current version of the Lucene, 4.0). My question here is , why it is not overwriting the existing index files by just updating it. I have reverted back my code (2.4 base) and tried to update a document couple of items, i

Re: IndexWriterConfig.OpenMode.CREATE vs OpenMode.APPEND (index files)

2013-02-01 Thread Michael McCandless
I use any other option to just overwrite the existing files in the > index directory such that there are only fewer files in the index directory. > > Thanks, > Sai. > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/IndexWriterConfig-O

Re: IndexWriterConfig.OpenMode.CREATE vs OpenMode.APPEND (index files)

2013-01-31 Thread saisantoshi
. Thanks, Sai. -- View this message in context: http://lucene.472066.n3.nabble.com/IndexWriterConfig-OpenMode-CREATE-vs-OpenMode-APPEND-index-files-tp4037766p4037796.html Sent from the Lucene - Java Users mailing list archive at Nabble.com

Re: IndexWriterConfig.OpenMode.CREATE vs OpenMode.APPEND (index files)

2013-01-31 Thread Michael McCandless
://blog.mikemccandless.com On Thu, Jan 31, 2013 at 2:54 PM, saisantoshi wrote: > It's _0.si ( typo) > > For second update, create = "false". > > Thanks, > Sai. > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/IndexWriterCo

Re: IndexWriterConfig.OpenMode.CREATE vs OpenMode.APPEND (index files)

2013-01-31 Thread saisantoshi
It's _0.si ( typo) For second update, create = "false". Thanks, Sai. -- View this message in context: http://lucene.472066.n3.nabble.com/IndexWriterConfig-OpenMode-CREATE-vs-OpenMode-APPEND-index-files-tp4037766p4037785.html Sent from the Lucene - Java Users mailing

Re: IndexWriterConfig.OpenMode.CREATE vs OpenMode.APPEND (index files)

2013-01-31 Thread Michael McCandless
WriterConfig. > > Appreciate if you could shed some light on this as the docs are not very > clear. > > Thanks, > Sai. > > > > > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/IndexWriterConfig-OpenMode-CREATE-vs-Ope

IndexWriterConfig.OpenMode.CREATE vs OpenMode.APPEND (index files)

2013-01-31 Thread saisantoshi
/IndexWriterConfig-OpenMode-CREATE-vs-OpenMode-APPEND-index-files-tp4037766.html Sent from the Lucene - Java Users mailing list archive at Nabble.com. - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional

Re: How to load only part of index files on hard drive to ram

2012-01-09 Thread Ian Lea
You could certainly run some warm up queries, but you can't tell lucene which bits of an index to load into RAM. There are a couple of classes in the org.apache.lucene.index contrib section that can split indexes but I doubt they are relevant here. Why the obsession with RAMDirectory? In my exp

Re: Strategy for large index files

2012-01-08 Thread Petite Abeille
On Jan 8, 2012, at 6:32 AM, Cheng wrote: > Hi, my servlet application is running a large index of 20G. I don't think > it can be loaded to RAM at one time. > > What are the general strategies to improve the search and write performance? Got money? http://www.ramsan.com/ http://www.fusionio.co

Re: Strategy for large index files

2012-01-08 Thread Ian Lea
Read the FAQ. http://wiki.apache.org/lucene-java/LuceneFAQ -- Ian. On Sun, Jan 8, 2012 at 5:32 AM, Cheng wrote: > Hi, my servlet application is running a large index of 20G. I don't think > it can be loaded to RAM at one time. > > What are the general strategies to improve the search and write

Strategy for large index files

2012-01-07 Thread Cheng
Hi, my servlet application is running a large index of 20G. I don't think it can be loaded to RAM at one time. What are the general strategies to improve the search and write performance? Thanks

Re: index files naming

2011-01-03 Thread Bernd Fehling
Hi Simon, thanks a lot for your good explanation. Best wishes, Bernd Am 03.01.2011 13:51, schrieb Simon Willnauer: > Hey Bernd, > > On Mon, Jan 3, 2011 at 1:35 PM, Bernd Fehling > wrote: >> Dear list, >> >> some questions about the names of the index files. &g

Re: index files naming

2011-01-03 Thread Simon Willnauer
Hey Bernd, On Mon, Jan 3, 2011 at 1:35 PM, Bernd Fehling wrote: > Dear list, > > some questions about the names of the index files. > With an older Lucene/Solr 4.x version from trunk my index looks like: > _2t1.fdt > _2t1.fdx > _2t1.fnm > _2t1.frq > _2t1.nrm >

index files naming

2011-01-03 Thread Bernd Fehling
Dear list, some questions about the names of the index files. With an older Lucene/Solr 4.x version from trunk my index looks like: _2t1.fdt _2t1.fdx _2t1.fnm _2t1.frq _2t1.nrm _2t1.prx _2t1.tii _2t1.tis segments_2 segments.gen With a most recent version from trunk it looks like: _3a9.fdt _3a9

Re: Index files not deleted after optimization

2009-11-05 Thread Ganesh
ubject: Re: Index files not deleted after optimization > On Tue, Nov 3, 2009 at 9:45 AM, Ganesh wrote: >> My IndexReader and Searcher is open all the time. I am reopening it at >> constant interval. >> >> Below are the code sequence. >> >> 1. DB optimize >

Re: Index files not deleted after optimization

2009-11-03 Thread Michael McCandless
On Tue, Nov 3, 2009 at 9:45 AM, Ganesh wrote: > My IndexReader and Searcher is open all the time. I am reopening it at > constant interval. > > Below are the code sequence. > > 1. DB optimize > 2. Close writer > 3. Open writer > 4. Reopen new reader > 5. Close old reader > 6. Close old searcher.

Re: Index files not deleted after optimization

2009-11-03 Thread Ganesh
09 3:22 PM Subject: Re: Index files not deleted after optimization It depends on the relative timing. If the old IndexReader is still open when the optimize completes then the files it has open cannot be deleted. But, if that IndexReader hadn't been reopened in a while, it's possible it d

Re: Index files not deleted after optimization

2009-11-03 Thread Michael McCandless
se(); > searcher.close(); > > Regards > Ganesh > > - Original Message - > From: "Michael McCandless" > To: > Sent: Monday, November 02, 2009 6:03 PM > Subject: Re: Index files not deleted after optimization > > > Something must still have these file ha

Re: Index files not deleted after optimization

2009-11-03 Thread Ganesh
nt: Monday, November 02, 2009 6:03 PM Subject: Re: Index files not deleted after optimization Something must still have these file handles open at the time the optimization completed. EG do you have a reader open on this index? Mike On Mon, Nov 2, 2009 at 6:54 AM, Ganesh wrote: > Hello all, &

Re: Index files not deleted after optimization

2009-11-02 Thread Michael McCandless
Something must still have these file handles open at the time the optimization completed. EG do you have a reader open on this index? Mike On Mon, Nov 2, 2009 at 6:54 AM, Ganesh wrote: > Hello all, > > I am using Lucene 2.4.1 and My app is running inside Tomcat. > > In Windows, after database o

Index files not deleted after optimization

2009-11-02 Thread Ganesh
Hello all, I am using Lucene 2.4.1 and My app is running inside Tomcat. In Windows, after database optimization, the old db files are not getting deleted. I enabled the info stream and found the below entries. I used ProcessExplorer from SysInternals to view the lock file, but old db files are

Re: How to avoid huge index files

2009-09-10 Thread Ted Stockwell
009 2:18:35 PM > Subject: Re: How to avoid huge index files > > > Is it possible to upload to GAE an already exist index? My index is data I'm > collecting for long time, and I prefer not to give it up. > >

Re: How to avoid huge index files

2009-09-10 Thread Dvora
in Google App Engine >> > (http://code.google.com/appengine/), which limits files length to be >> > smaller than 10MB. > > > > > > - > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org &

Re: How to avoid huge index files

2009-09-10 Thread Ted Stockwell
Another alternative is storing the indexes in the Google Datastore, I think Compass already supports that (though I have not used it). Also, I have successfully run Lucene on GAE using GaeVFS (http://code.google.com/p/gaevfs/) to store the index in the Datastore. (I developed a Lucene Directory

RE: How to avoid huge index files

2009-09-10 Thread Dvora
28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > >> From: Dvora [mailto:barak.ya...@gmail.com] >> Sent: Thursday, September 10, 2009 1:23 PM >> To: java-user@lucene.apache.org >> Subject: Re: How to avoid huge index files >> >> >> Hi aga

RE: How to avoid huge index files

2009-09-10 Thread Uwe Schindler
Re: How to avoid huge index files > > > Hi again, > > Can you add some details and guidelines how to implement that? Different > files types have different structure, is such spliting doable without > knowing Lucene internals? > > > Michael McCandless-2 wrote:

Re: How to avoid huge index files

2009-09-10 Thread Dvora
t;>>> Hello again, >>>> >>>> Can someone please comment on that, whether what I'm looking is >>>> possible >>>> or >>>> not? >>>> >>>> >>>> Dvora wrote: >>>>> >>>>&g

Re: How to avoid huge index files

2009-09-10 Thread Michael McCandless
ble >>> or >>> not? >>> >>> >>> Dvora wrote: >>>> >>>> Hello, >>>> >>>> I'm using Lucene2.4. I'm developing a web application that using Lucene >>>> (via compass) to do the searches. >

Re: How to avoid huge index files

2009-09-10 Thread Dvora
> >>> I'm using Lucene2.4. I'm developing a web application that using Lucene >>> (via compass) to do the searches. >>> I'm intending to deploy the application in Google App Engine >>> (http://code.google.com/appengine/), which limits files l

Re: How to avoid huge index files

2009-09-10 Thread Michael McCandless
the searches. >> I'm intending to deploy the application in Google App Engine >> (http://code.google.com/appengine/), which limits files length to be >> smaller than 10MB. I've read about the various policies supported by >> Lucene to limit the file sizes, but on matte

Re: How to avoid huge index files

2009-09-09 Thread Dvora
y the application in Google App Engine > (http://code.google.com/appengine/), which limits files length to be > smaller than 10MB. I've read about the various policies supported by > Lucene to limit the file sizes, but on matter which policy I used and > which parameters, the index

How to avoid huge index files

2009-09-08 Thread Dvora
ad about the various policies supported by Lucene to limit the file sizes, but on matter which policy I used and which parameters, the index files still grew to be lot more the 10MB. Looking at the code, I've managed to limit the cfs files (predicting the file size in CompoundFileWriter before closing

Re: deletion of index-files fails

2009-03-07 Thread Michael McCandless
which is how I found the problem). So, getRefCount() would be good! thanks, Paul -Original Message- From: Erick Erickson Sent: Mar 6, 2009 9:01 PM To: java-user@lucene.apache.org Subject: Re: deletion of index-files fails OK, I understand now. Like I said, anything you deem approp

Re: deletion of index-files fails

2009-03-06 Thread rolarenfan
- >From: Erick Erickson >Sent: Mar 6, 2009 9:01 PM >To: java-user@lucene.apache.org >Subject: Re: deletion of index-files fails > >OK, I understand now. Like I said, anything you deem appropriate. > >Best >Erick > >On Fri, Mar 6, 2009 at 5:45 PM, Michael McCand

Re: deletion of index-files fails

2009-03-06 Thread Erick Erickson
re and/or raise a JIRA... >>>> >>>> Best >>>> Erick >>>> >>>> On Fri, Mar 6, 2009 at 11:40 AM, Michael McCandless < >>>> luc...@mikemccandless.com> wrote: >>>> >>>> >>>> OK, phew! Tha

Re: deletion of index-files fails

2009-03-06 Thread Michael McCandless
Sent: Mar 6, 2009 4:23 AM To: java-user@lucene.apache.org Cc: rolaren...@earthlink.net Subject: Re: deletion of index-files fails If truly the IndexWriter & all IndexReaders are closed, then they should no longer be holding open files. Maybe triple check that you've indeed closed everything.

Re: deletion of index-files fails

2009-03-06 Thread Erick Erickson
>>> once >>>> I removed that, the call to IndexReader.close() actually worked and then >>>> the >>>> deletion did so too. Thanks; sorry to trouble you. >>>> >>>> -Paul >>>> >>>> -Original Message--

Re: deletion of index-files fails

2009-03-06 Thread Michael McCandless
so too. Thanks; sorry to trouble you. -Paul -Original Message- From: Michael McCandless Sent: Mar 6, 2009 4:23 AM To: java-user@lucene.apache.org Cc: rolaren...@earthlink.net Subject: Re: deletion of index-files fails If truly the IndexWriter & all IndexReaders are closed, then they shou

Re: deletion of index-files fails

2009-03-06 Thread Erick Erickson
ef() ... once >> I removed that, the call to IndexReader.close() actually worked and then the >> deletion did so too. Thanks; sorry to trouble you. >> >> -Paul >> >> -Original Message- >> >>> From: Michael McCandless >>&g

Re: deletion of index-files fails

2009-03-06 Thread Michael McCandless
worked and then the deletion did so too. Thanks; sorry to trouble you. -Paul -Original Message- From: Michael McCandless Sent: Mar 6, 2009 4:23 AM To: java-user@lucene.apache.org Cc: rolaren...@earthlink.net Subject: Re: deletion of index-files fails If truly the IndexWriter &

Re: deletion of index-files fails

2009-03-06 Thread rolarenfan
-Original Message- >From: Michael McCandless >Sent: Mar 6, 2009 4:23 AM >To: java-user@lucene.apache.org >Cc: rolaren...@earthlink.net >Subject: Re: deletion of index-files fails > > >If truly the IndexWriter & all IndexReaders are closed, then they >should no longe

Re: deletion of index-files fails

2009-03-06 Thread rolarenfan
rom: Ian Lea >Sent: Mar 6, 2009 4:07 AM >To: java-user@lucene.apache.org, rolaren...@earthlink.net >Subject: Re: deletion of index-files fails > >What OS are you running? What version of lucene? Are you sure that >you have privilege to delete the files that it is failing on? That

Re: deletion of index-files fails

2009-03-06 Thread Michael McCandless
are part of the index you are trying to remove? That something else doesn't have the files open? It seems likely that you are on Windows and that something is holding on to the files. I believe that Windows won't let you delete open files. You could try calling File.deleteOnExit() for the

Re: deletion of index-files fails

2009-03-06 Thread Ian Lea
thing is holding on to the files. I believe that Windows won't let you delete open files. You could try calling File.deleteOnExit() for the index files. -- Ian. On Fri, Mar 6, 2009 at 2:19 AM, wrote: > So, I have a (small) Lucene index, all fine; I use it a bit, and then (on app >

deletion of index-files fails

2009-03-05 Thread rolarenfan
So, I have a (small) Lucene index, all fine; I use it a bit, and then (on app shutdown) want to delete its files and the containing directory (the index is intended as a temp object). At some earlier time this was working just fine, using java.io.File.delete(). Now however, some of the files get

RE: How to make a query that associates 2 index files

2008-05-06 Thread Michael Siu
fly and be unique per different content. Thanks. -m -Original Message- From: Erick Erickson [mailto:[EMAIL PROTECTED] Sent: Tuesday, May 06, 2008 12:46 PM To: java-user@lucene.apache.org Subject: Re: How to make a query that associates 2 index files Sure, just include different fields in

Re: How to make a query that associates 2 index files

2008-05-06 Thread Erick Erickson
-Original Message- > From: Erick Erickson [mailto:[EMAIL PROTECTED] > Sent: Tuesday, May 06, 2008 9:36 AM > To: java-user@lucene.apache.org > Subject: Re: How to make a query that associates 2 index files > > You don't. You really have to roll your own solution here, the

RE: How to make a query that associates 2 index files

2008-05-06 Thread Michael Siu
Message- From: Erick Erickson [mailto:[EMAIL PROTECTED] Sent: Tuesday, May 06, 2008 9:36 AM To: java-user@lucene.apache.org Subject: Re: How to make a query that associates 2 index files You don't. You really have to roll your own solution here, there's no "inter-index" a

Re: How to make a query that associates 2 index files

2008-05-06 Thread Chris Lu
> > > > I am a newbie to Lucene. I have a question for making a query that > associate > 2 index files: > > > > - One index has the content index for a list of documents and a key to the > document. That means the Lucene document of this index contains 2 fields: &

Re: How to make a query that associates 2 index files

2008-05-06 Thread Erick Erickson
t place? Perhaps there's a more specific answer waiting out there... Best Erick On Tue, May 6, 2008 at 12:14 PM, Michael Siu <[EMAIL PROTECTED]> wrote: > Hi, > > > > I am a newbie to Lucene. I have a question for making a query that > associate > 2 index files: >

How to make a query that associates 2 index files

2008-05-06 Thread Michael Siu
Hi, I am a newbie to Lucene. I have a question for making a query that associate 2 index files: - One index has the content index for a list of documents and a key to the document. That means the Lucene document of this index contains 2 fields: the 'content' and the 'k

  1   2   >