RE: Index merging and optimizing

2008-01-15 Thread spring
> But it also seems that the parallel/not parallel decision is > something you control on the back end, so I'm not sure the user > is involved in the merge question at all. In other words, you could > easily split the indexing task up amongst several machines and/or > processes and combine all the

Re: Index merging and optimizing

2008-01-14 Thread Erick Erickson
OK, I think I'm getting a better handle here. I can't imagine how it would work to combine indexes that use *different* analyzers on the *same* field. Regardless of what Lucene did, you simply could NOT explain this to a user. To take a simple example, index part of your data for field1 with Keywor

RE: Index merging and optimizing

2008-01-14 Thread spring
> Then why would you want to combine them? > > I really think you need to explain what you're trying to accomplish > rather then obsess on the details. I have to create indexes in parallel because the amount of data is very high. Then I want to merge them into bigger indexes an move them to the s

Re: Index merging and optimizing

2008-01-14 Thread Erick Erickson
Then why would you want to combine them? I really think you need to explain what you're trying to accomplish rather then obsess on the details. Erick On Jan 14, 2008 10:17 AM, <[EMAIL PROTECTED]> wrote: > > I admit I've never used IndexMergeTool, I've always used > > IndexWriter.AddIndexex and

RE: Index merging and optimizing

2008-01-14 Thread spring
> I admit I've never used IndexMergeTool, I've always used > IndexWriter.AddIndexex and then execute > IndexWriter.optimize(). > > And I've seen no problems. That call takes no > analyzer. So you take the first index an add a remaining indexes via addIndexes? What happens if the indexes were crea

Re: Index merging and optimizing

2008-01-14 Thread Erick Erickson
I admit I've never used IndexMergeTool, I've always used IndexWriter.AddIndexex and then execute IndexWriter.optimize(). And I've seen no problems. That call takes no analyzer. Erick On Jan 14, 2008 6:12 AM, <[EMAIL PROTECTED]> wrote: > > See org.apache.lucene.misc.IndexMergeTool > > Thank you.

RE: Index merging and optimizing

2008-01-14 Thread spring
> See org.apache.lucene.misc.IndexMergeTool Thank you. But this uses a hardcoded analyzer and deprecated API-Calls. How does the used analyzer effect the merge process? Is everything reindexed with this new analyzer again? Does this make sense? What if the sources indexes had other analyzers us

Re: Index merging and optimizing

2008-01-13 Thread Erick Erickson
See IndexWriter.AddIndexes. See org.apache.lucene.misc.IndexMergeTool Erick On Jan 13, 2008 12:10 PM, <[EMAIL PROTECTED]> wrote: > Hi, > > are there any ready to use tools out there which I can use for merging and > optimzing? > > I have seen that Luke can optimize, but not merge? > > Or do I ha

Index merging and optimizing

2008-01-13 Thread spring
Hi, are there any ready to use tools out there which I can use for merging and optimzing? I have seen that Luke can optimize, but not merge? Or do I have to write my own utility? Thank you - To unsubscribe, e-mail: [EMAIL PRO