und)... since you're using these classes for
>> writing, it'll mean you can freely swap in different codecs.
>>
>> The only thing you can do further is to conflate your custom code with
>> the codec, ie, so that you make a single chain that directly writes
>> i
dex files. But I'm not sure you'll gain much performance by doing
> so... (and then you can't [as easily] swap codecs).
>
> Have you profiled to see where the time is being spent?
>
> Mike
>
> On Thu, Mar 25, 2010 at 7:40 PM, britske <[hidden
> email]<htt
Hi,
perhaps first some background:
I need to speed-up indexing for an particular application which has a pretty
unsual schema: besides the normal stored and indexed fields we have about
20.000 fields per document which are all indexed/ non-stored sInts.
Obviously indexing was really slow wit
eally want is Y. And most of the posts
> I've seen wondering about doc IDs were exactly that, but yours
> clearly isn't.
>
> And I'm going to have to defer any other comments to people who
> know more about it than I do
>
> Erick
>
> On Wed, Nov 4, 2
please ignore the garbage at the end ;-)
Britske wrote:
>
> This issue is related to post: "merging Parallel indexes (can
> indexWriter.addIndexesNoOptimize be used?)"
>
> Among another thing described in the post above, I'm experimenting with a
> com
;
> All the advice I've ever seen says that if you want to keep track of
> documents, you assign and index your own ID. You can get the
> doc ID from your unique term quite efficiently if you need to.
>
> HTH
> Erick
>
> On Wed, Nov 4, 2009 at 9:23 AM, Britske wrote
Just to clarify question changed the subject:
addIndexesNoOptimize on shards --> is docid deterministic and calculable?
(IF docids of shards seperately are known)
Britske wrote:
>
> Hi,
>
> say I have:
> - Indexreader[] readers = {reader1, reader2, reader3} //containi
Hi,
say I have:
- Indexreader[] readers = {reader1, reader2, reader3} //containing all
different docs
- I know the internal docids of documents in reader1, reader2, reader3
seperately
Does doing IndexWriter.addIndexesNoOptimize(Indexreader[] readers) on these
readers give me a determinstic and
n large indexes is not very efficient for searching ,
> but I need to use parallel indexes because one of them is huge but
> evolve very slowly compared to the others.
>
> Jérôme
>
> Britske a écrit :
>> Given two parallel indexes which contain the same products but differe
se addIndexes(IndexReader[]),
> passing in your ParallelReader?
>
> Mike
>
> On Wed, Nov 4, 2009 at 6:46 AM, Britske wrote:
>>
>> Thanks, but it's already guaranteed that the indexes are in sync. So I
>> could
>> (and do) use parallelReader to search them
ache.org/jira/browse/LUCENE-1879
>
> It does not actually merge the indexes, but rather keeps 2 parallel
> indexes in sync so you can use ParallelReader to search them
> coherently.
>
> Mike
>
> On Tue, Nov 3, 2009 at 1:46 PM, Britske wrote:
>>
>> Given two para
Given two parallel indexes which contain the same products but different
fields, one with slowly changing fields and one with fields which are
updated regularly:
Is it possible to periodically merge these to form a single index? (thereby
representing a frozen snapshot in time)
For example: Ca
Given two parallel indexes which contain the same products but different
fields, one with slowly changing fields and one with fields which are
updated regularly:
Is it possible to periodically merge these to form a single index? (thereby
representing a frozen snapshot in time)
For example: Can
Given two parallel indexes one with slowly changing fields and one with
fields which are updated regularly.
Is it possible to periodically merge these to form a single index? (thereby
representing a frozen snapshot in time)
For example: Can indexWriter.addIndexesNoOptimize handle this, or was
ad();
>
>
> Or you have to add more options in
> FieldSortedHitQueue.CacheEntry.buildData(IndexReader reader) and more
> static comparator builders in FieldSortedHitQueue that use the right
> CacheKeys. Obviously not very extensibility friendly at the moment. I'm
> sur
thought
> about it, but I'm hoping an API tweak can alleviate some of this.
>
> - Mark
>
> Britske wrote:
>> Hi,
>>
>> I recently saw activity on LUCENE-831 (Complete overhaul of FieldCache
>> API/Implementation) which I have interest in.
>> I posted
g my own cache implementation
> in
> lucene 2.3.2. What I'd really like to see in the new cache implementation
> is
> easier pluggability and extension of the lucene classes, which is
> currently
> not possible due to visibility issues.
>
> My 2 cents.
>
> 2008/11/14
Hi,
I recently saw activity on LUCENE-831 (Complete overhaul of FieldCache
API/Implementation) which I have interest in.
I posted previously on this with my concern that given the current default
cache I sometimes get OOM-errors because I have a lot of fields which are
sorted on, which ultimate
eopening Readers with field caches, it can be waaay faster.
>
>
> - Mark
>
> On Thu, 2008-04-17 at 05:14 -0700, Britske wrote:
>> I've seen some recent activity on LUCENE-831 "Complete overhaul of
>> FieldCache
>> API" and read that it must be
r if this hasn't yet been taken into
account. As far as I can follow it, this was one of the initial intentions.
Thanks,
Britske
--
View this message in context:
http://www.nabble.com/Does-LUCENE-831%29-%22Complete-overhaul-of-FieldCache-API%22-provide-fieldcache-offloading-to-disk--tp167435
First my question:
Is there an (experimental / patch-version) lucene-fieldcache available which
uses some kind of eviction-strategy (LRU or whatever) so that OOM's would
never happen in my case, but instead some sorts would simply get evicted?
Now the background for those who are interested:
21 matches
Mail list logo
