Re: FieldCache & DocValues Filter

2013-06-06 Thread Robert Muir
Yes: thats right. On Thu, Jun 6, 2013 at 3:23 AM, Arun Kumar K wrote: > Hi, > > Thanks Robert ! This info is exactly what i need. > > Just for getting myself clear. > If the field is a DocValue field the FieldCacheTermsFilter will use the > existing DocValues Field. For Normal Fields the filter

Re: FieldCache & DocValues Filter

2013-06-06 Thread Arun Kumar K
Hi, Thanks Robert ! This info is exactly what i need. Just for getting myself clear. If the field is a DocValue field the FieldCacheTermsFilter will use the existing DocValues Field. For Normal Fields the filter will create a DocValues for that field using FieldCache. Arun On Thu, Jun 6, 2013

Re: FieldCache & DocValues Filter

2013-06-06 Thread Michael McCandless
On Thu, Jun 6, 2013 at 5:35 AM, Robert Muir wrote: > Its confusing: I think we should rename FieldCacheXXX to DocValuesXXX. +1 Mike McCandless http://blog.mikemccandless.com - To unsubscribe, e-mail: java-user-unsubscr...@luc

Re: FieldCache & DocValues Filter

2013-06-06 Thread Robert Muir
FieldCacheTermsFilter will use your docvalues field. Its confusing: I think we should rename FieldCacheXXX to DocValuesXXX. On Thu, Jun 6, 2013 at 2:22 AM, Arun Kumar K wrote: > Hi Guys, > > I was trying to better the filtering mechanism for my use case. > When i use the existing filters like F

Re: FieldCache

2011-10-22 Thread Simon Willnauer
I think i'd try to use a bitset instead of a string for your categories, is that possible? how many categories do you have roughly? simon On Sat, Oct 22, 2011 at 6:01 AM, Peyman Faratin wrote: > Hi > > I have a field that is indexed as follows > > for(String c: article.getCategories()){ >      

RE: FieldCache and 2.9

2010-05-11 Thread Carl Austin
Carl -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: 11 May 2010 14:41 To: java-user@lucene.apache.org Subject: Re: FieldCache and 2.9 You are requesting the FieldCache entry from the top-level reader and hence a whole new FieldCache

Re: FieldCache and 2.9

2010-05-11 Thread Yonik Seeley
You are requesting the FieldCache entry from the top-level reader and hence a whole new FieldCache entry must be created. Lucene 2.9 sorting requests FieldCache entries at the segment level and hence reuses entries for those segments that haven't changed. -Yonik Apache Lucene Eurocon 2010 18-21 Ma

Re: FieldCache memory estimation - term values are interned?

2010-05-01 Thread Koji Sekiguchi
Yonik Seeley wrote: On Sat, May 1, 2010 at 8:23 PM, Koji Sekiguchi wrote: Yonik Seeley wrote: Values are not interned, but in a single field cache entry (String[]) the same String object is used for all docs with that same value. Yeah, you are right. Because I could see the arb

Re: FieldCache memory estimation - term values are interned?

2010-05-01 Thread Yonik Seeley
On Sat, May 1, 2010 at 8:23 PM, Koji Sekiguchi wrote: > Yonik Seeley wrote: >> Values are not interned, but in a single field cache entry (String[]) >> the same String object is used for all docs with that same value. > > Yeah, you are right. Because I could see the arbitrary two Strings that have

Re: FieldCache memory estimation - term values are interned?

2010-05-01 Thread Koji Sekiguchi
Yonik Seeley wrote: 2010/4/30 Koji Sekiguchi : Are Strings that are got via FieldCache.DEFAULT.getStrings( reader, field ) interned? Since I have a requirement for having FieldCaches of some fields in 250M docs index, I'd like to estimate memory consumed by FieldCache. By looking at FieldCa

Re: FieldCache memory estimation - term values are interned?

2010-05-01 Thread Yonik Seeley
2010/4/30 Koji Sekiguchi : > Are Strings that are got via FieldCache.DEFAULT.getStrings( reader, > field ) interned? > > Since I have a requirement for having FieldCaches of some > fields in 250M docs index, I'd like to estimate memory > consumed by FieldCache. > > By looking at FieldCacheImpl sour

Re: FieldCache cost too much time. how can I preload the the custom fieldCache when new segment exits!

2010-02-27 Thread Michael McCandless
Sounds like you should simply open & warm the reader in a background thread... You might want to use the SearcherManager class from upcoming Lucene in Action 2nd edition (NOTE: I'm a co-author). You can download the source code @ http://manning.com/hatcher3. Mike ---

Re: FieldCache Question

2009-02-04 Thread Todd Benge
On Wed, Feb 4, 2009 at 10:41 AM, Mark Miller wrote: > Todd Benge wrote: > >> >> The intent is to reduce the amount of memory that is held in cache. As it >> is now, it looks like there is an array of comparators for each index >> reader. Most of the data in the array appears to be the same for

Re: FieldCache Question

2009-02-04 Thread Mark Miller
Todd Benge wrote: The intent is to reduce the amount of memory that is held in cache. As it is now, it looks like there is an array of comparators for each index reader. Most of the data in the array appears to be the same for each cache so there is duplication for each type ( string, float).

Re: FieldCache Question

2009-02-04 Thread Todd Benge
On Wed, Feb 4, 2009 at 10:01 AM, Mark Miller wrote: > Todd Benge wrote: >> >> Hi, >> >> I've been looking into the FieldCache API because of memory problems >> we've been seeing in our production environment. >> >> We use various different sorts so over time the cache builds up and >> servers stop

Re: FieldCache Question

2009-02-04 Thread Mark Miller
Todd Benge wrote: Hi, I've been looking into the FieldCache API because of memory problems we've been seeing in our production environment. We use various different sorts so over time the cache builds up and servers stop responding. I decided to apply the patch for JIRA 831: https://issues.ap

Re: FieldCache performance

2008-04-13 Thread Otis Gospodnetic
Angel, Have you stepped through this with a debugger? That might reveal something. Have you tried doing kill -QUIT while waiting for those slow calls you mention to return? Perhaps this will show that the slow calls spend their time somewhere where the faster calls never go. Otis -- Sematext

Re: FieldCache Implementations

2007-12-03 Thread Thom Nelson
I have implemented a custom version of FieldCache to handle multi-valued fields, but this requires an interface change so it isn't applicable to what you're suggesting. However, it would be great to have a standard solution for handling multiple values. Grant Ingersoll wrote: Does any out the

Re: fieldcache gives OOM. Deos a LRU-style fieldcache exist?

2007-11-12 Thread Mark Miller
https://issues.apache.org/jira/browse/LUCENE-831 This patch should be a good start for you. Not sure how stale it is at this point though. Britske wrote: First my question: Is there an (experimental / patch-version) lucene-fieldcache available which uses some kind of eviction-strategy (LRU

Re: FieldCache for Search

2007-07-28 Thread Erick Erickson
You might try lazy loading. See IndexReader.*document *(int n, FieldSelector fieldSelector), particularly the FieldSelector. It allows you to selectively load only the fields you want. Otherwise, I'm sure if you looked in the unit tests you'd find examples of how to use FieldCache. If your field

Re: FieldCache: flush cache explicitly

2007-03-13 Thread Otis Gospodnetic
John - a bug with code is best. No gods here. Otis - Original Message From: John Wang <[EMAIL PROTECTED]> To: java-user@lucene.apache.org Sent: Friday, March 9, 2007 2:22:35 AM Subject: FieldCache: flush cache explicitly I think the api should allow for explicitly flush the fieldcache.

RE: FieldCache: flush cache explicitly

2007-03-09 Thread Ramana Jelda
Yeap. I strongly support John. I knew when I reopen indexes. Then what is the reason to wait for garbage collector.. And that too, FieldCache uses WeakHashMap and that may lead some memory leaks. Jelda > -Original Message- > From: John Wang [mailto:[EMAIL PROTECTED] > Sent: Friday, March

Re: FieldCache and Sort

2005-06-06 Thread Yonik Seeley
Why do we keep the lookup array around? The actual field value is needed to sort results from multiple searchers (multisearcher). -Yonik On 6/1/05, John Wang <[EMAIL PROTECTED]> wrote: > Hi: > >In the current Lucene sorting implementation, FieldCache is used to > retrieve 2 arrays, the looku