On Wed, Feb 4, 2009 at 10:41 AM, Mark Miller wrote:
> Todd Benge wrote:
>
>>
>> The intent is to reduce the amount of memory that is held in cache. As it
>> is now, it looks like there is an array of comparators for each index
>> reader. Most of the data in the ar
On Wed, Feb 4, 2009 at 10:01 AM, Mark Miller wrote:
> Todd Benge wrote:
>>
>> Hi,
>>
>> I've been looking into the FieldCache API because of memory problems
>> we've been seeing in our production environment.
>>
>> We use various differen
Hi,
I've been looking into the FieldCache API because of memory problems
we've been seeing in our production environment.
We use various different sorts so over time the cache builds up and
servers stop responding.
I decided to apply the patch for JIRA 831:
https://issues.apache.org/jira/browse
ussion of payloads (it is where I initially learned about them).
> In that thread should also be an explanation of the solution I ended up
> using for implementing payloads, so maybe that would be helpful for you.
>
> - Greg
>
> On Sun, Dec 14, 2008 at 11:47 PM, Todd Benge wrote
Hi,
I've been reading about payloads because I think we may be able to use
them to solve a problem we're having involving sorts.
>From the API, Payloads are set on Token classes. During indexing,
Fields are added to Documents so I think I'm missing the connection of
how the payloads are set into
Yup - goofed that up.
Thanks,
Todd
2008/11/5 Grant Ingersoll <[EMAIL PROTECTED]>:
>
> On Nov 5, 2008, at 12:04 PM, Todd Benge wrote:
>
>> I think it's like this:
>>
>> Sort sort = new Sort("FIELDNAME", true);
>> TopFieldDocs docs = searcher
I think it's like this:
Sort sort = new Sort("FIELDNAME", true);
TopFieldDocs docs = searcher.searcher(query, null, n, sort); // n is
the number of documents you want to retrieve
ScoreDoc[] hits = docs.scoreDocs;
for (int i = 0; i:
> hi.
> I have a question :)
>
> In lucene 2.3.X I did use Sort
Thanks Pablo.
I'll be flying to New Orleans tomorrow for ApacheCon and would love
the opportunity to talk with others about architectures others are
using.
Todd
On 11/4/08, PabloS <[EMAIL PROTECTED]> wrote:
>
> Sure Todd,
>
> the idea basically consist in the following:
>
> - Subclassing FIeld
Pablo,
Would you mind adding a little more detail about how you're working
around the problem?
I'm still evaluating our different options so am interested in what you did.
Todd
On Mon, Nov 3, 2008 at 2:37 PM, PabloS <[EMAIL PROTECTED]> wrote:
>
> Thanks hossman, but I've already 'solved' the pr
some great points (he the lucene master), especially possibly
> turning off norms if you can, but for an index like that i'd reccomwnd solr.
> Solr sharding can be scaled to billions (min a billion or two anyway) with
> few limitations (of course there are a few). Plus it has furt
o 32 bits x num docs - shorts, 32 bits x num docs, etc.
>
> So you have those field caches, plus the IndexReader terminfo, term stuff,
> plus whatever RAM your app needs beyond Lucene. 4 gig might just not *quite*
> cut it is my guess.
>
> Todd Benge wrote:
>>
>> Ther
There's usually only a couple sort fields and a bunch of terms in the
various indices. The terms are user entered on various media so the
number of terms is very large.
Thanks for the help.
Todd
On 10/29/08, Todd Benge <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I'm the le
Hi,
I'm the lead engineer for search on a large website using lucene for search.
We're indexing about 300M documents in ~ 100 indices. The indices add
up to ~ 60G.
The indices are sorted into 4 different Multisearcher with the largest
handling ~50G.
The code is basically like the following:
13 matches
Mail list logo
