Re: Question about many fields within a single index

2009-12-31 Thread Jason Tesser
right we do analyze a number of fields. We use the WHiteSpace whenever we have a text field. So maybe 5 on average per guy. Can be more of course. Thanks, Jason Tesser dotCMS Lead Development Manager 1-305-858-1422 On Wed, Dec 30, 2009 at 10:44 PM, Tom Hill wrote: > Hi - > > One thing to cons

Re: Question about many fields within a single index

2009-12-30 Thread Tom Hill
Hi - One thing to consider is field norms. If your fields aren't analyzed, this doesn't apply to you. But if you do have norms, I believe that it's one by per field with norms x number of documents. It doesn't matter if the field occurs in a document or not, it's nTotalFields x nDocs. So, an ind

Re: Question about many fields within a single index

2009-12-30 Thread Renaud Delbru
Hi, just sharing some personal experiences in this domain, We performed some benchmarks in a similar setup (indexing millions of documents with thousands of fields) to measure the impact of large number of fields on a Lucene index. We observed that more you have fields, more the dictionary wil

Re: Question about many fields within a single index

2009-12-30 Thread Erick Erickson
As far as I know, no problem. There's no penalty that I know of for having this kind of setup. Of course your mileage may vary, and a relevant question is "why do you care?" That is, if your total index is 100M in size, pretty much no matter how Lucene implements the internal data structures you wo