I'm trying to get a feel for the impact of changing the termIndexInterval from 
the default of 128 to 1024 (8 * 128).  This reduces the size of the tii file by 
1/8th but in the worst case requires doing a linear scan of 1024 terms instead 
of 128 in memory.   I'm not so concerned about the performance impact of the 
in-memory scan, but I was trying to get an idea about how this affects disk 
I/O. i.e. assuming a term is not in the tii file, we need to  load 1024 terms 
from the tis file instead of 128.

I looked at the output of a CheckIndex on one of our very large segments to get 
the number of terms in the segment (see below) and got about 2.7 billion terms. 
(We have lots of dirty OCR from 400 languages) .  The tis file is about  24.7 
GB. I divided the size of the tis file for that segment in bytes by the number 
of terms to get the average number of bytes/term:

(24.7 * (10^9) bytes ) / (2.7 * (10^9) terms) = 9 bytes/term.

This is the average size of a term entry in the tis file (assuming CheckIndex 
and ls outputs are correct).
This seems too small.   Looking at the Lucene File formats doc (excerpt below), 
if we assume that everything other than the Suffix of the term takes a VInt 
that only occupies 1 byte, we have 6 bytes for that data, which leaves only 3 
bytes for the String that holds the Suffix.

What am I missing here?

Tom Burton-West


-------------------------------------------------------------------------------------------------------

>From the Lucene File formats doc:

TermInfo --> <Term, DocFreq, FreqDelta, ProxDelta, SkipDelta>
Term --> <PrefixLength, Suffix, FieldNum>
Suffix --> String
PrefixLength, DocFreq, FreqDelta, ProxDelta, SkipDelta
--> VInt

1 of 2: name=_2cj docCount=708,639
    compound=false
    hasProx=true
    numFiles=9
    size (MB)=393,395.313
    diagnostics = {optimize=true, mergeFactor=9, os.version=2.6.18-238.1.1.el5, 
os=Linux, mergeDocStores=true, lu
cene.version=3.1-SNAPSHOT 1036094 - 2010-11-19 16:01:10, source=merge, 
os.arch=amd64, java.version=1.6.0_20, java
.vendor=Sun Microsystems Inc.}
    has deletions [delFileName=_2cj_2.del]
    test: open reader.........OK [24 deleted docs]
    test: fields..............OK [55 fields]
    test: field norms.........OK [17 fields]
    test: terms, freq, prox...OK [2,723,440,775 terms; 35740903735 terms/docs 
pairs; 154861967859 tokens]
    test: stored fields.......OK [11040443 total field count; avg 15.58 fields 
per doc]
    test: term vectors........OK [0 total vector count; avg 0 term/freq vector 
fields per doc]

[xxx@shotz-1 index]$ ls -l _2cj.tis
-rw-rw-r-- 1 tomcat dlps 24,775,378,328 Mar 12 17:16 _2cj.tis

Reply via email to