Hi Everyone
The stats I sent through earlier were erroneous due to fact the date
range query selected fewer records than stated.
The correct stats are:
Lucene 3.0 Stats:
Search conducted using Lucene's Realtime search feature
(writer.getReader() for each search)
Analyzer: Russian Analyzer
Total Docs: 26.04M (emails data - all attachments, body content indexed)
Index Size: 37G
Query: body: test AND date: [200901010101 to 201003220225] with
descending sort on date.
Lucene Mem Usage: 32 MB (when sorted on date)
Search Speed: 0.48 (unsorted)
Search Speed: 0.49 (sorted on YYYYMMDDHHSS date)
Hardware / Software:
Index stored on 4 SAS HDD hitachi RAID 10
16G RAM
2x Xeon 4 core 2.4Gz
OS FreeBSD 7.2
Filesystem UFS2 gjournal
From Yourkit, memory usage is as follows:
Name Number of Objects Shallow Size
org.apache.lucene.index.TermInfo 370610 14824400
org.apache.lucene.index.Term 370505 11856160
com.stimulus.archiva.search.LuceneResult 20000 960000
org.apache.lucene.search.FieldDoc 10000 320000
org.apache.lucene.search.ScoreDoc 10000 240000
org.apache.lucene.index.FieldInfo 1027 41080
org.apache.lucene.index.SegmentReader$Norm 840 67200
org.apache.lucene.document.Field 578 36992
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput 415 49800
org.apache.lucene.index.CompoundFileReader$CSIndexInput 380 36480
org.apache.lucene.util.UnicodeUtil$UTF8Result 320 10240
org.apache.lucene.index.TermBuffer 315 17640
org.apache.lucene.util.UnicodeUtil$UTF16Result 315 12600
org.apache.lucene.index.CompoundFileReader$FileEntry 280 8960
org.apache.lucene.util.CloseableThreadLocal 260 8320
org.apache.lucene.index.FreqProxTermsWriter$PostingList 256 12288
org.apache.lucene.index.SegmentInfo 185 19240
org.apache.lucene.index.SegmentReader$FieldsReaderLocal 140 5600
org.apache.lucene.index.ReadOnlySegmentReader 105 12600
org.apache.lucene.index.SegmentReader$Ref 105 2520
org.apache.lucene.index.SegmentTermEnum 105 11760
org.apache.lucene.search.FieldCacheImpl$Entry 105 3360
org.apache.lucene.index.TermInfosReader$ThreadResources 70 2240
org.apache.lucene.util.cache.SimpleLRUCache 70 1680
org.apache.lucene.util.cache.SimpleLRUCache$1 70 6160
org.apache.lucene.index.FieldsReader 50 4400
org.apache.lucene.index.IndexFileDeleter$RefCount 42 1344
org.apache.lucene.document.Document 41 1312
org.apache.lucene.util.SimpleStringInterner$Entry 41 1640
org.apache.lucene.index.FieldInfos 37 1480
org.apache.lucene.index.CompoundFileReader 35 2520
org.apache.lucene.index.SegmentReader 35 4200
org.apache.lucene.index.SegmentReader$CoreReaders 35 4480
org.apache.lucene.index.TermInfo[] 35 2963448
org.apache.lucene.index.TermInfosReader 35 3360
org.apache.lucene.index.Term[] 35 2963448
org.apache.lucene.search.FieldCache$CreationPlaceholder 35 840
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor 35
2240
org.apache.lucene.index.RawPostingList[] 30 8184
org.apache.lucene.index.TermsHashPerField 24 3648
org.apache.lucene.analysis.WhitespaceAnalyzer 16 512
org.apache.lucene.document.Fieldable[] 13 416
org.apache.lucene.index.DocFieldProcessorPerField 12 672
org.apache.lucene.index.DocInverterPerField 12 768
org.apache.lucene.index.FreqProxTermsWriterPerField 12 864
org.apache.lucene.index.NormsWriterPerField 12 864
org.apache.lucene.index.TermVectorsTermsWriterPerField 12 960
org.apache.lucene.util.AttributeSource$State 12 384
org.apache.lucene.util.Version 8 256
org.apache.lucene.index.SegmentInfos 7 616
org.apache.lucene.document.FieldSelectorResult 6 192
org.apache.lucene.analysis.CharArraySet$UnmodifiableCharArraySet 5
160
org.apache.lucene.analysis.tokenattributes.OffsetAttributeImpl 5 120
org.apache.lucene.analysis.tokenattributes.TermAttributeImpl 5 160
org.apache.lucene.analysis.tokenattributes.PositionIncrementAttributeImpl
4 96
org.apache.lucene.index.BufferedDeletes 4 224
org.apache.lucene.index.TermsHash 4 320
org.apache.lucene.analysis.LowerCaseFilter 3 192
org.apache.lucene.analysis.PerFieldAnalyzerWrapper 3 144
org.apache.lucene.analysis.SimpleAnalyzer 3 96
org.apache.lucene.analysis.StopFilter 3 264
org.apache.lucene.analysis.ru.RussianAnalyzer 3 144
org.apache.lucene.analysis.ru.RussianAnalyzer$SavedStreams 3 120
org.apache.lucene.analysis.ru.RussianLetterTokenizer 3 288
org.apache.lucene.analysis.ru.RussianStemFilter 3 216
org.apache.lucene.analysis.ru.RussianStemmer 3 96
org.apache.lucene.index.IndexReader[] 3 912
org.apache.lucene.index.ReadOnlyDirectoryReader 3 384
org.apache.lucene.index.SegmentReader[] 3 912
org.apache.lucene.search.Sort 3 72
org.apache.lucene.search.SortField 3 168
org.apache.lucene.search.SortField[] 3 96
org.apache.lucene.search.TermQuery 3 96
org.apache.lucene.util.NamedThreadFactory 3 120
org.apache.lucene.analysis.CharReader 2 80
org.apache.lucene.index.ByteBlockPool 2 112
org.apache.lucene.index.ConcurrentMergeScheduler 2 112
org.apache.lucene.index.DocFieldProcessor 2 96
org.apache.lucene.index.DocFieldProcessorPerField[] 2 432
org.apache.lucene.index.DocInverter 2 80
org.apache.lucene.index.DocumentsWriter 2 608
org.apache.lucene.index.DocumentsWriter$ByteBlockAllocator 2 64
org.apache.lucene.index.DocumentsWriter$DocWriter[] 2 208
org.apache.lucene.index.DocumentsWriter$SkipDocWriter 2 64
org.apache.lucene.index.DocumentsWriter$WaitQueue 2 112
org.apache.lucene.index.DocumentsWriterThreadState[] 2 56
org.apache.lucene.index.FreqProxTermsWriter 2 80
org.apache.lucene.index.IndexFileDeleter 2 192
org.apache.lucene.index.IndexFileDeleter$CommitPoint 2 176
org.apache.lucene.index.IndexWriter 2 592
org.apache.lucene.index.IndexWriter$MaxFieldLength 2 64
org.apache.lucene.index.IndexWriter$ReaderPool 2 64
org.apache.lucene.index.IntBlockPool 2 112
org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy 2 32
org.apache.lucene.index.LogByteSizeMergePolicy 2 112
org.apache.lucene.index.NormsWriter 2 48
org.apache.lucene.index.StoredFieldsWriter 2 128
org.apache.lucene.index.StoredFieldsWriter$PerDoc[] 2 64
org.apache.lucene.index.TermVectorsTermsWriter 2 176
org.apache.lucene.index.TermVectorsTermsWriter$PerDoc[] 2 64
org.apache.lucene.index.TermsHashPerThread 2 176
org.apache.lucene.queryParser.QueryParser$Operator 2 64
org.apache.lucene.search.BooleanClause 2 64
org.apache.lucene.search.NumericRangeQuery 2 160
org.apache.lucene.search.ParallelMultiSearcher 2 144
org.apache.lucene.search.QueryWrapperFilter 2 48
org.apache.lucene.search.ScoreDoc[] 2 48
org.apache.lucene.search.Searchable[] 2 64
org.apache.lucene.store.NIOFSDirectory 2 96
org.apache.lucene.store.NativeFSLock 2 128
org.apache.lucene.store.NativeFSLockFactory 2 80
org.apache.lucene.analysis.CharArraySet 1 32
org.apache.lucene.analysis.LowerCaseTokenizer 1 96
org.apache.lucene.document.Field$Index$1 1 32
org.apache.lucene.document.Field$Index$2 1 32
org.apache.lucene.document.Field$Index$3 1 32
org.apache.lucene.document.Field$Index$4 1 32
org.apache.lucene.document.Field$Index$5 1 32
org.apache.lucene.document.Field$Index[] 1 64
org.apache.lucene.document.Field$Store$1 1 32
org.apache.lucene.document.Field$Store$2 1 32
org.apache.lucene.document.Field$Store[] 1 40
org.apache.lucene.document.Field$TermVector$1 1 32
org.apache.lucene.document.Field$TermVector$2 1 32
org.apache.lucene.document.Field$TermVector$3 1 32
org.apache.lucene.document.Field$TermVector$4 1 32
org.apache.lucene.document.Field$TermVector$5 1 32
org.apache.lucene.document.Field$TermVector[] 1 64
org.apache.lucene.document.FieldSelectorResult[] 1 72
org.apache.lucene.document.Field[] 1 24
org.apache.lucene.index.ByteSliceReader 1 80
org.apache.lucene.index.CharBlockPool 1 56
org.apache.lucene.index.DocFieldProcessorPerThread 1 112
org.apache.lucene.index.DocFieldProcessorPerThread$PerDoc[] 1 32
org.apache.lucene.index.DocInverterPerThread 1 72
org.apache.lucene.index.DocInverterPerThread$SingleTokenAttributeSource 1
64
org.apache.lucene.index.DocumentsWriter$1 1 16
org.apache.lucene.index.DocumentsWriter$DocState 1 72
org.apache.lucene.index.DocumentsWriterThreadState 1 48
org.apache.lucene.index.FieldInvertState 1 48
org.apache.lucene.index.FieldsWriter 1 48
org.apache.lucene.index.FreqProxTermsWriterPerThread 1 32
org.apache.lucene.index.IndexFileNameFilter 1 32
org.apache.lucene.index.NormsWriterPerThread 1 32
org.apache.lucene.index.ReusableStringReader 1 48
org.apache.lucene.index.SegmentWriteState 1 72
org.apache.lucene.index.StoredFieldsWriter$PerDoc 1 56
org.apache.lucene.index.StoredFieldsWriterPerThread 1 48
org.apache.lucene.index.TermVectorsTermsWriterPerThread 1 72
org.apache.lucene.queryParser.QueryParser$Operator[] 1 40
org.apache.lucene.search.BooleanClause$Occur$1 1 32
org.apache.lucene.search.BooleanClause$Occur$2 1 32
org.apache.lucene.search.BooleanClause$Occur$3 1 32
org.apache.lucene.search.BooleanClause$Occur[] 1 48
org.apache.lucene.search.BooleanQuery 1 40
org.apache.lucene.search.DefaultSimilarity 1 24
org.apache.lucene.search.DocIdSet$1 1 24
org.apache.lucene.search.DocIdSet$1$1 1 32
org.apache.lucene.search.FieldCache$1 1 16
org.apache.lucene.search.FieldCache$10 1 16
org.apache.lucene.search.FieldCache$2 1 16
org.apache.lucene.search.FieldCache$3 1 16
org.apache.lucene.search.FieldCache$4 1 16
org.apache.lucene.search.FieldCache$5 1 16
org.apache.lucene.search.FieldCache$6 1 16
org.apache.lucene.search.FieldCache$7 1 16
org.apache.lucene.search.FieldCache$8 1 16
org.apache.lucene.search.FieldCache$9 1 16
org.apache.lucene.search.FieldCacheImpl 1 32
org.apache.lucene.search.FieldCacheImpl$ByteCache 1 32
org.apache.lucene.search.FieldCacheImpl$DoubleCache 1 32
org.apache.lucene.search.FieldCacheImpl$FloatCache 1 32
org.apache.lucene.search.FieldCacheImpl$IntCache 1 32
org.apache.lucene.search.FieldCacheImpl$LongCache 1 32
org.apache.lucene.search.FieldCacheImpl$ShortCache 1 32
org.apache.lucene.search.FieldCacheImpl$StringCache 1 32
org.apache.lucene.search.FieldCacheImpl$StringIndexCache 1 32
org.apache.lucene.search.MultiTermQuery$1 1 32
org.apache.lucene.search.MultiTermQuery$ConstantScoreBooleanQueryRewrite 1
16
org.apache.lucene.search.MultiTermQuery$ConstantScoreFilterRewrite 1
16
org.apache.lucene.search.MultiTermQuery$ScoringBooleanQueryRewrite 1
16
org.apache.lucene.search.TopDocs 1 32
org.apache.lucene.store.RAMFile 1 56
org.apache.lucene.store.RAMOutputStream 1 72
org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory
1 16
org.apache.lucene.util.SimpleStringInterner 1 32
org.apache.lucene.util.SimpleStringInterner$Entry[] 1 8216
org.apache.lucene.util.UnicodeUtil$UTF8Result[] 1 40
org.apache.lucene.util.Version[] 1 88
34562928 <mailto:=...@sum%28c1:C193%29>
32.96
