Re: DocValues memory usage

2013-03-28 Thread Peter Keegan
This is wierd. I indexed using DiskDocValuesFormat as the default codec and observed 16K qps with BinaryDocValuesField. But with a simple StoredField, I observed a much higher 30K qps. When I added both fields (BinaryDocValuesField and StoredField) to the index, I observed only 100 qps on each fiel

Re: DocValues memory usage

2013-03-26 Thread Michael McCandless
DiskDocValuesFormat is the right thing to use: it loads certain things into RAM, eg the compressed bits that tell it the addresses of the bytes on disk, but then leaves the actual bytes on disk. I believe the old DirectSource was more extreme as it left the addresses on disk too, so there were 2 s

Re: DocValues memory usage

2013-03-26 Thread Duke
I made the same experiment and got same result. Then I used per-field codec with DiskDocValuesFormat, it works like DirectSource in 4.0.0, but I'm not feeling confident with this usage. Anyone can say more about removing DirectSource API? On 2013-3-26, at 22:59, Peter Keegan wrote: > Inspir