Re: SortingAtomicReader alternate to Tim-Sort...

2015-05-06 Thread Adrien Grand
Sorry for the delay, I opened https://issues.apache.org/jira/browse/LUCENE-6469. It can go to trunk and 5.x (the value of x depending on when it's ready :)). On Thu, Apr 30, 2015 at 9:02 AM, Ravikumar Govindarajan wrote: >> >> Would you like to submit a patch that changes SortingMergePolicy to >>

Re: SortingAtomicReader alternate to Tim-Sort...

2015-04-30 Thread Ravikumar Govindarajan
> > Would you like to submit a patch that changes SortingMergePolicy to > use the approach that you are proposing using bitsets instead of > sorting int[] arrays? Sure can do that. Can you open a ticket for this, as I don't know what versions this can go in? -- Ravi On Tue, Apr 28, 2015 at 6:

Re: SortingAtomicReader alternate to Tim-Sort...

2015-04-28 Thread Adrien Grand
On Tue, Apr 21, 2015 at 10:00 AM, Ravikumar Govindarajan wrote: > Thanks for the comments… > > My only >> concern about using the FixedBitSet is that it would make sorting each >> postings list run in O(maxDoc) but maybe we can make it better by >> using SparseFixedBitSet > > > Yes I was also thin

Re: SortingAtomicReader alternate to Tim-Sort...

2015-04-24 Thread Ravikumar Govindarajan
Thanks. Glad that it has been pro-actively identified and fixed -- Ravi On Thu, Apr 23, 2015 at 10:34 AM, Robert Muir wrote: > On Tue, Apr 21, 2015 at 4:00 AM, Ravikumar Govindarajan > wrote: > > > b) CompressingStoredFieldsReader did not store the last decoded 32KB > chunk. > > Our segments a

Re: SortingAtomicReader alternate to Tim-Sort...

2015-04-22 Thread Robert Muir
On Tue, Apr 21, 2015 at 4:00 AM, Ravikumar Govindarajan wrote: > b) CompressingStoredFieldsReader did not store the last decoded 32KB chunk. > Our segments are already sorted before participating in a merge. On mostly > linear merge, we ended up decoding the same chunk again and again. Simply > s

Re: SortingAtomicReader alternate to Tim-Sort...

2015-04-21 Thread Ravikumar Govindarajan
Thanks for the comments… My only > concern about using the FixedBitSet is that it would make sorting each > postings list run in O(maxDoc) but maybe we can make it better by > using SparseFixedBitSet Yes I was also thinking about this. But we are on 4.x and did not take the plunge. But as you sa

Re: SortingAtomicReader alternate to Tim-Sort...

2015-04-20 Thread Adrien Grand
I like these ideas, the int[] we are using today are wasteful. My only concern about using the FixedBitSet is that it would make sorting each postings list run in O(maxDoc) but maybe we can make it better by using SparseFixedBitSet (added in 5.0, given your code snippets I assume you are still on 4