Re: Flex API - Debugging Segment Merge

2010-03-27 Thread Michael McCandless
Phew, I'm glad to hear you got to the bottom of it! Good sleuthing. And I'm looking forwarded to your results and hopefully patches that make these various encoding techniques available as flex codecs :) Mike On Fri, Mar 26, 2010 at 6:08 PM, Renaud Delbru wrote: > Hi Michael, > > On 25/03/10 1

Re: Flex API - Debugging Segment Merge

2010-03-26 Thread Renaud Delbru
Hi Michael, On 25/03/10 19:15, Michael McCandless wrote: I am using one single thread for indexing: reading sequentially the list of wikipedia articles, putting the content into a single field, and add the document to the index. Commit is done every 10K documents. Are you using contrib/be

Re: Flex API - Debugging Segment Merge

2010-03-25 Thread Michael McCandless
On Thu, Mar 25, 2010 at 3:04 PM, Renaud Delbru wrote: > Hi Michael, > > On 25/03/10 18:45, Michael McCandless wrote: >> >> Hi Renaud, >> >> It's great that you're pushing flex forward so much :) You're making >> some cool sounding codecs!  I'm really looking forward to seeing >> indexing/searching

Re: Flex API - Debugging Segment Merge

2010-03-25 Thread Renaud Delbru
Hi Michael, On 25/03/10 18:45, Michael McCandless wrote: Hi Renaud, It's great that you're pushing flex forward so much :) You're making some cool sounding codecs! I'm really looking forward to seeing indexing/searching performance results on Wikipedia... I'll share them for sure whenever

Re: Flex API - Debugging Segment Merge

2010-03-25 Thread Michael McCandless
Hi Renaud, It's great that you're pushing flex forward so much :) You're making some cool sounding codecs! I'm really looking forward to seeing indexing/searching performance results on Wikipedia... It sounds most likely there's a bug in the PFor impl? (Since you don't hit this exception with th

Flex API - Debugging Segment Merge

2010-03-25 Thread Renaud Delbru
Hi, I am currently benchmarking various compression algorithms using the Sep Codec, but I got index corruption exception during the merge process, and I would need your help to debug it. I have reimplemented various algorithms like FOR, Simple9, VInt, PFor for the Sep IntBlock Codec. I am be