[
https://issues.apache.org/jira/browse/LUCENE-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431753#comment-13431753
]
Michael McCandless commented on LUCENE-3892:
--------------------------------------------
Shouldn't MIN_ENCODED_SIZE be MAX_ENCODED_SIZE? Ie the max number of
bytes encoding will ever require. And I think the same for
MIN -> MAX_DATA_SIZE? Or maybe MIN_REQUIRED_XXX?
I think readVIntBlock shouldn't be in ForUtil? Ie it's very
postings-format-specific and it's not using packed ints at all. Also
the "equivalent" readVIntBlock code for the positions case (in the
readPositions methods) is still in the BlockPackedPostingsReader. I
think it's great to have writeBlock/readBlock/skipBlock in ForUtil.
Do we really need to write/write the 32 format.getId(), numBits into
the postings file header? I guess it's either that or ... store the float
acceptableOverheadRatio (eg using Float.floatToIntBits I guess) and
have some back-compat enforced in the logic in
PackedInts.fastestFormatAndBits... hmm.
Hmm ... MIN_DATA_SIZE is 147 (PACKED_SINGLE_BLOCK, bpv=3), but
BLOCK_SIZE is 128 ... so I guess this means if we ever pick that
format (because acceptableOverheadRatio allowed us to), we're
encoding/decoding those extra 19 unused ints right? (I was just
trying to understand why we alloc all the int[] to MIN_DATA_SIZE not
BLOCK_SIZE...).
ForUtil.getMinRequiredBufferSize seems like dead code?
> Add a useful intblock postings format (eg, FOR, PFOR, PFORDelta,
> Simple9/16/64, etc.)
> -------------------------------------------------------------------------------------
>
> Key: LUCENE-3892
> URL: https://issues.apache.org/jira/browse/LUCENE-3892
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Michael McCandless
> Labels: gsoc2012, lucene-gsoc-12
> Fix For: 4.1
>
> Attachments: LUCENE-3892-BlockTermScorer.patch,
> LUCENE-3892-blockFor&hardcode(base).patch,
> LUCENE-3892-blockFor&packedecoder(comp).patch,
> LUCENE-3892-blockFor-with-packedints-decoder.patch,
> LUCENE-3892-blockFor-with-packedints-decoder.patch,
> LUCENE-3892-blockFor-with-packedints.patch, LUCENE-3892-bulkVInt.patch,
> LUCENE-3892-direct-IntBuffer.patch, LUCENE-3892-for&pfor-with-javadoc.patch,
> LUCENE-3892-handle_open_files.patch, LUCENE-3892-non-specialized.patch,
> LUCENE-3892-pfor-compress-iterate-numbits.patch,
> LUCENE-3892-pfor-compress-slow-estimate.patch, LUCENE-3892_for_byte[].patch,
> LUCENE-3892_for_int[].patch, LUCENE-3892_for_unfold_method.patch,
> LUCENE-3892_pfor_unfold_method.patch, LUCENE-3892_pulsing_support.patch,
> LUCENE-3892_settings.patch, LUCENE-3892_settings.patch
>
>
> On the flex branch we explored a number of possible intblock
> encodings, but for whatever reason never brought them to completion.
> There are still a number of issues opened with patches in different
> states.
> Initial results (based on prototype) were excellent (see
> http://blog.mikemccandless.com/2010/08/lucene-performance-with-pfordelta-codec.html
> ).
> I think this would make a good GSoC project.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
