You can see how we've done bookkeeping for some other vendored code from Daniel Lemire
https://github.com/apache/arrow/blob/master/LICENSE.txt#L313 On Wed, Apr 29, 2020 at 3:05 AM Antoine Pitrou <[email protected]> wrote: > > Hi Frank, > > Vendoring just the required lookup table sounds fine to me. > > Regards > > Antoine. > > > > Le 29/04/2020 à 08:25, Du, Frank a écrit : > > Hi All, > > > > Currently we are working on SIMD chance to improves speed of > DecodeSpaced on Jira<https://issues.apache.org/jira/browse/PARQUET-1841>, > the path for AVX512 is identified with the helper of vpcompressd/vpexpandd > instructions. I has uploaded the PR< > https://github.com/apache/arrow/pull/7029> for this path and other SIMD > unittest, benchmark things. > > > > For the SSE chance, thanks to help from Micah Kornfield, the path can be > achieved by a lookup table, method referenced from > https://github.com/lemire/simdprune. I has a draft implementation locally > which demonstrate double speed on epi32 path and 50% improvements on epi64. > > > > The issue is how to approach the reference, one way is vendor the whole > simdprune library while another choice is just introducing the lookup > table/script we really used. I personally like the later one as we need > expand the table to support both encode/decode, simdprune only has encode > table, also we need a new epi64 table to support the double type. Thoughts > are welcome😊 > > > > Thanks, > > Frank > > >
