hi Liya, Do you intend to be able to send RLE vectors using the IPC protocol? If so, we need to spend some time on Micah's discussion about sparseness and encodings/compression.
- Wes On Wed, Aug 21, 2019 at 7:33 AM Fan Liya <liya.fa...@gmail.com> wrote: > > Dear all, > > RLE (run length encoding) is a widely used encoding/decoding technique. > Compared with other encoding/decoding techniques, it is easier to work with > the encoded data. > > We want to provide an RLE vector implementation in Arrow. The design > details include: > > 1. RleVector implements ValueVector. > 2. the data structure of RleVector includes an inner vector, plus a > repetition buffer. > 3. we do not provide random access over the RleVector > 4. In the future, we will provide iterators to access the vector in > sequence. > 5. RleVector does not support update, but supports appending. > 6. In the future, we will provide encoder/decoder to efficiently transform > encoded/decoded vectors. > > Please give your valuable feedback. > > Best, > Liya Fan
