The SparseTensor stuff is something else entirely (that's matrices where the entries are mostly 0)
There isn't anything to help you right now aside from dictionary encoding — if your dictionary has 256 elements or less, you can use uint8 index type and thus have 1 byte per value. We've discussed implementing RLE in the project and so if we do that in the future then a random access data structure could be built on top of RLE (in principle) On Wed, Mar 24, 2021 at 8:53 AM Niranda Perera <niranda.per...@gmail.com> wrote: > > Hi Lykov, > > I believe there's an arrow sparse tensor abstraction. > > On Wed, Mar 24, 2021, 05:05 Kirill Lykov <lykov.kir...@gmail.com> wrote: > > > Hi, > > > > I wonder if there is an existing way to store floats/ints with many > > repetitions in some container (not sure about terminology). > > For example, I might have data like A=[1, 2, 2, 2, 3, 3, 3, 3, 3, 3, 1, 2] > > and i would like to store only B=[1, 2, 3, 1, 2] but from user > > perspective it behaves like container A. I know I can use dictionary but as > > far I understand it will store internally indices of the chosen elements so > > it makes sense more for binary data or structures. > > > > -- > > Best regards, > > Kirill Lykov > >
