Thanks for the answer. I asked about it because we need it and I was about writing a summer intern proposal for a student to work on it. Looks like it could work fine.
On Wed, Mar 24, 2021 at 3:49 PM Wes McKinney <wesmck...@gmail.com> wrote: > The SparseTensor stuff is something else entirely (that's matrices > where the entries are mostly 0) > > There isn't anything to help you right now aside from dictionary > encoding — if your dictionary has 256 elements or less, you can use > uint8 index type and thus have 1 byte per value. We've discussed > implementing RLE in the project and so if we do that in the future > then a random access data structure could be built on top of RLE (in > principle) > > On Wed, Mar 24, 2021 at 8:53 AM Niranda Perera <niranda.per...@gmail.com> > wrote: > > > > Hi Lykov, > > > > I believe there's an arrow sparse tensor abstraction. > > > > On Wed, Mar 24, 2021, 05:05 Kirill Lykov <lykov.kir...@gmail.com> wrote: > > > > > Hi, > > > > > > I wonder if there is an existing way to store floats/ints with many > > > repetitions in some container (not sure about terminology). > > > For example, I might have data like A=[1, 2, 2, 2, 3, 3, 3, 3, 3, 3, > 1, 2] > > > and i would like to store only B=[1, 2, 3, 1, 2] but from user > > > perspective it behaves like container A. I know I can use dictionary > but as > > > far I understand it will store internally indices of the chosen > elements so > > > it makes sense more for binary data or structures. > > > > > > -- > > > Best regards, > > > Kirill Lykov > > > > -- Best regards, Kirill Lykov
