Re: [Java] IPC stream write with re-stated dictionaries

2021-03-05 Thread Joris Peeters
Cheers. Made and self-assigned https://issues.apache.org/jira/browse/ARROW-11869. On Fri, Mar 5, 2021 at 1:44 AM Micah Kornfield wrote: > Hi Joris, > I do believe this is missing. I believe we worked around this for testing > by directly writing dictionary batches to the stream [1]. > > Thanks,

Re: [Java] IPC stream write with re-stated dictionaries

2021-03-04 Thread Micah Kornfield
Hi Joris, I do believe this is missing. I believe we worked around this for testing by directly writing dictionary batches to the stream [1]. Thanks, Micah [1] https://github.com/apache/arrow/blob/master/java/vector/src/test/java/org/apache/arrow/vector/ipc/TestArrowReaderWriter.java#L614 On Th

[Java] IPC stream write with re-stated dictionaries

2021-03-04 Thread Joris Peeters
Hello, For my use case I'm sending an Arrow IPC-stream from a server to a client, with some columns being dictionary-encoded. Dictionary-encoding happens on the fly, though, so the full dictionary isn't known yet at the beginning of the stream, but rather is computed for every batch, and Dictionar