> > > So, why would we allow dictionary replacement instead of have the > > emitter use a new dictionary id? Is it to optimize memory consumption > > on the receiver?
> The dictionary id's are set in the schema, so it's not possible to > change the dictionary id after the schema has been sent. It is conceivably possible to add another field to DictionaryBatch to indicate the dictionary is a replacement for another ID, but this seems like it would make the IPC protocol messier (i.e. we could eliminate isDelta). But I think this is more confusing from a protocol perspective. It might make implementations simpler though. On Sun, Oct 6, 2019 at 1:25 PM Wes McKinney <wesmck...@gmail.com> wrote: > On Sun, Oct 6, 2019 at 4:30 AM Antoine Pitrou <solip...@pitrou.net> wrote: > > > > On Sat, 5 Oct 2019 17:01:27 -0600 > > Micah Kornfield <emkornfi...@gmail.com> wrote: > > > I've opened a pull request [1] to clarify some recent conversations > about > > > semantics/edge cases for dictionary encoding [2][3] around interleaved > > > batches and when isDelta=False. > > > > > > Specifically, it proposes isDelta=False indicates dictionary > replacement. > > > > So, why would we allow dictionary replacement instead of have the > > emitter use a new dictionary id? Is it to optimize memory consumption > > on the receiver? > > The dictionary id's are set in the schema, so it's not possible to > change the dictionary id after the schema has been sent. > > > Regards > > > > Antoine. > > > > >
