Hi folks, I'm working on multi-batch dictionary with delta support in Java [1] and would like some clarifications. Given the "isDelta" flag in the dictionary message [2], when should this be set to "true"?
1) If we have dictionary with an ID of 1 that we want to delta encode and it is used across multiple batches, should the initial batch have `isDelta=false` then subsequent batches have `isDelta=true`? E.g. batch 1, dict 1, isDelta=false, dictVector=[a, b, c], indexVector=[0, 1, 1, 2] batch 2, dict 1, isDelta=true, dictVector=[d], indexVector=[2, 3, 0, 1] batch 3, dict 1, isDelta=true, dictVector=[e], indexVector=[0, 4] Or should the flag be true for the entire IPC flow? E.g. batch 1, dict 1, isDelta=true, dictVector=[a, b, c], indexVector=[0, 1, 1, 2] batch 2, dict 1, isDelta=true, dictVector=[d], indexVector=[2, 3, 0, 1] batch 3, dict 1, isDelta=true, dictVector=[e], indexVector=[0, 4, 3] Either works for me. 2) Could (in stream, not file IPCs) a single dictionary ever switch state across batches from delta to replacement mode or vice-versa? E.g. batch 1, dict 1, isDelta = true, dictVector=[a, b, c], indexVector=[0, 1, 1, 2] batch 2, dict 1, isDelta = true, dictVector=[d], indexVector=[2, 3, 0, 1] batch 3, dict 1, isDelta = false, dictVector=[c, a, d], indexVector=[0, 1, 2] I'd like to keep the protocol and API simple and assume switching is not allowed. This would mean the 2nd example above would be canonical. 3) Are replacement dictionaries required to be serialized for every batch or is a dictionary re-used across batches until a replacement is received? The CPP IPC API has 'unify_dictionaries' [3] that mentions "a column with a dictionary type must have the same dictionary in each record batch". I assume (and prefer) the latter, that replacements are serialized once and re-used. E.g. batch 1, dict 1, isDelta = false, dictVector=[a, b, c], indexVector=[0, 1, 1, 2] batch 2, dict 1, isDelta = false, dictVector=[], indexVector=[2, 1, 0, 1] // use previous dictionary batch 3, dict 1, isDelta = false, dictVector=[c, a, d], indexVector=[0, 1, 2] // replacement And I assume that 'unify_dictionaries' simply concatenates all dictionaries into a single vector serialized in the first batch (haven't looked at the code yet). 4) Is it valid for a delta dictionary to have an update in a subsequent batch even though the update is not used in that batch? A silly example would be: batch 1, dict 1, isDelta = true, dictVector=[a, b, c], indexVector=[0, 1, 1, 2] batch 2, dict 1, isDelta = true, dictVector=[d], indexVector=[null, null, null, null] batch 3, dict 1, isDelta = true, dictVector=[], indexVector=[0, 3, 2] Thanks for your help! [1] https://github.com/apache/arrow/pull/38423 [2] https://github.com/apache/arrow/blob/main/format/Message.fbs#L134 [3] https://arrow.apache.org/docs/cpp/api/ipc.html#_CPPv4N5arrow3ipc15IpcWriteOptions18unify_dictionariesE -- Chris Larsen
