Just on remark here.
The high-watermark could be disregarded. The decision about the forward
depends on the size of the aggregated map.
Only 1 element long maps would be unpacked and forwarded. 0 element maps
would be published as delete. Any other count
of map entries is in "waiting for correct deletes to arrive"-state.
On 04.09.2018 21:29, Adam Bellemare wrote:
It does look like I could replace the second repartition store and
highwater store with a groupBy and reduce. However, it looks like I would
still need to store the highwater value within the materialized store, to
compare the arrival of out-of-order records (assuming my understanding of
THIS is correct...). This in effect is the same as the design I have now,
just with the two tables merged together.
