Thank you Addison, this is very helpful.
On Fri, Mar 22, 2019 at 10:12 AM Addison Higham wrote:
> Our implementation has quite a bit more going on just to deal with
> serialization of types, but here is pretty much the core of what we do in
> (psuedo) scala:
>
> class DynamoSink[...](...) extend
Our implementation has quite a bit more going on just to deal with
serialization of types, but here is pretty much the core of what we do in
(psuedo) scala:
class DynamoSink[...](...) extends RichProcessFunction[T] with
ProcessingTimeCallback {
private var curTimer: Option[Long] = None
privat
Hi there,
We have implemented a dynamo sink, have had no real issues, but obviously,
it is at-least-once and so we work around that by just structuring our
transformations so that they produce idempotent writes.
What we do is pretty similar to what you suggest, we collect the records in
a buffer
