Re: Statefull system consumer

2015-12-17 Thread Yi Pan
Hi, Anton, It seems to me that the best option would possibly use the row number as the IncomingMessageEnvelope's offset. Then, when Samza commits the checkpoint, it will commit the row number as the offset. When the Samza job restarts, the row number would be read from the checkpoint topic and yo

Statefull system consumer

2015-12-10 Thread Anton Polyakov
Hi guys In my samza setup I need to create a custom SystemConsumer. Basically I need to source data from some database table but due to Oracle constraints I can’t use DataBus. So I created a consumer who polls database and pushes new changes into stream by extending BlockingEnvelopMap and call