Re: Passing records between two jobs

Hi Avihai, Rafi pointed out the two common approaches to deal with this situation. Let me expand a bit on those. 1) Transactional producing in to queues: There are two approaches to accomplish exactly-once producing into queues, 1) using a system with transactional support such as Kafka or 2) mai

Re: Passing records between two jobs

Hi Avihai, The problem is that every message queuing sink only provides at-least-once > guarantee > >From what I see, possible messaging queue which guarantees exactly-once is Kafka 0.11, while using the Kafka transactional messaging

Passing records between two jobs

Hello, We are planning a system that will be comprised of 3 different jobs: 1. Getting a stream of events, adding some metadata to the events, and outputting them to a temporary message queue. 2. Performing some calculations on the events we got from job 1, as required for product A. 3.