I do not mind the ordering as I have a Timestamp in all my messages and all my messaged land in a Timeseries database. So I understand that it is better to have just one Producer instance per JVM and use that to write to n number of topics. I mean even if I have 10,000 topics, I can just get away with a single Producer instance per JVM?
On Wed, May 25, 2016 at 8:41 AM, Ewen Cheslack-Postava <e...@confluent.io> wrote: > On Mon, Apr 25, 2016 at 6:34 AM, Joe San <codeintheo...@gmail.com> wrote: > > > I have an application that is currently running and is using Rx Streams > to > > move data. Now in this application, I have a couple of streams whose > > messages I would like to write to a single Kafka topic. Given this, I > have > > say Streams 1 to 5 as below: > > > > Stream1 - Takes in DataType A Stream2 - Takes in DataType B and so on > > > > Where these Streams are Rx Observers. All these data types that I get out > > of the stream are converted to a common JSON structure. I want this JSON > > structure to be pushed to a single Kafka topic. > > > > Now the questions are: > > > > 1. > > > > Should I create one KafkaProducer for each of those Streams or rather > Rx > > Observer instances? > > > > A single producer instance is fine. In fact, it may be better since you > share TCP connections and requests to produce data can be batched together. > > > > 2. > > > > What happens if multiple threads using its own instance of a > > KafkaProducer to write to the same topic? > > > > They can all write to the same topic, but their data will be arbitrarily > interleaved since there's no ordering guarantee across these producers. > > > > -- > Thanks, > Ewen >
