Rest Api is for you to have the standard interface , so that you can hide the implementation details of pushing data to Kafka topic .
You don't need number of producers same as data providers . You can have one Producer to send data to Kafka topic . Also , if you have requirement of dividing data to multiple Kafka topics, one for each data provider , then I think it would be over utilization of resources and would be difficult to manage. You may want to add data provider information in the payload so that you can have all data in one topic . It again depends on your requirements and SLA, Once you have data in topic , you can use KStreams / Kafka consumer and write data in NOSQL data store Thanks, Snehal On 13 July 2016 at 13:52, Luo, Chao <chao....@here.com> wrote: > Hi Snehal, > > Thanks for your input. They already have their own Java APIs to access > data. But why do I create rest API? What is the benefits? > > Say, if there are 500 data providers, do I need 500 producers at my end to > collect data? At least, the number of producers should be proportional to > number of data providers. In addition, I also need maybe 500 or more kafka > servers. You see, I think the system has too many producers. It is waste of > money. > > > Best, > Chao > > -----Original Message----- > From: Snehal Nagmote [mailto:nagmote.sne...@gmail.com] > Sent: Wednesday, July 13, 2016 3:38 PM > To: users@kafka.apache.org > Subject: Re: Role of Producer > > Hi Chao , > > To solve this problem , I can think of creating rest api . Your end point > can have one of the parameter as data provider if you want to send it to > different topics based on data provider . > > On backend , when you get data , you can send it to Kafka Topics, using > Kafka Producer at the end. > > Thanks, > Snehal > > > > On 13 July 2016 at 13:31, Luo, Chao <chao....@here.com> wrote: > > > Dear Kafka guys, > > > > I just started to build up a Kafka system two weeks ago. Here I have a > > question about how to design/implement the producer. > > > > In my system, there are many data providers. I need to collect > > real-time data from them and store it in a NoSQL database. The problem > > is that different data providers have their own Java APIs, and they > > will not use Kafka-client to send data directly to my Kafka servers. > > So I need to first collect data from them and feed it to the Kafka > > servers. I guess I need to finish data acquisition in the Producers. > > My question is that there are a great number of data providers so I > > also need a lot of producers??? Or is there any more efficient ways to > deal with it? > > > > Best, > > Chao > > >
