Hi, you can have single topic with multiple partitions. It looks like you have messages with key=region and value=sensor reading, and you want to run some aggregate, windowing operations by time. Kafka Streams is perfect fit for this use case.
On Wed, Mar 23, 2016 at 2:30 PM, Maria Musterfrau <daniela_4...@gmx.at> wrote: > Hi, > > Thank you for your reply. > > I would like to cluster the values (every message contains one value) > after their region and to add the values up in realtime afterwards (the > actual/last minute). > > I would like to use Kafka to subscribe the messages to a stream processing > framework like Storm or Spark Streaming and to use a time window because I > would like to add up the values of the last minute. > > Would it be better to use different topics for different regions or to use > one topic and to do the clustering in Storm or Spark Streaming afterwards? > > Thank you in advance. > > Regards, > Daniela > > > > Gesendet: Mittwoch, 23. März 2016 um 09:42 Uhr > Von: "Manikumar Reddy" <ku...@nmsworks.co.in> > An: "users@kafka.apache.org" <users@kafka.apache.org> > Betreff: Re: Topics in Kafka > Hi, > > 1. Based on your design, it can be one or more topics. You can design one > topic per region or > one topic for all region devices. > > 2. Yes, you need to listen to web socket messages and write to kafka > server using kafka producer. > In your use case, you can also send messages using Kafka Rest Proxy > from confluent. > http://docs.confluent.io/2.0.1/kafka-rest/docs/index.html > > > On Wed, Mar 23, 2016 at 1:59 PM, Maria Musterfrau <daniela_4...@gmx.at> > wrote: > > > Hi > > > > I am new and I have a question regarding Kafka. I have devices in > > different regions. Every device opens a websocket connection when it gets > > active and it sends its messages over the opened websocket connection to > a > > server. My question is: is every region a topic or is every websocket > > connection (means every device) a topic? Or can I choose which one is > used > > as topic? > > > > Is it true that I need a piece of code as Kafka cannot consume messages > > directly from the websocket connection? > > > > Thank you and regards, > > Daniela > > >
