Thank you all. I'll have a look at flume and also at akka-http and akka-streams since the MACs will send the data to a REST endpoint. El 29/01/2015 16:10, "Jeff Holoman" <jholo...@cloudera.com> escribió:
> Yeah if you're into Flume you can definitely do per event > modification/routing in an interceptor with relative ease. I don't know the > size of the total MAC addresses to look up (or actually why a hash > partitioning scheme wouldn't just work, but w/e I assume you have your > reasons). There's kind of an example of doing this here: > > > http://blog.cloudera.com/blog/2014/11/flafka-apache-flume-meets-apache-kafka-for-event-processing/ > > In the example in the blog it uses HBase to read some "profile" data. You > could sub this with whatever other store you wanted (Redis, Cassandra, > whatever) > > So you'd go: > > Your Systems -> Kafka (raw data) -> Flume Source->Interceptor->Kafka > Channel(Raw data to the correct topic by setting the Flume Event header) > > Of course you could code that all yourself too. Whatever floats your boat. > Writing interceptors is really easy and there's quite a few examples > around. > > Jeff > > On Thu, Jan 29, 2015 at 4:10 AM, David Morales <dmora...@stratio.com> > wrote: > > > Hi Toni, > > > > 1. Kafka can create topics on the fly, in case you need it. > > > > https://kafka.apache.org/08/configuration.html > > > > auto.create.topics.enabletrueEnable auto creation of topic on the server. > > If this is set to true then attempts to produce, consume, or fetch > metadata > > for a non-existent topic will automatically create it with the default > > replication factor and number of partitions. > > > > > > > > 2. About topic selection based on rules/dictionary, this must be solved > on > > your side. > > > > You can use custom-code in your app or an event transport solution, like > > Flume. > > > > Flume 1.6 now includes a sink for Kafka, and it already supports dynamic > > topics (by using a preprocessor) > > > > https://github.com/thilinamb/flume-ng-kafka-sink > > > > > > - > > > > *topic*[optional] > > - The topic in Kafka to which the messages will be published. If this > > topic is mentioned, every message will be published to the same > > topic. If > > dynamic topics are required, it's possible to use a preprocessor > > instead of > > a static topic. It's mandatory that either of the parameters > *topic* > > or *preprocessor* is provided, because the topic cannot be null > when > > publishing to Kafka. If none of these parameters are provided, > > the messages > > will be published to a default topic called default-flume-topic. > > > > > > > > Regards. > > > > > > > > > > 2015-01-29 0:16 GMT+01:00 Lakshmanan Muthuraman <lakshma...@tokbox.com>: > > > > > Hi Toni, > > > > > > Couple of thoughts. > > > > > > 1. Kafka behaviour need not be changed at run time. Your producers > which > > > push your MAC data into kafka should know to which topic it should > write. > > > Your producer can be flume, log stash or it can be your own custom > > written > > > java producer. > > > > > > As long as your producer know which topic to write, they can keep > > creating > > > new topics as new MAC data comes through your pipeline. > > > > > > On Wed, Jan 28, 2015 at 12:10 PM, Toni Cebrián <toni.cebr...@gmail.com > > > > > wrote: > > > > > > > Hi, > > > > > > > > I'm starting to weight different alternatives for data ingestion > > and > > > > I'd like to know whether Kafka meets the problem I have. > > > > Say we have a set of devices each with its own MAC and then we > > > receive > > > > data in Kafka. There is a dictionary defined elsewhere that says each > > MAC > > > > to which topic must publish. So I have basically 2 questions: > > > > New MACs keep comming and the dictionary must be updated accordingly. > > How > > > > could I change this Kafka behaviour during runtime? > > > > A problem for the future. Say that dictionaries are so big that they > > > don't > > > > fit in memory. Are there any patterns for bookkeeping internal data > > > > structures and how route to them? > > > > > > > > T. > > > > > > > > > > > > > > > -- > > > > David Morales de Frías :: +34 607 010 411 :: @dmoralesdf > > <https://twitter.com/dmoralesdf> > > > > > > <http://www.stratio.com/> > > Vía de las dos Castillas, 33, Ática 4, 3ª Planta > > 28224 Pozuelo de Alarcón, Madrid > > Tel: +34 91 828 6473 // www.stratio.com // *@stratiobd > > <https://twitter.com/StratioBD>* > > > > > > -- > Jeff Holoman > Systems Engineer > 678-612-9519 >
