Hi Alex, as Jun said it depends on your use case. Folks are doing everything under the sun now with Kafka now-a-days (which is awesome).
You can find Kafka with Accumulo, Akka, Camel, Camus, Cassandra, Druid, LogStash, Metrics, Samza, Spark, Storm and a whole lot more more. There is also *a lot *of custom development too with producers and consumers with all of the previous and other technologies for getting data in and out. If you are not already using Camel it may not make sense to use it, IMHO. You can just use one of the client libraries https://cwiki.apache.org/confluence/display/KAFKA/Clients or an integrated system https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem or the producer that comes with the project https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example and the consumer that comes with the project https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example As far as data going "in" you are going to find the most breadth of difference because a (the) goal is to get all of your data into Kafka and creating a unified log http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying . If you want to look at what happens after the data is in Kafka then the scope shrinks more into systems like Samza, Spark, Storm that process data in real time (or batch) or data stores directly like Druid, Cassandra, HBase, Accumulo, etc or Hadoop, S3, etc. I think if you can open up to what you are trying to-do (even if it is just learn how things works) you will find more specifics and assistance. Also be specific if you can to existing languages and technologies and use (even what data you are dealing with it is all relevant). /******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Thu, Jul 10, 2014 at 12:15 AM, Alex Li <alex.heqin...@gmail.com> wrote: > Thanks, Jun. I did read the uses and the presentations. Particularly for > operational monitoring data and other lower throughput streams, it feels > that camel or mule could help handle ingestion and output flows. > > > I'm wondering whether those etls are really used together with Kafka for a > unified integration in production. Read a few blogs and camel discussion > promoting camel-kafka, but have not heard any real-world use case. > > > > > Thanks. > > > > > HQ > — > Sent from Mailbox > > On Wed, Jul 9, 2014 at 8:49 PM, Jun Rao <jun...@gmail.com> wrote: > > > That depends on the use case. Perhaps you can start by looking at some of > > the presentations on Kafka usage in > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations > > Thanks, > > Jun > > On Wed, Jul 9, 2014 at 4:31 PM, HQ Li <alexli.paloa...@gmail.com> wrote: > >> Dear experts, > >> > >> I'm new to Kafka and am doing some study around overall real-time data > >> integration architecture. What is the common ways of pushing data into > >> Kafka? Does anyone use ESB or others to feed various message streams > into > >> Kafka in real-time / an event-drvien fashion? > >> > >> Thanks. > >> -HQ > >> >
