Hi Russell, On Jan 7, 2013, at 12:48pm, Russell Jurney wrote:
> Just to be clear - a Kafka 'Tap' of sorts exists in contrib: it scans > Hadoop records, which may be ETL'd first, and emits new Kafka events. Can you point me at the code? And just to confirm, you're talking about a Cascading Tap, right? -- Ken > On Mon, Jan 7, 2013 at 9:57 AM, Ken Krugler > <kkrugler_li...@transpac.com>wrote: > >> Hi Guy, >> >> On Jan 6, 2013, at 11:11pm, Guy Doulberg wrote: >> >>> Hi, >>> Thanks David, >>> >>> I am looking for a product (open source or not), something like Talend >> or Pentaho that in which I can design the ETL (from and to kafka), and run >> the the ETL in Storm/ IronCount or even maybe I can run it in Hadoop >> Map/Reduce. >> >> Interesting - we build ETLs on top of Hadoop using Cascading (open source >> workflow API), which has a lot of what it calls "Taps" for connecting to >> data sources and sinks. >> >> But I haven't heard of a Kafka Tap. Should be possible to implement, >> though. >> >> One issue is that Hadoop is batch oriented, so there's a bit of an >> impedance mismatch when you've got a streaming data source, but from >> experience it's possible to get that to work. >> >> -- Ken >> >>> The product should be complete and supports many connections to many >> data sources and targets, In that sense if you know of a connection to >> Talend or Pentaho it will be great. >>> >>> Thanks again. >>> , >>> >>> >>> On 01/07/2013 12:28 AM, David Arthur wrote: >>>> Storm has support for Kafka, if that's the sort of thing you're looking >>>> for. Maybe you could describe your use case a bit more? >>>> >>>> On Sunday, January 6, 2013, Guy Doulberg wrote: >>>> >>>>> Hi >>>>> >>>>> I am looking for an ETL tool that can connect to kafka, as a consumer >> and >>>>> as a producer, >>>>> >>>>> Have you heard of such a tool? >>>>> >>>>> Thanks >>>>> Guy >>>>> >>>>> >>> >> >> -------------------------- >> Ken Krugler >> +1 530-210-6378 >> http://www.scaleunlimited.com >> custom big data solutions & training >> Hadoop, Cascading, Cassandra & Solr >> >> >> >> >> >> > > > -- > Russell Jurney twitter.com/rjurney russell.jur...@gmail.com datasyndrome.com -------------------------------------------- http://about.me/kkrugler +1 530-210-6378 -------------------------- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr
