I previously posted a link to contrib in this thread. No, its not a cascading tap. Its a complete job. One to read kafka events to hdfs, one to generate kafka events from hdfs. ETL can happen in between. On Jan 7, 2013 1:51 PM, "Ken Krugler" <kkrugler_li...@transpac.com> wrote:
> Hi Russell, > > On Jan 7, 2013, at 12:48pm, Russell Jurney wrote: > > > Just to be clear - a Kafka 'Tap' of sorts exists in contrib: it scans > > Hadoop records, which may be ETL'd first, and emits new Kafka events. > > Can you point me at the code? > > And just to confirm, you're talking about a Cascading Tap, right? > > -- Ken > > > On Mon, Jan 7, 2013 at 9:57 AM, Ken Krugler <kkrugler_li...@transpac.com > >wrote: > > > >> Hi Guy, > >> > >> On Jan 6, 2013, at 11:11pm, Guy Doulberg wrote: > >> > >>> Hi, > >>> Thanks David, > >>> > >>> I am looking for a product (open source or not), something like Talend > >> or Pentaho that in which I can design the ETL (from and to kafka), and > run > >> the the ETL in Storm/ IronCount or even maybe I can run it in Hadoop > >> Map/Reduce. > >> > >> Interesting - we build ETLs on top of Hadoop using Cascading (open > source > >> workflow API), which has a lot of what it calls "Taps" for connecting to > >> data sources and sinks. > >> > >> But I haven't heard of a Kafka Tap. Should be possible to implement, > >> though. > >> > >> One issue is that Hadoop is batch oriented, so there's a bit of an > >> impedance mismatch when you've got a streaming data source, but from > >> experience it's possible to get that to work. > >> > >> -- Ken > >> > >>> The product should be complete and supports many connections to many > >> data sources and targets, In that sense if you know of a connection to > >> Talend or Pentaho it will be great. > >>> > >>> Thanks again. > >>> , > >>> > >>> > >>> On 01/07/2013 12:28 AM, David Arthur wrote: > >>>> Storm has support for Kafka, if that's the sort of thing you're > looking > >>>> for. Maybe you could describe your use case a bit more? > >>>> > >>>> On Sunday, January 6, 2013, Guy Doulberg wrote: > >>>> > >>>>> Hi > >>>>> > >>>>> I am looking for an ETL tool that can connect to kafka, as a consumer > >> and > >>>>> as a producer, > >>>>> > >>>>> Have you heard of such a tool? > >>>>> > >>>>> Thanks > >>>>> Guy > >>>>> > >>>>> > >>> > >> > >> -------------------------- > >> Ken Krugler > >> +1 530-210-6378 > >> http://www.scaleunlimited.com > >> custom big data solutions & training > >> Hadoop, Cascading, Cassandra & Solr > >> > >> > >> > >> > >> > >> > > > > > > -- > > Russell Jurney twitter.com/rjurney russell.jur...@gmail.com > datasyndrome.com > > -------------------------------------------- > http://about.me/kkrugler > +1 530-210-6378 > > > > > > > -------------------------- > Ken Krugler > +1 530-210-6378 > http://www.scaleunlimited.com > custom big data solutions & training > Hadoop, Cascading, Cassandra & Solr > > > > > >
