Koert I am very new to spark, is it ok to you to share the code base for dumping data into hdfs from kafka using spark ?
On Fri, Mar 20, 2015 at 12:20 AM, Koert Kuipers <ko...@tresata.com> wrote: > we load from kafka into hdfs using spark in batch mode, once a day. it's > very simple (74 lines of code) and works fine. > > On Fri, Mar 13, 2015 at 4:11 PM, Gwen Shapira <gshap...@cloudera.com> > wrote: > > > Camus uses MapReduce though. > > If Alberto uses Spark exclusively, I can see why installing MapReduce > > cluster (with or without YARN) is not a desirable solution. > > > > > > > > > > On Fri, Mar 13, 2015 at 1:06 PM, Thunder Stumpges <tstump...@ntent.com> > > wrote: > > > Sorry to go back in time on this thread, but Camus does NOT use YARN. > We > > have been using camus for a while on our CDH4 (no YARN) Hadoop cluster. > It > > really is fairly easy to set up, and seems to be quite good so far. > > > > > > -Thunder > > > > > > > > > -----Original Message----- > > > From: amiori...@gmail.com [mailto:amiori...@gmail.com] On Behalf Of > > Alberto Miorin > > > Sent: Friday, March 13, 2015 12:15 PM > > > To: users@kafka.apache.org > > > Cc: otis.gospodne...@gmail.com > > > Subject: Re: Alternative to camus > > > > > > We use spark on mesos. I don't want to partition our cluster because of > > one YARN job (camus). > > > > > > Best > > > > > > Alberto > > > > > > On Fri, Mar 13, 2015 at 7:43 PM, Otis Gospodnetic < > > otis.gospodne...@gmail.com> wrote: > > > > > >> Just curious - why - is Camus not suitable/working? > > >> > > >> Thanks, > > >> Otis > > >> -- > > >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management > > >> Solr & Elasticsearch Support * http://sematext.com/ > > >> > > >> > > >> On Fri, Mar 13, 2015 at 2:33 PM, Alberto Miorin > > >> <amiorin78+ka...@gmail.com > > >> > > > >> wrote: > > >> > > >> > I was wondering if anybody has already tried to mirror a kafka topic > > >> > to hdfs just copying the log files from the topic directory of the > > >> > broker (like 00000000000023244237.log). > > >> > > > >> > The file format is very simple : > > >> > https://twitter.com/amiorin/status/576448691139121152/photo/1 > > >> > > > >> > Implementing an InputFormat should not be so difficult. > > >> > > > >> > Any drawbacks? > > >> > > > >> > > > -- SunilKalva
