i can not just share this. take a look at KafkaRDD from our spark-kafka library, or starting with spark 1.3.0 you can use the KafkaRDD that is included with spark.
On Thu, Mar 19, 2015 at 2:58 PM, sunil kalva <sambarc...@gmail.com> wrote: > > Koert > I am very new to spark, is it ok to you to share the code base for dumping > data into hdfs from kafka using spark ? > > > On Fri, Mar 20, 2015 at 12:20 AM, Koert Kuipers <ko...@tresata.com> wrote: > >> we load from kafka into hdfs using spark in batch mode, once a day. it's >> very simple (74 lines of code) and works fine. >> >> On Fri, Mar 13, 2015 at 4:11 PM, Gwen Shapira <gshap...@cloudera.com> >> wrote: >> >> > Camus uses MapReduce though. >> > If Alberto uses Spark exclusively, I can see why installing MapReduce >> > cluster (with or without YARN) is not a desirable solution. >> > >> > >> > >> > >> > On Fri, Mar 13, 2015 at 1:06 PM, Thunder Stumpges <tstump...@ntent.com> >> > wrote: >> > > Sorry to go back in time on this thread, but Camus does NOT use YARN. >> We >> > have been using camus for a while on our CDH4 (no YARN) Hadoop cluster. >> It >> > really is fairly easy to set up, and seems to be quite good so far. >> > > >> > > -Thunder >> > > >> > > >> > > -----Original Message----- >> > > From: amiori...@gmail.com [mailto:amiori...@gmail.com] On Behalf Of >> > Alberto Miorin >> > > Sent: Friday, March 13, 2015 12:15 PM >> > > To: users@kafka.apache.org >> > > Cc: otis.gospodne...@gmail.com >> > > Subject: Re: Alternative to camus >> > > >> > > We use spark on mesos. I don't want to partition our cluster because >> of >> > one YARN job (camus). >> > > >> > > Best >> > > >> > > Alberto >> > > >> > > On Fri, Mar 13, 2015 at 7:43 PM, Otis Gospodnetic < >> > otis.gospodne...@gmail.com> wrote: >> > > >> > >> Just curious - why - is Camus not suitable/working? >> > >> >> > >> Thanks, >> > >> Otis >> > >> -- >> > >> Monitoring * Alerting * Anomaly Detection * Centralized Log >> Management >> > >> Solr & Elasticsearch Support * http://sematext.com/ >> > >> >> > >> >> > >> On Fri, Mar 13, 2015 at 2:33 PM, Alberto Miorin >> > >> <amiorin78+ka...@gmail.com >> > >> > >> > >> wrote: >> > >> >> > >> > I was wondering if anybody has already tried to mirror a kafka >> topic >> > >> > to hdfs just copying the log files from the topic directory of the >> > >> > broker (like 00000000000023244237.log). >> > >> > >> > >> > The file format is very simple : >> > >> > https://twitter.com/amiorin/status/576448691139121152/photo/1 >> > >> > >> > >> > Implementing an InputFormat should not be so difficult. >> > >> > >> > >> > Any drawbacks? >> > >> > >> > >> >> > >> > > > > -- > SunilKalva >
