we load from kafka into hdfs using spark in batch mode, once a day. it's
very simple (74 lines of code) and works fine.

On Fri, Mar 13, 2015 at 4:11 PM, Gwen Shapira <gshap...@cloudera.com> wrote:

> Camus uses MapReduce though.
> If Alberto uses Spark exclusively, I can see why installing MapReduce
> cluster (with or without YARN) is not a desirable solution.
>
>
>
>
> On Fri, Mar 13, 2015 at 1:06 PM, Thunder Stumpges <tstump...@ntent.com>
> wrote:
> > Sorry to go back in time on this thread, but Camus does NOT use YARN. We
> have been using camus for a while on our CDH4 (no YARN) Hadoop cluster. It
> really is fairly easy to set up, and seems to be quite good so far.
> >
> > -Thunder
> >
> >
> > -----Original Message-----
> > From: amiori...@gmail.com [mailto:amiori...@gmail.com] On Behalf Of
> Alberto Miorin
> > Sent: Friday, March 13, 2015 12:15 PM
> > To: users@kafka.apache.org
> > Cc: otis.gospodne...@gmail.com
> > Subject: Re: Alternative to camus
> >
> > We use spark on mesos. I don't want to partition our cluster because of
> one YARN job (camus).
> >
> > Best
> >
> > Alberto
> >
> > On Fri, Mar 13, 2015 at 7:43 PM, Otis Gospodnetic <
> otis.gospodne...@gmail.com> wrote:
> >
> >> Just curious - why - is Camus not suitable/working?
> >>
> >> Thanks,
> >> Otis
> >> --
> >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> >> Solr & Elasticsearch Support * http://sematext.com/
> >>
> >>
> >> On Fri, Mar 13, 2015 at 2:33 PM, Alberto Miorin
> >> <amiorin78+ka...@gmail.com
> >> >
> >> wrote:
> >>
> >> > I was wondering if anybody has already tried to mirror a kafka topic
> >> > to hdfs just copying the log files from the topic directory of the
> >> > broker (like 00000000000023244237.log).
> >> >
> >> > The file format is very simple :
> >> > https://twitter.com/amiorin/status/576448691139121152/photo/1
> >> >
> >> > Implementing an InputFormat should not be so difficult.
> >> >
> >> > Any drawbacks?
> >> >
> >>
>

Reply via email to