One other option is to use something like Druid, especially if you care
about doing arbitrary dimensional drilldowns.
http://druid.io
It reads from Kafka and can do simple rollups for you automatically
(meaning you don't need storm if all you are doing with Storm is a simple
"group by" style roll
Apologies for asking another question as a newbie without having really
tried stuff out, but actually one of our main reasons for wanting to use
kafka (not the linkedin use case) is exactly the fact that the "buffer" is
not just for buffering. We want to keep data for days to weeks, and be able
to
Anthony,
Is there a reason you wouldn't want to just push the data into something
built for cheap, long-term storage (like glacier, S3, or HDFS) and perhaps
"replay" from that instead of from the kafka brokers? I can't speak for
Jay, Jun or Neha, but I believe the expected usage of Kafka is essen
r in the same process.
>
> Jamie
>
> -----Original Message-
> From: Eric Tschetter [mailto:ched...@metamarkets.com]
> Sent: Thursday, February 14, 2013 2:44 PM
> To: users@kafka.apache.org
> Subject: Re: Loading kafka server within a Java process
>
> Jamie,
>
> As
Jamie,
As Jay says, this is definitely possible. We also do it at Metamarkets.
The one thing that we found odd when doing it though is that you cannot (or
couldn't with the version we are using, I'm not sure about 0.8) embed a
producer that produces for an embedded broker. You instead have to h
11:09 PM, Neha Narkhede wrote:
> Is that a trick question, Eric ? The answer is yes :-)
>
> Thanks,
> Neha
>
>
> On Thu, Jan 17, 2013 at 3:08 PM, Eric Tschetter >wrote:
>
> > Neha,
> >
> > Will the beta release include artifacts in maven? :)
>
Neha,
Will the beta release include artifacts in maven? :)
--Eric
On Thu, Jan 17, 2013 at 4:55 PM, Neha Narkhede wrote:
> We think we should be able to share a stable 0.8 beta with the community by
> the end of this month. This is also the time we will deploy 0.8 in
> production at LinkedIn, t
