I agree with you. We are looking for a simple solution for data from Kafka
to Hadoop. I have tried using Camus earlier (Non-Avro) and documentation is
lacking to make it work correctly, as we do not need to introduce another
component to the solution. In the meantime, can the Kafka Hadoop
Consu
its nice information and it is used for us.123trainings also provides hadoop
online trainings
http://123trainings.com/it-hadoop-bigdata-online-training.html";>hadoop
online training in india
com" ;
> "ao...@wikimedia.org" ; Felix GV ;
> Cosmin Lehene ; "us...@kafka.apache.org"
>
> Sent: Tuesday, August 13, 2013 1:03 PM
> Subject: Re: Kafka/Hadoop consumers and producers
>
> > What installs all the kafka dependencies under /usr/sh
"dibyendu.bhattacha...@pearson.com" ;
> "camus_...@googlegroups.com" ;
> "ao...@wikimedia.org" ; Felix GV ;
> Cosmin Lehene ; "us...@kafka.apache.org"
>
> Sent: Monday, August 12, 2013 7:00 PM
> Subject: Re: Kafka/Hadoop consumers and pr
_...@googlegroups.com" ;
> "ao...@wikimedia.org" ; Felix GV ;
> Cosmin Lehene ; "us...@kafka.apache.org"
>
> Sent: Monday, August 12, 2013 7:00 PM
> Subject: Re: Kafka/Hadoop consumers and producers
>
> We've done a bit of work over at Wikimedia to
yendu.bhattacha...@pearson.com" ;
"camus_...@googlegroups.com" ;
"ao...@wikimedia.org" ; Felix GV ;
Cosmin Lehene ; "us...@kafka.apache.org"
Sent: Monday, August 12, 2013 7:00 PM
Subject: Re: Kafka/Hadoop consumers and producers
We've done a bit of work over at
;
"camus_...@googlegroups.com" ;
"ao...@wikimedia.org" ; Felix GV ;
Cosmin Lehene
Sent: Monday, August 12, 2013 8:20 PM
Subject: Re: Kafka/Hadoop consumers and producers
Kam,
I am perfectly fine if you pick this up. After thinking about it for a
while, we are going to up
>> To: Ken Goodhope
>> Cc: Andrew Psaltis ;
>> dibyendu.bhattacha...@pearson.com; "camus_...@googlegroups.com"
>> ; "ao...@wikimedia.org"
>> ; Felix GV ; Cosmin Lehene
>> ; "dev@kafka.apache.org" ;
>> "us...@kafka.apache.org&
legroups.com"
> ; "ao...@wikimedia.org" ;
> Felix GV ; Cosmin Lehene ;
> "dev@kafka.apache.org" ; "us...@kafka.apache.org"
>
> Sent: Saturday, August 10, 2013 3:30 PM
> Subject: Re: Kafka/Hadoop consumers and producers
>
> So guys, j
quot;
>; "ao...@wikimedia.org"
>; Felix GV ; Cosmin Lehene
>; "dev@kafka.apache.org" ;
>"us...@kafka.apache.org"
>Sent: Saturday, August 10, 2013 3:30 PM
>Subject: Re: Kafka/Hadoop consumers and producers
>
>
>So guys, just to throw my 2 cents in:
>
>1.
ot;dev@kafka.apache.org" ; "us...@kafka.apache.org"
Sent: Saturday, August 10, 2013 3:30 PM
Subject: Re: Kafka/Hadoop consumers and producers
So guys, just to throw my 2 cents in:
1. We aren't deprecating anything. I just noticed that the Hadoop contrib
package wasn't getti
So guys, just to throw my 2 cents in:
1. We aren't deprecating anything. I just noticed that the Hadoop contrib
package wasn't getting as much attention as it should.
2. Andrew or anyone--if there is anyone using the contrib package who would
be willing to volunteer to kind of adopt it that would
Hi Ken,
I am also working on making the Camus fit for Non Avro message for our
requirement.
I see you mentioned about this patch
(https://github.com/linkedin/camus/commit/87917a2aea46da9d21c8f67129f6463af52f7aa8)
which supports custom data writer for Camus. But this patch is not pulled into
I just checked and that patch is in .8 branch. Thanks for working on back
porting it Andrew. We'd be happy to commit that work to master.
As for the kafka contrib project vs Camus, they are similar but not quite
identical. Camus is intended to be a high throughput ETL for bulk
ingestion of Kaf
Dibyendu,
According to the pull request: https://github.com/linkedin/camus/pull/15 it
was merged into the camus-kafka-0.8 branch. I have not checked if the code
was subsequently removed, however, two at least one the important files
from this patch
(camus-api/src/main/java/com/linkedin/camus/etl/Re
Felix,
The Camus route is the direction I have headed for allot of the reasons
that you described. The only wrinkle is we are still on Kafka 0.7.3 so I am
in the process of back porting this patch:
https://github.com/linkedin/camus/commit/87917a2aea46da9d21c8f67129f6463af52f7aa8
that
is descri
The contrib code is simple and probably wouldn't require too much work to
fix, but it's a lot less robust than Camus, so you would ideally need to do
some work to make it solid against all edge cases, failure scenarios and
performance bottlenecks...
I would definitely recommend investing in Camus
We also have a need today to ETL from Kafka into Hadoop and we do not currently
nor have any plans to use Avro.
So is the official direction based on this discussion to ditch the Kafka
contrib code and direct people to use Camus without Avro as Ken described or
are both solutions going to surv
Hi Andrew,
Camus can be made to work without avro. You will need to implement a message
decoder and and a data writer. We need to add a better tutorial on how to do
this, but it isn't that difficult. If you decide to go down this path, you can
always ask questions on this list. I try to make
Hi all,
Over at the Wikimedia Foundation, we're trying to figure out the best way to do
our ETL from Kafka into Hadoop. We don't currently use Avro and I'm not sure
if we are going to. I came across this post.
If the plan is to remove the hadoop-consumer from Kafka contrib, do you think
we s
IMHO, I think Camus should probably be decoupled from Avro before the
simpler contribs are deleted.
We don't actually use the contribs, so I'm not saying this for our sake,
but it seems like the right thing to do to provide simple examples for this
type of stuff, no...?
--
Felix
On Wed, Jul 3,
We can easily make a Camus configuration that would mimic the functionality
of the hadoop consumer in contrib. It may require the addition of a
BinaryWritable decoder, and a couple minor code changes. As for the
producer, we don't have anything in Camus that does what it does. But
maybe we shoul
I guess I am more concerned about the long term than the short term. I
think if you guys want to have all the Hadoop+Kafka stuff then we should
move the producer there and it sounds like it would be possible to get
similar functionality from the existing consumer code. I am not in a rush I
just wan
If the Hadoop consumer/producers use-case will remain relevant for Kafka
(I assume it will), it would make sense to have the core components (kafka
input/output format at least) as part of Kafka so that it could be built,
tested and versioned together to maintain compatibility.
This would also make
I think it makes sense to kill the hadoop consumer/producer code in Kafka,
given, as you said, Camus and the simplicity of the Hadoop producer.
/Sam
On Jul 2, 2013, at 5:01 PM, Jay Kreps wrote:
> We currently have a contrib package for consuming and producing messages
> from mapreduce (
> http
We currently have a contrib package for consuming and producing messages
from mapreduce (
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tree;f=contrib;h=e53e1fb34893e733b10ff27e79e6a1dcbb8d7ab0;hb=HEAD
).
We keep running into problems (e.g. KAFKA-946) that are basically due to
the fact tha
26 matches
Mail list logo
