Re: How to retrieve the HighWaterMark

Hi Florian I think you should be able to get it by calling consumer.seekToEnd() followed by consumer.position() for each topic partition. B > On 10 Feb 2016, at 09:23, Florian Hussonnois wrote: > > Hi all, > > I'm looking for a way to retrieve the HighWaterMark using the new API. > > Is th

Re: How to retrieve the HighWaterMark

As an aside - you should also be able to validate this against the replication-offset-checkpoint file for each topic partition, server side. > On 11 Feb 2016, at 09:02, Ben Stopford wrote: > > Hi Florian > > I think you should be able to get it by calling consumer.seekToEnd() followed > by co

Simple consumer offset help.

Hello All, I am new to Kafka, and confused in knowing things realated to offsets value that we set over partitions in specific topic using SimpleConsumer class. Would be great if anyone can tell me with example how to set an offset for specific partions , for reading from beginig or from last offs

Kangaroo projects with KafkaInputformat - Mapreduce.

Hi Team, Does any one use Kangaroo Project which uses SimpleConsumer of Kafka API, that provides support to integrate with MapReduce Jobs. Facing one issue related to getting an offsets for specific consumer. Also after Re launching, the committed offsets for a consumer is to be read, and start f

Consumer backwards compatibility

I have a 0.9.0 version of the Kafka consumer. Would that work against the 0.8.2 broker?

Callback Record Key

Hi, Is there a way to get the record key on the callback of the send() for a record? I would like to be able to identify for which of the Records that I have sent is the callback so I can ACK on the db that the record landed successfully in kafka. I am using 0.9.0. Thanks.

Re: Callback Record Key

Hi, Pass the key into the callback you provide to kafka. You then have it available when the callback is invoked. Cheers, Damian On 11 February 2016 at 10:59, Franco Giacosa wrote: > Hi, > > Is there a way to get the record key on the callback of the send() for a > record? I would like to be ab

Re: Callback Record Key

Thanks Damian. 2016-02-11 12:01 GMT+01:00 Damian Guy : > Hi, > Pass the key into the callback you provide to kafka. You then have it > available when the callback is invoked. > > Cheers, > Damian > > On 11 February 2016 at 10:59, Franco Giacosa wrote: > > > Hi, > > > > Is there a way to get the

Re: Where can I find the document for consumer metrics

On Thursday, December 17, 2015 at 18:08, Guozhang Wang wrote: > We should add a section for that. Siyuan can you file a JIRA? Did this ever happen? This documentation would be very helpful.

My producer works topic is Par: 0 ,Lead:1, Rep:1, Isr:1 BUT NOT Par: 0 ,Lead:2, Rep:2, Isr:2

My producer works topic is Par: 0 ,Lead:1, Rep:1, Isr:1 BUT NOT Par: 0 ,Lead:2, Rep:2, Isr:2 Hi, I have a kakfa cluster with 3 kafka nodes and 3 zk nodes. The producer is on AWS machine trying to push data on my kafka cluster running on my intranet servers. When the topic (JOB_AWS_14) is create

Connecting Kafka producer from remote machine and posting message on topics

Hi , We are using kafka for data streaming in our project. Where we have a remote (client) machine , from which client runs consumer and connect to our public ip ( hostname like marina.twin.com ) of kakfa -cluster is set up, and connect to respective topics and posts the data. In our server set up

LVM overhead

Hi everyone, I have multiple disks on my broker. Do you know if there's a noticeable overhead using LVM versus multiple log.dirs ? Thanks

Re: What is the best way to write Kafka data into HDFS?

Hello Steve, Thanks for the suggestion. Looks like this Git repo is not updated for more than 10 months. Is this project still supported? Where can I find current usage and performance metrics ? Thanks, R P From: steve.mo...@gmail.com on behalf of Stev

RE: What is the best way to write Kafka data into HDFS?

May be you can try Apache NiFi its quicker as well. Give a try good luck -Original Message- From: R P [mailto:hadoo...@outlook.com] Sent: 11 February 2016 16:09 To: users@kafka.apache.org Subject: Re: What is the best way to write Kafka data into HDFS? Hello Steve, Thanks for the sug

messages not consumed before topic partition assignment

I started a [java] consumer which subscribed to a topic "foo" (kafka is configured to create 2 partitions for topics by default), which previously did not exist. When subscribing I provided a ConsumerRebalanceListener which logs the information passed to the onPartitionsAssigned and onPartitionsRev

Re: LVM overhead

Hi, I suggest you run a micro benchmark and test it for your usecase. Should be pretty straight forward. Cheers, Jens – Skickat från Mailbox On Thu, Feb 11, 2016 at 4:24 PM, yazgoo wrote: > Hi everyone, > I have multiple disks on my broker. > Do you know if there's a noticeable over

ApacheCon NA 2016 - Important Dates!!!

Hello everyone! I hope this email finds you well.  I hope everyone is as excited about ApacheCon as I am! I'd like to remind you all of a couple of important dates, as well as ask for your assistance in spreading the word! Please use your social media platform(s) to get the word out! The more v

Re: messages not consumed before topic partition assignment

This seems to be related to "metadata.max.age.ms" [ http://kafka.apache.org/documentation.html#newconsumerconfigs], described as "The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new b

Re: What is the best way to write Kafka data into HDFS?

Check out Kafka Connect: http://www.confluent.io/blog/how-to-build-a-scalable-etl-pipeline-with-kafka-connect -Jay On Wed, Feb 10, 2016 at 5:09 PM, R P wrote: > Hello All, > New Kafka user here. What is the best way to write Kafka data into HDFS? > I have looked into following options and f

Re: LVM overhead

There will be some slight lvm overhead, depending on your configuration, but in my experience it will be negligible. I would suggest avoiding creating *any* snapshots when using LVM though as that will decrease performance pretty quickly. For kafka it is unlikely you will want to create any snaps

Regarding Issue with Kafka on Amazon EMR cluster

Hello, I have installed Apache Kafka on Amazon EMR cluster by bootstrap actions. I have the EMR cluster with one master node which has Zookeeper installed on it. Another two Core nodes. I am accessing the cluster using the SSH by Putty. Some how I having issues of Zookeeper and brokers going on d

Re: What is the best way to write Kafka data into HDFS?

Hey Jay, It's awesome to get reply from one of the key Kafka contributor :) . Thanks for suggesting Kafka Connect. How does Kafka-Connect deals with HDFS small files? ( I assume setting large flus.size allows user to maintain minimum HDFS file size. ) Does Kafka-Connect keep file handle ope

You're Invited: Apache Kafka Meetup on Tuesday, February 23 at LinkedIn's campus in Mountain View, CA

Hi Apache Kafka Open Source distros, LinkedIn invites you to attend an Apache Kafka meetup on Tuesday, February 23 at 6PM at LinkedIn's campus in Mountain View, CA. Here's are the details: - We have 3 great talks scheduled: - *Loose Coupling with the Jitney Proj

Re: Consumer backwards compatibility

The new Java consumer in 0.9.0 will not work with 0.8.2 since it depends on the group management protocol built into Kafka, but the older consumer should still work. -Jason On Thu, Feb 11, 2016 at 2:44 AM, Joe San wrote: > I have a 0.9.0 version of the Kafka consumer. Would that work against th

Re: Where can I find the document for consumer metrics

We have them in the Confluent docs: http://docs.confluent.io/2.0.0/kafka/monitoring.html#new-consumer-metrics. -Jason On Thu, Feb 11, 2016 at 4:40 AM, Avi Flax wrote: > On Thursday, December 17, 2015 at 18:08, Guozhang Wang wrote: > > We should add a section for that. Siyuan can you file a JIRA

0.9.0.1 RC1

This is the first candidate for release of Apache Kafka 0.9.0.1. This a bug fix release that fixes 70 issues. Release Notes for the 0.9.0.1 release https://home.apache.org/~junrao/kafka-0.9.0.1-candidate1/RELEASE_NOTES.html *** Please download, test and vote by Tuesday, Feb. 16, 7pm PT Kafka's K

Kafka 0.8.2.0 Log4j

How could I get rid of this warning? log4j:WARN No appenders could be found for logger (kafka.utils.VerifiableProperties). log4j:WARN Please initialize the log4j system properly. Any ideas how to get rid of this warning?