RE: Using OffsetRequest to get the Head and Tail of a partition in a single request.

I feared that was the case. Well they were certainly right when they said "this API is slightly funky" Thanks for the quick confirmation. Thunder -Original Message- From: Todd Palino [bonk...@gmail.com] Received: Tuesday, 18 Nov 2014, 9:43PM To: users@kafka.apache.org [users@kafka.apach

Re: ISR shrink to 0?

Ok, Makes sense. But if the node is not actually healthy (and underwent a hard crash) it would likely not be able to avoid an 'unclean' restart.what happens if unclean leader election is disabled, but there are no 'clean' partitions available? Jason On Wed, Nov 19, 2014 at 12:40 AM, Jun Rao

Re: How to recover from ConsumerRebalanceFailedException ?

Hi Jun, ZK cluster are up and running. What is best way to programmatically recover and I would try to exponential recovery process which I am willing to implement.So do you think monitoring "ZkClient-EventThread *" thread

Re: Using OffsetRequest to get the Head and Tail of a partition in a single request.

It is not possible, due to how the results for the offset request are stored within the broker and API (as a map). You will have to do 2 requests to get both offsets. -Todd > On Nov 18, 2014, at 8:52 PM, Thunder Stumpges wrote: > > Hey all, > > We are working on a .net client, and I have a

Re: ISR shrink to 0?

Yes, we will preserve the last replica in ISR. This way, we know which replica has all committed messages and can wait for it to come back as the leader, if unclean leader election is disabled. Thanks, Jun On Mon, Nov 17, 2014 at 11:06 AM, Jason Rosenberg wrote: > We have had 2 nodes in a 4 no

Re: selecting java producer (0.8.2 or 0.8.1.1?)

In the 0.8.2 final release, the java doc will be updated properly. See KAFKA-1769 Thanks, Jun On Tue, Nov 18, 2014 at 8:46 PM, Jason Rosenberg wrote: > So that java-doc link includes a new KafkaConsumer (but it seems in other > threads that's not being referred to as ready for use until 0.9, i

Re: How to recover from ConsumerRebalanceFailedException ?

Is your ZK service alive at that point? If not, you just need to monitor the ZK server properly. Thanks, Jun On Mon, Nov 17, 2014 at 2:30 PM, Bhavesh Mistry wrote: > Hi Kafka Team, > > > I get following exception due to ZK/Network issues intermittently. How do > I recover from consumer thread

Re: Topic Creation

Yes, that's the best option for now. The creation is async. Thanks, Jun On Tue, Nov 18, 2014 at 8:55 AM, Bryan Baugher wrote: > Hi everyone, > > I'm looking for the API to create/manager topics. Is this class[1] the one > I should be using? Also what if I'm wanting to create the topic > synchr

Re: Kafka consumer - want to print partition id along with message

You need to define msg as MessageAndMetadata. Thanks, Jun On Mon, Nov 17, 2014 at 10:43 PM, kade...@yahoo.com.INVALID < kade...@yahoo.com.invalid> wrote: > Hi, > > I am trying to learn more on Kafka consuming patterns if we have > partitions on the topic and testing few scenarios. I would like

Re: Enforcing Network Bandwidth Quote with New Java Producer

Either is fine. "outgoing-byte-rate" gives you the total and "byte-rate" gives you per topic. Thanks, Jun On Mon, Nov 17, 2014 at 3:58 PM, Bhavesh Mistry wrote: > Hi Jun, > > So If I set the "*metrics.sample.window.ms > *" to 1 minute, I will be able to get > c

Re: kafka application logdir

Try the following. KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:/mylog4jdir" /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties Thanks, Jun On Mon, Nov 17, 2014 at 11:11 AM, Jimmy John wrote: > How do I configure the application kafka log dir? > > Right now the default is

Using OffsetRequest to get the Head and Tail of a partition in a single request.

Hey all, We are working on a .net client, and I have a question about the OffsetRequest api (https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI) It seems to indicate that you would always get the "log end offset" (tail offset) reg

Re: selecting java producer (0.8.2 or 0.8.1.1?)

So that java-doc link includes a new KafkaConsumer (but it seems in other threads that's not being referred to as ready for use until 0.9, is that right?).Is there a way to know which parts of that javadoc are considered beta-ready in 0.8.2 and which are not? Jason On Tue, Nov 18, 2014 at 11:

Re: benchmark kafka on 10GbE network

Yeah this will involve some experimentation. The metrics are visible with jconsole or another jmx viewer. It may also be worth looking at the cpu usage per-thread (e.g. start top and press 't' I think). Another simple test for broker vs client as the bottleneck is just to start another producer

Re: selecting java producer (0.8.2 or 0.8.1.1?)

0.8.2-beta java doc https://archive.apache.org/dist/kafka/0.8.2-beta/java-doc/ /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop / On Nov

Re: selecting java producer (0.8.2 or 0.8.1.1?)

Hi Jun, Is this the official java doc for the new producer (www.trieuvan.com)? I'm not seeing any links to it (or any documentation) on the apache kafka site (am I overlooking it)? Should there be a link to it in the 0.8.2-beta documentation page? Jason On Tue, Nov 18, 2014 at 7:23 PM, Jun Rao

Re: ISR shrink to 0?

Not sure what happened, but the issue went away once revived the broker id on a new host But it does seem host D's ISR leadership could not be cleared until another member of the ISR came back.somehow D was stale and remained stuck (and clients therefore kept trying to connect to it)... J

Re: How to recover from ConsumerRebalanceFailedException ?

Hi Kakfa team, So just monitor "ZkClient-EventThread *" threads via ThreadInfo[] threads = ManagementFactory.getThreadMXBean().; and if this ZkClient-EventThread

Re: benchmark kafka on 10GbE network

Thanks Jay for the quick response. Yes, it's a single producer and consumer both configured with multiple threads but I'm not using the new producer. CPU is typically 50% utilized on client and merely used on broker. Disks aren't busy either as a lot of data are cached in memory. Would you please

Re: selecting java producer (0.8.2 or 0.8.1.1?)

The new producer in 0.8.2 is considered stable, although it's relatively new. Compared with the old producer, it has the following features. 1. Use non-blocking socket to send requests to the broker. So uses fewer threads and have better throughput. 2. Bound the memory consumption. 3. Support a ca

Re: Getting Simple consumer details using MBean

If you connect to the jvm of the consumer, you will see the following mbean. "kafka.consumer":type="FetchRequestAndResponseMetrics",name="SimpleConsumerShell-AllBrokersFetchRequestRateAndTimeMs" If you want to monitor the consumption rate on the broker, you can look at the byte out rate and the f

Re: Consumer and offset management support in 0.8.2 and 0.9

Yes it should be backwards compatible. So for e.g., you should be able to use an 0.8.1 client with an 0.8.2 broker. In general, you should not upgrade your clients until after the brokers have been upgraded. However, you can point an 0.8.2 client at an 0.8.1 broker. One wire protocol change I'm awa

Re: Consumer and offset management support in 0.8.2 and 0.9

Hi Joel, Thanks for all the clarifications! Just another question on this: will 0.8.2 be backwards compatible with 0.8.1, just as 0.8.1 was with 0.8? Generally speaking, would there be any concerns with using the 0.8.2 consumer with a 0.8.1 broker, for instance? Marius On Tue, Nov 18, 2014 at 2

Re: Consumer and offset management support in 0.8.2 and 0.9

Inline.. On Tue, Nov 18, 2014 at 04:26:04AM -0500, Marius Bogoevici wrote: > Hello everyone, > > I have a few questions about the current status and future of the Kafka > consumers. > > We have been working to adding Kafka support in Spring XD [1], currently > using the high level consumer via S

Re: partition auto-rebalance

The imbalance is measured wrt preferred leaders. i.e., for every partition, the first replica in the assigned replica list (as shown in the output of kafka-topics.sh) is called the preferred replica. On each broker, the auto-balancer counts the number of partitions led by that broker for which the

Re: partition auto-rebalance

Hello Wes, The document here is a bit misleading indeed: http://kafka.apache.org/documentation.html#brokerconfigs In Kafka a partition has a replica list {A,B,C..} and the first replica would be the leader of the partition. When it is not the case, for example since A is down B becomes the leade

Re: benchmark kafka on 10GbE network

Hey Manu, I'm not aware of a benchmark on 10GbE. I'd love to see that though. Diving into the results may help us find bottlenecks hidden by the slower network. Can you figure out where the bottleneck is in your test? I assume this is a single producer and consumer instance and you are using the

Topic Creation

Hi everyone, I'm looking for the API to create/manager topics. Is this class[1] the one I should be using? Also what if I'm wanting to create the topic synchronously? [1] - https://github.com/apache/kafka/blob/0.8.1.1/core/src/main/scala/kafka/admin/AdminUtils.scala#L149-L157 Bryan

partition auto-rebalance

I'm trying to understand the config options for auto-rebalancing. This is what we have in /etc/kafka/server.properties for all the nodes: auto.leader.rebalance.enable=true leader.imbalance.per.broker.percentage=10 leader.imbalance.check.interval.seconds=300 We have 10 nodes for this topic which

Kafka consumer - want to print partition id along with message

Hi,  I am trying to learn more on Kafka consuming patterns if we have partitions on the topic and testing few scenarios. I would like to print the message when the consumer consumes along with the partition id. How to get this done ?  We have simple consumer code (high level)    public void ru

benchmark kafka on 10GbE network

Hi all, I have been trying out kafka benchmarks described in Jay's benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines . I'm able to get similar results on a 4-node GbE

Consumer and offset management support in 0.8.2 and 0.9

Hello everyone, I have a few questions about the current status and future of the Kafka consumers. We have been working to adding Kafka support in Spring XD [1], currently using the high level consumer via Spring Integration Kafka [2]. We are working on adding features such as: - the ability to c

Re: Kafka broker cannot start after running out of disk space

Thanks Guozhang for the pointer to the mapped NIO. The issue in my case was related to the disk still being out of space (I thought I did free up some, but I actually didn't). Curiously, I ran out of space on two occasions. In one case the error message was clear "No space left on device", and in a