Question re patched yammer metrics used by Kafka 0.8 and future plans

Hi, W < users@kafka.apache.org>e are looking at using Kafka, and currently experimenting with 0.8 We use Yammer Metrics 2.2.0 in our code base. The Kafka producer code currently has dependencies on a patched version of metrics which appears to be a version somewhere between 2.2.0 and 3.0, which i

Re: Question re patched yammer metrics used by Kafka 0.8 and future plans

So just to be clear, I'm referring to these jars in core/lib: metrics-annotation-3.0.0-c0c8be71.jar metrics-core-3.0.0-c0c8be71.jar On Thu, Jun 27, 2013 at 11:58 AM, David Black wrote: > Hi, > > W <+users@kafka.apache.org>e are looking at using Kafka, and currently > experimenting with 0.8 > >

Re: Question re patched yammer metrics used by Kafka 0.8 and future plans

Actually, the 0.8.0 beta1 release (from the latest 0.8 branch) depends on metrics 2.2.0. A couple of months ago, the 0.8 branch was using an unreleased version of metrics-core. When we heard that people have class conflicts with their existing metrics dependency, we downgraded on metrics-core in Ka

Re: How to get partition information from Kafka Stream?

Currently, partition # is not exposed in MessageAndMetaData. We do plan to expose such information in a future release. How do you plan to use this info? Thanks, Jun On Thu, Jun 27, 2013 at 7:13 AM, Hussain Pirosha < hussain.piro...@impetus.co.in> wrote: > After retrieving a kafka stream or ka

How to get partition information from Kafka Stream?

After retrieving a kafka stream or kafka message how to get the corresponding partition number to which it belongs ? I am using kafka version 0.7.2. More specifically kafka.consumer.KafkaStream and kafka.message.MessageAndMetaData classes, does not provide API to retrieve partition number. Are t

RE: How to get partition information from Kafka Stream?

A source produces data in kafka at different stages of execution. Topic is partitioned by number of execution stages. Partition number shall give us that information. Eg. 0 specifies data received by source, #last partition specifies processed data delivered by source etc. Thanks, Hussain

Re: How to get partition information from Kafka Stream?

Could you just use a topic for each stage? The nice thing about topics is you don't even have to tell the Kafka cluster about the topics beforehand, so if you a new stage appears, just start writing to that topic. Kafka is great. :-) Phiilp On Thu, Jun 27, 2013 at 7:27 AM, Hussain Pirosha wrote

Kafka 0.8 on Windows issues

While I was able to spin it up on OSX and Ubuntu without any issues, when running this on Windows, I get the following error message when I execute the consumer: Š WARN [console-consumer-8585_tardis8-1372348635138-29591e5d-leader-finder-thread], Failed to add fetcher for [test,0] to broker id:0,ho

Re: Kafka 0.8 on Windows issues

Quick follow up ­ this looks a lot like KAFKA-876 so I'm trying that out now. From: Denny Lee Date: Thursday, June 27, 2013 9:10 AM To: "users@kafka.apache.org" Subject: Kafka 0.8 on Windows issues While I was able to spin it up on OSX and

Re: Kafka 0.8 on Windows issues

Oh, quick follow up ­ I created a kafka-topics.bat and ran it and noticed something hinke related to the topics description [2013-06-27 10:04:17,066] INFO Starting ZkClient event thread. (org.I0Itec.zkcli ent.ZkEventThread) [2013-06-27 10:04:17,068] INFO Opening socket connection to server /10.0.1

Kafka & Vendor Support Options

Hello all, Are there any vendors who provide commercial support for Kafka?

Re: Truncated messages in fetch response

Jay. I assume this is problem exists in the consumer. How this can this problem be triggered so I could test my high level consumer. Thanks On Jun 26, 2013, at 9:21 AM, Jay Kreps wrote: > Yeah, that is true. I thought I documented that, but looking at the > protocol docs, it looks like I didn'

Re: Kafka 0.8 on Windows issues

Interesting, I logged into your vm and i can repro with the console bat. I never tried the console bat, and just wrote my own producer / consumer using the java API. I'll try copying it over to the VM and see if it differs. Tim Sent from my iPhone On Jun 27, 2013, at 10:12 AM, Denny Lee wrote

Re: Kafka 0.8 on Windows issues

Oh that's interesting - much appreciated! > On Jun 27, 2013, at 1:35 PM, Timothy Chen wrote: > > Interesting, I logged into your vm and i can repro with the console bat. > > I never tried the console bat, and just wrote my own producer / consumer > using the java API. I'll try copying it over

Re: Truncated messages in fetch response

Vadim, I don't know under exactly what conditions it happens but the behavior in this test seems to reliably reproduce it: https://github.com/v-a/poseidon/commit/d0ac928e0967e1eaf5b92b403103c4f0dc8fd7f7 -Bob On 27 June 2013 14:29, Vadim Keylis wrote: > Jay. I assume this is problem exists in

Re: Truncated messages in fetch response

Jay is correct. ..it manifests itself when the size threshold condition is violated. For erlang client, this is fixed in 0.4.6 of mps. Github.com/milindparikh/mps Regards Milind On Jun 27, 2013 3:30 PM, "Bob Potter" wrote: > Vadim, > > I don't know under exactly what conditions it happens but

Re: Kafka & Vendor Support Options

I think InfoChimps is doing some support/consulting around Kafka. -Jay On Thu, Jun 27, 2013 at 10:30 AM, d0t wrote: > Hello all, > > Are there any vendors who provide commercial support for Kafka?

Re: Truncated messages in fetch response

This should happen when there is a backlog of data larger than the fetch size the consumer is using. Also, just to be clear this problem is something client implementation needs to handle but not something the user of the client needs to handle. -Jay On Thu, Jun 27, 2013 at 12:29 PM, Vadim Keyli

Re: Kafka 0.8 Failover Behavior

Jun, Does kafka provides ability to configure broker to be in in-sync before become availalble? Is it possible in case of all brokers crash to find out which node has the most recent data to initiate proper startup procedure? Thanks, Vadim On Fri, Jun 21, 2013 at 8:24 PM, Jun Rao wrote: > Hi,

site updates

Hey Folks, I did a pass on the website. Changes: 1. Rewrote the 0.8 quickstart and included a section on running in distributed mode. 2. Fixed up the styles a bit. 3. Fixed the bizarre menu thing with 0.7 and 0.8 specific docs. 4. Re-wrote the copy on the front page. I would love to get any feedb

Re: site updates

Looks much better. 1. We need to update FAQ for 0.8 2. We should probably have a separate section for implementation. 3. The migration tool explanation seems to be hard to get to. On 6/27/13 5:40 PM, "Jay Kreps" wrote: >Hey Folks, > >I did a pass on the website. Changes: >1. Rewrote the 0.8 q

RE: How to get partition information from Kafka Stream?

Great Suggestion ! Hope creating too many topics does not have any performance impact on the broker. Will it ? Thanks, Hussain -Original Message- From: Philip O'Toole [mailto:phi...@loggly.com] Sent: Thursday, June 27, 2013 9:03 PM To: users@kafka.apache.org Subject: Re: How to get part

[ANNOUNCEMENT] Apache Kafka 0.8.0-beta1 Released

The Apache Kafka team is pleased to announce the release of Kafka 0.8.0-beta1 Apache Kafka is a distributed publish-subscribe messaging system . You can download the release from: http://kafka.apache.org/downloads.html The full change log is available here: https://dist.apache.org/repos/dist/rel

Re: Kafka 0.8 on Windows issues

This issue was fixed in kafka-903. The fix is included in the 0.8.0-beta1 release. Do you see the same issue in that version? Thanks, Jun On Thu, Jun 27, 2013 at 9:45 AM, Denny Lee wrote: > Quick follow up ­ this looks a lot like KAFKA-876 >

Re: Kafka 0.8 Failover Behavior

The broker still in ISR in ZK has all committed data. Thanks, Jun On Thu, Jun 27, 2013 at 5:04 PM, Vadim Keylis wrote: > Jun, > Does kafka provides ability to configure broker to be in in-sync before > become availalble? > Is it possible in case of all brokers crash to find out which node has

Re: Kafka 0.8 on Windows issues

Ah - I've been grabbing the 0.8 release itself. Let me snag the 0.8.0-beta1 release - thanks very much! On 6/27/13 11:14 PM, "Jun Rao" wrote: >This issue was fixed in kafka-903. The fix is included in the 0.8.0-beta1 >release. Do you see the same issue in that version? > >Thanks, > >Jun > > >O