Consumer not getting data when there is a big lag in the topic

Hi, I am been facing some JAVA high level consumer related issues lately and would like to understand more on this. We have 9 bare-metals (48 core, 250 GB, Terabytes of Hard disks) running *Kafka 0.8.2* and 5 independent VM (8 core, 60 GB) running zookeeper. I have a topic that has key as metadat

Re: Consumer not getting data when there is a big lag in the topic

Dinesh, Your fetch.message.max.bytes is 188743680 < 155MB, but you said some messages can be as large as 180MB. Could you try to set it to be larger than, say 200MB and see if it helps? Guozhang On Tue, Jan 13, 2015 at 4:18 AM, dinesh kumar wrote: > Hi, > I am been facing some JAVA high level

Re: Create topic, then fetch offset

Henri, FetchOffsetRequest is used for retrieving the last committed offset of a certain consumer group; for your use case you would send OffsetRequest. That said the error code is misleading, it may be a bug in not indicating the right error code. Guozhang On Fri, Jan 9, 2015 at 4:10 AM, Henri

network connection between kafka nodes

Hello, Kafka experts I have a production cluster which has three nodes(.100, .101, .102) I am using a C# producer to publish data to kafka brokers, it works for a while but started to lose connection error to 2 nodes of cluster. Here is the C# producer error: [2015-01-13 01:49:49,786] ERROR [Cons

Re: [DISCUSS] 0.8.2-beta2 release

Thanks for everyone's help, we have resolved all known blockers for 0.8.2 (except for fixing the docs, which can be done in parallel with voting). I will be doing some basic testing on the 0.8.2 branch. If that looks good, I will cut an RC for 0.8.2 and start the voting process today. Thanks, Jun

Re: Consumer not getting data when there is a big lag in the topic

Hi Guozhang, Sorry for the misinformation. We have file sizes around 50 - 100 MB. So we set *fetch.message.max.bytes* conservatively around 188743680. Can you please explain me the reason for this behavior? Thanks, Dinesh On 13 January 2015 at 21:42, Guozhang Wang wrote: > Dinesh, > > Your fet

Re: Consumer not getting data when there is a big lag in the topic

We are using 0.8.1.1 version of Kafka and *not 0.8.2 *as mentioned above. Thanks, Dinesh On 13 January 2015 at 23:35, dinesh kumar wrote: > Hi Guozhang, > Sorry for the misinformation. We have file sizes around 50 - 100 MB. So we > set *fetch.message.max.bytes* conservatively around 188743680.

Re: Thread safety of Encoder implementations

Hi Guozhang, Thanks for you response. We've only got one producer client (per Kryo instance) but the producer client is configured (via the broker.list config) to produce to two Kafka brokers. When we create the Producer, we pass in an instance of the serializer. What if we used the serializer.clas

Re: Create topic, then fetch offset

Hi, Thanks for your reply, Guozhang, you are right. I eventually got it working, but it required TWO changes to what I initially tried: 1) Use OffsetRequest, not OffsetFetchRequest, like Guozhang pointed out. 2) Create topics by sending a TopicMetadataRequest and having auto.create.topics.ena

How to handle authentication system wide with kafka as the transport layer?

Hello I've been reading about kafka and its use in a distributed system, authentication handling remains a mystery to me. Indeed, when the user connects to the website, many services are triggered : how do they make sure the user is authenticated and authorized ? I mainly see two options : - eac

Re: Create topic, then fetch offset

It looks like you are not adding a ZkStringSerializer to your zkClient. ZkClient zkClient = new ZkClient(ZK_CONN_STRING); in Kafka TopicCommand uses "new ZkClient(opts.options.valueOf(opts.zkConnectOpt), 3, 3, ZKStringSerializer)" because of this although your topic is getting created and

Re: Create topic, then fetch offset

Also I don’t think you need to try the TopicMetadataRequest when auto.create.topics.enable=true, explicitly. I suppose if you try to produce using a topic name Kafka should automatically create the topic for you. Thanks, Mayuresh On Jan 13, 2015, at 2:28 PM, Henri Pihkala wrote: > Hi, > > T

Re: Create topic, then fetch offset

You can import ZKStringSerializer from kafka.utils.ZkClient or write your own similar string serializer like this https://gist.github.com/harshach/7b5447c39168eb6062e0 On Tue, Jan 13, 2015, at 04:31 PM, Harsha wrote: > It looks like you are not adding a ZkStringSerializer to your zkClient. > > Z

Re: Create topic, then fetch offset

Internally producer sends a TopicMetadataRequest which creates the topic if auto.create.topics.enable is true. On Tue, Jan 13, 2015, at 04:49 PM, Harsha wrote: > > You can import ZKStringSerializer from kafka.utils.ZkClient or write > your own similar string serializer like this > https://gist.gi

Javadoc errors in MetricName when building with Java 8

Is Java 8 supported for building Kafka? Or do you only support Java 7? I just noticed that the latest code on the 0.8.2 branch fails on the javadoc in the new MetricName class due to Java 8's javadoc tool being much stricter when checking comments (especially around things like HTML tags). For exa