Re: Kafka dotnet SDK suggestion

Another .Net option is to use the Confluent Rest Proxy and this: https://github.com/josephjeganathan/Confluent.RestClient -Mike On Wed, Oct 7, 2015 at 9:51 AM, sugumar analysis wrote: > Hi All, > > We are developing Messaging system with Kafka using DotNet (C#). > > We found there are 2 differe

Re: Call to Future.get() hangs for producer

HI Guozhang, Thanks for replying. I have minimal configuration at this point. I am only trying to get the new producer working with a single kafka node with the replication factor on my topic set to 1. I am only setting the bootstrap.servers variable so the acks looks like it would be 1. I have

Re: Urban Airship Connect

The first level Kafkas are actually two separate clusters, and do about avg 50k messages/s, max 120k messages/s on 4 node clusters. The second level cluster doesn't handle all that load, as we filter down to the subset of our customers using the service. We currently use a retention window of 7 da

Re: Urban Airship Connect

Hi, Nice to know another usage of Kafka Could you tell how much load does the first and second level Kafka clusters intake? What is your write throughput, and end-to-end latency, for how long do you hold data in Kafka and how many minimum in-sync replicas you've chosen? Thanks, Prabhjot On Oct

Urban Airship Connect

Hi folks, I thought this list might find a case study interesting. Urban Airship just released a new product, Connect, that uses Kafka heavily. We released some discussion of the why and how we built it at https://www.urbanairship.com/blog/why-we-built-urban-airship-connect and https://www.urbanai

Re: Is kafka 8 and LeaderNotAvailableError

Zoo.cfg looks OK to me Please post your reply whether it worked after changing those values If not, we'll figure out something Also, I forgot to tell earlier, please check the size of that partition's directory on both the replicas. If the size differs by a substantial amount, it clears one thin

Re: Is kafka 8 and LeaderNotAvailableError

Thanks for the help :) tickTime=3000 initLimit=10 syncLimit=5 dataDir=/var/lib/zookeeper clientPort=2181 server.1=45.55.xxx.xxx:2888:3888 server.3=104.236.xxx.xxx:2888:3888 server.2=192.241.xxx.xxx:2888:3888 On Sat, Oct 10, 2015 at 3:02 AM, Prabhjot Bharaj wrote: > Hi, > > The error is due to

Re: Is kafka 8 and LeaderNotAvailableError

Hi, The error is due to some kind of timeout happening, during fetching of metadata. Random successes and failures can also be explained, if we come to know which timeout parameter is the culprit Default value for replica.fetch.wait.max.ms is 500. Try changing it to some higher value Although I'v

Re: Is kafka 8 and LeaderNotAvailableError

Hi I updated the metadata.broker.list=104.236.xxx.xxx:9092,104.236.xxx.xxx:9092 in producer.properties. When I run the code I get random fails and success success: [ProduceResponse(topic='topic-test-production', partition=0, error=0, offset=1)] fail: kafka.common.LeaderNotAvailableError: TopicMet

Fetcher threads blocked, no messages read from stream

Hi everyone, I'm currently experiencing a situation that looks very similar to this[1]. I'm running Kafka 0.8.2.1 and we use it with Storm but don't use Storm's kafka spout since it doesn't quite work for our needs. I've been able to reproduce the issue locally by creating a storm topology with 1

Re: Is kafka 8 and LeaderNotAvailableError

PS I see this in one of the kafka log files Thanks [2015-10-09 13:41:52,873] WARN [ReplicaFetcherThread-0-21441], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 58; ClientId: ReplicaFetcherThread-0-21441; ReplicaId: 26680; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: [topic-de

Re: Is kafka 8 and LeaderNotAvailableError

Is this a total failure on kafka or python kakfa-python? I am using ubuntu 14.04 on digital ocean and zk 3.4.6 Using UFW all zk and kafka servers have access I added my ip address from home to kafka and zk servers using UFW so there should be no filewall issues Below is my kafka server.propertie

Re: Is kafka 8 and LeaderNotAvailableError

Hi, Your metadata response is empty. Here is an output for my kafka cluster: *>>> kafka = KafkaClient(['96.a.a.a:9092','96.b.b.b:9092','96.c.c.c:9092','96.d.d.d:9092','96.e.e.e:9092'])* *2015-10-09 21:16:53,575.575.392961502:kafka.client:140735282348800:DEBUG:8705:Request 1: []* *2015-10-09 21:

Re: Kafka Mirror to consume data from beginning of topics

Hi Leo Set auto.offset.reset=smallest in your consumer.config B > On 8 Oct 2015, at 18:47, Clelio De Souza wrote: > > Hi there, > > I am trying to setup a Kafka Mirror mechanism, but it seems the consumer > from the source Kafka cluster only reads from new incoming data to the > topics, i.e.

Re: Dual commit with zookeeper and kafka

Hi Alexey Whether you commit offsets to Kafka itself (stored in offsets topic) or ZK or both depends on the settings in two properties: offset.storage and dual.commit.enabled (here ). Currently ZK is the default. Commits happen eith

Re: Is kafka 8 and LeaderNotAvailableError

ps here is the output the I add logging to the code import logging logging.basicConfig( format='%(asctime)s.%(msecs)s:%(name)s:%(thread)d:%(levelname)s:%(process)d:%(message)s', level=logging.DEBUG ) 2015-10-09 20:59:33,656.656.464099884:kafka.client:139860093155072:DEBUG:25285:Request 1: []

Re: Is kafka 8 and LeaderNotAvailableError

from kafka import SimpleProducer from kafka import KafkaClient kafka_host_list = ['192.241.xxx.xxx:9092','45.55.xxx.xxx:9092','107.170.xxx.xxx:9092'] kafka = KafkaClient(kafka_host_list) producer = SimpleProducer(kafka, async=False) test_topic = 'topic-test-production' test_payload = 'test' print p

Re: Is kafka 8 and LeaderNotAvailableError

Hi, Please share the code snippet for Kafka python Regards, Prabhjot On Oct 9, 2015 4:30 PM, "David Montgomery" wrote: > This is the python error > > kafka.common.LeaderNotAvailableError: > TopicMetadata(topic='topic-test-production', error=5, partitions=[]) > > > On Fri, Oct 9, 2015 at 6:55 PM

Re: Is kafka 8 and LeaderNotAvailableError

This is the python error kafka.common.LeaderNotAvailableError: TopicMetadata(topic='topic-test-production', error=5, partitions=[]) On Fri, Oct 9, 2015 at 6:55 PM, David Montgomery wrote: > Well i masked my true domain but the hostnames are valid and do work. I > do see the topics creates in

Re: Is kafka 8 and LeaderNotAvailableError

Well i masked my true domain but the hostnames are valid and do work. I do see the topics creates in ZK brokers/topics/topic-test-production/partisions/0/state {"controller_epoch":10,"leader":26665,"version":1,"leader_epoch":2,"isr":[26665]} brokers/topics/topic-test-production/partisions/1/state

Re: Is kafka 8 and LeaderNotAvailableError

Hi, Is this a valid zookeeper connection string 1.zk.do.production.test:2181,2.zk.do.production.test:2181,3.zk.do.production.test:2181 Is 2.zk.do.production.test or 3.zk.do.production.test a valid hostname?? Regards, Prabhjot On Oct 9, 2015 1:37 PM, "David Montgomery" wrote: > PS I added one

Re: How to verify offsets topic exists?

If I'm not mistaken, replication factor of a topic does not get stored in ZK - on creation replication factor gets translated to topic replica assignment which gets stored in ZK. Again, please correct me if wrong - it seems that only during topic creation it is verified that all replicas in replic

Re: Is kafka 8 and LeaderNotAvailableError

PS I added one more server now 3 kafka servers. still kafka is not working. Is there a known bug? On Fri, Oct 9, 2015 at 3:23 PM, David Montgomery wrote: > i, > > I have 3 ZK servers and 2 kafka servers with 2 partitions. > > I am using kafka_2.11-0.8.2.1 > > Here is how I am creating a topic:

Is kafka 8 and LeaderNotAvailableError

i, I have 3 ZK servers and 2 kafka servers with 2 partitions. I am using kafka_2.11-0.8.2.1 Here is how I am creating a topic: bin/kafka-topics.sh --zookeeper 1.zk.do.production.test:2181,2.zk.do.production.test:2181,3.zk.do.production.test:2181 --create --topic topic-test-production --partitio