Is this a total failure on kafka or python kakfa-python?
I am using ubuntu 14.04 on digital ocean and zk 3.4.6
Using UFW all zk and kafka servers have access
I added my ip address from home to kafka and zk servers using UFW so there
should be no filewall issues
Below is my kafka server.properties from my chef template
broker.id=<%=@broker_id%>
advertised.host.name=<%=@ipaddress%>
port=9092
host.name=<%=@ipaddress%>
num.network.threads=2
num.io.threads=2
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=2
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.segment.bytes=536870912
log.cleanup.interval.mins=1
zookeeper.connect=<%=@zookeeper%>
zookeeper.connection.timeout.ms=1000000
default.replication.factor=2
delete.topic.enable=true
unclean.leader.election.enable=true
I just rebuilt my zk cluster and kafka cluster so this is a cold start.
so from a cold start I ran my python code to create a topic. Got a
leadererror and ran again
I then ran
bin/kafka-topics.sh --zookeeper 1.zk.do.production.sf.ftest.com:2181,
2.zk.do.production.sf.test.com:2181,3.zk.do.production.sf.test.com:2181
--describe --topic topic-debug-production
Topic:topic-debug-production PartitionCount:2 ReplicationFactor:2 Configs:
Topic: topic-debug-production Partition: 0 Leader: 21441 Replicas:
21441,26680 Isr: 21441,26680
Topic: topic-debug-production Partition: 1 Leader: 26680 Replicas:
26680,21441 Isr: 26680
So when I run the python code again I get the below:
import logging
logging.basicConfig(
format='%(asctime)s.%(msecs)s:%(name)s:%(thread)d:%(levelname)s:%(process)d:%(message)s',
level=logging.DEBUG
)
from kafka import SimpleProducer
from kafka import KafkaClient
kafka_host_list = '104.236.xxx.xxx:9092,104.236.xxx.xxx:9092'
kafka = KafkaClient(kafka_host_list)
producer = SimpleProducer(kafka, async=False)
test_topic = 'topic-debug-production'
test_payload = 'test'
response = producer.send_messages(test_topic,test_payload)
print response
2015-10-10
00:17:37,599.599.545001984:kafka.client:140388282590976:DEBUG:20143:Request
1: []
2015-10-10
00:17:37,600.600.193977356:kafka.conn:140388282590976:DEBUG:20143:Reinitializing
socket connection for 104.236.167.64:9092
2015-10-10
00:17:37,778.778.523921967:kafka.conn:140388282590976:DEBUG:20143:About to
send 30 bytes to Kafka, request 1
2015-10-10
00:17:37,779.779.033899307:kafka.conn:140388282590976:DEBUG:20143:Reading
response 1 from Kafka
2015-10-10
00:17:37,779.779.129981995:kafka.conn:140388282590976:DEBUG:20143:About to
read 4 bytes from Kafka
2015-10-10
00:17:37,955.955.996990204:kafka.conn:140388282590976:DEBUG:20143:Read 4/4
bytes from Kafka
2015-10-10
00:17:37,956.956.175088882:kafka.conn:140388282590976:DEBUG:20143:About to
read 12 bytes from Kafka
2015-10-10
00:17:37,956.956.24089241:kafka.conn:140388282590976:DEBUG:20143:Read 12/12
bytes from Kafka
2015-10-10
00:17:37,956.956.319093704:kafka.client:140388282590976:DEBUG:20143:Response
1: MetadataResponse(brokers=[], topics=[])
2015-10-10
00:17:37,956.956.377983093:kafka.client:140388282590976:DEBUG:20143:Updating
broker metadata: []
2015-10-10
00:17:37,956.956.424951553:kafka.client:140388282590976:DEBUG:20143:Updating
topic metadata: []
2015-10-10
00:17:37,956.956.562995911:kafka.client:140388282590976:DEBUG:20143:Request
2: ['topic-debug-production']
2015-10-10
00:17:37,956.956.639051437:kafka.conn:140388282590976:DEBUG:20143:About to
send 54 bytes to Kafka, request 2
2015-10-10
00:17:37,956.956.777095795:kafka.conn:140388282590976:DEBUG:20143:Reading
response 2 from Kafka
2015-10-10
00:17:37,956.956.829071045:kafka.conn:140388282590976:DEBUG:20143:About to
read 4 bytes from Kafka
2015-10-10
00:17:38,142.142.988920212:kafka.conn:140388282590976:DEBUG:20143:Read 4/4
bytes from Kafka
2015-10-10
00:17:38,143.143.167972565:kafka.conn:140388282590976:DEBUG:20143:About to
read 42 bytes from Kafka
2015-10-10
00:17:38,143.143.234968185:kafka.conn:140388282590976:DEBUG:20143:Read
42/42 bytes from Kafka
2015-10-10
00:17:38,143.143.313884735:kafka.client:140388282590976:DEBUG:20143:Response
2: MetadataResponse(brokers=[],
topics=[TopicMetadata(topic='topic-debug-production', error=5,
partitions=[])])
2015-10-10
00:17:38,143.143.378973007:kafka.client:140388282590976:DEBUG:20143:Updating
broker metadata: []
2015-10-10
00:17:38,143.143.425941467:kafka.client:140388282590976:DEBUG:20143:Updating
topic metadata: [TopicMetadata(topic='topic-debug-production', error=5,
partitions=[])]
Traceback (most recent call last):
File
"/home/ubuntu/workspace/feed-tests/tests/post_druid_feeds_kafka_write.py",
line 13, in <module>
response = producer.send_messages(test_topic,test_payload)
File
"/usr/local/lib/python2.7/dist-packages/kafka_python-0.9.5_dev-py2.7.egg/kafka/producer/simple.py",
line 52, in send_messages
partition = self._next_partition(topic)
File
"/usr/local/lib/python2.7/dist-packages/kafka_python-0.9.5_dev-py2.7.egg/kafka/producer/simple.py",
line 36, in _next_partition
self.client.load_metadata_for_topics(topic)
File
"/usr/local/lib/python2.7/dist-packages/kafka_python-0.9.5_dev-py2.7.egg/kafka/client.py",
line 383, in load_metadata_for_topics
kafka.common.check_error(topic_metadata)
File
"/usr/local/lib/python2.7/dist-packages/kafka_python-0.9.5_dev-py2.7.egg/kafka/common.py",
line 233, in check_error
raise error_class(response)
kafka.common.LeaderNotAvailableError:
TopicMetadata(topic='topic-debug-production', error=5, partitions=[])
On Fri, Oct 9, 2015 at 11:53 PM, Prabhjot Bharaj <prabhbha...@gmail.com>
wrote:
> Hi,
>
> Your metadata response is empty. Here is an output for my kafka cluster:
>
> *>>> kafka =
>
> KafkaClient(['96.a.a.a:9092','96.b.b.b:9092','96.c.c.c:9092','96.d.d.d:9092','96.e.e.e:9092'])*
>
> *2015-10-09
> 21:16:53,575.575.392961502:kafka.client:140735282348800:DEBUG:8705:Request
> 1: []*
>
> *2015-10-09
>
> 21:16:53,580.580.756902695:kafka.conn:140735282348800:DEBUG:8705:Reinitializing
> socket connection for 96.b.b.b:9092*
>
> *2015-10-09
> 21:16:53,981.981.142044067:kafka.conn:140735282348800:DEBUG:8705:About to
> send 30 bytes to Kafka, request 1*
>
> *2015-10-09
> 21:16:53,981.981.415987015:kafka.conn:140735282348800:DEBUG:8705:Reading
> response 1 from Kafka*
>
> *2015-10-09
> 21:16:53,981.981.508016586:kafka.conn:140735282348800:DEBUG:8705:About to
> read 4 bytes from Kafka*
>
> *2015-10-09
> 21:16:54,596.596.358060837:kafka.conn:140735282348800:DEBUG:8705:Read 4/4
> bytes from Kafka*
>
> *2015-10-09
> 21:16:54,596.596.518039703:kafka.conn:140735282348800:DEBUG:8705:About to
> read 389 bytes from Kafka*
>
> *2015-10-09
> 21:16:54,596.596.612930298:kafka.conn:140735282348800:DEBUG:8705:Read
> 389/389 bytes from Kafka*
>
> *2015-10-09
> 21:16:54,596.596.745014191:kafka.client:140735282348800:DEBUG:8705:Response
> 1: MetadataResponse(brokers=[BrokerMetadata(nodeId=1, host='96.a.a.a.a',
> port=9092), BrokerMetadata(nodeId=6, host='96.d.d.d', port=9092),
> BrokerMetadata(nodeId=2, host='96.b.b.b', port=9092),
> BrokerMetadata(nodeId=7, host='96.e.e.e', port=9092),
> BrokerMetadata(nodeId=4, host='96.c.c.c', port=9092)],
> topics=[TopicMetadata(topic='part_1_repl_3', error=0,
> partitions=[PartitionMetadata(topic='part_1_repl_3', partition=0, leader=2,
> replicas=(2, 6, 7), isr=(2, 6, 7), error=0)]),
> TopicMetadata(topic='part_1_repl_3_4', error=0,
> partitions=[PartitionMetadata(topic='part_1_repl_3_4', partition=0,
> leader=6, replicas=(6,), isr=(6,), error=0)])])*
>
> *2015-10-09
> 21:16:54,596.596.867084503:kafka.client:140735282348800:INFO:8705:Updating
> broker metadata: [BrokerMetadata(nodeId=1, host='96.a.a.a.a', port=9092),
> BrokerMetadata(nodeId=6, host='96.d.d.d', port=9092),
> BrokerMetadata(nodeId=2, host='96.b.b.b', port=9092),
> BrokerMetadata(nodeId=7, host='96.e.e.e', port=9092),
> BrokerMetadata(nodeId=4, host='96.c.c.c', port=9092)] *
>
> *2015-10-09
> 21:16:54,596.596.991062164:kafka.client:140735282348800:INFO:8705:Updating
> topic metadata: [TopicMetadata(topic='part_1_repl_3', error=0,
> partitions=[PartitionMetadata(topic='part_1_repl_3', partition=0, leader=2,
> replicas=(2, 6, 7), isr=(2, 6, 7), error=0)]),
> TopicMetadata(topic='part_1_repl_3_4', error=0,
> partitions=[PartitionMetadata(topic='part_1_repl_3_4', partition=0,
> leader=6, replicas=(6,), isr=(6,), error=0)])]*
>
> Is your cluster up ?
>
> Are you behind some firewall ?
>
> By the way, I'm using kafka-python-0.9.4 from my mac
>
> Regards,
>
> Prabhjot
>
> On Fri, Oct 9, 2015 at 7:32 PM, David Montgomery <
> davidmontgom...@gmail.com>
> wrote:
>
> > ps here is the output the I add logging to the code
> >
> > import logging
> > logging.basicConfig(
> >
> >
> >
> format='%(asctime)s.%(msecs)s:%(name)s:%(thread)d:%(levelname)s:%(process)d:%(message)s',
> > level=logging.DEBUG
> > )
> >
> > 2015-10-09
> >
> 20:59:33,656.656.464099884:kafka.client:139860093155072:DEBUG:25285:Request
> > 1: []
> > 2015-10-09
> >
> >
> 20:59:33,657.657.119989395:kafka.conn:139860093155072:DEBUG:25285:Reinitializing
> > socket connection for 45.55.xxx.xxx:9092
> > 2015-10-09
> > 20:59:33,928.928.045034409:kafka.conn:139860093155072:DEBUG:25285:About
> to
> > send 30 bytes to Kafka, request 1
> > 2015-10-09
> > 20:59:33,928.928.556919098:kafka.conn:139860093155072:DEBUG:25285:Reading
> > response 1 from Kafka
> > 2015-10-09
> > 20:59:33,928.928.649902344:kafka.conn:139860093155072:DEBUG:25285:About
> to
> > read 4 bytes from Kafka
> > 2015-10-09
> > 20:59:34,184.184.187889099:kafka.conn:139860093155072:DEBUG:25285:Read
> 4/4
> > bytes from Kafka
> > 2015-10-09
> > 20:59:34,184.184.477090836:kafka.conn:139860093155072:DEBUG:25285:About
> to
> > read 12 bytes from Kafka
> > 2015-10-09
> > 20:59:34,184.184.587001801:kafka.conn:139860093155072:DEBUG:25285:Read
> > 12/12 bytes from Kafka
> > 2015-10-09
> >
> >
> 20:59:34,184.184.746026993:kafka.client:139860093155072:DEBUG:25285:Response
> > 1: MetadataResponse(brokers=[], topics=[])
> > 2015-10-09
> >
> >
> 20:59:34,184.184.829950333:kafka.client:139860093155072:DEBUG:25285:Updating
> > broker metadata: []
> > 2015-10-09
> >
> 20:59:34,184.184.88407135:kafka.client:139860093155072:DEBUG:25285:Updating
> > topic metadata: []
> > 2015-10-09
> >
> 20:59:34,185.185.06193161:kafka.client:139860093155072:DEBUG:25285:Request
> > 2: ['topic-test-production']
> > 2015-10-09
> > 20:59:34,185.185.146093369:kafka.conn:139860093155072:DEBUG:25285:About
> to
> > send 53 bytes to Kafka, request 2
> > 2015-10-09
> > 20:59:34,185.185.309886932:kafka.conn:139860093155072:DEBUG:25285:Reading
> > response 2 from Kafka
> > 2015-10-09
> > 20:59:34,185.185.367107391:kafka.conn:139860093155072:DEBUG:25285:About
> to
> > read 4 bytes from Kafka
> > 2015-10-09
> > 20:59:34,445.445.18995285:kafka.conn:139860093155072:DEBUG:25285:Read 4/4
> > bytes from Kafka
> > 2015-10-09
> > 20:59:34,445.445.672988892:kafka.conn:139860093155072:DEBUG:25285:About
> to
> > read 41 bytes from Kafka
> > 2015-10-09
> > 20:59:34,445.445.739984512:kafka.conn:139860093155072:DEBUG:25285:Read
> > 41/41 bytes from Kafka
> > 2015-10-09
> >
> >
> 20:59:34,445.445.817947388:kafka.client:139860093155072:DEBUG:25285:Response
> > 2: MetadataResponse(brokers=[],
> > topics=[TopicMetadata(topic='topic-test-production', error=5,
> > partitions=[])])
> > 2015-10-09
> >
> >
> 20:59:34,445.445.895910263:kafka.client:139860093155072:DEBUG:25285:Updating
> > broker metadata: []
> > 2015-10-09
> >
> 20:59:34,445.445.94502449:kafka.client:139860093155072:DEBUG:25285:Updating
> > topic metadata: [TopicMetadata(topic='topic-test-production', error=5,
> > partitions=[])]
> >
> >
> > On Fri, Oct 9, 2015 at 7:00 PM, David Montgomery <
> > davidmontgom...@gmail.com>
> > wrote:
> >
> > > This is the python error
> > >
> > > kafka.common.LeaderNotAvailableError:
> > > TopicMetadata(topic='topic-test-production', error=5, partitions=[])
> > >
> > >
> > > On Fri, Oct 9, 2015 at 6:55 PM, David Montgomery <
> > > davidmontgom...@gmail.com> wrote:
> > >
> > >> Well i masked my true domain but the hostnames are valid and do
> work. I
> > >> do see the topics creates in ZK
> > >>
> > >>
> > >> brokers/topics/topic-test-production/partisions/0/state
> >
> {"controller_epoch":10,"leader":26665,"version":1,"leader_epoch":2,"isr":[26665]}
> > >>
> > >> brokers/topics/topic-test-production/partisions/1/state
> >
> {"controller_epoch":10,"leader":26665,"version":1,"leader_epoch":3,"isr":[26665]}
> > >> for example
> > {"version":1,"partitions":{"1":[2028,26665],"0":[26665,2028]}}
> > >>
> > >> But now I do get the below on describe:
> > >> roduction PartitionCount:2 ReplicationFactor:2 Configs:
> > >> Topic: topic-test-production Partition: 0 Leader: 26665 Replicas:
> > >> 26665,2028 Isr: 26665
> > >> Topic: topic-test-production Partition: 1 Leader: 26665 Replicas:
> > >> 2028,26665 Isr: 26665
> > >>
> > >> But in kafka-python I still get this error
> > >> kafka.common.LeaderNotAvailableError:
> > >> TopicMetadata(topic='topic-test-production', error=5, partitions=[])
> > >>
> > >>
> > >>
> > >> On Fri, Oct 9, 2015 at 5:31 PM, Prabhjot Bharaj <
> prabhbha...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> Is this a valid zookeeper connection string
> > >>>
> > >>>
> > >>>
> >
> 1.zk.do.production.test:2181,2.zk.do.production.test:2181,3.zk.do.production.test:2181
> > >>>
> > >>> Is 2.zk.do.production.test or 3.zk.do.production.test a valid
> > hostname??
> > >>>
> > >>> Regards,
> > >>> Prabhjot
> > >>> On Oct 9, 2015 1:37 PM, "David Montgomery" <
> davidmontgom...@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > PS I added one more server now 3 kafka servers. still kafka is not
> > >>> > working. Is there a known bug?
> > >>> >
> > >>> > On Fri, Oct 9, 2015 at 3:23 PM, David Montgomery <
> > >>> > davidmontgom...@gmail.com>
> > >>> > wrote:
> > >>> >
> > >>> > > i,
> > >>> > >
> > >>> > > I have 3 ZK servers and 2 kafka servers with 2 partitions.
> > >>> > >
> > >>> > > I am using kafka_2.11-0.8.2.1
> > >>> > >
> > >>> > > Here is how I am creating a topic:
> > >>> > >
> > >>> > > bin/kafka-topics.sh --zookeeper
> > >>> > >
> > >>> >
> > >>>
> >
> 1.zk.do.production.test:2181,2.zk.do.production.test:2181,3.zk.do.production.test:2181
> > >>> > > --create --topic topic-test-production --partitions 2
> > >>> > --replication-factor 2
> > >>> > >
> > >>> > > here is what I describe:
> > >>> > >
> > >>> > > bin/kafka-topics.sh
> > >>> > > --zookeeper
> > >>> >
> > >>>
> >
> .zk.do.production.test:2181,2.zk.do.production.test:2181,3.zk.do.production.test:2181
> > >>> > --describe
> > >>> > > --topic topic-test-production
> > >>> > >
> > >>> > > Topic:topic-test-production PartitionCount:2 ReplicationFactor:2
> > >>> Configs:
> > >>> > > Topic: topic-test-production Partition: 0 Leader: none Replicas:
> > >>> > > 26665,2028 Isr:
> > >>> > > Topic: topic-test-production Partition: 1 Leader: none Replicas:
> > >>> > > 2028,26665 Isr:
> > >>> > >
> > >>> > > When i try to write to kafka in python I get this error.
> > >>> > >
> > >>> > > kafka.common.LeaderNotAvailableError:
> > >>> > > TopicMetadata(topic='topic-test-production', error=5,
> > partitions=[])
> > >>> > >
> > >>> > > What is wrong with kafka?
> > >>> > >
> > >>> > > Thanks
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
>
>
> --
> ---------------------------------------------------------
> "There are only 10 types of people in the world: Those who understand
> binary, and those who don't"
>
