About: zookeeper-shell.sh localhost:2181 get /brokers/ids/11 The result:
zookeeper-shell.sh n1.test.com:2181 Connecting to n1.test.com:2181 Welcome to ZooKeeper! JLine support is disabled WATCHER:: WatchedEvent state:SyncConnected type:None path:null WATCHER:: WatchedEvent state:SaslAuthenticated type:None path:null On Thu, Aug 10, 2017 at 4:43 AM, Ascot Moss <ascot.m...@gmail.com> wrote: > FYI, about zookeeper, I used my existing zookeeper (as I have existing > zookeeper up and running, which is also used for hbase) > > zookeeper versoom: 3.4.10 > > zoo.cfg > ###### > > tickTime=2000 > > initLimit=10 > > syncLimit=5 > > dataDir=/usr/local/zookeeper/data > > dataLogDir=/usr/local/zookeeper/datalog > > clientPort=2181 > > maxClientCnxns=60 > > server.1=n1.test.com:2888:3888 > > server.2=n2.test.com:2888:3888 > > server.3=n3.test.com:2888:3888 > > authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider > > jaasLoginRenew=3600000 > > requireClientAuthScheme=sasl > > zookeeper.allowSaslFailedClients=false > > kerberos.removeHostFromPrincipal=true > > ###### > > > > On Thu, Aug 10, 2017 at 4:35 AM, Ascot Moss <ascot.m...@gmail.com> wrote: > >> server.properties >> >> ###### >> >> broker.id=11 >> >> port=9093 >> >> host.name=n1 >> >> advertised.host.name=192.168.0.11 >> >> allow.everyone.if.no.acl.found=true >> >> super.users=User:CN=n1.test.com,OU=TEST,O=TEST,L=TEST,ST=TEST,C=TEST >> >> listeners=SSL://n1.test.com:9093 <http://n1.test.com:9092/> >> >> advertised.listeners=SSL://n1.test.com:9093 <http://n1.test.com:9092/> >> >> ssl.client.auth=required >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 >> >> ssl.keystore.type=JKS >> >> ssl.truststore.type=JKS >> >> security.inter.broker.protocol=SSL >> >> ssl.keystore.location=/home/kafka/kafka.server.keystore.jks >> >> ssl.keystore.password=Test2017 >> >> ssl.key.password=Test2017 >> >> ssl.truststore.location=/home/kafka/kafka.server.truststore.jks >> >> ssl.truststore.password=Test2017 >> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer >> >> principal.builder.class=org.apache.kafka.common.security.aut >> h.DefaultPrincipalBuilder >> >> num.replica.fetchers=4 >> >> replica.fetch.max.bytes=1048576 >> >> replica.fetch.wait.max.ms=500 >> >> replica.high.watermark.checkpoint.interval.ms=5000 >> >> replica.socket.timeout.ms=30000 >> >> replica.socket.receive.buffer.bytes=65536 >> >> replica.lag.time.max.ms=10000 >> >> controller.socket.timeout.ms=30000 >> >> controller.message.queue.size=10 >> >> default.replication.factor=3 >> >> log.dirs=/usr/log/kafka >> >> kafka.logs.dir=/usr/log/kafka >> >> num.partitions=20 >> >> message.max.bytes=1000000 >> >> auto.create.topics.enable=true >> >> log.index.interval.bytes=4096 >> >> log.index.size.max.bytes=10485760 >> >> log.retention.hours=720 >> >> log.flush.interval.ms=10000 >> >> log.flush.interval.messages=20000 >> >> log.flush.scheduler.interval.ms=2000 >> >> log.roll.hours=168 >> >> log.retention.check.interval.ms=300000 >> >> log.segment.bytes=1073741824 >> >> delete.topic.enable=true >> >> socket.request.max.bytes=104857600 >> >> socket.receive.buffer.bytes=1048576 >> >> socket.send.buffer.bytes=1048576 >> >> num.io.threads=8 >> >> num.network.threads=8 >> >> queued.max.requests=16 >> >> fetch.purgatory.purge.interval.requests=100 >> >> producer.purgatory.purge.interval.requests=100 >> >> zookeeper.connect=n1:2181,n2:2181,n3:2181 >> >> zookeeper.connection.timeout.ms=2000 >> >> zookeeper.sync.time.ms=2000 >> >> ###### >> >> >> >> >> >> producer.properties >> >> ###### >> >> bootstrap.servers=n1.test.com:9093 <http://n1.test.com:9092/> >> >> security.protocol=SSL >> >> ssl.truststore.location=/home/kafka/kafka.client.truststore.jks >> >> ssl.truststore.password=testkafka >> >> ssl.keystore.location=/home/kafka/kafka.client.keystore.jks >> >> ssl.keystore.password=testkafka >> >> ssl.key.password=testkafka >> ##### >> >> >> (I had tried to switch to another port, 9093 is the correct port) >> >> On Thu, Aug 10, 2017 at 4:28 AM, M. Manna <manme...@gmail.com> wrote: >> >>> Your openssl test is showing connected with port 9092. but your previous >>> messages show 9093 - is there some typo issues? Where is SSL running >>> >>> Please share the following and don't leave any details out. This will >>> only >>> create more assumptions. >>> >>> 1) server.properties >>> 2) Zookeeper.properties >>> >>> Also, run the following command (when the cluster is running) >>> zookeeper-shell.sh localhost:2181 >>> get /brokers/ids/11 >>> >>> Does it show that your broker #11 is connected? >>> >>> >>> >>> >>> On 9 August 2017 at 21:17, Ascot Moss <ascot.m...@gmail.com> wrote: >>> >>> > Dear Manna, >>> > >>> > >>> > What's the status of your SSL? Have you verified that the setup is >>> working? >>> > Yes, I used " >>> > >>> > openssl s_client -debug -connect n1.test.com:9092 -tls1 >>> > Output: >>> > >>> > CONNECTED(00000003) >>> > >>> > write to 0x853e70 [0x89fd43] (155 bytes => 155 (0x9B)) >>> > >>> > 0000 - 16 03 01 00 96 01 00 00-92 03 01 59 8b 6d 0d b1 >>> ...........Y.m.. >>> > ... >>> > >>> > Server certificate >>> > >>> > -----BEGIN CERTIFICATE----- >>> > >>> > CwwCSEsxGT............ >>> > >>> > -----END CERTIFICATE----- >>> > >>> > --- >>> > >>> > SSL handshake has read 2470 bytes and written 161 bytes >>> > >>> > --- >>> > >>> > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA >>> > >>> > PSK identity hint: None >>> > >>> > Start Time: 1502309645 >>> > >>> > Timeout : 7200 (sec) >>> > >>> > Verify return code: 19 (self signed certificate in certificate >>> chain) >>> > >>> > --- >>> > >>> > Regards >>> > >>> > On Wed, Aug 9, 2017 at 10:29 PM, M. Manna <manme...@gmail.com> wrote: >>> > >>> > > Hi, >>> > > >>> > > What's the status of your SSL? Have you verified that the setup is >>> > working? >>> > > >>> > > You can enable rough logins using log4j.properties file supplier with >>> > kafka >>> > > and set the root logging level to DEBUG. This prints out more info to >>> > trace >>> > > things. Also, you can enable security logging by adding >>> > > -Djavax.security.debug=all >>> > > >>> > > Please share your producer/broker configs with us. >>> > > >>> > > Kindest Regards, >>> > > M. Manna >>> > > >>> > > On 9 August 2017 at 14:38, Ascot Moss <ascot.m...@gmail.com> wrote: >>> > > >>> > > > Hi, >>> > > > >>> > > > >>> > > > I have setup Kafka 0.10.2.1 with SSL. >>> > > > >>> > > > >>> > > > Check Status: >>> > > > >>> > > > openssl s_client -debug -connect n1:9093 -tls1 >>> > > > >>> > > > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA >>> > > > >>> > > > ... SSL-Session: >>> > > > >>> > > > Protocol : TLSv1 >>> > > > >>> > > > PSK identity hint: None >>> > > > >>> > > > Start Time: 1502285690 >>> > > > >>> > > > Timeout : 7200 (sec) >>> > > > >>> > > > Verify return code: 19 (self signed certificate in certificate >>> > chain) >>> > > > >>> > > > >>> > > > Create Topic: >>> > > > >>> > > > kafka-topics.sh --create --zookeeper n1:2181,n2:2181,n3:2181 >>> > > > --replication-factor 3 --partitions 3 --topic test02 >>> > > > >>> > > > ERROR [ReplicaFetcherThread-2-111], Error for partition [test02,2] >>> to >>> > > > broker 1:org.apache.kafka.common.erro >>> rs.UnknownTopicOrPartitionExcepti >>> > > on: >>> > > > This server does not host this topic-partition. >>> > > > (kafka.server.ReplicaFetcherThread) >>> > > > >>> > > > However, if I run describe topic, I can see it is created >>> > > > >>> > > > >>> > > > >>> > > > Describe Topic: >>> > > > >>> > > > kafka-topics.sh --zookeeper n1:2181,n2:2181,n3:2181 --describe >>> --topic >>> > > > test02 >>> > > > >>> > > > Topic:test02 PartitionCount:3 ReplicationFactor:3 Configs: >>> > > > >>> > > > Topic: test02 Partition: 0 Leader: 12 Replicas: 12,13,11 Isr: >>> 12,13,11 >>> > > > >>> > > > Topic: test02 Partition: 1 Leader: 13 Replicas: 13,11,12 Isr: >>> 13,11,12 >>> > > > >>> > > > Topic: test02 Partition: 2 Leader: 11 Replicas: 11,12,13 Isr: >>> 11,12,13 >>> > > > >>> > > > >>> > > > Consumer: >>> > > > >>> > > > kafka-console-consumer.sh --bootstrap-server n1:9093 >>> --consumer.config >>> > > > /home/kafka/config/consumer.n1.properties --topic test02 >>> > > --from-beginning >>> > > > >>> > > > >>> > > > >>> > > > Producer: >>> > > > >>> > > > kafka-console-producer.sh --broker-list n1:9093 --producer.config >>> > > > /homey/kafka/config/producer.n1.properties --sync --topic test02 >>> > > > >>> > > > ERROR Error when sending message to topic test02 with key: null, >>> > value: 0 >>> > > > bytes with error: >>> > > > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) >>> > > > >>> > > > org.apache.kafka.common.errors.TimeoutException: Expiring 1 >>> record(s) >>> > > for >>> > > > test02-1: 1506 ms has passed since batch creation plus linger time >>> > > > >>> > > > >>> > > > How to resolve it? >>> > > > >>> > > > Regards >>> > > > >>> > > >>> > >>> >> >> >
