Hi Pierre,
Do you see any errors in the server.log when this command
ran. Can you please open a thread here
https://community.hortonworks.com/answers/index.html .
Thanks,
Harsha
On Tue, Jun 7, 2016, at 09:22 AM, Pierre Labiausse wrote:
> Hi,
>
> I'm not able to use kafka-consumer-groups.sh to monitor the lag of my
> consumers when my cluster is kerberized.
>
> I'm using kafka version 0.9.0 installed on an hortonworks hdp 2.4.0
> cluster.
>
> I've replicated my setup on two sandboxes, one without kerberos and one
> with kerberos.
>
> On the one without kerberos, I'm able to get an answer from the
> consumer-groups client with the following command:
> ./kafka-consumer-groups.sh --list --new-consumer --bootstrap-server
> sandbox.hortonworks.com:6667
>
> On the one with kerberos activated, i've tried the same command without
> specifying the security-protocoln which fails with a java.io.EOFException
> when connecting to the broker.
> When specifying the security-protocol (PLAINTEXTSASL or SASL_PLAINTEXT),
> I'm seeing a java.io.IOException: Connection reset by peer (full log is
> below).
>
> In both the cases with kerberos, without activating the debug logs in the
> client, it appears to hang since it is stuck in an infinite loop of
> retrying to connect with the broker (and failing)
>
>
> Am I missing something in order to use this tool in a kerberized
> environment ? As of now, I'm not seeing any other way to monitor consumer
> offsets, since they are not stored in zookeeper anymore.
>
> Thanks in advance,
> Pierre
>
>
> ****** Full stack of execution with kerberos and security-protocol
> specified ******
> ./kafka-consumer-groups.sh --security-protocol PLAINTEXTSASL
> --new-consumer
> --bootstrap-server sandbox.hortonworks.com:6667 --list
> [2016-06-07 12 <callto:2016-06-07%2012>:42:26,951] INFO Successfully
> logged
> in. (org.apache.kafka.common.security.kerberos.Login)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:26,952] DEBUG It is a Kerberos
> ticket (org.apache.kafka.common.security.kerberos.Login)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:26,971] INFO TGT refresh
> thread
> started. (org.apache.kafka.common.security.kerberos.Login)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:26,981] DEBUG Found TGT Ticket
> (hex) =
> 0000: 61 82 01 5F 30 82 01 5B A0 03 02 01 05 A1 09 1B a.._0..[........
> 0010: 07 49 54 52 2E 4C 41 4E A2 1C 30 1A A0 03 02 01 .ITR.LAN..0.....
> 0020: 02 A1 13 30 11 1B 06 6B 72 62 74 67 74
> <callto:72%2062%2074%2067%2074>
> 1B 07 49 ...0...krbtgt..I
> 0030: 54 52 2E 4C 41 4E A3 82 01 29 30 82 01 25
> <callto:82%2001%2029%2030%2082%2001%2025> A0 03 TR.LAN...)0..%..
> 0040: 02 01 12 A1 03 02 01 01 A2 82 01 17 04 82 01 13
> <callto:82%2001%2017%2004%2082%2001%2013> ................
> 0050: D9 9F 09 9C F7 96 72 D2 5F 84 20 B9 D7 5D DC 7B ......r._. ..]..
> 0060: 8D 4F A0 03 DC D5 85 54 86 4D A1 A6 F1 31 5A BF .O.....T.M...1Z.
> 0070: F9 40 71 43 20 97 <callto:40%2071%2043%2020%2097> 7F 84 D6 F7 2D 93
> 16 27 06 B2 .@qC .....-..'..
> 0080: 42 45 9D DE C3 4C 61 B6 8B 9B B3 E5 F8 F7 EB 3E BE...La........>
> 0090: D6 53 AE 9D 5D E0 06 DA 75 E0 43 DE 28 9C DE CB .S..]...u.C.(...
> 00A0: CF 72 00 5A CC 69 20 82 5F C6 4F 1D 7F D0 1F FB .r.Z.i ._.O.....
> 00B0: 92 55 B6 31 69 F1 E8 5B FD 2B 22 F8 15 E0 5D 84 .U.1i..[.+"...].
> 00C0: 5A 1A 2B 6D 0B 90 93 97 5B 06 EC 30 37 3C BB 71 Z.+m....[..07<.q
> 00D0: 0B 23 24 67 F2 70 ED 1A E2 FF 6F 3A 12 0F B2 1D .#$g.p....o:....
> 00E0: AD B9 C9 2C 24 B3 89 B3 90 22 8F 5C 1E AE 86 99 ...,$....".\....
> 00F0: 1A B5 B4 4A 3E 1D 6F 73 FD CB 60 D3 E3 76 71 6C ...J>.os..`..vql
> 0100: 90 B5 EA 4A D3 74 87 0E 02 9E C4 6D 0E 49 A2 47 ...J.t.....m.I.G
> 0110: A4 2A FA CD D4 96 65 F3 FC E1 FB 9A 6F A1 0F 0E .*....e.....o...
> 0120: AF 6F 9F 9F D5 7C 5A 29 FE BF 84 18 2E CC 7F 0C .o....Z)........
> 0130: 07 53 D2 F9 0A 44 DA 8E 3C B6 90 C0 71 69 5C CA .S...D..<...qi\.
> 0140: 9F E1 FE 23 71 C1 B7 B1 1A 7D 84 BD 33 AA ED A6 ...#q.......3...
> 0150: 9A CE 08 A9 9B 6E 29 54 B5 6B 06 9A 4D 4C 5F 3A .....n)T.k..ML_:
> 0160: CF A6 FF ...
>
> Client Principal = kafka/sandbox.hortonworks....@itr.lan
> Server Principal = krbtgt/itr....@itr.lan
> Session Key = EncryptionKey: keyType=18 keyBytes (hex dump)=
> 0000: 8E 4E 45 F8 0D B4 33 0C ED C5 7C A2 2D E2 C2 19 .NE...3.....-...
> 0010: 87 CC 27 68 72 B1 5B F8 C4 7D E8 BF EC F0 E9 F4 ..'hr.[.........
>
>
> Forwardable Ticket true
> Forwarded Ticket false
> Proxiable Ticket false
> Proxy Ticket false
> Postdated Ticket false
> Renewable Ticket true
> Initial Ticket true
> Auth Time = Tue Jun 07 12:36:46 UTC 2016
> Start Time = Tue Jun 07 12:36:46 UTC 2016
> End Time = Wed Jun 08 12:36:46 UTC 2016
> Renew Till = Tue Jun 07 12:36:46 UTC 2016
> Client Addresses Null . (org.apache.kafka.common.security.kerberos.Login)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:26,981] INFO TGT valid
> starting
> at: Tue Jun 07 12:36:46 UTC 2016
> (org.apache.kafka.common.security.kerberos.Login)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:26,981] INFO TGT expires: Wed
> Jun 08 12:36:46 UTC 2016
> (org.apache.kafka.common.security.kerberos.Login)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:26,981] INFO TGT refresh
> sleeping until: Tue Jun 07 14:40:41 UTC 2016
> (org.apache.kafka.common.security.kerberos.Login)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,036] DEBUG Updated cluster
> metadata version 1 to Cluster(nodes = [Node(-1, sandbox.hortonworks.com,
> 6667)], partitions = []) (org.apache.kafka.clients.Metadata)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,044] DEBUG Added sensor
> with
> name connections-closed: (org.apache.kafka.common.metrics.Metrics)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,046] DEBUG Added sensor
> with
> name connections-created: (org.apache.kafka.common.metrics.Metrics)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,047] DEBUG Added sensor
> with
> name bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,047] DEBUG Added sensor
> with
> name bytes-sent: (org.apache.kafka.common.metrics.Metrics)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,048] DEBUG Added sensor
> with
> name bytes-received: (org.apache.kafka.common.metrics.Metrics)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,048] DEBUG Added sensor
> with
> name select-time: (org.apache.kafka.common.metrics.Metrics)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,048] DEBUG Added sensor
> with
> name io-time: (org.apache.kafka.common.metrics.Metrics)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,082] DEBUG Initiating
> connection to node -1 at sandbox.hortonworks.com:6667.
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,091] DEBUG Added sensor
> with
> name node--1.bytes-sent (org.apache.kafka.common.metrics.Metrics)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,091] DEBUG Added sensor
> with
> name node--1.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,091] DEBUG Added sensor
> with
> name node--1.latency (org.apache.kafka.common.metrics.Metrics)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,091] DEBUG Completed
> connection to node -1 (org.apache.kafka.clients.NetworkClient)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,182] DEBUG Sending metadata
> request ClientRequest(expectResponse=true, callback=null,
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=3,client_id=admin-1},
> body={topics=[]}), isInitiatedByNetworkClient,
> createdTimeMs=1465303347182,
> sendTimeMs=0) to node -1 (org.apache.kafka.clients.NetworkClient)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,187] DEBUG Connection with
> sandbox.hortonworks.com/10.0.2.15 disconnected
> (org.apache.kafka.common.network.Selector)
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384)
> at
> org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:108)
> at
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:81)
> at
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
> at
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:160)
> at
> org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:141)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
> at
> kafka.admin.AdminClient.kafka$admin$AdminClient$$send(AdminClient.scala:52)
> at
> kafka.admin.AdminClient$$anonfun$sendAnyNode$1.apply(AdminClient.scala:67)
> at
> kafka.admin.AdminClient$$anonfun$sendAnyNode$1.apply(AdminClient.scala:64)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:64)
> at kafka.admin.AdminClient.findAllBrokers(AdminClient.scala:93)
> at kafka.admin.AdminClient.listAllGroups(AdminClient.scala:101)
> at kafka.admin.AdminClient.listAllGroupsFlattened(AdminClient.scala:122)
> at
> kafka.admin.AdminClient.listAllConsumerGroupsFlattened(AdminClient.scala:126)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.list(ConsumerGroupCommand.scala:322)
> at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:73)
> at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,190] DEBUG Node -1
> disconnected. (org.apache.kafka.clients.NetworkClient)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,190] TRACE Cancelled
> request
> ClientRequest(expectResponse=true,
> callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@3528b777,
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=2,client_id=admin-1},
> body={topics=[]}), createdTimeMs=1465303347182, sendTimeMs=1465303347184)
> due to node -1 being disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,190] TRACE Cancelled
> request
> ClientRequest(expectResponse=true, callback=null,
> request=RequestSend(header={api_key=3,api_version=0,correlation_id=3,client_id=admin-1},
> body={topics=[]}), isInitiatedByNetworkClient,
> createdTimeMs=1465303347182,
> sendTimeMs=1465303347182) due to node -1 being disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-07 12 <callto:2016-06-07%2012>:42:27,190] DEBUG Give up sending
> metadata request since no node is available
> (org.apache.kafka.clients.NetworkClient)
