Harsh

I followed this blog 
(http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption)
 and got an environment via vagrant setup, no issues. I’ll poke around what the 
differences are and if find the issue will post.
Thanks for your help anyway.

Cheers
Oleg
On Feb 23, 2016, at 4:06 PM, Oleg Zhurakousky 
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:

Yeah, I noticed the localhost as well, but I’ve changed it since to FQDN and it 
is still the same including 'sname is 
zookeeper/localh...@oleg.com<mailto:zookeeper/localh...@oleg.com>’

Oleg

On Feb 23, 2016, at 4:00 PM, Harsha <ka...@harsha.io<mailto:ka...@harsha.io>> 
wrote:

whats your zookeeper.connect in server.properties  looks like. Did you
use the hostname or localhost
-Harsha

On Tue, Feb 23, 2016, at 12:01 PM, Oleg Zhurakousky wrote:
Still digging, but here is more info that may help

2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
Found ticket for 
kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com> to go to
krbtgt/oleg....@oleg.com<mailto:krbtgt/oleg....@oleg.com> expiring on Wed Feb 
24 00:59:24 EST 2016
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for 
kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com> to go to
krbtgt/oleg....@oleg.com<mailto:krbtgt/oleg....@oleg.com> expiring on Wed Feb 
24 00:59:24 EST 2016
Service ticket not found in the subject
Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 17 16 23.
CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
KrbKdcReq send: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88, 
timeout=30000, number of retries =3, #bytes=660
KDCCommunication: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88, 
timeout=30000,Attempt =1, #bytes=660
KrbKdcReq send: #bytes read=183
KdcAccessibility: remove ubuntu.oleg.com<http://ubuntu.oleg.com>
KDCRep: init() encoding tag is 126 req type is 13
KRBError:
cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000
sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000
suSec is 248635
error code is 7
error Message is Server not found in Kerberos database
cname is kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>
sname is zookeeper/localh...@oleg.com<mailto:zookeeper/localh...@oleg.com>
msgType is 30

On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky 
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:

No joy. the same error

KafkaServer {
     com.sun.security.auth.module.Krb5LoginModule required
     debug=true
     useKeyTab=true
     storeKey=true
     keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
     
principal="kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>";
};
Client {
    com.sun.security.auth.module.Krb5LoginModule required
    debug=true
    useKeyTab=true
    serviceName=zookeeper
    storeKey=true
    keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
    
principal="kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>";
};
On Feb 23, 2016, at 2:41 PM, Harsha <m...@harsha.io<mailto:m...@harsha.io>> 
wrote:

My bad it should be under Client section

Client {
   com.sun.security.auth.module.Krb5LoginModule required
   debug=true
   useKeyTab=true
   storeKey=true
   serviceName=zookeeper
   keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
   
principal="kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>";
};

-Harsha

On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
can you try adding "serviceName=zookeeper" to KafkaServer section like
KafkaServer {
    com.sun.security.auth.module.Krb5LoginModule required
    debug=true
    useKeyTab=true
    storeKey=true
    serviceName=zookeeper
    keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
    
principal="kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>";
};

On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
More info

I am starting both services as myself ‘oleg’. Validated that both key tab
files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
and Kafka as ‘kafka’

Oleg

On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky 
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:

Harsha

Thanks for following up. Here is is:
oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat  kafka_server_jaas.conf
KafkaServer {
   com.sun.security.auth.module.Krb5LoginModule required
   debug=true
   useKeyTab=true
   storeKey=true
   keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
   
principal="kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>";
};
Client {
  com.sun.security.auth.module.Krb5LoginModule required
  debug=true
  useKeyTab=true
  storeKey=true
  keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
  
principal="kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>";
};

oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat  zookeeper_jaas.conf
Server {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
storeKey=true
useTicketCache=false
principal="zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com>";
};

Cheers
Oleg

On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io<mailto:ka...@harsha.io>> 
wrote:

Oleg,
  Can you post your jaas configs. Its important that serviceName
  must match the principal name with which zookeeper is running.
  Whats the principal name zookeeper service is running with.
-Harsha

On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
Hey guys, first post here so bare with me

Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
http://kafka.apache.org/documentation.html#security_sasl and i seem to be
very close, but not quite there yet.

ZOOKEEPER
Starting Zookeeper seems to be OK (below is the relevant part of the log)
. . .
[2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
(org.apache.zookeeper.server.ZooKeeperServer)
Debug is  true storeKey true useTicketCache false useKeyTab true
doNotPrompt false ticketCache is null isInitiator true KeyTab is
/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
refreshKrb5Config is false principal is
zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com><mailto:zookeeper/ubuntu.oleg....@oleg.com>
tryFirstPass is false useFirstPass is false storePass is false clearPass
is false
principal is
zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com><mailto:zookeeper/ubuntu.oleg....@oleg.com>
Will use keytab
Commit Succeeded

[2016-02-23 13:22:40,541] INFO successfully logged in.
(org.apache.zookeeper.Login)
[2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
(org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-02-23 13:22:40,544] INFO TGT refresh thread started.
(org.apache.zookeeper.Login)
[2016-02-23 13:22:40,554] INFO TGT valid starting at:        Tue Feb 23
13:22:40 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:22:40,554] INFO TGT expires:                  Tue Feb 23
23:22:40 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
21:47:35 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:23:09,012] INFO Accepted socket connection from
/127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-02-23 13:23:09,025] INFO Client attempting to establish new session
at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-02-23 13:23:09,026] INFO Creating new log file: log.57
(org.apache.zookeeper.server.persistence.FileTxnLog)
. . .


KAFKA
Starting Kafka server is not going well yet although I see that
interaction with Kerberos is successful (see relevant log below. the
error is at the bottom)
. . .
[2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
[2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
(kafka.server.KafkaServer)
[2016-02-23 13:26:11,519] INFO JAAS File name:
/home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
(org.I0Itec.zkclient.ZkClient)
[2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2016-02-23 13:26:11,527] INFO Client
environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
GMT (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
Corporation (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client
environment:java.home=/usr/lib/jvm/java-8-oracle/jre
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client
environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
. . . . .
[2016-02-23 13:26:11,531] INFO Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client
environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client
environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,532] INFO Initiating client connection,
connectString=localhost:2181 sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
(org.I0Itec.zkclient.ZkClient)
Debug is  true storeKey true useTicketCache false useKeyTab true
doNotPrompt false ticketCache is null isInitiator true KeyTab is
/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
refreshKrb5Config is false principal is
kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com><mailto:kafka/ubuntu.oleg....@oleg.com>
tryFirstPass is false useFirstPass is false storePass is false clearPass
is false
principal is
kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com><mailto:kafka/ubuntu.oleg....@oleg.com>
Will use keytab
Commit Succeeded

[2016-02-23 13:26:11,734] INFO successfully logged in.
(org.apache.zookeeper.Login)
[2016-02-23 13:26:11,735] INFO TGT refresh thread started.
(org.apache.zookeeper.Login)
[2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
(org.apache.zookeeper.client.ZooKeeperSaslClient)
[2016-02-23 13:26:11,743] INFO Opening socket connection to server
localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2016-02-23 13:26:11,748] INFO Socket connection established to
localhost/127.0.0.1:2181, initiating session
(org.apache.zookeeper.ClientCnxn)
[2016-02-23 13:26:11,752] INFO TGT valid starting at:        Tue Feb 23
13:26:11 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:26:11,752] INFO TGT expires:                  Tue Feb 23
23:26:11 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
21:40:22 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:26:11,761] INFO Session establishment complete on server
localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
[2016-02-23 13:26:11,773] ERROR An error:
(java.security.PrivilegedActionException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not
found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper
Client will go to AUTH_FAILED state.
(org.apache.zookeeper.client.ZooKeeperSaslClient)
[2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
member failed: javax.security.sasl.SaslException: An error:
(java.security.PrivilegedActionException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not
found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper
Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
[2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
(org.I0Itec.zkclient.ZkClient)
[2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
. . .

Any pointers?

Cheers
Oleg











Reply via email to