Seems to be a zookeeper setup issue. Mind pastebin'ing your config (for 3 zookeeper servers) ?
Please also check zookeeper server log. Cheers On Mon, Nov 17, 2014 at 11:58 AM, eluiggi <[email protected]> wrote: > I have tried that as is one of the suggestions from Cloudera manager. > However, adding the servers results in none of them able to talk to > zookeeper (not even the one on the sharing the same node) and therefore > Hbase completely down. The master throws an exception related to the one > thrown by the region servers. > > 2014-11-17 14:50:20,590 INFO org.apache.zookeeper.ClientCnxn: Opening > socket > connection to server ip-10-146-188-157.ec2.internal/10.146.188.157:2181. > Will not attempt to authenticate using SASL (unknown error) > 2014-11-17 14:50:20,591 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to > ip-10-146-188-157.ec2.internal/10.146.188.157:2181, initiating session > 2014-11-17 14:50:20,592 INFO org.apache.zookeeper.ClientCnxn: Unable to > read > additional data from server sessionid 0x0, likely server has closed socket, > closing socket connection and attempting reconnect > 2014-11-17 14:50:22,576 INFO org.apache.zookeeper.ClientCnxn: Opening > socket > connection to server ip-10-164-167-107.ec2.internal/10.164.167.107:2181. > Will not attempt to authenticate using SASL (unknown error) > 2014-11-17 14:51:00,726 INFO org.apache.zookeeper.ClientCnxn: Client > session > timed out, have not heard from server in 40032ms for sessionid 0x0, closing > socket connection and attempting reconnect > 2014-11-17 14:51:00,826 WARN > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient > ZooKeeper, > > quorum=ip-10-146-194-138.ec2.internal:2181,ip-10-146-188-157.ec2.internal:2181,ip-10-164-167-107.ec2.internal:2181, > exception=org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /hbase > 2014-11-17 14:51:00,827 ERROR > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper create > failed after 4 attempts > 2014-11-17 14:51:00,828 ERROR > org.apache.hadoop.hbase.master.HMasterCommandLine: Master exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster > at > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2775) > at > > org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:184) > at > > org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:134) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2789) > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /hbase > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) > at > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:489) > at > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:468) > at > > org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1233) > at > > org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1211) > at > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174) > at > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167) > at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:472) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2770) > ... 5 more > > One other test that I made was to connect to the zookeeper from one of the > region server nodes using zkCli.sh. It looks like the connection is > established but sockets are closed and reopen constantly as the timeout > limit is reached. > > Thanks for the help! > > > > -- > View this message in context: > http://apache-hbase.679495.n3.nabble.com/ConnectionLossException-KeeperErrorCode-ConnectionLoss-for-hbase-master-tp4066034p4066039.html > Sent from the HBase User mailing list archive at Nabble.com. >
