Sridhar, The logs from solix4 are very interesting:
solix 4: 2015-04-08 22:48:21,308 INFO [regionserver60020-SendThread( solix1.abcsupply.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server solix1.abcsupply. com/10.1.252.86:2181, sessionid = 0x14c9c32c6fd0004, negotiated timeout = 40000 ... 2015-04-08 22:48:21,530 INFO [regionserver60020] regionserver.HRegionServer: reportForDuty to master=solix1.abcsupply.com,60000,1428551050776 with port=60020, startcode=1428551291142 2015-04-08 22:48:21,777 WARN [regionserver60020] regionserver.HRegionServer: error telling master we are up com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused Which mean there is a live master since we can get a znode from ZooKeeper (coincidentally zk connected to solix1 ZK server) and that znode is telling us that the master in solix1 but we cannot connect to the master. Have you verified if iptables or any other firewall rule is blocking the connectivity between the RSs and the master? thanks, esteban. -- Cloudera, Inc. On Fri, Apr 10, 2015 at 12:29 AM, sridhararao mutluri <[email protected]> wrote: > Hi Team, > > I uploaded al lregion logs in below address.The Hregions are up on three > nodes and i saw erros "connection refused" > > https://gist.github.com/drmsr6/7e40958db1ea364e3541 > > Thanks, > Sridhar > > > > From: [email protected] > > Date: Fri, 10 Apr 2015 02:02:07 -0500 > > Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of > Hbase > > To: [email protected] > > CC: [email protected]; [email protected] > > > > the master log shows that no region servers ever come online. can you > > upload the region server logs? > > > > On Fri, Apr 10, 2015 at 1:15 AM, sridhararao mutluri <[email protected] > > > > wrote: > > > > > Team, > > > > > > Please find master log url and suggest any solution: > > > > > > https://gist.github.com/drmsr6/a25de9e60bb6229f796e > > > > > > Thanks, > > > Sridhar > > > > > > > > > ------------------------------ > > > Date: Thu, 9 Apr 2015 07:46:40 -0500 > > > Subject: RE: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of > Hbase > > > From: [email protected] > > > To: [email protected] > > > CC: [email protected]; [email protected] > > > > > > > > > Please post a link to updated master logs. > > > > > > -- > > > Sean > > > On Apr 9, 2015 5:50 AM, "sridhararao mutluri" <[email protected]> > wrote: > > > > > > Hi Esteban, > > > > > > We increased maximum files and with reboot ulimit -n shows across all > > > clusters > > > > > > [hadoop@solix1 bin]$ ulimit -n > > > 65535 > > > > > > But unluckily we are getting same error now also: > > > > > > hbase(main):001:0> create 'buses', 'vi' > > > > > > > > > *ERROR: java.io.IOException: Table Namespace Manager not ready yet, try > > > again later* > > > at > > > > org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179) > > > at > > > org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735) > > > at > > > org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774) > > > at > > > > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470) > > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) > > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) > > > at > > > > org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74) > > > at > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > > at java.lang.Thread.run(Thread.java:745) > > > > > > Do we downgrade JDK from 1.8 to 1.7 and validate.Please suggest. > > > > > > Thanks, > > > Sridhar > > > > > > > > > > > > ------------------------------ > > > From: [email protected] > > > To: [email protected] > > > Subject: RE: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of > Hbase > > > Date: Thu, 9 Apr 2015 06:44:10 +0530 > > > > > > Hi Esteban, > > > > > > Thanks much for your speedy suggestion.I will implement and will come > back > > > to you. > > > > > > Regards, > > > > > > Dr.Sridhar > > > > > > > > > ------------------------------ > > > From: [email protected] > > > Date: Wed, 8 Apr 2015 18:06:21 -0700 > > > Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of > Hbase > > > To: [email protected] > > > > > > > > > what i meant to say is that you should try to change the ulimit to 64k > > > > > > -- > > > Cloudera, Inc. > > > > > > > > > On Wed, Apr 8, 2015 at 6:06 PM, Esteban Gutierrez < > [email protected]> > > > wrote: > > > > > > > > > > > > cool... so the HBase master started, but you might want to bump your > > > ulimits -n by 60 times, 1024 of max open files is too low of hbase. > > > > > > > > > -- > > > Cloudera, Inc. > > > > > > > > > On Wed, Apr 8, 2015 at 5:55 PM, sridhararao mutluri < > [email protected]> > > > wrote: > > > > > > Hi Esteban, > > > > > > Please find the link: > > > > > > https://gist.github.com/drmsr6/bcf439f09cac95b59eca > > > > > > > > > I am waiting for final inputs.:) > > > > > > Thanks, > > > Dr.Sridhar, > > > 919573453136 > > > > > > > > > ------------------------------ > > > From: [email protected] > > > Date: Wed, 8 Apr 2015 11:54:04 -0700 > > > Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of > Hbase > > > To: [email protected] > > > > > > > > > > > > Sridhar, > > > > > > what is the URL to the log file? > > > > > > thanks > > > > > > > > > -- > > > Cloudera, Inc. > > > > > > > > > On Tue, Apr 7, 2015 at 6:35 PM, sridhararao mutluri < > [email protected]> > > > wrote: > > > > > > Hi Esteban, > > > > > > I pasted logs in github.com: > > > > > > > > > > > > Thanks, > > > Sridhar > > > > > > > > > > > > > From: [email protected] > > > > Date: Tue, 7 Apr 2015 08:44:55 -0700 > > > > Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of > > > Hbase > > > > To: [email protected] > > > > > > > > > > > Sridhar, > > > > > > > > What do you see in the HBase Master logs? The exception you are > getting > > > > from the HBase Master is just a side effect and not the real cause? > Is it > > > > possible for you to upload the HBase Master logs to a site like > > > pastebin.com > > > > or gist.github.com so we can look at? > > > > > > > > cheers, > > > > esteban. > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Cloudera, Inc. > > > > > > > > > > > > On Tue, Apr 7, 2015 at 8:03 AM, sridhararao mutluri < > [email protected]> > > > > wrote: > > > > > > > > > Team, > > > > > The port the HBase Master should bind to 60000 > > > > > Thanks,Sridhar> From: [email protected] > > > > > > Date: Tue, 7 Apr 2015 16:40:54 +0200 > > > > > > Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 > HA:Issues of > > > > > Hbase > > > > > > To: [email protected] > > > > > > CC: [email protected] > > > > > > > > > > > > <property> > > > > > > <name>hbase.master</name> > > > > > > <value>hdfs://cluster1:60000</value> > > > > > > </property> > > > > > > what is it? > > > > > > > > > > > > 2015-04-07 16:34 GMT+02:00 sridhararao mutluri < > [email protected]>: > > > > > > > > > > > > > Hi, > > > > > > > This is my hbase-site.xml: > > > > > > > <configuration> <property> <name>hbase.master</name> > > > > > > > <value>hdfs://cluster1:60000</value> </property> > > > > > > > <property> <name>hbase.rootdir</name> > > > > > > > <value>hdfs://mycluster/hbase</value> </property> > > > > > <property> > > > > > > > <name>hbase.cluster.distributed</name> > > > > > > > <value>true</value> </property> <property> > > > > > > > <name>hbase.zookeeper.property.clientPort</name> > > > > > > > <value>2181</value> </property> <property> > > > > > > > <name>hbase.zookeeper.quorum</name> > > > > > > > <value>cluster1,cluster2,cluster3</value> </property> > > > > > > > <property> <name>hbase.zookeeper.property.dataDir</name> > > > > > > > <value>/hadoop/hdfs/zookeeper/data/zk1</value> > > > > > > > </property></configuration> > > > > > > > Thanks,Sridhar > > > > > > > > Date: Tue, 7 Apr 2015 07:09:44 -0700 > > > > > > > > Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 > > > HA:Issues of > > > > > > > Hbase > > > > > > > > From: [email protected] > > > > > > > > To: [email protected] > > > > > > > > CC: [email protected] > > > > > > > > > > > > > > > > bq. <property> <name>hbase.rootdir</name> > > > > > > > > <value>hdfs://mycluster/hbase</value> </property> > > > > > <property> > > > > > > > > > > > > > > > > Looks like there is a property missing at the end of the > line. > > > > > > > > > > > > > > > > You showed snippet from shell output. Have you checked master > > > log ? > > > > > > > > > > > > > > > > Cheers > > > > > > > > > > > > > > > > On Tue, Apr 7, 2015 at 5:16 AM, sridhararao mutluri < > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Team, > > > > > > > > > I am trying to use hbase 0.98 distributed mode with zk > 3.4.6 & > > > > > hadoop > > > > > > > ha > > > > > > > > > 2.6.(JDK 1.8) > > > > > > > > > I am having following issue and little help in google pages > > > also > > > > > > > > > I tried to start zk first after clearing zk data dir and > tried > > > to > > > > > start > > > > > > > > > master first and rs later and no luck > > > > > > > > > I used mycluster/hbase in hbase-site.xml and no luck to > > > me.tried > > > > > to put > > > > > > > > > hdfs-site.xml/core-site.xml in $Hbase_home/conf also. > > > > > > > > > I noticed all hadoop.*jars in $HBASE_HOME/lib are 2.2 of > hadoop > > > > > where > > > > > > > as > > > > > > > > > we are using 2.6 and tried to copy those hadoop jars ..but > no > > > luck. > > > > > > > > > A New new error is coming: > > > > > > > > > > > > > > > > > > hbase(main):002:0> create 'cars', 'vi' > > > > > > > > > ERROR: java.io.IOException: Table Namespace Manager not > ready > > > yet, > > > > > try > > > > > > > > > again later at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179) > > > > > > > > > at > > > > > > > > > > > > > > > org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735) > > > > > > > > > at > > > > > > > > > > org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774) > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470) > > > > > > > > > at > > > > > > > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) > > > > > > > > > at > > > > > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74) > > > > > > > > > at > > > > > > > > > > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > > > > > > > > at java.lang.Thread.run(Thread.java:745)ZK are running on 3 > > > > > > > servers.I > > > > > > > > > tired to stop hbase and then stopped zk in 3 clusters and > > > cleared > > > > > zk > > > > > > > data > > > > > > > > > and started fresh. > > > > > > > > > My bash_profile on classpath:export > > > > > > > > > > CLASSPATH=$CLASSPATH:/home/hadoop/hadoop-2.6.0/lib/*:.export > > > > > > > > > CLASSPATH=$CLASSPATH:/home/hadoop/hbase-0.98.4/lib/*:. > > > > > > > > > my hbase_site.xml > > > > > > > > > <property> <name>hbase.rootdir</name> > > > > > > > > > <value>hdfs://mycluster/hbase</value> </property> > > > > > > > <property> > > > > > > > > > and hadoop core-site is same. > > > > > > > > > Any incompatibility between JDK1.8 with hbase or hadoop > 2.6? > > > > > > > > > Please suggest any solution. > > > > > > > > > Thanks,Sridhar > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Sean >
