Gabriel,
Thanks for taking the pain to create the test case. I appreciate it. I am
currently not using secondary indexing. I will try out the work around you
suggested and post result here. If not I will upgrade.
~Ashish
On Thu, Aug 7, 2014 at 2:07 PM, Gabriel Reid <gabriel.r...@gmail.com> wrote:
> Hi Ashish,
>
> Thanks for the additional info. I just made an actual test case of
> this, and I can reproduce the issue locally. It appears to be related
> to the secondary indexing coprocessor, and only seems to be an issue
> with HBase 0.94.x and Phoenix 3.x.
>
> A temporary work-around for this is to disable the secondary indexing
> coprocessor (if you're not relying on secondary indexing). You can do
> this programmatically as follows:
>
> HBaseAdmin hbaseAdmin = new HBaseAdmin(getTestClusterConfig());
>
> HTableDescriptor htd = hbaseAdmin.getTableDescriptor(tableName);
> htd.removeCoprocessor("org.apache.phoenix.hbase.index.Indexer");
>
> hBaseAdmin.disableTable(tableName);
> hBaseAdmin.modifyTable(tableName, htd);
> hBaseAdmin.enableTable(tableName);
>
> Another option is to upgrade to HBase 0.98 and Phoenix 4, but that is
> obviously a bigger issue.
>
> Are you currently making use of secondary indexing?
>
> - Gabriel
>
> On Wed, Aug 6, 2014 at 8:57 PM, ashish tapdiya <ashishtapd...@gmail.com>
> wrote:
> > Gabriel,
> >
> > Nothing in region server's or master logs. However, i do see some stuff
> in
> > zookeeper's log,
> >
> > 2014-08-06 12:15:30,621 [myid:] - INFO
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] -
> > Accepted socket connection from /192.168.1.102:38727
> > 2014-08-06 12:15:30,623 [myid:] - INFO
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client
> > attempting to establish new session at /192.168.1.102:38$
> > 2014-08-06 12:15:30,645 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617]
> -
> > Established session 0x14658117d544452 with negotiated timeout 40000 for
> > client /192.168.1.10$
> > 2014-08-06 12:20:58,741 [myid:] - WARN
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught
> end
> > of stream exception
> > EndOfStreamException: Unable to read additional data from client
> sessionid
> > 0x14658117d544452, likely client has closed socket
> > at
> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
> > at
> >
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> > at java.lang.Thread.run(Thread.java:744
> >
> >
> >
> > But everything works fine if i do scan,get.
> >
> > Thanks,
> > ~Ashish
> >
> >
> > On Wed, Aug 6, 2014 at 1:24 AM, Gabriel Reid <gabriel.r...@gmail.com>
> wrote:
> >>
> >> Is there anything in the regionserver logs that gives any additional
> >> information (e.g. an additional stack trace?)
> >>
> >> On Tue, Aug 5, 2014 at 9:23 PM, ashish tapdiya <ashishtapd...@gmail.com
> >
> >> wrote:
> >> > Hi Gabriel,
> >> >
> >> > Thanks for your response. Below is the stack trace
> >> >
> >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> >> > attempts=14, exceptions:
> >> > Tue Aug 05 14:11:49 CDT 2014,
> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
> java.io.IOException:
> >> > java.io.IOException
> >> > : java.lang.NullPointerException
> >> > Tue Aug 05 14:11:50 CDT 2014,
> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
> java.io.IOException:
> >> > java.io.IOException
> >> > : java.lang.NullPointerException
> >> > Tue Aug 05 14:11:51 CDT 2014,
> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
> java.io.IOException:
> >> > java.io.IOException
> >> > : java.lang.NullPointerException
> >> > Tue Aug 05 14:11:53 CDT 2014,
> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
> java.io.IOException:
> >> > java.io.IOException
> >> > : java.lang.NullPointerException
> >> > Tue Aug 05 14:11:55 CDT 2014,
> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
> java.io.IOException:
> >> > java.io.IOException
> >> > : java.lang.NullPointerException
> >> > Tue Aug 05 14:11:59 CDT 2014,
> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
> java.io.IOException:
> >> > java.io.IOException
> >> > : java.lang.NullPointerException
> >> > Tue Aug 05 14:12:03 CDT 2014,
> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
> java.io.IOException:
> >> > java.io.IOException
> >> > : java.lang.NullPointerException
> >> > Tue Aug 05 14:12:11 CDT 2014,
> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
> java.io.IOException:
> >> > java.io.IOException
> >> > : java.lang.NullPointerException
> >> > Tue Aug 05 14:12:27 CDT 2014,
> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
> java.io.IOException:
> >> > java.io.IOException
> >> > : java.lang.NullPointerException
> >> > Tue Aug 05 14:12:59 CDT 2014,
> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
> java.io.IOException:
> >> > java.io.IOException
> >> > : java.lang.NullPointerException
> >> > Tue Aug 05 14:14:03 CDT 2014,
> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
> java.io.IOException:
> >> > java.io.IOException
> >> > : java.lang.NullPointerException
> >> > Tue Aug 05 14:15:07 CDT 2014,
> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
> java.io.IOException:
> >> > java.io.IOException
> >> > : java.lang.NullPointerException
> >> > Tue Aug 05 14:16:12 CDT 2014,
> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
> java.io.IOException:
> >> > java.io.IOException
> >> > : java.lang.NullPointerException
> >> > Tue Aug 05 14:17:16 CDT 2014,
> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
> java.io.IOException:
> >> > java.io.IOException
> >> > : java.lang.NullPointerException
> >> >
> >> > at
> >> >
> >> >
> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:187)
> >> > at
> >> > org.apache.hadoop.hbase.client.HTable.checkAndPut(HTable.java:946)
> >> > at HAdminTest.testCheckNPut(HAdminTest.java:150)
> >> > at HAdminTest.main(HAdminTest.java:257)
> >> >
> >> >
> >> > Thanks,
> >> > ~Ashish
> >> >
> >> >
> >> > On Tue, Aug 5, 2014 at 1:21 PM, Gabriel Reid <gabriel.r...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi Ashish,
> >> >>
> >> >> Could you post the full stack trace you're getting when the
> >> >> checkAndPut fails? No immediate reason I can think of as to why this
> >> >> would happen.
> >> >>
> >> >> - Gabriel
> >> >>
> >> >> On Tue, Aug 5, 2014 at 7:57 PM, ashish tapdiya
> >> >> <ashishtapd...@gmail.com>
> >> >> wrote:
> >> >> > Folks,
> >> >> >
> >> >> > any intuition why this is happening.
> >> >> >
> >> >> > Thanks,
> >> >> > ~Ashish
> >> >> >
> >> >> >
> >> >> > On Thu, Jul 31, 2014 at 8:29 PM, ashish tapdiya
> >> >> > <ashishtapd...@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> I am using checkandput operation through HBase API on a table
> >> >> >> created
> >> >> >> in
> >> >> >> Phoenix. However, it fails after 14 client retries. Any insight on
> >> >> >> why
> >> >> >> it is
> >> >> >> happening and if there is a work around?
> >> >> >>
> >> >> >> I tried checkandput on a table created in HBase and it works.
> Also,
> >> >> >> scan,
> >> >> >> get using HBase API work fine with tables created in Phoenix.
> >> >> >>
> >> >> >> HBase version: 0.94
> >> >> >> Phoenix version: 3.0
> >> >> >>
> >> >> >> Thanks,
> >> >> >> ~Ashish
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>
