Hi Ashish,
Thanks for the additional info. I just made an actual test case of
this, and I can reproduce the issue locally. It appears to be related
to the secondary indexing coprocessor, and only seems to be an issue
with HBase 0.94.x and Phoenix 3.x.
A temporary work-around for this is to disable the secondary indexing
coprocessor (if you're not relying on secondary indexing). You can do
this programmatically as follows:
HBaseAdmin hbaseAdmin = new HBaseAdmin(getTestClusterConfig());
HTableDescriptor htd = hbaseAdmin.getTableDescriptor(tableName);
htd.removeCoprocessor("org.apache.phoenix.hbase.index.Indexer");
hBaseAdmin.disableTable(tableName);
hBaseAdmin.modifyTable(tableName, htd);
hBaseAdmin.enableTable(tableName);
Another option is to upgrade to HBase 0.98 and Phoenix 4, but that is
obviously a bigger issue.
Are you currently making use of secondary indexing?
- Gabriel
On Wed, Aug 6, 2014 at 8:57 PM, ashish tapdiya <ashishtapd...@gmail.com> wrote:
> Gabriel,
>
> Nothing in region server's or master logs. However, i do see some stuff in
> zookeeper's log,
>
> 2014-08-06 12:15:30,621 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] -
> Accepted socket connection from /192.168.1.102:38727
> 2014-08-06 12:15:30,623 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client
> attempting to establish new session at /192.168.1.102:38$
> 2014-08-06 12:15:30,645 [myid:] - INFO [SyncThread:0:ZooKeeperServer@617] -
> Established session 0x14658117d544452 with negotiated timeout 40000 for
> client /192.168.1.10$
> 2014-08-06 12:20:58,741 [myid:] - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end
> of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x14658117d544452, likely client has closed socket
> at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:744
>
>
>
> But everything works fine if i do scan,get.
>
> Thanks,
> ~Ashish
>
>
> On Wed, Aug 6, 2014 at 1:24 AM, Gabriel Reid <gabriel.r...@gmail.com> wrote:
>>
>> Is there anything in the regionserver logs that gives any additional
>> information (e.g. an additional stack trace?)
>>
>> On Tue, Aug 5, 2014 at 9:23 PM, ashish tapdiya <ashishtapd...@gmail.com>
>> wrote:
>> > Hi Gabriel,
>> >
>> > Thanks for your response. Below is the stack trace
>> >
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
>> > attempts=14, exceptions:
>> > Tue Aug 05 14:11:49 CDT 2014,
>> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a, java.io.IOException:
>> > java.io.IOException
>> > : java.lang.NullPointerException
>> > Tue Aug 05 14:11:50 CDT 2014,
>> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a, java.io.IOException:
>> > java.io.IOException
>> > : java.lang.NullPointerException
>> > Tue Aug 05 14:11:51 CDT 2014,
>> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a, java.io.IOException:
>> > java.io.IOException
>> > : java.lang.NullPointerException
>> > Tue Aug 05 14:11:53 CDT 2014,
>> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a, java.io.IOException:
>> > java.io.IOException
>> > : java.lang.NullPointerException
>> > Tue Aug 05 14:11:55 CDT 2014,
>> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a, java.io.IOException:
>> > java.io.IOException
>> > : java.lang.NullPointerException
>> > Tue Aug 05 14:11:59 CDT 2014,
>> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a, java.io.IOException:
>> > java.io.IOException
>> > : java.lang.NullPointerException
>> > Tue Aug 05 14:12:03 CDT 2014,
>> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a, java.io.IOException:
>> > java.io.IOException
>> > : java.lang.NullPointerException
>> > Tue Aug 05 14:12:11 CDT 2014,
>> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a, java.io.IOException:
>> > java.io.IOException
>> > : java.lang.NullPointerException
>> > Tue Aug 05 14:12:27 CDT 2014,
>> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a, java.io.IOException:
>> > java.io.IOException
>> > : java.lang.NullPointerException
>> > Tue Aug 05 14:12:59 CDT 2014,
>> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a, java.io.IOException:
>> > java.io.IOException
>> > : java.lang.NullPointerException
>> > Tue Aug 05 14:14:03 CDT 2014,
>> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a, java.io.IOException:
>> > java.io.IOException
>> > : java.lang.NullPointerException
>> > Tue Aug 05 14:15:07 CDT 2014,
>> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a, java.io.IOException:
>> > java.io.IOException
>> > : java.lang.NullPointerException
>> > Tue Aug 05 14:16:12 CDT 2014,
>> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a, java.io.IOException:
>> > java.io.IOException
>> > : java.lang.NullPointerException
>> > Tue Aug 05 14:17:16 CDT 2014,
>> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a, java.io.IOException:
>> > java.io.IOException
>> > : java.lang.NullPointerException
>> >
>> > at
>> >
>> > org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:187)
>> > at
>> > org.apache.hadoop.hbase.client.HTable.checkAndPut(HTable.java:946)
>> > at HAdminTest.testCheckNPut(HAdminTest.java:150)
>> > at HAdminTest.main(HAdminTest.java:257)
>> >
>> >
>> > Thanks,
>> > ~Ashish
>> >
>> >
>> > On Tue, Aug 5, 2014 at 1:21 PM, Gabriel Reid <gabriel.r...@gmail.com>
>> > wrote:
>> >>
>> >> Hi Ashish,
>> >>
>> >> Could you post the full stack trace you're getting when the
>> >> checkAndPut fails? No immediate reason I can think of as to why this
>> >> would happen.
>> >>
>> >> - Gabriel
>> >>
>> >> On Tue, Aug 5, 2014 at 7:57 PM, ashish tapdiya
>> >> <ashishtapd...@gmail.com>
>> >> wrote:
>> >> > Folks,
>> >> >
>> >> > any intuition why this is happening.
>> >> >
>> >> > Thanks,
>> >> > ~Ashish
>> >> >
>> >> >
>> >> > On Thu, Jul 31, 2014 at 8:29 PM, ashish tapdiya
>> >> > <ashishtapd...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> I am using checkandput operation through HBase API on a table
>> >> >> created
>> >> >> in
>> >> >> Phoenix. However, it fails after 14 client retries. Any insight on
>> >> >> why
>> >> >> it is
>> >> >> happening and if there is a work around?
>> >> >>
>> >> >> I tried checkandput on a table created in HBase and it works. Also,
>> >> >> scan,
>> >> >> get using HBase API work fine with tables created in Phoenix.
>> >> >>
>> >> >> HBase version: 0.94
>> >> >> Phoenix version: 3.0
>> >> >>
>> >> >> Thanks,
>> >> >> ~Ashish
>> >> >
>> >> >
>> >
>> >
>
>
