Gabriel,
Suggested work around of disabling secondary works like a charm. Thanks for
your help.
~Ashish
On Thu, Aug 7, 2014 at 2:59 PM, ashish tapdiya <ashishtapd...@gmail.com>
wrote:
> Gabriel,
>
> Thanks for taking the pain to create the test case. I appreciate it. I am
> currently not using secondary indexing. I will try out the work around you
> suggested and post result here. If not I will upgrade.
>
> ~Ashish
>
>
> On Thu, Aug 7, 2014 at 2:07 PM, Gabriel Reid <gabriel.r...@gmail.com>
> wrote:
>
>> Hi Ashish,
>>
>> Thanks for the additional info. I just made an actual test case of
>> this, and I can reproduce the issue locally. It appears to be related
>> to the secondary indexing coprocessor, and only seems to be an issue
>> with HBase 0.94.x and Phoenix 3.x.
>>
>> A temporary work-around for this is to disable the secondary indexing
>> coprocessor (if you're not relying on secondary indexing). You can do
>> this programmatically as follows:
>>
>> HBaseAdmin hbaseAdmin = new HBaseAdmin(getTestClusterConfig());
>>
>> HTableDescriptor htd = hbaseAdmin.getTableDescriptor(tableName);
>> htd.removeCoprocessor("org.apache.phoenix.hbase.index.Indexer");
>>
>> hBaseAdmin.disableTable(tableName);
>> hBaseAdmin.modifyTable(tableName, htd);
>> hBaseAdmin.enableTable(tableName);
>>
>> Another option is to upgrade to HBase 0.98 and Phoenix 4, but that is
>> obviously a bigger issue.
>>
>> Are you currently making use of secondary indexing?
>>
>> - Gabriel
>>
>> On Wed, Aug 6, 2014 at 8:57 PM, ashish tapdiya <ashishtapd...@gmail.com>
>> wrote:
>> > Gabriel,
>> >
>> > Nothing in region server's or master logs. However, i do see some stuff
>> in
>> > zookeeper's log,
>> >
>> > 2014-08-06 12:15:30,621 [myid:] - INFO
>> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] -
>> > Accepted socket connection from /192.168.1.102:38727
>> > 2014-08-06 12:15:30,623 [myid:] - INFO
>> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] -
>> Client
>> > attempting to establish new session at /192.168.1.102:38$
>> > 2014-08-06 12:15:30,645 [myid:] - INFO
>> [SyncThread:0:ZooKeeperServer@617] -
>> > Established session 0x14658117d544452 with negotiated timeout 40000 for
>> > client /192.168.1.10$
>> > 2014-08-06 12:20:58,741 [myid:] - WARN
>> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught
>> end
>> > of stream exception
>> > EndOfStreamException: Unable to read additional data from client
>> sessionid
>> > 0x14658117d544452, likely client has closed socket
>> > at
>> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
>> > at
>> >
>> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>> > at java.lang.Thread.run(Thread.java:744
>> >
>> >
>> >
>> > But everything works fine if i do scan,get.
>> >
>> > Thanks,
>> > ~Ashish
>> >
>> >
>> > On Wed, Aug 6, 2014 at 1:24 AM, Gabriel Reid <gabriel.r...@gmail.com>
>> wrote:
>> >>
>> >> Is there anything in the regionserver logs that gives any additional
>> >> information (e.g. an additional stack trace?)
>> >>
>> >> On Tue, Aug 5, 2014 at 9:23 PM, ashish tapdiya <
>> ashishtapd...@gmail.com>
>> >> wrote:
>> >> > Hi Gabriel,
>> >> >
>> >> > Thanks for your response. Below is the stack trace
>> >> >
>> >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
>> after
>> >> > attempts=14, exceptions:
>> >> > Tue Aug 05 14:11:49 CDT 2014,
>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>> java.io.IOException:
>> >> > java.io.IOException
>> >> > : java.lang.NullPointerException
>> >> > Tue Aug 05 14:11:50 CDT 2014,
>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>> java.io.IOException:
>> >> > java.io.IOException
>> >> > : java.lang.NullPointerException
>> >> > Tue Aug 05 14:11:51 CDT 2014,
>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>> java.io.IOException:
>> >> > java.io.IOException
>> >> > : java.lang.NullPointerException
>> >> > Tue Aug 05 14:11:53 CDT 2014,
>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>> java.io.IOException:
>> >> > java.io.IOException
>> >> > : java.lang.NullPointerException
>> >> > Tue Aug 05 14:11:55 CDT 2014,
>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>> java.io.IOException:
>> >> > java.io.IOException
>> >> > : java.lang.NullPointerException
>> >> > Tue Aug 05 14:11:59 CDT 2014,
>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>> java.io.IOException:
>> >> > java.io.IOException
>> >> > : java.lang.NullPointerException
>> >> > Tue Aug 05 14:12:03 CDT 2014,
>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>> java.io.IOException:
>> >> > java.io.IOException
>> >> > : java.lang.NullPointerException
>> >> > Tue Aug 05 14:12:11 CDT 2014,
>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>> java.io.IOException:
>> >> > java.io.IOException
>> >> > : java.lang.NullPointerException
>> >> > Tue Aug 05 14:12:27 CDT 2014,
>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>> java.io.IOException:
>> >> > java.io.IOException
>> >> > : java.lang.NullPointerException
>> >> > Tue Aug 05 14:12:59 CDT 2014,
>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>> java.io.IOException:
>> >> > java.io.IOException
>> >> > : java.lang.NullPointerException
>> >> > Tue Aug 05 14:14:03 CDT 2014,
>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>> java.io.IOException:
>> >> > java.io.IOException
>> >> > : java.lang.NullPointerException
>> >> > Tue Aug 05 14:15:07 CDT 2014,
>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>> java.io.IOException:
>> >> > java.io.IOException
>> >> > : java.lang.NullPointerException
>> >> > Tue Aug 05 14:16:12 CDT 2014,
>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>> java.io.IOException:
>> >> > java.io.IOException
>> >> > : java.lang.NullPointerException
>> >> > Tue Aug 05 14:17:16 CDT 2014,
>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>> java.io.IOException:
>> >> > java.io.IOException
>> >> > : java.lang.NullPointerException
>> >> >
>> >> > at
>> >> >
>> >> >
>> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:187)
>> >> > at
>> >> > org.apache.hadoop.hbase.client.HTable.checkAndPut(HTable.java:946)
>> >> > at HAdminTest.testCheckNPut(HAdminTest.java:150)
>> >> > at HAdminTest.main(HAdminTest.java:257)
>> >> >
>> >> >
>> >> > Thanks,
>> >> > ~Ashish
>> >> >
>> >> >
>> >> > On Tue, Aug 5, 2014 at 1:21 PM, Gabriel Reid <gabriel.r...@gmail.com
>> >
>> >> > wrote:
>> >> >>
>> >> >> Hi Ashish,
>> >> >>
>> >> >> Could you post the full stack trace you're getting when the
>> >> >> checkAndPut fails? No immediate reason I can think of as to why this
>> >> >> would happen.
>> >> >>
>> >> >> - Gabriel
>> >> >>
>> >> >> On Tue, Aug 5, 2014 at 7:57 PM, ashish tapdiya
>> >> >> <ashishtapd...@gmail.com>
>> >> >> wrote:
>> >> >> > Folks,
>> >> >> >
>> >> >> > any intuition why this is happening.
>> >> >> >
>> >> >> > Thanks,
>> >> >> > ~Ashish
>> >> >> >
>> >> >> >
>> >> >> > On Thu, Jul 31, 2014 at 8:29 PM, ashish tapdiya
>> >> >> > <ashishtapd...@gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> I am using checkandput operation through HBase API on a table
>> >> >> >> created
>> >> >> >> in
>> >> >> >> Phoenix. However, it fails after 14 client retries. Any insight
>> on
>> >> >> >> why
>> >> >> >> it is
>> >> >> >> happening and if there is a work around?
>> >> >> >>
>> >> >> >> I tried checkandput on a table created in HBase and it works.
>> Also,
>> >> >> >> scan,
>> >> >> >> get using HBase API work fine with tables created in Phoenix.
>> >> >> >>
>> >> >> >> HBase version: 0.94
>> >> >> >> Phoenix version: 3.0
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> ~Ashish
>> >> >> >
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>>
>
>
