Re: Snapshot failure

2014-10-29 Thread Ian Brooks
Hi, I applied the suggested timeout changes and it lasted longer before it started failing, but its now just throwing the same errors on "procedure.Subprocedure: Subprocedure pool is full!" Looking through the logs, it looks like the nodes may not correctly updating they completion state after

Re: Snapshot failure

2014-10-17 Thread Ian Brooks
Hi, I got a change finally to apply the timeout changes and it lasted longer before it started failing, but its now just throwing the same errors on "procedure.Subprocedure: Subprocedure pool is full!" Looking through the logs, it looks like the nodes may not correctly updating they completion

Re: Snapshot failure

2014-10-16 Thread Ian Brooks
Hi, I got a change finally to apply the timeout changes and it lasted longer before it started failing, but its now just throwing the same errors on "procedure.Subprocedure: Subprocedure pool is full!" Looking through the logs, it looks like the nodes may not correctly updating they completion

Re: Snapshot failure

2014-09-23 Thread Ted Yu
Here're the config parameters related to controlling snapshot timeout: hbase.snapshot.master.timeoutMillis 60 This is the time HBase master waits for the snapshot operation to complete. Do not confuse this hbase.snapshot.master.timeout.millis, which although

Re: Snapshot failure

2014-09-23 Thread Ian Brooks
Hi, I'm running snapshots on 7 tables every hour, the last run managed 15 rounds of snapshots before starting to fail. I did notice the following while getting the success count 2014-09-10 11:01:55,453 DEBUG [rs(##,16020,1410126483043)-snapshot-pool151-thread-1] snapshot.FlushSnap

Re: Snapshot failure

2014-09-23 Thread Ted Yu
Subprocedure pool is full. How many snapshot requests did you submit before seeing the following ? Cheers On Sep 23, 2014, at 2:28 AM, Ian Brooks wrote: > Hi, > > I'm seeing an issue on our hbase cluster which is preventing snapshots from > working. So far the only way i can get it working

Snapshot failure

2014-09-23 Thread Ian Brooks
Hi, I'm seeing an issue on our hbase cluster which is preventing snapshots from working. So far the only way i can get it working again is to restart all the regionservers which is not ideal. I'm running hbase 0.98.3 on hadoop 2.4.0, the stacktraces are >From host given the snapshot command vi