Re: HBase issues since upgrade from 0.92.4 to 0.94.6

In the end we increased the heap allocation for HBase region servers to 4GB (from it's default 1GB) and it seems to work now. On Mon, Jul 15, 2013 at 1:28 PM, Jamal B wrote: > I believe that your workload after the upgrade caused the process to exceed > it's 1 GB memory allocation, and your jvm

Re: HBase issues since upgrade from 0.92.4 to 0.94.6

I believe that your workload after the upgrade caused the process to exceed it's 1 GB memory allocation, and your jvm flag -XX:OnOutOfMemoryError=kill -9 %p worked as expected and killed it. I would remove the kill hook, or at least put out some sort of log entry to the syslog or something before

Re: HBase issues since upgrade from 0.92.4 to 0.94.6

Checked now. It is 0.94.6.1 - Original Message - From: lars hofhansl To: "user@hbase.apache.org" Cc: Sent: Sunday, July 14, 2013 6:55 AM Subject: Re: HBase issues since upgrade from 0.92.4 to 0.94.6 Didn't check, but I sincerely hope that CDH 4.3.0 ships with HBase

Re: HBase issues since upgrade from 0.92.4 to 0.94.6

Didn't check, but I sincerely hope that CDH 4.3.0 ships with HBase 0.94.6.1 (and not 0.94.6). From: David Koch To: user@hbase.apache.org Sent: Friday, July 12, 2013 3:09 AM Subject: HBase issues since upgrade from 0.92.4 to 0.94.6 Hello, NOTE: I poste

Re: HBase issues since upgrade from 0.92.4 to 0.94.6

David, you can set -Xmx1g if your JDK is 6 or above. dont need to set specify bytes. On Jul 13, 2013 12:16 AM, "David Koch" wrote: > Hello, > > This is the command that is used to launch the region servers: > > /usr/java/jdk1.7.0_25/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m > -Djava.

Re: HBase issues since upgrade from 0.92.4 to 0.94.6

Hello, This is the command that is used to launch the region servers: /usr/java/jdk1.7.0_25/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m -Djava.net.preferIPv4Stack=true -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=7

Re: HBase issues since upgrade from 0.92.4 to 0.94.6

I do think your JVM on the RS crashed. do you have GC log? do you set MR *mapred*.map.tasks.*speculative.execution=false *when you using map jobs to read or write HBASE? and if you have a heavy read/write load, how did you tune the HBase? such as block cache size, compaction, memstore etc. On F

Re: HBase issues since upgrade from 0.92.4 to 0.94.6

Thank you for your responses. With respect to the version of Java I found that Cloudera recommend1.7.x for CDH4.3. On Fri, Jul 12, 2013 at 1:32 PM, Jean-Marc S

Re: HBase issues since upgrade from 0.92.4 to 0.94.6

David, java.io.IOException: Premature EOF from inputStream at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194) for this error, generally client always ask for bytes from the stream, but sever has been shut down, so there maybe network issue or JVM crashed or some others. I don't think this

Re: HBase issues since upgrade from 0.92.4 to 0.94.6

Might want to run memtest also, just to be sure there is no memory issue. It should not since it was working fine with 0.92.4, but costs nothing... the last version of Java 6 is 45... Might also worst to give it a try if you are running with 1.6. 2013/7/12 Asaf Mesika > You need to see the jvm

Re: HBase issues since upgrade from 0.92.4 to 0.94.6

You need to see the jvm crash in .out log file and see if maybe its the .so native Hadoop code that making the problem. In our case we Downgraded from jvm 1.6.0-37 to 33 and it solved the issue. On Friday, July 12, 2013, David Koch wrote: > Hello, > > NOTE: I posted the same message in the the C

Re: HBase issues since upgrade from 0.92.4 to 0.94.6

Hi David, I will recommand you to run: - FSCK from your os (fsck.ext4) on this node; - FSCK from Hadoop on your HDFS - HBCK from HBase Seems your node has some troubles to read something, just want to see if there is related issues. JM 2013/7/12 David Koch > Hello, > > NOTE: I posted the same

HBase issues since upgrade from 0.92.4 to 0.94.6

Hello, NOTE: I posted the same message in the the Cloudera group. Since upgrading from CDH 4.0.1 (HBase 0.92.4) to 4.3.0 (HBase 0.94.6) we systematically experience problems with region servers crashing silently under workloads which used to pass without problems. More specifically, we run about