Hi sathya,
I agree with Nik: Ouch.
Do you see this file? It doesn't look right as you mention:
/hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.tmp/eca51fd20f464525a3a611fdcd61496e
It is 455GB and that times 3 can easily cause the DNs to run out of space.
It would be really useful if you could provide all the configurations that
the RS is using along with some jstacks. Perhaps just loop curl to grab
the output from the /dump servlet from the RS:
$ (for i in {1..30}; do echo curl http://rs:port/dump ; sleep 10; done) >
rs.dumps.txt
regards,
esteban.
--
Cloudera, Inc.
On Tue, Jan 13, 2015 at 2:53 PM, Sean Busbey <[email protected]> wrote:
> what are the configuration settings for
>
> * dfs.data.dir / dfs.datanode.data.dir
> * dfs.datanode.du.reserved
>
> For each of the directories in the first list, can you get the output of
> "tune2fs -l" ?
>
> Can you get us any non-default settings for the hbase cluster and table?
> Are you using compression on the CF?
>
>
> On Tue, Jan 13, 2015 at 1:37 PM, sathyafmt <[email protected]> wrote:
>
> > Yes, we normally run a 4 node hbase instance with 500G on each of the
> nodes
> > for HDFS. Here's the hadoop fs -ls from a single node instance.
> >
> > sathya
> > =====
> >
> > user@node1:/var/lib$ hadoop fs -ls -R /hbase
> > drwxr-xr-x - hbase hbase 0 2015-01-05 10:59 /hbase/.tmp
> > drwxr-xr-x - hbase hbase 0 2015-01-05 10:59 /hbase/WALs
> > drwxr-xr-x - hbase hbase 0 2015-01-06 13:00
> > /hbase/WALs/node1,60020,1420484330328
> > -rw-r--r-- 3 hbase hbase 29728249 2015-01-06 13:00
> >
> >
> /hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420574422072
> > -rw-r--r-- 3 hbase hbase 9 2015-01-06 12:59
> >
> >
> /hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420577990013.meta
> > -rw-r--r-- 3 hbase hbase 9 2015-01-06 13:00
> >
> >
> /hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420578022196
> > drwxr-xr-x - hbase hbase 0 2015-01-06 11:36 /hbase/archive
> > drwxr-xr-x - hbase hbase 0 2015-01-05 10:53 /hbase/corrupt
> > drwxr-xr-x - hbase hbase 0 2014-12-17 06:38 /hbase/data
> > drwxr-xr-x - hbase hbase 0 2014-12-17 06:45
> /hbase/data/default
> > drwxr-xr-x - hbase hbase 0 2015-01-05 10:59
> > /hbase/data/default/tsdb
> > drwxr-xr-x - hbase hbase 0 2014-12-17 06:45
> > /hbase/data/default/tsdb/.tabledesc
> > -rw-r--r-- 3 hbase hbase 283 2014-12-17 06:45
> > /hbase/data/default/tsdb/.tabledesc/.tableinfo.0000000001
> > drwxr-xr-x - hbase hbase 0 2014-12-17 06:45
> > /hbase/data/default/tsdb/.tmp
> > drwxr-xr-x - hbase hbase 0 2015-01-06 11:34
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71
> > -rwxr-xr-x 3 hbase hbase 50 2014-12-30 16:14
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.regioninfo
> > drwxr-xr-x - hbase hbase 0 2015-01-06 12:34
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.tmp
> >
> > -rwxr-xr-x 3 hbase hbase 488820965376 2015-01-06 12:34
> >
> >
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.tmp/eca51fd20f464525a3a611fdcd61496e
> >
> > -drwxr-xr-x - hbase hbase 0 2015-01-05 10:53
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/recovered.edits
> > drwxr-xr-x - hbase hbase 0 2015-01-06 12:34
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t
> > -rwxr-xr-x 3 hbase hbase 977014 2015-01-06 12:34
> >
> >
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/0eea561cb7b841a7b7dbb6ead58f4556
> > -rwxr-xr-x 3 hbase hbase 775564 2015-01-05 10:53
> >
> >
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/130b7f0ca2e646e29771367740713911
> > -rwxr-xr-x 3 hbase hbase 216732 2015-01-06 11:34
> >
> >
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/3bbb590e4f1e4b57a9cca41a80271738
> > drwxr-xr-x - hbase hbase 0 2015-01-05 13:00
> > /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453
> > -rwxr-xr-x 3 hbase hbase 50 2014-12-30 16:14
> > /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.regioninfo
> > drwxr-xr-x - hbase hbase 0 2015-01-06 13:16
> > /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.tmp
> > -rwxr-xr-x 3 hbase hbase 0 2015-01-06 13:16
> >
> >
> /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.tmp/20679e20318c479ca30d980d9d26293e
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4067589.html
> > Sent from the HBase User mailing list archive at Nabble.com.
> >
>
>
>
> --
> Sean
>
