The fourth parameter should be of type Class.
On Nov 2, 2013, at 8:58 PM, Bharat Shetty wrote:
> Hi,
>
> I'm trying to export data from HBase table into HDFS using map-reduce in
> the Text format. Note that I have already tried the export tool that comes
> with HBase, which dumps data into the
Hi,
I'm trying to export data from HBase table into HDFS using map-reduce in
the Text format. Note that I have already tried the export tool that comes
with HBase, which dumps data into the HDFS as bytes in a sequence file.
However, my use-case requires Text data to be dumped into the HDFS.
I'm r
There appear to be significant performance advantages to using Hadoop 1.2.1.
I was just debugging why the VerifyReplication test takes 190s in 0.94 vs. 39s
in HBase trunk.
After some debugging it turns out the difference is due to the different
default Hadoop versions (0.94 uses 1.0.4, trunk uses
@Ted: okay, thanks for the information
@Asaf: It seems to work if I compress the bytes by my self. I use snappy
for that ( https://code.google.com/p/snappy/ ). The 120mb BitSet is
compressed to a 5mb byte array. So far the hbase server did not crashed.
Thanks!
kind regards
2013/11/2 Ted Yu
Compression happens on server.
See src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java (0.94)
In 0.96 and beyond, see http://hbase.apache.org/book.html#rpc.configs
Cheers
On Sat, Nov 2, 2013 at 9:46 AM, John wrote:
> You mean I should use the BitSet, transform it into bytes and the
You mean I should use the BitSet, transform it into bytes and then compress
it by my own in the map-function? Hmmm ... I could try it. What is the best
way to compress it in java?
BTW. I'm not sure how exactly the hbase compression works. As I mentioned I
have allready enabled the LZO compression
If mean, if you take all those bytes if the bit set and zip them, wouldn't
you reduce it significantly? Less traffic on the wire, memory in HBase, etc.
On Saturday, November 2, 2013, John wrote:
> I already use LZO compression in HBase. Or do you mean a compressed Java
> object? Do you know an im
I already use LZO compression in HBase. Or do you mean a compressed Java
object? Do you know an implementation?
kind regards
2013/11/2 Asaf Mesika
> I would try to compress this bit set.
>
> On Nov 2, 2013, at 2:43 PM, John wrote:
>
> > Hi,
> >
> > thanks for your answer! I increase the "Map
I would try to compress this bit set.
On Nov 2, 2013, at 2:43 PM, John wrote:
> Hi,
>
> thanks for your answer! I increase the "Map Task Maximum Heap Size" to 2gb
> and it seems to work. The OutOfMemoryEroror is gone. But the HBase Region
> server are now crashing all the time :-/ I try to stor
Hi,
thanks for your answer! I increase the "Map Task Maximum Heap Size" to 2gb
and it seems to work. The OutOfMemoryEroror is gone. But the HBase Region
server are now crashing all the time :-/ I try to store the bitvector
(120mb in size) for some rows. This seems to be very memory intensive, the
10 matches
Mail list logo
