Re: HBase checksum vs HDFS checksum

On Tue, Apr 29, 2014 at 11:53 AM, Stack wrote: > On Tue, Apr 29, 2014 at 1:54 AM, Krishna Rao wrote: > >> Thank you for your reply Anoop. >> >> However, the confusing is, unfortunately, still there because of the >> following (from >> here

Re: HBase checksum vs HDFS checksum

On Tue, Apr 29, 2014 at 1:54 AM, Krishna Rao wrote: > Thank you for your reply Anoop. > > However, the confusing is, unfortunately, still there because of the > following (from > here > ): > > "For optimal performance when short-circu

Re: HBase checksum vs HDFS checksum

Hi Ted, I had read those, but I'm confused about how this will affect non-HBase HDFS data. With HDFS checksumming off won't it affect data integrity? Krishna On 24 April 2014 15:54, Ted Yu mailto:yuzhih...@gmail.com>> wrote: Please take a look at the following: http://hbase.apache.org/book.h

Re: HBase checksum vs HDFS checksum

Thank you for your reply Anoop. However, the confusing is, unfortunately, still there because of the following (from here ): "For optimal performance when short-circuit reads are enabled, it is recommended that HDFS checksums are disa

Re: HBase checksum vs HDFS checksum

HBase using its own checksum handling doesn't directly affect HDFS. It will still maintain checksum info. The diff is at the read time.. HBase will open reader with checksum validation false and it will do checksum validation on its own. So using hbase handled checksum in a cluster should not a

Re: HBase checksum vs HDFS checksum

Hi Ted, I had read those, but I'm confused about how this will affect non-HBase HDFS data. With HDFS checksumming off won't it affect data integrity? Krishna On 24 April 2014 15:54, Ted Yu wrote: > Please take a look at the following: > > http://hbase.apache.org/book.html#perf.hdfs.configs.lo

Re: HBase checksum vs HDFS checksum

Please take a look at the following: http://hbase.apache.org/book.html#perf.hdfs.configs.localread http://hbase.apache.org/book.html#hbase.regionserver.checksum.verify On Thu, Apr 24, 2014 at 5:55 AM, Krishna Rao wrote: > Hi all, > > I understand that there is a significant improvement gain wh

Re: HBase CheckSum vs Hadoop CheckSum

o: user@hbase.apache.org > Subject: Re: HBase CheckSum vs Hadoop CheckSum > > Thanks for your replies. Few seconds I was feeling unsecured ;) > > Seems the default period for the DataBlockScanner is 3 weeks: > static final long DEFAULT_SCAN_PERIOD_HOURS = 21*24L; > > And

RE: HBase CheckSum vs Hadoop CheckSum

JM Pls check "dfs.datanode.scan.period.hours" -Anoop- From: Jean-Marc Spaggiari [jean-m...@spaggiari.org] Sent: Tuesday, February 26, 2013 7:04 PM To: user@hbase.apache.org Subject: Re: HBase CheckSum vs Hadoop CheckSum Thanks for your re

Re: HBase CheckSum vs Hadoop CheckSum

Thanks for your replies. Few seconds I was feeling unsecured ;) Seems the default period for the DataBlockScanner is 3 weeks: static final long DEFAULT_SCAN_PERIOD_HOURS = 21*24L; And I have not found anyway to modify that. I will continue to search and might drop a msg on hadoop list if I still

RE: HBase CheckSum vs Hadoop CheckSum

I was typing a reply and by the time Liang replied :) Ya agree with him. It is only the HDFS client (At RS) not doing the checksum verification based on the HDFS stored checksum. Instead HBase only check for the correctness by comparing with stored checksum values. Still the periodic operation o

Re: HBase Checksum

Agreed. One should be able to monitor these things. Mind filing a jira describing your experience? From: Jean-Marc Spaggiari To: user@hbase.apache.org; lars hofhansl Sent: Friday, February 1, 2013 1:09 PM Subject: Re: HBase Checksum Thanks for the

Re: HBase Checksum

ly one IO is needed per block. > > > > > From: Robert Dyer > To: Hbase-User > Sent: Friday, February 1, 2013 11:37 AM > Subject: Re: HBase Checksum > > Yes that log is a debug level log, as I saw in the source.  But I too > enabled DEBUG an

Re: HBase Checksum

IOs. With the checksum handled by HBase only one IO is needed per block. From: Robert Dyer To: Hbase-User Sent: Friday, February 1, 2013 11:37 AM Subject: Re: HBase Checksum Yes that log is a debug level log, as I saw in the source.  But I too enabled DEBUG

Re: HBase Checksum

ent side (RS) which tells abt creating new BlockReaderLocal . If >> >> you >> >> can see this then sure the local read is happening. >> >> >> >> Also check DN log. If local read happening, then you will not see >> >> read >> >> request related lo

Re: HBase Checksum

>> can see this then sure the local read is happening. > >> > >> Also check DN log. If local read happening, then you will not see read > >> request related logs for the HFile at the DN side. > >> You check your no# of HFiles and names for checking the lo

Re: HBase Checksum

data locality. >> >> -Anoop- >> >> From: Robert Dyer [psyb...@gmail.com] >> Sent: Friday, February 01, 2013 11:10 AM >> To: Hbase-User >> Subject: Re: HBase Checksum >> >> Not trying to hijack your thread

Re: HBase Checksum

e you sure that when you tested, u have data locality? Region movements > across RSs can break the full data locality. > > -Anoop- > > From: Robert Dyer [psyb...@gmail.com] > Sent: Friday, February 01, 2013 11:10 AM > To: Hbase-Us

RE: HBase Checksum

. -Anoop- From: Robert Dyer [psyb...@gmail.com] Sent: Friday, February 01, 2013 11:10 AM To: Hbase-User Subject: Re: HBase Checksum Not trying to hijack your thread here... But can you verify via logs that the shortcircuit is working? Because I enabled

Re: HBase Checksum

Not trying to hijack your thread here... But can you verify via logs that the shortcircuit is working? Because I enabled shortcircuit but I sure didn't see any performance increase. I haven't tried enabling hbase checksum yet but I'd like to be able to verify that works too. On Thu, Jan 31, 20

RE: HBase Checksum

You can check with HDFS level logs whether the checksum meta file is getting read to the DFS client? In the HBase handled checksum, this should not happen. Have you noticed any perf gain when you configure the HBase handled checksum option? -Anoop- From: