Re: Urgent help with a down node.

Bryan Hughes Mon, 08 Jul 2013 08:13:23 -0700

Andrew,

Thanks for the tip on how to use Google. :) But that was not myoriginal question. I wanted to understand in more detail from the Bashofolks what


2013-07-07 12:51:42 =ERROR REPORT====

Hintfile'./data/bitcask/22835963083295358096932575511191922182123945984/3.bitcask.hint'contains pointer 16555635 566 that is greater than total data size 16556032


and

2013-07-07 12:54:43 =ERROR REPORT====
Bad datafile entry, discarding(383/566 bytes)

meant to my system. For example, did I lose data and if so, how do Iknow what data was lost? More importantly is if this is data lost, howdid it happen. I ran fsck on all the disks and checked the health ofthe system - which is all good.


The later information was included for completeness.

Bryan

On 7/8/13 12:49 AM, Andrew Berman wrote:

Bryan,

What version of Erlang? You should check this out:https://github.com/basho/riak_kv/issues/411


BTW - Google is your friend, which is how I found the above issue :)

--Andrew

On Sun, Jul 7, 2013 at 3:01 PM, Bryan Hughes <br...@go-factory.net<mailto:br...@go-factory.net>> wrote:


    Hi Mark,

    DOH - sorry for the lack of detail.  Didnt have enough coffee this
    morning.

    OS:     CentOS release 6.3 (Final)
    Riak:   Riak 1.2.1

    Hadnt had a chance to upgrade to 1.3 yet.

    Got the node back up - but not entirely sure why which is a little
    concerning.  Been verifying the data, and everything looks
    intact.  When I try to run riak-admin status, I get the following
    (note I am not entirely sure this was the case when we first set
    the node up):

    $ riak-admin status
    Status failed, see log for details

    The logs shows:

    2013-07-07 14:55:03.858 [error]
    <0.12982.0>@riak_kv_console:status:173 Status failed
    error:function_clause
    2013-07-07 14:55:03.858 [error] emulator Error in process
    <0.12983.0> on node 'riak@127.0.0.1 <mailto:riak@127.0.0.1>' with
    exit value:
    
{badarg,[{erlang,system_info,[global_heaps_size],[]},{riak_kv_stat,system_stats,0,[{file,"src/riak_kv_stat.erl"},{line,421}]},{riak_kv_stat,produce_stats,0,[{file,"src/riak_kv_stat.erl"},{line,320}]},{timer,tc,3,[{file,"timer...


    This is on a dev cluster with an out-of-the box configuration
    using bitcask.

    Thanks!

    Bryan


    On 7/7/13 2:51 PM, Mark Phillips wrote:

    Hi Bryan,

    I remember seeing something similar on the list a while ago. I'll
    dig through the archives (Riak.markmail.org
    <http://Riak.markmail.org>) if I have a few minutes later tonight.

    In the mean time, what version of Riak is this? And what OS?

    Mark

    On Sunday, July 7, 2013, Bryan Hughes wrote:

        Anyone familiar with this error message?

        2013-07-07 12:51:42 =ERROR REPORT====
        Hintfile
        
'./data/bitcask/22835963083295358096932575511191922182123945984/3.bitcask.hint'
        contains pointer 16555635 566 that is greater than total data
        size 16556032
        2013-07-07 12:51:45 =ERROR REPORT====
        Hintfile
        
'./data/bitcask/114179815416476790484662877555959610910619729920/3.bitcask.hint'
        contains pointer 17817310 567 <tel:17817310%20567> that is
        greater than total data size 17817600
        2013-07-07 12:51:46 =ERROR REPORT====
        Hintfile
        
'./data/bitcask/159851741583067506678528028578343455274867621888/3.bitcask.hint'
        contains pointer 7573448 567 <tel:7573448%20567> that is
        greater than total data size 7573504
        2013-07-07 12:51:46 =ERROR REPORT====
        Bad datafile entry 1:
        
{ok,<<131,104,2,109,0,0,0,9,65,80,73,67,79,85,78,84,83,109,0,0,0,33,55,56,54,57,52,49,56,49,94,103,111,115,101,114,118,105,99,101,95,99>>}
        2013-07-07 12:51:56 =ERROR REPORT====
        Hintfile
        
'./data/bitcask/730750818665451459101842416358141509827966271488/3.bitcask.hint'
        contains pointer 13229833 581 that is greater than total data
        size 13230080
        2013-07-07 12:52:05 =ERROR REPORT====
        Hintfile
        
'./data/bitcask/1187470080331358621040493926581979953470445191168/3.bitcask.hint'
        contains pointer 23465420 578 that is greater than total data
        size 23465984
        2013-07-07 12:52:06 =ERROR REPORT====
        Hintfile
        
'./data/bitcask/1210306043414653979137426502093171875652569137152/3.bitcask.hint'
        contains pointer 27733824 578 that is greater than total data
        size 27734016
        2013-07-07 12:52:07 =ERROR REPORT====
        Hintfile
        
'./data/bitcask/1233142006497949337234359077604363797834693083136/3.bitcask.hint'
        contains pointer 15014008 578 <tel:15014008%20578> that is
        greater than total data size 15014586
        2013-07-07 12:54:43 =ERROR REPORT====
        Bad datafile entry, discarding(383/566 bytes)
        2013-07-07 12:54:45 =ERROR REPORT====
        Bad datafile entry, discarding(276/567 bytes)
        2013-07-07 12:54:46 =ERROR REPORT====
        Bad datafile entry, discarding(42/567 bytes)
        2013-07-07 12:54:57 =ERROR REPORT====
        Bad datafile entry, discarding(233/581 bytes)
        2013-07-07 12:55:06 =ERROR REPORT====
        Bad datafile entry, discarding(550/578 bytes)
        2013-07-07 12:55:07 =ERROR REPORT====
        Bad datafile entry, discarding(178/578 bytes)
        2013-07-07 12:56:00 =ERROR REPORT====
        Error in process <0.1536.0> on node 'riak@127.0.0.1' with
        exit value:
        
{badarg,[{erlang,system_info,[global_heaps_size],[]},{riak_kv_stat,system_stats,0,[{file,"src/riak_kv_stat.erl"},{line,421}]},{riak_kv_stat,produce_stats,0,[{file,"src/riak_kv_stat.erl"},{line,320}]},{timer,tc,3,[{file,"timer...

--

        Bryan Hughes
        *Go Factory*
        http://www.go-factory.net

        /"Internet Class, Enterprise Grade"/

--

    Bryan Hughes
    CTO and Founder / *Go Factory*
    (415) 515-7916 <tel:%28415%29%20515-7916>


    http://www.go-factory.net

    /"Internet Class, Enterprise Grade"/



    _______________________________________________
    riak-users mailing list
    riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
    http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Urgent help with a down node.

Reply via email to