Hi Bryan,
thanks for this. I tried it but to be honest I cannot see any specific
stuff in the logs (on the specific host).
I attached the logfile from the specific node. If you think it is
also/more important to look into the logfiles on the other nodes I can
send them as well.. but a quick look into all of them (searching for
"2i" and "index") didn't show anything unusual.. the only stuff was
2014-08-07 05:44:11.298 UTC [debug]
<0.969.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=633697975561446187189878970435575840553939501056
(preflist)={610862012478150829092946394924383918371815555072,12}
and searching for errors didn't show more than you see in the attached
files:
$ for host in kriak46-{1..7} kriak47-{1..6}; do echo $host; ssh $host "grep
'^2014-08-07 05' /var/log/riak/console.log | grep -i error" ; done
kriak46-1
2014-08-07 05:38:28.596 UTC [error] <0.8949.566> ** Node
'c_24556_riak@10.46.109.201' not responding **
2014-08-07 05:42:36.197 UTC [error] <0.24823.566> ** Node
'c_26945_riak@10.46.109.201' not responding **
2014-08-07 05:43:16.213 UTC [error] <0.26434.566> ** Node
'c_27071_riak@10.46.109.201' not responding **
2014-08-07 05:48:14.284 UTC [error] <0.1697.0> gen_server <0.1697.0> terminated
with reason: bad argument in call to eleveldb:async_write(#Ref<0.0.567.170046>,
<<>>,
[{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
[]) in eleveldb:write/3 line 155
2014-08-07 05:48:14.284 UTC [error] <0.1697.0> CRASH REPORT Process <0.1697.0>
with 0 neighbours exited with reason: bad argument in call to
eleveldb:async_write(#Ref<0.0.567.170046>, <<>>,
[{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
[]) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747
2014-08-07 05:48:14.284 UTC [error] <0.1692.0> Supervisor
{<0.1692.0>,poolboy_sup} had child riak_core_vnode_worker started with
{riak_core_vnode_worker,start_link,undefined} at <0.1697.0> exit with reason
bad argument in call to eleveldb:async_write(#Ref<0.0.567.170046>, <<>>,
[{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
[]) in eleveldb:write/3 line 155 in context child_terminated
2014-08-i7 05:50:11.390 UTC [error] <0.20983.567> ** Node
'c_32188_riak@10.46.109.201' not responding **
kriak46-2
kriak46-3
kriak46-4
kriak46-5
kriak46-6
kriak46-7
kriak47-1
kriak47-2
kriak47-3
kriak47-4
kriak47-5
kriak47-6
You mentioned the partition repair stuff.. do you think I need to try
out the full repair? Is this maybe a way to fix it? Because it is quiet
hard to do this on the cluster (~15 TB of data with AAE stuff and
tombstones and maybe ~10 TB without tombstones and AAE stuff) and I
don't want to start doing this if it won't help.
Cheers
Simon
On Wed, Aug 06, 2014 at 01:08:36PM +0100, bryan hunt wrote:
> Simon,
>
> If you want to get more verbose logging information, you could perform the
> following to change the logging level, to debug, then run `repair-2i`, and
> finally switching back to the normal logging level.
>
> - `riak attach`
> - `(riak@nodename)1> SetDebug = fun() -> {node(),
> lager:set_loglevel(lager_file_backend, "/var/log/riak/console.log", debug)}
> end.`
> - `(riak@nodename)2> rp(rpc:multicall(erlang, apply, [SetDebug,[]])).`
> (don't forget the period at the end of these statements)
> - Hit CTRL+C twice to quit from the node
>
> You can then revert back to the normal `info` logging level by running the
> following command via `riak attach`:
>
> - `riak attach`
> - `(riak@nodename)1> SetInfo = fun() -> {node(),
> lager:set_loglevel(lager_file_backend, "/var/log/riak/console.log", info)}
> end.`
> - `(riak@nodename)2> rp(rpc:multicall(erlang, apply, [SetInfo,[]])).`
> (don't forget the period at the end of these statements)
> - Hit CTRL+C twice to quit from a the node
>
> Please also see the docs for info on `riak attach` monitoring of repairs.
>
> http://docs.basho.com/riak/1.4.9/ops/running/recovery/repairing-partitions/#Monitoring-Repairs
>
> Repairs can also be monitored using the `riak-admin transfers` command.
>
> http://docs.basho.com/riak/1.4.9/ops/running/recovery/repairing-partitions/#Running-a-Repair
>
> Best Regards,
>
> Bryan Hunt
>
> Bryan Hunt - Client Services Engineer - Basho Technologies Limited -
> Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
>
2014-08-07 05:43:04.469 UTC [notice] <0.57.0>@lager_file_backend:128 Changed
loglevel of /var/log/riak/console.log to debug
2014-08-07 05:43:14.156 UTC [info] <0.27496.566>@riak_kv_2i_aae:init:139
Starting 2i repair at speed 100 for partitions
[319703483166135013357056057156686910549735243776]
2014-08-07 05:43:14.156 UTC [info]
<0.27497.566>@riak_kv_2i_aae:repair_partition:257 Acquired lock on partition
319703483166135013357056057156686910549735243776
2014-08-07 05:43:14.156 UTC [info]
<0.27497.566>@riak_kv_2i_aae:repair_partition:259 Repairing indexes in
partition 319703483166135013357056057156686910549735243776
2014-08-07 05:43:14.239 UTC [info]
<0.27497.566>@riak_kv_2i_aae:create_index_data_db:324 Creating temporary
database of 2i data in /var/lib/riak/anti_entropy/2i/tmp_db
2014-08-07 05:43:14.283 UTC [info]
<0.27497.566>@riak_kv_2i_aae:create_index_data_db:361 Grabbing all index data
for partition 319703483166135013357056057156686910549735243776
2014-08-07 05:43:14.289 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue:
{822094670998632891489572718402909198556462055424,879184578706871286731904157180889004011771920384,{822094670998632891489572718402909198556462055424,12}}
2014-08-07 05:43:16.213 UTC [error] <0.26434.566> ** Node
'c_27071_riak@10.46.109.201' not responding **
** Removing (timedout) connection **
2014-08-07 05:43:29.290 UTC [debug] <0.28227.566>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:43:29.290 UTC [debug]
<0.28227.566>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:43:29.290 UTC [debug]
<0.28227.566>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:43:29.290 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={833512652540280570538039006158505159647524028416,12}
2014-08-07 05:43:30.633 UTC [debug]
<0.28227.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:43:30.633 UTC [debug]
<0.28227.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:43:30.633 UTC [debug]
<0.28227.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {833512652540280570538039006158505159647524028416,12}
2014-08-07 05:43:33.673 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:43:44.291 UTC [debug] <0.29028.566>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:43:44.291 UTC [debug]
<0.29028.566>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:43:44.291 UTC [debug]
<0.29028.566>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:43:44.291 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={839221643311104410062272150036303140193055014912,12}
2014-08-07 05:43:45.929 UTC [debug]
<0.29028.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:43:45.929 UTC [debug]
<0.29028.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:43:45.929 UTC [debug]
<0.29028.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {839221643311104410062272150036303140193055014912,12}
2014-08-07 05:43:49.312 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:43:59.292 UTC [debug] <0.29871.566>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:43:59.292 UTC [debug]
<0.29871.566>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:43:59.292 UTC [debug]
<0.29871.566>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:43:59.292 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={844930634081928249586505293914101120738586001408,12}
2014-08-07 05:43:59.822 UTC [debug]
<0.29871.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:43:59.822 UTC [debug]
<0.29871.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:43:59.822 UTC [debug]
<0.29871.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {844930634081928249586505293914101120738586001408,12}
2014-08-07 05:44:02.535 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:44:14.293 UTC [debug] <0.30743.566>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:44:14.293 UTC [debug]
<0.30743.566>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:44:14.293 UTC [debug]
<0.30743.566>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:44:14.293 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={850639624852752089110738437791899101284116987904,12}
2014-08-07 05:44:14.669 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
822094670998632891489572718402909198556462055424 :: normal
2014-08-07 05:44:22.126 UTC [debug]
<0.30743.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:44:22.126 UTC [debug]
<0.30743.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:44:22.126 UTC [debug]
<0.30743.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {850639624852752089110738437791899101284116987904,12}
2014-08-07 05:44:25.055 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:44:29.294 UTC [debug] <0.31540.566>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:44:29.294 UTC [debug]
<0.31540.566>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:44:29.294 UTC [debug]
<0.31540.566>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:44:29.294 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={856348615623575928634971581669697081829647974400,12}
2014-08-07 05:44:30.567 UTC [debug]
<0.31540.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:44:30.568 UTC [debug]
<0.31540.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:44:30.568 UTC [debug]
<0.31540.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {856348615623575928634971581669697081829647974400,12}
2014-08-07 05:44:33.509 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:44:44.295 UTC [debug] <0.32261.566>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:44:44.295 UTC [debug]
<0.32261.566>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:44:44.295 UTC [debug]
<0.32261.566>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:44:44.295 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={862057606394399768159204725547495062375178960896,12}
2014-08-07 05:44:52.849 UTC [debug]
<0.32261.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:44:52.849 UTC [debug]
<0.32261.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:44:52.849 UTC [debug]
<0.32261.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {862057606394399768159204725547495062375178960896,12}
2014-08-07 05:44:55.580 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:44:59.296 UTC [debug] <0.281.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:44:59.296 UTC [debug]
<0.281.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:44:59.296 UTC [debug]
<0.281.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:44:59.296 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={884893569477695126256137301058686984557302906880,3}
2014-08-07 05:45:09.895 UTC [debug]
<0.281.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:45:09.895 UTC [debug]
<0.281.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'} and
{896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:45:09.895 UTC [debug]
<0.281.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {884893569477695126256137301058686984557302906880,3}
2014-08-07 05:45:11.747 UTC [info]
<0.281.567>@riak_kv_exchange_fsm:key_exchange:206 Repaired 2 keys during active
anti-entropy exchange of {884893569477695126256137301058686984557302906880,3}
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:45:11.747 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:45:14.297 UTC [debug] <0.1111.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:45:14.297 UTC [debug]
<0.1111.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:45:14.297 UTC [debug]
<0.1111.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:45:14.297 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={890602560248518965780370444936484965102833893376,3}
2014-08-07 05:45:24.617 UTC [debug]
<0.1111.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:45:24.617 UTC [debug]
<0.1111.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:45:24.617 UTC [debug]
<0.1111.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {890602560248518965780370444936484965102833893376,3}
2014-08-07 05:45:26.307 UTC [debug]
<0.1404.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=685078892498860742907977265335757665463718379520
(preflist)={673660910957213063859510977580161704372656406528,3}
2014-08-07 05:45:29.298 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue:
{890602560248518965780370444936484965102833893376,913438523331814323877303020447676887284957839360,{884893569477695126256137301058686984557302906880,12}}
2014-08-07 05:45:29.964 UTC [info]
<0.1111.567>@riak_kv_exchange_fsm:key_exchange:206 Repaired 1 keys during
active anti-entropy exchange of
{890602560248518965780370444936484965102833893376,3} between
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'} and
{896311551019342805304603588814282945648364879872,'riak@10.46.109.202'}
2014-08-07 05:45:29.964 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:45:44.299 UTC [debug] <0.2988.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:45:44.299 UTC [debug]
<0.2988.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:45:44.299 UTC [debug]
<0.2988.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{913438523331814323877303020447676887284957839360,'riak@10.46.109.205'}
2014-08-07 05:45:44.299 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={890602560248518965780370444936484965102833893376,12}
2014-08-07 05:45:48.779 UTC [debug]
<0.2988.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:45:48.779 UTC [debug]
<0.2988.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {913438523331814323877303020447676887284957839360,'riak@10.46.109.205'}
2014-08-07 05:45:48.779 UTC [debug]
<0.2988.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {890602560248518965780370444936484965102833893376,12}
2014-08-07 05:45:56.046 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:45:59.300 UTC [debug] <0.4233.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:45:59.300 UTC [debug]
<0.4233.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:45:59.300 UTC [debug]
<0.4233.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{919147514102638163401536164325474867830488825856,'riak@10.46.109.206'}
2014-08-07 05:45:59.300 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={856348615623575928634971581669697081829647974400,12}
2014-08-07 05:46:01.552 UTC [debug]
<0.4233.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:46:01.552 UTC [debug]
<0.4233.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {919147514102638163401536164325474867830488825856,'riak@10.46.109.206'}
2014-08-07 05:46:01.552 UTC [debug]
<0.4233.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {856348615623575928634971581669697081829647974400,12}
2014-08-07 05:46:06.327 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:46:14.301 UTC [debug] <0.5306.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:46:14.301 UTC [debug]
<0.5306.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:46:14.301 UTC [debug]
<0.5306.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{919147514102638163401536164325474867830488825856,'riak@10.46.109.206'}
2014-08-07 05:46:14.301 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={862057606394399768159204725547495062375178960896,12}
2014-08-07 05:46:15.236 UTC [debug]
<0.5306.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:46:15.236 UTC [debug]
<0.5306.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {919147514102638163401536164325474867830488825856,'riak@10.46.109.206'}
2014-08-07 05:46:15.236 UTC [debug]
<0.5306.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {862057606394399768159204725547495062375178960896,12}
2014-08-07 05:46:20.366 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:46:29.302 UTC [debug] <0.6563.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:46:29.302 UTC [debug]
<0.6563.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:46:29.302 UTC [debug]
<0.6563.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{919147514102638163401536164325474867830488825856,'riak@10.46.109.206'}
2014-08-07 05:46:29.302 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={873475587936047447207671013303091023466240933888,12}
2014-08-07 05:46:44.303 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue:
{890602560248518965780370444936484965102833893376,919147514102638163401536164325474867830488825856,{879184578706871286731904157180889004011771920384,12}}
2014-08-07 05:46:52.665 UTC [debug]
<0.6563.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:46:52.665 UTC [debug]
<0.6563.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {919147514102638163401536164325474867830488825856,'riak@10.46.109.206'}
2014-08-07 05:46:52.665 UTC [debug]
<0.6563.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {873475587936047447207671013303091023466240933888,12}
2014-08-07 05:46:57.385 UTC [info]
<0.6563.567>@riak_kv_exchange_fsm:key_exchange:206 Repaired 13 keys during
active anti-entropy exchange of
{873475587936047447207671013303091023466240933888,12} between
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'} and
{919147514102638163401536164325474867830488825856,'riak@10.46.109.206'}
2014-08-07 05:46:57.385 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:46:59.304 UTC [debug] <0.9160.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:46:59.315 UTC [debug]
<0.9160.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:46:59.315 UTC [debug]
<0.9160.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{919147514102638163401536164325474867830488825856,'riak@10.46.109.206'}
2014-08-07 05:46:59.315 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={890602560248518965780370444936484965102833893376,12}
2014-08-07 05:46:59.368 UTC [info]
<0.588.0>@riak_kv_entropy_manager:perhaps_log_throttle_change:826 Changing AAE
throttle from 0 -> 10 msec/key, based on maximum vnode mailbox size 246 from
'riak@10.47.109.202'
2014-08-07 05:47:00.465 UTC [debug]
<0.9160.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:47:00.465 UTC [debug]
<0.9160.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {919147514102638163401536164325474867830488825856,'riak@10.46.109.206'}
2014-08-07 05:47:00.465 UTC [debug]
<0.9160.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {890602560248518965780370444936484965102833893376,12}
2014-08-07 05:47:05.461 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:47:14.305 UTC [debug] <0.10096.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:47:14.312 UTC [debug]
<0.10096.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:47:14.312 UTC [info]
<0.588.0>@riak_kv_entropy_manager:perhaps_log_throttle_change:826 Changing AAE
throttle from 10 -> 0 msec/key, based on maximum vnode mailbox size 1 from
'riak@10.47.109.206'
2014-08-07 05:47:14.312 UTC [debug]
<0.10096.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{924856504873462002925769308203272848376019812352,'riak@10.47.109.206'}
2014-08-07 05:47:14.312 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={862057606394399768159204725547495062375178960896,12}
2014-08-07 05:47:15.540 UTC [debug]
<0.10096.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:47:15.540 UTC [debug]
<0.10096.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {924856504873462002925769308203272848376019812352,'riak@10.47.109.206'}
2014-08-07 05:47:15.540 UTC [debug]
<0.10096.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {862057606394399768159204725547495062375178960896,12}
2014-08-07 05:47:29.306 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue:
{890602560248518965780370444936484965102833893376,924856504873462002925769308203272848376019812352,{873475587936047447207671013303091023466240933888,12}}
2014-08-07 05:47:44.307 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue:
{890602560248518965780370444936484965102833893376,924856504873462002925769308203272848376019812352,{879184578706871286731904157180889004011771920384,12}}
2014-08-07 05:47:46.669 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:47:59.308 UTC [debug] <0.13725.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:47:59.315 UTC [debug]
<0.13725.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
2014-08-07 05:47:59.315 UTC [debug]
<0.13725.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{924856504873462002925769308203272848376019812352,'riak@10.47.109.206'}
2014-08-07 05:47:59.315 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={884893569477695126256137301058686984557302906880,12}
2014-08-07 05:48:01.948 UTC [debug]
<0.13725.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:48:01.948 UTC [debug]
<0.13725.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'riak@10.46.109.201'}
and {924856504873462002925769308203272848376019812352,'riak@10.47.109.206'}
2014-08-07 05:48:01.948 UTC [debug]
<0.13725.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {884893569477695126256137301058686984557302906880,12}
2014-08-07 05:48:14.284 UTC [error] <0.1697.0> gen_server <0.1697.0> terminated
with reason: bad argument in call to eleveldb:async_write(#Ref<0.0.567.170046>,
<<>>,
[{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
[]) in eleveldb:write/3 line 155
2014-08-07 05:48:14.284 UTC [info]
<0.27496.566>@riak_kv_2i_aae:next_partition:160 Finished 2i repair:
Total partitions: 1
Finished partitions: 1
Speed: 100
Total 2i items scanned: 0
Total tree objects: 0
Total objects fixed: 0
With errors:
Partition: 319703483166135013357056057156686910549735243776
Error: index_scan_timeout
2014-08-07 05:48:14.284 UTC [error] <0.1697.0> CRASH REPORT Process <0.1697.0>
with 0 neighbours exited with reason: bad argument in call to
eleveldb:async_write(#Ref<0.0.567.170046>, <<>>,
[{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
[]) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747
2014-08-07 05:48:14.284 UTC [error] <0.1692.0> Supervisor
{<0.1692.0>,poolboy_sup} had child riak_core_vnode_worker started with
{riak_core_vnode_worker,start_link,undefined} at <0.1697.0> exit with reason
bad argument in call to eleveldb:async_write(#Ref<0.0.567.170046>, <<>>,
[{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
[]) in eleveldb:write/3 line 155 in context child_terminated
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
