Hey guys,
I'm encountering a timeout when attempting a write using `riak-admin test`.
(02:01:08) [riak1] ~ $ time riak-admin test
Failed to write test value: {error,timeout}
real 1m1.561s
user 0m0.325s
sys 0m0.103s
(02:02:39) [riak1] ~ $
This is one node of a 4 node cluster. The other nodes all work perfectly
fine when attempting this. What I also find very weird is that this node
was the second node to be created, nodes created after it (riak{2,3})
complete their tests fine and do not appear to have any issues.
vm.args and app.config are generated by puppet and are used with identical
values (except for the node name) by all nodes.
Values of note may be:
{pb_backlog, 128},
{ring_creation_size, 256},
I've also tried leaving the cluster from this node, waiting for the ring to
re-stabilize and show it successfully leaving. I then removed the entire
/var/lib/riak/*, started riak back up, rejoined the cluster. (Prior to
rejoining, I did a riak-admin test and it succeeded.)
I've provided a tail of all the logs within /var/log/riak below immediately
proceeding the execution of the riak-admin test command.
==> /var/log/riak/console.log <==
2013-11-23 13:45:53.597 [info]
<0.157.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,anti_entropy} = enabled_v1
2013-11-23 13:45:53.798 [info]
<0.157.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,handoff_data_encoding} = encode_raw
2013-11-23 13:45:54.187 [info]
<0.157.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,object_format} = v1
2013-11-23 13:45:54.405 [info]
<0.157.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,secondary_index_version} = v2
2013-11-23 13:45:54.683 [info]
<0.157.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,vclock_data_encoding} = encode_zlib
2013-11-23 13:45:55.066 [info]
<0.157.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_kv,crdt} = [pncounter]
2013-11-23 13:45:56.141 [info]
<0.157.0>@riak_core_capability:process_capability_changes:530 New
capability: {riak_control,member_info_version} = v1
2013-11-23 13:45:56.277 [info] <0.7.0> Application riak_control started on
node 'r...@riak1.tyne.io'
2013-11-23 13:45:56.277 [info] <0.7.0> Application erlydtl started on node '
r...@riak1.tyne.io'
2013-11-23 13:46:11.607 [info] <0.538.0>@riak_core:wait_for_service:464
Wait complete for service riak_kv (14 seconds)
==> /var/log/riak/crash.log <==
2013-11-23 02:01:43 =ERROR REPORT====
Hintfile
'/var/lib/riak/bitcask/1444374665018431399630985401082889078019339583488/33.bitcask.hint'
invalid
2013-11-23 02:01:43 =ERROR REPORT====
Hintfile
'/var/lib/riak/bitcask/1375866775768545325340187674549313311472967745536/29.bitcask.hint'
invalid
2013-11-23 02:01:43 =ERROR REPORT====
Hintfile
'/var/lib/riak/bitcask/1124671181852296386273929343926202167469604339712/42.bitcask.hint'
invalid
2013-11-23 02:01:43 =ERROR REPORT====
Hintfile
'/var/lib/riak/bitcask/1421538701935136041534052825571697155837215637504/30.bitcask.hint'
invalid
2013-11-23 02:01:43 =ERROR REPORT====
Hintfile
'/var/lib/riak/bitcask/1330194849601954609146322523526929467108719853568/35.bitcask.hint'
invalid
==> /var/log/riak/error.log <==
2013-11-23 02:01:43.869 [error] <0.2775.0> Hintfile
'/var/lib/riak/bitcask/1353030812685249967243255099038121389290843799552/15.bitcask.hint'
invalid
2013-11-23 02:01:43.869 [error] <0.2723.0> Hintfile
'/var/lib/riak/bitcask/1261686960352068534855524796993353700562348015616/34.bitcask.hint'
invalid
2013-11-23 02:01:43.869 [error] <0.2593.0> Hintfile
'/var/lib/riak/bitcask/1170343108018887102467794494948586011833852231680/27.bitcask.hint'
invalid
2013-11-23 02:01:43.869 [error] <0.2623.0> Hintfile
'/var/lib/riak/bitcask/1216015034185477818661659645970969856198100123648/37.bitcask.hint'
invalid
2013-11-23 02:01:43.870 [error] <0.2704.0> Hintfile
'/var/lib/riak/bitcask/1238850997268773176758592221482161778380224069632/33.bitcask.hint'
invalid
2013-11-23 02:01:43.870 [error] <0.2855.0> Hintfile
'/var/lib/riak/bitcask/1444374665018431399630985401082889078019339583488/33.bitcask.hint'
invalid
2013-11-23 02:01:43.870 [error] <0.2805.0> Hintfile
'/var/lib/riak/bitcask/1375866775768545325340187674549313311472967745536/29.bitcask.hint'
invalid
2013-11-23 02:01:43.870 [error] <0.2561.0> Hintfile
'/var/lib/riak/bitcask/1124671181852296386273929343926202167469604339712/42.bitcask.hint'
invalid
2013-11-23 02:01:43.870 [error] <0.2823.0> Hintfile
'/var/lib/riak/bitcask/1421538701935136041534052825571697155837215637504/30.bitcask.hint'
invalid
2013-11-23 02:01:43.871 [error] <0.2759.0> Hintfile
'/var/lib/riak/bitcask/1330194849601954609146322523526929467108719853568/35.bitcask.hint'
invalid
==> /var/log/riak/run_erl.log <==
run_erl [2259] Sat Nov 23 13:45:49 2013
Args before exec of shell:
run_erl [2259] Sat Nov 23 13:45:49 2013
argv[0] = sh
run_erl [2259] Sat Nov 23 13:45:49 2013
argv[1] = -c
run_erl [2259] Sat Nov 23 13:45:49 2013
argv[2] = exec /usr/sbin/riak console
Here's some information about the installation and cluster:
(02:04:36) [riak1] ~ $ riak version
1.4.2
(02:08:07) [riak1] ~ $ riak-admin ringready
TRUE All nodes agree on the ring ['r...@riak.tyne.io','r...@riak1.tyne.io',
'r...@riak2.tyne.io','r...@riak3.tyne.io']
(02:07:31) [riak1] ~ $ riak-admin member-status
================================= Membership
==================================
Status Ring Pending Node
-------------------------------------------------------------------------------
valid 25.0% -- 'r...@riak.tyne.io'
valid 25.0% -- 'r...@riak1.tyne.io'
valid 25.0% -- 'r...@riak2.tyne.io'
valid 25.0% -- 'r...@riak3.tyne.io'
-------------------------------------------------------------------------------
Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
Touch test:
(02:07:00) [riak1] ~ $ for dir in /var/lib/riak/*/; do touch $dir/test;
stat $dir/test; done
File: `/var/lib/riak/anti_entropy//test'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 802h/2050d Inode: 1704796 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2013-11-23 14:07:31.284244924 -0600
Modify: 2013-11-23 14:07:31.284244924 -0600
Change: 2013-11-23 14:07:31.284244924 -0600
File: `/var/lib/riak/bitcask//test'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 802h/2050d Inode: 1705089 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2013-11-23 14:07:31.310243834 -0600
Modify: 2013-11-23 14:07:31.310243834 -0600
Change: 2013-11-23 14:07:31.310243834 -0600
File: `/var/lib/riak/kv_vnode//test'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 802h/2050d Inode: 1705150 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2013-11-23 14:07:31.316243674 -0600
Modify: 2013-11-23 14:07:31.316243674 -0600
Change: 2013-11-23 14:07:31.316243674 -0600
File: `/var/lib/riak/leveldb//test'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 802h/2050d Inode: 1705153 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2013-11-23 14:07:31.322243576 -0600
Modify: 2013-11-23 14:07:31.322243576 -0600
Change: 2013-11-23 14:07:31.322243576 -0600
File: `/var/lib/riak/ring//test'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 802h/2050d Inode: 1705154 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2013-11-23 14:07:31.326243490 -0600
Modify: 2013-11-23 14:07:31.326243490 -0600
Change: 2013-11-23 14:07:31.326243490 -0600
Any help would be greatly appreciated!
Thanks,
Andrew
--
[Andrew Tynefield]
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
