Hey guys,
Sorry for the bump, but, I'm kind of at a loss on how to troubleshoot this
further. In addition to all the previously mentioned stuff, I have since
left the cluster on riak1, deleted that VM, completely reprovisioned it and
the exact same issues are occurring. Please let me know if there's anything
else I can provide to help anyone help me.
Thanks,
Andrew
On Sat, Nov 23, 2013 at 2:14 PM, Andrew Tynefield <atynefi...@gmail.com>wrote:
> Hey guys,
>
> I'm encountering a timeout when attempting a write using `riak-admin test`.
>
> (02:01:08) [riak1] ~ $ time riak-admin test
>
> Failed to write test value: {error,timeout}
>
> real 1m1.561s
> user 0m0.325s
> sys 0m0.103s
>
> (02:02:39) [riak1] ~ $
>
>
> This is one node of a 4 node cluster. The other nodes all work perfectly
> fine when attempting this. What I also find very weird is that this node
> was the second node to be created, nodes created after it (riak{2,3})
> complete their tests fine and do not appear to have any issues.
>
> vm.args and app.config are generated by puppet and are used with identical
> values (except for the node name) by all nodes.
>
> Values of note may be:
> {pb_backlog, 128},
> {ring_creation_size, 256},
>
>
> I've also tried leaving the cluster from this node, waiting for the ring
> to re-stabilize and show it successfully leaving. I then removed the entire
> /var/lib/riak/*, started riak back up, rejoined the cluster. (Prior to
> rejoining, I did a riak-admin test and it succeeded.)
>
> I've provided a tail of all the logs within /var/log/riak below
> immediately proceeding the execution of the riak-admin test command.
>
> ==> /var/log/riak/console.log <==
> 2013-11-23 13:45:53.597 [info]
> <0.157.0>@riak_core_capability:process_capability_changes:530 New
> capability: {riak_kv,anti_entropy} = enabled_v1
> 2013-11-23 13:45:53.798 [info]
> <0.157.0>@riak_core_capability:process_capability_changes:530 New
> capability: {riak_kv,handoff_data_encoding} = encode_raw
> 2013-11-23 13:45:54.187 [info]
> <0.157.0>@riak_core_capability:process_capability_changes:530 New
> capability: {riak_kv,object_format} = v1
> 2013-11-23 13:45:54.405 [info]
> <0.157.0>@riak_core_capability:process_capability_changes:530 New
> capability: {riak_kv,secondary_index_version} = v2
> 2013-11-23 13:45:54.683 [info]
> <0.157.0>@riak_core_capability:process_capability_changes:530 New
> capability: {riak_kv,vclock_data_encoding} = encode_zlib
> 2013-11-23 13:45:55.066 [info]
> <0.157.0>@riak_core_capability:process_capability_changes:530 New
> capability: {riak_kv,crdt} = [pncounter]
> 2013-11-23 13:45:56.141 [info]
> <0.157.0>@riak_core_capability:process_capability_changes:530 New
> capability: {riak_control,member_info_version} = v1
> 2013-11-23 13:45:56.277 [info] <0.7.0> Application riak_control started on
> node 'r...@riak1.tyne.io'
> 2013-11-23 13:45:56.277 [info] <0.7.0> Application erlydtl started on node
> 'r...@riak1.tyne.io'
> 2013-11-23 13:46:11.607 [info] <0.538.0>@riak_core:wait_for_service:464
> Wait complete for service riak_kv (14 seconds)
>
> ==> /var/log/riak/crash.log <==
> 2013-11-23 02:01:43 =ERROR REPORT====
> Hintfile
> '/var/lib/riak/bitcask/1444374665018431399630985401082889078019339583488/33.bitcask.hint'
> invalid
> 2013-11-23 02:01:43 =ERROR REPORT====
> Hintfile
> '/var/lib/riak/bitcask/1375866775768545325340187674549313311472967745536/29.bitcask.hint'
> invalid
> 2013-11-23 02:01:43 =ERROR REPORT====
> Hintfile
> '/var/lib/riak/bitcask/1124671181852296386273929343926202167469604339712/42.bitcask.hint'
> invalid
> 2013-11-23 02:01:43 =ERROR REPORT====
> Hintfile
> '/var/lib/riak/bitcask/1421538701935136041534052825571697155837215637504/30.bitcask.hint'
> invalid
> 2013-11-23 02:01:43 =ERROR REPORT====
> Hintfile
> '/var/lib/riak/bitcask/1330194849601954609146322523526929467108719853568/35.bitcask.hint'
> invalid
>
> ==> /var/log/riak/error.log <==
> 2013-11-23 02:01:43.869 [error] <0.2775.0> Hintfile
> '/var/lib/riak/bitcask/1353030812685249967243255099038121389290843799552/15.bitcask.hint'
> invalid
> 2013-11-23 02:01:43.869 [error] <0.2723.0> Hintfile
> '/var/lib/riak/bitcask/1261686960352068534855524796993353700562348015616/34.bitcask.hint'
> invalid
> 2013-11-23 02:01:43.869 [error] <0.2593.0> Hintfile
> '/var/lib/riak/bitcask/1170343108018887102467794494948586011833852231680/27.bitcask.hint'
> invalid
> 2013-11-23 02:01:43.869 [error] <0.2623.0> Hintfile
> '/var/lib/riak/bitcask/1216015034185477818661659645970969856198100123648/37.bitcask.hint'
> invalid
> 2013-11-23 02:01:43.870 [error] <0.2704.0> Hintfile
> '/var/lib/riak/bitcask/1238850997268773176758592221482161778380224069632/33.bitcask.hint'
> invalid
> 2013-11-23 02:01:43.870 [error] <0.2855.0> Hintfile
> '/var/lib/riak/bitcask/1444374665018431399630985401082889078019339583488/33.bitcask.hint'
> invalid
> 2013-11-23 02:01:43.870 [error] <0.2805.0> Hintfile
> '/var/lib/riak/bitcask/1375866775768545325340187674549313311472967745536/29.bitcask.hint'
> invalid
> 2013-11-23 02:01:43.870 [error] <0.2561.0> Hintfile
> '/var/lib/riak/bitcask/1124671181852296386273929343926202167469604339712/42.bitcask.hint'
> invalid
> 2013-11-23 02:01:43.870 [error] <0.2823.0> Hintfile
> '/var/lib/riak/bitcask/1421538701935136041534052825571697155837215637504/30.bitcask.hint'
> invalid
> 2013-11-23 02:01:43.871 [error] <0.2759.0> Hintfile
> '/var/lib/riak/bitcask/1330194849601954609146322523526929467108719853568/35.bitcask.hint'
> invalid
>
> ==> /var/log/riak/run_erl.log <==
> run_erl [2259] Sat Nov 23 13:45:49 2013
> Args before exec of shell:
> run_erl [2259] Sat Nov 23 13:45:49 2013
> argv[0] = sh
> run_erl [2259] Sat Nov 23 13:45:49 2013
> argv[1] = -c
> run_erl [2259] Sat Nov 23 13:45:49 2013
> argv[2] = exec /usr/sbin/riak console
>
>
>
> Here's some information about the installation and cluster:
>
> (02:04:36) [riak1] ~ $ riak version
> 1.4.2
>
> (02:08:07) [riak1] ~ $ riak-admin ringready
> TRUE All nodes agree on the ring ['r...@riak.tyne.io','r...@riak1.tyne.io
> ',
> 'r...@riak2.tyne.io','r...@riak3.tyne.io
> ']
>
> (02:07:31) [riak1] ~ $ riak-admin member-status
> ================================= Membership
> ==================================
> Status Ring Pending Node
>
> -------------------------------------------------------------------------------
> valid 25.0% -- 'r...@riak.tyne.io'
> valid 25.0% -- 'r...@riak1.tyne.io'
> valid 25.0% -- 'r...@riak2.tyne.io'
> valid 25.0% -- 'r...@riak3.tyne.io'
>
> -------------------------------------------------------------------------------
> Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
>
>
> Touch test:
>
> (02:07:00) [riak1] ~ $ for dir in /var/lib/riak/*/; do touch $dir/test;
> stat $dir/test; done
> File: `/var/lib/riak/anti_entropy//test'
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: 802h/2050d Inode: 1704796 Links: 1
> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2013-11-23 14:07:31.284244924 -0600
> Modify: 2013-11-23 14:07:31.284244924 -0600
> Change: 2013-11-23 14:07:31.284244924 -0600
> File: `/var/lib/riak/bitcask//test'
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: 802h/2050d Inode: 1705089 Links: 1
> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2013-11-23 14:07:31.310243834 -0600
> Modify: 2013-11-23 14:07:31.310243834 -0600
> Change: 2013-11-23 14:07:31.310243834 -0600
> File: `/var/lib/riak/kv_vnode//test'
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: 802h/2050d Inode: 1705150 Links: 1
> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2013-11-23 14:07:31.316243674 -0600
> Modify: 2013-11-23 14:07:31.316243674 -0600
> Change: 2013-11-23 14:07:31.316243674 -0600
> File: `/var/lib/riak/leveldb//test'
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: 802h/2050d Inode: 1705153 Links: 1
> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2013-11-23 14:07:31.322243576 -0600
> Modify: 2013-11-23 14:07:31.322243576 -0600
> Change: 2013-11-23 14:07:31.322243576 -0600
> File: `/var/lib/riak/ring//test'
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: 802h/2050d Inode: 1705154 Links: 1
> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2013-11-23 14:07:31.326243490 -0600
> Modify: 2013-11-23 14:07:31.326243490 -0600
> Change: 2013-11-23 14:07:31.326243490 -0600
>
>
> Any help would be greatly appreciated!
>
> Thanks,
> Andrew
>
> --
> [Andrew Tynefield]
>
--
[Andrew Tynefield]
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
