Hello Riak friends,
I'm rolling out a new Riak ring in EC2 on a set of m1.xlarges. I'm planning on
having six nodes. I wrote some automation that provisions the servers, installs
Riak, arranges the ring, etc., and after standing up three nodes I decided to
see how things are going.
Got Riak up and running on each machine just fine but there's something weird
in my logs that I don't quite get:
$ tail -n 5 /var/log/riak/error.log
2012-11-03 19:22:18.023 [error] <0.8981.0> CRASH REPORT Process <0.8981.0> with
0 neighbours exited with reason: no match of right hand value {error,timeout}
in riak_core_handoff_receiver:handle_call/3 line 59 in gen_server2:terminate/6
line 873
2012-11-03 19:22:18.030 [error] <0.158.0> Supervisor
riak_core_handoff_receiver_sup had child riak_core_handoff_receiver started
with {riak_core_handoff_receiver,start_link,undefined} at <0.8981.0> exit with
reason no match of right hand value {error,timeout} in
riak_core_handoff_receiver:handle_call/3 line 59 in context child_terminated
2012-11-03 19:22:23.028 [error] <0.8993.0> gen_server <0.8993.0> terminated
with reason:
{timeout,{gen_server2,call,[<0.9040.0>,{set_socket,#Port<0.8109>}]}}
2012-11-03 19:22:23.031 [error] <0.8993.0> CRASH REPORT Process
riak_core_handoff_listener with 1 neighbours exited with reason:
{timeout,{gen_server2,call,[<0.9040.0>,{set_socket,#Port<0.8109>}]}} in
gen_server:terminate/6 line 747
2012-11-03 19:22:23.035 [error] <0.160.0> Supervisor
riak_core_handoff_listener_sup had child riak_core_handoff_listener started
with riak_core_handoff_listener:start_link() at <0.8993.0> exit with reason
{timeout,{gen_server2,call,[<0.9040.0>,{set_socket,#Port<0.8109>}]}} in context
child_terminated
Otherwise, ring_status, member_status, and ring_ready return OK:
$ /usr/sbin/riak-admin ring_status
Attempting to restart script through sudo -H -u riak
================================== Claimant ===================================
Claimant: 'reporting_prod001@ my_private_ip1'
Status: up
Ring Ready: true
============================== Ownership Handoff ==============================
No pending changes.
============================== Unreachable Nodes ==============================
All nodes are up and reachable
$ /usr/sbin/riak-admin member_status
Attempting to restart script through sudo -H -u riak
================================= Membership ==================================
Status Ring Pending Node
-------------------------------------------------------------------------------
valid 33.6% -- 'reporting_prod001@my_private_ip1'
valid 33.2% -- 'reporting_prod002@ my_private_ip2'
valid 33.2% -- 'reporting_prod003@ my_private_ip3'
-------------------------------------------------------------------------------
Valid:3 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
$ /usr/sbin/riak-admin ringready
Attempting to restart script through sudo -H -u riak
TRUE All nodes agree on the ring ['reporting_prod001@ my_private_ip1',
'reporting_prod002@ my_private_ip2',
'reporting_prod003@ my_private_ip3']
Additionally, if I try to curl either my ELB from my machine or the private IP
from one of the cluster members, I get back the content I would (mostly) expect:
$ curl --insecure https://my-elb.aws.amazon.com:8080/
<html><body><ul><li><a href="/admin">admin_cluster</a></li><li><a
href="/admin">admin_cluster_down</a></li><li><a
href="/admin">admin_cluster_join</a></li><li><a
href="/admin">admin_fallbacks</a></li><li><a
href="/admin">admin_gui</a></li><li><a href="/admin">admin_node</a></li><li><a
href="/admin">admin_node_leave</a></li><li><a
href="/admin">admin_node_stop</a></li><li><a
href="/admin">admin_overview</a></li><li><a
href="/admin">admin_ring</a></li><li><a
href="/buckets">riak_kv_wm_buckets</a></li><li><a
href="/riak">riak_kv_wm_buckets</a></li><li><a
href="/buckets">riak_kv_wm_index</a></li><li><a
href="/buckets">riak_kv_wm_keylist</a></li><li><a
href="/buckets">riak_kv_wm_link_walker</a></li><li><a
href="/riak">riak_kv_wm_link_walker</a></li><li><a
href="/mapred">riak_kv_wm_mapred</a></li><li><a
href="/buckets">riak_kv_wm_object</a></li><li><a
href="/riak">riak_kv_wm_object</a></li><li><a
href="/ping">riak_kv_wm_ping</a></li><li><a
href="/buckets">riak_kv_wm_props</a></li><li><a
href="/stats">riak_kv_wm_stats</a></li></ul></body></html>
So…it appears as if everything is OK. But what are the log messages about…?
Something to do with our ELB maybe?
Cheers,
Brad Heller | Engineering Lead | Cloudability.com | 541-231-1514 | Skype:
brad.heller | @bradhe | @cloudability
We're hiring! http://cloudability.com/jobs
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
