Upgraded riak 2.0.2-> 2.0.4
trying to add nodes to cluster( was single node )
all new nodes located on another host, but multi-nodes installation from
source
each new node have config settings tuned like
nodename = riak1@10.0.0.28
platform_data_dir = /var/lib/riak1
listener.http.internal = 0.0.0.0:18098
listener.protobuf.internal = 0.0.0.0:18087
handoff.port = 18099
search.solr.port = 18093
search.solr.jmx_port = 18985
search=on
after adding this nodes to cluster for long time was nothing was
transferred for new nodes.
I was down this nodes and force-remove it from cluster.
After this strange things happens.
On new nodes:
Even after restart cleaning entire data dir and start again, in folder yz
indexes somehow appears again.
logs contains only
2015-01-20 13:15:04.815 [info] <0.352.0>@riak_core:wait_for_service:483
Wait complete for service riak_kv (10 seconds)
2015-01-20 13:15:06.660 [info] <0.547.0>@yz_index:local_create:189 Created
index contacts_index with schema contacts
2015-01-20 13:15:08.683 [info] <0.547.0>@yz_index:local_create:189 Created
index users_index with schema users
Main Node now have bunch of errors on start and periodically after:
015-01-20 13:29:00.382 [error] <0.871.0>@riak_core_vnode:vnode_command:348
riak_kv_vnode command failed
{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_map,'-filter_unique/4-fun-1-',4,[{file,"src/ria
k_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{l
ine,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"orddict.erl"},{line,170}]}]}
2015-01-20 13:29:00.382 [error] <0.650.0>@riak_core_vnode:vnode_command:348
riak_kv_vnode command failed
{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_map,'-filter_unique/4-fun-1-',4,[{file,"src/ria
k_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{l
ine,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"orddict.erl"},{line,170}]}]}
2015-01-20 13:29:00.383 [error] <0.4520.0> gen_fsm <0.4520.0> in state
waiting_local_vnode terminated with reason: no case clause matching
{vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_
map,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_
dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}}
in riak_kv_put_fsm:waiting_local_vnode/2 line 5
67
2015-01-20 13:29:00.384 [error] <0.4520.0> CRASH REPORT Process <0.4520.0>
with 0 neighbours exited with reason: no case clause matching
{vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_ma
p,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt
_map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}}
in riak_kv_put_fsm:waiting_local_vnode/2 line 567
in gen_fsm:terminate/7 line 622
2015-01-20 13:29:00.384 [error] <0.4521.0> gen_fsm <0.4521.0> in state
waiting_local_vnode terminated with reason: no case clause matching
{vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_
map,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_
dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}}
in riak_kv_put_fsm:waiting_local_vnode/2 line 5
67
2015-01-20 13:29:00.384 [error] <0.4521.0> CRASH REPORT Process <0.4521.0>
with 0 neighbours exited with reason: no case clause matching
{vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_ma
p,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt
_map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}}
in riak_kv_put_fsm:waiting_local_vnode/2 line 567
in gen_fsm:terminate/7 line 622
2015-01-20 13:29:00.385 [error] <0.871.0> gen_fsm <0.871.0> in state active
terminated with reason: bad record dict in dict:filter_dict/2 line 464
2015-01-20 13:29:00.385 [error] <0.871.0> CRASH REPORT Process <0.871.0>
with 1 neighbours exited with reason: bad record dict in dict:filter_dict/2
line 464 in gen_fsm:terminate/7 line 622
2015-01-20 13:29:00.385 [error] <0.167.0> Supervisor riak_core_vnode_sup
had child undefined started with {riak_core_vnode,start_link,undefined} at
<0.871.0> exit with reason bad record dict in dict:filter_dict/2 line 464
in context
child_terminated
2015-01-20 13:29:00.385 [error] <0.988.0> gen_fsm <0.988.0> in state ready
terminated with reason: bad record dict in dict:filter_dict/2 line 464
2015-01-20 13:29:00.385 [error] <0.988.0> CRASH REPORT Process <0.988.0>
with 10 neighbours exited with reason: bad record dict in
dict:filter_dict/2 line 464 in gen_fsm:terminate/7 line 622
2015-01-20 13:29:00.385 [error] <0.650.0> gen_fsm <0.650.0> in state active
terminated with reason: bad record dict in dict:filter_dict/2 line 464
2015-01-20 13:29:00.386 [error] <0.650.0> CRASH REPORT Process <0.650.0>
with 1 neighbours exited with reason: bad record dict in dict:filter_dict/2
line 464 in gen_fsm:terminate/7 line 622
2015-01-20 13:29:00.386 [error] <0.989.0> Supervisor
{<0.989.0>,poolboy_sup} had child riak_core_vnode_worker started with
riak_core_vnode_worker:start_link([{worker_module,riak_core_vnode_worker},{worker_args,[593735040165679310520
246963290989976735222595584,...]},...]) at undefined exit with reason bad
record dict in dict:filter_dict/2 line 464 in context shutdown_error
2015-01-20 13:29:00.386 [error] <0.989.0> gen_server <0.989.0> terminated
with reason: bad record dict in dict:filter_dict/2 line 464
2015-01-20 13:29:00.386 [error] <0.989.0> CRASH REPORT Process <0.989.0>
with 0 neighbours exited with reason: bad record dict in dict:filter_dict/2
line 464 in gen_server:terminate/6 line 744
2015-01-20 13:29:00.386 [error] <0.167.0> Supervisor riak_core_vnode_sup
had child undefined started with {riak_core_vnode,start_link,undefined} at
<0.650.0> exit with reason bad record dict in dict:filter_dict/2 line 464
in context
child_terminated
2015-01-20 13:29:00.386 [error] <0.696.0> gen_fsm <0.696.0> in state ready
terminated with reason: bad record dict in dict:filter_dict/2 line 464
2015-01-20 13:29:00.386 [error] <0.696.0> CRASH REPORT Process <0.696.0>
with 10 neighbours exited with reason: bad record dict in
dict:filter_dict/2 line 464 in gen_fsm:terminate/7 line 622
2015-01-20 13:29:00.386 [error] <0.702.0> Supervisor
{<0.702.0>,poolboy_sup} had child riak_core_vnode_worker started with
riak_core_vnode_worker:start_link([{worker_module,riak_core_vnode_worker},{worker_args,[228359630832953580969
325755111919221821239459840,...]},...]) at undefined exit with reason bad
record dict in dict:filter_dict/2 line 464 in context shutdown_error
2015-01-20 13:29:00.386 [error] <0.702.0> gen_server <0.702.0> terminated
with reason: bad record dict in dict:filter_dict/2 line 464
2015-01-20 13:29:00.386 [error] <0.702.0> CRASH REPORT Process <0.702.0>
with 0 neighbours exited with reason: bad record dict in dict:filter_dict/2
line 464 in gen_server:terminate/6 line 744
any advice?
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
