Hi Dan,
Thanks for your reply.
Indeed, I saw the "insufficient_vnodes_available" error message with
only one node on the machine.
What we did was just install the official 1.0.2's deb version and
modifying vm.args and app.config.
The only changes in our setting are:
A. app.config:
- (http ip ..
- {pb_ib ..
- storage_backend points to storage_backend, riak_kv_eleveldb_backend
B. vm.args
-name
-setcookie
And the db ran smoothly for many days, then crashed for the 1st time.
It got recovered after we adjusted the ulimit.
Then it crashed again with the error message as above.
We don't have the other vnodes up-and-running in the machine at this
moment, though I was thinking it might be a solution and trying.
I tried to add an additional node and join the original node, but
fails. Then I leave it as it is.
Odd is, the DB comes back.
Here is our riak-admin info:
1. We don't have MapReduce procedures in it.
2. "#riak-admin status" shows:
Attempting to restart script through sudo -u riak
1-minute stats for 'riak@my-ip-address'
-------------------------------------------
vnode gets : 138
vnode_puts : 78
vnode_index_reads : 0
vnode_index_writes : 0
vnode_index_writes_postings : 0
vnode_index_deletes : 0
vnode_index_deletes_postings : 0
read_repairs : 1
vnode_gets_total : 490203
vnode_puts_total : 199436
vnode_index_reads_total : 0
vnode_index_writes_total : 0
vnode_index_writes_postings_total : 0
vnode_index_deletes_total : 0
vnode_index_deletes_postings_total : 0
node_gets : 42
node_gets_total : 163369
node_get_fsm_time_mean : 409
node_get_fsm_time_median : 397
node_get_fsm_time_95 : 674
node_get_fsm_time_99 : 734
node_get_fsm_time_100 : 748
node_puts : 24
node_puts_total : 66478
node_put_fsm_time_mean : 968
node_put_fsm_time_median : 1000
node_put_fsm_time_95 : 1599
node_put_fsm_time_99 : 1719
node_put_fsm_time_100 : 1735
node_get_fsm_siblings_mean : 1
node_get_fsm_siblings_median : 1
node_get_fsm_siblings_95 : 1
node_get_fsm_siblings_99 : 1
node_get_fsm_siblings_100 : 1
node_get_fsm_objsize_mean : 3609
node_get_fsm_objsize_median : 1280
node_get_fsm_objsize_95 : 29952
node_get_fsm_objsize_99 : 31385
node_get_fsm_objsize_100 : 31422
read_repairs_total : 1
coord_redirs_total : 0
cpu_nprocs : 441
cpu_avg1 : 51
cpu_avg5 : 33
cpu_avg15 : 54
mem_total : 8383463424
mem_allocated : 8280829952
disk : [{"/",724427316,3},
{"/dev",4088508,1},
{"/dev/shm",4093488,0},
{"/var/run",4093488,1},
{"/var/lock",4093488,0},
{"/lib/init/rw",4093488,0},
{"/var/lib/ureadahead/debugfs",724427316,3},
{"/boot",256665,20}]
nodename : 'riak@my-ip-address'
connected_nodes : []
sys_driver_version : <<"1.5">>
sys_global_heaps_size : 0
sys_heap_type : private
sys_logical_processors : 8
sys_otp_release : <<"R14B03">>
sys_process_count : 1225
sys_smp_support : true
sys_system_version : <<"Erlang R14B03 (erts-5.8.4) [source] [64-bit]
[smp:8:8] [rq:8] [async-threads:64] [kernel-poll:true]">>
sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
sys_threads_enabled : true
sys_thread_pool_size : 64
sys_wordsize : 8
ring_members : ['riak@my-ip-address']
ring_num_partitions : 64
ring_ownership : <<"[{'riak@my-ip-address',64}]">>
ring_creation_size : 64
storage_backend : riak_kv_eleveldb_backend
ssl_version : <<"4.1.5">>
public_key_version : <<"0.12">>
runtime_tools_version : <<"1.8.5">>
basho_stats_version : <<"1.0.1">>
riak_search_version : <<"1.0.2">>
merge_index_version : <<"1.0.1">>
luwak_version : <<"1.1.2">>
skerl_version : <<"1.1.0">>
riak_kv_version : <<"1.0.2">>
bitcask_version : <<"1.3.0">>
luke_version : <<"0.2.4">>
erlang_js_version : <<"1.0.0">>
mochiweb_version : <<"1.5.1">>
inets_version : <<"5.6">>
riak_pipe_version : <<"1.0.2">>
riak_core_version : <<"1.0.2">>
riak_sysmon_version : <<"1.0.0">>
webmachine_version : <<"1.9.0">>
crypto_version : <<"2.0.3">>
os_mon_version : <<"2.2.6">>
cluster_info_version : <<"1.2.0">>
sasl_version : <<"2.1.9.4">>
lager_version : <<"0.9.4">>
basho_metrics_version : <<"1.0.0">>
stdlib_version : <<"1.17.4">>
kernel_version : <<"2.14.4">>
executing_mappers : 0
memory_total : 1088568880
memory_processes : 1041629696
memory_processes_used : 1041593360
memory_system : 46939184
memory_atom : 1031585
memory_atom_used : 999738
memory_binary : 2593768
memory_code : 9301939
memory_ets : 656408
3. "# riak-admin ring_status" shows:
Attempting to restart script through sudo -u riak
================================== Claimant ===================================
Claimant: 'riak@my-ip-address'
Status: up
Ring Ready: true
============================== Ownership Handoff ==============================
No pending changes.
============================== Unreachable Nodes ==============================
All nodes are up and reachable
4. "# riak-admin member_status" shows
Attempting to restart script through sudo -u riak
================================= Membership ==================================
Status Ring Pending Node
-------------------------------------------------------------------------------
valid 100.0% -- 'riak@my-ip-address'
-------------------------------------------------------------------------------
Valid:1 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
Thanks,
Walter
2011/11/30 Dan Reverri <d...@basho.com>:
> Hi Walter,
> It is not recommended to run more than 1 node per machine. Also, creating a
> new node by copying/pasting an existing node is not a good idea.
> Were you seeing "insufficient vnodes" before you tried adding nodes?
> Was the node under any load? MapReduce, ListKeys, etc.
> What does "riak-admin status" return on the original node?
> What does "riak-admin ring_status" return on the original node?
> What does "riak-admin "member_status" return on the original node?
> I am concerned that the addition of copied nodes may be creating additional
> issues. I would recommend getting the cluster back to a steady state with a
> single node. Once this is complete we can review the issues and determine
> what the problem might be.
> Thanks,
> Dan
> Daniel Reverri
> Developer Advocate
> Basho Technologies, Inc.
> d...@basho.com
>
>
> On Mon, Nov 28, 2011 at 3:06 PM, Chung-Hung (Walter) Tsai
> <waltert...@gmail.com> wrote:
>>
>> Hi Dan,
>>
>> Yes, I use non-default name parameter and reip it.
>>
>> Please find my setting and setup info as below:
>>
>> Version: 1.0.2, .deb
>> Linux: Unbuntu 10.4, 64bits
>> Vnode: I didn't specify the other 2 nodes specifically, just used
>> default setting and did riak start.
>> The DB has a bumping due to insufficient file descriptor and got
>> resolved after I set ulimit and sysctl for system-wide parameters.
>> It ran ok for several days but since yesterday, it gave us the error
>> message regarding insufficient vnodes errors.
>>
>> I'm trying to setup 2 additional nodes by copying and modding
>> necessary files (e.g. /usr/sbin/riak to /usr/sbin/riakN) manually now.
>> But they each become a single cluster with just a single member in it
>> and doesn't allow other vnodes to join.
>> When I launch "riak-admin", it shows something like " ... 100% ...
>> riak@MY-IP-ADDRESS"
>>
>>
>> Thanks,
>> Walter
>>
>> 2011/11/29 Dan Reverri <d...@basho.com>:
>> > Hi Walter,
>> > Have you made any changes to your configuration recently such as
>> > changing
>> > the "-name" parameter in vm.args?
>> > Thanks,
>> > Dan
>> > Daniel Reverri
>> > Developer Advocate
>> > Basho Technologies, Inc.
>> > d...@basho.com
>> >
>> >
>> > On Mon, Nov 28, 2011 at 1:04 AM, Chung-Hung (Walter) Tsai
>> > <waltert...@gmail.com> wrote:
>> >>
>> >> Hello everyone,
>> >>
>> >> We just saw many errors like this:
>> >>
>> >> 2011-11-27 23:02:01.129 [error] <0.700.0> webmachine error:
>> >> path="/buckets/.. (SKIPPED).....ui"
>> >> {error,{error,insufficient_vnodes_available}}
>> >>
>> >> The DB went well for several days, but now our riak cannot be
>> >> connected from our code since the errors appear.
>> >> Does anyone know how to resolve the issue?
>> >>
>> >>
>> >> Thanks,
>> >> Walter
>> >>
>> >> _______________________________________________
>> >> riak-users mailing list
>> >> riak-users@lists.basho.com
>> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>> >
>
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
