You'll get insufficient vnodes messages when not enough riak nodes are up and running. The 'riak start' command returns before the node is servicing requests. In your log you'll see something similar to this
2013-08-19 15:33:59.337 [info] <0.7.0> Application riak_kv started on node
'c1n1@127.0.0.1'
Is there a chance you ran your test soon after a restart? If you have a
large amount of data written into bitcask, it can take several minutes for
everything to start up.
Jon
On Tue, Aug 20, 2013 at 6:09 PM, Idan Shinberg <idan.shinb...@idomoo.com>wrote:
> One small update -
>
> I've tried GET operations
>
> yet they seldom fail ( again , I'm guessing on fails that have blocks on
> the missing mode )
>
> for example
>
> *s3cmd --force get s3://riak.test/test/21*
> *s3://riak.test/test/21 -> ./21 [1 of 1]*
> * 3145728 of 9437184 33% in 0s 24.33 MB/s failed*
> *WARNING: Retrying failed request: /test/21 (string indices must be
> integers, not str)*
> *WARNING: Waiting 3 sec...*
> *s3://riak.test/test/21 -> ./21 [1 of 1]*
> * 3145728 of 9437184 33% in 0s 0.00 B/s failed*
> *WARNING: Retrying failed request: /test/21 (string indices must be
> integers, not str)*
> *WARNING: Waiting 6 sec...*
>
> logs the following message :
>
> *2013-08-21 00:02:35.768 [error]
> <0.741.0>@riak_cs_get_fsm:waiting_chunks:311 riak_cs_get_fsm: Cannot get S3
> <<"riak.test">> <<"test/21">> block#
> {<<212,253,202,96,63,227,78,187,153,104,106,213,103,21,228,199>>,3}:
> {error,notfound}*
> *2013-08-21 00:02:35.768 [error] <0.299.0> CRASH REPORT Process <0.299.0>
> with 0 neighbours exited with reason:
> {normal,{gen_fsm,sync_send_event,[<0.741.0>,get_next_chunk,infinity]}} in
> gen_fsm:sync_send_event/3 line 214*
>
> Could it be that Riak is not replicating data coming from Riak-CS ?
>
>
> Regards,
>
> Idan Shinberg
>
>
> System Architect
>
> Idomoo Ltd.
>
>
>
> Mob +972.54.562.2072
>
> email idan.shinb...@idomoo.com
>
> web www.idomoo.com
>
> [image: Description: cid:FC66336E-E750-4C2D-B6E3-985D5A06B5BE@idomoo.co.il]
>
>
> On Wed, Aug 21, 2013 at 2:47 AM, Idan Shinberg
> <idan.shinb...@idomoo.com>wrote:
>
>> Hi there riak users
>>
>> I've been running some tests on riak-cs on a 3 node riak cluster ( 3
>> servers , each host riak and riak-cs pointing at the local riak . One of
>> the instance is also hosting the stanchion )
>> Software versions :
>> - Riak is 1.4.1
>> - Riak-CS is 1.4.0
>> - s3cmd is 1.5.0 alpha 3
>>
>> n_val was either set to be 2 or 3 ( tried both )for default bucket ,
>> yet whenever I kill a single riak node ( simulate a server outage )
>> I come across the following error when try to use s3cmd get path:
>> *
>> *
>> *s3cmd ls s3://riak.test/test/1*
>> *
>> *
>> *WARNING: Retrying failed request: /?prefix=test/1&delimiter=/ ()*
>> *WARNING: Waiting 3 sec...*
>> *WARNING: Retrying failed request: /?prefix=test/1&delimiter=/ ()*
>> *WARNING: Waiting 6 sec...*
>> *
>> *
>> Looking into Riak-CS error logs , I find these errors
>> *
>> *
>> *
>> 2013-08-20 23:39:40.240 [error] <0.14674.0> gen_fsm <0.14674.0> in state
>> waiting_object_list terminated with reason:
>> <<"{error,insufficient_vnodes_available}">>
>> 2013-08-20 23:39:40.240 [error] <0.14674.0> CRASH REPORT Process
>> <0.14674.0> with 1 neighbours exited with reason:
>> <<"{error,insufficient_vnodes_available}">> in gen_fsm:terminate/7 line 611
>>
>> This of course implies of missing v_nodes found on the single dead riak
>> node... :/
>>
>> *
>> Writing to riak using s3cmd keeps working flawlessly . And of course
>> after I revive the dead node
>> misplaced data is re-distributed ( ring of vnodes are redistributed to
>> reflect requested distribution ).
>>
>> I might be missing something here , but ain't the whole point of
>> replication is to avoid such issues with a single node failure ? ( assuming
>> n_val is greater the 1 )
>> Isn't a 3 Node Cluster with n_val =>2 supposed to withstand a single node
>> failure ?
>> Have I missed something in configuring Riak-CS ?
>>
>> Any help will be appreciated , this is really the final missing piece of
>> our RiAK puzzle :)
>>
>> Regards,
>>
>> Idan Shinberg
>>
>>
>> System Architect
>>
>> Idomoo Ltd.
>>
>>
>>
>> Mob +972.54.562.2072
>>
>> email idan.shinb...@idomoo.com
>>
>> web www.idomoo.com
>>
>> [image: Description:
>> cid:FC66336E-E750-4C2D-B6E3-985D5A06B5BE@idomoo.co.il]
>>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
--
Jon Meredith
VP, Engineering
Basho Technologies, Inc.
jmered...@basho.com
<<image001.jpg>>
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
