Absolute consistency
Hello, Riak is based on eventual consistency concept, but sometimes it's desirable to have absolute consistency. I though it would be possible to implement this on top of Riak with the help of pr and wr parameters of fetch and store, but when testing this I sometimes get worrying results. I have at test setup with Riak nodes in three virtual machines running on my desktop machine. The test bucket has n_val=3 and allow_mult=true. On my desktop machine I constantly read from every node like this: watch curl -v "'http://192.168.100.21:8098/riak/test/foo?pr=quorum'" When I pause two of the virtual machines, the remaining one starts to fail with error message "PR-value unsatisfied: 1/2", just as expected. But occasionally it will returns 200 OK for about 10 seconds, before it starts returning "PR-value unsatisfied: 1/2". I can't predictably reproduce this situation, but it's seems to happen when I resume a machine that was paused when the other machines where running too. Pausing virtual machines to simulate downtime might be a bit unrealistic. Still it concerns me that Riak can return stale data. Here is an example: First I wrote a value when all nodes where up. Then I pause one of the nodes and wrote a new value. Then I paused the two remaining nodes and resume the first node. For a few seconds the first node then returned 200 OK with the old value when though I supplied pr=quorum. I have also been able to do successful writes with w, dw and pw=quorum with only one machine running. I thing the node also returned 200 OK when the object was read with pr=quorum right after. The tests where done with Riak 1.0.0-1 running on Ubuntu 10.10 64 bit. -- Thomas Bakketun, Senior software developer T. +47 21 53 69 40 Copyleft Solutions www.copyleftsolutions.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Absolute consistency
On 2012-01-05 22:52, Tim Robinson wrote: > On a side note, it looks like we've completely highjacked the > "Absolute consistency" question initially proposed. Yes, the answers so far doesn't explain the behaviour I have observed. If that particular key would happen to have two of it's primary replica on the same node, then that single node would be able to answer fetch request with pw=quorum even if it's the only node currently running. But that node would also be a have to be online for pw=quorum read of the key to be successful. This not what I observed, quorum reads are successful when any two of the three nodes are online. Also the surprising results I get only last for a few seconds, and it's not one particular node that gives the surprising results. Another thing I observed. Sometimes the vclock of an object starts to flap between two different values. The object value is the same. Here's an example: a85hYGBgzmDKBVIc6TXMFn6HL3ZmMCUxMLC65bEyzM6pPsEHlRWoNl7nd/jyeohsLVB2JZKsoK2Av9+qLz4Q2UNA2U8g2SwA a85hYGBgzmDKBVIcAtXG6/wOX16fwZTEwMBam8fKsDKn+gQfVFbQVsDfb9UXH4jsIaDsJyTZ9BpmC7/DFzshsm5A2dkg2SwA No writes are done when this happens. One node always returns the first vclock, while the two others occupationally returns the second one. I have not supplied any client-id when writing to the key. -- Thomas Bakketun, Senior software developer T. +47 21 53 69 40 Copyleft Solutions www.copyleftsolutions.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Corruption of leveldb database files
Hello,
We have found a bug in leveldb that causes corruption of the CURRENT
file of leveldb databases. A bug report has been filed:
http://code.google.com/p/leveldb/issues/detail?id=68
This bug can easily be reproduced by restarting restarting Riak and
force of the computer within 30 to 60 seconds. After reboot at least one
corrupt leveldb database can be expected. When this happens Riak fails
to start and logs this message:
Failed to start riak_kv_eleveldb_backend Reason: {db_open,"Corruption:
CURRENT file does not end with newline"}
Switching on the sync flag for leveldb has no effect.
--
Thomas Bakketun, Senior software developer
T. +47 21 53 69 40
Copyleft Solutions
www.copyleftsolutions.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Corruption of leveldb database files
Hello,
A fix for this bug in leveldb has been released now. I would recommend
that a service release of Riak with this fix included is done as soon as
possible.
On 2012-01-16 17:52, David Smith wrote:
> Thomas,
>
> Thanks for the heads up -- I saw that bug report on the Level repo this
> morning.
>
> D.
>
> On Mon, Jan 16, 2012 at 9:50 AM, Thomas Bakketun
> wrote:
>> Hello,
>>
>> We have found a bug in leveldb that causes corruption of the CURRENT
>> file of leveldb databases. A bug report has been filed:
>> http://code.google.com/p/leveldb/issues/detail?id=68
>>
>> This bug can easily be reproduced by restarting restarting Riak and
>> force of the computer within 30 to 60 seconds. After reboot at least one
>> corrupt leveldb database can be expected. When this happens Riak fails
>> to start and logs this message:
>>
>> Failed to start riak_kv_eleveldb_backend Reason: {db_open,"Corruption:
>> CURRENT file does not end with newline"}
>>
>> Switching on the sync flag for leveldb has no effect.
>>
>> --
>> Thomas Bakketun, Senior software developer
>> T. +47 21 53 69 40
>> Copyleft Solutions
>> www.copyleftsolutions.com
>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
--
Thomas Bakketun, Senior software developer
T. +47 21 53 69 40
Copyleft Solutions
www.copyleftsolutions.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
