Re: Migration from memcachedb to riak
( first post here, hi everybody... ) If you don't need MR, 2i, etc, then BitCask will be faster. You just need to make sure all your keys fit in memory, which should not be a problem. How many keys do you have and what's their average length ? About the values,you can save a lot of space by choosing an appropriate serialization. We use Sereal[1] to serialize our data, and it's small enough that we don't need to compress it further (it can automatically use snappy to compress further). There is a php client [2] If you use leveldb, it can compress using snappy, but I've been a bit disappointed by snappy, because it didn't work well with our data. If you serialize your php object as verbose string (I don't know what's the usual way to serialize php objects), then you should probably benchmark different compressions algorithms on the application side. [1]: https://github.com/Sereal/Sereal/wiki/Sereal-Comparison-Graphs [2]: https://github.com/tobyink/php-sereal/tree/master/PHP On 10 July 2013 10:49, Edgar Veiga wrote: > Hello all! > > I have a couple of questions that I would like to address all of you guys, > in order to start this migration the best as possible. > > Context: > - I'm responsible for the migration of a pure key/value store that for now > is being stored on memcacheDB. > - We're serializing php objects and storing them. > - The total size occupied it's ~2TB. > > - The idea it's to migrate this data to a riak cluster with elevelDB > backend (starting with 6 nodes, 256 partitions. This thing is scaling very > fast). > - We only need to access the information by key. *We won't need neither > map/reduces, searches or secondary indexes*. It's a pure key/value store! > > My questions are: > - Do you have any riak fine tunning tip regarding this use case (due to > the fact that we will only use the key/value capabilities of riak)? > - It's expected that those 2TB would be reduced due to the levelDB > compression. Do you think we should compress our objects to on the client? > > Best regards, > Edgar Veiga > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Migration from memcachedb to riak
On 10 July 2013 11:03, Edgar Veiga wrote: > Hi Guido. > > Thanks for your answer! > > Bitcask it's not an option due to the amount of ram needed.. We would need > a lot more of physical nodes so more money spent... > Why is it not an option? If you use Bitcask, then each node needs to store its keys in memory. It's usually not a lot. In a precedent email I asked you the average lenght of *keys*, but you gave us the average length of *values* :) We have 1 billion keys and fits on a 5 nodes Ring. ( check out http://docs.basho.com/riak/1.2.0/references/appendices/Bitcask-Capacity-Planning/). Our bucket names are 1 letter, our keys are 10 chars long. What does a typical key look like ? Also, what are you using to serialize your php objects? Maybe you could paste a typical value somewhere as well Damien > > Instead we're using less machines with SSD disks to improve elevelDB > performance. > > Best regards > > > > On 10 July 2013 09:58, Guido Medina wrote: > >> Well, I rushed my answer before, if you want performance, you probably >> want Bitcask, if you want compression then LevelDB, the following links >> should help you decide better: >> >> http://docs.basho.com/riak/1.2.0/tutorials/choosing-a-backend/Bitcask/ >> http://docs.basho.com/riak/1.2.0/tutorials/choosing-a-backend/LevelDB/ >> >> Or multi, use one as default and then the other for specific buckets: >> >> http://docs.basho.com/riak/1.2.0/tutorials/choosing-a-backend/Multi/ >> >> HTH, >> >> Guido. >> >> >> >> On 10/07/13 09:53, Guido Medina wrote: >> >> Then you are better off with Bitcask, that will be the fastest in your >> case (no 2i, no searches, no M/R) >> >> HTH, >> >> Guido. >> >> On 10/07/13 09:49, Edgar Veiga wrote: >> >> Hello all! >> >> I have a couple of questions that I would like to address all of you >> guys, in order to start this migration the best as possible. >> >> Context: >> - I'm responsible for the migration of a pure key/value store that for >> now is being stored on memcacheDB. >> - We're serializing php objects and storing them. >> - The total size occupied it's ~2TB. >> >> - The idea it's to migrate this data to a riak cluster with elevelDB >> backend (starting with 6 nodes, 256 partitions. This thing is scaling very >> fast). >> - We only need to access the information by key. *We won't need neither >> map/reduces, searches or secondary indexes*. It's a pure key/value store! >> >> My questions are: >> - Do you have any riak fine tunning tip regarding this use case (due to >> the fact that we will only use the key/value capabilities of riak)? >> - It's expected that those 2TB would be reduced due to the levelDB >> compression. Do you think we should compress our objects to on the client? >> >> Best regards, >> Edgar Veiga >> >> >> ___ >> riak-users mailing >> listriak-users@lists.basho.comhttp://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >> >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Migration from memcachedb to riak
Hi, Indeed you're using very big keys. If you can't change the keys, then yes you'll have to use leveldb. However I wonder why you need keys that long :) On 10 July 2013 13:04, Edgar Veiga wrote: > Hi Damien, > > Well let's dive into this a little bit. > > I told you guys that bitcask was not an option due to a bad past > experiencie with couchbase (sorry, in the previous post I wrote couchdb), > that uses the same architecture as bitcask, keys in memory and values in > disk. > > We started the migration to couchbase, and were already using 3 physical > nodes and only had imported 5% of the real data! That was one of the main > reasons for choosing a solution like riak + leveldb, to delete the keys fit > in memory bottleneck.. > > Now, here's a tipical key (It's large :D, x=letter and 0=numbers ): > > xxx___0x_00_000_000xx > > We are using the php serialize native function! > > Best regards > > > > On 10 July 2013 11:43, damien krotkine wrote: > >> >> >> >> On 10 July 2013 11:03, Edgar Veiga wrote: >> >>> Hi Guido. >>> >>> Thanks for your answer! >>> >>> Bitcask it's not an option due to the amount of ram needed.. We would >>> need a lot more of physical nodes so more money spent... >>> >> >> Why is it not an option? >> >> If you use Bitcask, then each node needs to store its keys in memory. >> It's usually not a lot. In a precedent email I asked you the average lenght >> of *keys*, but you gave us the average length of *values* :) >> >> We have 1 billion keys and fits on a 5 nodes Ring. ( check out >> http://docs.basho.com/riak/1.2.0/references/appendices/Bitcask-Capacity-Planning/). >> Our bucket names are 1 letter, our keys are 10 chars long. >> >> What does a typical key look like ? Also, what are you using to serialize >> your php objects? Maybe you could paste a typical value somewhere as well >> >> Damien >> >> >>> >>> Instead we're using less machines with SSD disks to improve elevelDB >>> performance. >>> >>> Best regards >>> >>> >>> >>> On 10 July 2013 09:58, Guido Medina wrote: >>> >>>> Well, I rushed my answer before, if you want performance, you probably >>>> want Bitcask, if you want compression then LevelDB, the following links >>>> should help you decide better: >>>> >>>> http://docs.basho.com/riak/1.2.0/tutorials/choosing-a-backend/Bitcask/ >>>> http://docs.basho.com/riak/1.2.0/tutorials/choosing-a-backend/LevelDB/ >>>> >>>> Or multi, use one as default and then the other for specific buckets: >>>> >>>> http://docs.basho.com/riak/1.2.0/tutorials/choosing-a-backend/Multi/ >>>> >>>> HTH, >>>> >>>> Guido. >>>> >>>> >>>> >>>> On 10/07/13 09:53, Guido Medina wrote: >>>> >>>> Then you are better off with Bitcask, that will be the fastest in your >>>> case (no 2i, no searches, no M/R) >>>> >>>> HTH, >>>> >>>> Guido. >>>> >>>> On 10/07/13 09:49, Edgar Veiga wrote: >>>> >>>> Hello all! >>>> >>>> I have a couple of questions that I would like to address all of you >>>> guys, in order to start this migration the best as possible. >>>> >>>> Context: >>>> - I'm responsible for the migration of a pure key/value store that for >>>> now is being stored on memcacheDB. >>>> - We're serializing php objects and storing them. >>>> - The total size occupied it's ~2TB. >>>> >>>> - The idea it's to migrate this data to a riak cluster with elevelDB >>>> backend (starting with 6 nodes, 256 partitions. This thing is scaling very >>>> fast). >>>> - We only need to access the information by key. *We won't need >>>> neither map/reduces, searches or secondary indexes*. It's a pure >>>> key/value store! >>>> >>>> My questions are: >>>> - Do you have any riak fine tunning tip regarding this use case (due to >>>> the fact that we will only use the key/value capabilities of riak)? >>>> - It's expected that those 2TB would be reduced due to the levelDB >>>> compression. Do you think we should compress our objects to on the client? >>>> >>>> Best regards, >>>> Edgar Veiga >>>> >>>> >>>> ___ >>>> riak-users mailing >>>> listriak-users@lists.basho.comhttp://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>>> >>>> >>>> ___ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>> >>> ___ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >> > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
design a read-only data storage cluster
Hi, I'm trying to use Riak for - basically - a short-lived data storage system. Here are my prerequisites: - keys will contain a timestamp and some properties. I can work on making them short - values will be binary blobs. From few bytes to few MB - values are read-only : once written they are never updated or overwritten - key/values are constantly being added - items should expire after some times, few weeks if we can afford the disk space - keys should be indexed using 2i to be found via different aspects - if possible I'd like to be able to use normal HD, and avoid having to use expensive SSDs My Questions: - what kind of backend to use ? should I use multibackend? - It's the first time I use Riak for strongly date/time related read-only keys. Are there any tips & tricks? First option is to use leveldb as storage backend. And use an external script to expire (delete) keys that are too old (one of the secondary index being the timestamp, it's easy). However it seems that deleted keys in leveldb may not be claimed before a very long time, and some tests showed that real space used by the ring tend to always grow (space never reclamied?) and I've seen some threads about this lately. Second option would be to use multibackend, and one leveldb bucket that would only serve at indexing keys, and a bitcask backend to store the data. The number of keys is small enough that we can have them twice, and they should fit in memory for bitcask. I've read that using bitcask for big values is not recommended because file merging process would take a lot of time. However it seems that these merging issues wouldn't happen for values that are never updated/rewritten. The advantage of using bitcask would be to have automatic expiration. I'm maybe completely wrong about some (all) of my assumption, and that's why I'm seeking advices on this mailing list. Do you have any ideas on how to build such system . Maybe some of you have built a very similar one ? Thanks, dams. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: design a read-only data storage cluster
On Sat, Dec 21, 2013, at 06:51 AM, Matthew Von-Maszewski wrote: On Dec 21, 2013, at 4:12 AM, Damien Krotkine <[1]dkrotk...@gmail.com> wrote: First option is to use leveldb as storage backend. And use an external script to expire (delete) keys that are too old (one of the secondary index being the timestamp, it's easy). However it seems that deleted keys in leveldb may not be claimed before a very long time, and some tests showed that real space used by the ring tend to always grow (space never reclamied?) and I've seen some threads about this lately. The issue about leveldb not reclaiming disk space for a very long time is true. This is the way Google wrote leveldb. Basho has recently added code to leveldb to address the problem. Basho's change is discussed here: [2]https://github.com/basho/leveldb/wiki/mv-aggressive-delete The new code is part of the mv-agrressive-delete branch in the basho/leveldb repository at github. The branch is available now if you wish to manually include it within your Riak build. The branch is also part of the upcoming Riak 2.0 release. Thank you for the information. I'll try to use Bitcask for now. References 1. mailto:dkrotk...@gmail.com 2. https://github.com/basho/leveldb/wiki/mv-aggressive-delete ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
PUT / GET priority
Hi, I'm stress testing Riak on a reasonable cluster size (10 nodes), where 1 process constantly adds data, and various other processes read it, but never update it (nor delete it). I can see that when I add more processes to read the data, the processing PUTing the data is affected, and its performance drops quite a bit. That was expected, but I was wondering if there is a way to force Riak to handle PUT operations in priority, or any idea you may have. My goal is to keep the data feeding process performance good, even if some reader processes are pretty slow. It's ok if it's a hack or not documented, I just would like to know what my options are. Also, is it possible to setup GET throttling ? Thanks, dams ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Still merge errors in patched riak_kv_bitcask_backend
Hi,
I'm using Riak 2.0 and the latest riak_kv_bitcask_backend.beam patch
from this very mailing list [0], but I'm still experiencing merge errors:
$ md5sum /usr/lib64/riak/lib/basho-patches/riak_kv_bitcask_backend.beam
751dcf22ba4ce17630db062b0c646305
riak attach shows:
> code:which(riak_kv_bitcask_backend).
"/usr/lib64/riak/lib/basho-patches/riak_kv_bitcask_backend.beam"
Alas, I still get a lot of merging errors [1]. Note that I use a merging
strategy that is maybe not too common: I make sure Riak waits before a
file is completely full of out-dated keys, before it tries to merge it.
Any hint ? Shall I open a github issue ?
Thanks,
dams.
[0]
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2014-September/015897.html
[1]
2014-09-25 14:07:16.291 [info] <0.16612.8> All keys expired in:
"/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/112.bitcask.data"
scheduling file for deletion
2014-09-25 14:07:16.298 [error] <0.16612.8> Failed to merge
{["/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/96.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/94.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/92.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/91.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/90.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/89.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/125.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/124.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/123.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/122.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/121.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/120.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/118.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/117.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/116.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/114.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/112.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/110.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/108.bitcask.data","/var/lib/riak/bitcask_mult_events/822094670998632891489572718402909198556462055424/106.bi...",...],...}:
{generic_failure,error,badarg,[{bitcask_nifs,keydir_remove_int,[<<>>,{tombstone,<<2,0,6,101,118,101,110,116,115,49,52,49,49,49,54,49,53,56,49,58,49,58,69,88,80,58,58,48>>},1411161602,114,<<133,167,151,16,0,0,0,0>>,1411654036],[]},{bitcask,'-expiry_merge/4-fun-0-',7,[{file,"src/bitcask.erl"},{line,1912}]},{bitcask_fileops,fold_hintfile_loop,5,[{file,"src/bitcask_fileops.erl"},{line,660}]},{bitcask_fileops,fold_file_loop,8,[{file,"src/bitcask_fileops.erl"},{line,720}]},{bitcask_fileops,fold_hintfile,3,[{file,"src/bitcask_fileops.erl"},{line,624}]},{bitcask,expiry_merge,4,[{file,"src/bitcask.erl"},{line,1915}]},{bitcask,merge1,4,[{file,"src/bitcask.erl"},{line,686}]},{bitcask,merge,3,[{file,"src/bitcask.erl"},{line,566}]}]}
--
Damien Krotkine, IT Developer, Paris, France.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Broken beam for bitcask patch on basho website
Hi, On the official website, http://docs.basho.com/riak/latest/downloads/ mentions that bitcask users should use this patch : http://s3.amazonaws.com/downloads.basho.com/patches/bitcask-2.0-merge-crash/riak_kv_bitcask_backend.beam However, the link will let you download a beam file that actually contains html... I think it would be handy to fix the link at some point :) Cheers, dams ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
measuring gossip bandwidth
Hi, I'd like to (at least approximately know) how much gossiping (and other things that are non strictly data-copying related) uses of the network bandwidth. Is there any information I can lookup in logs or via the console, or even any experiment that I can do to measure it? I am operating clusters in a bandwidth-constrainted environment, and I'd like to keep bandwidth usage to a minimum Thanks, dams ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak CS expiration
Hi, As far as I understand, Riak CS uses bitcask for data storage and leveldb for metadata. Trying to implement expiration of data in riak cs, bitcask expiration works fine, but of course metadata are still in the leveldb backend. Is there any way to expire metadata from riak cs automatically ? Ideally, I'd like to be able to install a hook or callback on the bitcask compaction. Is that possible ? Thanks, dams ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
upgrading a node in a 27 nodes cluster
Hi,
In a cluster with 27 nodes, running 2.0.0, I upgraded one node to 2.1.1
(riak-2.1.1-1.el6.x86_64.rpm). Since them, this node is producing these
kind of logs, every couple of minutes:
2015-05-27 14:56:31.978 [info]
<0.94.0>@riak_core_sysmon_handler:handle_event:92 monitor long_schedule
<0.1201.0>
[{initial_call,{riak_core_vnode,init,1}},{almost_current_function,{gen,do_call,4}},{message_queue_len,1}]
[{timeout,1329},{in,{gen,do_call,4}},{out,{gen,do_call,4}}]
Any idea what this is and if it's harmful? I'd like to double check
before upgrading the other nodes
Thanks
dams.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Upgrading 1 node in a 27 nodes cluster
Hi,
In a cluster with 27 nodes, running 2.0.0, I upgraded one node to 2.1.1
(riak-2.1.1-1.el6.x86_64.rpm). Since them, this node is producing these
kind of logs, every couple of minutes:
2015-05-27 14:56:31.978 [info]
<0.94.0>@riak_core_sysmon_handler:handle_event:92 monitor long_schedule
<0.1201.0>
[{initial_call,{riak_core_vnode,init,1}},{almost_current_function,{gen,do_call,4}},{message_queue_len,1}]
[{timeout,1329},{in,{gen,do_call,4}},{out,{gen,do_call,4}}]
Any idea what this is and if it's harmful? I'd like to double check
before upgrading the other nodes
Thanks
dams.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: upgrading a node in a 27 nodes cluster
Thanks Matt, I'll silence it then.
Cheers, Damien
On Mon, Jun 1, 2015, at 05:44 PM, Matt Brender wrote:
>
> Hey Damien,
>
> I followed up internally and had Brian Sparrow share the details:
>
> This is just saying that an operation took longer than a configurable
> threshold. It can safely be ignored. I believe the default threshold
> is 1ms. Looks like this was disabled in earlier versions but was re-
> enabled in vm settings in 2.1.
>
> _
> http://erldocs.com/current/erts/erlang.html?i=2&search=system_mon#system_monitor/2_
>
> Cheers, Matt
>
>
> --
> Matt Brender | Developer Advocacy Lead
> [t]@mjbrender[1]
>
> On Wednesday, May 27, 2015 at 11:26 AM, Damien Krotkine
> , wrote:
>> Hi,
>>
>>
>>
>> In a cluster with 27 nodes, running 2.0.0, I upgraded one node
>> to 2.1.1
>>
>> (riak-2.1.1-1.el6.x86_64.rpm). Since them, this node is
>> producing these
>>
>> kind of logs, every couple of minutes:
>>
>>
>>
>> 2015-05-27 14:56:31.978 [info]
>>
>> <.94.0>@riak_core_sysmon_handler:handle_event:92 monitor
>> long_schedule
>>
>> <.1201.0>
>>
>> [{initial_call,{riak_core_vnode,init,1}},{almost_current_function,{g-
>> en,do_call,4}},{message_queue_len,1}]
>>
>> [{timeout,1329},{in,{gen,do_call,4}},{out,{gen,do_call,4}}]
>>
>>
>>
>>
>>
>> Any idea what this is and if it's harmful? I'd like to double check
>>
>> before upgrading the other nodes
>>
>>
>>
>> Thanks
>>
>> dams.
>>
>>
>>
>> ___
>>
>> riak-users mailing list
>>
>> riak-users@lists.basho.com
>>
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
Links:
1. http://twitter.com/mjbrender
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Bitcask expiration issue
Hi, I'm seeing strange things with bitcask expiration: When a new node joins a Ring, it gets data from the other nodes to rebalance. It seems that the starting date to count expiration of this data is the time when they were copied to the new node, *not* the date they were originally created in the Ring For instance, on a Ring with 2 weeks expiration: I write 1 key at T=0 in a Ring with 3 nodes. At T+7days I add a new node 4. This node gets the ownership of the key. It seems that this key will expire on node 4 only at T+21 instead of T+14 That's annoying because we have a lot of data that is constantly being written and that *must* be expired at proper time. Here is a graph (warning, it shows the free disk space, not the disk space used) of data expiration when adding new nodes. https://www.dropbox.com/s/1od5xo5wxsvk11j/render_1.png?dl=0 Is it a wrong configuration ? Here is my bitcask expiration configuration: multi_backend.bitcask_mult_events.storage_backend = bitcask multi_backend.bitcask_mult_events.bitcask.data_root = /var/lib/riak/bitcask_mult_events multi_backend.bitcask_mult_events.bitcask.expiry = 8d multi_backend.bitcask_mult_events.bitcask.expiry.grace_time = 1d multi_backend.bitcask_mult_events.bitcask.io_mode = erlang multi_backend.bitcask_mult_events.bitcask.max_file_size = 4GB multi_backend.bitcask_mult_events.bitcask.merge.policy = always multi_backend.bitcask_mult_events.bitcask.merge.thresholds.dead_bytes = 4GB multi_backend.bitcask_mult_events.bitcask.merge.thresholds.fragmentation = 100 multi_backend.bitcask_mult_events.bitcask.merge.triggers.dead_bytes = 4GB multi_backend.bitcask_mult_events.bitcask.merge.triggers.fragmentation = 100 Maybe the configuration is not right, let me know if you think so. What I try to do here is to limit disk usage by compacting a file only if it contains only expired keys. Our data is pure readonly. Thanks, dams. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Bitcask expiration issue
Hi, [ this is a possible duplicate, but it seems that messages sent by my other email account (damien.krotk...@booking.com don't reach list recipients - if I'm wrong, let me know - so I'm sending it again with this address ] I'm seeing strange things with bitcask expiration: When a new node joins a Ring, it gets data from the other nodes to rebalance. It seems that the starting date to count expiration of this data is the time when they were copied to the new node, *not* the date they were originally created in the Ring For instance, on a Ring with 2 weeks expiration: I write 1 key at T=0 in a Ring with 3 nodes. At T+7days I add a new node 4. This node gets the ownership of the key. It seems that this key will expire on node 4 only at T+21 instead of T+14 That's annoying because we have a lot of data that is constantly being written and that *must* be expired at proper time. Here is a graph (warning, it shows the free disk space, not the disk space used) of data expiration when adding new nodes. https://www.dropbox.com/s/1od5xo5wxsvk11j/render_1.png?dl=0 Is it a wrong configuration ? Here is my bitcask expiration configuration: multi_backend.bitcask_mult_events.storage_backend = bitcask multi_backend.bitcask_mult_events.bitcask.data_root = /var/lib/riak/bitcask_mult_events multi_backend.bitcask_mult_events.bitcask.expiry = 8d multi_backend.bitcask_mult_events.bitcask.expiry.grace_time = 1d multi_backend.bitcask_mult_events.bitcask.io_mode = erlang multi_backend.bitcask_mult_events.bitcask.max_file_size = 4GB multi_backend.bitcask_mult_events.bitcask.merge.policy = always multi_backend.bitcask_mult_events.bitcask.merge.thresholds.dead_bytes = 4GB multi_backend.bitcask_mult_events.bitcask.merge.thresholds.fragmentation = 100 multi_backend.bitcask_mult_events.bitcask.merge.triggers.dead_bytes = 4GB multi_backend.bitcask_mult_events.bitcask.merge.triggers.fragmentation = 100 Maybe the configuration is not right, let me know if you think so. What I try to do here is to limit disk usage by compacting a file only if it contains only expired keys. Our data is pure readonly. EDIT: on IRC I got this reply from EvilEngel: EvilEngel> Hey dams. Unfortunately that is how Bitcask expiry works. the timestamp is internal and not part of the object, so re-writing it restarts the TTL. This is worrying: it implies that any new node added to a cluster should have enough disk space to be able to store *twice* the amount of expirable data it is supposed to store. This is ( in my opinion ) an important fact that should be stressed in the documentation at least, and if possible, fixed. One way would be to have handoffs transfer transmit the original timestamp in addition to the data, so that it's properly set on the target node. Thanks, dams. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Ring resizing space requirement
Hi, I need to do a ring-resizing on a live cluster. The documentation ( http://docs.basho.com/riak/latest/ops/advanced/ring-resizing/ ) is not very detailed, and I'm missing one important information: how much free space do I need on the nodes to perform the ring resizing properly ? Having enough space seems critical as per the documentation, but there is no way to compute the free space required. Any information on that and also feedback on ring resizing on live cluster would be much apreciated. Thanks, dams ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
spurious secondary index writes
Hi, On my cluster, I am using the Multi backend, with a bunch of bitcask backends configured and a leveldb one. However, I am *never* writing to the leveldb backend. And I'm never writing any 2i However, riak-admin status is reporting secondary index writes: sudo riak-admin status | grep index index_fsm_active : 0 index_fsm_create : 0 index_fsm_create_error : 0 vnode_index_deletes : 0 vnode_index_deletes_postings : 0 vnode_index_deletes_postings_total : 0 vnode_index_deletes_total : 0 vnode_index_reads : 0 vnode_index_reads_total : 0 vnode_index_refreshes : 0 vnode_index_refreshes_total : 0 vnode_index_writes : 3484 vnode_index_writes_postings : 0 vnode_index_writes_postings_total : 0 vnode_index_writes_total : 67341668 I don't understand how this can happen. Is Riak using secondary indexes internally, or is the metric being polluted byt other values ? I ned to make sure my cluster is not creating secondary indexes :) Thanks, Damien Krotkine ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Stalled handoffs
Have you checked riak-admin transfer-limit ? if it's at zero handoffs will be blocked. > Le 21 sept. 2015 à 15:41, Nico Revin a écrit : > > Hi! > > I have the following issue: > # riak-admin ringready > TRUE All nodes agree on the ring ['riak@1', > 'riak@2', > 'riak@3', > 'riak@4'] > > # riak-admin transfers > 'riak@4' waiting to handoff 8 partitions > 'riak@3' waiting to handoff 17 partitions > 'riak@2' waiting to handoff 17 partitions > 'riak@1' waiting to handoff 26 partitions > > Active Transfers: > > > # riak-admin member-status > = Membership > == > Status RingPendingNode > --- > valid 0.0% 25.0%'riak@1' > valid 32.8% 25.0%'riak@2' > valid 34.4% 25.0%'riak@3' > valid 32.8% 25.0%'riak@4' > --- > Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 > > # riak-admin ring-status > == Claimant > === > Claimant: 'riak@2' > Status: up > Ring Ready: true > > == Ownership Handoff > == > Owner: riak@2 > Next Owner: riak@1 > > Index: 0 > Waiting on: [] > Complete: [riak_kv_vnode,riak_pipe_vnode] > > Index: 274031556999544297163190906134303066185487351808 > Waiting on: [] > Complete: [riak_kv_vnode,riak_pipe_vnode] > > Index: 548063113999088594326381812268606132370974703616 > Waiting on: [] > Complete: [riak_kv_vnode,riak_pipe_vnode] > > Index: 822094670998632891489572718402909198556462055424 > Waiting on: [riak_kv_vnode] > Complete: [riak_pipe_vnode] > > Index: 1096126227998177188652763624537212264741949407232 > Waiting on: [riak_kv_vnode] > Complete: [riak_pipe_vnode] > > Index: 1370157784997721485815954530671515330927436759040 > Waiting on: [riak_kv_vnode] > Complete: [riak_pipe_vnode] > > --- > Owner: riak@2 > Next Owner: riak@3 > > Index: 685078892498860742907977265335757665463718379520 > Waiting on: [riak_kv_vnode] > Complete: [riak_pipe_vnode,riak_search_vnode] > > Index: 959110449498405040071168171470060731649205731328 > Waiting on: [riak_kv_vnode] > Complete: [riak_pipe_vnode,riak_search_vnode] > > --- > Owner: riak@2 > Next Owner: riak@4 > > Index: 1438665674247607560106752257205091097473808596992 > Waiting on: [riak_kv_vnode] > Complete: [riak_pipe_vnode,riak_search_vnode] > > --- > Owner: riak@3 > Next Owner: riak@1 > > Index: 91343852333181432387730302044767688728495783936 > Waiting on: [] > Complete: [riak_kv_vnode,riak_pipe_vnode] > > Index: 365375409332725729550921208179070754913983135744 > Waiting on: [] > Complete: [riak_kv_vnode,riak_pipe_vnode] > > Index: 639406966332270026714112114313373821099470487552 > Waiting on: [] > Complete: [riak_kv_vnode,riak_pipe_vnode] > > Index: 913438523331814323877303020447676887284957839360 > Waiting on: [riak_kv_vnode] > Complete: [riak_pipe_vnode] > > Index: 1187470080331358621040493926581979953470445191168 > Waiting on: [] > Complete: [riak_kv_vnode,riak_pipe_vnode] > > --- > Owner: riak@3 > Next Owner: riak@2 > > Index: 1118962191081472546749696200048404186924073353216 > Waiting on: [riak_kv_vnode] > Complete: [riak_pipe_vnode,riak_search_vnode] > Index: 1392993748081016843912887106182707253109560705024 > Waiting on: [riak_kv_vnode] > Complete: [riak_pipe_vnode,riak_search_vnode] > --- > Owner: riak@3 > Next Owner: riak@4 > > Index: 707914855582156101004909840846949587645842325504 > Waiting on: [riak_kv_vnode] > Complete: [riak_pipe_vnode,riak_search_vnode] > > Index: 125597796958124469533129165311720016817029120 > Waiting on: [riak_kv_vnode] > Complete: [riak_pipe_vnode,riak_search_vnode] > > --- > Owner: riak@4 > Next Owner: riak@1 > > Index: 182687704666362864775460604089535377456991567872 > Waiting on: [riak_search_vnode] > Complete: [riak_kv_vnode,riak_pipe_vnode] > > Index: 456719261665907161938651510223838443642478919680 > Waiting on: [riak_search_vnode] > Complete: [riak_kv_vnode,riak_pipe_vnode] > > Index: 730750818665451459101842416358141509827966271488 > Waiting on: [riak_search_vnode] > Complete: [riak_kv_vnode,riak_pipe_vnode] > > Index:
Re: SOLR - indexing JSON
Hi Joe,
I have a similar setup, and in my case, "indexed_s" is properly indexed.
Are you sure that your data is really what you think it is ( i.e. real
JSON, with the right mimetype, etc) ?
dams.
Joe Olson wrote:
Using the default YZ index schema, I know I can index:
dataset={
indexed_s:"Blah"
}
I also know I can index:
dataset={
indexed_s:"Blah",
notindexed: 52
}
However, when I add:
dataset={
indexed_s:"Blah",
notindexed: 52,
otherstuff:{"something":1, "something_else":2}
}
the indexing fails. Please note all I really want to index is field
'indexed_s' - nothing else. Why would adding the sub dictionary
'otherstuff' (not indexed) wreck the indexing of field 'indexed_s'? Is
there a way around this? I really want search to ignore 'otherstuff' -
and it should as it does not follow the *_* format.
Thanks!
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: limit of bucket/solr indexes
Alexander Popov wrote: 1. does Riak have restrictions on bucket numbers? In practical life no, as long as you use bucket-types properly. 2. same for SOLR indexes? I assume you'll want one index per bucket. SOLR indexes are mostly limited by the RAM and disk space that you can throw at them Interesting not theoretical, but practical limits when it start works bad. Use case: we want split users bucket to bucket-per-team model, this bucket have also solr index on top. Can it work for 1000 teams? 10k? 100k? You should create a unique bucket-type, and have all these team buckets under that same bucket-type. That would work fine for 10k or 100k. For 100k, especially if you have 100k SOLR indexes, I recommend that you have a solid cluster, something like dozens of boxes with a lot of RAM (100G looks good) and CPUs (8 or 12) Of course these numbers are completely guestimate, because I have no idea what your workload will be. Cheers, dams ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Sorting of data using 2i pagination
AFAIK secondary index pagination sorting doesn't support descending order. Maybe you can simple have an additional 2i whith your value inverted ? Grigory Fateyev wrote: Hello! We use 2i pagination and it is very good. The only thing that bothering me is sorting. Now data returns in ascending order but we need descending order. How can we do it? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: LucidWorks Banana Integration
Hi Mark, I have successfully integrated Banana with Riak 2.0 Solr implementation. I simply configured a nginx to act as proxy between Riak Search / Solr / What banana expects. So basically: - Install Riak 2, java, and enable Riak Search (follow basho doc) - Install banana - install nginx and use this as a base : https://github.com/glickbot/riak-banana/blob/b9bd32242ee0a6ee133fb2804b1976a4fcd73f82/puppet/modules/riakbanana/templates/nginx.conf.erb - configure banana to point to the solr on your riak search. If you need more help, feel free to ask, dams Mark Schmidt wrote: Has anyone successfully integrated Banana with the Riak 2.0 Solr implementation? Regards, -Mark Schmidt ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Solr requests and Riak Search (was: Re: LucidWorks Banana Integration)
I feel like the topic changed a bit, so I changed the email subject. Mark Schmidt wrote: Hi Dams, I appreciate the assistance. I was able to turn up Banana fairly easily using the steps you laid out below, but I have a few questions regarding communicating with my Riak environment. We have a 6 Riak nodes sitting behind a proxy that handles both HTTP and protocol buffer load balancing. I’ve turned up a new standalone Riak node (hosts Banana/Nginx) outside of the cluster. First of all, you're using a proxy. It's common practice, and should not impact your solr requests, as long as the proxy properly forward the request (riak KV, search or solr, on any port) to one of the node. I personally am not a big fan of using a proxy in front of Riak nodes, although it's not a technical issue here. Most of the time people are using proxies in front of clusters (like Riak) to have the proxy answer the question "what are the nodes that are up and running ?" instead of answering this question on the hosts initiating the request. Although it looks like a great idea and seems to simplify things, it brings issues, like network traffic concentration, or potential SPOF. To work around that, the proxy has to be made redundant, and not configured as pass-through, etc. A better approach is - imho - to use something like Zookeeper or any lighter equivalent solution to have real-time knowledge of which nodes are up and running, and share that knowledge with all the hosts that are going to use Riak. They can then do round-robin or random selection of Riak nodes among the ones that are up and running. Anyway, that's a different topic than the present email. 1)Should the Nginx config file be pointed at my HAProxy IP that handles the Riak node load balancing, or do I need to incorporate additional settings in the config to handle the 6 Riak Solr nodes? The nginx configuration should point to your HAProxy IP. Distributed solr requests will be forwarded to the other nodes properly, as long as they are allowed to (firewal rules) on 8098 and 8093. 2)Should I use the Riak HTTP interface port (8098) or the Solr interface port (8093) in the Nginx config file? You should use 8098 ports for all queries 3)Is there any way to perform faceted queries or other more advanced query functionality against the Riak Solr nodes? Searching through the conversation archives, it sounds like we may be able to query the Solr nodes themselves outside of the Riak API. In a nutshell: yes! *all* Solr API is available through riak search, because the Riak API is just forwarding to solr. I recommend to *not* use any special Riak client to query Riak Search, but instead use plain Http, using any http client that your language provide. I recommend reading this page again http://docs.basho.com/riak/latest/dev/using/search/ but clicking on HTTP :) curl "$RIAK_HOST/search/query/famous?wt=json&q=age_i:%5B30%20TO%20*%5D" This is the example given. In my company, I've been using facets queries, stats, etc... The only limitation is the Solr version that is bundled with Riak Search (hopefully it'll be upgraded in a later release ). So back to nginx rules: The first rule is to allow banana to query riak search: it thinks that it's talking to a regular Solr, so you have to have a nginx rule to fix that. I think you've got that part right. In my setup I added an additional rule to allow using the solr admin web interface, whic give some interesting figures and options, and it's useful for debugging. So I added a rule to say that if it starts with "internal_solr", in this case instead of forwarding to 8098 it continues on 8093. But that's the only rule I added from the configuration I pointed to you. Here is an example of a request that I do using Riak Search : I do a query on a node, on port 80: $RIAK_HOST:80/solr/query/$SOLR_INDEX?stats.field=nr_requests_count_l&stats.facet=datacenter_s&q=*:*&rows=0&stats=true&fq=epoch_l:1449480300&fq=timeseries_s:requests&wt=json&indent=false the nginx rule ransforms that into : $RIAK_HOST:8098/search/$SOLR_INDEX?stats.field=nr_requests_count_l&stats.facet=datacenter_s&q=*:*&rows=0&stats=true&fq=epoch_l:1449480300&fq=timeseries_s:requests&wt=json&indent=false Disclaimer: I manually edited the request so it's probably not 100% valid, but at least you get the idea of what we can do : I'm using the solr stats features *with* facets at the same time. In this case I4m only interested by the stats (min/max/sum_of_squares/average) and not the actual results, so I set row=0. So basically all the solr power is there :) Hope that helps and sorry for the somewhat late answer, dams Thanks again Dams, -Mark Schmidt *From:*Damien Krotkine [mailto:dam...@krotkine.com] *Sent:* Saturday, November 28, 2015 5:
Re: Two quick questions about X-RIak-Meta-* headers....
Hi Joe, First of all, what Dmitri says makes a lot of sense. From what I understand, you are trying to avoid wasting network bandwidth by transferring data where you only need the metadata of your keys. As Dmitri pointed out, if your replication factor is 3 (default), then Riak will internally query all the replica full *data* and metadata, then only return the metadata to the client. From a network perspective, you'll save network bandwidth *outside* of your cluster, but you'll use quite a lot of network bandwidth *inside* of your cluster. Maybe that's not an issue. But if it is, read on: 1/ first solution : as Dmitri suggested, the best approach is to decouple your metadata and data, if possible. If your key is "my_stuff", then have your data stored in the bucket "data" and the metadata stored in the bucket "metadata". So that you'd fetch "metadata/my_stuff", then fetch "data/my_stuff". This should make your life way easier, but the issue is that you loose the strong relationship between data and metadata. To try to mitigate this: - when writing your data, start by writing "data/my_stuff" with conservative parameters ( w=3 for instance ) then wait for the successful write before storing the metadata. So that when the metadata is there, there is a very high chance that the data is there as well. - when updating your data: ry to be smart, like marking the metadata as invalid or unavailable, while you change the data underneath then update the metadata - be fault tolerant on your application: if you fetch some metadata, but the data is missing, retry, or wait, or gracefully fallback. - be fault tolerant again: when fetching some data, have it contain a header or an id that must match with the metadata. If it doesn't match, you need to wait/retry/fallback - if you don't want/can't handle that on the client side, it's possible to enrich the Riak API and have Riak do the bookeeping itself. If you're interested, let me know. 2/ second solution: you don't want / can't separate metadata and data. In this case, you can try to reduce the network usage - first, reduce the internal network usage inside the cluster. When querying the metadata, if you're using the PB API, you can pass "n_val" as parameter of your request. If you pass n_val=1, then Riak will not query the 3 replicas to fetch the value, instead it'll fetch only one replica, saving a lot of internal bandwidth. - second, you can have your client query one of the primary node (where one of the replica is) for a given key. Coupled with passing n_val=1, Riak will not transfer anything on the internal network. you can check out https://github.com/dams/riak_preflists to find the primary nodes. Dmitri Zagidulin wrote: Hi Joe, 1. Yes, it's possible (with the HTTP HEAD request, or the client library equivalent (I'm pretty sure all the client libraries expose the 'return only the headers' part of the object fetch -- see the Optional Parameters head=true section of the PB API http://docs.basho.com/riak/latest/dev/references/protocol-buffers/fetch-object/ )). However, this is not going to be very helpful in your case. Behind the scenes, a HEAD request still requests all replicas of an object -- *full* replicas, including the value. It's just that the node coordinating the request drops the actual object value before returning the metadata/headers to the client. So, if you use this 'just give me the metadata' request, you're only saving on the cost of shipping the object value down the wire from the cluster to the client. But you're still incurring the cost of all 3 copies of the object (so, 3-4MB, in your case) being transferred over the network between the nodes, as a result of that HEAD request. 2. I don't know that there is a specific size limit on the object header values. However, this question is definitely a red flag -- it's very likely that you're trying to use the custom headers in a way that they weren't intended for. Can you describe in more detail what's your use case? (As in, what are you trying to store as metadata, and why would retrieving just the headers be useful to you.) Don't forget, that if you need to store a LOT of metadata on an object, and you can't do it within the object value itself (for example, when storing binary images), you can simply store a separate metadata object, in a different bucket, using the same key as the object. For example, if I'm storing my primary objects in the 'blobs' bucket, I can also store a JSON object with corresponding metadata in a 'blobs-meta' object, like so: /buckets/blobs/keys/blob123 --> binary object value /buckets/blobs-meta/keys/blob123 --> json metadata object The downsides of this setup is that you're now doing 2 writes for each object (one to the blobs bucket, and one to the meta bucket). But the benefits are considerable: - You can store arbitrarily large metadata object (you're not abusing object headers by stuffi
Re: epmd listener port
There is an easier way to do it, by using the advanced config file
(usually /etc/riak/advanced.config )
Here is an example, where I use advanced.config to add a vm_args
configuration fields. So I suppose you could use the same syntax to add
your -env option:
[
{vm_args, [
{'-s my_app', ""},
{'-s my_other_app', ""}
]}
].
Note to Basho people: this is not very well documented (if at all), and
I've seen many people ask about it (be it for vm options, or adding
paths, etc). It might be useful to ad some documentation section about it.
Cheers,
dams
Christopher Meiklejohn wrote:
On Tue, Feb 16, 2016 at 3:24 PM, David Byron wrote:
http://docs.basho.com/riak/latest/community/faqs/operations/#is-it-possible-to-change-the-port-on-which-the-erl
says to add
-env ERL_EPMD_PORT
to vm.args, but I get the sense from
http://docs.basho.com/riak/latest/ops/advanced/configs/configuration-files/
that the goal is to avoid vm.args in preference to riak.conf.
My struggle is that I don't see a way to set the epmd port listener via
riak.conf. Is there a way? If not, are the new/latest docs missing a
mention of the correct location for vm.args?
riak.conf is generated by the cuttlefish extension, which apparently
does not have control for environment variables [1], so, I don't
believe there is a way to do this out of the box. However, you could
modify the schema files to add this cuttlefish transformation [2], if
you want to, or you could call the Riak initialization commands with
the EPMD environment variable set.
[1] https://github.com/basho/cuttlefish/issues/135
[2] https://github.com/basho/riak/blob/develop/rel/files/riak.schema
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Massive json schema update
Hi Guillaume, If I understand correctly you need to change all the values of your JSON data. How many keys are we talking about, how big are the data, and in how many buckets are the keys? Also, is your cluster in production yet? > Le 7 juin 2016 à 18:43, Guillaume Boddaert > a écrit : > > Hi, > > I'd like to patch my current riak collection to rename a field inside a JSON > schema, how can I achieve that from command line on the riak server itself ? > Is there some kind of map/reduce mechanism that allow any json record to be > updated then saved to the riak cluster ? > > Guillaume > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Massive json schema update
My advice : use the fetch-update loop that you already know. You won't get better performance or reliability by using a MapReduce. I understand that your values are indexed in Search. So either use a search query and update them all, or you could also do a list_keys in stream mode, and updated the values. Guillaume Boddaert wrote: Well, We are talking about ~30 million items, in a single bucket. Those are standard tweets, I need to alter all of them because of an invalid structure. My cluster is not in production, but I am willing to start validating a procedure that will be used at some point on a production cluster, that's a pre-release rehearsal. I can write my own script that read/update every entries in big scan of the bucket, but It may suffer from network lag, or other external stuffs. I'd really like to learn how to alter a set of document in the erlang console for instance, in order to validate that I can patch data easily in something that would looks more like an SQL command (Yes, that's not how it works, understand a map-reduce script in a mongo shell for instance) in the Riak core than a programmatic fetch/update loop that I already know. Guillaume On 08/06/2016 08:49, Damien Krotkine wrote: Hi Guillaume, If I understand correctly you need to change all the values of your JSON data. How many keys are we talking about, how big are the data, and in how many buckets are the keys? Also, is your cluster in production yet? Le 7 juin 2016 à 18:43, Guillaume Boddaert a écrit : Hi, I'd like to patch my current riak collection to rename a field inside a JSON schema, how can I achieve that from command line on the riak server itself ? Is there some kind of map/reduce mechanism that allow any json record to be updated then saved to the riak cluster ? Guillaume ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: RiakTS and Multi-Backend
For what it's worth, I've successfully used KV features inside of Riak TS and tested it quite a lot, including with a heavy load. As John said, I didn't use multi-backend and I disabled AAE. Riak TS was happy when using Gets, Sets, bucket types, and including CRDTs (I tested only the Set CRDTs). So you can go ahead and test these features, they should work fine, at least for now. That said, I would not recommend using this in production in the long term. Instead, use a different cluster for KV. However, different cluster doesn't have to mean different hardware. You could setup Riak KV and Riak TS alongside on the same nodes (you need some work to have separate configuration, data directory, different cookies, etc, but it's not a lot of work). Granted, both clusters will share resources, but having them running on the same nodes has a lot of benefits in term of maintenance and administration. John Daily writes: > There are several important KV components such as AAE and Search with which > TS integration has not been sufficiently tested. At this time we still > recommend running multiple clusters as mixed use workloads in Riak TS are > currently not supported. > > Your internal testing may reveal that small supporting KV datasets may work > as expected and allow you to augment your TS data in a single cluster. We > are interested in your results as this will help us to improve and expand > Riak TS moving forward. > > -John > >> On Oct 12, 2016, at 1:04 PM, Junk, Damion A wrote: >> >> So are you suggesting that it's not advisable to use even KV backed by >> LevelDB in a TS instance, or is it more that performance is unknown and >> therefore not guaranteed / supported? If the data is otherwise "safe", this >> may still be a better option for us than running separate clusters. >> >> Are others who need both a general KV store and the TS functionality running >> multiple clusters to handle this use case? >> >> Thanks for the quick reply! >> >> >> Damion >> >>> On Oct 12, 2016, at 11:53 AM, John Daily wrote: >>> >>> We have not done any work to support the multi-backend, hence the error >>> you’re seeing. TS depends exclusively on leveldb. >>> >>> We’re not recommending the use of KV functionality in the TS product yet, >>> because the latter is still changing rapidly and we will need to go back >>> and fix some basic KV mechanisms. We’re also not yet sure of the >>> performance characteristics if both KV and TS are in use under heavy load. >>> >>> In short: I apologize, but we’re not really ready to support your use case >>> yet. >>> >>> -John >>> >>>> On Oct 12, 2016, at 12:17 PM, Junk, Damion A wrote: >>>> >>>> Hello all - >>>> >>>> I was wondering if it is possible to use RiakTS with a multi-backend >>>> configuration. >>>> >>>> I have an existing set of applications using RiakKV and Bitcask, but we're >>>> now wanting to start using some of the TS features on a new project. >>>> Setting up the multi-backend configuration with TS seems to work fine (our >>>> app existing app reads the Bitcask buckets and SOLR indices without >>>> issue), and we can even create a TS schema. We have LevelDB as the default >>>> backend, so presumably this is what was used during the "create table" >>>> query. >>>> >>>> When trying to query the TS table, we're seeing: >>>> >>>> 2016-10-12 10:02:41.372 [error] <0.1410.0> gen_fsm <0.1410.0> in state >>>> active terminated with reason: call to undefined function >>>> riak_kv_multi_backend:range_scan/4 from riak_kv_vnode:list/7 line 1875 >>>> >>>> And then in the client code, no return until finally an Exception (Riak >>>> Java Client). >>>> >>>> >>>> Thanks for any assistance! >>>> >>>> >>>> Damion >>>> ___ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >> > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com -- Cheers, Damien Krotkine ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: anti_entropy and solr taking up suspiciously large amounts of space
On Mon, Apr 3, 2017, at 22:20, Matthew Von-Maszewski wrote: > Rohit, > > My apologies for the delayed reply. Too many conflicting demands on > my time the past two weeks. > > I reviewed the riak-debug package you shared. I also discussed its > contents with other Riak developers. > > There does not appear to be anything unexpected. The anti_entropy > bloat is due to the bitcask backend not actively communicating TTL > expirations to AAE. This is a known issue. Similarly, bitcask is not > communicating TTL expirations to solr. (The leveldb backend recently > added expiry / TTL. And it fails the same way in this scenario as > bitcask.) > > We have engineering designs in the works that will eventually correct > this situation. But the designs do not translate to code that you can > use today. My apologies. Hi Matthew, I've been experiencing the same kind of issues on some application prototype I'm working on. I tried using leveldb backend (through multi- backend) with TTL, coupled to Riak Search, and indeed Solr data are not deleted. Do you know if there is any ticket to follow ? (I searched for a github issue but couldn't find any), or any rough ETA on a fix on this ? Also, are there any best practises about deleting data from KV and Solr, especially, is search.queue.batch.minimum taken in account when deleting keys ? Thanks, Damien > > --> The 100% accurate approach today is to disable bitcask's TTL and > create external jobs that prune your data via Delete operations. Yes, > this is going to create a bunch of extra disk operations. But I am > being honest. > > --> You could reduce only the anti_entropy disk usage by changing the > "anti_entropy" setting in riak.conf from "active" to "passive". But > this does nothing for solr. > > Matthew > >> On Mar 22, 2017, at 10:56 AM, Matthew Von-Maszewski >> wrote: >> >> Rohit, >> >> Would you please run “riak-debug” on one server from the command line >> and send the tar.gz file it creates directly to me. Do not copy the >> email list. >> >> Notes on its usage are here: >> http://docs.basho.com/riak/kv/2.2.1/using/cluster-operations/inspecting-node/#riak-debug >> >> The resulting debug package will give me and others at Basho a better >> picture of the problem. The alternative is a about twenty rounds of >> “what about this, oh, then what about that”. >> >> Thanks, >> Matthew >> >> >> >>> On Mar 21, 2017, at 9:53 PM, Rohit Sanbhadti >>> wrote: >>> >>> Matthew, >>> >>> To clarify, this happens on all nodes in our cluster (10+ nodes) >>> although the exact size varies by 10s of GB. I’ve rolling restarted >>> the cluster the last time this happened (last week) with no >>> significant change in size, although the output from riak-admin aae- >>> status and riak-admin search aae-status shows empty after the >>> restart. >>> >>> -- >>> Rohit S. >>> >>> On 3/21/17, 5:25 PM, "Matthew Von-Maszewski" >>> wrote: >>> >>> Rohit, >>> >>> If you restart the node does the elevated anti_entropy size >>> decline after the restart? >>> >>> Matthew >>> >>> On Mar 21, 2017, at 8:00 PM, Rohit Sanbhadti wrote: Running Riak 2.2.0 on Ubuntu 16.04.1, we’ve noticed that anti_entropy is taking up way too much space on all of our nodes. We use multi_backend with mostly bitcask backends (relevant part of config pasted below). Has anyone seen this before, or does anyone have an idea what might be causing this? storage_backend = multi multi_backend.bitcask_1day.storage_backend = bitcask multi_backend.bitcask_1day.bitcask.data_root = /var/lib/riak/bitcask_1day multi_backend.bitcask_1day.bitcask.expiry = 1d multi_backend.bitcask_1day.bitcask.expiry.grace_time = 1h multi_backend.bitcask_2day.storage_backend = bitcask multi_backend.bitcask_2day.bitcask.data_root = /var/lib/riak/bitcask_2day multi_backend.bitcask_2day.bitcask.expiry = 2d multi_backend.bitcask_2day.bitcask.expiry.grace_time = 1h multi_backend.bitcask_4day.storage_backend = bitcask multi_backend.bitcask_4day.bitcask.data_root = /var/lib/riak/bitcask_4day multi_backend.bitcask_4day.bitcask.expiry = 4d multi_backend.bitcask_4day.bitcask.expiry.grace_time = 1h multi_backend.bitcask_8day.storage_backend = bitcask multi_backend.bitcask_8day.bitcask.data_root = /var/lib/riak/bitcask_8day multi_backend.bitcask_8day.bitcask.expiry = 8d multi_backend.bitcask_8day.bitcask.expiry.grace_time = 1h multi_backend.leveldb_mult.storage_backend = leveldb multi_backend.leveldb_mult.leveldb.maximum_memory.percent = 30 multi_backend.leveldb_mult.leveldb.data_root = /var/lib/riak/leveldb_mult multi_backend.bitcask_mult.storage_backend = bitcask multi_backend.bitcask_mult.bitcask.data_root = /var/lib/riak/bitcask_mult multi_backend.defaul
Re: Riak and the demise of Basho
+1 for Apache2 On Thu, Sep 7, 2017, at 19:05, Neeraj Sharma wrote: > +1 for Apache 2. Thanks for making it open source. > > On Thu, Sep 7, 2017 at 3:36 AM, Bill Barnhill > wrote: > > +1 for Apache 2. Thank you so much for saving all the important technology > > developed by the great developers from Basho. > > > > On Wed, Sep 6, 2017 at 5:31 PM, Heinz N. Gies wrote: > >> > >> +1 For Apache 2 :) > >> > >> On 6. Sep 2017, at 20:27, Jordan West wrote: > >> > >> Really excited to see this news. Thanks Bet365! +1 to apache2. > >> > >> Jordan > >> > >> -- > >> Sent from my iPhone > >> > >> On September 6, 2017 at 10:46:50 AM, AJAX DoneBy Jack (ajaxd...@gmail.com) > >> wrote: > >>> > >>> +2 Apache 2 > >>> > >>> On Wed, Sep 6, 2017 at 1:45 PM Bryan Hunt > >>> wrote: > > +1 Apache 2 > > > On Wed, 6 Sep 2017, 18:09 Tom Santero wrote: > > > > +1 Apache 2 > > > > On Wed, Sep 6, 2017 at 11:10 AM, Gokhan Boranalp > > wrote: > >> > >> +1 for Apache 2 > >> i wish we could use GPL 3 :( > >> > >> > >> On Sep 6, 2017 4:55 PM, "Drew Kerrigan" wrote: > >>> > >>> +1 to Apache 2 for MDC and other relevant EE repos. > >>> > >>> On Wed, Sep 6, 2017 at 9:02 AM DeadZen wrote: > > Sept 16 is software freedom day, a release that day might be a nice > idea. > > On Wed, Sep 6, 2017 at 8:37 AM, Christopher Meiklejohn > wrote: > > Given that all of the code except for MDC (and maybe JMX) is under > > Apache 2, I would assume that those components would follow the > > already existing license of the other components. > > > > Is there a plan to change the original license on the open source > > product from Apache 2? > > > > Thanks, > > Christopher > > > > On Wed, Sep 6, 2017 at 2:13 PM, wrote: > >> Hi, > >> > >> A quick status update. Today bet365 have signed the agreement to > >> purchase all Basho IP. We expect the agreement to be ratified in > >> the US > >> courts next week. Once cleared, our intention is to open source > >> all code, > >> help rebuild the community and collaboratively take the > >> development of RIAK > >> forward. > >> > >> In the coming weeks we will hopefully answer the questions people > >> have and will be calling on the community to help forge the > >> initial RIAK > >> Roadmap. > >> > >> One of the initial questions we have for the community is which > >> OSS license would people like applied to the code? Our thought is > >> the most > >> open and permissive. > >> > >> Andy. > >> > >> Andrew Deane > >> Systems Development Manager - Middleware > >> Hillside (Technology) Limited > >> andrew.de...@bet365.com > >> bet365.com > >> -Original Message- > >> From: riak-users [mailto:riak-users-boun...@lists.basho.com] On > >> Behalf Of martin@bet365.com > >> Sent: 24 August 2017 16:18 > >> To: riak-users@lists.basho.com > >> Cc: Martin Davies > >> Subject: FW: Riak and the demise of Basho > >> > >> Hi > >> > >> I have been asked to forward the below message from Martin > >> Davies, the CEO of Technology for bet365. > >> > >> Kind Regards > >> > >> Martin Cox > >> > >> From: Martin Davies > >> Sent: 24 August 2017 16:11 > >> To: Martin Cox > >> Subject: Riak and the demise of Basho > >> > >> Hi, > >> I have been wanting to make you aware for a few weeks now > >> that we have reached an agreement, in principle, to buy all of > >> Basho's > >> remaining assets (except support contracts) from the receiver. Up > >> until this > >> afternoon, I was constrained by some confidentiality needs of the > >> receiver > >> and was unable to speak. > >> > >> We have agreed a price for the assets and are almost at the > >> end of sorting out the legal agreement. Once this is complete, > >> this will > >> then need to be processed through the courts which, I am advised, > >> should > >> take a week or so. > >> > >> It is our intention to open source all of Basho's products > >> and all of the source code that they have been working on. We'll > >> do this as > >> quickly as we are able to organise it, and we would appreciate > >> some input > >> from the community on how
Re: Riak-2.2.5 progress update
Hi, I'm probably coming late to the party, but we have a big number of riak boxes running at work, in meta-clusters, so some rings are redundantly storing data. I could move one of them to the RC and compare its performance/errors/whatever with the non upgraded rings, if you people think it's useful. Poke me if that's useful. Cheers On Wed, Mar 28, 2018, at 09:07, martin@bet365.com wrote: > Awesome progress. > > We're happy to run generalised load tests against RC and share results > when completed - probably in the following week. > > Cheers > > From: riak-users [riak-users-boun...@lists.basho.com] on behalf of > Russell Brown [russell.br...@icloud.com] > Sent: 27 March 2018 19:04 > To: Fred Dushin > Cc: riak-users > Subject: Re: Riak-2.2.5 progress update > > Hey Fred, > > I can probably share my configs, yes. I have 4 configs! > > current is riak-2.2.5, the develop-2.2 branch of basho/riak for all > configs > previous is either riak-2.2.3 or riak-2.0.5 (and I have 4 configs as > there has to be an EE config, for the repl upgrade tests, since > riak-2.2.3 didn’t have repl, but riak_ee-2.2.3 did) > legacy is always 1.4.12 > > And since some tests have explicit version there are also explicit > entries for 2.0.6, 2.0.4, 2.0.2, in each config. > > Does that help? > > Cheers > > Russell > > On 27 Mar 2018, at 18:51, Fred Dushin wrote: > > > @russeldb do you have ordained values for `current`, `previous`, and > > `legacy` to test against? > > > > Always the :bane: of riak_test > > > > -Fred > > > >> On Mar 27, 2018, at 1:47 PM, Russell Brown wrote: > >> > >> Giddyup died when basho was shuttered. These test runs have all been on > >> private infrastructure that doesn’t have a public facing interface. > >> > >> I guess I could probably cut and paste logs into a gist for you. But I’d > >> have to take the time to sanitize them, just in case. I’d rather not take > >> that time unless absolutely necessary. > >> > >> On 27 Mar 2018, at 18:43, Bryan Hunt > >> wrote: > >> > >>> Could you share URL for those outstanding failures please. B > >>> > On 27 Mar 2018, at 16:37, Russell Brown wrote: > > Hi Again, > > More progress update. All the PRs are merged (thanks Bryan Hunt (ESL) > and Nick/Martin (bet365)). > > I’m planning on tagging Riak with riak-2.2.5RC1 this week. > > We haven’t been able to run load tests. I’m not 100% sure that all > basho’s releases had extensive load tests run, though I know that > later releases had a perf team, and MvM always load tested leveldb. > > The aim is to have those willing parties that deploy riak, and > therefore have perf testing already set up, to test and report > back. The NHS will run load tests and report results back. I hope that > others can do the same. > To the end we’ll probably tag RC1-noperftest. > > There are a few failing riak tests (was Riak ever released without a > failing riak-test?) If you have the time/capacity to run riak-test, > and you’re interested in helping out, get in touch and I’ll help you > get started. > > The failures, should one pique your interest: > > datatypes - riak667_mixed-eleveldb > ensemble - ensemble_basic3-eleveldb ensemble_basic4-eleveldb > yoko - yz_crdt-eleveldb yz_solr_upgrade_downgrade-eleveldb > > Let me know if you want to look into ensemble or yoko. > > Still aiming to have this tagged by end-of-week. > > Cheers > > Russell > > On 2 Mar 2018, at 09:44, Russell Brown wrote: > > > Hi, > > > > Just an update on the progress of riak-2.2.5 release. I realize we said > > "end of 2017" and then "end of Jan" and then "end of Feb" and here we > > are, 1st March, spring is upon us, and still no 2.2.5. I thought it best > > to at least keep you posted. > > > > Why no release? Well, we're not quite finished yet. In terms of what is > > left: > > > > - a few PRs need review and merge against the upstream Basho repo (see > > [1] if you want to help there); > > - Opening of PRs for gsets; > > - Docs for the changes; > > - Release notes; > > - Tagging; > > - A final round of testing (and fixing?) after all is merged; > > - Client support. This is crucial, but all the above work only has > > client support in the Basho erlang clients. I'm hoping the community > > that uses the Java/Python/Ruby etc clients can step up here. But we can > > release Riak before all client work is done. > > > > The optimist in me says 2 weeks. > > > > What do I mean "release"? > > > > From my point of view the release is the tags. After that I'm sincerely > > hoping ESL will continue to kindly build and host the actual artifacts. > > > > What's in the release? > > > >>
Re: Riak-2.2.5 progress update
Hi, I have installed RC2 (thanks to Nicholas' rpms) on one of our rings (5 boxes, heavy continuous read, heavy continuous write, centos7, bitcask, expiration, no compaction), and everything is running smoothly (same metrics as before, no errors so far). The only minor user experience issue was https://github.com/basho/riak/issues/940 On Thu, Apr 12, 2018, at 05:26, Nicholas Adams wrote: > Dear All, > We have placed a basic set of package builds online at > https://files.tiot.jp/riak/kv/2.2/2.2.5rc2/ > These builds were very simply a clone, checkout, make locked-all and > make package locked-all so should provide the same functionality as > building Riak 2.2.5rc2 from source. > Good luck and please share your feedback. > > Best regards, > > Nicholas > > -Original Message- > From: riak-users On Behalf Of > Russell Brown > Sent: 10 April 2018 18:53 > To: Bryan Hunt > Cc: Damien Krotkine ; riak-users us...@lists.basho.com> > Subject: Re: Riak-2.2.5 progress update > > More update. > > There were some minor changes that missed the rc1, so we added them > > * yokozuna/746: [remove -XX:+UseStringCache] > (https://github.com/basho/yokozuna/pull/746) > * yokozuna/747: [Remove jvm directive from test too] > (https://github.com/basho/yokozuna/pull/747) > * riak_repl/782: [Change ETS queue table permissions to protected] > (https://github.com/basho/riak_repl/pull/782) > > And today tagged RC2 (riak-2.2.5rc2) > > Same rules as before: > > At this point I DID NOT tag all the dependencies so it is _VERY > IMPORTANT_ that the rebar.config.lock file is used when you build a > release to test (e.g. `make locked-all`) > > If anyone (Tiot? ESL?) can/wants to build packages please post to this > list about location for those who'd rather test against packages. > > We've got perf test results upcoming, and then we'll tag final. > > Thanks for your patience, almost there now. > > Cheers > > Russell > > On 29 Mar 2018, at 18:18, Bryan Hunt wrote: > > > From our side, it's bank holiday weekend now, so we shall start building > > packages on Monday/Tuesday and share them out via package cloud. > > Will keep you updated. > > B > > > >> On 29 Mar 2018, at 16:15, Russell Brown wrote: > >> > >> > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
