Re: RiakCS usage data not found
Thanks Hector
I actually did the same steps as you mentioned expect I run Riak-CS in a
real virtual machine.
When I looking into the error log of Riak-CS when I doing riak-cs-access
flush an error was thrown (when doing other file operations are fine):
*2013-12-11 10:14:56.442 [error] <0.12276.0> gen_fsm <0.12276.0> in state
idle terminated with reason: bad argument in gen_fsm:send_event/2 line 198*
*2013-12-11 10:14:56.442 [error] <0.12276.0> CRASH REPORT Process
<0.12276.0> with 0 neighbours exited with reason: bad argument in
gen_fsm:send_event/2 line 198 in gen_fsm:terminate/7 line 611*
when doing riak-cs-storage batch no error was thrown with this message:
*2013-12-11 10:13:46.097 [info] <0.286.0>@riak_cs_storage_d:calculating:150
Finished storage calculation in 0 seconds.*
I tried Riak-CS-1.4.0 and latest Riak-CS, both get no luck. was that some
configuration issue?
Thanks.
Gavin
On Mon, Dec 9, 2013 at 10:08 PM, Hector Castro wrote:
> Hi Gavin,
>
> How have you configured your `storage_schedule`? It's possible that
> the job for calculating storage statistics hasn't occurred (or
> finished) yet.
>
> You can manually trigger storage usage calculation by invoking the
> following command:
>
> sudo riak-cs-storage batch
>
> From there, you can retrieve statistics from the `usage` bucket with
> `s3cmd`:
>
> s3cmd get
> s3://riak-cs/usage/.abj.20130501T00Z.20130531T00Z
>
> Also, here is a script I wrote up to access usage and storage
> statistics with `s3cmd`:
>
> https://gist.github.com/hectcastro/126b5657f228096775c6
>
> --
> Hector
>
>
> On Mon, Dec 9, 2013 at 1:25 AM, Gavin Huang
> wrote:
> > BTW, what I can see from the riak-cs log is:
> >
> > 2013-12-09 22:04:40.836 [error] <0.17758.42> No WM route: 'GET'
> > /riak-cs/usage/GRRYE2NHIXTNPFP89NMJ/abj/20131210T00Z/20131210T16Z
> > {5,{"x-amz-date",{"X-Amz-Date","Mon, 09 Dec 2013 06:10:17
> > +"},{"authorization",{'Authorization',"AWS
> >
> MESG2VCANJVU7XTX-9YC:K7v4mFdvvAZ8a5LJHqfc/TZLy/w="},{"accept-encoding",{'Accept-Encoding',"identity"},nil,nil},{"host",{'Host',"
> riak-cs.s3.amazonaws.com:8080
> "},nil,nil}},{"x-rcs-rewrite-path",{"x-rcs-rewrite-path","/riak-cs/usage/GRRYE2NHIXTNPFP89NMJ/abj/20131210T00Z/20131210T16Z"},nil,nil}}}
> >
> >
> >
> > On Mon, Dec 9, 2013 at 10:50 AM, Gavin Huang
> wrote:
> >>
> >> Hi,
> >> I'm trying to get the usage and access statistics from Riak-CS.
> >> I configured "storage_schedule"(restarted riak-cs), I manually start
> >> riak-cs-storage (it takes a while first time, but getting more quick
> >> afterwards).
> >>
> >> I tried to access the storage via s3cmd:
> >> s3cmd get s3://riak-cs/usage/[adminkey]
> >> s3cmd get s3://riak-cs/usage/[adminkey]/aj
> >> s3cmd get
> >> s3://riak-cs/usage/[adminkey]/abj/20131201T14Z/20131210T16Z
> >>
> >> all request get the response of:
> >> ERROR: S3 error: 404 (Object Not Found):
> >>
> >> is there anything I missed?
> >>
> >> Thanks.
> >> Gavin
> >
> >
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
Hi Matthew, it took around 11hours for the first node to finish the compaction. The second node is running already 12 hours and is still doing compaction. Besides that I wonder because the fsm_put time on the new 1.4.2 host is much higher (after the compaction) than on an old 1.3.1 (both are running in the cluster right now and another one is doing the compaction/upgrade while it is in the cluster but not directly accessible because it is out of the Loadbalancer): 1.4.2: node_put_fsm_time_mean : 2208050 node_put_fsm_time_median : 39231 node_put_fsm_time_95 : 17400382 node_put_fsm_time_99 : 50965752 node_put_fsm_time_100 : 59537762 node_put_fsm_active : 5 node_put_fsm_active_60s : 364 node_put_fsm_in_rate : 5 node_put_fsm_out_rate : 3 node_put_fsm_rejected : 0 node_put_fsm_rejected_60s : 0 node_put_fsm_rejected_total : 0 1.3.1: node_put_fsm_time_mean : 5036 node_put_fsm_time_median : 1614 node_put_fsm_time_95 : 8789 node_put_fsm_time_99 : 38258 node_put_fsm_time_100 : 384372 any clue why this could/should be? Cheers Simon On Tue, 10 Dec 2013 17:21:07 +0100 Simon Effenberg wrote: > Hi Matthew, > > thanks!.. that answers my questions! > > Cheers > Simon > > On Tue, 10 Dec 2013 11:08:32 -0500 > Matthew Von-Maszewski wrote: > > > 2i is not my expertise, so I had to discuss you concerns with another Basho > > developer. He says: > > > > Between 1.3 and 1.4, the 2i query did change but not the 2i on-disk format. > > You must wait for all nodes to update if you desire to use the new 2i > > query. The 2i data will properly write/update on both 1.3 and 1.4 machines > > during the migration. > > > > Does that answer your question? > > > > > > And yes, you might see available disk space increase during the upgrade > > compactions if your dataset contains numerous delete "tombstones". The > > Riak 2.0 code includes a new feature called "aggressive delete" for > > leveldb. This feature is more proactive in pushing delete tombstones > > through the levels to free up disk space much more quickly (especially if > > you perform block deletes every now and then). > > > > Matthew > > > > > > On Dec 10, 2013, at 10:44 AM, Simon Effenberg > > wrote: > > > > > Hi Matthew, > > > > > > see inline.. > > > > > > On Tue, 10 Dec 2013 10:38:03 -0500 > > > Matthew Von-Maszewski wrote: > > > > > >> The sad truth is that you are not the first to see this problem. And > > >> yes, it has to do with your 950GB per node dataset. And no, nothing to > > >> do but sit through it at this time. > > >> > > >> While I did extensive testing around upgrade times before shipping 1.4, > > >> apparently there are data configurations I did not anticipate. You are > > >> likely seeing a cascade where a shift of one file from level-1 to > > >> level-2 is causing a shift of another file from level-2 to level-3, > > >> which causes a level-3 file to shift to level-4, etc … then the next > > >> file shifts from level-1. > > >> > > >> The bright side of this pain is that you will end up with better write > > >> throughput once all the compaction ends. > > > > > > I have to deal with that.. but my problem is now, if I'm doing this > > > node by node it looks like 2i searches aren't possible while 1.3 and > > > 1.4 nodes exists in the cluster. Is there any problem which leads me to > > > an 2i repair marathon or could I easily wait for some hours for each > > > node until all merges are done before I upgrade the next one? (2i > > > searches can fail for some time.. the APP isn't having problems with > > > that but are new inserts with 2i indices processed successfully or do > > > I have to do the 2i repair?) > > > > > > /s > > > > > > one other good think: saving disk space is one advantage ;).. > > > > > > > > >> > > >> Riak 2.0's leveldb has code to prevent/reduce compaction cascades, but > > >> that is not going to help you today. > > >> > > >> Matthew > > >> > > >> On Dec 10, 2013, at 10:26 AM, Simon Effenberg > > >> wrote: > > >> > > >>> Hi @list, > > >>> > > >>> I'm trying to upgrade our Riak cluster from 1.3.1 to 1.4.2 .. after > > >>> upgrading the first node (out of 12) this node seems to do many merges. > > >>> the sst_* directories changes in size "rapidly" and the node is having > > >>> a disk utilization of 100% all the time. > > >>> > > >>> I know that there is something like that: > > >>> > > >>> "The first execution of 1.4.0 leveldb using a 1.3.x or 1.2.x dataset > > >>> will initiate an automatic conversion that could pause the startup of > > >>> each node by 3 to 7 minutes. The leveldb data in "level #1" is being > > >>> adjusted such that "level #1" can operate as an overlapped data level > > >>> instead of as a sorted data level. The conversion is simply the > > >>> reduction of the number of files in "level #1" to being less than eight > > >>> via normal compaction of data from "level #1" into "level #2". This is > > >>> a one time conversion." > > >>> > > >>> but it l
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
I need to ask other developers as they arrive for the new day. Does not make sense to me. How many nodes do you have? How much RAM do you have in each node? What are your settings for max_open_files and cache_size in the app.config file? Maybe this is as simple as leveldb using too much RAM in 1.4. The memory accounting for maz_open_files changed in 1.4. Matthew Von-Maszewski On Dec 11, 2013, at 6:28, Simon Effenberg wrote: > Hi Matthew, > > it took around 11hours for the first node to finish the compaction. The > second node is running already 12 hours and is still doing compaction. > > Besides that I wonder because the fsm_put time on the new 1.4.2 host is > much higher (after the compaction) than on an old 1.3.1 (both are > running in the cluster right now and another one is doing the > compaction/upgrade while it is in the cluster but not directly > accessible because it is out of the Loadbalancer): > > 1.4.2: > > node_put_fsm_time_mean : 2208050 > node_put_fsm_time_median : 39231 > node_put_fsm_time_95 : 17400382 > node_put_fsm_time_99 : 50965752 > node_put_fsm_time_100 : 59537762 > node_put_fsm_active : 5 > node_put_fsm_active_60s : 364 > node_put_fsm_in_rate : 5 > node_put_fsm_out_rate : 3 > node_put_fsm_rejected : 0 > node_put_fsm_rejected_60s : 0 > node_put_fsm_rejected_total : 0 > > > 1.3.1: > > node_put_fsm_time_mean : 5036 > node_put_fsm_time_median : 1614 > node_put_fsm_time_95 : 8789 > node_put_fsm_time_99 : 38258 > node_put_fsm_time_100 : 384372 > > > any clue why this could/should be? > > Cheers > Simon > > On Tue, 10 Dec 2013 17:21:07 +0100 > Simon Effenberg wrote: > >> Hi Matthew, >> >> thanks!.. that answers my questions! >> >> Cheers >> Simon >> >> On Tue, 10 Dec 2013 11:08:32 -0500 >> Matthew Von-Maszewski wrote: >> >>> 2i is not my expertise, so I had to discuss you concerns with another Basho >>> developer. He says: >>> >>> Between 1.3 and 1.4, the 2i query did change but not the 2i on-disk format. >>> You must wait for all nodes to update if you desire to use the new 2i >>> query. The 2i data will properly write/update on both 1.3 and 1.4 machines >>> during the migration. >>> >>> Does that answer your question? >>> >>> >>> And yes, you might see available disk space increase during the upgrade >>> compactions if your dataset contains numerous delete "tombstones". The >>> Riak 2.0 code includes a new feature called "aggressive delete" for >>> leveldb. This feature is more proactive in pushing delete tombstones >>> through the levels to free up disk space much more quickly (especially if >>> you perform block deletes every now and then). >>> >>> Matthew >>> >>> >>> On Dec 10, 2013, at 10:44 AM, Simon Effenberg >>> wrote: >>> Hi Matthew, see inline.. On Tue, 10 Dec 2013 10:38:03 -0500 Matthew Von-Maszewski wrote: > The sad truth is that you are not the first to see this problem. And > yes, it has to do with your 950GB per node dataset. And no, nothing to > do but sit through it at this time. > > While I did extensive testing around upgrade times before shipping 1.4, > apparently there are data configurations I did not anticipate. You are > likely seeing a cascade where a shift of one file from level-1 to level-2 > is causing a shift of another file from level-2 to level-3, which causes > a level-3 file to shift to level-4, etc … then the next file shifts from > level-1. > > The bright side of this pain is that you will end up with better write > throughput once all the compaction ends. I have to deal with that.. but my problem is now, if I'm doing this node by node it looks like 2i searches aren't possible while 1.3 and 1.4 nodes exists in the cluster. Is there any problem which leads me to an 2i repair marathon or could I easily wait for some hours for each node until all merges are done before I upgrade the next one? (2i searches can fail for some time.. the APP isn't having problems with that but are new inserts with 2i indices processed successfully or do I have to do the 2i repair?) /s one other good think: saving disk space is one advantage ;).. > > Riak 2.0's leveldb has code to prevent/reduce compaction cascades, but > that is not going to help you today. > > Matthew > > On Dec 10, 2013, at 10:26 AM, Simon Effenberg > wrote: > >> Hi @list, >> >> I'm trying to upgrade our Riak cluster from 1.3.1 to 1.4.2 .. after >> upgrading the first node (out of 12) this node seems to do many merges. >> the sst_* directories changes in size "rapidly" and the node is having >> a disk utilization of 100% all the time. >> >> I know that there is something like that: >> >> "The first execution of 1.4.0 leveldb using a 1.3.x or 1.2.x dataset >> will initiate an automatic
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
Hi Matthew Memory: 23999 MB ring_creation_size, 256 max_open_files, 100 riak-admin status: memory_total : 276001360 memory_processes : 191506322 memory_processes_used : 191439568 memory_system : 84495038 memory_atom : 686993 memory_atom_used : 686560 memory_binary : 21965352 memory_code : 11332732 memory_ets : 10823528 Thanks for looking! Cheers Simon On Wed, 11 Dec 2013 06:44:42 -0500 Matthew Von-Maszewski wrote: > I need to ask other developers as they arrive for the new day. Does not make > sense to me. > > How many nodes do you have? How much RAM do you have in each node? What are > your settings for max_open_files and cache_size in the app.config file? > Maybe this is as simple as leveldb using too much RAM in 1.4. The memory > accounting for maz_open_files changed in 1.4. > > Matthew Von-Maszewski > > > On Dec 11, 2013, at 6:28, Simon Effenberg wrote: > > > Hi Matthew, > > > > it took around 11hours for the first node to finish the compaction. The > > second node is running already 12 hours and is still doing compaction. > > > > Besides that I wonder because the fsm_put time on the new 1.4.2 host is > > much higher (after the compaction) than on an old 1.3.1 (both are > > running in the cluster right now and another one is doing the > > compaction/upgrade while it is in the cluster but not directly > > accessible because it is out of the Loadbalancer): > > > > 1.4.2: > > > > node_put_fsm_time_mean : 2208050 > > node_put_fsm_time_median : 39231 > > node_put_fsm_time_95 : 17400382 > > node_put_fsm_time_99 : 50965752 > > node_put_fsm_time_100 : 59537762 > > node_put_fsm_active : 5 > > node_put_fsm_active_60s : 364 > > node_put_fsm_in_rate : 5 > > node_put_fsm_out_rate : 3 > > node_put_fsm_rejected : 0 > > node_put_fsm_rejected_60s : 0 > > node_put_fsm_rejected_total : 0 > > > > > > 1.3.1: > > > > node_put_fsm_time_mean : 5036 > > node_put_fsm_time_median : 1614 > > node_put_fsm_time_95 : 8789 > > node_put_fsm_time_99 : 38258 > > node_put_fsm_time_100 : 384372 > > > > > > any clue why this could/should be? > > > > Cheers > > Simon > > > > On Tue, 10 Dec 2013 17:21:07 +0100 > > Simon Effenberg wrote: > > > >> Hi Matthew, > >> > >> thanks!.. that answers my questions! > >> > >> Cheers > >> Simon > >> > >> On Tue, 10 Dec 2013 11:08:32 -0500 > >> Matthew Von-Maszewski wrote: > >> > >>> 2i is not my expertise, so I had to discuss you concerns with another > >>> Basho developer. He says: > >>> > >>> Between 1.3 and 1.4, the 2i query did change but not the 2i on-disk > >>> format. You must wait for all nodes to update if you desire to use the > >>> new 2i query. The 2i data will properly write/update on both 1.3 and 1.4 > >>> machines during the migration. > >>> > >>> Does that answer your question? > >>> > >>> > >>> And yes, you might see available disk space increase during the upgrade > >>> compactions if your dataset contains numerous delete "tombstones". The > >>> Riak 2.0 code includes a new feature called "aggressive delete" for > >>> leveldb. This feature is more proactive in pushing delete tombstones > >>> through the levels to free up disk space much more quickly (especially if > >>> you perform block deletes every now and then). > >>> > >>> Matthew > >>> > >>> > >>> On Dec 10, 2013, at 10:44 AM, Simon Effenberg > >>> wrote: > >>> > Hi Matthew, > > see inline.. > > On Tue, 10 Dec 2013 10:38:03 -0500 > Matthew Von-Maszewski wrote: > > > The sad truth is that you are not the first to see this problem. And > > yes, it has to do with your 950GB per node dataset. And no, nothing to > > do but sit through it at this time. > > > > While I did extensive testing around upgrade times before shipping 1.4, > > apparently there are data configurations I did not anticipate. You are > > likely seeing a cascade where a shift of one file from level-1 to > > level-2 is causing a shift of another file from level-2 to level-3, > > which causes a level-3 file to shift to level-4, etc … then the next > > file shifts from level-1. > > > > The bright side of this pain is that you will end up with better write > > throughput once all the compaction ends. > > I have to deal with that.. but my problem is now, if I'm doing this > node by node it looks like 2i searches aren't possible while 1.3 and > 1.4 nodes exists in the cluster. Is there any problem which leads me to > an 2i repair marathon or could I easily wait for some hours for each > node until all merges are done before I upgrade the next one? (2i > searches can fail for some time.. the APP isn't having problems with > that but are new inserts with 2i indices processed successfully or do > I have to do the 2i repair?) > > /s > > one other good think: saving disk space is one advantage ;).. > > > > > > Riak 2.0's l
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
Ok, I am now suspecting that your servers are either using swap space (which is slow) or your leveldb file cache is thrashing (opening and closing multiple files per request). How many servers do you have and do you use Riak's active anti-entropy feature? I am going to plug all of this into a spreadsheet. Matthew Von-Maszewski On Dec 11, 2013, at 7:09, Simon Effenberg wrote: > Hi Matthew > > Memory: 23999 MB > > ring_creation_size, 256 > max_open_files, 100 > > riak-admin status: > > memory_total : 276001360 > memory_processes : 191506322 > memory_processes_used : 191439568 > memory_system : 84495038 > memory_atom : 686993 > memory_atom_used : 686560 > memory_binary : 21965352 > memory_code : 11332732 > memory_ets : 10823528 > > Thanks for looking! > > Cheers > Simon > > > > On Wed, 11 Dec 2013 06:44:42 -0500 > Matthew Von-Maszewski wrote: > >> I need to ask other developers as they arrive for the new day. Does not >> make sense to me. >> >> How many nodes do you have? How much RAM do you have in each node? What >> are your settings for max_open_files and cache_size in the app.config file? >> Maybe this is as simple as leveldb using too much RAM in 1.4. The memory >> accounting for maz_open_files changed in 1.4. >> >> Matthew Von-Maszewski >> >> >> On Dec 11, 2013, at 6:28, Simon Effenberg wrote: >> >>> Hi Matthew, >>> >>> it took around 11hours for the first node to finish the compaction. The >>> second node is running already 12 hours and is still doing compaction. >>> >>> Besides that I wonder because the fsm_put time on the new 1.4.2 host is >>> much higher (after the compaction) than on an old 1.3.1 (both are >>> running in the cluster right now and another one is doing the >>> compaction/upgrade while it is in the cluster but not directly >>> accessible because it is out of the Loadbalancer): >>> >>> 1.4.2: >>> >>> node_put_fsm_time_mean : 2208050 >>> node_put_fsm_time_median : 39231 >>> node_put_fsm_time_95 : 17400382 >>> node_put_fsm_time_99 : 50965752 >>> node_put_fsm_time_100 : 59537762 >>> node_put_fsm_active : 5 >>> node_put_fsm_active_60s : 364 >>> node_put_fsm_in_rate : 5 >>> node_put_fsm_out_rate : 3 >>> node_put_fsm_rejected : 0 >>> node_put_fsm_rejected_60s : 0 >>> node_put_fsm_rejected_total : 0 >>> >>> >>> 1.3.1: >>> >>> node_put_fsm_time_mean : 5036 >>> node_put_fsm_time_median : 1614 >>> node_put_fsm_time_95 : 8789 >>> node_put_fsm_time_99 : 38258 >>> node_put_fsm_time_100 : 384372 >>> >>> >>> any clue why this could/should be? >>> >>> Cheers >>> Simon >>> >>> On Tue, 10 Dec 2013 17:21:07 +0100 >>> Simon Effenberg wrote: >>> Hi Matthew, thanks!.. that answers my questions! Cheers Simon On Tue, 10 Dec 2013 11:08:32 -0500 Matthew Von-Maszewski wrote: > 2i is not my expertise, so I had to discuss you concerns with another > Basho developer. He says: > > Between 1.3 and 1.4, the 2i query did change but not the 2i on-disk > format. You must wait for all nodes to update if you desire to use the > new 2i query. The 2i data will properly write/update on both 1.3 and 1.4 > machines during the migration. > > Does that answer your question? > > > And yes, you might see available disk space increase during the upgrade > compactions if your dataset contains numerous delete "tombstones". The > Riak 2.0 code includes a new feature called "aggressive delete" for > leveldb. This feature is more proactive in pushing delete tombstones > through the levels to free up disk space much more quickly (especially if > you perform block deletes every now and then). > > Matthew > > > On Dec 10, 2013, at 10:44 AM, Simon Effenberg > wrote: > >> Hi Matthew, >> >> see inline.. >> >> On Tue, 10 Dec 2013 10:38:03 -0500 >> Matthew Von-Maszewski wrote: >> >>> The sad truth is that you are not the first to see this problem. And >>> yes, it has to do with your 950GB per node dataset. And no, nothing to >>> do but sit through it at this time. >>> >>> While I did extensive testing around upgrade times before shipping 1.4, >>> apparently there are data configurations I did not anticipate. You are >>> likely seeing a cascade where a shift of one file from level-1 to >>> level-2 is causing a shift of another file from level-2 to level-3, >>> which causes a level-3 file to shift to level-4, etc … then the next >>> file shifts from level-1. >>> >>> The bright side of this pain is that you will end up with better write >>> throughput once all the compaction ends. >> >> I have to deal with that.. but my problem is now, if I'm doing this >> node by node it looks like 2i searches aren't possible while 1.3 and >> 1.4 nodes exists in the cluster. Is there any problem which leads me to >> an 2i repai
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
Sorry I forgot the half of it.. seffenberg@kriak46-1:~$ free -m total used free sharedbuffers cached Mem: 23999 23759239 0184 16183 -/+ buffers/cache: 7391 16607 Swap:0 0 0 We have 12 servers.. datadir on the compacted servers (1.4.2) ~ 765 GB AAE is enabled. I attached app.config and vm.args. Cheers Simon On Wed, 11 Dec 2013 07:33:31 -0500 Matthew Von-Maszewski wrote: > Ok, I am now suspecting that your servers are either using swap space (which > is slow) or your leveldb file cache is thrashing (opening and closing > multiple files per request). > > How many servers do you have and do you use Riak's active anti-entropy > feature? I am going to plug all of this into a spreadsheet. > > Matthew Von-Maszewski > > > On Dec 11, 2013, at 7:09, Simon Effenberg wrote: > > > Hi Matthew > > > > Memory: 23999 MB > > > > ring_creation_size, 256 > > max_open_files, 100 > > > > riak-admin status: > > > > memory_total : 276001360 > > memory_processes : 191506322 > > memory_processes_used : 191439568 > > memory_system : 84495038 > > memory_atom : 686993 > > memory_atom_used : 686560 > > memory_binary : 21965352 > > memory_code : 11332732 > > memory_ets : 10823528 > > > > Thanks for looking! > > > > Cheers > > Simon > > > > > > > > On Wed, 11 Dec 2013 06:44:42 -0500 > > Matthew Von-Maszewski wrote: > > > >> I need to ask other developers as they arrive for the new day. Does not > >> make sense to me. > >> > >> How many nodes do you have? How much RAM do you have in each node? What > >> are your settings for max_open_files and cache_size in the app.config > >> file? Maybe this is as simple as leveldb using too much RAM in 1.4. The > >> memory accounting for maz_open_files changed in 1.4. > >> > >> Matthew Von-Maszewski > >> > >> > >> On Dec 11, 2013, at 6:28, Simon Effenberg > >> wrote: > >> > >>> Hi Matthew, > >>> > >>> it took around 11hours for the first node to finish the compaction. The > >>> second node is running already 12 hours and is still doing compaction. > >>> > >>> Besides that I wonder because the fsm_put time on the new 1.4.2 host is > >>> much higher (after the compaction) than on an old 1.3.1 (both are > >>> running in the cluster right now and another one is doing the > >>> compaction/upgrade while it is in the cluster but not directly > >>> accessible because it is out of the Loadbalancer): > >>> > >>> 1.4.2: > >>> > >>> node_put_fsm_time_mean : 2208050 > >>> node_put_fsm_time_median : 39231 > >>> node_put_fsm_time_95 : 17400382 > >>> node_put_fsm_time_99 : 50965752 > >>> node_put_fsm_time_100 : 59537762 > >>> node_put_fsm_active : 5 > >>> node_put_fsm_active_60s : 364 > >>> node_put_fsm_in_rate : 5 > >>> node_put_fsm_out_rate : 3 > >>> node_put_fsm_rejected : 0 > >>> node_put_fsm_rejected_60s : 0 > >>> node_put_fsm_rejected_total : 0 > >>> > >>> > >>> 1.3.1: > >>> > >>> node_put_fsm_time_mean : 5036 > >>> node_put_fsm_time_median : 1614 > >>> node_put_fsm_time_95 : 8789 > >>> node_put_fsm_time_99 : 38258 > >>> node_put_fsm_time_100 : 384372 > >>> > >>> > >>> any clue why this could/should be? > >>> > >>> Cheers > >>> Simon > >>> > >>> On Tue, 10 Dec 2013 17:21:07 +0100 > >>> Simon Effenberg wrote: > >>> > Hi Matthew, > > thanks!.. that answers my questions! > > Cheers > Simon > > On Tue, 10 Dec 2013 11:08:32 -0500 > Matthew Von-Maszewski wrote: > > > 2i is not my expertise, so I had to discuss you concerns with another > > Basho developer. He says: > > > > Between 1.3 and 1.4, the 2i query did change but not the 2i on-disk > > format. You must wait for all nodes to update if you desire to use the > > new 2i query. The 2i data will properly write/update on both 1.3 and > > 1.4 machines during the migration. > > > > Does that answer your question? > > > > > > And yes, you might see available disk space increase during the upgrade > > compactions if your dataset contains numerous delete "tombstones". The > > Riak 2.0 code includes a new feature called "aggressive delete" for > > leveldb. This feature is more proactive in pushing delete tombstones > > through the levels to free up disk space much more quickly (especially > > if you perform block deletes every now and then). > > > > Matthew > > > > > > On Dec 10, 2013, at 10:44 AM, Simon Effenberg > > wrote: > > > >> Hi Matthew, > >> > >> see inline.. > >> > >> On Tue, 10 Dec 2013 10:38:03 -0500 > >> Matthew Von-Maszewski wrote: > >> > >>> The sad truth is that you are not the first to see this problem. And > >>> yes, it has to do with your 950GB per node dataset. And no, nothing > >>> to do but sit through it at this time. > >>> > >>> While I did extensive
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
Also some side notes: "top" is even better on new 1.4.2 than on 1.3.1 machines.. IO utilization of disk is mostly the same (round about 33%).. but 95th percentile of response time for get (avg over all nodes): before upgrade: 29ms after upgrade: almost the same 95th percentile of response time for put (avg over all nodes): before upgrade: 60ms after upgrade: 1548ms but this is only because of 2 of 12 nodes are on 1.4.2 and are really slow (17000ms) Cheers, Simon On Wed, 11 Dec 2013 13:45:56 +0100 Simon Effenberg wrote: > Sorry I forgot the half of it.. > > seffenberg@kriak46-1:~$ free -m > total used free sharedbuffers cached > Mem: 23999 23759239 0184 16183 > -/+ buffers/cache: 7391 16607 > Swap:0 0 0 > > We have 12 servers.. > datadir on the compacted servers (1.4.2) ~ 765 GB > > AAE is enabled. > > I attached app.config and vm.args. > > Cheers > Simon > > On Wed, 11 Dec 2013 07:33:31 -0500 > Matthew Von-Maszewski wrote: > > > Ok, I am now suspecting that your servers are either using swap space > > (which is slow) or your leveldb file cache is thrashing (opening and > > closing multiple files per request). > > > > How many servers do you have and do you use Riak's active anti-entropy > > feature? I am going to plug all of this into a spreadsheet. > > > > Matthew Von-Maszewski > > > > > > On Dec 11, 2013, at 7:09, Simon Effenberg wrote: > > > > > Hi Matthew > > > > > > Memory: 23999 MB > > > > > > ring_creation_size, 256 > > > max_open_files, 100 > > > > > > riak-admin status: > > > > > > memory_total : 276001360 > > > memory_processes : 191506322 > > > memory_processes_used : 191439568 > > > memory_system : 84495038 > > > memory_atom : 686993 > > > memory_atom_used : 686560 > > > memory_binary : 21965352 > > > memory_code : 11332732 > > > memory_ets : 10823528 > > > > > > Thanks for looking! > > > > > > Cheers > > > Simon > > > > > > > > > > > > On Wed, 11 Dec 2013 06:44:42 -0500 > > > Matthew Von-Maszewski wrote: > > > > > >> I need to ask other developers as they arrive for the new day. Does not > > >> make sense to me. > > >> > > >> How many nodes do you have? How much RAM do you have in each node? > > >> What are your settings for max_open_files and cache_size in the > > >> app.config file? Maybe this is as simple as leveldb using too much RAM > > >> in 1.4. The memory accounting for maz_open_files changed in 1.4. > > >> > > >> Matthew Von-Maszewski > > >> > > >> > > >> On Dec 11, 2013, at 6:28, Simon Effenberg > > >> wrote: > > >> > > >>> Hi Matthew, > > >>> > > >>> it took around 11hours for the first node to finish the compaction. The > > >>> second node is running already 12 hours and is still doing compaction. > > >>> > > >>> Besides that I wonder because the fsm_put time on the new 1.4.2 host is > > >>> much higher (after the compaction) than on an old 1.3.1 (both are > > >>> running in the cluster right now and another one is doing the > > >>> compaction/upgrade while it is in the cluster but not directly > > >>> accessible because it is out of the Loadbalancer): > > >>> > > >>> 1.4.2: > > >>> > > >>> node_put_fsm_time_mean : 2208050 > > >>> node_put_fsm_time_median : 39231 > > >>> node_put_fsm_time_95 : 17400382 > > >>> node_put_fsm_time_99 : 50965752 > > >>> node_put_fsm_time_100 : 59537762 > > >>> node_put_fsm_active : 5 > > >>> node_put_fsm_active_60s : 364 > > >>> node_put_fsm_in_rate : 5 > > >>> node_put_fsm_out_rate : 3 > > >>> node_put_fsm_rejected : 0 > > >>> node_put_fsm_rejected_60s : 0 > > >>> node_put_fsm_rejected_total : 0 > > >>> > > >>> > > >>> 1.3.1: > > >>> > > >>> node_put_fsm_time_mean : 5036 > > >>> node_put_fsm_time_median : 1614 > > >>> node_put_fsm_time_95 : 8789 > > >>> node_put_fsm_time_99 : 38258 > > >>> node_put_fsm_time_100 : 384372 > > >>> > > >>> > > >>> any clue why this could/should be? > > >>> > > >>> Cheers > > >>> Simon > > >>> > > >>> On Tue, 10 Dec 2013 17:21:07 +0100 > > >>> Simon Effenberg wrote: > > >>> > > Hi Matthew, > > > > thanks!.. that answers my questions! > > > > Cheers > > Simon > > > > On Tue, 10 Dec 2013 11:08:32 -0500 > > Matthew Von-Maszewski wrote: > > > > > 2i is not my expertise, so I had to discuss you concerns with another > > > Basho developer. He says: > > > > > > Between 1.3 and 1.4, the 2i query did change but not the 2i on-disk > > > format. You must wait for all nodes to update if you desire to use > > > the new 2i query. The 2i data will properly write/update on both 1.3 > > > and 1.4 machines during the migration. > > > > > > Does that answer your question? > > > > > > > > > And yes, you might see available disk space increase during the > > > upgrade compactions if your dataset contains numerous delete > >
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
Oh and at the moment they are waiting for some handoffs and I see errors in logfiles: 2013-12-11 13:41:47.948 UTC [error] <0.7157.24>@riak_core_handoff_sender:start_fold:269 hinted_handoff transfer of riak_kv_vnode from 'riak@10.46.109.202' 468137243207554840987117797979434404733540892672 but I remember that somebody else had this as well and if I recall correctly it disappeared after the full upgrade was done.. but at the moment it's hard to think about upgrading everything at once.. (~12hours 100% disk utilization on all 12 nodes will lead to real slow puts/gets) What can I do? Cheers Simon PS: transfers output: 'riak@10.46.109.202' waiting to handoff 17 partitions 'riak@10.46.109.201' waiting to handoff 19 partitions (these are the 1.4.2 nodes) On Wed, 11 Dec 2013 14:39:58 +0100 Simon Effenberg wrote: > Also some side notes: > > "top" is even better on new 1.4.2 than on 1.3.1 machines.. IO > utilization of disk is mostly the same (round about 33%).. > > but > > 95th percentile of response time for get (avg over all nodes): > before upgrade: 29ms > after upgrade: almost the same > > 95th percentile of response time for put (avg over all nodes): > before upgrade: 60ms > after upgrade: 1548ms > but this is only because of 2 of 12 nodes are > on 1.4.2 and are really slow (17000ms) > > Cheers, > Simon > > On Wed, 11 Dec 2013 13:45:56 +0100 > Simon Effenberg wrote: > > > Sorry I forgot the half of it.. > > > > seffenberg@kriak46-1:~$ free -m > > total used free sharedbuffers cached > > Mem: 23999 23759239 0184 16183 > > -/+ buffers/cache: 7391 16607 > > Swap:0 0 0 > > > > We have 12 servers.. > > datadir on the compacted servers (1.4.2) ~ 765 GB > > > > AAE is enabled. > > > > I attached app.config and vm.args. > > > > Cheers > > Simon > > > > On Wed, 11 Dec 2013 07:33:31 -0500 > > Matthew Von-Maszewski wrote: > > > > > Ok, I am now suspecting that your servers are either using swap space > > > (which is slow) or your leveldb file cache is thrashing (opening and > > > closing multiple files per request). > > > > > > How many servers do you have and do you use Riak's active anti-entropy > > > feature? I am going to plug all of this into a spreadsheet. > > > > > > Matthew Von-Maszewski > > > > > > > > > On Dec 11, 2013, at 7:09, Simon Effenberg > > > wrote: > > > > > > > Hi Matthew > > > > > > > > Memory: 23999 MB > > > > > > > > ring_creation_size, 256 > > > > max_open_files, 100 > > > > > > > > riak-admin status: > > > > > > > > memory_total : 276001360 > > > > memory_processes : 191506322 > > > > memory_processes_used : 191439568 > > > > memory_system : 84495038 > > > > memory_atom : 686993 > > > > memory_atom_used : 686560 > > > > memory_binary : 21965352 > > > > memory_code : 11332732 > > > > memory_ets : 10823528 > > > > > > > > Thanks for looking! > > > > > > > > Cheers > > > > Simon > > > > > > > > > > > > > > > > On Wed, 11 Dec 2013 06:44:42 -0500 > > > > Matthew Von-Maszewski wrote: > > > > > > > >> I need to ask other developers as they arrive for the new day. Does > > > >> not make sense to me. > > > >> > > > >> How many nodes do you have? How much RAM do you have in each node? > > > >> What are your settings for max_open_files and cache_size in the > > > >> app.config file? Maybe this is as simple as leveldb using too much > > > >> RAM in 1.4. The memory accounting for maz_open_files changed in 1.4. > > > >> > > > >> Matthew Von-Maszewski > > > >> > > > >> > > > >> On Dec 11, 2013, at 6:28, Simon Effenberg > > > >> wrote: > > > >> > > > >>> Hi Matthew, > > > >>> > > > >>> it took around 11hours for the first node to finish the compaction. > > > >>> The > > > >>> second node is running already 12 hours and is still doing compaction. > > > >>> > > > >>> Besides that I wonder because the fsm_put time on the new 1.4.2 host > > > >>> is > > > >>> much higher (after the compaction) than on an old 1.3.1 (both are > > > >>> running in the cluster right now and another one is doing the > > > >>> compaction/upgrade while it is in the cluster but not directly > > > >>> accessible because it is out of the Loadbalancer): > > > >>> > > > >>> 1.4.2: > > > >>> > > > >>> node_put_fsm_time_mean : 2208050 > > > >>> node_put_fsm_time_median : 39231 > > > >>> node_put_fsm_time_95 : 17400382 > > > >>> node_put_fsm_time_99 : 50965752 > > > >>> node_put_fsm_time_100 : 59537762 > > > >>> node_put_fsm_active : 5 > > > >>> node_put_fsm_active_60s : 364 > > > >>> node_put_fsm_in_rate : 5 > > > >>> node_put_fsm_out_rate : 3 > > > >>> node_put_fsm_rejected : 0 > > > >>> node_put_fsm_rejected_60s : 0 > > > >>> node_put_fsm_rejected_total : 0 > > > >>> > > > >>> > > > >>> 1.3.1: > > > >>> > > > >>> node_put_fsm_time_mean : 5036 > > > >>> node_put_fsm_time_median : 1614 > > > >>> node_put_fsm_time_95 : 8789 > > > >>> node_pu
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
The real Riak developers have arrived on-line for the day. They are telling me that all of your problems are likely due to the extended upgrade times, and yes there is a known issue with handoff between 1.3 and 1.4. They also say everything should calm down after all nodes are upgraded. I will review your system settings now and see if there is something that might make the other machines upgrade quicker. So three more questions: - what is the average size of your keys - what is the average size of your value (data stored) - in regular use, are your keys accessed randomly across their entire range, or do they contain a date component which clusters older, less used keys Matthew On Dec 11, 2013, at 8:43 AM, Simon Effenberg wrote: > Oh and at the moment they are waiting for some handoffs and I see > errors in logfiles: > > > 2013-12-11 13:41:47.948 UTC [error] > <0.7157.24>@riak_core_handoff_sender:start_fold:269 hinted_handoff > transfer of riak_kv_vnode from 'riak@10.46.109.202' > 468137243207554840987117797979434404733540892672 > > but I remember that somebody else had this as well and if I recall > correctly it disappeared after the full upgrade was done.. but at the > moment it's hard to think about upgrading everything at once.. > (~12hours 100% disk utilization on all 12 nodes will lead to real slow > puts/gets) > > What can I do? > > Cheers > Simon > > PS: transfers output: > > 'riak@10.46.109.202' waiting to handoff 17 partitions > 'riak@10.46.109.201' waiting to handoff 19 partitions > > (these are the 1.4.2 nodes) > > > On Wed, 11 Dec 2013 14:39:58 +0100 > Simon Effenberg wrote: > >> Also some side notes: >> >> "top" is even better on new 1.4.2 than on 1.3.1 machines.. IO >> utilization of disk is mostly the same (round about 33%).. >> >> but >> >> 95th percentile of response time for get (avg over all nodes): >> before upgrade: 29ms >> after upgrade: almost the same >> >> 95th percentile of response time for put (avg over all nodes): >> before upgrade: 60ms >> after upgrade: 1548ms >>but this is only because of 2 of 12 nodes are >>on 1.4.2 and are really slow (17000ms) >> >> Cheers, >> Simon >> >> On Wed, 11 Dec 2013 13:45:56 +0100 >> Simon Effenberg wrote: >> >>> Sorry I forgot the half of it.. >>> >>> seffenberg@kriak46-1:~$ free -m >>> total used free sharedbuffers cached >>> Mem: 23999 23759239 0184 16183 >>> -/+ buffers/cache: 7391 16607 >>> Swap:0 0 0 >>> >>> We have 12 servers.. >>> datadir on the compacted servers (1.4.2) ~ 765 GB >>> >>> AAE is enabled. >>> >>> I attached app.config and vm.args. >>> >>> Cheers >>> Simon >>> >>> On Wed, 11 Dec 2013 07:33:31 -0500 >>> Matthew Von-Maszewski wrote: >>> Ok, I am now suspecting that your servers are either using swap space (which is slow) or your leveldb file cache is thrashing (opening and closing multiple files per request). How many servers do you have and do you use Riak's active anti-entropy feature? I am going to plug all of this into a spreadsheet. Matthew Von-Maszewski On Dec 11, 2013, at 7:09, Simon Effenberg wrote: > Hi Matthew > > Memory: 23999 MB > > ring_creation_size, 256 > max_open_files, 100 > > riak-admin status: > > memory_total : 276001360 > memory_processes : 191506322 > memory_processes_used : 191439568 > memory_system : 84495038 > memory_atom : 686993 > memory_atom_used : 686560 > memory_binary : 21965352 > memory_code : 11332732 > memory_ets : 10823528 > > Thanks for looking! > > Cheers > Simon > > > > On Wed, 11 Dec 2013 06:44:42 -0500 > Matthew Von-Maszewski wrote: > >> I need to ask other developers as they arrive for the new day. Does not >> make sense to me. >> >> How many nodes do you have? How much RAM do you have in each node? >> What are your settings for max_open_files and cache_size in the >> app.config file? Maybe this is as simple as leveldb using too much RAM >> in 1.4. The memory accounting for maz_open_files changed in 1.4. >> >> Matthew Von-Maszewski >> >> >> On Dec 11, 2013, at 6:28, Simon Effenberg >> wrote: >> >>> Hi Matthew, >>> >>> it took around 11hours for the first node to finish the compaction. The >>> second node is running already 12 hours and is still doing compaction. >>> >>> Besides that I wonder because the fsm_put time on the new 1.4.2 host is >>> much higher (after the compaction) than on an old 1.3.1 (both are >>> running in the cluster right now and another one is doing the >>> compaction/upgrade while it is in the cluster but not directly >>> accessible because it is out of the Loadbalancer): >>>
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
Hi Matthew, thanks for all your time and work.. see inline for answers.. On Wed, 11 Dec 2013 09:17:32 -0500 Matthew Von-Maszewski wrote: > The real Riak developers have arrived on-line for the day. They are telling > me that all of your problems are likely due to the extended upgrade times, > and yes there is a known issue with handoff between 1.3 and 1.4. They also > say everything should calm down after all nodes are upgraded. > > I will review your system settings now and see if there is something that > might make the other machines upgrade quicker. So three more questions: > > - what is the average size of your keys bucket names are between 5 and 15 characters (only ~ 10 buckets).. key names are normally something like 26iesj:hovh7egz > > - what is the average size of your value (data stored) I have to guess.. but mean is (from Riak) 12kb but 95th percentile is at 75kb and in theory we have a limit of 1MB (then it will be split up) but sometimes thanks to sibblings (we have to buckets with allow_mult) we have also some 7MB in MAX but this will be reduced again (it's a new feature in our app which has to many parallel wrights below of 15ms). > > - in regular use, are your keys accessed randomly across their entire range, > or do they contain a date component which clusters older, less used keys normally we don't search but retrieve keys by key name.. and we have data which is up to 6 months old and normally we access mostly new/active/hot data and not all the old ones.. besides this we have a job doing a 2i query every 5mins and another one doing this maybe once an hour.. both don't work while the upgrade is ongoing (2i isn't working). Cheers Simon > > Matthew > > > On Dec 11, 2013, at 8:43 AM, Simon Effenberg > wrote: > > > Oh and at the moment they are waiting for some handoffs and I see > > errors in logfiles: > > > > > > 2013-12-11 13:41:47.948 UTC [error] > > <0.7157.24>@riak_core_handoff_sender:start_fold:269 hinted_handoff > > transfer of riak_kv_vnode from 'riak@10.46.109.202' > > 468137243207554840987117797979434404733540892672 > > > > but I remember that somebody else had this as well and if I recall > > correctly it disappeared after the full upgrade was done.. but at the > > moment it's hard to think about upgrading everything at once.. > > (~12hours 100% disk utilization on all 12 nodes will lead to real slow > > puts/gets) > > > > What can I do? > > > > Cheers > > Simon > > > > PS: transfers output: > > > > 'riak@10.46.109.202' waiting to handoff 17 partitions > > 'riak@10.46.109.201' waiting to handoff 19 partitions > > > > (these are the 1.4.2 nodes) > > > > > > On Wed, 11 Dec 2013 14:39:58 +0100 > > Simon Effenberg wrote: > > > >> Also some side notes: > >> > >> "top" is even better on new 1.4.2 than on 1.3.1 machines.. IO > >> utilization of disk is mostly the same (round about 33%).. > >> > >> but > >> > >> 95th percentile of response time for get (avg over all nodes): > >> before upgrade: 29ms > >> after upgrade: almost the same > >> > >> 95th percentile of response time for put (avg over all nodes): > >> before upgrade: 60ms > >> after upgrade: 1548ms > >>but this is only because of 2 of 12 nodes are > >>on 1.4.2 and are really slow (17000ms) > >> > >> Cheers, > >> Simon > >> > >> On Wed, 11 Dec 2013 13:45:56 +0100 > >> Simon Effenberg wrote: > >> > >>> Sorry I forgot the half of it.. > >>> > >>> seffenberg@kriak46-1:~$ free -m > >>> total used free sharedbuffers cached > >>> Mem: 23999 23759239 0184 16183 > >>> -/+ buffers/cache: 7391 16607 > >>> Swap:0 0 0 > >>> > >>> We have 12 servers.. > >>> datadir on the compacted servers (1.4.2) ~ 765 GB > >>> > >>> AAE is enabled. > >>> > >>> I attached app.config and vm.args. > >>> > >>> Cheers > >>> Simon > >>> > >>> On Wed, 11 Dec 2013 07:33:31 -0500 > >>> Matthew Von-Maszewski wrote: > >>> > Ok, I am now suspecting that your servers are either using swap space > (which is slow) or your leveldb file cache is thrashing (opening and > closing multiple files per request). > > How many servers do you have and do you use Riak's active anti-entropy > feature? I am going to plug all of this into a spreadsheet. > > Matthew Von-Maszewski > > > On Dec 11, 2013, at 7:09, Simon Effenberg > wrote: > > > Hi Matthew > > > > Memory: 23999 MB > > > > ring_creation_size, 256 > > max_open_files, 100 > > > > riak-admin status: > > > > memory_total : 276001360 > > memory_processes : 191506322 > > memory_processes_used : 191439568 > > memory_system : 84495038 > > memory_atom : 686993 > > memory_atom_used : 686560 > > memory_binary : 21965352 > > memory_code : 11332732 > > memory_ets : 10823528 > > > > Thanks for look
Re: RiakCS usage data not found
Gavin,
After some more digging, it looks like the issue you're facing is an
open issue against Riak CS:
https://github.com/basho/riak_cs/issues/746
A pull request for the issue has been supplied and will be in the next release:
https://github.com/basho/riak_cs/pull/747
--
Hector
On Wed, Dec 11, 2013 at 5:22 AM, Gavin Huang wrote:
> Thanks Hector
> I actually did the same steps as you mentioned expect I run Riak-CS in a
> real virtual machine.
>
> When I looking into the error log of Riak-CS when I doing riak-cs-access
> flush an error was thrown (when doing other file operations are fine):
>
> 2013-12-11 10:14:56.442 [error] <0.12276.0> gen_fsm <0.12276.0> in state
> idle terminated with reason: bad argument in gen_fsm:send_event/2 line 198
> 2013-12-11 10:14:56.442 [error] <0.12276.0> CRASH REPORT Process <0.12276.0>
> with 0 neighbours exited with reason: bad argument in gen_fsm:send_event/2
> line 198 in gen_fsm:terminate/7 line 611
>
> when doing riak-cs-storage batch no error was thrown with this message:
>
> 2013-12-11 10:13:46.097 [info] <0.286.0>@riak_cs_storage_d:calculating:150
> Finished storage calculation in 0 seconds.
>
>
> I tried Riak-CS-1.4.0 and latest Riak-CS, both get no luck. was that some
> configuration issue?
>
> Thanks.
> Gavin
>
>
>
> On Mon, Dec 9, 2013 at 10:08 PM, Hector Castro wrote:
>>
>> Hi Gavin,
>>
>> How have you configured your `storage_schedule`? It's possible that
>> the job for calculating storage statistics hasn't occurred (or
>> finished) yet.
>>
>> You can manually trigger storage usage calculation by invoking the
>> following command:
>>
>> sudo riak-cs-storage batch
>>
>> From there, you can retrieve statistics from the `usage` bucket with
>> `s3cmd`:
>>
>> s3cmd get
>> s3://riak-cs/usage/.abj.20130501T00Z.20130531T00Z
>>
>> Also, here is a script I wrote up to access usage and storage
>> statistics with `s3cmd`:
>>
>> https://gist.github.com/hectcastro/126b5657f228096775c6
>>
>> --
>> Hector
>>
>>
>> On Mon, Dec 9, 2013 at 1:25 AM, Gavin Huang
>> wrote:
>> > BTW, what I can see from the riak-cs log is:
>> >
>> > 2013-12-09 22:04:40.836 [error] <0.17758.42> No WM route: 'GET'
>> >
>> > /riak-cs/usage/GRRYE2NHIXTNPFP89NMJ/abj/20131210T00Z/20131210T16Z
>> > {5,{"x-amz-date",{"X-Amz-Date","Mon, 09 Dec 2013 06:10:17
>> > +"},{"authorization",{'Authorization',"AWS
>> >
>> > MESG2VCANJVU7XTX-9YC:K7v4mFdvvAZ8a5LJHqfc/TZLy/w="},{"accept-encoding",{'Accept-Encoding',"identity"},nil,nil},{"host",{'Host',"riak-cs.s3.amazonaws.com:8080"},nil,nil}},{"x-rcs-rewrite-path",{"x-rcs-rewrite-path","/riak-cs/usage/GRRYE2NHIXTNPFP89NMJ/abj/20131210T00Z/20131210T16Z"},nil,nil}}}
>> >
>> >
>> >
>> > On Mon, Dec 9, 2013 at 10:50 AM, Gavin Huang
>> > wrote:
>> >>
>> >> Hi,
>> >> I'm trying to get the usage and access statistics from Riak-CS.
>> >> I configured "storage_schedule"(restarted riak-cs), I manually start
>> >> riak-cs-storage (it takes a while first time, but getting more quick
>> >> afterwards).
>> >>
>> >> I tried to access the storage via s3cmd:
>> >> s3cmd get s3://riak-cs/usage/[adminkey]
>> >> s3cmd get s3://riak-cs/usage/[adminkey]/aj
>> >> s3cmd get
>> >> s3://riak-cs/usage/[adminkey]/abj/20131201T14Z/20131210T16Z
>> >>
>> >> all request get the response of:
>> >> ERROR: S3 error: 404 (Object Not Found):
>> >>
>> >> is there anything I missed?
>> >>
>> >> Thanks.
>> >> Gavin
>> >
>> >
>> >
>> > ___
>> > riak-users mailing list
>> > riak-users@lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
bucket props question
Hi All,
Is it at all possible to have default bucket props be based on the
bucket names? I ask because I'm trying to use Riak to store key/value
data and use the buckets to separate the data based on day and for a
couple of different projects, for example:
buckets/project-year-month-day/key
The issue becomes I have one project that needs 'allow_mult' to true,
and another that needs 'last_write_wins' set to true (requiring that
'allow_mult' be false). So using the default_buckets_props will only get
me half way there.
Does anyone know of a way I can do this? The only solution I can think
of off the top of my head is to create a cron job that creates the
bucket sooner than when the clients start writing to it.
Thank you in advance,
Bryce
PS -- can anyone tell me what:
{multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]}}
does?
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
Hi Matthew,
On Wed, 11 Dec 2013 18:38:49 +0100
Matthew Von-Maszewski wrote:
> Simon,
>
> I have plugged your various values into the attached spreadsheet. I assumed
> a vnode count to allow for one of your twelve servers to die (256 ring size /
> 11 servers).
Great, thanks!
>
> The spreadsheet suggests you can safely raise your max_open_files from 100 to
> 170. I would suggest doing this for the next server you upgrade. If part of
> your problem is file cache thrashing, you should see an improvement.
I will try this out.. starting the next server in 3-4 hours.
>
> Only if max_open_files helps, you should then consider adding
> {sst_block_size, 32767} to the eleveldb portion of app.config. This setting,
> given your value sizes, would likely half the size of the metadata held in
> the file cache. It only impacts the files newly compacted in the upgrade,
> and would gradually increase space in the file cache while slowing down the
> file cache thrashing.
So I'll do this at the over-next server if the next server is fine.
>
> What build/packaging of Riak do you use, or do you build from source?
Using the debian packages from the basho site..
I'm really wondering why the "put" performance is that bad.
Here are the changes which were introduced/changed only on the new
upgraded servers:
+fsm_limit => 5,
--- our '+P' is set to 262144 so more than 3x fsm_limit which was
--- stated somewhere
+# after finishing the upgrade this should be switched to v1 !!!
+object_format => '__atom_v0',
- '-env ERL_MAX_ETS_TABLES' => 8192,
+ '-env ERL_MAX_ETS_TABLES' => 256000, # old package used 8192
but 1.4.2 raised it to this high number
+ '-env ERL_MAX_PORTS' => 64000,
+ # Treat error_logger warnings as warnings
+ '+W' => 'w',
+ # Tweak GC to run more often
+ '-env ERL_FULLSWEEP_AFTER' => 0,
+ # Force the erlang VM to use SMP
+ '-smp' => 'enable',
+ #
Cheers
Simon
>
> Matthew
>
>
>
> On Dec 11, 2013, at 9:48 AM, Simon Effenberg
> wrote:
>
> > Hi Matthew,
> >
> > thanks for all your time and work.. see inline for answers..
> >
> > On Wed, 11 Dec 2013 09:17:32 -0500
> > Matthew Von-Maszewski wrote:
> >
> >> The real Riak developers have arrived on-line for the day. They are
> >> telling me that all of your problems are likely due to the extended
> >> upgrade times, and yes there is a known issue with handoff between 1.3 and
> >> 1.4. They also say everything should calm down after all nodes are
> >> upgraded.
> >>
> >> I will review your system settings now and see if there is something that
> >> might make the other machines upgrade quicker. So three more questions:
> >>
> >> - what is the average size of your keys
> >
> > bucket names are between 5 and 15 characters (only ~ 10 buckets)..
> > key names are normally something like 26iesj:hovh7egz
> >
> >>
> >> - what is the average size of your value (data stored)
> >
> > I have to guess.. but mean is (from Riak) 12kb but 95th percentile is
> > at 75kb and in theory we have a limit of 1MB (then it will be split up)
> > but sometimes thanks to sibblings (we have to buckets with allow_mult)
> > we have also some 7MB in MAX but this will be reduced again (it's a new
> > feature in our app which has to many parallel wrights below of 15ms).
> >
> >>
> >> - in regular use, are your keys accessed randomly across their entire
> >> range, or do they contain a date component which clusters older, less used
> >> keys
> >
> > normally we don't search but retrieve keys by key name.. and we have
> > data which is up to 6 months old and normally we access mostly
> > new/active/hot data and not all the old ones.. besides this we have a
> > job doing a 2i query every 5mins and another one doing this maybe once
> > an hour.. both don't work while the upgrade is ongoing (2i isn't
> > working).
> >
> > Cheers
> > Simon
> >
> >>
> >> Matthew
> >>
> >>
> >> On Dec 11, 2013, at 8:43 AM, Simon Effenberg
> >> wrote:
> >>
> >>> Oh and at the moment they are waiting for some handoffs and I see
> >>> errors in logfiles:
> >>>
> >>>
> >>> 2013-12-11 13:41:47.948 UTC [error]
> >>> <0.7157.24>@riak_core_handoff_sender:start_fold:269 hinted_handoff
> >>> transfer of riak_kv_vnode from 'riak@10.46.109.202'
> >>> 468137243207554840987117797979434404733540892672
> >>>
> >>> but I remember that somebody else had this as well and if I recall
> >>> correctly it disappeared after the full upgrade was done.. but at the
> >>> moment it's hard to think about upgrading everything at once..
> >>> (~12hours 100% disk utilization on all 12 nodes will lead to real slow
> >>> puts/gets)
> >>>
> >>> What can I do?
> >>>
> >>> Cheers
> >>> Simon
> >>>
> >>> PS: transfers output:
> >>>
> >>> 'riak@10.46.109.202' waiting to handoff 17 partitions
> >>> 'riak@10.46.109.201' waiting to handoff 19
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
An additional thought: if increasing max_open_files does NOT help, try
removing +S 4:4 from the vm.args. Typically +S setting helps leveldb, but one
other user mentioned that the new sorted 2i queries needed more CPU in the
Erlang layer.
Summary:
- try increasing max_open_files to 170
- helps: try setting sst_block_size to 32768 in app.config
- does not help: try removing +S from vm.args
Matthew
On Dec 11, 2013, at 1:58 PM, Simon Effenberg wrote:
> Hi Matthew,
>
> On Wed, 11 Dec 2013 18:38:49 +0100
> Matthew Von-Maszewski wrote:
>
>> Simon,
>>
>> I have plugged your various values into the attached spreadsheet. I assumed
>> a vnode count to allow for one of your twelve servers to die (256 ring size
>> / 11 servers).
>
> Great, thanks!
>
>>
>> The spreadsheet suggests you can safely raise your max_open_files from 100
>> to 170. I would suggest doing this for the next server you upgrade. If
>> part of your problem is file cache thrashing, you should see an improvement.
>
> I will try this out.. starting the next server in 3-4 hours.
>
>>
>> Only if max_open_files helps, you should then consider adding
>> {sst_block_size, 32767} to the eleveldb portion of app.config. This
>> setting, given your value sizes, would likely half the size of the metadata
>> held in the file cache. It only impacts the files newly compacted in the
>> upgrade, and would gradually increase space in the file cache while slowing
>> down the file cache thrashing.
>
> So I'll do this at the over-next server if the next server is fine.
>
>>
>> What build/packaging of Riak do you use, or do you build from source?
>
> Using the debian packages from the basho site..
>
> I'm really wondering why the "put" performance is that bad.
> Here are the changes which were introduced/changed only on the new
> upgraded servers:
>
>
> +fsm_limit => 5,
> --- our '+P' is set to 262144 so more than 3x fsm_limit which was
> --- stated somewhere
> +# after finishing the upgrade this should be switched to v1 !!!
> +object_format => '__atom_v0',
>
> - '-env ERL_MAX_ETS_TABLES' => 8192,
> + '-env ERL_MAX_ETS_TABLES' => 256000, # old package used 8192
> but 1.4.2 raised it to this high number
> + '-env ERL_MAX_PORTS' => 64000,
> + # Treat error_logger warnings as warnings
> + '+W' => 'w',
> + # Tweak GC to run more often
> + '-env ERL_FULLSWEEP_AFTER' => 0,
> + # Force the erlang VM to use SMP
> + '-smp' => 'enable',
> + #
>
> Cheers
> Simon
>
>
>>
>> Matthew
>>
>>
>>
>> On Dec 11, 2013, at 9:48 AM, Simon Effenberg
>> wrote:
>>
>>> Hi Matthew,
>>>
>>> thanks for all your time and work.. see inline for answers..
>>>
>>> On Wed, 11 Dec 2013 09:17:32 -0500
>>> Matthew Von-Maszewski wrote:
>>>
The real Riak developers have arrived on-line for the day. They are
telling me that all of your problems are likely due to the extended
upgrade times, and yes there is a known issue with handoff between 1.3 and
1.4. They also say everything should calm down after all nodes are
upgraded.
I will review your system settings now and see if there is something that
might make the other machines upgrade quicker. So three more questions:
- what is the average size of your keys
>>>
>>> bucket names are between 5 and 15 characters (only ~ 10 buckets)..
>>> key names are normally something like 26iesj:hovh7egz
>>>
- what is the average size of your value (data stored)
>>>
>>> I have to guess.. but mean is (from Riak) 12kb but 95th percentile is
>>> at 75kb and in theory we have a limit of 1MB (then it will be split up)
>>> but sometimes thanks to sibblings (we have to buckets with allow_mult)
>>> we have also some 7MB in MAX but this will be reduced again (it's a new
>>> feature in our app which has to many parallel wrights below of 15ms).
>>>
- in regular use, are your keys accessed randomly across their entire
range, or do they contain a date component which clusters older, less used
keys
>>>
>>> normally we don't search but retrieve keys by key name.. and we have
>>> data which is up to 6 months old and normally we access mostly
>>> new/active/hot data and not all the old ones.. besides this we have a
>>> job doing a 2i query every 5mins and another one doing this maybe once
>>> an hour.. both don't work while the upgrade is ongoing (2i isn't
>>> working).
>>>
>>> Cheers
>>> Simon
>>>
Matthew
On Dec 11, 2013, at 8:43 AM, Simon Effenberg
wrote:
> Oh and at the moment they are waiting for some handoffs and I see
> errors in logfiles:
>
>
> 2013-12-11 13:41:47.948 UTC [error]
> <0.7157.24>@riak_core_handoff_sender:start_fold:269 hinted_handoff
> transfer of riak_kv_vnode from
Re: bucket props question
Hi Bryce,
You generally want to avoid creating too many buckets with custom bucket
properties, as this gets stored in Riak's ring data. A large number of
custom buckets will degrade cluster performance.
Is there any reason why you don't just create two buckets, with the desired
LWW/allow_mult=true settings and then prepend the timestamp to your key?
Tom
On Wed, Dec 11, 2013 at 1:47 PM, Bryce Verdier wrote:
> Hi All,
>
> Is it at all possible to have default bucket props be based on the bucket
> names? I ask because I'm trying to use Riak to store key/value data and use
> the buckets to separate the data based on day and for a couple of different
> projects, for example:
>
> buckets/project-year-month-day/key
>
> The issue becomes I have one project that needs 'allow_mult' to true, and
> another that needs 'last_write_wins' set to true (requiring that
> 'allow_mult' be false). So using the default_buckets_props will only get me
> half way there.
>
> Does anyone know of a way I can do this? The only solution I can think of
> off the top of my head is to create a cron job that creates the bucket
> sooner than when the clients start writing to it.
>
> Thank you in advance,
>
> Bryce
>
> PS -- can anyone tell me what:
> {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]}}
> does?
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
I will do..
but one other thing:
watch Every 10.0s: sudo riak-admin status | grep put_fsm
node_put_fsm_time_mean : 2208050
node_put_fsm_time_median : 39231
node_put_fsm_time_95 : 17400382
node_put_fsm_time_99 : 50965752
node_put_fsm_time_100 : 59537762
node_put_fsm_active : 5
node_put_fsm_active_60s : 364
node_put_fsm_in_rate : 5
node_put_fsm_out_rate : 3
node_put_fsm_rejected : 0
node_put_fsm_rejected_60s : 0
node_put_fsm_rejected_total : 0
this is not changing at all.. so maybe my expectations are _wrong_?! So
I will start searching around if there is a "status" bug or I'm
looking in the wrong place... maybe there is no problem while searching
for one?! But I see that at least the app has some issues on GET and
PUT (more on PUT).. so I would like to know how fast the things are..
but "status" isn't working.. argh...
Cheers
Simon
On Wed, 11 Dec 2013 14:32:07 -0500
Matthew Von-Maszewski wrote:
> An additional thought: if increasing max_open_files does NOT help, try
> removing +S 4:4 from the vm.args. Typically +S setting helps leveldb, but
> one other user mentioned that the new sorted 2i queries needed more CPU in
> the Erlang layer.
>
> Summary:
> - try increasing max_open_files to 170
> - helps: try setting sst_block_size to 32768 in app.config
> - does not help: try removing +S from vm.args
>
> Matthew
>
> On Dec 11, 2013, at 1:58 PM, Simon Effenberg
> wrote:
>
> > Hi Matthew,
> >
> > On Wed, 11 Dec 2013 18:38:49 +0100
> > Matthew Von-Maszewski wrote:
> >
> >> Simon,
> >>
> >> I have plugged your various values into the attached spreadsheet. I
> >> assumed a vnode count to allow for one of your twelve servers to die (256
> >> ring size / 11 servers).
> >
> > Great, thanks!
> >
> >>
> >> The spreadsheet suggests you can safely raise your max_open_files from 100
> >> to 170. I would suggest doing this for the next server you upgrade. If
> >> part of your problem is file cache thrashing, you should see an
> >> improvement.
> >
> > I will try this out.. starting the next server in 3-4 hours.
> >
> >>
> >> Only if max_open_files helps, you should then consider adding
> >> {sst_block_size, 32767} to the eleveldb portion of app.config. This
> >> setting, given your value sizes, would likely half the size of the
> >> metadata held in the file cache. It only impacts the files newly
> >> compacted in the upgrade, and would gradually increase space in the file
> >> cache while slowing down the file cache thrashing.
> >
> > So I'll do this at the over-next server if the next server is fine.
> >
> >>
> >> What build/packaging of Riak do you use, or do you build from source?
> >
> > Using the debian packages from the basho site..
> >
> > I'm really wondering why the "put" performance is that bad.
> > Here are the changes which were introduced/changed only on the new
> > upgraded servers:
> >
> >
> > +fsm_limit => 5,
> > --- our '+P' is set to 262144 so more than 3x fsm_limit which was
> > --- stated somewhere
> > +# after finishing the upgrade this should be switched to v1 !!!
> > +object_format => '__atom_v0',
> >
> > - '-env ERL_MAX_ETS_TABLES' => 8192,
> > + '-env ERL_MAX_ETS_TABLES' => 256000, # old package used 8192
> > but 1.4.2 raised it to this high number
> > + '-env ERL_MAX_PORTS' => 64000,
> > + # Treat error_logger warnings as warnings
> > + '+W' => 'w',
> > + # Tweak GC to run more often
> > + '-env ERL_FULLSWEEP_AFTER' => 0,
> > + # Force the erlang VM to use SMP
> > + '-smp' => 'enable',
> > + #
> >
> > Cheers
> > Simon
> >
> >
> >>
> >> Matthew
> >>
> >>
> >>
> >> On Dec 11, 2013, at 9:48 AM, Simon Effenberg
> >> wrote:
> >>
> >>> Hi Matthew,
> >>>
> >>> thanks for all your time and work.. see inline for answers..
> >>>
> >>> On Wed, 11 Dec 2013 09:17:32 -0500
> >>> Matthew Von-Maszewski wrote:
> >>>
> The real Riak developers have arrived on-line for the day. They are
> telling me that all of your problems are likely due to the extended
> upgrade times, and yes there is a known issue with handoff between 1.3
> and 1.4. They also say everything should calm down after all nodes are
> upgraded.
>
> I will review your system settings now and see if there is something
> that might make the other machines upgrade quicker. So three more
> questions:
>
> - what is the average size of your keys
> >>>
> >>> bucket names are between 5 and 15 characters (only ~ 10 buckets)..
> >>> key names are normally something like 26iesj:hovh7egz
> >>>
>
> - what is the average size of your value (data stored)
> >>>
> >>> I have to guess.. but mean is (from Riak) 12kb but 95th percentile is
> >>> at 75kb and in theory we have a limit of 1MB (then it will be split up)
> >>> but sometimes
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
The real Riak developers have suggested this might be your problem with stats
being stuck:
https://github.com/basho/riak_core/pull/467
The fix is included in the upcoming 1.4.4 maintenance release (which is overdue
so I am not going to bother guessing when it will actually arrive).
Matthew
On Dec 11, 2013, at 2:47 PM, Simon Effenberg wrote:
> I will do..
>
> but one other thing:
>
> watch Every 10.0s: sudo riak-admin status | grep put_fsm
> node_put_fsm_time_mean : 2208050
> node_put_fsm_time_median : 39231
> node_put_fsm_time_95 : 17400382
> node_put_fsm_time_99 : 50965752
> node_put_fsm_time_100 : 59537762
> node_put_fsm_active : 5
> node_put_fsm_active_60s : 364
> node_put_fsm_in_rate : 5
> node_put_fsm_out_rate : 3
> node_put_fsm_rejected : 0
> node_put_fsm_rejected_60s : 0
> node_put_fsm_rejected_total : 0
>
> this is not changing at all.. so maybe my expectations are _wrong_?! So
> I will start searching around if there is a "status" bug or I'm
> looking in the wrong place... maybe there is no problem while searching
> for one?! But I see that at least the app has some issues on GET and
> PUT (more on PUT).. so I would like to know how fast the things are..
> but "status" isn't working.. argh...
>
> Cheers
> Simon
>
>
> On Wed, 11 Dec 2013 14:32:07 -0500
> Matthew Von-Maszewski wrote:
>
>> An additional thought: if increasing max_open_files does NOT help, try
>> removing +S 4:4 from the vm.args. Typically +S setting helps leveldb, but
>> one other user mentioned that the new sorted 2i queries needed more CPU in
>> the Erlang layer.
>>
>> Summary:
>> - try increasing max_open_files to 170
>> - helps: try setting sst_block_size to 32768 in app.config
>> - does not help: try removing +S from vm.args
>>
>> Matthew
>>
>> On Dec 11, 2013, at 1:58 PM, Simon Effenberg
>> wrote:
>>
>>> Hi Matthew,
>>>
>>> On Wed, 11 Dec 2013 18:38:49 +0100
>>> Matthew Von-Maszewski wrote:
>>>
Simon,
I have plugged your various values into the attached spreadsheet. I
assumed a vnode count to allow for one of your twelve servers to die (256
ring size / 11 servers).
>>>
>>> Great, thanks!
>>>
The spreadsheet suggests you can safely raise your max_open_files from 100
to 170. I would suggest doing this for the next server you upgrade. If
part of your problem is file cache thrashing, you should see an
improvement.
>>>
>>> I will try this out.. starting the next server in 3-4 hours.
>>>
Only if max_open_files helps, you should then consider adding
{sst_block_size, 32767} to the eleveldb portion of app.config. This
setting, given your value sizes, would likely half the size of the
metadata held in the file cache. It only impacts the files newly
compacted in the upgrade, and would gradually increase space in the file
cache while slowing down the file cache thrashing.
>>>
>>> So I'll do this at the over-next server if the next server is fine.
>>>
What build/packaging of Riak do you use, or do you build from source?
>>>
>>> Using the debian packages from the basho site..
>>>
>>> I'm really wondering why the "put" performance is that bad.
>>> Here are the changes which were introduced/changed only on the new
>>> upgraded servers:
>>>
>>>
>>> +fsm_limit => 5,
>>> --- our '+P' is set to 262144 so more than 3x fsm_limit which was
>>> --- stated somewhere
>>> +# after finishing the upgrade this should be switched to v1 !!!
>>> +object_format => '__atom_v0',
>>>
>>> - '-env ERL_MAX_ETS_TABLES' => 8192,
>>> + '-env ERL_MAX_ETS_TABLES' => 256000, # old package used 8192
>>> but 1.4.2 raised it to this high number
>>> + '-env ERL_MAX_PORTS' => 64000,
>>> + # Treat error_logger warnings as warnings
>>> + '+W' => 'w',
>>> + # Tweak GC to run more often
>>> + '-env ERL_FULLSWEEP_AFTER' => 0,
>>> + # Force the erlang VM to use SMP
>>> + '-smp' => 'enable',
>>> + #
>>>
>>> Cheers
>>> Simon
>>>
>>>
Matthew
On Dec 11, 2013, at 9:48 AM, Simon Effenberg
wrote:
> Hi Matthew,
>
> thanks for all your time and work.. see inline for answers..
>
> On Wed, 11 Dec 2013 09:17:32 -0500
> Matthew Von-Maszewski wrote:
>
>> The real Riak developers have arrived on-line for the day. They are
>> telling me that all of your problems are likely due to the extended
>> upgrade times, and yes there is a known issue with handoff between 1.3
>> and 1.4. They also say everything should calm down after all nodes are
>> upgraded.
>>
>> I will review your system settings now and see if there is something
>> that might make the other machines upgrade quicker. So three more
>> questions:
>>
>> - what is
Re: bucket props question
Hi Bryce,
Unfortunately Riak 2.0 final is not yet available but I would be curious to
know if the upcoming Bucket Types [1] [2] feature would help you model your
problem. You could create a Bucket Type for your allow_mult=true projects
and another for lww=true. so you would have something like (extending the
notation you used in your email):
* bucket_types/allowsibs/buckets/project-year-month-day/keys/key
* bucket_types/nosibs/buckets/project-year-month-day/keys/key
where "allowsibs" and "nosibs" are the two bucket types, respectively.
Cheers,
Jordan
[1]
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2013-November/013847.html
[2] https://github.com/basho/riak/issues/362
On Wed, Dec 11, 2013 at 11:45 AM, Tom Santero wrote:
> Hi Bryce,
>
> You generally want to avoid creating too many buckets with custom bucket
> properties, as this gets stored in Riak's ring data. A large number of
> custom buckets will degrade cluster performance.
>
> Is there any reason why you don't just create two buckets, with the
> desired LWW/allow_mult=true settings and then prepend the timestamp to your
> key?
>
> Tom
>
>
> On Wed, Dec 11, 2013 at 1:47 PM, Bryce Verdier wrote:
>
>> Hi All,
>>
>> Is it at all possible to have default bucket props be based on the bucket
>> names? I ask because I'm trying to use Riak to store key/value data and use
>> the buckets to separate the data based on day and for a couple of different
>> projects, for example:
>>
>> buckets/project-year-month-day/key
>>
>> The issue becomes I have one project that needs 'allow_mult' to true, and
>> another that needs 'last_write_wins' set to true (requiring that
>> 'allow_mult' be false). So using the default_buckets_props will only get me
>> half way there.
>>
>> Does anyone know of a way I can do this? The only solution I can think of
>> off the top of my head is to create a cron job that creates the bucket
>> sooner than when the clients start writing to it.
>>
>> Thank you in advance,
>>
>> Bryce
>>
>> PS -- can anyone tell me what:
>> {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]}}
>> does?
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: bucket props question
Based on my understanding of the Bucket Types feature - yes, this feature
would solve the problem.
---
Jeremiah Peschka - Founder, Brent Ozar Unlimited
MCITP: SQL Server 2008, MVP
Cloudera Certified Developer for Apache Hadoop
On Wed, Dec 11, 2013 at 12:39 PM, Jordan West wrote:
> Hi Bryce,
>
> Unfortunately Riak 2.0 final is not yet available but I would be curious
> to know if the upcoming Bucket Types [1] [2] feature would help you model
> your problem. You could create a Bucket Type for your allow_mult=true
> projects and another for lww=true. so you would have something like
> (extending the notation you used in your email):
>
> * bucket_types/allowsibs/buckets/project-year-month-day/keys/key
> * bucket_types/nosibs/buckets/project-year-month-day/keys/key
>
> where "allowsibs" and "nosibs" are the two bucket types, respectively.
>
> Cheers,
>
> Jordan
>
> [1]
> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2013-November/013847.html
> [2] https://github.com/basho/riak/issues/362
>
>
> On Wed, Dec 11, 2013 at 11:45 AM, Tom Santero wrote:
>
>> Hi Bryce,
>>
>> You generally want to avoid creating too many buckets with custom bucket
>> properties, as this gets stored in Riak's ring data. A large number of
>> custom buckets will degrade cluster performance.
>>
>> Is there any reason why you don't just create two buckets, with the
>> desired LWW/allow_mult=true settings and then prepend the timestamp to your
>> key?
>>
>> Tom
>>
>>
>> On Wed, Dec 11, 2013 at 1:47 PM, Bryce Verdier wrote:
>>
>>> Hi All,
>>>
>>> Is it at all possible to have default bucket props be based on the
>>> bucket names? I ask because I'm trying to use Riak to store key/value data
>>> and use the buckets to separate the data based on day and for a couple of
>>> different projects, for example:
>>>
>>> buckets/project-year-month-day/key
>>>
>>> The issue becomes I have one project that needs 'allow_mult' to true,
>>> and another that needs 'last_write_wins' set to true (requiring that
>>> 'allow_mult' be false). So using the default_buckets_props will only get me
>>> half way there.
>>>
>>> Does anyone know of a way I can do this? The only solution I can think
>>> of off the top of my head is to create a cron job that creates the bucket
>>> sooner than when the clients start writing to it.
>>>
>>> Thank you in advance,
>>>
>>> Bryce
>>>
>>> PS -- can anyone tell me what:
>>> {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]}}
>>> does?
>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
So I think I have no real chance to get good numbers. I can see a
little bit through the app monitoring but I'm not sure if I can see
real differences about the 100 -> 170 open_files increase.
I will try to change the value on the already migrated nodes as well to
see if this improves the stuff I can see..
Any other ideas?
Cheers
Simon
On Wed, 11 Dec 2013 15:37:03 -0500
Matthew Von-Maszewski wrote:
> The real Riak developers have suggested this might be your problem with stats
> being stuck:
>
> https://github.com/basho/riak_core/pull/467
>
> The fix is included in the upcoming 1.4.4 maintenance release (which is
> overdue so I am not going to bother guessing when it will actually arrive).
>
> Matthew
>
> On Dec 11, 2013, at 2:47 PM, Simon Effenberg
> wrote:
>
> > I will do..
> >
> > but one other thing:
> >
> > watch Every 10.0s: sudo riak-admin status | grep put_fsm
> > node_put_fsm_time_mean : 2208050
> > node_put_fsm_time_median : 39231
> > node_put_fsm_time_95 : 17400382
> > node_put_fsm_time_99 : 50965752
> > node_put_fsm_time_100 : 59537762
> > node_put_fsm_active : 5
> > node_put_fsm_active_60s : 364
> > node_put_fsm_in_rate : 5
> > node_put_fsm_out_rate : 3
> > node_put_fsm_rejected : 0
> > node_put_fsm_rejected_60s : 0
> > node_put_fsm_rejected_total : 0
> >
> > this is not changing at all.. so maybe my expectations are _wrong_?! So
> > I will start searching around if there is a "status" bug or I'm
> > looking in the wrong place... maybe there is no problem while searching
> > for one?! But I see that at least the app has some issues on GET and
> > PUT (more on PUT).. so I would like to know how fast the things are..
> > but "status" isn't working.. argh...
> >
> > Cheers
> > Simon
> >
> >
> > On Wed, 11 Dec 2013 14:32:07 -0500
> > Matthew Von-Maszewski wrote:
> >
> >> An additional thought: if increasing max_open_files does NOT help, try
> >> removing +S 4:4 from the vm.args. Typically +S setting helps leveldb, but
> >> one other user mentioned that the new sorted 2i queries needed more CPU in
> >> the Erlang layer.
> >>
> >> Summary:
> >> - try increasing max_open_files to 170
> >> - helps: try setting sst_block_size to 32768 in app.config
> >> - does not help: try removing +S from vm.args
> >>
> >> Matthew
> >>
> >> On Dec 11, 2013, at 1:58 PM, Simon Effenberg
> >> wrote:
> >>
> >>> Hi Matthew,
> >>>
> >>> On Wed, 11 Dec 2013 18:38:49 +0100
> >>> Matthew Von-Maszewski wrote:
> >>>
> Simon,
>
> I have plugged your various values into the attached spreadsheet. I
> assumed a vnode count to allow for one of your twelve servers to die
> (256 ring size / 11 servers).
> >>>
> >>> Great, thanks!
> >>>
>
> The spreadsheet suggests you can safely raise your max_open_files from
> 100 to 170. I would suggest doing this for the next server you upgrade.
> If part of your problem is file cache thrashing, you should see an
> improvement.
> >>>
> >>> I will try this out.. starting the next server in 3-4 hours.
> >>>
>
> Only if max_open_files helps, you should then consider adding
> {sst_block_size, 32767} to the eleveldb portion of app.config. This
> setting, given your value sizes, would likely half the size of the
> metadata held in the file cache. It only impacts the files newly
> compacted in the upgrade, and would gradually increase space in the file
> cache while slowing down the file cache thrashing.
> >>>
> >>> So I'll do this at the over-next server if the next server is fine.
> >>>
>
> What build/packaging of Riak do you use, or do you build from source?
> >>>
> >>> Using the debian packages from the basho site..
> >>>
> >>> I'm really wondering why the "put" performance is that bad.
> >>> Here are the changes which were introduced/changed only on the new
> >>> upgraded servers:
> >>>
> >>>
> >>> +fsm_limit => 5,
> >>> --- our '+P' is set to 262144 so more than 3x fsm_limit which was
> >>> --- stated somewhere
> >>> +# after finishing the upgrade this should be switched to v1 !!!
> >>> +object_format => '__atom_v0',
> >>>
> >>> - '-env ERL_MAX_ETS_TABLES' => 8192,
> >>> + '-env ERL_MAX_ETS_TABLES' => 256000, # old package used 8192
> >>> but 1.4.2 raised it to this high number
> >>> + '-env ERL_MAX_PORTS' => 64000,
> >>> + # Treat error_logger warnings as warnings
> >>> + '+W' => 'w',
> >>> + # Tweak GC to run more often
> >>> + '-env ERL_FULLSWEEP_AFTER' => 0,
> >>> + # Force the erlang VM to use SMP
> >>> + '-smp' => 'enable',
> >>> + #
> >>>
> >>> Cheers
> >>> Simon
> >>>
> >>>
>
> Matthew
>
>
>
> On Dec 11, 2013, at 9:48 AM, Simon Effenberg
> wrote:
>
> > Hi Matthew,
> >
> > thanks
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
One of the core developers says that the following line should stop the stats
process. It will then be automatically started, without the stuck data.
exit(whereis(riak_core_stat_calc_sup), kill), profit().
Matthew
On Dec 11, 2013, at 4:50 PM, Simon Effenberg wrote:
> So I think I have no real chance to get good numbers. I can see a
> little bit through the app monitoring but I'm not sure if I can see
> real differences about the 100 -> 170 open_files increase.
>
> I will try to change the value on the already migrated nodes as well to
> see if this improves the stuff I can see..
>
> Any other ideas?
>
> Cheers
> Simon
>
> On Wed, 11 Dec 2013 15:37:03 -0500
> Matthew Von-Maszewski wrote:
>
>> The real Riak developers have suggested this might be your problem with
>> stats being stuck:
>>
>> https://github.com/basho/riak_core/pull/467
>>
>> The fix is included in the upcoming 1.4.4 maintenance release (which is
>> overdue so I am not going to bother guessing when it will actually arrive).
>>
>> Matthew
>>
>> On Dec 11, 2013, at 2:47 PM, Simon Effenberg
>> wrote:
>>
>>> I will do..
>>>
>>> but one other thing:
>>>
>>> watch Every 10.0s: sudo riak-admin status | grep put_fsm
>>> node_put_fsm_time_mean : 2208050
>>> node_put_fsm_time_median : 39231
>>> node_put_fsm_time_95 : 17400382
>>> node_put_fsm_time_99 : 50965752
>>> node_put_fsm_time_100 : 59537762
>>> node_put_fsm_active : 5
>>> node_put_fsm_active_60s : 364
>>> node_put_fsm_in_rate : 5
>>> node_put_fsm_out_rate : 3
>>> node_put_fsm_rejected : 0
>>> node_put_fsm_rejected_60s : 0
>>> node_put_fsm_rejected_total : 0
>>>
>>> this is not changing at all.. so maybe my expectations are _wrong_?! So
>>> I will start searching around if there is a "status" bug or I'm
>>> looking in the wrong place... maybe there is no problem while searching
>>> for one?! But I see that at least the app has some issues on GET and
>>> PUT (more on PUT).. so I would like to know how fast the things are..
>>> but "status" isn't working.. argh...
>>>
>>> Cheers
>>> Simon
>>>
>>>
>>> On Wed, 11 Dec 2013 14:32:07 -0500
>>> Matthew Von-Maszewski wrote:
>>>
An additional thought: if increasing max_open_files does NOT help, try
removing +S 4:4 from the vm.args. Typically +S setting helps leveldb, but
one other user mentioned that the new sorted 2i queries needed more CPU in
the Erlang layer.
Summary:
- try increasing max_open_files to 170
- helps: try setting sst_block_size to 32768 in app.config
- does not help: try removing +S from vm.args
Matthew
On Dec 11, 2013, at 1:58 PM, Simon Effenberg
wrote:
> Hi Matthew,
>
> On Wed, 11 Dec 2013 18:38:49 +0100
> Matthew Von-Maszewski wrote:
>
>> Simon,
>>
>> I have plugged your various values into the attached spreadsheet. I
>> assumed a vnode count to allow for one of your twelve servers to die
>> (256 ring size / 11 servers).
>
> Great, thanks!
>
>>
>> The spreadsheet suggests you can safely raise your max_open_files from
>> 100 to 170. I would suggest doing this for the next server you upgrade.
>> If part of your problem is file cache thrashing, you should see an
>> improvement.
>
> I will try this out.. starting the next server in 3-4 hours.
>
>>
>> Only if max_open_files helps, you should then consider adding
>> {sst_block_size, 32767} to the eleveldb portion of app.config. This
>> setting, given your value sizes, would likely half the size of the
>> metadata held in the file cache. It only impacts the files newly
>> compacted in the upgrade, and would gradually increase space in the file
>> cache while slowing down the file cache thrashing.
>
> So I'll do this at the over-next server if the next server is fine.
>
>>
>> What build/packaging of Riak do you use, or do you build from source?
>
> Using the debian packages from the basho site..
>
> I'm really wondering why the "put" performance is that bad.
> Here are the changes which were introduced/changed only on the new
> upgraded servers:
>
>
> +fsm_limit => 5,
> --- our '+P' is set to 262144 so more than 3x fsm_limit which was
> --- stated somewhere
> +# after finishing the upgrade this should be switched to v1 !!!
> +object_format => '__atom_v0',
>
> - '-env ERL_MAX_ETS_TABLES' => 8192,
> + '-env ERL_MAX_ETS_TABLES' => 256000, # old package used 8192
> but 1.4.2 raised it to this high number
> + '-env ERL_MAX_PORTS' => 64000,
> + # Treat error_logger warnings as warnings
> + '+W' => 'w',
> + # Tweak GC to run more often
> + '-env ERL_FULLSWEEP_AFTER' => 0,
> + # Force the erlang VM to use SMP
Re: Upgrade from 1.3.1 to 1.4.2 => high IO
Cool..
gave me an exception about
** exception error: undefined shell command profit/0
but it worked and now I have new data.. thanks a lot!
Cheers
Simon
On Wed, 11 Dec 2013 17:05:29 -0500
Matthew Von-Maszewski wrote:
> One of the core developers says that the following line should stop the stats
> process. It will then be automatically started, without the stuck data.
>
> exit(whereis(riak_core_stat_calc_sup), kill), profit().
>
> Matthew
>
> On Dec 11, 2013, at 4:50 PM, Simon Effenberg
> wrote:
>
> > So I think I have no real chance to get good numbers. I can see a
> > little bit through the app monitoring but I'm not sure if I can see
> > real differences about the 100 -> 170 open_files increase.
> >
> > I will try to change the value on the already migrated nodes as well to
> > see if this improves the stuff I can see..
> >
> > Any other ideas?
> >
> > Cheers
> > Simon
> >
> > On Wed, 11 Dec 2013 15:37:03 -0500
> > Matthew Von-Maszewski wrote:
> >
> >> The real Riak developers have suggested this might be your problem with
> >> stats being stuck:
> >>
> >> https://github.com/basho/riak_core/pull/467
> >>
> >> The fix is included in the upcoming 1.4.4 maintenance release (which is
> >> overdue so I am not going to bother guessing when it will actually arrive).
> >>
> >> Matthew
> >>
> >> On Dec 11, 2013, at 2:47 PM, Simon Effenberg
> >> wrote:
> >>
> >>> I will do..
> >>>
> >>> but one other thing:
> >>>
> >>> watch Every 10.0s: sudo riak-admin status | grep put_fsm
> >>> node_put_fsm_time_mean : 2208050
> >>> node_put_fsm_time_median : 39231
> >>> node_put_fsm_time_95 : 17400382
> >>> node_put_fsm_time_99 : 50965752
> >>> node_put_fsm_time_100 : 59537762
> >>> node_put_fsm_active : 5
> >>> node_put_fsm_active_60s : 364
> >>> node_put_fsm_in_rate : 5
> >>> node_put_fsm_out_rate : 3
> >>> node_put_fsm_rejected : 0
> >>> node_put_fsm_rejected_60s : 0
> >>> node_put_fsm_rejected_total : 0
> >>>
> >>> this is not changing at all.. so maybe my expectations are _wrong_?! So
> >>> I will start searching around if there is a "status" bug or I'm
> >>> looking in the wrong place... maybe there is no problem while searching
> >>> for one?! But I see that at least the app has some issues on GET and
> >>> PUT (more on PUT).. so I would like to know how fast the things are..
> >>> but "status" isn't working.. argh...
> >>>
> >>> Cheers
> >>> Simon
> >>>
> >>>
> >>> On Wed, 11 Dec 2013 14:32:07 -0500
> >>> Matthew Von-Maszewski wrote:
> >>>
> An additional thought: if increasing max_open_files does NOT help, try
> removing +S 4:4 from the vm.args. Typically +S setting helps leveldb,
> but one other user mentioned that the new sorted 2i queries needed more
> CPU in the Erlang layer.
>
> Summary:
> - try increasing max_open_files to 170
> - helps: try setting sst_block_size to 32768 in app.config
> - does not help: try removing +S from vm.args
>
> Matthew
>
> On Dec 11, 2013, at 1:58 PM, Simon Effenberg
> wrote:
>
> > Hi Matthew,
> >
> > On Wed, 11 Dec 2013 18:38:49 +0100
> > Matthew Von-Maszewski wrote:
> >
> >> Simon,
> >>
> >> I have plugged your various values into the attached spreadsheet. I
> >> assumed a vnode count to allow for one of your twelve servers to die
> >> (256 ring size / 11 servers).
> >
> > Great, thanks!
> >
> >>
> >> The spreadsheet suggests you can safely raise your max_open_files from
> >> 100 to 170. I would suggest doing this for the next server you
> >> upgrade. If part of your problem is file cache thrashing, you should
> >> see an improvement.
> >
> > I will try this out.. starting the next server in 3-4 hours.
> >
> >>
> >> Only if max_open_files helps, you should then consider adding
> >> {sst_block_size, 32767} to the eleveldb portion of app.config. This
> >> setting, given your value sizes, would likely half the size of the
> >> metadata held in the file cache. It only impacts the files newly
> >> compacted in the upgrade, and would gradually increase space in the
> >> file cache while slowing down the file cache thrashing.
> >
> > So I'll do this at the over-next server if the next server is fine.
> >
> >>
> >> What build/packaging of Riak do you use, or do you build from source?
> >
> > Using the debian packages from the basho site..
> >
> > I'm really wondering why the "put" performance is that bad.
> > Here are the changes which were introduced/changed only on the new
> > upgraded servers:
> >
> >
> > +fsm_limit => 5,
> > --- our '+P' is set to 262144 so more than 3x fsm_limit which was
> > --- stated somewhere
> > +# after finishing the upgrade this should be switched to v1 !!!
> > +object_format
Fwd: 404 Error: Object Not Found
> > Can you please try running the following command from within the dev > directory: > > $ ./dev1/bin/riak ping > > When I run it locally using your exact configs (downloaded from > pastebin), I see: > > $ ./dev1/bin/riak ping > Node 'dev1@127.0.0.1\r' not responding to pings. > > (Note the \r) > > If I start dev2 (using the config populated by Riak's `make devrel > DEVNODES=2` command), I get: > > $ ./dev2/bin/riak ping > Node 'dev2@127.0.0.1' not responding to pings. > > Can you please ensure that all carriage returns are removed from your > Riak configuration files? > > Hector, Apologies for the delayed response. I've been traveling internationally over the last ten days and in my haste thought I had sent the reply, when I actually had only written the response. I do not have any carriage returns in my actual config file, I believe that was added when pasting to pastebin. When I run "./dev1/bin/riak ping" I get "pong". However, I'm running riak on ubuntu 12.10 on vagrant. And I have to start riak with "sudo" otherwise while it does start up, it does not bind to the configured ports. Thanks. -Ari ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: riak key sharding
Hi Georgio, There are many possible ways to do something like this. Riak CS in particular chunks large files into immutable data blocks, and has manifests pointing to those blocks to track versions of files. Manifests and blocks are each stored in their own riak object. There are some tricks around manifest merging that take into account the eventually consistent nature of Riak CS, and there are quite a few other gotchas around building a system like this on top of Riak, including undocumented performance hacks. Riak CS additionally provides S3 API compatibility and is a commercially supported product, built by people that also work on Riak and strive to make the product as fast and reliable as possible on a daily basis. And crazier than that, it's free and open source! It doesn't sound like your goal is to build an object store, but to build a good product on top of an object store. In this case, I'd say just use Riak CS and develop your application. If CS doesn't meet your measured needs for latency than build your own store, or better yet help fix Riak CS :) Best of luck, Andrew On Tue, Dec 10, 2013 at 5:29 AM, Georgio Pandarez wrote: > Hi, > > I have noticed that Riak-CS can shard (that is split) large keys > automatically across nodes. I would like to achieve a similar outcome with > Riak itself. Is there any best practice to achieve this? Could a portion of > Riak-CS be used or should I just bite the bullet and use Riak-CS? > > Latency is key for my application and I wanted to avoid the additional > layer Riak-CS provides. > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: RiakCS usage data not found
Thanks Hector
I got lucky today, after I tried RiakCS-1.4.3, I finally got storage and
access stat data. although the error message is still there.
I notice that if a config the storage schedule to 0600 when 05, it won't
execute one hours later even i restart riak-cs, but will execute after one
day, is that true and is it by design?
Thanks.
Gavin
On Wed, Dec 11, 2013 at 11:02 PM, Hector Castro wrote:
> Gavin,
>
> After some more digging, it looks like the issue you're facing is an
> open issue against Riak CS:
>
> https://github.com/basho/riak_cs/issues/746
>
> A pull request for the issue has been supplied and will be in the next
> release:
>
> https://github.com/basho/riak_cs/pull/747
>
> --
> Hector
>
>
> On Wed, Dec 11, 2013 at 5:22 AM, Gavin Huang
> wrote:
> > Thanks Hector
> > I actually did the same steps as you mentioned expect I run Riak-CS in a
> > real virtual machine.
> >
> > When I looking into the error log of Riak-CS when I doing riak-cs-access
> > flush an error was thrown (when doing other file operations are fine):
> >
> > 2013-12-11 10:14:56.442 [error] <0.12276.0> gen_fsm <0.12276.0> in state
> > idle terminated with reason: bad argument in gen_fsm:send_event/2 line
> 198
> > 2013-12-11 10:14:56.442 [error] <0.12276.0> CRASH REPORT Process
> <0.12276.0>
> > with 0 neighbours exited with reason: bad argument in
> gen_fsm:send_event/2
> > line 198 in gen_fsm:terminate/7 line 611
> >
> > when doing riak-cs-storage batch no error was thrown with this message:
> >
> > 2013-12-11 10:13:46.097 [info]
> <0.286.0>@riak_cs_storage_d:calculating:150
> > Finished storage calculation in 0 seconds.
> >
> >
> > I tried Riak-CS-1.4.0 and latest Riak-CS, both get no luck. was that some
> > configuration issue?
> >
> > Thanks.
> > Gavin
> >
> >
> >
> > On Mon, Dec 9, 2013 at 10:08 PM, Hector Castro wrote:
> >>
> >> Hi Gavin,
> >>
> >> How have you configured your `storage_schedule`? It's possible that
> >> the job for calculating storage statistics hasn't occurred (or
> >> finished) yet.
> >>
> >> You can manually trigger storage usage calculation by invoking the
> >> following command:
> >>
> >> sudo riak-cs-storage batch
> >>
> >> From there, you can retrieve statistics from the `usage` bucket with
> >> `s3cmd`:
> >>
> >> s3cmd get
> >> s3://riak-cs/usage/.abj.20130501T00Z.20130531T00Z
> >>
> >> Also, here is a script I wrote up to access usage and storage
> >> statistics with `s3cmd`:
> >>
> >> https://gist.github.com/hectcastro/126b5657f228096775c6
> >>
> >> --
> >> Hector
> >>
> >>
> >> On Mon, Dec 9, 2013 at 1:25 AM, Gavin Huang
> >> wrote:
> >> > BTW, what I can see from the riak-cs log is:
> >> >
> >> > 2013-12-09 22:04:40.836 [error] <0.17758.42> No WM route: 'GET'
> >> >
> >> >
> /riak-cs/usage/GRRYE2NHIXTNPFP89NMJ/abj/20131210T00Z/20131210T16Z
> >> > {5,{"x-amz-date",{"X-Amz-Date","Mon, 09 Dec 2013 06:10:17
> >> > +"},{"authorization",{'Authorization',"AWS
> >> >
> >> >
> MESG2VCANJVU7XTX-9YC:K7v4mFdvvAZ8a5LJHqfc/TZLy/w="},{"accept-encoding",{'Accept-Encoding',"identity"},nil,nil},{"host",{'Host',"
> riak-cs.s3.amazonaws.com:8080
> "},nil,nil}},{"x-rcs-rewrite-path",{"x-rcs-rewrite-path","/riak-cs/usage/GRRYE2NHIXTNPFP89NMJ/abj/20131210T00Z/20131210T16Z"},nil,nil}}}
> >> >
> >> >
> >> >
> >> > On Mon, Dec 9, 2013 at 10:50 AM, Gavin Huang
> >> > wrote:
> >> >>
> >> >> Hi,
> >> >> I'm trying to get the usage and access statistics from Riak-CS.
> >> >> I configured "storage_schedule"(restarted riak-cs), I manually start
> >> >> riak-cs-storage (it takes a while first time, but getting more quick
> >> >> afterwards).
> >> >>
> >> >> I tried to access the storage via s3cmd:
> >> >> s3cmd get s3://riak-cs/usage/[adminkey]
> >> >> s3cmd get s3://riak-cs/usage/[adminkey]/aj
> >> >> s3cmd get
> >> >> s3://riak-cs/usage/[adminkey]/abj/20131201T14Z/20131210T16Z
> >> >>
> >> >> all request get the response of:
> >> >> ERROR: S3 error: 404 (Object Not Found):
> >> >>
> >> >> is there anything I missed?
> >> >>
> >> >> Thanks.
> >> >> Gavin
> >> >
> >> >
> >> >
> >> > ___
> >> > riak-users mailing list
> >> > riak-users@lists.basho.com
> >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >> >
> >
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Can riak use multiple disk partitions ?
Hi all , I'm new to riak. and there are four disk partitions on our server, every partition is ITB. so I want to know if riak support multiple disk partition usage and how to configure it. thanks in advance ! -- 不学习，不知道 ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
build Riak 2 Java client (on windows)
Hi Riak users, I want to start checking out Riak 2 and for that I need the new Java client I downloaded it from Git but it will not build. It seems to be missing a few dependencies (One of them was protobuf which I actually downloaded and built but it did not sync) Is there anywhere I can download the Jar or get detailed instructions of how to build the project? Thanks, Shimon ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: build Riak 2 Java client (on windows)
Hi Shimon, As noted in the README, the new version of the Java client (v2.0) is a work in progress and not yet usable; while the core is mostly complete there's currently no user API. Work on that is progressing, and we hope to have a release candidate available at the end of the month. The current 1.4.2 version of the client will work with Riak 2.0 preview, but unfortunately does not support the new features in Riak 2.0 Thanks, - Roach On Wed, Dec 11, 2013 at 11:54 PM, Shimon Benattar wrote: > Hi Riak users, > > I want to start checking out Riak 2 and for that I need the new Java client > > I downloaded it from Git but it will not build. > It seems to be missing a few dependencies (One of them was protobuf which I > actually downloaded and built but it did not sync) > > Is there anywhere I can download the Jar or get detailed instructions of how > to build the project? > > Thanks, > > Shimon > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
