How to calculate storage statistics properly
Hello there
We have a Riak/Riak CS cluster and want to calculate S3 storage
statistics per user on a hourly basis. As far as I understand, the
proper way to do so is to set up a schedule in riak-cs app.config
{storage_schedule, []} section
We distribute configs to our nodes from a common template. Would it be
ok to add same {storage_schedule, [...]} on all nodes or only one node
should initiate storage recalculation at a time? How to set this
schedule up properly in a 'headless' way so no single node is
responsible for initiating recalculation?
--
Best regards
Anton Khalikov
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: very slow write speed to riak-cs
Hi Luke
Is it safe to allow clients to connect directly to Riak CS as your
suggestion is kinda against official docs
(http://docs.basho.com/riakcs/latest/cookbooks/configuration/Load-Balancing-and-Proxy-Configuration/)?
I also think our problem might be caused by ring_creation_size 256 while
we have only 8 nodes up currently (the cluster is planned to grow twice
in the future). Yes I am Stanislav's colleague :)
pb_backlog is set to 512 at the moment. Are you suggesting to lower it?
Luke Bakken
3 Apr 2014 20:18
Hi Stanislav,Could
you configure your clients to connect directly to Riak CS instead of
through your proxy? A colleague of mine suggested that the {error,closed} and 104 error code
in s3cmd appear to be caused by the connection between the client and
Riak CS being closed or terminated unexpectedly.
I would still recommend keeping {pb_backlog,
256} as that will allow for more simultaneous connections to be
handled going forward.
--Luke BakkenCSElbak...@basho.com
___riak-users
mailing listriak-users@lists.basho.comhttp://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
-- Best regards
Anton Khalikov
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: very slow write speed to riak-cs
Luke, it would be quite difficult to connect directly to riak cs nodes because riak cs is set up to run in DMZ so it would take us to reconfigure the whole network. So let's assume you are right in your thinking and it is proxy (tengine) causing our problems. What would you recommend to use as a proxy instead? Original nginx? haproxy? May be specific version of them? We decided to use tengine because of proxy_request_buffering feature which bypasses the original nginx behavior and allows it to pass PUT requests directly to backend nodes instead of buffering entire request body in a temp file before passing it to backend (http://wiki.nginx.org/HttpProxyModule). So tengine had twice as fast PUT performance comparing to nginx. About pt_backlog value: got it, thank you. -- Best regards Anton Khalikov ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: riak random block missing on GET after some hour
Hello Charles
We had the same problem with 1.4.8 so we decided to upgrade the whole
cluster to 1.4.10. No evidences of this problem yet
Charles Bijon <mailto:bijon.char...@gmail.com>
23 Jul 2014 15:55
hello,
we have a problem since 4 days.
The story :
We did a migration of our 38 old machines to 45 new servers to
increase our production capacity. To do this, we make update of RIAK
(1.4.8 -> 1.4.9 -> 1.4.10). Today we have Riak (1.4.10) riakcs (1.4.5)
and stanchion (1.4.3).
Now something strange appeared our storage : New data put on the Riak
cluster become corrupted over time ( AAE enabled and was enabled
during the migration ) .
We have this message in log : error]
<0.13320.0>@riak_cs_get_fsm:waiting_chunks:311 riak_cs_get_fsm: Cannot
get S3 <<"blabla">> <<"blabla/blabla/blabla/blabla.foo">> block#
{<<94,144,214,192,123,131,68,132,142,55,30,108,189,81,242,106>>,0}:
{error,notfound}
Yesterday I deactivated AAE to test if the problem continues and we
put the dataagain to rebuild storage partialy.
The riakdiag is ok and also the ring-status
Is someone already had this trouble ?
Is it advisable to go back to the 1.4.8 version ?
Is what I have to restore AAE? And under what conditions?
If i should back to the 1.4.8 version of Riak, how to without loose
something ? Is this the right approach? Is that corrupt data will
become viable after the rear back?
It's a little hell right now, I really need a helping hand.
Regards,
Charles
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
--
Best regards
Anton Khalikov
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Version compatibility
Hello Riak users and Basho guys I've found an interesting page today: http://docs.basho.com/riakcs/latest/cookbooks/Version-Compatibility/ It says the latest working and recommended combination of Riak CS packages is: riak=1.4.10, stanchion=1.4.5 and riak-cs=1.5.0. Unfortunately I am unable to find stanchion=1.4.5 package for Debian Wheezy. Apt just gives me the following choices: # apt-cache policy stanchion stanchion: Installed: 1.4.3-1 Candidate: 1.5.0-1 Version table: 1.5.0-1 0 500 http://apt.basho.com/ wheezy/main amd64 Packages *** 1.4.3-1 0 500 http://apt.basho.com/ wheezy/main amd64 Packages 100 /var/lib/dpkg/status 1.4.2-1 0 500 http://apt.basho.com/ wheezy/main amd64 Packages 1.4.1-1 0 500 http://apt.basho.com/ wheezy/main amd64 Packages 1.4.0-1 0 500 http://apt.basho.com/ wheezy/main amd64 Packages So I am not sure if I can upgrade to Riak CS 1.5.0 (we have 1.4.5 at the moment) and also should I upgrade Stanchion to 1.5.0 or leave 1.4.3 version. Any suggestions? -- Best regards Anton Khalikov ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Version compatibility
I see. Thank you! Eric Redmond <mailto:eredm...@basho.com> 4 Aug 2014 00:13 Sorry, stanchion 1.5.0 for CS 1.5.0, else stanchion 1.4.3 for CS 1.4.5 Eric Redmond, Engineer @ Basho On Sun, Aug 03, 2014 at 11:09 AM, Eric Redmond <mailto:eredm...@basho.com>> wrote: Eric Redmond <mailto:eredm...@basho.com> 4 Aug 2014 00:09 This is a documentation error. As of today, use versions 1.4.5. Eric Redmond, Engineer @ Basho On Sun, Aug 03, 2014 at 10:59 AM, Anton Khalikov <mailto:an...@khalikov.ru>> wrote: -- Best regards Anton Khalikov ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Hardware recommendations?
Hello there As our Riak CS cluster grows, we went through several performance problems which caused me to think we went a wrong way with planning our hardware nodes configuration probably. If we want to have an s3 cluster with 30-40 Tb capacity, 50-100 millions of keys (files) stored on few thousands of buckets with n_val=3 which way to choose hardware configuration for nodes would be better: 1. more nodes with less memory and hdd capacity each. Let's say 18 nodes with 40 gb ram and 8 tb storage per node 2. less nodes with more memory and hdd capacity each. Let's say 12 nodes with 64 gb ram and 12 tb storage per node 3. even less nodes with even more memory and hdd capacity each. Well, for example 6 nodes with 128 gb ram and 24 tb storage each Or is there a dependency between total cluster memory and total cluster storage capacity? What is our maximum storage capacity if we have 768 gb ram in total in our cluster in other words? -- Best regards Anton Khalikov ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Small bug in riak/riak-cs debian init script
Hello Basho guys
I think we have found a small bug in riak/riak-cs debian init scripts
(/etc/init.d/riak{,-cs}). When used in CRON, ie for restarting riak
process on a schedule, init scripts fail because they don't set PATH
variable to recommended value. Without setting proper PATH, init script
is unable to find start-stop-daemon.
Recommended value for PATH for debian init scripts can be found in
/etc/init.d/ssh and many other debian init scripts and looks like:
export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"
or
PATH=/sbin:/bin:/usr/sbin:/usr/bin
--
Best regards
Anton Khalikov
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
