RE: Riak CS Backup Solutions

ate to another cluster. 3. Create something yourself to work at the application level that would mirror all commands. Hope this helps, Nicholas From: riak-users On Behalf Of Bryan Hunt Sent: 25 April 2018 22:50 To: riak-users Subject: Re: Riak CS Backup Solutions Anthony, Your snapshot solution is

Re: Riak CS Backup Solutions

Anthony, Your snapshot solution is not bad if you just want a daily version you can roll back to for disaster recovery. I presume : 1. You’re running on VM’s 2. You stop the world at the time of the snapshot Riak CS makes multiple (3 by default) copies of your data and generally speaking,

Re: riak-cs fails to start after reimporting Docker container

Done. Here it is: https://github.com/basho/riak_cs/issues/1333 2017-03-08 16:21 GMT+01:00 Jon Brisbin : > Might be good to make sure this info is encapsulated into a GitHub issue >> so we don't loose track of it. >> >> >> In the end, the ONE thing to fix is the cumbersome error message. A >> prop

Re: riak-cs fails to start after reimporting Docker container

> > Might be good to make sure this info is encapsulated into a GitHub issue > so we don't loose track of it. > > > In the end, the ONE thing to fix is the cumbersome error message. A propre > error message would have saved me days of "work"... > Should I create a new issue for that ? > Please do.

Re: riak-cs fails to start after reimporting Docker container

2017-03-08 15:53 GMT+01:00 Jon Brisbin : > FWIW- All work done so far on Riak-in-Docker is for purposes of testing > and ad-hoc cluster creation and NOT for use in "production" environments > (for any definition of the word "production"). Things like exposing the > conf files on a volume which are

Re: riak-cs fails to start after reimporting Docker container

FWIW- All work done so far on Riak-in-Docker is for purposes of testing and ad-hoc cluster creation and NOT for use in "production" environments (for any definition of the word "production"). Things like exposing the conf files on a volume which are more production-oriented haven't been in scope of

Re: riak-cs fails to start after reimporting Docker container

Hello, I finally found the problem : the riak and riak-cs config files where stored in my app container, not in my persistent data container (where the riak data is). Whenever I upgraded the app container to a newly generated one, the conf files and especially the admin user credentials did not ma

Re: riak-cs fails to start after reimporting Docker container

On 6 March 2017 at 03:49, Toby Corkindale wrote: > > I tried quite hard to get Riak to work reliably in a Docker container, in a long-term-use kind of way. > Riak would never shutdown cleanly, though, and so at startup there would always be lots of lock files left around that had to be deleted fir

Re: riak-cs fails to start after reimporting Docker container

I tried quite hard to get Riak to work reliably in a Docker container, in a long-term-use kind of way. Riak would never shutdown cleanly, though, and so at startup there would always be lots of lock files left around that had to be deleted first. Riak is not well-behaved after a rough shutdown --

Re: riak-cs fails to start after reimporting Docker container

I haven't tried CS in a container yet. Could you provide the Dockerfiles and compose files or the commands you use to start the services? jb On Wed, Feb 15, 2017 at 4:49 PM Jean-Marc Le Roux wrote: > Hi, > > inspecting the logs further, I get this in /etc/riak/console.log even > before running

Re: riak-cs fails to start after reimporting Docker container

Hi, inspecting the logs further, I get this in /etc/riak/console.log even before running riak-admin repair-2i : 2017-02-15 23:41:12.441 [warning] <0.714.0> Hintfile '/var/lib/riak/bitcask/205523667749658222872393179600727299639115513856/2.bitcask.hint' invalid 2017-02-15 23:41:12.441 [warning] <0

Re: riak-cs fails to start after reimporting Docker container

Forgot to mention ACLs are alright AFAIK : root@b4394bf1de78:/var/lib/riak# ls -la total 52 drwxr-xr-x. 10 riak riak 179 Feb 9 23:43 . drwxr-xr-x. 1 root root 95 Feb 15 20:48 .. -r. 1 riak riak 20 Feb 9 01:00 .erlang.cookie drwxrwxr-x. 67 riak riak 8192 Feb 15 21:31 anti_entropy d

Re: riak-cs fails to start after reimporting Docker container

Hi, I'll try to send the log archive ASAP. Here is what I get in /var/log/riak/error.log after running riak-admin repair-2i : 2017-02-15 22:09:06.535 [error] <0.3287.0>@riak_kv_2i_aae:repair_partition:297 Failed to acquire hashtree lock on partition 1255977969581244695331291653117200168170291

Re: riak-cs fails to start after reimporting Docker container

Hi Jean-Marc - Can you provide a complete archive of the log directory? I wonder if another file might have more information. -- Luke Bakken Engineer lbak...@basho.com On Thu, Feb 9, 2017 at 1:58 AM, Jean-Marc Le Roux wrote: > > Hello, > > here is the original github issue : > > https://github.

Re: Riak CS race condition at start-up (was: Riak-CS issues when Riak endpoint fails-over to new server)

Thanks Luke. We're using the official Basho Ubuntu packages for Ubuntu 14.04 LTS. These don't seem to perform that step; they just start Riak CS simultaneously with Riak KV. I'm surprised if we're the only people who hit this. Don't people hit these issues in your other production deployments, or

Re: Riak CS race condition at start-up (was: Riak-CS issues when Riak endpoint fails-over to new server)

Hi Toby, The process you use to run "riak-cs start" could use the "riak-admin wait-for-service riak_kv" command to ensure Riak is ready first: http://docs.basho.com/riak/kv/2.2.0/using/admin/riak-admin/#wait-for-service -- Luke Bakken Engineer lbak...@basho.com On Thu, Jan 19, 2017 at 5:38 PM,

Re: Riak CS - admin keys changing

Hi Toby, If you put another user into the config, that's all it takes to make them the admin user. There's no special value that's set in the database itself. Any user can be an admin user, it doesn't even have to be the first one created. It's just whatever user you have set in the config. Ki

Re: Riak CS - admin keys changing

Hi, I have a follow-up question around this security aspect. If the riak-cs.conf and stanchion.conf files are changed so that their admin.key and admin.secret match a different user (eg. not that first-created admin user) then will that user now have admin-like privileges? Or are the admin-abilit

Re: Riak CS - admin keys changing

Thanks, Luke! On Fri, 13 Jan 2017 at 12:10 Luke Bakken wrote: Hi Toby, When you create the user, the data is stored in Riak (and is the authoritative location). The values must match in the config files to provide credentials used when connecting to various parts of your CS cluster. -- Luke Ba

Re: Riak CS - admin keys changing

Hi Toby, When you create the user, the data is stored in Riak (and is the authoritative location). The values must match in the config files to provide credentials used when connecting to various parts of your CS cluster. -- Luke Bakken Engineer lbak...@basho.com On Thu, Jan 12, 2017 at 3:47 PM,

Re: Riak-CS issues when Riak endpoint fails-over to new server

Hi Shaun, We've been running Riak CS since its early days, so it's possible best practice has changed.. but I'm sure at some point it was suggested to put a haproxy between CS and KV to guard against the issues of start-up race condition, individual KV losses, and brief KV restarts. I'm sure we use

Re: Riak-CS issues when Riak endpoint fails-over to new server

Hi Toby, > I thought that it was recommended AGAINST talking to a co-located Riak on the same host? I'm not sure where you heard that from, but that's not the case. We do discourage running other parts of an application on the same hosts, such as client front-ends for example. From the top of m

Re: Riak-CS issues when Riak endpoint fails-over to new server

Hi Magnus, I thought that it was recommended AGAINST talking to a co-located Riak on the same host? The reason being, the local Riak will take longer to start up than Riak CS, once you have a sizeable amount of data. This means Riak CS starts up, fails to connect to Riak, and exits. You also end up

Re: Riak-CS issues when Riak endpoint fails-over to new server

Hi Toby, As far as I know Riak CS has none of the more advanced retry capabilities that Riak KV has. However, in the design of CS there seems to be an assumption that a CS instance will talk to a co-located KV node on the same host. To achieve high availability, in CS deployments HAProxy is often

Re: Riak-CS issues when Riak endpoint fails-over to new server

Hello all, Now that we're all back from the end-of-year holidays, I'd like to bump this question up. I feel like this has been a long-standing problem with Riak CS not handling dropped TCP connections. Last time the cause was haproxy dropping idle TCP connections after too long, but we solved that

Re: Riak CS: avoiding RAM overflow and OOM killer

Hi Alexander, Thanks a lot for your input. I have a few follow-up questions. > Stupid math: > > 3e7 x 3 (replication) / 9 = 1e7 minimum objects per node ( absolutely more > due to obj > 1MB size ) > > 1e7 x ~400 bytes per obj in ram = 4e9 ram per node just for bitcask. Aka 4 > GB. > > You alread

Re: Riak CS: avoiding RAM overflow and OOM killer

Hello DeadZen, Yes, networking interconnect becomes a bigger issue with more nodes in the cluster. A Riak cluster is actually a fully meshed network of erlang virtual machines. Multiple 1/10 gig nics dedicated to inter/intra networking are your friends. That said, we have many customers running

Re: Riak CS: avoiding RAM overflow and OOM killer

ok I loled at this. then got worries trump could win a node election. anyways. 24gigs per riak server is not a bad safe bet. Erlang in general is ram heavy. It uses it more effectively then most languages wrt concurrency, but ram is the fuel for concurrency and buffer for operations, especially du

Re: Riak CS: avoiding RAM overflow and OOM killer

Hi Alexander, Thanks for responding. > How many nodes? We currently have 9 nodes in our cluster. > How much ram per node? Each node has 4GB of ram and 4GB of swap. The memory levels (ram + swap) on each node are currently between 4GB and 5.5GB. > How many objects (files)? What is the average

Re: Riak CS: avoiding RAM overflow and OOM killer

Hi Daniel, Ya, I'm not surprised you're having issues. 4GB ram is woefully underspecd. 😔 🤓Stupid math: 3e7 x 3 (replication) / 9 = 1e7 minimum objects per node ( absolutely more due to obj > 1MB size ) 1e7 x ~400 bytes per obj in ram = 4e9 ram per node just for bitcask. Aka 4 GB. You already

Re: Riak CS: avoiding RAM overflow and OOM killer

Hi Daniel, How many nodes? -You should be using 5 minimum if you using the default config. There are reasons. How much ram per node? -As you noted, in Riak CS, 1MB file chunks are stored in bitcask. Their key names and some overhead consume memory. How many objects (files)? What is the average f

Re: Riak CS: avoiding RAM overflow and OOM killer

I found a similar question from over a year ago ( http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-July/017327.html), and it sounds like leveldb is the way to go, although possibly not well tested. Has anything changed with regard to Basho's (or anyone else) experience with using le

Re: Riak-CS False 503 Service Unavailable Error on Client

Hi Anthony, Looking at the linked issue it appears that the 503 response can be returned erroneously when communication between a Riak CS and Riak node has an error ("If the first member of the preflist is down"). Is there anything predictable about these errors? You say they come from 1 client o

Re: Riak-cs / stanchion won't find credentials

Also, just for reference I am using last riak/riak-cs/stanchion available on packagecloud On Wed, May 4, 2016 at 11:34 PM, Jhonny Everson wrote: > Hi, > > I am setting up a new cluster. I followed all the setup instructions (I > think). I created admin user as doc says, then updated riak-cs.conf

Re: riak-cs

did some investigation - this problem seems to be the same as described in https://github.com/basho/riak/issues/415 (and others) - using the erlang interface i retrieved the bucket list as: [<<"moss.buckets">>,<<"moss.access">>,<<"moss.users">>, <<48,111,58,16,121,107,99,149,64,231,6,8,234,204

Re: Riak CS/Stanchion troubleshooting (Retrieval of user record)

Hi, > I think that I need to extend my Riak cluster with more nodes to increase > performance. It can be a simple solution if your cluster just faced over capacity on most/all nodes. However, it would be better to make sure a fact by reading logs, and monitoring system resources. > I think that

Re: Riak CS/Stanchion troubleshooting (Retrieval of user record)

Hi, Sorry for my long silence. Kazhuiro thank you for your answer. The situation is more clear. I think that I need to extend my Riak cluster with more nodes to increase performance. The reason for my opinion is: Nov 19 11:42:40 localhost haproxy[24678]: 172.18.103.31:49608 [19/Nov/2015:11:42:40.

Re: riak-cs sync buckets from S3 to Riak-cs

Thanks a lot for your help! 2015-11-17 3:50 GMT+01:00 Shunichi Shinohara : > Hi Alberto, > > I didn't look into boto implementation, but I suspect that COPY Object API > does NOT work between different S3-like systems. > The actual interface definition of the API is [1] and source bucket/key > is

Re: riak-cs sync buckets from S3 to Riak-cs

Hi Alberto, I didn't look into boto implementation, but I suspect that COPY Object API does NOT work between different S3-like systems. The actual interface definition of the API is [1] and source bucket/key is just a string in the x-amz-copy-source header. The request went into the system that in

Re: riak-cs sync buckets from S3 to Riak-cs

Hello. Thanks for your help Dmitri. Perhaps this is not the correct place to ask this, but maybe someone had have the same problem. I build a test environment with two nodes of RIAK-CS, not in cluster, and I'm trying to move objects from one to another, to a bucket with the same name (testbucket

Re: Riak CS/Stanchion troubleshooting (Retrieval of user record)

Hi, ha_proxy's timeout settings often causes disconnected errors on a Riak CS deployment by high work load. termination_stat [1] in tcplog [2] lets you know if timeout happens or not. > 2015-11-13 13:13:09.514 [error] > <0.11264.1387>@riak_cs_wm_common:maybe_create_user:222 Retrieval of user >

Re: riak-cs sync buckets from S3 to Riak-cs

Hi Alberto, >From what I understand, the state of the art in terms of migration of objects from Amazon S3 to Riak CS is -- writing migration scripts. Either as shell scripts (using s3cmd), or language-specific libraries like boto (or even just the S3 SDKs). And the scripts would consist of: 1) get

Re: Riak CS - Unable to create/view bucket details using dragon disk

Riak cs fast track would answer your question On Oct 13, 2015 21:30, "Shawn Debnath" wrote: > I have been using s3curl.sh with success to do the dirty work with Riak > CS. Here’s a snippet from my docker setup files. I use the same methodology > in production. > > 187 # CREATE CS BUCKET AND APP

Re: Riak CS - Unable to create/view bucket details using dragon disk

I have been using s3curl.sh with success to do the dirty work with Riak CS. Here’s a snippet from my docker setup files. I use the same methodology in production. 187 # CREATE CS BUCKET AND APPLY ACL 188 ## 189 bin/s3curl.pl \ 190 --debug \ 191 --id ${RIAK_ADMIN_KEY} \

Re: Riak CS - Unable to create/view bucket details using dragon disk

Hi Shino, I tried multiple times with the access key and secret key configured in riak-cs.conf. But I'm unable to create or view buckets using s3 cmd. Could you let me know if there is any documentation for this? My aim to create buckets, upload objects to riak buckets from a HTTP interface sim

Re: Riak CS - Unable to create/view bucket details using dragon disk

Thanks for detailed information. You got 403 then authentication failure or authorization failure. First to check is StringToSign on both side. s3cmd with debug flag showed StringToSign at "DEBUG: SignHeaders" line. For Riak CS, you can see it by setting log level to debug like: [debug] STS: ["G

Re: Riak CS - Unable to create/view bucket details using dragon disk

Seem like the response is 403 forbidden, have no idea why. Did you get any stack trace this time? the response seems legit and that it managed to parse it? /J On 2 October 2015 at 18:35, G wrote: > Hey Johan, > > I have executed s3cmd with --debug option. > > Please find the output. > > Test acc

Re: Riak CS - Unable to create/view bucket details using dragon disk

Hey Johan, I have executed s3cmd with --debug option. Please find the output. Test access with supplied credentials? [Y/n] Y Please wait, attempting to list all buckets... DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Fri, 02 Oct 2015 16:32:44 +\n/' DEBUG: CreateRequest: resource[uri]=/ DEBUG:

Re: Riak CS - Unable to create/view bucket details using dragon disk

Haven't used S3 or Dragon disk, but looking at the python stacktrace and the code at github suggests some things. First the actuall from riak is caught correctly: File "/usr/share/s3cmd/S3/S3.py", line 624, in send_request raise S3Error(response) and its later when it tries to parse

Re: Riak CS - Unable to create/view bucket details using dragon disk

Dragon disk version - 1.05 S3cmd - 1.1.0-beta3 I think dragon disk is using HTTP to get the data.. Once I add the Riak account in dragon disk below is the output from dragon disk. It is able to issue GET request to riak, but no reply from riak.

Re: Riak CS - Unable to create/view bucket details using dragon disk

I asked some questions but the error stack did not answer any. s/Dragon disk/s3cmd/ version of my questions: - Does s3cmd actually attempt to connect to Riak CS? (instead of AWS S3) - If yes, how does TCP communication go on? - If TCP is ok, how is HTTP request/response? - If HTTP is ok, what i

Re: Riak CS - Unable to create/view bucket details using dragon disk

I get below error when I try to create a bucket using s3cmd Test access with supplied credentials? [Y/n] Y Please wait, attempting to list all buckets... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Success. Encryption and decryption worked fine :

Re: Riak CS - Unable to create/view bucket details using dragon disk

Hi Shino, Thanks for your reply. I ran s3cmd and created /root/.s3cfg(uploaded file to this post). I have selected others3 compatible storage and used the same access key/secret key in dragon disk. But it is unable to connect to riak. I'm able to access AWS S3 storage from the same app(dragon d

Re: Riak CS - Unable to create/view bucket details using dragon disk

Hi, Sorry but I don't have much to say about Dragon Disk. If you can access to Riak CS by Rest API, then I think Riak CS is (almost) working well. Hmm... the random items that should be investigated at first are: - Does Dragon disk actually attempt to connect to Riak CS? (instead of AWS S3) - I

Re: riak-cs auth_module riak_cs_s3_passthru_auth startup problem

Hi Kent, riak_cs_s3_passthru_auth is for internal use and does not work as auth_module. It will be possible to make it work as auth_module but some effort of refactering and (maybe) some additional implementation. It can be a workaround to use bucket policy to permit GET Object and PUT Object for

Re: RIAK-CS / FreeBSD 10.2 - Start error

On Tue, Sep 15, 2015 at 12:47 PM, Kota Uenishi wrote: > The message says your Riak is not properly configured. Please read [1] > and set default bucket properties as buckets.default.allow_mult=true. > > [1] > http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak/#Allowing-f

Re: RIAK-CS / FreeBSD 10.2 - Start error

The message says your Riak is not properly configured. Please read [1] and set default bucket properties as buckets.default.allow_mult=true. [1] http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak/#Allowing-for-Sibling-Creation. On Tue, Sep 15, 2015 at 10:48 AM, Outback

Re: Riak CS Node Shutdown Because it Ran Out Data Space - Unbalanced With Other Nodes in Cluster

On 03/08/2015 17:55, Valenti, Anthony wrote: > The ring size is actually 64. This is from riak-admin status: > > ring_num_partitions : 64 > ring_ownership : <<"[{'riak@192.168.40.13',12},\n > {'riak@192.168.40.14',12},\n {'riak@192.168.40.15',12},\n > {'riak@192.168.40.27',16},\n {'riak@192.168

RE: Riak CS Node Shutdown Because it Ran Out Data Space - Unbalanced With Other Nodes in Cluster

The ring size is actually 64. This is from riak-admin status: ring_num_partitions : 64 ring_ownership : <<"[{'riak@192.168.40.13',12},\n {'riak@192.168.40.14',12},\n {'riak@192.168.40.15',12},\n {'riak@192.168.40.27',16},\n {'riak@192.168.40.28',12}]">> ring_creation_size : 64 We added more sp

Re: Riak CS Node Shutdown Because it Ran Out Data Space - Unbalanced With Other Nodes in Cluster

On 03/08/2015 16:44, Valenti, Anthony wrote: > BTW, this is the ring distribution on the nodes: > > root@CORP-RIAK07:~# riak-admin member-status > = Membership > == > Status RingPendingNode > -

RE: Riak CS Node Shutdown Because it Ran Out Data Space - Unbalanced With Other Nodes in Cluster

BTW, this is the ring distribution on the nodes: root@CORP-RIAK07:~# riak-admin member-status = Membership == Status RingPendingNode --- valid

Re: Riak CS Node Shutdown Because it Ran Out Data Space - Unbalanced With Other Nodes in Cluster

On 31/07/2015 19:51, Valenti, Anthony wrote: > We have a Riak CS node that ran out of space on the data mount and the > service shutdown. This is strange since the other nodes in the cluster seem > to be about 76%-78% full. Why is it so out of balance and how can it be > fixed? > Hi ! I fac

Re: Riak-CS RAM usage

Bitcask keeps all the *keys* in RAM. That's like an index in database terms. So even if you have 9Gb of data, you should only have a tiny fraction of that in keys, in memory. So you might have some other problem going on there? Riak uses quite a bit of memory for LevelDB buffers, by default, so m

Re: Riak-CS RAM usage

Anita, To use leveldb instead of bitcask for blocks, just stop using multi backend and remove all multi-backend related configuration. You'll need fresh install as CS does not automatically move data from bitcask to leveldb when you change backend. I would like you to know that Basho have not yet

Re: Riak CS docs have incorrect packagecloud URL

Thanks Toby! Just submitted PR to fix the urls: https://github.com/basho/basho_docs/pull/1713 On Mon, Jun 8, 2015 at 9:02 PM, Toby Corkindale wrote: > Hi Matt, > This page contains the incorrect URLs: > > http://docs.basho.com/riakcs/latest/cookbooks/installing/Installing-Riak-CS/ > > If you s

Re: Riak CS docs have incorrect packagecloud URL

Hi Matt, This page contains the incorrect URLs: http://docs.basho.com/riakcs/latest/cookbooks/installing/Installing-Riak-CS/ If you search on that page for this text, you'll find it right underneath. "The resulting file should hold contents like the following" Cheers, Toby On Thu, 4 Jun 2015 at

Re: Riak CS 2 backend config confusion

Hi Toby, Current versions for Riak CS system are: Riak CS 2.0.x and it is tested with Riak 2.0.x. Sorry for confusion, but the document you pointed is for the combination. You can use the configuration steps using advanced.config in the doc. We are now in development of Riak CS 2.1 and prefix_mul

Re: Riak CS docs have incorrect packagecloud URL

For what it's worth, it looks good on the packagecloud directions as of the most recent update [1]: > curl -s > https://packagecloud.io/install/repositories/basho/riak-cs/script.deb.sh | sudo bash > sudo apt-get install riak-cs=2.0.1-1 [1] https://packagecloud.io/basho/riak-cs/packages/ubunt

Re: Riak CS docs have incorrect packagecloud URL

Hey Toby, Which documentation pointed you to the wrong URL? And did the dash address the issue? We can and should update that documentation. On a related note, our packaging is up for discussion. A few of us started discussing it here [1]. I'd love for you to weigh in. [1] https://github.com/bas

Re: Riak-cs cluster

Hi Mohamad. Good questions. You can install Stanchion on the HA proxy load balancer node, or a separate standalone node, whichever you prefer. Stanchion exists only to provide serialization (essentially, a gatekeeper process) for the creation of new users and new buckets, only. As you mentioned,

Re: Riak CS need for 5 servers?

Thanks for the links and additional explanation Chris. I will go through them. One more question. My hosting provider sets a hard limit on the harddisk size at the time of order. I go with, say, 20TB harddisk per server and order 5 of them, I can get Riak running but I will still be limited to 20T

Re: Riak CS need for 5 servers?

> On Apr 3, 2015, at 11:24 AM, Shankar Dhanasekaran > wrote: > > Thanks Chris for your answer. I still don't comprehend what is meant by > this is the minimum amount of nodes needed to ensure all copies of each > object are stored on different physical nodes. [2] > > Why can't a 3 node clust

Re: Riak CS need for 5 servers?

Thanks Chris for your answer. I still don't comprehend what is meant by > this is the minimum amount of nodes needed to ensure all copies of each > object are stored on different physical nodes. [2] Why can't a 3 node cluster store the data in each of them? I am reading the little riak book as w

Re: Riak CS need for 5 servers?

> On Apr 3, 2015, at 10:38 AM, Shankar Dhanasekaran > wrote: > > hi, > For the organization that I am working for, it seems like 25tb of storage is > needed. Reading riak docs, it's mentioned that at least 5 servers should be > put in place. > > what is the harm in having only 2 riak nodes?

Re: Riak-CS Node.js client

Thanks! I'll stay tuned ;) till there I will patch my code an pray not to introduce any issues here.. The patch is bellow for anyone to try (just send the decoded string to be signed) >From a439f6126317a2b66fc08baf31b24b47e8ec4ed9 Mon Sep 17 00:00:00 2001 From: "Patrick F. Marques" Date: Thu, 26

Re: Riak-CS Node.js client

> The s3 signer uses the signs the "canonicalizedResource" and that have the > query parameters already encoded, so I tried to replace the "%3D" by the "=" > and it already works. Yey! The culprit is here. Most client mistakenly encodes Multipart uploadId although it is already supposed to be ur

Re: Riak-CS Node.js client

Hi, thanks for your help Uenishi. I'm using Riak 1.5.2, and AWS Node.js SDK 2.1.14 and the example code I'm running is bellow. I have beed trying with and without forcing a singing version. With some debug I found that the default is the use the s3 signer If I force v2 I have another error, "

Re: Riak-CS Node.js client

Hi, My 6th sense says you're hitting this problem: https://github.com/basho/riak_cs/issues/1063 Could you give me an example of code or debug print of Node.js client that includes the source string before being signed by a secret key? Otherwise maybe that client is just using v4 authentication w

Re: Riak CS expiration

On 7 February 2015 at 06:21, Kota Uenishi wrote: > Bitcask doesn't have such interface like hooking anything. As of > internal data structure of Riak CS built on top of Riak, bitcask keys > don't correspond simply to leveldb key. Say, things like this cannot > happen: "all tombstones of blocks of

Re: Riak CS expiration

Bitcask doesn't have such interface like hooking anything. As of internal data structure of Riak CS built on top of Riak, bitcask keys don't correspond simply to leveldb key. Say, things like this cannot happen: "all tombstones of blocks of this object were collected and space were reclaimed by mer

Re: Riak CS Won't Start After Upgrade

Hi Anthony, I have filed the following request to clarify this better in the documentation: https://github.com/basho/basho_docs/issues/1490 -- Luke Bakken Engineer lbak...@basho.com On Fri, Jan 16, 2015 at 9:59 AM, Valenti, Anthony wrote: > Thank you. That worked. I think that the rolling upg

RE: Riak CS Won't Start After Upgrade

Thank you. That worked. I think that the rolling upgrade page needs to be changed to mention that. Step 6 does mention the add_paths, but it is not clear what needs to be changed. It mentions .beam files, but I had no idea what that is, if I am using them or what I should be changing. Antho

Re: Riak CS Won't Start After Upgrade

If your Riak CS version changed, ensure that add_paths points to the correct directory: http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak/ There is a version string embedded in the path that will change on upgrades. -- Luke Bakken Engineer lbak...@basho.com On Fri, Ja

RE: Riak CS Won't Start After Upgrade

Of course you have to have Riak started. It looked like it started normally, but then it failed. The error is complaining about an invalid storage backend, which didn't change. 2015-01-16 10:13:44.911 [info] <0.7.0> Application lager started on node 'riak@192.168.40.28' 2015-01-16 10:13:44.93

RE: Riak CS Won't Start After Upgrade

BTW, the components were upgraded to the following versions: Riak CS - 1.5.3 Riak - 1.4.10 Stanchion - 1.5.0 Anthony Inmar Confidentiality Note:  This e-mail and any attachments are confidential and intended to be viewed and used solely by the inten

Re: RIAK-CS Unable to create bucket using s3cmd - AccessDenied

Thank-you, that corrected the issue. On Monday, January 12, 2015 9:00 PM, Shunichi Shinohara wrote: Hi Sellmy, New versions of s3cmd uses AWS v4 authentication [1] but Riak CS does not support it yet [2]. Tentatively, please add following one line to your .s3cfg file: signature_v2 =

Re: RIAK-CS Unable to create bucket using s3cmd - AccessDenied

Hi Sellmy, New versions of s3cmd uses AWS v4 authentication [1] but Riak CS does not support it yet [2]. Tentatively, please add following one line to your .s3cfg file: signature_v2 = True [1] https://github.com/s3tools/s3cmd/issues/402 [2] https://github.com/basho/riak_cs/issues/897 Thanks, Shi

Re: Riak CS Authorization Problems

Lucas, Thank you for being interested in Riak CS. > s3cmd ls and retrieving the admin user works fine but I am not able to create a new bucket. That is typical observation when Stanchion is not correctly configured. I'm afraid you didn't put admin key and secret into Stanchion's app.config. Othe

RE: Riak-CS Not Inheriting Permissions?

Bakken [mailto:lbak...@basho.com] Sent: Tuesday, October 14, 2014 11:52 AM To: Valenti, Anthony Cc: riak-users@lists.basho.com Subject: Re: Riak-CS Not Inheriting Permissions? Hi Anthony, When you put files, are you also including any ACL information? http://docs.basho.com/riakcs/latest/cookbooks

Re: Riak-CS Not Inheriting Permissions?

Hi Anthony, When you put files, are you also including any ACL information? http://docs.basho.com/riakcs/latest/cookbooks/Access-Control-Lists/ -- Luke Bakken Engineer / CSE lbak...@basho.com On Tue, Oct 14, 2014 at 8:07 AM, Valenti, Anthony wrote: > Does anyone have an idea why this is worki

Re: Riak-CS Not Inheriting Permissions?

Does anyone have an idea why this is working this way or if it is functioning as designed? Am I expecting it to do something that it doesn't do? If so, then how can I get it do what I am expecting. We are running Riak-CS on version 1.4.8. Here is the situation. We have a bucket that is owned

Re: Riak CS with hadoop over s3 protocol

According to the stacktrace you pasted, your software seems modified to use some EMR RPC calls, which is not included in OSS hadoop [1]. I can't imagine anything precise, but it looks like some reporting API calls to Amazon's EMR service and your job seems to be accessing that RPC with unauthentica

Re: Riak CS with hadoop over s3 protocol

Hi Kota/John/Andrew, Thanks for your suggestions. So this is what i've tried with unsuccessful results. -* jets3t.properties file* s3service.s3-endpoint= s3service.s3-endpoint-http-port=8080 s3service.disable-dns-buckets=true s3service.s3-endpoint-virtual-path=/ httpclient.proxy-autodetect=fals

Re: Riak CS with hadoop over s3 protocol

I played on Hadoop MapReduce on Riak CS, and it actually worked with the latest 1.5 beta package. Hadoop relies S3 connectivity on jets3t, so if MapR uses vanilla jets3t it will work. I believe so because MapR works on EMR (which usually extracts data from S3). Technically, you can add several opt

Re: Riak CS with hadoop over s3 protocol

This blog post on configuring S3 clients to work with CS may be useful: http://basho.com/riak-cs-proxy-vs-direct-configuration/ Sent from my iPhone On Jul 31, 2014, at 2:53 PM, Andrew Stone wrote: Hi Charles, AFAIK we haven't ever tested Riak Cs with the MapR connector. However, if MapR works

Re: Riak CS with hadoop over s3 protocol

Hi Charles, AFAIK we haven't ever tested Riak Cs with the MapR connector. However, if MapR works with S3, you should just have to change the IP to point to a load balancer in front of your local Riak CS cluster. I'm unaware of how to change that setting in MapR though. It seems like a question for

Re: Riak CS

Mohan, I might be a bit confused on what your intent is, but it sounds like your task is to download a large group of files from S3 for processing and you are considering Riak CS for that processing work. If that is the case I am not sure Riak CS is the right fit for that job. Riak itself has a

Re: Riak CS

I thought the general idea floating in my ignorant mind can help. We have other storage systems apart from S3 like FTPS, HTTPS etc. I thought the task of crawling remote storage systems and processing files naturally lent itself to distributed MR jobs and a DFS. That is when I came across Riak CS.

Re: riak-cs-gc won't start

Any example configuration maybe can apply to small space server ? On Fri, Jul 11, 2014 at 3:17 PM, Shunichi Shinohara wrote: > It seems GC works well :) > > > I try to delete those file and check the size on the server won't reduce > > If you care about disk usage of the server that riak is run

Re: riak-cs-gc won't start

It seems GC works well :) > I try to delete those file and check the size on the server won't reduce If you care about disk usage of the server that riak is running on, there is another factor. You need backend compaction to reduce disk usage. It will require putting/deleting many objects or larg

  1   2   >