Howto add Users without anonymous / whats the deal with URLs
Hi there, I managed to install a 5 Node RIAK CS Cluster and it seems to work fine. Anyway, there are some things I still do not understand: I managed to add, revoke, remove users through riak-cs-controll, but I don't really like to use this tool, as for me it feels like another possible security hole (I want to use riak cs on public servers for public available files). I likely understood s3cmd, for getting informations about Buckets, Objects and Users. I also got amazon-s3-php-class to work, to upload files and create buckets. But I'm not really familiar with the concept of s3curl, the proxying and the URL (s3.amazonaws.com). Can I simply put my own URL in front of my cluster (round-robin at best) and push to that cluster through a http Proxy (like varnish, sitting on each node to cache hight frequently fetched files) and which console tool I have to use, with which parameters? Same for the users. Which console tool do I have to use with which params to add, revoke, remove users? -- Andre Lohmann Alte Gärtnerei 20a 24232 Schönkirchen Tel.: +49 (0) 431 260 93 833 mobil: +49 (0) 176 55 910 913 mail: lohmann.an...@gmail.com LinkedIn: http://de.linkedin.com/pub/andre-lohmann/26/a60/a92 Xing: http://www.xing.com/profile/Andre_Lohmann ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
local_put_failed error
Recently my Riak nodes started returning errors local_put_failed. It happens on many nodes. Half of my nodes are Riak 1.3.1, half upgraded 1.3.2 (this setup worked for more than a month without any problems). I found in http://docs.basho.com/riak/latest/ops/running/recovery/errors/ that this can be somehow related to LevelDB memory and node restart is recommended. The problem is that it happens on many nodes. Should I restart all of them and will it actually solve the problem (I'm afraid it will just appear again later)? I've run risk-admin diag -- nothing. Looked at the riak logs on all nodes -- nothing except ALIVE. Why this error happens and how to solve this issue? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: local_put_failed error
Some more information. Error doesn't seem to be related to particular node/partition since put operations succeed on retries. Errors appear few times a second while I'm making about 500-1000 write requests a second. There were no increase in cluster load before this error started to appear (in fact it started at the lowest night load). How can I diagnose a source of error? 2013/9/10 Vladimir Shabanov > Recently my Riak nodes started returning errors local_put_failed. It > happens on many nodes. > > Half of my nodes are Riak 1.3.1, half upgraded 1.3.2 (this setup worked > for more than a month without any problems). > > I found in > http://docs.basho.com/riak/latest/ops/running/recovery/errors/ > that this can be somehow related to LevelDB memory and node restart is > recommended. > > The problem is that it happens on many nodes. Should I restart all of them > and will it actually solve the problem (I'm afraid it will just appear > again later)? > > I've run risk-admin diag -- nothing. Looked at the riak logs on all nodes > -- nothing except ALIVE. > > Why this error happens and how to solve this issue? > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Howto add Users without anonymous / whats the deal with URLs
Hi Andre,We are working to improve the user management tooling for Riak CS. A pretty much read-only prototype CLI can be found under our basho-labs GitHub organization. [0]If you don't want to use Riak CS Control to manage users, your best alternatives right now are:- s3curl- fogExamples of how to use Fog to manage users can be found on our docs site. [1] In order to use s3curl against Riak CS setup in a proxy configuration you'll want to look at the account management section on the CS docs site. [2] The arguments to s3curl would look something like the ones in this gist. [3]As an alternative to the proxy configuration, you can configure Riak CS in a "direct" configuration that gets rid of the s3.amazonaws.com requirement altogether. [4]-- Hector [0] https://github.com/basho-labs/riak-cs-user-mgmt[1] http://docs.basho.com/riakcs/latest/cookbooks/fog/[2] http://docs.basho.com/riakcs/latest/cookbooks/Account-Management/[3] https://gist.github.com/hectcastro/6514828[4] http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak-CS/#Proxy-vs-Direct-ConfigurationOn September 10, 2013 at 3:00:16 AM, Andre Lohmann (lohmann.an...@gmail.com) wrote: Hi there,I managed to install a 5 Node RIAK CS Cluster and it seems to work fine.Anyway, there are some things I still do not understand: I managed to add, revoke, remove users through riak-cs-controll, but I don't really like to use this tool, as for me it feels like another possible security hole (I want to use riak cs on public servers for public available files). I likely understood s3cmd, for getting informations about Buckets, Objects and Users.I also got amazon-s3-php-class to work, to upload files and create buckets. But I'm not really familiar with the concept of s3curl, the proxying and the URL (s3.amazonaws.com). Can I simply put my own URL in front of my cluster (round-robin at best) and push to that cluster through a http Proxy (like varnish, sitting on each node to cache hight frequently fetched files) and which console tool I have to use, with which parameters? Same for the users. Which console tool do I have to use with which params to add, revoke, remove users? -- Andre LohmannAlte Gärtnerei 20a24232 SchönkirchenTel.: +49 (0) 431 260 93 833mobil: +49 (0) 176 55 910 913 mail: lohmann.an...@gmail.comLinkedIn: http://de.linkedin.com/pub/andre-lohmann/26/a60/a92 Xing: http://www.xing.com/profile/Andre_Lohmann ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Should content-type not be required?
Suppose I want to just store keys in a bucket without any body, this make sense in scenarios where the key completely identify the entity. Is it possible to use the riak http api without including the content-type header? Looking at the http specifications, content–type is not mandatory or suggested when the body is empty: http://www.w3.org/Protocols/rfc2616/rfc2616-sec7.html#sec7.2.1 and it's probably a good argument to say that if the entity message is empty then the content type doesn't really make sense. Personally I think that just the existence of an http entity - body or not body – is sufficient to justify a "type"; it's unfortunate that any other higher type (entity type maybe?) - decoupled from what is defined as content - exist in the spec. Framework like spray are quite strict about this, not setting any content type when the content is empty: https://github.com/spray/spray/blob/master/spray-http/src/main/scala/spray/http/HttpEntity.scala#L74 how to deal with such situation? Adding a fake content because of this data store constraint doesn't seems right. Thanks, Vincenzo. -- If your e-mail inbox is out of control, check out http://sanebox.com/t/mmzve. I love it. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Should content-type not be required?
The content-type is important for Riak KV: Various clients will use it to identify the difference between a response where the whole body is the key, or a response that contains siblings (if you have allow_mult=true). I suggest finding or creating a content type, so that this is clearer to your app. It will also allow you to version your objects better. Here are a few suggestions (of course replace the text between the < >): - application/octet-stream - this is usually used for binary data, and is the easiest thing to set the content-type to. - application/vnd.. - a vendor-specific type, which you can create yourself - application/vnd...v - another vendor-specific type, which supports versioning. You don't have to use the information in your final app, but it is used by riak and riak clients, so that's why we require it. -- Sam Elliott Engineer sam.elli...@basho.com -- On Tuesday, 10 September 2013 at 5:03PM, Vincenzo Vitale wrote: > Suppose I want to just store keys in a bucket without any body, this make > sense in scenarios where the key completely identify the entity. Is it > possible to use the riak http api without including the content-type header? > Looking at the http specifications, content–type is not mandatory or > suggested when the body is empty: > http://www.w3.org/Protocols/rfc2616/rfc2616-sec7.html#sec7.2.1 > and it's probably a good argument to say that if the entity message is empty > then the content type doesn't really make sense. > Personally I think that just the existence of an http entity - body or not > body – is sufficient to justify a "type"; it's unfortunate that any other > higher type (entity type maybe?) - decoupled from what is defined as content > - exist in the spec. > Framework like spray are quite strict about this, not setting any content > type when the content is empty: > https://github.com/spray/spray/blob/master/spray-http/src/main/scala/spray/http/HttpEntity.scala#L74 > how to deal with such situation? Adding a fake content because of this data > store constraint doesn't seems right. > > > Thanks, > Vincenzo. > > -- > If your e-mail inbox is out of control, check out http://sanebox.com/t/mmzve. > I love it. > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Should content-type not be required?
Hi Sam, thanks for the fast reply. In my company we normally use your third suggestion and indeed it's a robust strategy. At the moment we are relying on a riak client based on spray ( https://github.com/agemooij/riak-scala-client) so even if we set the Content-Type - when the body is empty - it doesn't get to the final request being dispatched. Our workaround is to have a content body equal to the one byte carriage return. It's not a big deal... Writing to the mailing list was mainly because curious to know your thoughts about this :) Have a good day, V. On Wed, Sep 11, 2013 at 12:03 AM, Sam Elliott wrote: > The content-type is important for Riak KV: Various clients will use it to > identify the difference between a response where the whole body is the key, > or a response that contains siblings (if you have allow_mult=true). > > I suggest finding or creating a content type, so that this is clearer to > your app. It will also allow you to version your objects better. Here are a > few suggestions (of course replace the text between the < >): > - application/octet-stream - this is usually used for binary data, and is > the easiest thing to set the content-type to. > - application/vnd.. - a > vendor-specific type, which you can create yourself > - application/vnd...v > - another vendor-specific type, which supports versioning. > > You don't have to use the information in your final app, but it is used by > riak and riak clients, so that's why we require it. > > -- > Sam Elliott > Engineer > sam.elli...@basho.com > -- > > > On Tuesday, 10 September 2013 at 5:03PM, Vincenzo Vitale wrote: > > > Suppose I want to just store keys in a bucket without any body, this > make sense in scenarios where the key completely identify the entity. Is it > possible to use the riak http api without including the content-type header? > > Looking at the http specifications, content–type is not mandatory or > suggested when the body is empty: > > http://www.w3.org/Protocols/rfc2616/rfc2616-sec7.html#sec7.2.1 > > and it's probably a good argument to say that if the entity message is > empty then the content type doesn't really make sense. > > Personally I think that just the existence of an http entity - body or > not body – is sufficient to justify a "type"; it's unfortunate that any > other higher type (entity type maybe?) - decoupled from what is defined as > content - exist in the spec. > > Framework like spray are quite strict about this, not setting any > content type when the content is empty: > > > https://github.com/spray/spray/blob/master/spray-http/src/main/scala/spray/http/HttpEntity.scala#L74 > > how to deal with such situation? Adding a fake content because of this > data store constraint doesn't seems right. > > > > > > Thanks, > > Vincenzo. > > > > -- > > If your e-mail inbox is out of control, check out > http://sanebox.com/t/mmzve. I love it. > > ___ > > riak-users mailing list > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > -- If your e-mail inbox is out of control, check out http://sanebox.com/t/mmzve. I love it. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Download stops at certain megabyte boundary (RiakCS)
Martin, I think you have already read this page: http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak/#Setting-up-the-Proper-Riak-Backend As it says, you can use "add_paths" to set load path. -- Shunichi Shinohara Basho Japan KK On Wed, Sep 11, 2013 at 2:05 AM, Martin Alpers wrote: > Thank you Shino for your patient assistance. > > I did follow Seth's advice, and upgrading resulted in smooth downloads, as > stated in my reply to his mail. > Additionally, I upgraded the one noed I am working on to the very recent > 1.4.1 version. (Riak is 1.4.2) > I still use the packages for Debian Squeeze. > > Outputs of console.log and crash.log are attached to this email. > Although I am not an expert with these, the term "invalid_storage_backend" > appears in both of them, and riak used to start just fine before. > > I also tried to find the files: > root@riak2:~# find /usr/lib/riak -name "*kv_multi*" > /usr/lib/riak/lib/riak_kv-1.4.2-0-g61ac9d8/ebin/riak_kv_multi_backend.beam > root@riak2:~# find /usr/lib/riak-cs -name "*kv_multi*" > /usr/lib/riak-cs/lib/riak_cs-1.4.1/ebin/riak_cs_kv_multi_backend.beam > > So is adding "-pz /usr/lib/riak-cs/lib/riak_cs-1.4.1/ebin/" to > /etc/riak/vm.args, as suggested here: > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2010-October/002211.html > the right thing to do? > > Kind regards, > Martin > > On 13/09/10.10:48:1378806537, Shunichi Shinohara wrote: >> Martin, >> >> Thank you for more information. >> I found a mis-configuration of riak's app.config. The backend of riak for >> Riak CS is riak_cs_kv_multi_backend (look out for _cs_). This is the custom >> riak_kv backend for Riak CS. The error of "s3cmd ls" was caused by this. >> >> As for MB boundary stop, the issue >> https://github.com/basho/riak_cs/issues/613 >> may be related. Although not direct way to treat it, some points which may >> help: >> >> - As Seth said, upgrade to 1.4, which can reduce network communication >> between riak nodes >> - Set pb_backlog to 256 or higher in riak_api section of riak's app.config >> >> Regards, >> Shino >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > -- > Greetings, Martin Alpers > -- > martin-alp...@web.de > Mobile: 0176/66185173, but I prefer typing to talking (: > Jabber: martin.alp...@jabber.org > My mails are signed using GPG to verify their origin; request my public key > (10216CFB). > See also: http://apps.opendatacity.de/stasi-vs-nsa/ ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Should content-type not be required?
I'm interested to know why you're trying to store an empty body in Riak. Surely just don't even make the request to Riak? I guess I could be overlooking something obvious. Sam -- Sam Elliott Engineer sam.elli...@basho.com -- On Tuesday, 10 September 2013 at 6:29PM, Vincenzo Vitale wrote: > Hi Sam, > thanks for the fast reply. > > In my company we normally use your third suggestion and indeed it's a robust > strategy. > > At the moment we are relying on a riak client based on spray > (https://github.com/agemooij/riak-scala-client) so even if we set the > Content-Type - when the body is empty - it doesn't get to the final request > being dispatched. > > Our workaround is to have a content body equal to the one byte carriage > return. > It's not a big deal... > > Writing to the mailing list was mainly because curious to know your thoughts > about this :) > > > Have a good day, > V. > > > > On Wed, Sep 11, 2013 at 12:03 AM, Sam Elliott (mailto:sam.elli...@basho.com)> wrote: > > The content-type is important for Riak KV: Various clients will use it to > > identify the difference between a response where the whole body is the key, > > or a response that contains siblings (if you have allow_mult=true). > > > > I suggest finding or creating a content type, so that this is clearer to > > your app. It will also allow you to version your objects better. Here are a > > few suggestions (of course replace the text between the < >): > > - application/octet-stream - this is usually used for binary data, and is > > the easiest thing to set the content-type to. > > - application/vnd.. - a > > vendor-specific type, which you can create yourself > > - application/vnd...v > > - another vendor-specific type, which supports versioning. > > > > You don't have to use the information in your final app, but it is used by > > riak and riak clients, so that's why we require it. > > > > -- > > Sam Elliott > > Engineer > > sam.elli...@basho.com (mailto:sam.elli...@basho.com) > > -- > > > > > > On Tuesday, 10 September 2013 at 5:03PM, Vincenzo Vitale wrote: > > > > > Suppose I want to just store keys in a bucket without any body, this make > > > sense in scenarios where the key completely identify the entity. Is it > > > possible to use the riak http api without including the content-type > > > header? > > > Looking at the http specifications, content–type is not mandatory or > > > suggested when the body is empty: > > > http://www.w3.org/Protocols/rfc2616/rfc2616-sec7.html#sec7.2.1 > > > and it's probably a good argument to say that if the entity message is > > > empty then the content type doesn't really make sense. > > > Personally I think that just the existence of an http entity - body or > > > not body – is sufficient to justify a "type"; it's unfortunate that any > > > other higher type (entity type maybe?) - decoupled from what is defined > > > as content - exist in the spec. > > > Framework like spray are quite strict about this, not setting any content > > > type when the content is empty: > > > https://github.com/spray/spray/blob/master/spray-http/src/main/scala/spray/http/HttpEntity.scala#L74 > > > how to deal with such situation? Adding a fake content because of this > > > data store constraint doesn't seems right. > > > > > > > > > Thanks, > > > Vincenzo. > > > > > > -- > > > If your e-mail inbox is out of control, check out > > > http://sanebox.com/t/mmzve. I love it. > > > ___ > > > riak-users mailing list > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > > (mailto:riak-users@lists.basho.com) > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > > > -- > If your e-mail inbox is out of control, check out http://sanebox.com/t/mmzve. > I love it. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
