Re: Object not found after successful PUT on S3 API

Matthew, That's very helpful, thank you! I agree, the ring size change is probably the most significant factor, and we probably should not have done that on this cluster considering it is the smallest of our production clusters and the growth rate is very low. Daniel On Tue, May 2, 2017 at 11:01

Re: Object not found after successful PUT on S3 API

Daniel, 1G of ram is an interesting and untested challenge. Others have succeeded in getting regular Riak KV to operate on a Raspberry PI with 2g of Ram. Here are the 3 things you can do in an attempt to drive down required RAM (in order of priority): 1. reduce the ring size: in riak.conf s

Re: Object not found after successful PUT on S3 API

On 28 April 2017 at 15:15, Matthew Von-Maszewski wrote: > Daniel, > > Something is wrong. All instances of leveldb within a node share the > total memory configuration. The memory is equally divided between all > active vnodes. It is possible to create an OOM situation if total RAM is > low an

Re: Object not found after successful PUT on S3 API

Daniel, Something is wrong. All instances of leveldb within a node share the total memory configuration. The memory is equally divided between all active vnodes. It is possible to create an OOM situation if total RAM is low and vnodes count per node is high relative to RAM size. The best ne

Re: Object not found after successful PUT on S3 API

Hi Luke, I'm reviving this thread from March where we discussed a new backend configuration for our riak cluster. We have had a chance to test out the new recommended configuration, and so far we have not been successful in limiting the RAM usage of leveldb with multi_backend. We have tried variou

Re: Object not found after successful PUT on S3 API

I apologize for the confusion and defer to Luke on all the technical Riak things. On Fri, Mar 10, 2017 at 11:16 Luke Bakken wrote: > Just to clarify ... > > What Alexander is suggesting is what Daniel is currently using, and > what I suspect may be causing Daniel's issues. > > If you wish to ru

Re: Object not found after successful PUT on S3 API

Just to clarify ... What Alexander is suggesting is what Daniel is currently using, and what I suspect may be causing Daniel's issues. If you wish to run a leveldb-only Riak CS cluster, you still *must* use the advanced.config file and the riak_cs_kv_multi_backend, and the other settings that I m

Re: Object not found after successful PUT on S3 API

Hi Daniel, Riak CS uses multi by default. By default the manifests are stored in leveldb and the blobs/chunks are stored in bitcask. If you're looking to force everything to level you should remove multi and use level as the backend setting. As Luke noted elsewhere, this configuration hasn't be

Re: Object not found after successful PUT on S3 API

Hi Luke, Again, thanks for your help. We are currently preparing to move all objects into a new cluster using the S3 API. One question on configuration: currently I have "storage_backend = leveldb" in my riak.conf. I assume that on the new cluster, in addition to using the advanced.config you prov

Fwd: Object not found after successful PUT on S3 API

> Thanks for taking the time to look into this Luke. I should have asked more questions when I setup the configuration for leveldb backend since there is no clear documentation for how configure CS with leveldb only. The reason for this is that a leveldb-only configuration is neither supported nor

Re: Object not found after successful PUT on S3 API

Thanks for taking the time to look into this Luke. I should have asked more questions when I setup the configuration for leveldb backend since there is no clear documentation for how configure CS with leveldb only. In your current configuration, you are not using the expected prefix for > the bloc

Re: Object not found after successful PUT on S3 API

Hi Daniel, Thanks for providing all of that information. You are missing important configuration for riak_kv that can only be provided in an /etc/riak/advanced.config file. Please see the following document, especially the section to which I link here: http://docs.basho.com/riak/cs/2.1.1/cookboo

Re: Object not found after successful PUT on S3 API

Also I just realized I gave the wrong platform_data_dir in the config-files.zip I sent yesterday. The correct path is shown in riak-env.txt as well as generated.configs.tbz attachments sent later. On Tue, Mar 7, 2017 at 9:47 AM, Daniel Miller wrote: > Responses inline. > > On Mon, Mar 6, 2017 at

Re: Object not found after successful PUT on S3 API

Responses inline. On Mon, Mar 6, 2017 at 3:04 PM, Luke Bakken wrote: > Hi Daniel, > > Two questions: > > * Do you happen to have an /etc/riak/app.config file present? > No. Not sure if relevant, but I did notice that /etc/riak-cs/advanced.config does exist, which contradicts with what I said e

Re: Object not found after successful PUT on S3 API

Hi Daniel, Two questions: * Do you happen to have an /etc/riak/app.config file present? * On one of your Riak nodes, could you please execute the following commands: riak attach rp(application:get_all_env(riak_kv)). Copy the output of the previous command and attach as a separate file to your

Re: Object not found after successful PUT on S3 API

Hi Luke, I do not have an advanced.config file since I switched to leveldb storage backend. Generated configs attached. Hopefully not relevant, the data root is on an ecryptfs volume. On Mon, Mar 6, 2017 at 1:23 PM, Luke Bakken wrote: > Hi Daniel - > > Did you forget to include the advanced.co

Re: Object not found after successful PUT on S3 API

Hi Daniel - Did you forget to include the advanced.config file in your archive of configuration files? I only see three *.conf.j2 files. The reason I ask is that the following settings are critical to Riak CS functioning correctly: http://docs.basho.com/riak/cs/2.1.1/cookbooks/configuration/riak-

Re: Object not found after successful PUT on S3 API

Genuinely stumped then. I’m surprised that dvv_enabled=false is the default as sibling explosion is bad. I don’t know the CS code very well, but I assume a not_found means that either the manifest or some chunk is not found. I wonder if you can get the manifest and then see if any/all of the ch

Re: Object not found after successful PUT on S3 API

> Would be good to know the riak version Riak 2.1.1 Riak CS 2.1.0 Stanchion 2.1.0 > why the dvv_enabled bucket property is set to false, please? Looks like that's the default . I haven't changed it. > Also, is there multi

Re: Object not found after successful PUT on S3 API

Hi, Would be good to know the riak version, and why the dvv_enabled bucket property is set to false, please? Also, is there multi-datacentre replication involved? Do you re-use your keys, for example, have the keys in question been created, deleted, and then re-created? Cheers Russell On 6 Ma

Re: Object not found after successful PUT on S3 API

Hi Luke, Sorry for the late response and thanks for following up. I haven't seen it happen since. At this point I'm going to wait and see if it happens again and hopefully get more details about what might be causing it. Daniel On Thu, Feb 9, 2017 at 1:02 PM, Luke Bakken wrote: > Hi Daniel - >

Re: Object not found after successful PUT on S3 API

Hi Daniel - I don't have any ideas at this point. Has this scenario happened again? -- Luke Bakken Engineer lbak...@basho.com On Wed, Jan 25, 2017 at 2:11 PM, Daniel Miller wrote: > Thanks for the quick response, Luke. > > There is nothing unusual about the keys. The format is a name + UUID +

Re: Object not found after successful PUT on S3 API

Thanks for the quick response, Luke. There is nothing unusual about the keys. The format is a name + UUID + some other random URL-encoded charaters, like most other keys in our cluster. There are no errors near the time of the incident in any of the logs (the last [error] is from over a month bef

Re: Object not found after successful PUT on S3 API

Hi Daniel - This is a strange scenario. I recommend looking at all of the log files for "[error]" or other entries at about the same time as these PUTs or 404 responses. Is there anything unusual about the key being used? -- Luke Bakken Engineer lbak...@basho.com On Wed, Jan 25, 2017 at 6:40 AM

Object not found after successful PUT on S3 API

I have a 9-node Riak CS cluster that has been working flawlessly for about 3 months. The cluster configuration, including backend and bucket parameters such as N-value are using default settings. I'm using the S3 API to communicate with the cluster. Within the past week I had an issue where two ob