Re: Riak replication and quorum

Peter, wrote my replies inline. Mathias Meyer Developer Advocate, Basho Technologies On Freitag, 13. Mai 2011 at 20:05, Peter Fales wrote: > Sean, > > Thanks to you and Ben for clarifying how that works. Since that was > so helpful, I'll ask a followup question, and also a question on > a m

Re: Clarification on key filters

Jeremiah, I sure hope you have not been drinking mouthwas. The wiki is indeed showing single key filters, and that's confusing. In hindsight, it confused me too when I worked with key filters from the wiki page for the first time. I'll make sure we put some proper examples on that page to clar

Re: Old values on put call

Anthony, currently there's no simple way to achieve what you're after. The API is modelled around a GET before you PUT model. You can specify the option return_body when putting new data into Riak, like so: {ok, RObj} = riak_pb_socket:put (.., NewObject, [return_body]), That returns the freshl

Re: riaksearch: using index docs in place of real objects

Greg, Riak Search stores indexed documents in Riak KV too, as serialized Erlang terms. You can easily verify that by requesting a document from http://riak.host:8098/riak/_rsid_/key. So whenever you query something through the Solr interface the documents you get back are fetched from these bu

Re: Clarification on key filters

Thanks for the clarification. I'll use Ripple to generate some 'canonical' strings to work with while I do other, more nefarious, things with them. Much appreciated. --- Jeremiah Peschka Founder, Brent Ozar PLF On Thursday, May 26, 2011 at 2:42 AM, Mathias Meyer wrote: > Jeremiah, > > I sur

Re: Riak doesn't use consistent hashing

That sounds quite disconcerting. What happens to the performance of the cluster when this occurs?* Jonathan Langevin Systems Administrator Loom Inc. Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com - www.loomlearning.com - Skype: intel352 * On Thu, May 2

Re: Riak doesn't use consistent hashing

Performance is fine. However requests get a "not found" response for an extended period of time. See http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-May/thread.html#4078for previous discussion of what sounds like the same issue. On Thu, May 26, 2011 at 6:57 AM, Jonathan Langevin

Riak Client Resources, Deleting a Key Doesn't Remove it from bucket.keys

All - I just started working with Riak, and am using the riak-client Ruby gem. When I delete a key from a bucket, and try to fetch the value associated with that key, I get a 404 error (which is reasonable). However, it remains in the bucket's list of keys (i.e. the value returned by bucket.ke

Re: Riak Client Resources, Deleting a Key Doesn't Remove it from bucket.keys

Keith, There was a pull-request issue out for this on the Github project (https://github.com/seancribbs/ripple/pull/168). For various reasons, the list of keys is memoized in the Riak::Bucket instance. Passing :reload => true to the #keys method will cause it to refresh. I like to discourage

Re: Riak Client Resources, Deleting a Key Doesn't Remove it from bucket.keys

How long is the key list cached like that, naturally?* Jonathan Langevin Systems Administrator Loom Inc. Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com - www.loomlearning.com - Skype: intel352 * On Thu, May 26, 2011 at 10:35 AM, Sean Cribbs wrote: >

Re: Riak doesn't use consistent hashing.

Hi, Greg. Thanks for your thoughtful analysis and the pull request. On Thu, May 26, 2011 at 1:54 AM, Greg Nelson wrote: > However, the skipping bit isn't part of > Riak's preflist calculation.  Instead, nodes claim partitions in such a way > as to be spaced out by target_n_val, to obviate the n

Re: Riak Client Resources, Deleting a Key Doesn't Remove it from bucket.keys

Sean - Thanks for responding so quickly. I posted a response on github (https://github.com/seancribbs/ripple/pull/168). Regards, Keith On May 26, 2011, at 10:35 AM, Sean Cribbs wrote: > Keith, > > There was a pull-request issue out for this on the Github project > (https://github.com/seanc

Re: Riak Client Resources, Deleting a Key Doesn't Remove it from bucket.keys

With recent commits ( https://github.com/seancribbs/ripple/compare/35d7323fb0e179c8c971...da3ab71a19d194c65a7b ), it is cached until you either refresh it manually by passing :reload => true or a block (for streaming key lists). This was the compromise reached in that pull-request. All of this

Re: Riak Client Resources, Deleting a Key Doesn't Remove it from bucket.keys

Agreed. In fact, jrecursive pointed out to me last week that vnode operations are synchronous. That means that when you call list-keys, not only is it going to take a long time (right now upwards of 5 minutes) to complete, but while each vnode is returning its list of keys *it blocks any other

Re: Riak Client Resources, Deleting a Key Doesn't Remove it from bucket.keys

On May 26, 2011, at 12:40 PM, Sean Cribbs wrote: > With recent commits ( > https://github.com/seancribbs/ripple/compare/35d7323fb0e179c8c971...da3ab71a19d194c65a7b > ), it is cached until you either refresh it manually by passing :reload => > true or a block (for streaming key lists). This was

Re: Riak Client Resources, Deleting a Key Doesn't Remove it from bucket.keys

A cache seems legitimate for performance, but perhaps the cache could additionally be maintained for inserts/deletes? At least then the cache is still being used, but is also accurate. I don't know how expensive that would be though, but hopefully less expensive than a key list reload, correct? *

Re: Riak Client Resources, Deleting a Key Doesn't Remove it from bucket.keys

Additionally, perhaps the automatically updating cache (regarding inserts/deletes) could be an optionally enabled behavior? As there are cases where it could be needlessly expensive (i.e. - high write/delete scenarios), especially when someone does not use the key listing feature. *

Re: Riak Client Resources, Deleting a Key Doesn't Remove it from bucket.keys

In software products that have containment metaphors, how often do we see a function return a cached value rather than the up-to-date value, especially for products that manage shared data? Pretty frequently, actually. Every Ruby ORM I've used caches associations by default. Even when listing is

Riak KV ETS Backend?

Hi all, I'm posting to the list for the first time after talking with Mark (ironically, at MongoSF the other day), about a use-case I'm considering Riak for. Quickly, I should probably say I've only played with Riak on a side project or two, and dabbled with Riak Core thanks to the new GitHub blog

Re: Riak Client Resources, Deleting a Key Doesn't Remove it from bucket.keys

Kyle, you bring up a good point that I feel strongly about -- most people use list-keys as a substitute for better solutions (also known as an anti-pattern). In most cases what they really need is one of: * Better key/schema design, so keys are at least guessable if not knowable. * Secondary in

Re: Riak Client Resources, Deleting a Key Doesn't Remove it from bucket.keys

On Thu, May 26, 2011 at 11:21 AM, Sean Cribbs wrote: > With all of this discussion it has been pointed out to me there are two > issues at hand, possibly conflated as one: > > * Which is the least surprise, caching the key list or the incurring the > large cost of the operation? Or is it that it

Re: Riak doesn't use consistent hashing.

Excellent. Let me know what I can do to help. -Greg On Thursday, May 26, 2011 at 7:37 AM, Justin Sheehy wrote: > Hi, Greg. > > Thanks for your thoughtful analysis and the pull request. > > On Thu, May 26, 2011 at 1:54 AM, Greg Nelson wrote: > > > However, the skipping bit isn't part of > > Ri

Re: Riak doesn't use consistent hashing

The "not found" issue is a different one, but related. The issue there is that when a node joins the ring, the ring state is immediately changed. However, it takes time to handoff partitions to new owners. During that time, if a request comes in for data which has > r of its replicas on partitio

Re: Riak Client Resources, Deleting a Key Doesn't Remove it from bucket.keys

> > 1) I would rather be hit by a large cost that I can see and feel instead of > trying to run down hidden keys from a stale cache (reflect on chasing memory > corruptions...) I think that's fair; it emphasizes the fact that if you shouldn't use it once, you shouldn't be using it twice! (or

Re: Riak Client Resources, Deleting a Key Doesn't Remove it from bucket.keys

I think that configuration key should be a bit more verbose :-)* Jonathan Langevin Systems Administrator Loom Inc. Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com - www.loomlearning.com - Skype: intel352 * On Thu, May 26, 2011 at 3:12 PM, Sean Cribbs w

Re: riaksearch: using index docs in place of real objects

This behavior is specific to the Solr interface. It first fetches document IDs matching the criteria and then fetches the documents from Riak KV. Using the Erlang interface you can fetch just the IDs. It would certainly make sense to add an option like that, but it'd be inconsistent with Solr's

Re: riaksearch: using index docs in place of real objects

Out of curiosity what is the key in this URL? http://riak.host:8098/riak/_rsid_/key On Thu, May 26, 2011 at 9:42 AM, Mathias Meyer wrote: > Greg, > > Riak Search stores indexed documents in Riak KV too, as serialized Erlang > terms. You can easily verify that by requesting a document from > htt

Re: riaksearch: using index docs in place of real objects

Eric, I believe the key is the document id, which will be the same as the key of the corresponding object in . -Greg On Thu, May 26, 2011 at 12:41 PM, Eric Moritz wrote: > Out of curiosity what is the key in this URL? > http://riak.host:8098/riak/_rsid_/key > > On Thu, May 26, 2011 at 9:42 AM, M

Re: riaksearch: using index docs in place of real objects

That is correct, Greg. It's either determined by the key used to store the object in Riak KV (given the precommit hook is used), or by a key specified when indexing directly into Riak Search, using e.g. the Solr or the Erlang API. There'll always be a key required, and that'll be used to look up

riak locking and out of memory

Hi, I am playing around with riak and have set up a 2 node cluster. One node on a desktop ubuntu, and the other node on a macbook pro os/x 10.5. On the macbook I looped across 400meg files using bash and curl to upload them as documents into a bucket: for a in *.gz; do curl -v http://127.0.0.1

Re: riak locking and out of memory

Hi, Ron. On Thu, May 26, 2011 at 4:33 PM, Ron Yang wrote: > On the macbook I looped across 400meg files using bash and curl to > upload them as documents into a bucket: There are other details in your post that I might comment on, but I will focus on the main point. What you describe here simp

Re: Issues with capacity planning pages on wiki

Hi Justin, Thanks for the reply. Good to know you may have some partial solutions for the sizings of items. Out use case may long term require us to write out own backend just for space efficiency, but I'm hoping we can make it quite far with bitcask. I've got enough on my plate at the mome

Eric Moritz Added As A Wiki Committer

Hey All - Some good news from the community I wanted to pass along: we just added Eric Moritz as a Committer to the Riak Wiki. There's a short blog post about it here ---> http://blog.basho.com/2011/05/26/Eric-Moritz-is-Now-A-Wiki-Committer Be sure to give Eric a (virtual) high five next time yo

hidding buckets and keys

hello all, http://IP:8098/riak?buckets=true [ will show all available buckets on Riak ] http://IP:8098/riak/bucketname?keys=true&props=false [ will show all available keys on a bucket ] to me, this proves a very big security risk, as if somebody discovers your Riak server's IP, is very easy to

Re: hidding buckets and keys

Rohman, In our case, the only nodes that are allowed to hit the Riak cluster are those of our applications. We do not allow access to the Riak nodes from the public Internet. Firewall rules are in place to prevent this in some cases, and in others the Riak nodes themselves are on internal networks

Re: hidding buckets and keys

Hi Rohman, It is not recommended that you deploy Riak on the public internet. Keep all access private and then implement iptables on each individual node securing access to upstream clients. Ports to keep in mind - http(s) port (8098) protocol buffers port (8099) epmd (4369) forcing the range

Re: hidding buckets and keys

"In our case, the only nodes that are allowed to hit the Riak cluster are those of our applications"... what if your app is more complex than that and you have thousands of servers all around the world ( different datacenters, different networks ) with crawlers, scanners, blackboxes, etc... all com

Re: hidding buckets and keys

On Thu, May 26, 2011 at 8:10 PM, Antonio Rohman Fernandez < roh...@mahalostudio.com> wrote: > what if apart from webservers with a web-app i want to build > iPhone/iPad/Android apps that access Riak directly? Unfortunately, Riak just isn't designed for that. You might be able to work around it

Re: hidding buckets and keys

On 27 May 2011, at 07:10, Antonio Rohman Fernandez wrote: > "In our case, the only nodes that are allowed to hit the Riak cluster are > those of our applications"... what if your app is more complex than that and > you have thousands of servers all around the world ( different datacenters, > d

Re: hidding buckets and keys

"riak only available on localhost and nginx facing the outside world"... that sounds like something worth trying! thanks.even i still think it could be great to have some options to enable/disable those "?buckets=true" and "?keys=true"Rohman On Fri, 27 May 2011 07:40:45 +0100, Russell Brown wrote: