Hi Bogunov, This is a lengthy email, and I'm still a bit hazy from all the holiday egg nog, but I'll do my best to respond here...
On Tue, Dec 13, 2011 at 5:59 PM, Bogunov <bogu...@gmail.com> wrote: > Hi, > I have been doing research about Riak use cases and patterns (talked to a > few people with production usage experience). > I have tried using it for developing some project from scratch a few months > ago, but received to much resistance from my team, so dropped it =). Thought > i think the technology developed by basho is great, so i want to write an > article or maybe give a talk about it at some local developer conference =) > Bummer about your team. Keep at it. They will come around. Try this out: spin up a 3-5 node cluster for some internal service used by the dissenters, kill a few nodes, and don't tell them for a few days :) As far as writing an article or giving a local talk, that sounds like a great idea. (I'll make sure to reserve you a spot in the Recap if your efforts are published.) > So from my own experience and words of those, who i`m talked with - I > developed such a picture in my head: > > Riak works very well as "real" k\v: > - user session, user data streams(with append only data) (like facebook > wall), and likes. > not so good for relational\indexed data: > - before 2i indexes you have to roll own your own indexes using riak(but >2 > disk reads can be quite costly) or redis\postregsql\mysql (and support their > own sharding\migration\etc, which kind of forces you to write your own > sharding logic anyway, which really doesn't make sense as you expected riak > to take care of that part). > - with 2i indexes you are doing mini-M\R job and this have some performance > penalties. > So this is mostly correct. Riak is (in my very biased opinion) the most-production ready, distributed k/v store available out there. If your data model and access patterns can fit into a k/v model (and you would be surprised how many can), then I say look no further. I wouldn't necessarily call 2i a "mini-m/r job". There are a lot of nuances here and I apologize for being pedantic. 2i and listkeys both use the coverage query. m/r can use either 2i or listkeys as an input. For the HTTP interface to Riak, there is an endpoint to just the 2i component, but the protocol buffers interface currently requires you to use the m/r interface. Performance will vary based on app design, cluster resources, etc. It's also worth nothing that in the next release, we are aiming to make the http 2i interface and the m/r interface equally-performant. InfoQ published an interview last week [1] with Andy Gross and myself that talks about use cases and best practices that might be worth reading if you haven't stumbled onto it already. > Not as well as m\r engine: > - it is used for multiget (i suppose for languages like php, because doing > map-reduce is easier than doing multi-thread reading) > - it is used for run-once tasks like building index. > but : > - using it ad-hoc for analytics seems to be too slow ( why riak doesn't > support "pre"-reduce on "map"-vnode like actual google M\R ?). [actually > this statement based on 0.14 usages, don't know anybody who has used > riak-pipe ] > This all depends on what you are classifying as an "m/r engine." Riak 1.x's MapReduce is based on riak_pipe, which is a significant improvement over the previous m/r framework. As it stands now, pipe has not been optimized for the type of processing that might lend itself to the work that a traditional m/r engine would do. But with enough time and developer resources, it's not unlikely that we would build in more support for this AND add some interfaces to the various client libraries to make it more accessible. If you're Erlang skills are reasonably strong, you can plug directly into the pipe framework (I've spoken with a handful of people who are doing just this) and take advantage of pipe's processing primitives. I would checkout the Pipe README [2] or ask nicely on the list for ideas on how to do this. But on the whole Riak's m/r is designed for realtime querying over a known key set, not long running analytics. > RiakSearch have been developed for a: checkbox "we have full-text search". > I will compare it with Sphinx(http://sphinxsearch.com/) as i got quite a lot > experience with it(used it extensively and know author personally =)). > I wouldn't call RiakSearch something that was written for a "checkbox." It was written to solve a set of problems associated with distributed, full-text search. If you're familiar with Sphinx, you should probably use it. I don't have any direct experience with it but I'm told it's quite powerful. > RiakSearch does have this magical "availability" built-in, but only for > writes: if you lost shard`s data you have to "reindex" ALL your dataset > completely, because you don't know which part of the dataset, that shard > held (if you lost sphinx`s shard you will have to do this too, but because > sharding will be configured in sphinx`s conf manually, you won't need to > reindex all data, only lost part). > Re: built-in availability for writes - correct. But we have plans to address this, and there is an open issue you can keep an eye on to track the progress of this. [3] > If Sphinx`s shard dies you will have to wait for it to timeout, but you can > configure timeout, but as far as i understand you can't configure timeout in > RiakSearch. > > RiakSearch can consume a lot of memory while searching low cardinality terms > because it uses term-based sharding and there is no way to fix this (except > inline fields), sphinx uses fixed amount of memory for sorting and > weighting(and document-base sharding) (this obviously leads to approximate > results + limit to returned ids, but for full-text search this is > acceptable), can i configure same fixed memory usage for RiakSearch request > ?. Right. We advise people to stay away from low cardinality terms. Have a look at the Search/2i/MapReduce comparison [4] for more info on why and how to mitigate this. > Plus sphinx has configurable ranking, grouping, sorting and weighting, and > plain faster. Thought sphinx has accumulated at least 10-20 man-years of > development and more features =) > > So here are the questions: > 1) I think I'm a little biased here, but why would i use RiakSearch at all ? > I think using sphinx\solr(elastic search) will give me more benefits as they > are specialized full-text search engines with most of the search-related > problems solved already. And RiakSearch will force me to reindex lost > node\data as any other search engine, so practically no gain from that > feature . The main selling point I can see for someone like you (who is already steeped in Sphinx) is that Riak and Riak Search are fully-integrated, meaning you won't have to run a separate search system on top of your Riak cluster. But like I said, if you're keen on Sphinx, no reason to switch now. > 2) Does anybody uses Riak`s M\R on daily basis for analytics ? Or everybody > tend to use specialized data-warehouse products for that matter ? I know of few companies who use m/r in this capacity, but I'm in no position to speak on their behalf... Anyone have a moment to share some stories/opinions? > 3) Anybody uses 2i heavily ? Using sharded redis\*sql seems to be wrong, but > that was the only way before 2i ? Maybe there are trickier way to do this > (keeping indexes at ets\lets backend ) ? How majority of users do this ? See my response to the previous question. > 4) What happens with 2i then one of "covering" vnode fails ?Can i configure > timeout for it ? If the physical node on which the vnonde resides is unavailable, the coverage query will use vnodes on other nodes to satisfy it. If the vnode fails during the query, the request will most-likely timeout as we don't currently monitor this and you'll probably get an error like: {error, insufficient_vnodes_available} > 5) Theoretical question regarding quorums: what happens if I issue a (w = 2) > put, first vnode aknowledge write, but the second will fail, and quorum will > trigger failure, does first node will roll-back write (I assume - no) ? > If it doesn't then will it be deleted during read-repair ? Because having > sibling of write which i aknowledged as failed, and "replayed" later can > cause too much inconsistency. What is the right way to handle this ? The write shouldn't fail assuming you have at least one physical node up. Riak will send the write to another vnode that will then hold onto it until the physical node holding the partition in question comes back online or is removed from the cluster for good. > 6) Does anybody uses some sort of sorting\pagination with Riak ? Do you > prebuild specialized sorted index for it to work ? Using RiakSearch for it > doesn't seems to be a good idea. There was recent thread about pagination and Search on the list recenlty [5] that might help you out. > Thanks in advance =) No problem. Keep the questions coming. Mark [1] http://www.infoq.com/news/2011/12/andy-gross-riak-database [2] https://github.com/basho/riak_pipe [3] https://issues.basho.com/show_bug.cgi?id=1304 [4] http://wiki.basho.com/MapReduce-Search-2i-Comparison.html [5] http://riak.markmail.org/search/?q=pagination%2C%20search%20#query:pagination%2C%20search%20+page:1+mid:v4yghl4tromitu6m+state:results _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
