Re: How do I improve Level DB performance?

2012-05-10 Thread David Smith
On Thu, May 10, 2012 at 2:33 PM, Tim Haines wrote: > I've set up a new cluster, and have been doing pre-deployment benchmarks on > it. The benchmark I was running slowly sunk from 1000 TPS to 250 TPS over > the course of the single 8 hour benchmark doing 1 read+1 update using 1k > values.  I'm wo

Re: 50-100 ops/sec on a 5 Node cluster

2012-06-26 Thread David Smith
What does your basho_bench config file look like? Seeing a low number of ops/sec is usually an indicator that you're hitting a bottleneck on the I/O path -- typically due to random reads. LevelDB has a number of characteristics that can make this worse -- what happens if you use bitcask? D. On T

Innostore users?

2012-06-26 Thread David Smith
Hi all, As the mid-July release of Riak looms large, the Basho team has been discussing our continued support of the Innostore backend. Embedded InnoDB was long ago abandoned by Oracle. We have not been actively working on it either. We believe that alternative designs (like LevelDB) are the futu

Re: Innostore users?

2012-07-09 Thread David Smith
Just a brief follow up -- I have yet to hear from anyone that they are using Innostore in a production setting. As such, we will NOT be releasing Innostore as part of the Riak 1.2 release later this month. Thanks! D. On Tue, Jun 26, 2012 at 3:45 PM, David Smith wrote: > Hi all, > > A

Re: reg. storageon Localfile system

2012-07-10 Thread David Smith
Hi Sangeetha, If you are using one of the Linux packages for Riak, you should be able to look in /var/lib/riak for the data files. Otherwise, consult app.config for the data_root directory. Hope that helps, D. On Mon, Jul 9, 2012 at 4:46 AM, wrote: > Hi team, > > > > Iam very

Re: Key expiration

2012-09-20 Thread David Smith
Sorry -- there is not currently an event that notifies external systems when keys are being expired. The expiration is done at merge time, so you could potentially get a very large number of events all at once. D. On Thu, Sep 20, 2012 at 12:14 PM, Mike Oxford wrote: > When a key comes up on expi

Re: Riak compile error on Fedora 12 (32 bit)

2011-02-11 Thread David Smith
On Fri, Feb 11, 2011 at 6:05 PM, Mike Stoddart wrote: > I'm trying to compile Riak on Fedora 12 (32 bit) but I'm getting an error: > ==> rel (generate) > ERROR: Unable to generate spec: Undefined applications: [inets] Try installing the erlang-inets package via yum. It's unfortunate they break u

Re: Riak compile error on Fedora 12 (32 bit)

2011-02-11 Thread David Smith
nks David, I already had erlang-inets installed. I even tried > reinstalling it but it didn't make any difference. > > Mike > > On Fri, Feb 11, 2011 at 9:39 PM, David Smith wrote: >> On Fri, Feb 11, 2011 at 6:05 PM, Mike Stoddart wrote: >>> I'm trying to

Re: Riak compile error on Fedora 12 (32 bit)

2011-02-12 Thread David Smith
ht. > > Thanks > Mike > > > > -- Sent from my Palm Pre > ____ > On 12 Feb 2011 12:33 a.m., David Smith wrote: > > It might not be perfect, but I spun up a VM + buildslave for fedora > and build an RPM this evening: > > http://dl.dropbox.

Re: Riak Java Client

2011-02-16 Thread David Smith
On Tue, Feb 15, 2011 at 5:37 PM, Mark Wolfe wrote: > Gday > Been messing around on some personal projects which use riak with Java and > was interested to know if their was a roadmap or task list of the java > client. This is one of the things we're working on solidifying and would love to get fe

Re: 0.14 crash on OS X

2011-02-17 Thread David Smith
sasl-error.log is usually the first place of interest, but be warned it gets overwritten between restarts. I would suggest trying to start up with "console" method (after making sure other process is gone) and see what happens. D. On Thu, Feb 17, 2011 at 10:09 AM, francisco treacy wrote: > Riak

Re: Riak compile error on Fedora 12 (32 bit)

2011-02-17 Thread David Smith
e most appreciated. Could you add the FC13 RPMs to the official > download page or are older systems frowned upon? > > Thanks > Mike > > On Sat, Feb 12, 2011 at 9:37 AM, David Smith wrote: >> I tried out a FC13 on 12 the other day and it _seemed_ to work fine. >> But that was

Re: Riak compile error on Fedora 12 (32 bit)

2011-02-18 Thread David Smith
rch-0.14.0-1.fc13.x86_64.rpm Hope that helps, D. On Fri, Feb 18, 2011 at 5:50 AM, Mike Stoddart wrote: > Great thanks! Any chance you could put up a 64 bit riak search for > FC12/13 as well please? Sorry to be a nuisance! > > Thanks > Mike > > On Thu, Feb 17, 2011 at 4:15

Re: Can't start Riak Search

2011-02-18 Thread David Smith
I should note that we've not had a chance to validate these fedora packages to the degree I'd like -- this was simply an early-morning-coffee-sipping-I'm-gonna-try-and-build-packages approach. :) When 0.14.1 lands, we should have sorted out these issues. Thanks, D. On Fri, Feb 18, 2011 at 8:22 A

Re: /stats throws a 500 error

2011-03-09 Thread David Smith
There was a bug fixed in 0.14.1 release: https://issues.basho.com/show_bug.cgi?id=971 which has this symptom. Without knowing what version of Riak, the amount of load you are exerting on the server, etc. it's hard to say definitively if you are experiencing this specific issue. :) D. On Tue, Ma

Re: Bitcask - large keydirs

2011-03-11 Thread David Smith
TLDR: Yes, good idea. We'd welcome any patches/pull requests you have ;) There are significant tradeoffs here, particularly as the accesses devolve into a uniformly random pattern. The rebalancing and secondary disk hits would also have some interesting impacts to the predictability of our lat

Re: Multicore machine: one node vs several nodes

2011-04-22 Thread David Smith
On Fri, Apr 22, 2011 at 9:25 AM, Mikhail Sobolev wrote: > Hi, > > I wonder what setup would be better for a machine with 16 cores. > > Please note that I do not put any meaning for "better" here. :) My general philosophy is to start simple and construct a baseline so that such statements can be q

Re: crash after single insert

2011-05-09 Thread David Smith
On Sat, May 7, 2011 at 9:25 AM, Gary William Flake wrote: > That was it, Nico.  Thanks. > > I know we did a forced shutdown this week, which was probably the cause.  But > I would have thought that riak would have taken care of its own lock file > bookkeeping on restarting. Bitcask does: https

Re: crash after single insert

2011-05-09 Thread David Smith
ag, den 09.05.2011, 07:06 -0600 schrieb David Smith: >> On Sat, May 7, 2011 at 9:25 AM, Gary William Flake wrote: >> > That was it, Nico.  Thanks. >> > >> > I know we did a forced shutdown this week, which was probably the cause.   >> > But I would have

Re: When is the next Riak Search stable release

2011-05-19 Thread David Smith
Hi Odelon, This is something we're working on and plan to ship as part of upcoming 0.14.2 release. Thanks, D. On Tue, May 3, 2011 at 11:49 AM, Odelon Miranda wrote: > Hi, > > Just like to ask if anyone knows when will be the next riak-search release? > > Thanks, > > Odie > > -- > We stretch ou

Re: Issues with capacity planning pages on wiki

2011-05-23 Thread David Smith
On Mon, May 23, 2011 at 9:39 PM, Anthony Molinaro wrote: > >      Bitcask-Capacity-Planning     Cluster-Capacity-Planning   Reality > RAM            34.9 GB                        34.9 GB             70 GB > Disk          102 GB                          18.49 GB           341 GB > > So it looks to

Re: luwak not properly overwriting data

2011-06-06 Thread David Smith
r 20, 2011 at 4:04 PM, francisco treacy >> wrote: >>> >>> Hi all, >>> >>> Any updates regarding this bug?  It's constantly biting me. >>> >>> Thanks! >>> Francisco >>> >>> >>> 2010/11/30 David Smith

riak_search moving forward

2011-06-06 Thread David Smith
Hi all, One of the things we've got cooking at Basho HQ is a plan to roll the functionality currently in the riak_search project into the default Riak distribution. This means that using riak_search will become a matter of flipping a configuration switch versus running a whole separate instance. T

Re: riak_search moving forward

2011-06-07 Thread David Smith
On Tue, Jun 7, 2011 at 9:02 AM, Jasinek, Jason wrote: > > Are there plans to add in additional Erlang Analyzers that would provide > additional terms?  Analyzers that I would like to see added would including > stemming and ngrams. Hi Jason, We're not currently planning on implementing those s

Re: riaksearch memory growth issues

2011-06-07 Thread David Smith
Gordon, Thanks for the test case. I've queued it up for review by a dev, as time permits. D. On Tue, Jun 7, 2011 at 1:33 PM, Gordon Tillman wrote: > Guys I have put together a simple test to reproduce the error that we are > seeing. > It is on github here: > https://github.com/gordyt/riaksearch

Re: bitcask hash algo

2011-06-08 Thread David Smith
On Tue, Jun 7, 2011 at 11:39 PM, Aaron Blohowiak wrote: > as far as i can tell, bitcask c_src is using the murmurhash2 algo, which has > a known flaw ( https://sites.google.com/site/murmurhash/murmurhash2flaw ).. > while this is not *likely* to cause an issue, I was wondering if there was a > reas

Next few months in Riak HEAD

2011-06-13 Thread David Smith
All, The next few months will see a number of major new features and changes dropping into the tip (aka HEAD) of the Riak codebase. The team at Basho is very excited to get these new pieces of code out there and we believe that this upcoming release will make Riak even more reliable and easier to

Re: Errors when using Basho Bench make results

2011-06-13 Thread David Smith
This really does sound like an environmental issue. As a workaround, you could try running it w/ Rscript --vanilla from command line. D. On Thu, Jun 9, 2011 at 3:36 PM, Ken Perkins wrote: > I've followed the directions verbatim for setting up basho bench. I > successfully ran a test, but now I g

Re: Solr Search

2011-06-17 Thread David Smith
Hi Jason, Gary Flake and the crew at Clipboard submitted a pull request on this ticket a while back: http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-May/004219.html We merged in the changes (with a little tweaking) in this pull request: https://github.com/basho/riak_search/pull/5

Re: Riak or Cassandra for this...

2011-06-28 Thread David Smith
On Tue, Jun 28, 2011 at 9:17 AM, Evans, Matthew wrote: > > The disadvantages of Riak are: > > 1) Write performance. We need to handle ~50,000 writes per second. I would note that we are investigating the use of LevelDB in lieu of InnoDB due to superior write performance and better crash recovery.

Re: Lots of bitcask files for a vnode, unable to merge

2011-06-30 Thread David Smith
The fix is quite easy -- you can safely delete any zero-length files (while the node is down!). How it got to that state is less clear. The "every three minutes" suggests that the vnode triggers a merge only to find there is nothing to actually merge. It is odd, however, that it doesn't clean up t

Re: Lots of bitcask files for a vnode, unable to merge

2011-07-01 Thread David Smith
On Fri, Jul 1, 2011 at 4:30 AM, Nico Meyer wrote: > > Bitcask doesn't handle corrupted files very well in all cases, which is > unfortunate. I patched our version of Riak/bitcask, to gracefully handle all Yes, we've recently corrected some of these edge cases here: https://github.com/basho/bitca

Re: Future plans

2011-07-01 Thread David Smith
Hi Stanislavs, On Thu, Jun 30, 2011 at 5:15 PM, Staņislavs Koikovs wrote: > > * Distributed counters; > * Geospatial indexing support in search; > * Offline / batch processing for data. As you noted, we're sorta busy with the existing features for the next major release. :) Insofar as counters g

LevelDB driver

2011-07-01 Thread David Smith
Hi all, Yesterday, we merged in support for the LevelDB (http://code.google.com/p/leveldb/) as a backend driver for Riak. Initial benchmarking of LevelDB suggests that it is competitive to InnoDB (aka innostore) for most use cases. The recovery model is much better, as LevelDB uses an append-only

Re: LevelDB driver

2011-07-01 Thread David Smith
On Fri, Jul 1, 2011 at 12:53 PM, Will Moss wrote: > This is very cool--glad you guys decided to bundle this in. The linked post > and the Google Code page both suggest that it will have a much more > efficient on-disk representation than InnoDB, do you have an specific > numbers on overhead per ke

Re: LevelDB driver

2011-07-12 Thread David Smith
On Mon, Jul 11, 2011 at 4:24 PM, Will Moss wrote: > I know you guys are busy working on various new features, is this going to > make it into the next production release? Is there a date for that? Is there > somewhere to look at your road map so I don't have to spam the list? Hi Will, Yes, the

Re: Slow performance on search-cmd set-schema

2011-07-12 Thread David Smith
On Tue, Jul 12, 2011 at 7:51 PM, Ken Perkins wrote: > What could be causing the delay on the rackspace VM? Is it purely a VM > performance issue, or is there something else going on. The blessing of the cloud: It hides the Hard Stuff. The curse of the cloud: It hides the Hard Stuff. :) First ord

Re: Riak Recap for July 20 - 24

2011-07-26 Thread David Smith
On Tue, Jul 26, 2011 at 10:12 AM, Kev Burns wrote: > Right on. > http://www.slideshare.net/rklophaus/querying-riak-just-got-easier-introducing-secondary-indices > Looking forward to seeing this in master :) > https://github.com/basho/riak_kv/commits/rtk-dev1 > 0.15? 1.0. D. -- Dave Smith Direc

Re: Heart

2011-08-03 Thread David Smith
Generally, I would advise against using heart as it has no rate limiting when restarting, which can lead to vicious cycles of restart. The presence of "heart" in the riak scripts was my attempt (way back when) to ensure that the generic Erlang scripts provided access to as much functionality as po

Re: Riak 1.0 - what to expect?

2011-08-04 Thread David Smith
Jonathan, We're planning to ship 1.0 at the end of September. A couple of the items that are planned for this release: * Secondary indicies * Improved MapReduce system (based on riak_pipe) * riak_search merged into base product (versus a separate download) * Logging system overhaul There are (a

Re: Crashed node has Bitcask merge errors on restart

2011-08-05 Thread David Smith
Hi Jeff, I believe you are encountering BZ 1097 (http://issues.basho.com/1097), where a suddenly truncated bitcask file can cause problems when attempting to merge. The truncation is typically the result of underlying O/S or hardware failure and simply means that the last record in a bitcask file

Re: Crashed node has Bitcask merge errors on restart

2011-08-05 Thread David Smith
On Fri, Aug 5, 2011 at 6:49 AM, Jeff Pollard wrote: > Update: now the node has crashed, due to the following lines in the > sasl-error.log (see below).  I've also attached the crash dump to this > email. > Real quickly though, just to confirm - If we wanted to restore the node from > a recent back

Re: GET vs map reduce

2011-08-19 Thread David Smith
As Matt and Jacques noted, M/R is really not intended to be used in this manner (i.e. for multi-get), particularly if you're interested in latency. Generally, you will wind up involving more nodes on a M/R request and shipping the "get" results across those nodes as the system tries to distribute M

Re: Riak Bitcask merging

2011-08-23 Thread David Smith
On Tue, Aug 23, 2011 at 2:47 AM, Nathan Wilken wrote: > > Does this make sense?  I'd have thought my app.config would keep the bitcasks > small, but it seems they just grow bigger and never get cleared of much > expired data.  The manual config I use above immediately shrinks the data > files d

Re: "..including key filters ... is an expensive operation that should not be used in production"

2011-08-23 Thread David Smith
On Tue, Aug 23, 2011 at 10:47 AM, Craig Muth wrote: >> until some upcoming changes land, and always on 0.14.x > When will the non-blocking key filtering land in a stable release? Non-blocking key _listing_ will land in master sometime before end of August. There are a number of changes across the

Re: git://github.com/basho/riak.git - master branch - 2 questions

2011-08-29 Thread David Smith
On Tue, Aug 9, 2011 at 4:54 AM, Joseph Wayne Norton wrote: > > Q1 - For a new installation, is there a missing mkdir for the data dir? >     OR is somewhere else the data directory being created before the > automatic creation of the ssl_distribution.args_file file? I believe the first was a tran

Re: innostore performance tuning

2011-08-30 Thread David Smith
On Tue, Aug 30, 2011 at 5:31 PM, David Koblas wrote: > > The one issue that I see with LevelDB in a big insert job like this is file > descriptors, I've currently got about 9,000 files open on the machine, I'm > guessing that I'll run out of file descriptors before the load is done.   Is > there a

Re: Bitcask with sync_strategy o_sync seem to have no effect

2011-08-30 Thread David Smith
On Tue, Aug 30, 2011 at 8:57 PM, Tatsuya Kawano wrote: > The current behavior could be dangerous if someone rely on {sync_strategy, > o_sync} for durability. > > My colleague checked Bitcask implementation and found that it uses "fcntl > (2)" system call to set O_SYNC flag to the file descripto

Re: Bitcask with sync_strategy o_sync seem to have no effect

2011-09-05 Thread David Smith
On Tue, Aug 30, 2011 at 8:57 PM, Tatsuya Kawano wrote: > My colleague checked Bitcask implementation and found that it uses "fcntl > (2)" system call to set O_SYNC flag to the file descriptor. Unfortunately, > O_SYNC being set by this way will be ignored by kernel and has no effect. To > make

Riak 1.0 pre-release series

2011-09-07 Thread David Smith
Hi All, I'm proud to announce that we're taking the first steps towards releasing Riak 1.0 with the creation of the 1.0 branch and initial package builds. With 1.0, we will be doing two series of builds before the final release. The first stage is "pre-release" (aka PR) builds that represent the

Re: Riak 1.0 pre-release series

2011-09-07 Thread David Smith
> repos? (i.e. - apt-get for Debian, for instance) > > * > <http://www.loomlearning.com/> > Jonathan Langevin > Systems Administrator > Loom Inc. > Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com - > www.loomlearning.com - Skype: intel352 > * &

Re: Riak 1.0 pre-release series

2011-09-07 Thread David Smith
n > Systems Administrator > Loom Inc. > Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com - > www.loomlearning.com - Skype: intel352 > * > > > > On Wed, Sep 7, 2011 at 10:55 PM, David Smith wrote: > >> Hi Jonathan, >> >> We're working

Re: Riak 1.0 pre-release series

2011-09-08 Thread David Smith
On Thu, Sep 8, 2011 at 5:41 AM, Greg Stein wrote: > I just checked out the link (provided below) and did not find a > *source* distribution. Could you work that into the distribution? Yes, there should be one of those -- will ping Jared. > In my particular situation, I'm working on Mac OS 10.5

Re: Bitcask with sync_strategy o_sync seem to have no effect

2011-09-08 Thread David Smith
On Tue, Sep 6, 2011 at 9:14 PM, Tatsuya Kawano wrote: > Hi David, > > OK, so there is no easy fix on Linux. I think generating the warning > message will help. Thanks for look into this. I would note that it's possible to configure a periodic sync still with bitcask on Linux; it's just going to b

Re: Riak 1.0.0pre3 build is now available for download

2011-09-14 Thread David Smith
Out of curiosity, what shell are you using? That looks like a shell problem to me... D. On Wed, Sep 14, 2011 at 7:25 PM, Jeremiah Peschka wrote: > I get errors trying to build eleveldb on OS X 10.7.1 > > ==> eleveldb (compile) > : command not foundform: line 2: > : command not foundform: line 5:

Re: Riak 1.0.0pre3 build is now available for download

2011-09-14 Thread David Smith
t;Jeremiah Peschka" > wrote: >> Bash >> >> Jeremiah Peschka >> Founder, Brent Ozar PLF >> On Sep 14, 2011 10:17 PM, "David Smith" wrote: >>> Out of curiosity, what shell are you using? That looks like a shell >>> problem to me... >

Re: Riak 1.0.0pre3 build is now available for download

2011-09-15 Thread David Smith
this file is generated by build_detect_platform to set build flags > include build_config.mk > --- > Jeremiah Peschka - Founder, Brent Ozar PLF, LLC > Microsoft SQL Server MVP > > On Sep 14, 2011, at 10:28 PM, David Smith wrote: > >> On Wed, Sep 14, 2011 at 9:21 PM, Jeremiah

Re: Precommit Hook Difficulties

2011-09-19 Thread David Smith
On Sat, Sep 17, 2011 at 5:11 PM, Greg Pascale wrote: > Hi, > I'm trying to write a simple precommit hook to modify a JSON object by > removing certain fields. The simplest way to do this, I figure, is to decode > the object's value with mochijson2, remove the fields I don't want, > re-encode it, a

Re: pre3 and pre4 build issue

2011-09-20 Thread David Smith
A quickfix for this issue is to simply copy: deps/sext/priv/git.vsn -> deps/sext/priv/vsn.git We'll fix this for the next build. Thanks, D. On Tue, Sep 20, 2011 at 10:45 AM, Dan Reverri wrote: > Hi TJ, > Thank you for reporting this issue. We have a bug filed for this issue and > are working

Re: Search precommit hook bug?

2011-09-21 Thread David Smith
On Tue, Sep 20, 2011 at 12:41 PM, Elias Levy wrote: > Now from what I've been able to find,{error,emfile} usually means you are > out of file descriptors.  Yes? Yes. If you are running a default configuration, however, you will need considerably more than 4096 file handles. You can tighten this

Re: Search precommit hook bug?

2011-09-22 Thread David Smith
On Wed, Sep 21, 2011 at 10:26 AM, Elias Levy wrote: > Any idea what kind of performance hit one takes by lowering the number of > open file handles? I just submitted a pull request to lower the default to 10; should show up in next RC. As for performance, in the tests I've run I've not observed

Re: leveldb crash on 1.0.0 rc1

2011-09-23 Thread David Smith
On Thu, Sep 22, 2011 at 7:07 PM, Phil Sorber wrote: > > From watching top when the nodes crashed, it looked IO related. The IO > wait went up and then there was only one thread of beam.smp running > and it was using 100% CPU. Sometimes the nodes would recover after the > benchmark was stopped, som

Re: leveldb crash on 1.0.0 rc1

2011-09-23 Thread David Smith
from all 4 machines. > > On Fri, Sep 23, 2011 at 7:47 AM, David Smith wrote: >> On Thu, Sep 22, 2011 at 7:07 PM, Phil Sorber wrote: >>> >>> From watching top when the nodes crashed, it looked IO related. The IO >>> wait went up and then there was only one thre

Re: secondary indexes - no multi_backend?

2011-09-29 Thread David Smith
Artem, This is a known issue that didn't make the cutoff for 1.0. We'll be working to address this in the point releases following. D. On Thu, Sep 29, 2011 at 3:04 PM, Artem wrote: >> We are pleased to announce the availability of the second release >> candidate for Riak 1.0. > > The RC2 doesn'

Re: Timeout when storing

2011-09-30 Thread David Smith
IIRC, {error, emfile} indicates that the max # of ports (in the erlang VM) is being exceeded. Try bumping up ERL_MAX_PORTS in vm.args. D. On Thu, Sep 29, 2011 at 10:52 PM, Jim Adler wrote: > Thanks Sean.  I added the ulimit -n 10240 to /etc/default/riak, restarted > riak, but that didn't work. >

Riak 1.0

2011-09-30 Thread David Smith
Hi All, I am very pleased to announce that Riak 1.0 is tagged and officially ready for downloading. Pre-built installations and source tarballs are available at: http://downloads.basho.com/riak/CURRENT Release notes are here: https://github.com/basho/riak/blob/1.0/RELEASE-NOTES.org There is a

Re: Timeout when storing

2011-10-03 Thread David Smith
707326989568713251046585937826284568576/1316879145.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1316995340.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1317493005.bitcask.data/var/lib/riak/bitcask/60515302170732698956871325104

Next Riak release

2011-10-10 Thread David Smith
Hi All, Riak 1.0 has now been out for 10 days. It is shaping up to be a great success and warmly received by the community -- thank you! With your help, we have identified a few bugs that warrant a patch release: 1228 - riak_core_ring_manager crashes with function clause and state of 'ok' 1234 -

Re: Riak 1.0 LevelDB issue

2011-10-12 Thread David Smith
A couple of questions... 1. What does your app.config look like? 2. Can you capture an output of lsof on the riak process? Thanks, D. On Fri, Oct 7, 2011 at 1:10 PM, Dan wrote: > Hi, > > after storing more than one million keys (random key ids, 200 byte datas) > over http to a riak standalone

Re: Do not expose Riak to the Internet

2011-10-20 Thread David Smith
On Thu, Oct 20, 2011 at 11:27 AM, Aphyr wrote: > I *do* know that multitenant mongo is vulnerable to trivial > denial-of-service vulnerabilities, thanks to a global write lock and > gleefully executing javascript everywhere. While we're talking DoS, it's > I think we can all agree it's a bad ide

Re: Severe problems when adding a new node

2011-10-28 Thread David Smith
Hi John, et. al. On Fri, Oct 28, 2011 at 5:03 PM, John Axel Eriksson wrote: > > I don't want to be too hard on you fine people of Basho and you provide a > really great system in Riak and I understand what you're aiming for, but if > anything as bad as this ever happens in the future you might w

Re: Eleveldb backend randomly chewing all disk

2011-11-15 Thread David Smith
On Mon, Oct 24, 2011 at 5:29 PM, Elias Levy wrote: > This morning, while performing a query against one of the nodes, Riak > crashes.  I then notice there is no disk space left.  Upon investigating, I > see that all disk space is being consumed by the eleveldb backend.  All 45GB > of of.  Looking

Re: Eleveldb backend randomly chewing all disk

2011-11-16 Thread David Smith
On Tue, Nov 15, 2011 at 6:01 PM, Jeremy Raymond wrote: > I've seen issues when leveldb runs out of file handles. The leveldb > log then fills with error messages. Hmm -- this could be. However, I would expect that the Erlang VM would eventually shutdown with errors about emfile in that situation.

Re: rebar and validate_app_modules property

2011-11-17 Thread David Smith
On Wed, Nov 16, 2011 at 11:45 PM, Dmitry Demeshchuk wrote: > Hi, guys. > > I've recently noticed that newest rebar requires .app.src files to > list the modules to be compiled, unless {validate_app_modules, false} > property is set in rebar.config: > https://github.com/basho/rebar/blob/master/src/

Re: Integrating Riak

2011-11-22 Thread David Smith
On Tue, Nov 22, 2011 at 4:36 PM, Andrew Berman wrote: > 4.  Does it make more sense to just install a riak package and use the > erlang pb client?  Seems like it would be less efficient especially since > these will live on the same machine. This is the preferred way to attack this problem. Separ

Re: RiakEDS - one way replication?

2011-12-06 Thread David Smith
On Tue, Dec 6, 2011 at 8:19 AM, Kresten Krab Thorup wrote: > Hi folks, > > Quick question - can Riak EDS do one-way replication, or does it always have > to be two-way? Yup, it can be one-way. D. -- Dave Smith Director, Engineering Basho Technologies, Inc. diz...@basho.com __

Re: Riak Recap for December 2 - 4

2011-12-06 Thread David Smith
On Mon, Dec 5, 2011 at 3:48 PM, Mikhail Sobolev wrote: > I'm curious whether it anyhow suggests that there's something wrong with > leveldb backend (perhaps, in specific use cases??)...  (A colleague of > mine and I are considering using 2i (which is only available with > leveldb backend) and any

Re: Strange merge time variance

2011-12-19 Thread David Smith
On Thu, Dec 15, 2011 at 10:51 AM, Dmitry Groshev wrote: > I'm using standalone bitcask as a backend for our in-memory database > (not opensourced yet). I write ~1.5GB of data to it each 5 minutes > overwriting almost all keys and after that I run a merge as follows: If the database is in memory,

Luwak EOL

2011-12-19 Thread David Smith
Hi all, I wanted to take a moment and let the community-at-large know that we are going to end-of-life the Luwak functionality in Riak, as part of our 1.1 release in February. This simply means that the Luwak repo on Github will not be actively developed/supported by Basho, or included as a default

Re: Luwak EOL

2011-12-21 Thread David Smith
On Tue, Dec 20, 2011 at 11:08 PM, Paul Gross wrote: >> Does this mean there is still an intention to support storing larger >> values in Riak in the future? > > Or is this something the client libraries should implement? You could certainly implement the necessary chunking and such in client lib

Re: Status of Python library

2011-12-21 Thread David Smith
On Wed, Dec 21, 2011 at 4:31 PM, Andrew Fisher wrote: > I've noticed a lot of activity on the Python client library at the moment. > Is there any sense of when we're going to hit a stable build of this that > could be incorporated into a production environment using 1.0? Hi Andrew, Getting the

Re: Riak as a backup solution

2011-12-28 Thread David Smith
On Fri, Dec 23, 2011 at 3:55 AM, Luca Spiller wrote: > > Has anyone done anything like this before, and if so how has this > worked out for you? With Luwak support being deprecated I'm guessing > this isn't the direction Basho want to go with Riak, but it is an > interestion suggestion none the l

Re: LevelDB and open files count

2012-01-03 Thread David Smith
On Fri, Dec 30, 2011 at 1:35 PM, jeffrey k eliasen wrote: > In reading the LevelDB page on the wiki, it indicates as max ring size goes > up, the number of open files should go down. However, later (in the "Tips and > tricks" section) it says that there will be (by default) 20 open files per >

Re: Rebalancing pain and 100 percentile stat

2012-01-10 Thread David Smith
On Tue, Jan 10, 2012 at 11:17 AM, Elias Levy wrote: > Is this expected behavior?  Granted, these are EC2 boxes and Leveldb depends > heavily on disk, but I can't imagine folks using Riak on production his this > type of performance hit resulting from rebalancing. tl;dr - yes, adding nodes can cr

Re: Riak Control installed when upgrading to 1.0.2?

2012-01-10 Thread David Smith
On Tue, Jan 10, 2012 at 12:58 PM, Jason J. W. Williams wrote: > How does Riak Control differ from the management tools included in the > enterprise version? Riak Control is a redesign of those management tools and a release of that functionality to Open Source. We want it to be drop-dead easy for

Re: Stats for MR jobs and Riak Search

2012-01-12 Thread David Smith
Elias, +1, especially when it comes to search. Figuring out how to measure MR, where you'd really only have per-node stats, is a bit of a challenge. Do you have any specific stats you'd like to see? D. -- Dave Smith Director, Engineering Basho Technologies, Inc. diz...@basho.com __

Re: Luwak EOL

2012-01-12 Thread David Smith
On Thu, Jan 12, 2012 at 12:40 PM, Mikhail Sobolev wrote: >> >> Can you explain more detailed reason for EOL Luwak?  And is there some >> replacement or alternative solution for the part Luwak taking in Riak? > Seconding Zheng's request: would it be possible to hear the reasons for > Luwak EOL? Hi

Re: Corruption of leveldb database files

2012-01-16 Thread David Smith
Thomas, Thanks for the heads up -- I saw that bug report on the Level repo this morning. D. On Mon, Jan 16, 2012 at 9:50 AM, Thomas Bakketun wrote: > Hello, > > We have found a bug in leveldb that causes corruption of the CURRENT > file of leveldb databases. A bug report has been filed: > http:

Re: Riak in the cloud?

2012-01-18 Thread David Smith
Hi Tom, You can certainly run Riak in the cloud -- you simply need to be aware that there is likely to be a latency tradeoff, since you do not have direct control over the hardware and it's difficult to tune for available resources. In addition, there is no guarantee that multiple instances of the

Re: Corruption of leveldb database files

2012-01-26 Thread David Smith
t;> that a service release of Riak with this fix included is done as soon as >> possible. >> >> >> On 2012-01-16 17:52, David Smith wrote: >>> Thomas, >>> >>> Thanks for the heads up -- I saw that bug report on the Level repo this >>> m

Re: Corruption of leveldb database files

2012-01-26 Thread David Smith
Yes, the latest eleveldb now builds Snappy and you can enable/disable with {compression, true|false} setting. This will be part of 1.1 release of Riak (in next few weeks). D. On Thu, Jan 26, 2012 at 10:58 AM, Elias Levy wrote: > On Thu, Jan 26, 2012 at 6:27 AM, wrote: >> >> I just bumped the v

Re: does Riak cluster maintain fully connected Erlang network

2012-01-28 Thread David Smith
On Sat, Jan 28, 2012 at 8:18 AM, Sharas gpavarde wrote: > Hi > > I wonder does Riak cluster maintain fully connected Erlang network between > machines? > As far as I know Erlang node network has a practical limit of around 20 > machines > Does this limit influence Riak? Hi Sharas, Erlang/Riak do

Re: PBS: Better understanding latency vs. consistency trade-offs in Riak

2012-01-30 Thread David Smith
Hi Peter, Thanks for the email re: PBS -- it's quite interesting stuff. On Thu, Jan 19, 2012 at 6:03 PM, Peter Bailis wrote: > > Thus far, in addition to general Dynamo-style replication analysis, we've > developed a patch for Cassandra that performs the required latency profiling > and are inte

Re: HiPE and native libs

2012-02-02 Thread David Smith
Last time I benchmarked it, more than a year ago, it didn't make any meaningful difference in performance. That may have changed, or perhaps the compilation of Erlang core stuff wasn't done right -- benchmarks/data to the contrary would be welcome. Generally, I have the feeling (personally) that H

Re: Operations: Commodity hardware OK, or does Riak need ECC memory?

2012-02-03 Thread David Smith
On Fri, Feb 3, 2012 at 1:35 AM, Runar Jordahl wrote: > As far as I know, Riak does not provide any checksum on your data. I > hope I am wrong. Happily, you are wrong. :) For both the LevelDB and Bitcask backends, Riak calculates and verifies CRCs on data stored to disk. D. -- Dave Smith Direc

Re: Operations: Commodity hardware OK, or does Riak need ECC memory?

2012-02-03 Thread David Smith
On Fri, Feb 3, 2012 at 7:03 AM, Nico Meyer wrote: > Which of course doesn't only helps if data is corrupted on disk or on the > way to/from disk ;-) Good call, we should add that. Thanks, D. --- Dave Smith Director, Engineering Basho Technologies, Inc. diz...@basho.com ___

Re: Ruby client release v1.0.0

2012-02-03 Thread David Smith
Awesome, congrats! D. On Fri, Feb 3, 2012 at 8:33 AM, Sean Cribbs wrote: > Hey riak-users, > > I'm happy to announce the release of the Riak client for Ruby, version > 1.0.0.  There's a whole lot of great stuff in this release, so please do > check out the release > notes: https://github.com/bas

Re: Announcing Riak 1.1 release candidate 2

2012-02-10 Thread David Smith
Hi Mark, We are working on 1160, but the odds are we'll need to provide a patch shortly after the release. It's mostly an annoyance right now (i.e. no data is lost because of it), but an incorrect fix could become something more serious. At the very least, we'll provide a workaround script for 1.1

Re: standby cluster experiment

2012-02-13 Thread David Smith
On Mon, Jan 23, 2012 at 3:40 PM, John Loehrer wrote: > At this point one or more of the nodes will start misbehaving, and you > will see the data directory grow at the rate of about 100 MB per second > until it fills up the disk or until you restart riak on that node. > The disk usage problem was

Re: standby cluster experiment

2012-02-16 Thread David Smith
On Tue, Feb 14, 2012 at 3:05 PM, John Loehrer wrote: > as requested, I reproduced the bug, then straced the process to track it down > to a specific dir in leveldb: > > /var/lib/riak/leveldb/274031556999544297163190906134303066185487351808 > > Attaching all the non sst data files in that director

  1   2   >