Great, this explains why bitcask went merging crazy after we updated
from 0.14 to 1.1! The workaround was to constrain merging to the
nighttime, and since it kind of worked(TM) since then, I never came
round to look for the reason for the increased I/O load after the update.
I just checked, an
Am 26.09.2012 23:14, schrieb Anthony Molinaro:
On Wed, Sep 26, 2012 at 08:33:24PM +0200, Nico Meyer wrote:
Great, this explains why bitcask went merging crazy after we updated
from 0.14 to 1.1! The workaround was to constrain merging to the
nighttime, and since it kind of worked(TM) since then
Am 26.09.2012 23:20, schrieb Anthony Molinaro:
On Wed, Sep 26, 2012 at 10:04:54PM +0200, Kresten Krab Thorup wrote:
While we're discussing merge and expiry semantic here, I'd like to explain how
HanoiDB does it since that may be a reason for trying it out on your problem.
With the HanoiDB back
Hi Alfonso,
there simply is no connection between bitcasks key expiration feature
and the rest of Riak, including the search.
To say it bluntly, Riak has no 'real' key expiration that would work
like one would naively expect. That is, like someone magically coming
along and issuing a full fleg
Hi Kresten,
after the first node has been removed, you have a cluster and an extra
machine, haven't you ;-).
Lets take you example of 5 nodes.
So instead of having to do 5 node leaves and 5 joins, you can do 1 node
leave, 4 replaces and 1 join.
Cheers,
Nico
Am 28.09.2012 13:26, schrieb Kre
Hi Sebastian,
the way you proposed just works. I used this myself a lot lately.
For us the reason why we don't want to restart a node for this, is that
it takes about 40 minutes to load all keys into memory.
Cheers,
Nico
Am 23.10.2012 13:44, schrieb Sebastian Cohnen:
Hey list,
I'd like to r
Hi,
since we upgraded our riak cluster from 0.14 to 1.2, we see MapReduce
queries failing while any of the nodes is starting but not yet ready
(that is while the 'Waiting for service riak_kv' message still appears
in the logs).
This is quite problematic, since it takes almost 50 minutes for t
Hi Jeremiah!
Actually there should be no compaction at all if he only ever inserts
new keys, so the expire feature of bitcask won't help in this case.
Compactions/Merges only happen if keys have been updated or deleted.
Cheers,
Nico
Am Donnerstag, den 10.02.2011, 09:52 -0800 schrieb Jeremiah Pes
Hi Mike,
perhaps you can try to upgrade the protocol buffers library to at least
version 2.3.0. This is from the changelog for that version:
Python
* 10-25 times faster than 2.2.0, still pure-Python.
Cheers,
Nico
Am Montag, den 14.02.2011, 19:35 -0500 schrieb Mike Stoddart:
> Will do when I
gt;
> On Tuesday, February 15, 2011 at 1:46 AM, Nico Meyer wrote:
>
> > Hi Mike,
> >
> > perhaps you can try to upgrade the protocol buffers library to at
> > least
> > version 2.3.0. This is from the changelog for that version:
> >
> > Pytho
setup.py, so people are already most likely using
> protobufs-2.3.0.
>
>
> - Andy
>
> On Tuesday, February 15, 2011 at 3:09 AM, Nico Meyer wrote:
>
> > Hi Andy.
> >
> > I am not quite sure what you mean, is the protobuf library included
> > with
>
Whenever I see latencies which are roughly multiples of 40ms it screams
to me 'nagle algorithm'. I have seen this so often now, that the first
thing I check is, if the TCP_NODELAY option is set on the TCP socket on
both ends.
Am Freitag, den 25.02.2011, 08:14 -0600 schrieb Jeremiah Peschka:
> Here
Just out of curiosity I did some tests myself on one of our production
machines. We normally only use the ProtocolBuffers interface and a
thrift interface that we wrote ourselves.
If I fetch a small key (~250 bytes), the riak server becomes CPU bound
with about 20 concurrent requests, at which poi
Feb 25, 2011 at 2:43 PM, Nico Meyer wrote:
> > Just out of curiosity I did some tests myself on one of our production
> > machines. We normally only use the ProtocolBuffers interface and a
> > thrift interface that we wrote ourselves.
> >
> > If I fetch a small key (~
Hi Greg,
I don't think the vnodes will always die. I have seen some situations
(disk full, filesystem becoming read only due to device errors,
corrupted bitcask files after a mashine crash) where the vnode did not
crash, but the get and/or put requests returned errors.
Even if the process cras
meone could point me to the place where a
vnode failure is detected. As far as I can see, the heavy lifting
happens in riak_kv_util:try_cast/5 (
https://github.com/basho/riak_kv/blob/riak_kv-0.14.1/src/riak_kv_util.erl#L78),
which only checks if the whole node is up.
On 24.03.2011 00:56, Nico
Hi Anthony,
are you sure you are not including the filesystem cache in your
mem_allocated values? It will grow to use all of the free memory or the
total size of your bitcask data files, whichever is smaller.
We have about 100Mio keys per node, and riak uses about 7GB of RAM.
Cheers,
Nico
O
ith such evenly
> space data, everything tends to fall over at once. I just don't want that
> to happen with my riak cluster, so am wondering how to tell if you are close
> to needing to grow.
>
> Anyone have any ideas?
>
> -Anthony
>
>
> On Thu, Mar 24, 2011
e fewer than
> 4 bytes, or b) your ratio of keys to buckets is near 1.
>
>
> I'm sure I am missing something, though, and haven't poked into that
> part of the code yet.
>
>
> On Thursday, March 24, 2011 at 3:11 AM, Nico Meyer wrote:
>
> > Hi Anthony,
&
Hi Sean,
I have to object here. We have a cluster of 8 Core/64GB nodes with an
SSD drive for the bitcask dir. Each node holds on the order of 100Mio
keys. The complete bitcask directory is only about 60Gb big, so it fit
almost completely.
The time from starting the node until it start handling re
repair tables. YMMV
>
> Sean Cribbs
> Developer Advocate
> Basho Technologies, Inc.
> http://basho.com/
>
> On Apr 1, 2011, at 10:11 AM, Nico Meyer wrote:
>
> > Hi Sean,
> >
> > I have to object here. We have a cluster of 8 Core/64GB nodes with an
&g
Hi everyone,
I just want to note that I observed similar behaviour with a somewhat
larger clusters of 10 or so nodes. I first noticed that handoff activity
after node join (or leave for that matter) involved a lot more
partitions than I would have expected. By comparing the old and the new
ring fi
Hi Gary,
I had this problem before a few times.
It is cause by leftover files named 'bitcask.write.lock'. Normally theey
should be cleared automatically after a crash. They contain the OS
process ID of the erlang VM which created them, and if there is no such
process bitcask deletes the file.
Hi Dave,
I believe problem occours if there happens to be another process with
the same PID as the old (now gone) riak node. This can happen if the
machine was rebooted since the riak node crashed or if the PIDs wrapped,
they are only two bytes after all.
os_pid_exists/1 only checks for ANY proces
2011, at 7:10 AM, David Smith wrote:
Hmm...ok. Will have to ponder how we can fix that.
Thanks!
D.
On Mon, May 9, 2011 at 8:09 AM, Nico Meyer wrote:
Hi Dave,
I believe problem occours if there happens to be another process with
the same PID as the old (now gone) riak node. This can happen if t
Hi Anthony,
I think, I can explain at least a big chunk of the difference in RAM and
disk consumption you see.
Let start with RAM. I could of course be wrong here, but I believe the
/'static bitcask per key overhead/' is just plainly too small. Let me
explain why.
The bitcask_keydir_entry s
n, NC: (910) 241-0433 - jlange...@loomlearning.com
<mailto:jlange...@loomlearning.com> - www.loomlearning.com
<http://www.loomlearning.com/> - Skype: intel352
/*
*
On Wed, May 25, 2011 at 12:22 PM, Nico Meyer <mailto:nico.me...@adition.com>> wrote:
Hi Anthony,
I
Hi Michael,
Greg's advice is probably the best, if you really always want to read
back or update predefined groups of 1000 keys at once. It will increase
the rate at which you can write and read by a factor of 1000 ;-).
But if that's not what you want to do, and we really don't know what
you
Hello Jacques,
It's A.
Cheers,
Nico
Am 30.05.2011 18:43, schrieb Jacques:
Hello,
I'm working on a riak deployment and have a question. I've read that
the approximate storage size per key is 40 bytes plus the size of the
key. Does the total storage size include the bucket name AND the item
Hi Justin,
I wanted to write this earlier, but I just had to much on my plate:
Am 08.06.2011 16:11, schrieb Justin Sheehy:
On Thu, Mar 24, 2011 at 1:51 PM, Nico Meyer wrote:
The bigger concern for me would be the way the bucket/key tuple is
serialized:
Eshell V5.8 (abort with ^G)
1
Hello David,
this behaviour is quite expected if you think about how Riak works.
Assuming you use the default replication factor of n=3, each key is
stored on all of your three nodes. If you delete a key while one node
(let's call it A) is down, the key is deleted from the two nodes that
are s
27;s better (more consistent),
but not a real fix.
Kresten
On 16/06/2011, at 11.58, "Nico
Meyer"mailto:nico.me...@adition.com>> wrote:
Hello David,
this behaviour is quite expected if you think about how Riak works.
Assuming you use the default replication factor of n=3, each key
n Thursday, June 16, 2011 at 8:22 AM, Nico Meyer wrote:
The problem with unreachable nodes still remains, since you don't know
how long they will be gone. The only 'safe' minimum time to keep deleted
values is forever. This can be easily emulated in the application layer
by using a s
Sorry,
didn't see your mail in time. This is essentially what i was talking
about in my other mail.
So all is good then :-).
Cheers,
Nico
Am 16.06.2011 17:53, schrieb Andrew Thompson:
On Thu, Jun 16, 2011 at 08:40:45AM -0700, Greg Nelson wrote:
Well, it is kind of Riak specific. An impleme
Yes, but the same is true if you restore a relational DB from backup :-D
Am 16.06.2011 18:49, schrieb Les Mikesell:
On 6/16/2011 10:22 AM, Nico Meyer wrote:
Btw, this whole issue is not really Riak specific. It is essentially a
consequence of eventual consistency, where you have to make
I see you get my point ;-)
Am 16.06.2011 19:07, schrieb David Leimbach:
Or any data from backup? :-)
On Thu, Jun 16, 2011 at 10:04 AM, Nico Meyer <mailto:nico.me...@adition.com>> wrote:
Yes, but the same is true if you restore a relational DB from
backup :-D
Am 16.06.2
case where a value was
updated and the more current value is available on other nodes.
Restoring an old value don't matter then.
On 6/16/2011 12:07 PM, David Leimbach wrote:
Or any data from backup? :-)
On Thu, Jun 16, 2011 at 10:04 AM, Nico Meyer mailto:nico.me...@adition.com>> wrot
. ;-)
On Thursday, June 16, 2011 at 10:23 AM, Nico Meyer wrote:
I don't see any reason to only restore one node of your cluster.
That is the whole point of a replicated datastore. If you loose a node,
you shrug, replace it with a new one and go about your bus
Expiry time in bitcask is a global setting at the moment, not per key.
But even if it where different, this should behave exactly the same as
outright deletion (what else is expires now supposed to mean?). Hence
the same problems as David observed in the first place.
Which makes me think, actu
I was pretty sure I mentioned it quite early on in this discussion ;-).
Something about logical deletes vs. physical deletes.
Cheers,
Nico
Am Donnerstag, den 23.06.2011, 10:21 -0700 schrieb Greg Nelson:
> Something to keep in mind here -- which I don't think has been
> mentioned yet -- is the int
Hello,
I created a feature request for compression of values about a year ago
(https://issues.basho.com/show_bug.cgi?id=412), but unfortunately it
seems there was no interest in that.
For the time being we have to patch Riak to enable compression, though
we do it in the riak_kv_vnode module by cha
Hi Matthew,
although I really like Riak, I would seriously consider using Hadoop or
something similar (perhaps Disco http://discoproject.org/, if you don't
like Java), since you only need to read/write your data sequentially.
Assuming your example is representative of your typical log entry
(s
qual competence of both languages, but I can't imagine it's too hard to
implement that.
Matt
____
From: Nico Meyer [nico.me...@adition.com]
Sent: Wednesday, June 29, 2011 7:17 AM
To: riak-users@lists.basho.com
Cc: Evans, Matthew
Subject: Re: Riak or Cas
Hi Apyr,
I have had this problem in the past. Most likely the
1309482913.bitcask.data is corrupted somehow. I mostly see this problem
after a machine crash or disk problems, and in this case it is alway the
end of the file that is corrupted or truncated (the last record that
Riak was trying t
Hi Jacques,
please read my reply back in May, which should clear things up:
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-May/004292.html
Cheers,
Nico
On 16.08.2011 21:09, Jacques wrote:
We're utilizing Riak 14.2 and we're seeing higher memory consumption
than we expect.
W
usage over 100. I'm also notclear what the step changes
are. If you have any insights on these that would be great. Thanks!
On Aug 17, 2011 12:07 AM, "Nico Meyer" <mailto:nico.me...@adition.com>> wrote:
> Hi Jacques,
>
> please read my reply back in May, which
o be collected?
It sounds like you're generally saying that "it is what it is" for now
and just capacity plan around it.
thanks again,
Jacques
On Thu, Aug 18, 2011 at 3:31 AM, Nico Meyer <mailto:nico.me...@adition.com>> wrote:
Have you taken into account the size of
Hello Jim,
it sounds like the ring_creation_size in your app.conf file is
inconsistent with the actual ring size on your existing nodes. This
would happen, if the ring_creation_size in the config was changed after
the ring as been created. As the name implies, the setting is only used
when startin
Which of course doesn't only helps if data is corrupted on disk or on
the way to/from disk ;-)
Cheers,
Nico
Am 03.02.2012 14:06, schrieb David Smith:
On Fri, Feb 3, 2012 at 1:35 AM, Runar Jordahl wrote:
As far as I know, Riak does not provide any checksum on your data. I
hope I am wrong.
Ha
Fri, Feb 3, 2012 at 7:03 AM, Nico Meyer wrote:
Which of course doesn't only helps if data is corrupted on disk or on the
way to/from disk ;-)
Good call, we should add that.
Thanks,
D.
---
Dave Smith
Director, Engineering
Basho Technologies, Inc.
diz...@basho.com
--
Senior Backend Deve
Hello Michael,
the siblings returned by the Protobuf API have two properties calles
'last_mod' and 'last_mod_usec', which contain the (unix-)timestamp and
the microsend part for the time on which the sibling was last modified.
allow_mult_=true actually uses the contents of this field to return
Hello,
I just encountered a problem with one of our Riak nodes, which is caused
by a bug in either the disk controller or the firmware of our SSD disks.
Anyway, the obvious symptom is, that all writes to the disks suddenly
fail, which of course leads to truncated bitcask files. However, this
t
not_found, state()}'
Cheers,
Nico
Am 18.04.2012 12:18, schrieb Nico Meyer:
Hello,
I just encountered a problem with one of our Riak nodes, which is
caused by a bug in either the disk controller or the firmware of our
SSD disks.
Anyway, the obvious symptom is, that all writes to the di
cask directory while
writing in it, so i assume any not completely-written bitcask file can
cause it. Easy way looks like dropping bitcask directory .
On Wed, Apr 18, 2012 at 2:26 PM, Nico Meyer <mailto:nico.me...@adition.com>> wrote:
Oh, I forgot to mention:
Hi!
I have a question regarding the use of fallback nodes for get requests.
As I read the code, get requests use fallback nodes if any of the nodes
normally holding the key are not reachable. Also a simple test seemed to
confirm this.
What I mean is, if I do a get request for a key with R=N, and
to reason about
these things in German and face-to-face :-).
Cheers,
Nico
Am Montag, den 02.08.2010, 22:29 -0400 schrieb Justin Sheehy:
> Hi, Nico.
>
> On Mon, Aug 2, 2010 at 1:19 PM, Nico Meyer wrote:
>
> > What I mean is, if I do a get request for a key with R=N, and one of
I just saw https://issues.basho.com/show_bug.cgi?id=275 , which would
actually be just what I need.
Am Mittwoch, den 04.08.2010, 13:58 +0200 schrieb Nico Meyer:
> Hi Justin,
>
> I think we are coming from two different directions here, leading to
> some confusion. You seem to treat
One should also mention, that the overall performance of bitcask might
presumably become worse with a larger number of partitions per node.
If there is only one partition writing is basically linear on the disk,
an therefore extremely fast. If you on the other hand linearly write to
a large number
Hi,
I have also seen similar (not quite the same though) problems when
removing nodes from the cluster. Normally what happens, is that most
partitions are moved away from the node that was removed but the handoff
of some partitions will always produce an error like this:
=ERROR REPORT 10-Nov-
Hi,
Am Donnerstag, den 11.11.2010, 07:59 +0100 schrieb Sven Riedel:
> Hi,
> thanks for the detailed reply. So you would suggest that somehow the
> partition allocation got into an incosistent state across nodes. I'll
> have to check the logs to see if anything similar to your dump pops
> up.
>
>
Am Freitag, den 12.11.2010, 08:43 +0100 schrieb Sven Riedel:
> >
> > It's not a problem right away. But since the replicated data is not
> > actively synchronized in the background the keys that were not copied
> > until the node dies have one less replica. That is until they are read
> > at least
Hi Gary,
this is a known bug in the python client. See
https://issues.basho.com/show_bug.cgi?id=695
The fix is fairly simple and also mentioned in the bug report.
Cheers,
Nico
Am Montag, den 17.01.2011, 21:18 -0800 schrieb Gary William Flake:
> I am using the python client api and in some of my
Hi,
let me clarify the situation somewhat. The problem is not intermittent
in the sense that it occurs randomly. It depends mainly on the size of
the answer that riak sends. If the answer is bigger than the network MTU
the error will occur most of the time. If its smaller it occurs almost
never. T
Hi,
maybe I am wrong here, but from looking at the code of
riak_kv_multi_backend I get the impression that the backend name can be
anything (binary, atom, integer, ...), as long as it is the same in the
multi_backend config and the 'backend' property of the bucket.
The limitation to binaries come
Have you tried raising the limit for open filehandles (ulimit -n)?
Each TCP connection also uses one filehandle, and by default they will
linger around for 60 seconds after the connection is closed.
The PHP client uses the REST interface afaik, so I assume it will open a
new TCP connection for eac
ico.
> Much appreciated. I'm embarrassed that we opened up a load test
> process on the same host as the data store and didn't realize we were
> spammer ourselves :)
>
>
> On Fri, Jan 28, 2011 at 04:44, Nico Meyer wrote:
> > Have you tried raising the limit for open
66 matches
Mail list logo
