Re: Problem with deleting keys

ey(s)" if (bad_keys_counter + good_keys_counter) % 1000 == 0 end -- View this message in context: http://riak-users.197444.n3.nabble.com/Problem-with-deleting-keys-tp3069985p4031578.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Deleting Keys vs. Specifying a Null Value for a Key

I was pretty sure I mentioned it quite early on in this discussion ;-). Something about logical deletes vs. physical deletes. Cheers, Nico Am Donnerstag, den 23.06.2011, 10:21 -0700 schrieb Greg Nelson: > Something to keep in mind here -- which I don't think has been > mentioned yet -- is the int

Re: Deleting Keys vs. Specifying a Null Value for a Key

Something to keep in mind here -- which I don't think has been mentioned yet -- is the interplay with Bitcask. One may want to *actually* delete the key rather than store application-defined tombstones to keep from accumulating entries in Bitcask's in-memory keydir. On Thursday, June 23, 2011 a

Re: Deleting Keys vs. Specifying a Null Value for a Key

On 6/23/2011 11:16 AM, Jeremiah Peschka wrote: The bigger question, though, is "Why do I want to store a completely empty record?" In the context of riak, the answer might be that record changes are handled differently than record deletes. If you want the things that happen to make concurre

Re: Deleting Keys vs. Specifying a Null Value for a Key

probably store no data unless there is customization and just provide sane defaults in the app code to cope with an absence of data. TL;DR - magic numbers suck and NULLability is a holy war. --- Jeremiah Peschka Founder, Brent Ozar PLF, LLC On Thursday, June 23, 2011 at 9:00 AM, Keith Bennett w

Deleting Keys vs. Specifying a Null Value for a Key

Speaking of deleting keys, I have a question. It may be naive, as I'm a developer, but not very specialized in data storage issues, especially key/value stores. I don't think I've brought this up on this list, but if I have, I apologize, and feel free to ignore this message. (

Re: Problem with deleting keys

On Thu, Jun 23, 2011 at 09:59:38AM +0200, Runar Jordahl wrote: > As this discussion shows, deleting objects in an eventual consistency > database is complicated. The concept of multiple APIs to do a delete, > and the possibility of having objects reappear, can scare new > customers away from Riak.

Re: Problem with deleting keys

As this discussion shows, deleting objects in an eventual consistency database is complicated. The concept of multiple APIs to do a delete, and the possibility of having objects reappear, can scare new customers away from Riak. I understand delete is complicated, but Basho should make it as simple

Re: Problem with deleting keys

As Nico suggests below, although this is not an elegant solution, I would be satisfied with causing the deletes to fail unless all primary nodes are up. On 6/16/2011 12:09 PM, Nico Meyer wrote: > If all primary nodes are up, the problem is nonexistent. You could > certainly implement any number

Re: Problem with deleting keys

However this problem does get solved it needs to be backend agnostic and function at the riak level. -Alexander On Thu, Jun 16, 2011 at 14:02, Nico Meyer wrote: > Expiry time in bitcask is a global setting at the moment, not per key. But > even if it where different, this should behave exactly th

Re: Problem with deleting keys

On Thu, Jun 16, 2011 at 10:24 AM, Andrew Thompson wrote: > On Thu, Jun 16, 2011 at 09:24:28AM -0700, David Leimbach wrote: > > Riak doesn't have tombstones (as far as I know) so, you have to make sure > > all your nodes are up to do a delete. This, to me, seems like a > misfeature. > > > > Please

Re: Problem with deleting keys

Expiry time in bitcask is a global setting at the moment, not per key. But even if it where different, this should behave exactly the same as outright deletion (what else is expires now supposed to mean?). Hence the same problems as David observed in the first place. Which makes me think, actu

Re: Problem with deleting keys

Actually it would be nicer if my account would be the whole story, but we may be getting there at some point ;-) Am 16.06.2011 19:25, schrieb Greg Nelson: ...you shrug, replace it with a new one, and then trigger read-repair on all keys that were on the node, then go about your business. ;-)

Re: Problem with deleting keys

Agreed, if that's going to continue to play well with index/search, etc. Is everything involved with those operations distributed as well? On 6/16/2011 12:23 PM, Nico Meyer wrote: I don't see any reason to only restore one node of your cluster. That is the whole point of a replicated datastore

Re: Problem with deleting keys

In 0.14, would it be reasonable for the application to write its own special tombstone marker while at the same time setting an expiry time (== now) on the object? (Assuming bitcask backend...) On Thursday, June 16, 2011 at 10:24 AM, Andrew Thompson wrote: > On Thu, Jun 16, 2011 at 09:24:28AM -

Re: Problem with deleting keys

...you shrug, replace it with a new one, and then trigger read-repair on all keys that were on the node, then go about your business. ;-) On Thursday, June 16, 2011 at 10:23 AM, Nico Meyer wrote: > I don't see any reason to only restore one node of your cluster. > That is the whole point of a re

Re: Problem with deleting keys

On Thu, Jun 16, 2011 at 09:24:28AM -0700, David Leimbach wrote: > Riak doesn't have tombstones (as far as I know) so, you have to make sure > all your nodes are up to do a delete. This, to me, seems like a misfeature. > Please read my other responses in this thread, riak absolutely *does* have t

Re: Problem with deleting keys

I don't see any reason to only restore one node of your cluster. That is the whole point of a replicated datastore. If you loose a node, you shrug, replace it with a new one and go about your business. Am 16.06.2011 19:21, schrieb Les Mikesell: But we expect something different in the case wher

Re: Problem with deleting keys

But we expect something different in the case where a value was updated and the more current value is available on other nodes. Restoring an old value don't matter then. On 6/16/2011 12:07 PM, David Leimbach wrote: Or any data from backup? :-) On Thu, Jun 16, 2011 at 10:04 AM, Nico Meyer ma

Re: Problem with deleting keys

I see you get my point ;-) Am 16.06.2011 19:07, schrieb David Leimbach: Or any data from backup? :-) On Thu, Jun 16, 2011 at 10:04 AM, Nico Meyer > wrote: Yes, but the same is true if you restore a relational DB from backup :-D Am 16.06.2011 18:49,

Re: Problem with deleting keys

On Thu, Jun 16, 2011 at 09:19:51AM -0700, Greg Nelson wrote: > "A tombstone > is put and then an async get is fired off and if the async get finds all > the primary nodes in the preflist are up, it does the delete. If not, > the next time the key is fetched, it does the same check again and will >

Re: Problem with deleting keys

On Thu, Jun 16, 2011 at 06:09:37PM +0200, Kresten Krab Thorup wrote: > > On Jun 16, 2011, at 5:53 PM, Andrew Thompson wrote: > > > I've also > > added a 'deletedvclock' GET option to change the return behaviour for > > when a tombstone is found to instead return {deleted, VClock} so you can > > s

Re: Problem with deleting keys

Or any data from backup? :-) On Thu, Jun 16, 2011 at 10:04 AM, Nico Meyer wrote: > Yes, but the same is true if you restore a relational DB from backup :-D > > Am 16.06.2011 18:49, schrieb Les Mikesell: > > On 6/16/2011 10:22 AM, Nico Meyer wrote: >> >>> >>> >> >> >> Btw, this whole issue is n

Re: Problem with deleting keys

On Thu, Jun 16, 2011 at 9:49 AM, Les Mikesell wrote: > On 6/16/2011 10:22 AM, Nico Meyer wrote: > >> >> > > > Btw, this whole issue is not really Riak specific. It is essentially a >> consequence of eventual consistency, where you have to make a trade off >> between the amount of bookkeeping inf

Re: Problem with deleting keys

Yes, but the same is true if you restore a relational DB from backup :-D Am 16.06.2011 18:49, schrieb Les Mikesell: On 6/16/2011 10:22 AM, Nico Meyer wrote: Btw, this whole issue is not really Riak specific. It is essentially a consequence of eventual consistency, where you have to make a

Re: Problem with deleting keys

On 6/16/2011 10:22 AM, Nico Meyer wrote: Btw, this whole issue is not really Riak specific. It is essentially a consequence of eventual consistency, where you have to make a trade off between the amount of bookkeeping information you want to store and the maximum amount of time (or number o

Re: Problem with deleting keys

On Thu, Jun 16, 2011 at 2:57 AM, Nico Meyer wrote: > Hello David, > > this behaviour is quite expected if you think about how Riak works. > Assuming you use the default replication factor of n=3, each key is stored > on all of your three nodes. If you delete a key while one node (let's call > it

Re: Problem with deleting keys

"A tombstone is put and then an async get is fired off and if the async get finds all the primary nodes in the preflist are up, it does the delete. If not, the next time the key is fetched, it does the same check again and will do the delete then if the downed node is up." What if the key is never

Re: Problem with deleting keys

Sorry, didn't see your mail in time. This is essentially what i was talking about in my other mail. So all is good then :-). Cheers, Nico Am 16.06.2011 17:53, schrieb Andrew Thompson: On Thu, Jun 16, 2011 at 08:40:45AM -0700, Greg Nelson wrote: Well, it is kind of Riak specific. An impleme

Re: Problem with deleting keys

If all primary nodes are up, the problem is nonexistent. You could certainly implement any number of solutions inside Riak, but I cannot see that there is 'one right way(tm)' to do it. In such cases I am all in favor of providing all the needed primitives to implement whatever is best for your a

Re: Problem with deleting keys

On Jun 16, 2011, at 5:53 PM, Andrew Thompson wrote: > I've also > added a 'deletedvclock' GET option to change the return behaviour for > when a tombstone is found to instead return {deleted, VClock} so you can > safely override a tombstone, instead of triggering a merge (or creating > siblings).

Re: Problem with deleting keys

On Thu, Jun 16, 2011 at 08:40:45AM -0700, Greg Nelson wrote: > Well, it is kind of Riak specific. An implementation that treated DELETEs > like PUTs (tombstones w/ vector clocks for ordering), then this would not be > an issue, right? When no primary nodes are down, the tombstones can be > physi

Re: Problem with deleting keys

Well, it is kind of Riak specific. An implementation that treated DELETEs like PUTs (tombstones w/ vector clocks for ordering), then this would not be an issue, right? When no primary nodes are down, the tombstones can be physically deleted on the backend. A logical delete could never reappear i

Re: Problem with deleting keys

The problem with unreachable nodes still remains, since you don't know how long they will be gone. The only 'safe' minimum time to keep deleted values is forever. This can be easily emulated in the application layer by using a special value (or use Riak metadata for example). So it essentially a

Re: Problem with deleting keys

...when doing a delete, Riak actually stores a "deleted" record, but then it is too eagerly deleting it for real after that. There should be a configurable "zombie time" between requesting a delete and the "deleted record" being deleted for real; so that the deleted record's vector clock will sh

Re: Problem with deleting keys

Hello David, this behaviour is quite expected if you think about how Riak works. Assuming you use the default replication factor of n=3, each key is stored on all of your three nodes. If you delete a key while one node (let's call it A) is down, the key is deleted from the two nodes that are s

Re: Problem with deleting keys

I was seeing some similar behavior recently on a local devrel cluster. With a node down, introducing DELETEs into the mix seemed to cause a fair bit of wonkiness. Especially with allow_mult=true. I can consistently do something along these lines to reproduce at least one of the problems I was s

Problem with deleting keys

Erlang: R13B04 Riak: 0.14.2 I have a three node cluster, and while one node was down, I deleted every key in a certain bucket. Then, I started the node that was down, and it joined the cluster. Now, when do a listing on these keys in this bucket, and I get the entire list. I can also get the

Re: deleting keys

Good point, Eric. This is the primary reason that Ripple switched to ISO 8601 format as the default; also that other languages will often use ISO 8601, whereas RFC 822 is less common. Sean Cribbs Developer Advocate Basho Technologies, Inc. http://basho.com/ On Jun 1, 2011, at 10:33 AM, Eric Mo

Re: deleting keys

Depending on what you need the date for, you may be able to skip the parsing step. If you're using it for sorting or comparison, ISO8601 can be compared lexicographically and usually no parsing is needed, "earlier_isodate < later_isodate" will be true in Javascript even though they're strings. I

Re: deleting keys

Rob, Yes, you can use the provided iso8601.js file (only generated for you in a Rails 3 project, unfortunately, but you can grab it from inside the gem or on github) to parse ISO8601 Dates. You'll need to set the js_source_dir in app.config to point to the directory where this lives. If we upg

Re: deleting keys

Thanks, I was trying to use the number of keys as an indication of how many objects there were in a bucket but I see that is not the smartest approach. I didn't see anything that was an indication of the number of items in a bucket , is there anything if riak like this or do I need to create

Re: deleting keys

Robert, What Keith said is misleading -- that key cache was solely in the Ruby client driver and not part of Riak itself. In Riak, deletes have two phases; in the first, so-called "tombstones" are written to the partitions that own replicas of the key. The tombstone has special metadata marki

Re: deleting keys

Robert - Until a source code change a few days ago, riak would by default cache the keys reported to be in a bucket, so after fetching them once they would not be updated after deletions, additions, etc. The key is indeed gone, but the keys API did not report the change. If you go to the mess

deleting keys

Hi, When deleting a key from a bucket I'm noticing that the object associated with the key is gone but the key itself is still sticking around. I loop though all of the keys in a bucket and then call delete on each one, the object for the key is then gone so if I try to get the object for that