There are a few follow-ups on my mind:
Peter, it's good that you found an easy fix for now. I'd still love to
have a better answer on hand for you in the future.
John, could you share this rig you mention with me? I'll fork it to
Basho Labs [1] if you have it on GitHub already or help clean up an
Multi-backend bitcask with auto-expire for sure sounds like the most
future-proof solution.
For our part, we tend do delete keys in map-reduce jobs, since we have more
complex logic for determining when it is time to delete objects. In our
current setup, it takes about ~3 minutes to go through 1.5
Well, I’ve been looking to make my theoretical Erlang knowledge less
theoretical and somewhat more practical, so I wouldn’t say no. And this
approach is pretty much what we thought we’d use originally.
Since then it has come to light that our product folks have given us permission
to just delet
We've got an expiry worker rig I can likely pass over offline. Its not
overly clever.
Basic idea stream a feed of keys into a pool of workers that spin off
delete calls.
We feed this based on continuous search's of an expiry TTL field in all
keys.
It'd likely be better to run this from with the E
But then you need to use bitcask... And you have one ttl per backend I believe.
If that works for you...
-Alexander
@siculars
http://siculars.posthaven.com
Sent from my iRotaryPhone
> On Jun 4, 2015, at 12:24, Bryce Verdier wrote:
>
> I realize I'm kind of late to this party, but what about
DOH! I should have known that answer wouldn't have been that simple.
Best of luck.
Warm regards,
Bryce
On Thu, 4 Jun 2015 13:46:28 -0400
Peter Herndon wrote:
> Thanks, Bryce, but auto-expire relies on bitcask being the back-end,
> and we’re on leveldb.
>
> > On Jun 4, 2015, at 1:24 PM, Bryce
Mmm, I think we’re looking at deleting about 50 million keys per day. That’s a
completely back-of-envelope estimate, I haven’t done the actual math yet.
—Peter
> On Jun 4, 2015, at 3:28 AM, Daniel Abrahamsson
> wrote:
>
> Hi Peter,
>
> What is "large-scale" in your case? How many keys do you
Thanks, Bryce, but auto-expire relies on bitcask being the back-end, and we’re
on leveldb.
> On Jun 4, 2015, at 1:24 PM, Bryce Verdier wrote:
>
> I realize I'm kind of late to this party, but what about
> using the auto-expire feature and letting riak do the deletion of data
> for you?
>
> The
I realize I'm kind of late to this party, but what about
using the auto-expire feature and letting riak do the deletion of data
for you?
The link is for an older version, but I know the
functionality still exists in riak2.
http://docs.basho.com/riak/latest/community/faqs/developing/#how-can-i-auto
Hi Peter,
What is "large-scale" in your case? How many keys do you need to delete,
and how often?
//Daniel
On Wed, Jun 3, 2015 at 9:54 PM, Peter Herndon wrote:
> Interesting thought. It might work for us, it might not, I’ll have to
> check with our CTO to see whether the expense makes sense un
Interesting thought. It might work for us, it might not, I’ll have to check
with our CTO to see whether the expense makes sense under our circumstances.
Thanks!
—Peter
> On Jun 3, 2015, at 2:21 PM, Drew Kerrigan wrote:
>
> Another idea for a large-scale one-time removal of data, as well as an
That’s not really an option, I think. Our buckets are named “--MM”
and the keys are user login names. Re-using the date-based buckets wouldn’t
make much sense in our case.
> On Jun 3, 2015, at 2:48 PM, Igor Birman wrote:
>
> Can you recycle the keys rather than deleting them? We mark them
Can you recycle the keys rather than deleting them? We mark them as available
for deletion and keep a key recycle pool where we pull new keys from.
On Wednesday, June 3, 2015 12:37 PM, Peter Herndon
wrote:
Hi list,
We’re looking for the best way to handle large scale expiration of
Another idea for a large-scale one-time removal of data, as well as an
opportunity for a fresh start, would be to:
1. set up multi-data center replication between 2 clusters
2. implement a recv/2 hook on the sink which refuses data from the buckets
/ keys you would like to ignore / delete
3. trigg
Sadly, this is a production cluster already using leveldb as the backend. With
that constraint in mind, and rebuilding the cluster not really being an option
to enable multi-backends or bitcask, what would our best approach be?
Thanks!
—Peter
> On Jun 3, 2015, at 12:09 PM, Alexander Sicular
We are actively investigating better options for deletion of large amounts of
keys. As Sargun mentioned, deleting the data dir for an entire backend via an
operationalized rolling restart is probably the best approach right now for
killing large amounts of keys.
But if your key space can fit i
You could map your keys to a given bucket, and that bucket to a given
backend using multi_backend. There is some cost to having lots of backends
(memory overhead, FDs, etc...). When you want to do a mass drop, you could
down the node, and delete that given backend, and bring it up. Caveat: AAE,
MDC
17 matches
Mail list logo
