Re: Nightly Prune

2013-02-07 Thread Stephan Kepser
Hi Chad,

I recommend looking at secondary indexes. You can set up a secondary index
with the relevant date from your entry. Note that there is no data type
date, only string or integer. But you can easily convert a date into a
string or an integer for your query purposes. Secondary indexes even
provide you with a way to query for date ranges. And they are fast. So, I
think they'd serve your purpose.

Best,

Stephan

-- 
Dr. Stephan Kepser | Senior IT-Consultant

codecentric AG | Merscheider Straße 1 | 42699 Solingen | Deutschland
tel: +49 (0) 212.23362845 | fax: +49 (0) 212.23362879 | mobil: +49 (0)
151.52883635
www.codecentric.de | blog.codecentric.de | www.meettheexperts.de |
www.more4fi.de
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


Re: Nightly Prune

2013-02-08 Thread Stephan Kepser
Hi Chad,

yes, you're right: the use of secondary indexes requires LevelDB  as
backend. How much of an performance penalty this imposes to you I really
don't know. I'd still consider the use of secondary indexes because it is
an architecturally clean solution and hence simple to implement and
maintain. If you experience performance issues you still have the option to
scale out, i.e, use more nodes.

There may yet be another option to speed up write operations, depending on
your demands. You could use the memory backend for about a third of your
nodes and set the write quorum to 1. This turns the memory-only nodes into
something like a cache without any further need to administer this cache.
On the other hand I have to say I'm not sure you can enforce that at least
one replica of each data item goes to a node with LevelDB backend.

Best,

Stephan

2013/2/7 Chad Engler 

> I'm writing the prune script in Node, and the dates are stored as int
> timestamps so that isn't an issue.
>
> ** **
>
> I was under the impression that secondary indexes only worked on the
> LevelDB backend, we have a solid write throughput (maybe 10ish writes per
> second) and very little read (aside from the prune). Would we see
> significant performance degradation by switching from Bitcask?
>
> ** **
>
> -Chad
>
> ** **
>
> *From:* riak-users [mailto:riak-users-boun...@lists.basho.com] *On Behalf
> Of *Stephan Kepser
> *Sent:* Thursday, February 07, 2013 1:50 PM
> *To:* riak-users@lists.basho.com
> *Subject:* Re: Nightly Prune
>
> ** **
>
> Hi Chad,
>
> I recommend looking at secondary indexes. You can set up a secondary index
> with the relevant date from your entry. Note that there is no data type
> date, only string or integer. But you can easily convert a date into a
> string or an integer for your query purposes. Secondary indexes even
> provide you with a way to query for date ranges. And they are fast. So, I
> think they'd serve your purpose.
>
> Best,
>
> Stephan
>
> --
> Dr. Stephan Kepser | Senior IT-Consultant
>
> codecentric AG | Merscheider Straße 1 | 42699 Solingen | Deutschland
> tel: +49 (0) 212.23362845 | fax: +49 (0) 212.23362879 | mobil: +49 (0)
> 151.52883635
> www.codecentric.de | blog.codecentric.de | www.meettheexperts.de |
> www.more4fi.de
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com