Most efficient way to determine if 1000 specific keys exist?
Hey guys, Still a relative newbie here. I was hoping to be able to setup a MapReduce job that I could feed 1000 keys to, and have it tell me of the 1000, which keys exist in the bucket. I was hoping this could use the key index (such a thing exists right?) without having to read the objects. The methods I've tried for doing this fail when the first non-existing key is found though. Is there a way to do this? Or alternatively, is there a way to check for the presence of one key at a time without riak having to read the object? Cheers, Tim. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
How do I improve Level DB performance?
Hi there,
I've set up a new cluster, and have been doing pre-deployment benchmarks on
it. The benchmark I was running slowly sunk from 1000 TPS to 250 TPS over
the course of the single 8 hour benchmark doing 1 read+1 update using 1k
values. I'm wondering if anyone might have suggestions on how I can
improve this.
Here's a graph of the benchmark results: http://twitpic.com/9jh6hp/full
And the basho_bench config: https://gist.github.com/837239813dabb9c70c7a
This is on a cluster with 5 nodes of identical hardware, with the benchmark
running from a separate server. The nodes all have single proc quad-core
Sandy Beach CPUs, 8 GB of ram, and 600GB 15K RPM drives. They're running
Ubuntu 10.04 with the ext4 filesystem, and noop ioscheduler.
For the benchmark above, Riak itself was setup in LevelDB, with a ringsize
of 1024, and the default config settings for LevelDB.
There was about 10GB of benchmark data already in each node. While the
benchmarks are running, iowait hits about 20%, util about 80%, and each
server consumes about 4GB of ram.
Can anyone confirm this is expected performance from this kind of setup, or
offer suggestions on how to improve performance?
---
As an aside, I've since repaved the 5 nodes and adjusted it to a ring size
of 128 (I'm concerned about the scaling cap with this), and leveldb
settings like this:
{write_buffer_size, 16777216},
{max_open_files, 100},
{block_size, 262144},
{cache_size, 168430088}
I've set it running on another 8 hour benchmark to see what it turns up.
Cheers,
Tim Haines.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How do I improve Level DB performance?
Hi David,
I think I saw your name as one of the creators of the benchmarking tool -
thanks - compared to the efforts required to benchmark other systems, it's
been great.
On Thu, May 10, 2012 at 3:01 PM, David Smith wrote:
> On Thu, May 10, 2012 at 2:33 PM, Tim Haines wrote:
>
> > I've set up a new cluster, and have been doing pre-deployment benchmarks
> on
> > it. The benchmark I was running slowly sunk from 1000 TPS to 250 TPS over
> > the course of the single 8 hour benchmark doing 1 read+1 update using 1k
> > values. I'm wondering if anyone might have suggestions on how I can
> improve
> > this.
>
> Generally, this suggests that you are becoming seek-time bound. The
> test config, as specified, will generate a pretty huge number of
> not_founds which are (currently) crazy expensive w/ LevelDB,
> particularly as the dataset grows.
>
> Assuming you start with an empty database, a sample of this test will
> generate operations like so:
>
> Key 1000 - get -> not_found
> Key 1001 - update -> not_found + write
> Key 1002 - get -> not_found
> etc..
>
> I.e. the leveldb cache never gets a chance to be useful, because
> you're always writing new values and the cost of writing each new
> value goes up, since you have to thrash the cache to determine if
> you're ever seen the key that doesn't exist. :)
>
> The root problem here is going to be the key_generator --
> partitioned_sequential_int will just run through all the ints in order
> and never revisit a key.
>
>
Okay, I understand what you're saying here. There's no cache hits. I
thought about the tests I ran after the 8 hour benchmark, and although I
started the same test for a shorter duration again, it's likely that the
objects still weren't in the cache even though the data may have existed
because I'd already iterated over so many objects.
I'm assuming the benchmarking tool will use the same bucket for each run,
and the keys will grow in sequence. So if I run through the benchmark
writing out 100,000 keys, and then rerun the benchmark with 3 reads per one
update, I should see results where the read is successfully being filled by
the cache right?
I'll try this out. Even though this is occurring would you have expected
to see the performance drop from 500 to 250 over the course of the
benchmark?
>
> > {write_buffer_size, 16777216},
> > {max_open_files, 100},
> > {block_size, 262144},
> > {cache_size, 168430088}
>
> I strongly recommend not changing write_buffer_size; it can have
> unexpected latency side-effects when LevelDB compaction occurs.
> Smaller == more predictable.
>
>
Thanks for this - I will revert it back. One thing I haven't been able to
locate is what settings can be changed when the node has data in it. Is
this documented anywhere?
Does that help?
>
>
You've improved my understanding and given me something else to try. Thank
you!
Tim.
> D.
>
> --
> Dave Smith
> VP, Engineering
> Basho Technologies, Inc.
> diz...@basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How do I improve Level DB performance?
On Thu, May 10, 2012 at 3:01 PM, David Smith wrote:
> On Thu, May 10, 2012 at 2:33 PM, Tim Haines wrote:
>
> > I've set up a new cluster, and have been doing pre-deployment benchmarks
> on
> > it. The benchmark I was running slowly sunk from 1000 TPS to 250 TPS over
> > the course of the single 8 hour benchmark doing 1 read+1 update using 1k
> > values. I'm wondering if anyone might have suggestions on how I can
> improve
> > this.
>
> Generally, this suggests that you are becoming seek-time bound. The
> test config, as specified, will generate a pretty huge number of
> not_founds which are (currently) crazy expensive w/ LevelDB,
> particularly as the dataset grows.
>
> Assuming you start with an empty database, a sample of this test will
> generate operations like so:
>
> Key 1000 - get -> not_found
> Key 1001 - update -> not_found + write
> Key 1002 - get -> not_found
> etc..
>
> I.e. the leveldb cache never gets a chance to be useful, because
> you're always writing new values and the cost of writing each new
> value goes up, since you have to thrash the cache to determine if
> you're ever seen the key that doesn't exist. :)
>
> The root problem here is going to be the key_generator --
> partitioned_sequential_int will just run through all the ints in order
> and never revisit a key.
>
>
> > {write_buffer_size, 16777216},
> > {max_open_files, 100},
> > {block_size, 262144},
> > {cache_size, 168430088}
>
> I strongly recommend not changing write_buffer_size; it can have
> unexpected latency side-effects when LevelDB compaction occurs.
> Smaller == more predictable.
>
> Does that help?
>
> D.
>
> --
> Dave Smith
> VP, Engineering
> Basho Technologies, Inc.
> diz...@basho.com
>
Hey David,
With the adjusted ring size and settings, and adjusted to only do puts (so
no missed reads), my cluster is doing about 400 puts per second:
http://twitpic.com/9jnhlm/full
I have yet to insert much data, so as I understand it, as more levels are
created this is going to slow down even further. That still seems pretty
slow for a 5 node cluster.
I also ran a little benchmark on only reads that I knew would hit the
cache, and was achieving about 5000 reads per second.
Cheers,
Tim.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How do I improve Level DB performance?
David,
I ran the benchmark again for 9 hours overnight, just doing puts.
Performance fell steadily from 400 puts/s to 250 puts/s.
Graph: http://twitpic.com/9jtjmu/full
Cheers,
Tim.
On Thu, May 10, 2012 at 3:01 PM, David Smith wrote:
> On Thu, May 10, 2012 at 2:33 PM, Tim Haines wrote:
>
> > I've set up a new cluster, and have been doing pre-deployment benchmarks
> on
> > it. The benchmark I was running slowly sunk from 1000 TPS to 250 TPS over
> > the course of the single 8 hour benchmark doing 1 read+1 update using 1k
> > values. I'm wondering if anyone might have suggestions on how I can
> improve
> > this.
>
> Generally, this suggests that you are becoming seek-time bound. The
> test config, as specified, will generate a pretty huge number of
> not_founds which are (currently) crazy expensive w/ LevelDB,
> particularly as the dataset grows.
>
> Assuming you start with an empty database, a sample of this test will
> generate operations like so:
>
> Key 1000 - get -> not_found
> Key 1001 - update -> not_found + write
> Key 1002 - get -> not_found
> etc..
>
> I.e. the leveldb cache never gets a chance to be useful, because
> you're always writing new values and the cost of writing each new
> value goes up, since you have to thrash the cache to determine if
> you're ever seen the key that doesn't exist. :)
>
> The root problem here is going to be the key_generator --
> partitioned_sequential_int will just run through all the ints in order
> and never revisit a key.
>
>
> > {write_buffer_size, 16777216},
> > {max_open_files, 100},
> > {block_size, 262144},
> > {cache_size, 168430088}
>
> I strongly recommend not changing write_buffer_size; it can have
> unexpected latency side-effects when LevelDB compaction occurs.
> Smaller == more predictable.
>
> Does that help?
>
> D.
>
> --
> Dave Smith
> VP, Engineering
> Basho Technologies, Inc.
> diz...@basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How do I improve Level DB performance?
On Fri, May 11, 2012 at 9:13 AM, Ryan Zezeski wrote: > > > On Thu, May 10, 2012 at 11:14 PM, Tim Haines wrote: >> >> >> >> With the adjusted ring size and settings, and adjusted to only do puts >> (so no missed reads), my cluster is doing about 400 puts per second: >> http://twitpic.com/9jnhlm/full >> > > Actually, every put (put from a riak API level) does a read on the backend > [1]. This is needed to merge contents from the two objects [2]. > > Like Dave already mentioned the key generation strategy along with > leveldb's degrading performance on not-found means your benchmark will just > get worse the longer it runs. > > Are you testing an actual use case here? Do you envision 100M objects > being written in a constant stream? Will your objects have a median size > of 1000 bytes? Basho bench also provides a pareto key generator which uses > a fraction of the key space most of the time. I'm not sure it matches your > use case but thought I'd mention it is there. > > Hi Ryan, Thanks. Greg just mentioned the reads on puts too. I'd changed the config to 250 bytes (matching about what I store for a tweet), and reran it overnight, and observed performance drop from 400 puts/s to 250 puts/s. Right now my use case has me constantly writing about 200 new tweets per second, so unless I'm missing something, this throughput measurement is a realistic indicator for me. Tim. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How do I improve Level DB performance?
On Fri, May 11, 2012 at 9:20 AM, Tim Haines wrote: > > > On Fri, May 11, 2012 at 9:13 AM, Ryan Zezeski wrote: > >> >> >> On Thu, May 10, 2012 at 11:14 PM, Tim Haines wrote: >>> >>> >>> >>> With the adjusted ring size and settings, and adjusted to only do puts >>> (so no missed reads), my cluster is doing about 400 puts per second: >>> http://twitpic.com/9jnhlm/full >>> >> >> Actually, every put (put from a riak API level) does a read on the >> backend [1]. This is needed to merge contents from the two objects [2]. >> >> Like Dave already mentioned the key generation strategy along with >> leveldb's degrading performance on not-found means your benchmark will just >> get worse the longer it runs. >> >> Are you testing an actual use case here? Do you envision 100M objects >> being written in a constant stream? Will your objects have a median size >> of 1000 bytes? Basho bench also provides a pareto key generator which uses >> a fraction of the key space most of the time. I'm not sure it matches your >> use case but thought I'd mention it is there. >> >> > Hi Ryan, > > Thanks. Greg just mentioned the reads on puts too. I'd changed the config > to 250 bytes (matching about what I store for a tweet), and reran it > overnight, and observed performance drop from 400 puts/s to 250 puts/s. > Right now my use case has me constantly writing about 200 new tweets per > second, so unless I'm missing something, this throughput measurement is a > realistic indicator for me. > > Tim. > I guess I was hoping that someone could look at these results and say "Given the use case and the hardware, Riak should be performing 10x what you're seeing, so something is configured wrong." I'm not hearing that though. What I'm hearing is "Is that a realistic use case?". So given this use case, and the hardware I have, these are expected results? Tim. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How do I improve Level DB performance?
On Fri, May 11, 2012 at 10:05 AM, Ryan Zezeski wrote: > > > On Fri, May 11, 2012 at 12:42 PM, Tim Haines wrote: > >> >> I guess I was hoping that someone could look at these results and say >> "Given the use case and the hardware, Riak should be performing 10x what >> you're seeing, so something is configured wrong." I'm not hearing that >> though. What I'm hearing is "Is that a realistic use case?". >> >> So given this use case, and the hardware I have, these are expected >> results? >> >> Tim. >> > > Yea, I don't want to come of as dodging the question. I've seen lots of > people run benchmarks for use cases they don't even have. That doesn't > seem to be the case here. > > Ryan, thanks. I appreciate your response. I'm waiting to hear back from John this morning to see if I can engage one of your consultants to answer the question for me. Tim. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
