I get this error when I increase the number of threads accessing a single
connection. Something related to getting unexpected number of bytes on the
connection.
On Wednesday, June 27, 2012, Reid Draper <reiddra...@gmail.com> wrote:
>
> On Jun 27, 2012, at 8:41 AM, Yousuf Fauzan wrote:
>
> So I created an array of clients using the following code
> Clients = [riak.RiakClient(e, port=8087,
transport_class=riak.RiakPbcTransport) for e in NODES]
>
> Sounds like you're bringing your concurrency back down to 3 (because you
have three nodes). Give
> something like 10 connections _per_ node a try, so 30 connections.
>
> After this I assigned each thread a particular id ranging from 0 to
Number of Nodes
> So each thread now communicates with a single node.
> Even after this, I am getting <100 writes/sec
>
> On Wed, Jun 27, 2012 at 5:35 PM, Yousuf Fauzan <yousuffau...@gmail.com>
wrote:
>
> Oh! I think that may be an issue with my code then.
> Let me make some changes and get back to you.
>
> On Wed, Jun 27, 2012 at 5:25 PM, Reid Draper <reiddra...@gmail.com> wrote:
>
> On Jun 27, 2012, at 7:48 AM, Yousuf Fauzan wrote:
>
> This is great.
> I was loading data using Python. My code would spawn 10 threads and put
data in a queue. All threads would read data from this queue.
> However, all threads were hitting the same server/load balancer.
> I tried a different setup too. Where I spawned processes with each
process having its own queue. In this case too, all processes were hitting
the same server.
> I just now made a change to my code. So now I have 10 threads randomly
selecting a node and storing data in it.
> Again, I am getting around 50 writes/sec
>
> When the threads randomly pick a node, do they create a new connection to
it, or do they pull the connection from
> a pool? As you saw with the throughput difference between curl and
python, persistent connections make
> big difference.
>
> Could there be something wrong with the way I have written my loader
script?
>
> On Wed, Jun 27, 2012 at 5:10 PM, Russell Brown <russell.br...@mac.com>
wrote:
>
> On 27 Jun 2012, at 12:36, Yousuf Fauzan wrote:
>
> So I changed concurrency to 10 and put all the IPs of the nodes in basho
bench config.
> Throughput is now around 1500.
>
> I guess you can now try 5 or 15 concurrent workers and see which is
optimal for that set up to get a good feel for the sizing of any connection
pools for your application.
> You can also see how adding nodes and adding workers effects your results
to help you size the cluster you need for your expected usage.
> Cheers
> Russell
>
> On Wed, Jun 27, 2012 at 4:40 PM, Russell Brown <russell.br...@mac.com>
wrote:
>
> On 27 Jun 2012, at 12:09, Yousuf Fauzan wrote:
>
> I used examples/riakc_pb.config
> {mode, max}.
> {duration, 10}.
> {concurrent, 1}.
>
> Try upping this. On my local 3 node cluster with 8gb ram and an old,
cheap quad core per box I'd set concurrency to 10 workers.
>
> {driver, basho_bench_driver_riakc_pb}.
> {key_generator, {int_to_bin, {uniform_int, 10000}}}.
> {value_generator, {fixed_bin, 10000}}.
> {riakc_pb_ips, [{<IP of one of the nodes>}]}.
>
> I add all the IPs here, one entry per node.
>
> {riakc_pb_replies, 1}.
> {operations, [{get, 1}, {update, 1}]}.
>
> On Wed, Jun 27, 2012 at 4:37 PM, Russell Brown <russell.br...@mac.com>
wrote:
>
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
