Re: running erlang map phases via REST API

[Dan Reverri]

This commit looks interesting. A lot of changes were made to map reduce in
0.14 so I am not sure it will work in it's current form.

I think we would only need to include the changes in riak_kv_mapred_json.erl
and riak_kv_mapred_query.erl. Within riak_kv_mapred_query.erl we can convert
the function to a {qfun, Fun} phase. Thoughts?

Thanks,
Dan

Daniel Reverri
Developer Advocate
Basho Technologies, Inc.
d...@basho.com


On Wed, Jan 26, 2011 at 8:40 PM, Seth Falcon  wrote:
>
> Hi all,
>
> I too would find this useful.  And a while back I wrote a patch that
> enabled it.  I'm afraid the patch has likely suffered bit rot, but the
> ideas can at least be harvested:
>
> https://issues.basho.com/show_bug.cgi?id=806
>
> http://github.com/seth/riak_kv
>
> + seth
>
> --
> Seth Falcon | @sfalcon | http://userprimary.net/
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak map reduce returning all results

[Joshua Partogi]

Hi all,

I am trying to do map reduce with the ruby client as such:
client = Ripple.client
Riak::MapReduce.new(client).
  add("stocks").
  map("Riak.mapValuesJson", :arg => "GOOG", :keep => true)

where the JSON document inside the bucket is as such:

{ "abr" : "GOOG" }
{ "abr" : "AAPL" }

However instead of returning only one result it is returning all values in
the bucket.

What am I doing wrong here?

Thanks heaps for the help.

-- 
http://twitter.com/jpartogi 
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak map reduce returning all results

[Jeremiah Peschka]

mapValuesJson (
https://github.com/basho/riak_kv/blob/master/priv/mapred_builtins.js#L45)
takes a JSON formatted string and returns it as a JavaScript object. You'll
still need to do something with it to make the magic happen.

mr = Riak::MapReduce.new(client)
mr.add("stocks")
mr.map("function(value, keyData, arg) {
  var data = Riak.mapValuesJson(value)[0];
  if (data.abr == "GOOG") {
// magic happens here
return [data];
  }
}", :keep => true, :language => "JavaScript")



Jeremiah Peschka
Microsoft SQL Server MVP
MCITP: Database Developer, DBA


On Thu, Jan 27, 2011 at 7:37 AM, Joshua Partogi wrote:

> Hi all,
>
> I am trying to do map reduce with the ruby client as such:
> client = Ripple.client
> Riak::MapReduce.new(client).
>   add("stocks").
>   map("Riak.mapValuesJson", :arg => "GOOG", :keep => true)
>
> where the JSON document inside the bucket is as such:
>
> { "abr" : "GOOG" }
> { "abr" : "AAPL" }
>
> However instead of returning only one result it is returning all values in
> the bucket.
>
> What am I doing wrong here?
>
> Thanks heaps for the help.
>
> --
> http://twitter.com/jpartogi 
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Short read error on .store() with protobufs using Python client

[Jon Meredith]

Hi Bob/Nico/Gary

Thanks for submitting the pull request.  I took a look at the patch and
would prefer to keep the packet length check in, but just move it to the
right place.

I'm unable to reproduce the issue locally, but wondered if you'd be able to
try out the 
https://github.com/basho/riak-python-client/tree/jdm-msglen-fixbranch
and double-check it really resolves your issues.

Cheers, Jon.



On Sat, Jan 22, 2011 at 6:21 PM, Bob Feldbauer wrote:

> Thanks, Nico. I forked riak-python-client and created a pull request for
> the patch you described (removing the three lines for the length check in
> recv _pkt).
>
> - Bob Feldbauer
>
>
> On 1/22/2011 3:23 AM, Nico Meyer wrote:
>
>> Hi
>>
>> let me clarify the situation somewhat. The problem is not intermittent
>> in the sense that it occurs randomly. It depends mainly on the size of
>> the answer that riak sends. If the answer is bigger than the network MTU
>> the error will occur most of the time. If its smaller it occurs almost
>> never. The recv operation on the socket will usually return after every
>> received network packet, which means if you call recv fast enough it
>> will never return more bytes than the MTU of the conection.
>>
>> For get operations and put operations where return_body is True (the
>> default) this obviously depends on the size of the object one is
>> requesting/putting. So it seems to be itermittent if most objects are
>> smaller than about 1400 bytes (most LANs have an MTU of 1492), but some
>> are larger or multiple versions are returned for some of them.
>>
>> The fix I proposed in the bug report (removing the three line) is the
>> correct one. The while loop only terminates if the desired number of
>> bytes have been read or a socket error like a timeout or a closed
>> connection produces an exception.
>>
>> BTW, I have also observed short write errors when storing large objects
>> (larger than the default TCP send buffer I would guess, which is 16k at
>> least on Linux). The fix would be to also loop in snd_pkt() until all
>> data has been sent.
>>
>> Cheers,
>> Nico
>>
>>
>> Am Freitag, den 21.01.2011, 14:43 -0800 schrieb Bob Feldbauer:
>>
>>> Interesting. I was actually seeing this all the time, not just
>>> intermittently. Removing the lines as described in bug #695
>>> (https://issues.basho.com/show_bug.cgi?id=695) did resolve my issue.
>>> Thanks for the help, Gary.
>>>
>>> - Bob
>>>
>>> On 1/21/2011 2:28 PM, Gary Flake wrote:
>>>
 I had the same problem the other day and someone answered with the
 fix.  So search the archives on my name to find the precise answer.
 But the fix was to remove the three lines in pbc.py where the check
 and error message occur.   Basically this is a known issue and the
 check should not be there.

 -- GWF

 (sent from my cell phone.)



 On Jan 21, 2011, at 1:48 PM, Bob Feldbauer
 wrote:

  Thanks for the suggestion, Jon. This is occurring every time I try to
> do .store() via protobuf transport in the Python client (so yes, it is
> reproducible); however, there is no output in logs or riak console.
>
> - Bob
>
> On 1/21/2011 1:02 PM, Jon Meredith wrote:
>
>> Hi Bob,
>>
>> That sounds like the connection got closed as the client was readying.
>>
>> Have you checked through the server logs to verify that no processes
>> died or no other nodes went down?  Or if it's reproducible you should be
>> able to start the server with 'riak console' and see what gets printed.
>>
>> Cheers, Jon.
>> Basho Technologies
>>
>>
>> On Fri, Jan 21, 2011 at 11:36 AM, Bob Feldbauer<
>> b...@completefusion.com>   wrote:
>>
>> I'm using the riak-python-client with Riak 0.14. If I use HTTP
>> instead of PBC, my code works fine; however, with PBC I get a
>> "socket returned short read" error. Any help would be appreciated!
>> Code as follows (for debugging, I added .is_alive() which returns
>> as expected, prior to the error being displayed):
>>
>> client = riak.RiakClient(host='servername', port=8087,
>> transport_class=riak.RiakPbcTransport)
>> riakBucket = client.bucket('bucketname')
>> print client.is_alive()
>> ... [code to setup the key/data] ...
>> doc = riakBucket.new(key, data=result)
>> doc.store()
>>
>> And the full error shown:
>>
>> Traceback (most recent call last):
>>  File "riak_import-pb.py", line 23, in
>>doc.store()
>>  File
>> "/usr/local/lib/python2.6/dist-packages/riak/riak_object.py", line
>> 269, in store
>>Result = t.put(self, w, dw, return_body)
>>  File
>> "/usr/local/lib/python2.6/dist-packages/riak/transports/pbc.py",
>> line 190, in put
>>msg_code, resp = self.recv_msg()
>>  File
>

Re: Short read error on .store() with protobufs using Python client

[Bob Feldbauer]

Hi Jon,

Yes, your jdm-msglen-fix tree seems to resolve the issue for me; 
however, I'm only currently using (and likewise only testing) .store() 
operations with the python client. With the un-patched version, the 
issue occurred for me every time I tried to store data -- of course, my 
data is presumably all larger than a single packet.


Thanks,

Bob

On 1/27/2011 11:08 AM, Jon Meredith wrote:

Hi Bob/Nico/Gary

Thanks for submitting the pull request.  I took a look at the patch 
and would prefer to keep the packet length check in, but just move it 
to the right place.


I'm unable to reproduce the issue locally, but wondered if you'd be 
able to try out the 
https://github.com/basho/riak-python-client/tree/jdm-msglen-fix branch 
and double-check it really resolves your issues.


Cheers, Jon.



On Sat, Jan 22, 2011 at 6:21 PM, Bob Feldbauer > wrote:


Thanks, Nico. I forked riak-python-client and created a pull
request for the patch you described (removing the three lines for
the length check in recv _pkt).

- Bob Feldbauer


On 1/22/2011 3:23 AM, Nico Meyer wrote:

Hi

let me clarify the situation somewhat. The problem is not
intermittent
in the sense that it occurs randomly. It depends mainly on the
size of
the answer that riak sends. If the answer is bigger than the
network MTU
the error will occur most of the time. If its smaller it
occurs almost
never. The recv operation on the socket will usually return
after every
received network packet, which means if you call recv fast
enough it
will never return more bytes than the MTU of the conection.

For get operations and put operations where return_body is
True (the
default) this obviously depends on the size of the object one is
requesting/putting. So it seems to be itermittent if most
objects are
smaller than about 1400 bytes (most LANs have an MTU of 1492),
but some
are larger or multiple versions are returned for some of them.

The fix I proposed in the bug report (removing the three line)
is the
correct one. The while loop only terminates if the desired
number of
bytes have been read or a socket error like a timeout or a closed
connection produces an exception.

BTW, I have also observed short write errors when storing
large objects
(larger than the default TCP send buffer I would guess, which
is 16k at
least on Linux). The fix would be to also loop in snd_pkt()
until all
data has been sent.

Cheers,
Nico


Am Freitag, den 21.01.2011, 14:43 -0800 schrieb Bob Feldbauer:

Interesting. I was actually seeing this all the time, not just
intermittently. Removing the lines as described in bug #695
(https://issues.basho.com/show_bug.cgi?id=695) did resolve
my issue.
Thanks for the help, Gary.

- Bob

On 1/21/2011 2:28 PM, Gary Flake wrote:

I had the same problem the other day and someone
answered with the
fix.  So search the archives on my name to find the
precise answer.
But the fix was to remove the three lines in pbc.py
where the check
and error message occur.   Basically this is a known
issue and the
check should not be there.

-- GWF

(sent from my cell phone.)



On Jan 21, 2011, at 1:48 PM, Bob
Feldbauermailto:b...@completefusion.com>>   wrote:

Thanks for the suggestion, Jon. This is occurring
every time I try to do .store() via protobuf
transport in the Python client (so yes, it is
reproducible); however, there is no output in logs
or riak console.

- Bob

On 1/21/2011 1:02 PM, Jon Meredith wrote:

Hi Bob,

That sounds like the connection got closed as
the client was readying.

Have you checked through the server logs to
verify that no processes died or no other
nodes went down?  Or if it's reproducible you
should be able to start the server with 'riak
console' and see what gets printed.

Cheers, Jon.
Basho Technologies


On Fri, Jan 21, 2011 at 11:36 AM, Bob
Feldbauermailto:b...@completefusion.com>>>  

bug after ~10K get/store requests

[Paco NATHAN]

We're seeing a repeatable error -- running some tests with Riak
0.14.0-1 on Ubuntu, with PHP 5.3.3-1ubuntu9.3

We have a load test, using the PHP client, which stores then gets 1MM
objects. This is being run against both Riak and MySQL as part of a
performance evaluation suite.  The MySQL side runs fine, so the test
script seems fine.

On the Riak side, we generally get an exception from
RiakObject::populate() which throws 'Could not contact Riak Server..'
after having done between 10-20K store/get pairs, each with different
keys.  It doesn't always happen.

There's nothing else running on this system, and we see it both on our
VMware instances and on an EC2 m1.xlarge

I get the sense that our Riak client is causing a temporary DoS on our
Riak server...
Has anyone else seen this?

Maybe adding some retry logic into the client would resolve it?

Thanks,

Paco
IMVU

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com