On Dec 28, 2012, at 11:28 AM, Brian Roach <ro...@basho.com> wrote:
> Dietrich -
>
> I haven't seen this in testing or have had anyone report this; could I
> get some more info?
>
> What operations are timing out like this? Is that the complete message
> in the riak error.log? Which version of Riak are you running?
All socket operations. It looks as though those that open a new socket are
especially impacted. We are running 1.2.1 with the leveldb backend. Same 9 node
SSD cluster info I have posted to the list before but don't have access to all
of the details at the moment.
I suspect that there are additional timeouts to be configured and the previous
default values have been lowered. I tried bumping the requestTimeout to no
avail. This wouldn't explain the strange latency spikes (via /stats) seen as we
began rolling out the new driver.
>
> Do you see this simply dropping in the 1.0.6 client to your existing
> application, or is this after changing your code to use the new
> 'withoutFetch()' ?
1.0.6 with the inclusion of withoutFetch(). Haven't tried with just the driver
alone.
>
> Thanks,
> Brian Roach
>
> On Wed, Dec 26, 2012 at 7:28 PM, Dietrich Featherston <d...@d2fn.com> wrote:
>> I had rolled out an upgrade to a JVM app that uses rjc 1.0.5. We had
>> upgraded to 1.0.6 to take advantage of newly added abilities to do a
>> put without preceding it with a fetch in order to reduce operational
>> load on the cluster. However, after rolling out this change we
>> frequently see large rises in latency across the cluster (up to the
>> gen_fsm limit of 60s) and see the following in the riak logs
>>
>> [error] Unrecognized message {74392380,{error,timeout}}
>>
>> This is accompanied by repeated socket timeouts as seen by the
>> riak-java-client.
>>
>> Also worth mentioning, one of our nodes got into a state that the rjc
>> was unable to establish a tcp connection on the protobuf port to riak
>> on localhost. We were only able to fix this by restarting the riak
>> process on that node and inducing a fair amount of handoff.
>>
>> Any thoughts?
>>
>> Thanks,
>> D
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
