Hi Sean, This one really is a bug with 2i pagination on Equals term queries[1].
I've fixed it already[2], I'm very sorry that this one got out into the wild. It is something that worked then stopped working at some point in a merge. It was a pretty bad oversight that we didn't have a test for this condition to catch the regression. I've added a test[3] to the suite for the issue and verified the patch. I guess there will be a 1.4.1 soon. Apologies, again. Russell [1] https://github.com/basho/riak_kv/issues/611 [2] https://github.com/basho/riak_kv/pull/612 [3] https://github.com/basho/riak_test/pull/340 On 27 Jul 2013, at 01:11, Sean McKibben <grap...@graphex.com> wrote: > So when I try to use pagination, it doesn't seem to be picking up my > continuation. I'm having trouble parsing the json I get back using > stream=true (and there is still a timeout) so I went to just using > pagination. Perhaps I'm doing it wrong, (likely, it has been a long day) but > riak seems to be ignoring my continuation: > > (pardon the sanitization) > curl > 'http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?max_results=5' > {"keys":["1","2","3","4","5"],"continuation":"g20AAABAMDAwMDE1ZWVjMmNiZjY3Y2Y4YmU3ZTVkMWNiZTVjM2ZkYjg2YWU0MGIwNzNjMTE3NDYyZjEzMTNlMDQ5YmI2ZQ=="} > > curl > 'http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?max_results=5&continuation=g20AAABAMDAwMDE1ZWVjMmNiZjY3Y2Y4YmU3ZTVkMWNiZTVjM2ZkYjg2YWU0MGIwNzNjMTE3NDYyZjEzMTNlMDQ5YmI2ZQ==' > {"keys":["1","2","3","4","5"],"continuation":"g20AAABAMDAwMDE1ZWVjMmNiZjY3Y2Y4YmU3ZTVkMWNiZTVjM2ZkYjg2YWU0MGIwNzNjMTE3NDYyZjEzMTNlMDQ5YmI2ZQ=="} > > The same keys and continuation value are returned regardless of whether my > request contains a continuation value. I've tried swapping the order of > max_results and continuation without any luck. I also made sure that my > continuation value was url encoded. Hopefully I'm not missing something > obvious here. Well, come to think of it, hopefully I am missing something > obvious! > > Sean > > On Jul 26, 2013, at 6:43 PM, Russell Brown <russell.br...@mac.com> wrote: > >> For a work around you could use streaming and pagination. >> >> Request smaller pages of data (i.e. sub 60 seconds worth) and use streaming >> to get the results to your client sooner. >> >> In HTTP this would look like >> >> http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?max_results=10000&stream=true >> >> your results will include a continuation like >> >> {"continuation":"g2gCYgAAFXttAAAABDU0OTk="} >> >> and you can use that to get the next N results. Breaking your query up that >> way should duck the timeout. >> >> Furthermore, adding &stream=true will mean the first results is received >> very rapidly. >> >> I don't think the Ruby client is up to date for the new 2i features, but you >> could monkeypatch as before. >> >> Cheers >> >> Russell >> >> On 26 Jul 2013, at 19:00, Sean McKibben <grap...@graphex.com> wrote: >> >>> Thank you for looking in to this. This is a major problem for our >>> production cluster, and we're in a bit of a bind right now trying to figure >>> out a workaround in the interim. It sounds like maybe a mapreduce might >>> handle the timeout properly, so hopefully we can do that in the meantime. >>> If there is any way we can have a hotfix ASAP though, that would be >>> preferable. Certainly would not be a problem for us to edit a value in the >>> config file (and given the lack of support in the ruby client for the >>> timeout setting, the ability to edit the global default would be preferred). >>> In the ruby client i had to monkeypatch it like this to even submit the >>> timeout value, which is not ideal: >>> >>> module Riak >>> class Client >>> class HTTPBackend >>> def get_index(bucket, index, query) >>> bucket = bucket.name if Bucket === bucket >>> path = case query >>> when Range >>> raise ArgumentError, t('invalid_index_query', :value => >>> query.inspect) unless String === query.begin || Integer === query.end >>> index_range_path(bucket, index, query.begin, query.end) >>> when String, Integer >>> index_eq_path(bucket, index, query, 'timeout' => '260000') >>> else >>> raise ArgumentError, t('invalid_index_query', :value => >>> query.inspect) >>> end >>> response = get(200, path) >>> JSON.parse(response[:body])['keys'] >>> end >>> end >>> end >>> end >>> >>> Thanks for the update, >>> Sean >>> >>> >>> >>> On Jul 26, 2013, at 4:49 PM, Russell Brown <russell.br...@mac.com> wrote: >>> >>>> Hi Sean, >>>> I'm very sorry to say that you've found a featurebug. >>>> >>>> There was a fix put in here https://github.com/basho/riak_core/pull/332 >>>> >>>> But that means that the default timeout of 60 seconds is now honoured. In >>>> the past it was not. >>>> >>>> As far as I can see the 2i endpoint never accepted a timeout argument, and >>>> it still does not. >>>> >>>> The fix would be to add the timeout to the 2i API endpoints, and I'll do >>>> that straight away. >>>> >>>> In the meantime, I wonder if streaming the results would help, or if you'd >>>> still hit the overall timeout? >>>> >>>> Very sorry that you've run into this. Let me know if streaming helps, I've >>>> raised an issue here[1] if you want to track this bug >>>> >>>> Cheers >>>> >>>> Russell >>>> >>>> [1] https://github.com/basho/riak_kv/issues/610 >>>> >>>> >>>> On 26 Jul 2013, at 17:59, Sean McKibben <grap...@graphex.com> wrote: >>>> >>>>> I should have mentioned that I also tried: >>>>> curl -H "X-Riak-Timeout: 260000" >>>>> "http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?timeout=260000"; >>>>> -i >>>>> but still receive the 500 error below exactly at the 60 second mark. Is >>>>> this a bug? >>>>> >>>>> Secondary to getting this working at all, is this documented anywhere? >>>>> and any way to set this timeout using the ruby riak client? >>>>> >>>>> Stream may well work, but I'm going to have to make a number of changes >>>>> on the client side to deal with the results. >>>>> >>>>> Sean >>>>> >>>>> On Jul 26, 2013, at 3:53 PM, Brian Roach <ro...@basho.com> wrote: >>>>> >>>>>> Sean - >>>>>> >>>>>> The timeout isn't via a header, it's a query param -> &timeout=xxxx >>>>>> >>>>>> You can also use stream=true to stream the results. >>>>>> >>>>>> - Roach >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>> On Jul 26, 2013, at 3:43 PM, Sean McKibben <grap...@graphex.com> wrote: >>>>>> >>>>>>> We just upgraded to 1.4 and are having a big problem with some of our >>>>>>> larger 2i queries. We have a few key queries that takes longer than 60 >>>>>>> seconds (usually about 110 seconds) to execute, but after going to 1.4 >>>>>>> we can't seem to get around a 60 second timeout. >>>>>>> >>>>>>> I've tried: >>>>>>> curl -H "X-Riak-Timeout: 260000" >>>>>>> "http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?x-riak-timeout=260000"; >>>>>>> -i >>>>>>> >>>>>>> But I always get >>>>>>> HTTP/1.1 500 Internal Server Error >>>>>>> Vary: Accept-Encoding >>>>>>> Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact) >>>>>>> Date: Fri, 26 Jul 2013 21:41:28 GMT >>>>>>> Content-Type: text/html >>>>>>> Content-Length: 265 >>>>>>> Connection: close >>>>>>> >>>>>>> <html><head><title>500 Internal Server >>>>>>> Error</title></head><body><h1>Internal Server Error</h1>The server >>>>>>> encountered an error while processing this >>>>>>> request:<br><pre>{error,{error,timeout}}</pre><P><HR><ADDRESS>mochiweb+webmachine >>>>>>> web server</ADDRESS></body></html> >>>>>>> >>>>>>> Right at the 60 second mark. What can I set to give my secondary index >>>>>>> queries more time?? >>>>>>> >>>>>>> This is causing major problems for us :( >>>>>>> >>>>>>> Sean >>>>>>> _______________________________________________ >>>>>>> riak-users mailing list >>>>>>> riak-users@lists.basho.com >>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>> >>>>> _______________________________________________ >>>>> riak-users mailing list >>>>> riak-users@lists.basho.com >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>> >> >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
