> On 12 Oct 2015, at 13:33, Jan Lehnardt <j...@apache.org> wrote: > > Oops, hits end too early. > > > Okay -n 1 explains this
And again, butterfingers! …-n 1 explains this. I’m afraid this hides potential issues because we are testing a condition that is, while definitely common (presumably), not necessarily what still a lot of people are running. But maybe it’s okay for the tests like that, and we have to think about cluster-aware integration tests in a separate step. Would love to hear other opinions on this, maybe I’m just afraid for nothing, or it’s justified, I just don’t know. The (longer) experience PouchDB had here is definitely valuable, thanks Dale! Best Jan -- >> On 12 Oct 2015, at 13:32, Jan Lehnardt <j...@apache.org> wrote: >> >> >>> On 12 Oct 2015, at 13:30, Dale Harvey <d...@arandomurl.com> wrote: >>> >>> Started with >>> >>> python dev/run -n 1 --with-admin-party-please & >>> >>> and export COUCH_HOST='http://127.0.0.1:15984' >>> >>> >>> On 12 October 2015 at 13:26, Jan Lehnardt <j...@apache.org> wrote: >>> >>>> >>>>> On 12 Oct 2015, at 13:15, Dale Harvey <d...@arandomurl.com> wrote: >>>>> >>>>>> - instead of fixed test db names like `test_suite_db` that are used in >>>>> almost every test, >>>>>> we now generate a random database name for each test. The JS tests >>>>> expected db >>>>>> deletion to be synchronous, and that’s no longer the case in 2.0. >>>> Instead >>>>> of >>>>>> error-prone polling to wait for db deletion, we just use new db for >>>> every >>>>> test. >>>>>> An added benefit now is that (some) tests can run in parallel. >>>>> >>>>> Little confused by this, PouchDB uses consistent names in its test suite >>>>> (we had random >>>>> ones for a while and moved away), we very much depend on the database >>>> being >>>>> deleted >>>>> on the request returning and dont seem to have any problems with it >>>> against >>>>> master. >>>> >>>> Does the PouchDB test suite runs against the 5986 port, or with an n=1 >>>> cluster? >>>> This is with the default n=3 cluster against 5986. >>>> >>>> Best >>>> Jan >>>> -- >>>> >>>>> >>>>> On 12 October 2015 at 12:21, Jan Lehnardt <j...@apache.org> wrote: >>>>> >>>>>> >>>>>>> On 11 Oct 2015, at 18:53, Sebastian Rothbucher < >>>>>> sebastianrothbuc...@googlemail.com> wrote: >>>>>>> >>>>>>> Hi Jan, >>>>>>> >>>>>>> I transferred some of my findings to your branch - pls. find a PR for >>>> the >>>>>>> first few tests submitted 2 your branch. I put in TODOs where either >>>>>> there >>>>>>> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853) or we have >>>>>> to >>>>>>> do some more work (like making config available). >>>>>> >>>>>> Excellent, thanks! :) >>>>>> >>>>>> >>>>>>> Will from the Pouch team had the great idea of using -n 1 to mitigate >>>>>>> Heisenbugs due to distribution. I adopted it and it did work quite >>>> well. >>>>>> It >>>>>>> might make sense to put this into the dev/run call as a general rule, >>>>>> esp. >>>>>>> when checking for 201 (VS 202) status codes. Probably you know more on >>>>>>> that. >>>>>> >>>>>> We should carefully evaluate this :) >>>>>> >>>>>> >>>>>>> Anyways: I think we have to keep using _config on the local port as >>>> many >>>>>>> tests (like auth tests) depend on changed configurations. BTW: w/ -n 1 >>>>>> it's >>>>>>> only one call, but calling 3 _config(s) migt also work. >>>>>>> Hopefully (I just arrived at some test w/ that recently) we can spare >>>>>>> ourselves the _compact thing. I think we'd have to compact every single >>>>>>> shard via local port, right? for the first tests, we can go w/out as it >>>>>>> seems. Maybe we should stick to that for a while. >>>>>> >>>>>> My concern with this is that as far as I care, the local port is >>>> irrelevant >>>>>> to how CouchDB works. Of course it isn’t 100% irrelevant, but the >>>> cluster >>>>>> port is our publicised API for 2.0 and it’s the one that replaces the >>>> 1.x >>>>>> API for people upgrading. So having any tests that run on the local port >>>>>> of 2.0 don’t help us in any way (that’s not to say it still useful to >>>> have >>>>>> tests against them for the moment). >>>>>> >>>>>> Many of the _config-dependent tests substitute out the _users or >>>>>> _replicator >>>>>> databases, maybe we can find a workaround for those by just using the >>>> stock >>>>>> databases. >>>>>> >>>>>> For the other tests, maybe we split them out into their own “test >>>> suites” >>>>>> that start their own test cluster in the configuration they need and >>>> shut >>>>>> it down afterwards. >>>>>> >>>>>> Best >>>>>> Jan >>>>>> -- >>>>>> >>>>>> >>>>>>> >>>>>>> Looking fw 2 your thoughts >>>>>>> >>>>>>> Best >>>>>>> Sebastian >>>>>>> >>>>>>> P.S:: Maybe we can adopt some stuff from >>>>>>> >>>>>> >>>> https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest >>>>>>> >>>>>>> P.P.S.: Will keep working on it, just not in the next few days, >>>>>>> unfortunately ;-( >>>>>>> >>>>>>> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <j...@apache.org> wrote: >>>>>>> >>>>>>>> (I need your help deciding what to do with a bunch of tests, jump to >>>>>>>> “Discussion” below if you don’t care about the details before.) >>>>>>>> >>>>>>>> * * * >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I spent some more time getting the 1.x JavaScript tests running on the >>>>>>>> cluster port of 2.0. WIP here: >>>>>> https://github.com/apache/couchdb/pull/354 >>>>>>>> — I’ve gotten a lot further, but there is still a lot to do and >>>> everyone >>>>>>>> here can help, I’ve posted instructions on how to do so further down. >>>> No >>>>>>>> Erlang knowledge needed, this is ideal for folks who want to start >>>> with >>>>>>>> contributing to CouchDB. >>>>>>>> >>>>>>>> >>>>>>>> Notable changes: >>>>>>>> >>>>>>>> - instead of fixed test db names like `test_suite_db` that are used in >>>>>>>> almost every test, we now generate a random database name for each >>>> test. >>>>>>>> The JS tests expected db deletion to be synchronous, and that’s no >>>>>> longer >>>>>>>> the case in 2.0. Instead of error-prone polling to wait for db >>>>>> deletion, we >>>>>>>> just use new db for every test. An added benefit now is that (some) >>>>>> tests >>>>>>>> can run in parallel. >>>>>>>> >>>>>>>> - tests now clean up after themselves (incomplete): tests previously >>>>>> only >>>>>>>> collectively used three or four databases, and deleted and re-created >>>>>> them >>>>>>>> before each test was run. Now with the change of creating a new >>>> database >>>>>>>> per test, we also need to clean up databases at the end of a test, >>>>>> because >>>>>>>> otherwise we a) leave a lot of databases around and b) CouchDB runs >>>> out >>>>>> of >>>>>>>> file descriptors during the test run. The cleanup isn’t 100% done yet, >>>>>> so >>>>>>>> we should check for stray databases after JS test runs until we’ve got >>>>>> it >>>>>>>> cleaned. One particular aspect here is when a test fails, it never >>>>>> reaches >>>>>>>> the cleanup code, so we might have to wrap everything into a try/catch >>>>>>>> block. >>>>>>>> >>>>>>>> - I disabled all tests that still fail and made them so they print the >>>>>>>> reason why they are failing, so that fixing them should be a lot >>>> easier >>>>>>>> now. Some of them print just “TODO”, their error cause is yet to be >>>>>>>> determined. >>>>>>>> >>>>>>>> - couchdb.query() which previously executed temporary views, now under >>>>>> the >>>>>>>> hood creates a design doc with a random name and queries that view. >>>> This >>>>>>>> fixes all tests relying on temp views without having to change them >>>> all >>>>>>>> individually. This is because 2.0 doesn’t have temp views anymore >>>>>> (which is >>>>>>>> a good thing). >>>>>>>> >>>>>>>> - restartServer() is no longer used (mostly). >>>>>>>> >>>>>>>> I already filed a few bugs that showed things missing in the cluster >>>> API >>>>>>>> that we’d expect to be there: >>>>>>>> >>>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2849 >>>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2850 >>>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2851 >>>>>>>> >>>>>>>> Based on this I am ever more convinced we should get this test suite >>>>>> work >>>>>>>> against the 2.0 clustered API, because we’ll be finding some more >>>>>>>> inconsistencies that we should fix before we invite beta testers. >>>>>>>> >>>>>>>> >>>>>>>> ## Discussion: >>>>>>>> >>>>>>>> We have a whole number of tests that use the test-internal function >>>>>>>> run_on_modified_server. They use the _config API to test CouchDB in >>>>>>>> different configurations from the default one. All of these tests now >>>>>> throw >>>>>>>> an error that _config isn’t available on the clustered API. How should >>>>>> we >>>>>>>> handle them? >>>>>>>> >>>>>>>> The tests are: >>>>>>>> >>>>>>>> - erlang_views.js >>>>>>>> - jsonp.js >>>>>>>> - oauth_users_db.js >>>>>>>> - oauth_users_db.js >>>>>>>> - proxyauth.js >>>>>>>> - reader_acl.js >>>>>>>> - rewrite.js >>>>>>>> - security_validation.js >>>>>>>> - show_documents.js >>>>>>>> - users_db.js >>>>>>>> - view_errors.js >>>>>>>> >>>>>>>> >>>>>>>> And a few tests aside from the proper compaction tests call _compact >>>>>> which >>>>>>>> is also not available on the cluster. What to do with them? >>>>>>>> >>>>>>>> - recreate_doc.js >>>>>>>> - and a few more that are currently disabled for other reasons, sorry >>>> I >>>>>>>> don’t have a full list at this point. >>>>>>>> >>>>>>>> * * * >>>>>>>> >>>>>>>> >>>>>>>> Bottom line: It’d be great if other folks here could join in the fun. >>>> No >>>>>>>> Erlang needed, this is purely JavaScript and CouchDB HTTP API-work :) >>>>>>>> >>>>>>>> >>>>>>>> ## How to help: >>>>>>>> >>>>>>>> 1. Check out https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip >>>>>>>> 2. run `make clean && ./configure -c --disable-docs --disable-fauxton >>>> && >>>>>>>> make` >>>>>>>> 3. `make javascript` runs the whole test suite >>>>>>>> >>>>>>>> To run the whole test suite without waiting for rebar to confirm that >>>> no >>>>>>>> Erlang code needs to be compiled, run `./test/javascript/run` for the >>>>>> whole >>>>>>>> test suite, or `./test/javascript/run all_docs` to run an individual >>>>>> test >>>>>>>> (all_docs runs test/javascript/tests/all_docs.js, other file names >>>> match >>>>>>>> accordingly). >>>>>>>> >>>>>>>> Feel free to send me PRs against the branch on >>>>>>>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip, I’ll merge >>>>>>>> them, so the PR on apache/couchdb gets updated. >>>>>>>> >>>>>>>> I’m looking forward to your contributions! :) >>>>>>>> >>>>>>>> If you have any questions or have trouble getting started or just >>>> would >>>>>>>> like some hand-holding, let me know! >>>>>>>> >>>>>>>> >>>>>>>> Best >>>>>>>> Jan >>>>>>>> -- >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> -- >>>>>> Professional Support for Apache CouchDB: >>>>>> http://www.neighbourhood.ie/couchdb-support/ >>>>>> >>>>>> >>>> >>>> -- >>>> Professional Support for Apache CouchDB: >>>> http://www.neighbourhood.ie/couchdb-support/ >>>> >>>> >> >> -- >> Professional Support for Apache CouchDB: >> http://www.neighbourhood.ie/couchdb-support/ >> > > -- > Professional Support for Apache CouchDB: > http://www.neighbourhood.ie/couchdb-support/ > -- Professional Support for Apache CouchDB: http://www.neighbourhood.ie/couchdb-support/
