> On 12 Oct 2015, at 13:15, Dale Harvey <d...@arandomurl.com> wrote: > >> - instead of fixed test db names like `test_suite_db` that are used in > almost every test, >> we now generate a random database name for each test. The JS tests > expected db >> deletion to be synchronous, and that’s no longer the case in 2.0. Instead > of >> error-prone polling to wait for db deletion, we just use new db for every > test. >> An added benefit now is that (some) tests can run in parallel. > > Little confused by this, PouchDB uses consistent names in its test suite > (we had random > ones for a while and moved away), we very much depend on the database being > deleted > on the request returning and dont seem to have any problems with it against > master.
Does the PouchDB test suite runs against the 5986 port, or with an n=1 cluster? This is with the default n=3 cluster against 5986. Best Jan -- > > On 12 October 2015 at 12:21, Jan Lehnardt <j...@apache.org> wrote: > >> >>> On 11 Oct 2015, at 18:53, Sebastian Rothbucher < >> sebastianrothbuc...@googlemail.com> wrote: >>> >>> Hi Jan, >>> >>> I transferred some of my findings to your branch - pls. find a PR for the >>> first few tests submitted 2 your branch. I put in TODOs where either >> there >>> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853) or we have >> to >>> do some more work (like making config available). >> >> Excellent, thanks! :) >> >> >>> Will from the Pouch team had the great idea of using -n 1 to mitigate >>> Heisenbugs due to distribution. I adopted it and it did work quite well. >> It >>> might make sense to put this into the dev/run call as a general rule, >> esp. >>> when checking for 201 (VS 202) status codes. Probably you know more on >>> that. >> >> We should carefully evaluate this :) >> >> >>> Anyways: I think we have to keep using _config on the local port as many >>> tests (like auth tests) depend on changed configurations. BTW: w/ -n 1 >> it's >>> only one call, but calling 3 _config(s) migt also work. >>> Hopefully (I just arrived at some test w/ that recently) we can spare >>> ourselves the _compact thing. I think we'd have to compact every single >>> shard via local port, right? for the first tests, we can go w/out as it >>> seems. Maybe we should stick to that for a while. >> >> My concern with this is that as far as I care, the local port is irrelevant >> to how CouchDB works. Of course it isn’t 100% irrelevant, but the cluster >> port is our publicised API for 2.0 and it’s the one that replaces the 1.x >> API for people upgrading. So having any tests that run on the local port >> of 2.0 don’t help us in any way (that’s not to say it still useful to have >> tests against them for the moment). >> >> Many of the _config-dependent tests substitute out the _users or >> _replicator >> databases, maybe we can find a workaround for those by just using the stock >> databases. >> >> For the other tests, maybe we split them out into their own “test suites” >> that start their own test cluster in the configuration they need and shut >> it down afterwards. >> >> Best >> Jan >> -- >> >> >>> >>> Looking fw 2 your thoughts >>> >>> Best >>> Sebastian >>> >>> P.S:: Maybe we can adopt some stuff from >>> >> https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest >>> >>> P.P.S.: Will keep working on it, just not in the next few days, >>> unfortunately ;-( >>> >>> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <j...@apache.org> wrote: >>> >>>> (I need your help deciding what to do with a bunch of tests, jump to >>>> “Discussion” below if you don’t care about the details before.) >>>> >>>> * * * >>>> >>>> Hi all, >>>> >>>> I spent some more time getting the 1.x JavaScript tests running on the >>>> cluster port of 2.0. WIP here: >> https://github.com/apache/couchdb/pull/354 >>>> — I’ve gotten a lot further, but there is still a lot to do and everyone >>>> here can help, I’ve posted instructions on how to do so further down. No >>>> Erlang knowledge needed, this is ideal for folks who want to start with >>>> contributing to CouchDB. >>>> >>>> >>>> Notable changes: >>>> >>>> - instead of fixed test db names like `test_suite_db` that are used in >>>> almost every test, we now generate a random database name for each test. >>>> The JS tests expected db deletion to be synchronous, and that’s no >> longer >>>> the case in 2.0. Instead of error-prone polling to wait for db >> deletion, we >>>> just use new db for every test. An added benefit now is that (some) >> tests >>>> can run in parallel. >>>> >>>> - tests now clean up after themselves (incomplete): tests previously >> only >>>> collectively used three or four databases, and deleted and re-created >> them >>>> before each test was run. Now with the change of creating a new database >>>> per test, we also need to clean up databases at the end of a test, >> because >>>> otherwise we a) leave a lot of databases around and b) CouchDB runs out >> of >>>> file descriptors during the test run. The cleanup isn’t 100% done yet, >> so >>>> we should check for stray databases after JS test runs until we’ve got >> it >>>> cleaned. One particular aspect here is when a test fails, it never >> reaches >>>> the cleanup code, so we might have to wrap everything into a try/catch >>>> block. >>>> >>>> - I disabled all tests that still fail and made them so they print the >>>> reason why they are failing, so that fixing them should be a lot easier >>>> now. Some of them print just “TODO”, their error cause is yet to be >>>> determined. >>>> >>>> - couchdb.query() which previously executed temporary views, now under >> the >>>> hood creates a design doc with a random name and queries that view. This >>>> fixes all tests relying on temp views without having to change them all >>>> individually. This is because 2.0 doesn’t have temp views anymore >> (which is >>>> a good thing). >>>> >>>> - restartServer() is no longer used (mostly). >>>> >>>> I already filed a few bugs that showed things missing in the cluster API >>>> that we’d expect to be there: >>>> >>>> - https://issues.apache.org/jira/browse/COUCHDB-2849 >>>> - https://issues.apache.org/jira/browse/COUCHDB-2850 >>>> - https://issues.apache.org/jira/browse/COUCHDB-2851 >>>> >>>> Based on this I am ever more convinced we should get this test suite >> work >>>> against the 2.0 clustered API, because we’ll be finding some more >>>> inconsistencies that we should fix before we invite beta testers. >>>> >>>> >>>> ## Discussion: >>>> >>>> We have a whole number of tests that use the test-internal function >>>> run_on_modified_server. They use the _config API to test CouchDB in >>>> different configurations from the default one. All of these tests now >> throw >>>> an error that _config isn’t available on the clustered API. How should >> we >>>> handle them? >>>> >>>> The tests are: >>>> >>>> - erlang_views.js >>>> - jsonp.js >>>> - oauth_users_db.js >>>> - oauth_users_db.js >>>> - proxyauth.js >>>> - reader_acl.js >>>> - rewrite.js >>>> - security_validation.js >>>> - show_documents.js >>>> - users_db.js >>>> - view_errors.js >>>> >>>> >>>> And a few tests aside from the proper compaction tests call _compact >> which >>>> is also not available on the cluster. What to do with them? >>>> >>>> - recreate_doc.js >>>> - and a few more that are currently disabled for other reasons, sorry I >>>> don’t have a full list at this point. >>>> >>>> * * * >>>> >>>> >>>> Bottom line: It’d be great if other folks here could join in the fun. No >>>> Erlang needed, this is purely JavaScript and CouchDB HTTP API-work :) >>>> >>>> >>>> ## How to help: >>>> >>>> 1. Check out https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip >>>> 2. run `make clean && ./configure -c --disable-docs --disable-fauxton && >>>> make` >>>> 3. `make javascript` runs the whole test suite >>>> >>>> To run the whole test suite without waiting for rebar to confirm that no >>>> Erlang code needs to be compiled, run `./test/javascript/run` for the >> whole >>>> test suite, or `./test/javascript/run all_docs` to run an individual >> test >>>> (all_docs runs test/javascript/tests/all_docs.js, other file names match >>>> accordingly). >>>> >>>> Feel free to send me PRs against the branch on >>>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip, I’ll merge >>>> them, so the PR on apache/couchdb gets updated. >>>> >>>> I’m looking forward to your contributions! :) >>>> >>>> If you have any questions or have trouble getting started or just would >>>> like some hand-holding, let me know! >>>> >>>> >>>> Best >>>> Jan >>>> -- >>>> >>>> >>>> >>>> >> >> -- >> Professional Support for Apache CouchDB: >> http://www.neighbourhood.ie/couchdb-support/ >> >> -- Professional Support for Apache CouchDB: http://www.neighbourhood.ie/couchdb-support/
