ok this is a short update of Jan's description for new folks starting with fixing the tests (thank's in advance).
1. Check out https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip 2. cd into the repo and `git checkout js-tests-on-2.0-wip` 3. run `make clean if this is not the first time you build 4. `./configure -c --disable-docs --disable-fauxton && make` Now there are two options: * run the whole test suite `make javascript` * run a single test `./dev/run --with-admin-party-please` `./test/javascript/run test-filename` (in another tab) Cheers Andy P.S.: Not torpedoing Jan's great description in any way :-) On 12 October 2015 at 13:33, Jan Lehnardt <j...@apache.org> wrote: > Oops, hits end too early. > > > Okay -n 1 explains this > > On 12 Oct 2015, at 13:32, Jan Lehnardt <j...@apache.org> wrote: > > > > > >> On 12 Oct 2015, at 13:30, Dale Harvey <d...@arandomurl.com> wrote: > >> > >> Started with > >> > >> python dev/run -n 1 --with-admin-party-please & > >> > >> and export COUCH_HOST='http://127.0.0.1:15984' > >> > >> > >> On 12 October 2015 at 13:26, Jan Lehnardt <j...@apache.org> wrote: > >> > >>> > >>>> On 12 Oct 2015, at 13:15, Dale Harvey <d...@arandomurl.com> wrote: > >>>> > >>>>> - instead of fixed test db names like `test_suite_db` that are used > in > >>>> almost every test, > >>>>> we now generate a random database name for each test. The JS tests > >>>> expected db > >>>>> deletion to be synchronous, and that’s no longer the case in 2.0. > >>> Instead > >>>> of > >>>>> error-prone polling to wait for db deletion, we just use new db for > >>> every > >>>> test. > >>>>> An added benefit now is that (some) tests can run in parallel. > >>>> > >>>> Little confused by this, PouchDB uses consistent names in its test > suite > >>>> (we had random > >>>> ones for a while and moved away), we very much depend on the database > >>> being > >>>> deleted > >>>> on the request returning and dont seem to have any problems with it > >>> against > >>>> master. > >>> > >>> Does the PouchDB test suite runs against the 5986 port, or with an n=1 > >>> cluster? > >>> This is with the default n=3 cluster against 5986. > >>> > >>> Best > >>> Jan > >>> -- > >>> > >>>> > >>>> On 12 October 2015 at 12:21, Jan Lehnardt <j...@apache.org> wrote: > >>>> > >>>>> > >>>>>> On 11 Oct 2015, at 18:53, Sebastian Rothbucher < > >>>>> sebastianrothbuc...@googlemail.com> wrote: > >>>>>> > >>>>>> Hi Jan, > >>>>>> > >>>>>> I transferred some of my findings to your branch - pls. find a PR > for > >>> the > >>>>>> first few tests submitted 2 your branch. I put in TODOs where either > >>>>> there > >>>>>> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853) or we > have > >>>>> to > >>>>>> do some more work (like making config available). > >>>>> > >>>>> Excellent, thanks! :) > >>>>> > >>>>> > >>>>>> Will from the Pouch team had the great idea of using -n 1 to > mitigate > >>>>>> Heisenbugs due to distribution. I adopted it and it did work quite > >>> well. > >>>>> It > >>>>>> might make sense to put this into the dev/run call as a general > rule, > >>>>> esp. > >>>>>> when checking for 201 (VS 202) status codes. Probably you know more > on > >>>>>> that. > >>>>> > >>>>> We should carefully evaluate this :) > >>>>> > >>>>> > >>>>>> Anyways: I think we have to keep using _config on the local port as > >>> many > >>>>>> tests (like auth tests) depend on changed configurations. BTW: w/ > -n 1 > >>>>> it's > >>>>>> only one call, but calling 3 _config(s) migt also work. > >>>>>> Hopefully (I just arrived at some test w/ that recently) we can > spare > >>>>>> ourselves the _compact thing. I think we'd have to compact every > single > >>>>>> shard via local port, right? for the first tests, we can go w/out > as it > >>>>>> seems. Maybe we should stick to that for a while. > >>>>> > >>>>> My concern with this is that as far as I care, the local port is > >>> irrelevant > >>>>> to how CouchDB works. Of course it isn’t 100% irrelevant, but the > >>> cluster > >>>>> port is our publicised API for 2.0 and it’s the one that replaces the > >>> 1.x > >>>>> API for people upgrading. So having any tests that run on the local > port > >>>>> of 2.0 don’t help us in any way (that’s not to say it still useful to > >>> have > >>>>> tests against them for the moment). > >>>>> > >>>>> Many of the _config-dependent tests substitute out the _users or > >>>>> _replicator > >>>>> databases, maybe we can find a workaround for those by just using the > >>> stock > >>>>> databases. > >>>>> > >>>>> For the other tests, maybe we split them out into their own “test > >>> suites” > >>>>> that start their own test cluster in the configuration they need and > >>> shut > >>>>> it down afterwards. > >>>>> > >>>>> Best > >>>>> Jan > >>>>> -- > >>>>> > >>>>> > >>>>>> > >>>>>> Looking fw 2 your thoughts > >>>>>> > >>>>>> Best > >>>>>> Sebastian > >>>>>> > >>>>>> P.S:: Maybe we can adopt some stuff from > >>>>>> > >>>>> > >>> > https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest > >>>>>> > >>>>>> P.P.S.: Will keep working on it, just not in the next few days, > >>>>>> unfortunately ;-( > >>>>>> > >>>>>> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <j...@apache.org> > wrote: > >>>>>> > >>>>>>> (I need your help deciding what to do with a bunch of tests, jump > to > >>>>>>> “Discussion” below if you don’t care about the details before.) > >>>>>>> > >>>>>>> * * * > >>>>>>> > >>>>>>> Hi all, > >>>>>>> > >>>>>>> I spent some more time getting the 1.x JavaScript tests running on > the > >>>>>>> cluster port of 2.0. WIP here: > >>>>> https://github.com/apache/couchdb/pull/354 > >>>>>>> — I’ve gotten a lot further, but there is still a lot to do and > >>> everyone > >>>>>>> here can help, I’ve posted instructions on how to do so further > down. > >>> No > >>>>>>> Erlang knowledge needed, this is ideal for folks who want to start > >>> with > >>>>>>> contributing to CouchDB. > >>>>>>> > >>>>>>> > >>>>>>> Notable changes: > >>>>>>> > >>>>>>> - instead of fixed test db names like `test_suite_db` that are > used in > >>>>>>> almost every test, we now generate a random database name for each > >>> test. > >>>>>>> The JS tests expected db deletion to be synchronous, and that’s no > >>>>> longer > >>>>>>> the case in 2.0. Instead of error-prone polling to wait for db > >>>>> deletion, we > >>>>>>> just use new db for every test. An added benefit now is that (some) > >>>>> tests > >>>>>>> can run in parallel. > >>>>>>> > >>>>>>> - tests now clean up after themselves (incomplete): tests > previously > >>>>> only > >>>>>>> collectively used three or four databases, and deleted and > re-created > >>>>> them > >>>>>>> before each test was run. Now with the change of creating a new > >>> database > >>>>>>> per test, we also need to clean up databases at the end of a test, > >>>>> because > >>>>>>> otherwise we a) leave a lot of databases around and b) CouchDB runs > >>> out > >>>>> of > >>>>>>> file descriptors during the test run. The cleanup isn’t 100% done > yet, > >>>>> so > >>>>>>> we should check for stray databases after JS test runs until we’ve > got > >>>>> it > >>>>>>> cleaned. One particular aspect here is when a test fails, it never > >>>>> reaches > >>>>>>> the cleanup code, so we might have to wrap everything into a > try/catch > >>>>>>> block. > >>>>>>> > >>>>>>> - I disabled all tests that still fail and made them so they print > the > >>>>>>> reason why they are failing, so that fixing them should be a lot > >>> easier > >>>>>>> now. Some of them print just “TODO”, their error cause is yet to be > >>>>>>> determined. > >>>>>>> > >>>>>>> - couchdb.query() which previously executed temporary views, now > under > >>>>> the > >>>>>>> hood creates a design doc with a random name and queries that view. > >>> This > >>>>>>> fixes all tests relying on temp views without having to change them > >>> all > >>>>>>> individually. This is because 2.0 doesn’t have temp views anymore > >>>>> (which is > >>>>>>> a good thing). > >>>>>>> > >>>>>>> - restartServer() is no longer used (mostly). > >>>>>>> > >>>>>>> I already filed a few bugs that showed things missing in the > cluster > >>> API > >>>>>>> that we’d expect to be there: > >>>>>>> > >>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2849 > >>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2850 > >>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2851 > >>>>>>> > >>>>>>> Based on this I am ever more convinced we should get this test > suite > >>>>> work > >>>>>>> against the 2.0 clustered API, because we’ll be finding some more > >>>>>>> inconsistencies that we should fix before we invite beta testers. > >>>>>>> > >>>>>>> > >>>>>>> ## Discussion: > >>>>>>> > >>>>>>> We have a whole number of tests that use the test-internal function > >>>>>>> run_on_modified_server. They use the _config API to test CouchDB in > >>>>>>> different configurations from the default one. All of these tests > now > >>>>> throw > >>>>>>> an error that _config isn’t available on the clustered API. How > should > >>>>> we > >>>>>>> handle them? > >>>>>>> > >>>>>>> The tests are: > >>>>>>> > >>>>>>> - erlang_views.js > >>>>>>> - jsonp.js > >>>>>>> - oauth_users_db.js > >>>>>>> - oauth_users_db.js > >>>>>>> - proxyauth.js > >>>>>>> - reader_acl.js > >>>>>>> - rewrite.js > >>>>>>> - security_validation.js > >>>>>>> - show_documents.js > >>>>>>> - users_db.js > >>>>>>> - view_errors.js > >>>>>>> > >>>>>>> > >>>>>>> And a few tests aside from the proper compaction tests call > _compact > >>>>> which > >>>>>>> is also not available on the cluster. What to do with them? > >>>>>>> > >>>>>>> - recreate_doc.js > >>>>>>> - and a few more that are currently disabled for other reasons, > sorry > >>> I > >>>>>>> don’t have a full list at this point. > >>>>>>> > >>>>>>> * * * > >>>>>>> > >>>>>>> > >>>>>>> Bottom line: It’d be great if other folks here could join in the > fun. > >>> No > >>>>>>> Erlang needed, this is purely JavaScript and CouchDB HTTP API-work > :) > >>>>>>> > >>>>>>> > >>>>>>> ## How to help: > >>>>>>> > >>>>>>> 1. Check out > https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip > >>>>>>> 2. run `make clean && ./configure -c --disable-docs > --disable-fauxton > >>> && > >>>>>>> make` > >>>>>>> 3. `make javascript` runs the whole test suite > >>>>>>> > >>>>>>> To run the whole test suite without waiting for rebar to confirm > that > >>> no > >>>>>>> Erlang code needs to be compiled, run `./test/javascript/run` for > the > >>>>> whole > >>>>>>> test suite, or `./test/javascript/run all_docs` to run an > individual > >>>>> test > >>>>>>> (all_docs runs test/javascript/tests/all_docs.js, other file names > >>> match > >>>>>>> accordingly). > >>>>>>> > >>>>>>> Feel free to send me PRs against the branch on > >>>>>>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip, I’ll > merge > >>>>>>> them, so the PR on apache/couchdb gets updated. > >>>>>>> > >>>>>>> I’m looking forward to your contributions! :) > >>>>>>> > >>>>>>> If you have any questions or have trouble getting started or just > >>> would > >>>>>>> like some hand-holding, let me know! > >>>>>>> > >>>>>>> > >>>>>>> Best > >>>>>>> Jan > >>>>>>> -- > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> > >>>>> -- > >>>>> Professional Support for Apache CouchDB: > >>>>> http://www.neighbourhood.ie/couchdb-support/ > >>>>> > >>>>> > >>> > >>> -- > >>> Professional Support for Apache CouchDB: > >>> http://www.neighbourhood.ie/couchdb-support/ > >>> > >>> > > > > -- > > Professional Support for Apache CouchDB: > > http://www.neighbourhood.ie/couchdb-support/ > > > > -- > Professional Support for Apache CouchDB: > http://www.neighbourhood.ie/couchdb-support/ > > -- Andy Wenk Hamburg - Germany RockIt! GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588 https://people.apache.org/keys/committer/andywenk.asc
