nickva commented on issue #470: Scheduling Replicator URL: https://github.com/apache/couchdb/pull/470#issuecomment-297515901 @redgeoff Thanks for taking a look! `max_jobs` parameter would limit the number of running replication jobs at one time. File descriptors can be exhausted by other things not just replication tasks. In this case it is probably the databases opened. Also note each db by default is split into 8 shards (the `q` value). So that starts to add up. I gave your setup a try and got it to work with these settings: (I didn't try to minimize them, you can probably tweak them to find minimal values) `ulimit -n 8192` config settings: `couchdb.max_dbs_open = 3000` `cluster.q=1` `cluster.n=1` `replicator.max_jobs = 10` `replicator.worker_processes = 1` I put q=1, n=1 based on your use case in https://issues.apache.org/jira/browse/COUCHDB-3391 With all that set up. Started couch (I build a local scheduling replicator in Vagrant) ``` couchdb@ubuntu-xenial:~/couchdb$ ./bin/couchdb [info] 2017-04-26T19:22:44.457066Z couchdb@localhost <0.7.0> -------- Application couch_log started on node couchdb@localhost ... ``` Created `_users` and `_replicator` dbs. I am not sure if your test does this. Here is how it looks: ``` couchdb@ubuntu-xenial:/vagrant$ http put http://localhost:5984/_users HTTP/1.1 201 Created ... { "ok": true } couchdb@ubuntu-xenial:/vagrant$ http put http://localhost:5984/_replicator HTTP/1.1 201 Created ... { "ok": true } ``` Ran your demo: ``` npm run demo > [email protected] demo /vagrant > ./index.js ``` without any errors. And finally showing off the states-based filtering: ``` $ http http://localhost:5984/_scheduler/docs?states=running HTTP/1.1 200 OK ... { "docs": [ { "database": "_replicator", "doc_id": "7e801385699c0723d64ce2b684000378", "error_count": 0, "id": "2f691a8a3e337142691e7679e2c146a1+continuous", "info": null, "last_updated": "2017-04-26T19:14:07Z", "node": "couchdb@localhost", "proxy": null, "source": "http://localhost:5984/test_1493234045679_2_a/";, "start_time": "2017-04-26T19:14:07Z", "state": "running", "target": "http://localhost:5984/test_1493234045679_2_b/"; }, { "database": "_replicator", "doc_id": "7e801385699c0723d64ce2b684001144", "error_count": 0, "id": "2f42b5ccbfb20e82d319054af8cb7828+continuous", "info": null, "last_updated": "2017-04-26T19:14:08Z", "node": "couchdb@localhost", "proxy": null, "source": "http://localhost:5984/test_1493234045679_6_a/";, "start_time": "2017-04-26T19:14:08Z", "state": "running", "target": "http://localhost:5984/test_1493234045679_6_b/"; }, { "database": "_replicator", "doc_id": "7e801385699c0723d64ce2b684001d86", "error_count": 0, "id": "a7361651a4c5fbb64ae9dd601005dc11+continuous", "info": null, "last_updated": "2017-04-26T19:14:08Z", "node": "couchdb@localhost", "proxy": null, "source": "http://localhost:5984/test_1493234045679_4_a/";, "start_time": "2017-04-26T19:14:08Z", "state": "running", "target": "http://localhost:5984/test_1493234045679_4_b/"; }, ... ], "offset": 0, "total_rows": 1000 } ``` Without any crashes ``` http http://localhost:5984/_scheduler/docs?states=crashing HTTP/1.1 200 OK Cache-Control: must-revalidate Content-Type: application/json Date: Wed, 26 Apr 2017 19:26:03 GMT Server: CouchDB/2.0.0-49786f7 (Erlang OTP/18) Transfer-Encoding: chunked X-Couch-Request-ID: 465148b54d X-CouchDB-Body-Time: 0 { "docs": [], "offset": 0, "total_rows": 1000 } ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
With regards, Apache Git Services
