[
https://issues.apache.org/jira/browse/COUCHDB-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972770#comment-15972770
]
ASF subversion and git services commented on COUCHDB-3324:
----------------------------------------------------------
Commit d775c012517530b24dba4bf43c458be07613a36a in couchdb's branch
refs/heads/63012-scheduler from [~vatamane]
[ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=d775c01 ]
Add `_scheduler/{jobs,docs}` API endpoints
The `_scheduler/jobs` endpoint provides a view of all replications managed by
the scheduler. This endpoint includes more information on the replication than
the `_scheduler/docs` endpoint, including the history of state transitions of
the replication. This part was implemented by Benjamin Bastian.
The `_scheduler/docs` endpoint provides a view of all replicator docs which
have been seen by the scheduler. This endpoint includes useful information such
as the state of the replication and the coordinator node. The implemention of
`_scheduler/docs` mimics closely `_all_docs` behavior: similar pagination,
HTTP request processing and fabric / rexi setup. The algorithm is roughly
as follows:
* http endpoint:
- parses query args like it does for any view query
- parses states to filter by, states are kept in the `extra` query arg
* Call is made to couch_replicator_fabric. This is equivalent to
fabric:all_docs. Here the typical fabric / rexi setup is happening.
* Fabric worker is in `couch_replicator_fabric_rpc:docs/3`. This worker is
similar to fabric_rpc's all_docs handler. However it is a bit more intricate
to handle both replication document in terminal state as well as those which
are active.
- Before emitting it queries the state of the document to see if it is in a
terminal state. If it is, it filters it and decides if it should be
emitted or not.
- If the document state cannot be found from the document. It tries to
fetch active state from local node's doc processor via key based lookup.
If it finds, it can also filter it based on state and emit it or skip.
- If the document cannot be found in the node's local doc processor ETS
table, the row is emitted with a doc value of `undecided`. This will
let the coordinator fetch the state by possibly querying other nodes's
doc processors.
* Coordinator then starts handling messages. This also mostly mimics all_docs.
At this point the most interesting thing is handling `undecided` docs. If
one is found, then `replicator:active_doc/2` is queried. There, all nodes
where document shards live are queries. This is better than a previous
implementation where all nodes were queries all the time.
* The final work happens in `couch_replicator_httpd` where the emitting
callback is. There we only the doc is emitted (not keys, rows, values).
Another thing that happens is the `Total` value is decremented to
account for the always-present _design doc.
Because of this a bunch of stuff was removed. Including an extra view which
was build and managed by the previous implementation.
As a bonus, other view-related parameters such as skip and limit seems to
work out of the box and don't have to be implemented ad-hoc.
Also, most importantly many thanks to Paul Davis for suggesting this approach.
Jira: COUCHDB-3324
> Scheduling Replicator
> ---------------------
>
> Key: COUCHDB-3324
> URL: https://issues.apache.org/jira/browse/COUCHDB-3324
> Project: CouchDB
> Issue Type: New Feature
> Reporter: Nick Vatamaniuc
>
> Improve CouchDB replicator
> * Allow running a large number of replication jobs
> * Improve API with a focus on ease of use and performance. Avoid updating
> replication document with transient state updates. Instead create a proper
> API for querying replication states. At the same time provide a compatibility
> mode to let users keep existing behavior (of getting updates in documents).
> * Improve network resource usage and performance. Multiple connection to the
> same cluster could share socket connections
> * Handle rate limiting on target and source HTTP endpoints. Let replication
> request auto-discover rate limit capacity based on a proven algorithm such as
> Additive Increase / Multiplicative Decrease feedback control loop.
> * Improve performance by avoiding repeatedly retrying failing replication
> jobs. Instead use exponential backoff.
> * Improve recovery from long (but temporary) network failure. Currently if
> replications jobs fail to start 10 times in a row they will not be retried
> anymore. This is not always desirable. In case of a long enough DNS (or other
> network) failure replication jobs will effectively stop until they are
> manually restarted.
> * Better handling of filtered replications: Failing to fetch filters could
> block couch replicator manager, lead to message queue backups and memory
> exhaustion. Also, when replication filter code changes update replication
> accordingly (replication job ID should change in that case)
> * Provide better metrics to introspect replicator behavior.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
