[
https://issues.apache.org/jira/browse/COUCHDB-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jay Doane closed COUCHDB-3061.
------------------------------
Resolution: Fixed
This should be resolved by this commit:
https://github.com/apache/couchdb-couch/commit/6997c3c236ef9c492810410077fbdff1c303bca0
> Decrease search time for deeply buried headers
> ----------------------------------------------
>
> Key: COUCHDB-3061
> URL: https://issues.apache.org/jira/browse/COUCHDB-3061
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Jay Doane
> Assignee: Jay Doane
>
> When a db compaction is interrupted and restarted, it's possible for the most
> recent header written to the .compact.meta file to be buried under many GB
> from the end of file. For example, an interrupted compaction left the
> following files:
> {code}
> -rw-r--r-- 1 dbcore dbcore 47G Jun 27 04:29
> /srv/db/shards/40000000-7fffffff/opendi/yellow-ng-loadbalancer.1461156255.couch.compact.data
> -rw-r--r-- 1 dbcore dbcore 12G Jun 27 05:20
> /srv/db/shards/40000000-7fffffff/opendi/yellow-ng-loadbalancer.1461156255.couch.compact.meta
> {code}
> but
> {code}
> # grep -abo db_header yellow-ng-loadbalancer.1461156255.couch.compact.meta |
> tail -1
> 4364894251:db_header
> {code}
> which means the current algorithm must search through about 8GB of file
> before it gets to a header. I measured how long it takes, both on an SSD
> laptop and SSD server, getting similar numbers:
> {code}
> ([email protected])18>
> timer:tc(couch_file,test_find_header,["/srv/jdoane/yellow-ng-loadbalancer.1461156255.couch.compact.meta",
> default]).
> {328335338, ...
> ([email protected])27>
> timer:tc(couch_file,test_find_header,["/Users/jay/proj/ibm/sample-data/yellow-ng-loadbalancer.1461156255.couch.compact.meta",
> default]).
> {426650530, ...
> {code}
> which is between 328-427 seconds, or 19-25 MB/s.
> One reason for this relative slowness is because the current algorithm
> performs a disk read for every block it searches:
> https://github.com/apache/couchdb-couch/blob/master/src/couch_file.erl#L537-L539
> We can improve the speed by loading a "chunk" of many blocks into memory with
> a single read operation, and then search each block in memory, trading off
> memory for speed. Ideally, the tradeoff can be made configurable, so that
> existing speed/memory behavior can be retained by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
