janl opened a new pull request #1200: [DISCUSS] CouchDB Request Size Limits URL: https://github.com/apache/couchdb/pull/1200 *Note: the text below is written in a style that would allow it to be included in the CouchDB 2.2.0 documentation and/or release notes.* # CouchDB Request Size Limits There are multiple configuration variables for CouchDB that determine request size limits. This document explains the configuration variables, how they work together, and why they exist in the first place ## Why Limit Requests by Size Allowing requests of unlimited size to any network server is a [denial of service vector](wikipedia: denial of service). To allow safe operation of CouchDB, even on a network with hostile third parties, various request size limits exist. ## The Request Size Limits `max_http_request_size`: the maximum number bytes a request to a CouchDB server can have. `max_document_size`: the maximum number of bytes for a JSON document written to CouchDB. `max_attachment_size`: the maximum number of bytes for any one attachment written to CouchDB. ## Background There are three distinct ways of getting data into CouchDB: 1. The standard JSON Document API, which uses plain JSON, if binary data is involved, it has to be encoded as base64. The base64 option only exists for legacy reasons and it is not recommended to be used. 2. The standalone attachment API, which allows transferring of binary attachment data without encoding as base64. 3. The multipart HTTP API: it allows the mix of JSON data and binary attachment data without encoding as base64. The CouchDB replicator uses this. In version 2.1, CouchDB started enforcing a 64MB limit for `max_http_request_size` on all requests, but did not apply this to the standalone attachment API. This had the unfortunate side effect that one could create a doc that is smaller than `max_http_request_size` with an attachment that is bigger than `max_http_request_size`. In addition, one could create a doc with two or more attachments that were each smaller than `max_http_request_size` but together bigger than `max_http_request_size`. The result in this scenario now is that these documents could no longer be replicated to CouchDB nodes with the same default configuration (or even to the same node). Regardless to say, this is a very unfortunate user experience: create a number of documents with attachments, and at some not immediately obvious point, replications start failing. ### Large Documents and Attachments While CouchDB works reasonably well with almost any sort of JSON data sizes and attachment sizes. The development team makes recommendations as to the various limits for ideal and optimal uses. CouchDB users may vary from these recommendations, but will need to be okay with the resulting operational implications, like increased CPU & RAM usage as well as increased latency for many core operations. Before CouchDB 2.1.0 there were no real limits imposed, and before CouchDB 2.2.0 the available limits weren?t applied uniformly, leading to surprising behaviour as for example outlined above. CouchDB 2.2.0 and later aims to have a complete set of limits that avoids any unexpected behaviour, but the limits imposed won?t be set by default in order to preserve backwards compatibility. Starting with CouchDB 3.0.0 the recommended limits will be set by default and users migrating from earlier versions of CouchDB need to adjust them, if their use-case requires it. The CouchDB team might produce a utility script that would allow to determine the required settings from an existing CouchDB installation, if resources can be made available for this. Starting with CouchDB 2.2.0, the CouchDB distribution will come with an additional configuration file local.ini-recommended* with the developer-recommended defaults and explanations for what happens when these defaults are exceeded. *An alternative solution could avoid using the `max_attachments_per_doc` and reject attachment additions based on the existing doc + attachments size plus the new attachment size, but this PR/Discussion suggests that having another config value with sensible defaults here will nudge users into doing the right thing* ## Limits by Version In order to account for all use-cases and the interplay of the different APIs, CouchDB 2.2.0 introduces a new limit `max_attachments_per_document`. This allows the application of a formula to show the interplay of all limits: ``` max_http_request_size = max_document_size + multipart HTTP boundary data + max_attachments_per_doc * (max_attachment_size + multipart HTTP boundary data) ``` Using this formula, any doc update (JSON or attachments) can check whether it would exceed `max_http_request_size` which would cause replication to fail. CouchDB Version | `max_http_request_size` | `max_document_size` | `max_attachment_size` | `max_attachments_per_document*` ----------------|-----------------------|-------------------|---------------------|------------------------------ 2.0.0 and earlier | unlimited | 4GB | N/a | N/a 2.1.0 | 64MB | 4GB | Unlimited | N/a 2.2.0* | 64MB | 4GB | Unlimited | Unlimited 3.0.0* | 64MB | 4MB | 6MB | 10 * Proposed names and values The table shows the approximate sizes (sans HTTP multipart boundaries) for all limits. CouchDB versions earlier than 3.0.0 will still encounter the behaviour of not being able to replicate documents that have attachments that alone or together exceed `max_http_request_size`. # Implementation This draft implementation introduces the new `max_attachments_per_document` to show how it could work. Tests will need to be added to validate that all three API routes are covered (casual review suggests they are, but we do, of course need tests). I stopped short of adding tests so we can discuss the details of this suggestion first. # To 2.2.0 or not to 2.2.0 Since we started on [the 2.2.0 milestone](https://github.com/apache/couchdb/milestone/3), this might be too big a thing to discuss and finish. I?d be very okay with bumping this to 2.3.0 as long as we document the behaviour in the 2.2.0 release notes.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
