adam-spencer opened a new pull request, #5332: URL: https://github.com/apache/solr/pull/5332
https://issues.apache.org/jira/browse/SOLR-15680 > This remains a draft while I: > * Complete a code review > * Complete a documentation review > * Test on real S3 with KMS (only used mocks so far) # Description Adds optional client-side encryption to `S3BackupRepository`, following the approach Houston Putman outlined on [SOLR-15680](https://issues.apache.org/jira/browse/SOLR-15680): `solr.xml` options for a key, and the official Amazon S3 Encryption Client. Backup files are encrypted on the Solr node before upload and decrypted after download. Encryption is off by default and is enabled by one of two mutually exclusive options: - `s3.encryption.kms.keyId`: an AWS KMS key, optionally with `s3.encryption.kms.region` and `s3.encryption.kms.endpoint` - `s3.encryption.aes.keyFile`: a file holding a base64-encoded 256-bit AES key The behaviour of `S3OutputStream.flush()` has been updated for all uploads, holding back the last 1-16 buffered bytes for the final part. This means the upload waits until the buffer is greater than 5MB (rather than exactly 5MB as before). Object contents are unchanged. # Solution `S3StorageClient` keeps the plain client for listing, metadata and deletes, and uses the encryption client (`amazon-s3-encryption-client-java` 4.0.4) only for object content. - **Writes**: The encryption client encrypts a multipart upload as one AES-GCM stream. It needs the final part flagged as last and every other part to be a multiple of 16 bytes. `S3OutputStream` now holds back the final part until `close()`, and `flush()` uploads only a 16-byte-aligned prefix. - **Reads**: The JDK's AES-GCM doesn't release plaintext until a whole object is decrypted, which would hold each backup file in memory. Reads therefore use a BouncyCastle provider in delayed authentication mode. Writes stay on the JDK provider as it ran at least 20x faster than BouncyCastle in a local benchmark. - **Authentication**: The new `S3EncryptedInputStream` reads to the end of the object when the last plaintext byte is consumed, so failed tag checks surface as `IOException` and are not retried. - **Lengths**: Reported lengths exclude the 16-byte GCM tag, because Lucene's footer checks on restore depend on the plaintext length. - **Resuming**: The encryption client refuses ranged GETs, so dropped connections are resumed by re-reading the object from the start. - **No fallback**: Encrypted and unencrypted repositories refuse to read from eachother. - **Key storage**: Each file's wrapped data key is stored in the object's user metadata. Instruction files are disabled, so a missing key can be told apart from an unencrypted object. New dependencies in the module: - `amazon-s3-encryption-client-java`, with `kms` and `aws-json-protocol` - `netty-nio-client`, because the encryption client routes reads and writes through an `S3AsyncClient`. The SDK builds against Netty 4.1 and Solr pins 4.2, so `netty-codec-http2` is pinned to Solr's version and the empty `netty-codec` aggregator is excluded. - `saaj-impl`, from which the encryption client uses one MIME decoding class - `bcprov` Known limitations, also listed in the reference guide: - Parts of an encrypted upload are not retried individually, so a transient S3 error fails that file and with it the backup. - A single file is limited to about 64 GiB. - Restores are slower than unencrypted ones. 🤖 Code, tests, and parts of documentation were developed using Fable 5.2 in Claude Code and reviewed by me. I take full responsibililty for the work submitted. # Tests The module's suite passes with nightly tests enabled: 131 tests, none skipped, run against S3Mock in Docker. New tests: - `S3EncryptionTest` (19 tests): - round trips, including an object over 64 MiB and one larger than the heap - tampered ciphertext and a wrong key - each kind of repository refusing the other's objects - connection loss during a read - the KMS keyring with a mocked `KmsClient` - KMS and proxy configuration - `S3BackupRepositoryConfigTest` (9 tests): key file parsing and invalid option combinations. - `S3EncryptedBackupRepositoryTest` and `S3EncryptedIncrementalBackupTest` (`@Nightly`): the existing repository contract and incremental backup tests, run with encryption on. - `S3OutputStreamTest` and `S3OutputStreamMockitoTest`: new cases for part boundaries and `flush()`. - `S3ReadWriteTest.testNoNettyWithoutEncryption`: no Netty threads are started when encryption is off. # Checklist Please review the following and check all that apply: - [ ] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
