adam-spencer opened a new pull request, #5332:
URL: https://github.com/apache/solr/pull/5332

   https://issues.apache.org/jira/browse/SOLR-15680
   
   > This remains a draft while I:
   > * Complete a code review
   > * Complete a documentation review
   > * Test on real S3 with KMS (only used mocks so far)
   
   # Description
   
   Adds optional client-side encryption to `S3BackupRepository`, following the 
approach Houston Putman outlined on 
[SOLR-15680](https://issues.apache.org/jira/browse/SOLR-15680): `solr.xml` 
options for a key, and the official Amazon S3 Encryption Client. Backup files 
are encrypted on the Solr node before upload and decrypted after download.
   
   Encryption is off by default and is enabled by one of two mutually exclusive 
options:
   
   - `s3.encryption.kms.keyId`: an AWS KMS key, optionally with 
`s3.encryption.kms.region` and `s3.encryption.kms.endpoint`
   - `s3.encryption.aes.keyFile`: a file holding a base64-encoded 256-bit AES 
key
   
   The behaviour of `S3OutputStream.flush()` has been updated for all uploads, 
holding back the last 1-16 buffered bytes for the final part. This means the 
upload waits until the buffer is greater than 5MB (rather than exactly 5MB as 
before). Object contents are unchanged.
   
   # Solution
   
   `S3StorageClient` keeps the plain client for listing, metadata and deletes, 
and uses the encryption client (`amazon-s3-encryption-client-java` 4.0.4) only 
for object content.
   
   - **Writes**: The encryption client encrypts a multipart upload as one 
AES-GCM stream. It needs the final part flagged as last and every other part to 
be a multiple of 16 bytes. `S3OutputStream` now holds back the final part until 
`close()`, and `flush()` uploads only a 16-byte-aligned prefix.
   - **Reads**: The JDK's AES-GCM doesn't release plaintext until a whole 
object is decrypted, which would hold each backup file in memory. Reads 
therefore use a BouncyCastle provider in delayed authentication mode. Writes 
stay on the JDK provider as it ran at least 20x faster than BouncyCastle in a 
local benchmark.
   - **Authentication**: The new `S3EncryptedInputStream` reads to the end of 
the object when the last plaintext byte is consumed, so failed tag checks 
surface as `IOException` and are not retried.
   - **Lengths**: Reported lengths exclude the 16-byte GCM tag, because 
Lucene's footer checks on restore depend on the plaintext length.
   - **Resuming**: The encryption client refuses ranged GETs, so dropped 
connections are resumed by re-reading the object from the start.
   - **No fallback**: Encrypted and unencrypted repositories refuse to read 
from eachother.
   - **Key storage**: Each file's wrapped data key is stored in the object's 
user metadata. Instruction files are disabled, so a missing key can be told 
apart from an unencrypted object.
   
   New dependencies in the module:
   
   - `amazon-s3-encryption-client-java`, with `kms` and `aws-json-protocol`
   - `netty-nio-client`, because the encryption client routes reads and writes 
through an `S3AsyncClient`. The SDK builds against Netty 4.1 and Solr pins 4.2, 
so `netty-codec-http2` is pinned to Solr's version and the empty `netty-codec` 
aggregator is excluded.
   - `saaj-impl`, from which the encryption client uses one MIME decoding class
   - `bcprov`
   
   Known limitations, also listed in the reference guide:
   
   - Parts of an encrypted upload are not retried individually, so a transient 
S3 error fails that file and with it the backup.
   - A single file is limited to about 64 GiB.
   - Restores are slower than unencrypted ones.
   
   🤖 Code, tests, and parts of documentation were developed using Fable 5.2 in 
Claude Code and reviewed by me. I take full responsibililty for the work 
submitted.
   
   # Tests
   
   The module's suite passes with nightly tests enabled: 131 tests, none 
skipped, run against S3Mock in Docker.
   
   New tests:
   
   - `S3EncryptionTest` (19 tests):
     - round trips, including an object over 64 MiB and one larger than the heap
     - tampered ciphertext and a wrong key
     - each kind of repository refusing the other's objects
     - connection loss during a read
     - the KMS keyring with a mocked `KmsClient`
     - KMS and proxy configuration
   - `S3BackupRepositoryConfigTest` (9 tests): key file parsing and invalid 
option combinations.
   - `S3EncryptedBackupRepositoryTest` and `S3EncryptedIncrementalBackupTest` 
(`@Nightly`): the existing repository contract and incremental backup tests, 
run with encryption on.
   - `S3OutputStreamTest` and `S3OutputStreamMockitoTest`: new cases for part 
boundaries and `flush()`.
   - `S3ReadWriteTest.testNoNettyWithoutEncryption`: no Netty threads are 
started when encryption is off.
   
   # Checklist
   
   Please review the following and check all that apply:
   
   - [ ] I have reviewed the guidelines for [How to 
Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my 
code conforms to the standards described there to the best of my ability.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to