If decrypting before transmission we’ll want to require the cluster to have an internode authenticator setup, else a nefarious process could simply ask for data to be streamed to it to circumvent the encryption.
I agree it would be nice to have the nodes share the secret some way to avoid the additional performance penalties and potential security issues associated with decryption/encryption for transmission, but I don’t have any practical experience securely sharing secrets between nodes, so I’ll leave that decision to those who do. From: Jeremiah D Jordan <jerem...@datastax.com> Date: Monday, 15 November 2021 at 22:09 To: dev@cassandra.apache.org <dev@cassandra.apache.org> Subject: Re: Resurrection of CASSANDRA-9633 - SSTable encryption > On Nov 15, 2021, at 2:25 PM, Stefan Miklosovic > <stefan.mikloso...@instaclustr.com> wrote: > > On Mon, 15 Nov 2021 at 19:42, Jeremiah D Jordan > <jeremiah.jor...@gmail.com <mailto:jeremiah.jor...@gmail.com>> wrote: >> >> >> >>> On Nov 14, 2021, at 3:53 PM, Stefan Miklosovic >>> <stefan.mikloso...@instaclustr.com> wrote: >>> >>> Hey, >>> >>> there are two points we are not completely sure about. >>> >>> The first one is streaming. If there is a cluster of 5 nodes, each >>> node has its own unique encryption key. Hence, if a SSTable is stored >>> on a disk with the key for node 1 and this is streamed to node 2 - >>> which has a different key - it would not be able to decrypt that. Our >>> idea is to actually send data over the wire _decrypted_ however it >>> would be still secure if internode communication is done via TLS. Is >>> this approach good with you? >>> >> >> So would you fail startup if someone enabled sstable encryption but did not >> have TLS for internode communication? Another concern here is making sure >> zero copy streaming does not get triggered for this case. >> Have you considered having some way to distribute the keys to all nodes such >> that you don’t need to decrypt on the sending side? Having to do this will >> mean a lot more overhead for the sending side of a streaming operation. >> > > Yes, I would likely fail the start when encryption is enabled and > there is no TLS between nodes and yes, zero copy streaming should not > be triggered here. > > I have not considered that distribution. Honestly this seems like a > very complex setup. Due to the nature of Cassandra how one can easily > add / remove nodes, there would be a lot of hassle to distribute the > key of a new node to all other nodes somehow conveniently. I can't > even imagine how it would look in practice. One since the default implementation seems to be to just read the key out of a keystore, you could require having the same keystore on every node, and then it could just be the same key used by all nodes. Or there could actually be a different key for each node as the “key_alias” in the yaml, but every node has all the other nodes keys in its local keystore still. When adding a new node with a new key the operator would have to go add the new key to each existing keystore before adding the new node to the cluster. Another method could be to get the keys from a key server rather than a local file. But yes decrypt before streaming, stream encrypted using TLS, re-encrypt before writing to disk is an option. Just one that requires losing all the performance advantages gained when “do not decompress streaming” and then later “zero copy streaming” were implemented. > >>> The second question is about key rotation. If an operator needs to >>> roll the key because it was compromised or there is some policy around >>> that, we should be able to provide some way to rotate it. Our idea is >>> to write a tool (either a subcommand of nodetool (rewritesstables) >>> command or a completely standalone one in tools) which would take the >>> first, original key, the second, new key and dir with sstables as >>> input and it would literally took the data and it would rewrite it to >>> the second set of sstables which would be encrypted with the second >>> key. What do you think about this? >> >> I would rather suggest that “what key encrypted this” be part of the sstable >> metadata, and allow there to be multiple keys in the system. This way you >> can just add a new “current key” so new sstables use the new key, but >> existing sstables would use the old key. An operator could then trigger a >> “nodetool upgradesstables —all” to rewrite the existing sstables with the >> new “current key”. > > How would this key be added when Cassanra runs? Via JMX? So that means > JMX itself has to be secure to send that key to it or it would not > make sense. Or adding a new key would mean a node needs to go down and > we would somehow configure it on startup? But if a node needs to go > down first, we can just rewrite these tables while it is offline and > there is no need to do it that way. Given the current comments in the cassandra.yaml I would expect it to work like I just said: https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1328-L1332 <https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1328-L1332> # Enables encrypting data at-rest (on disk). Different key providers can be plugged in, but the default reads from # a JCE-style keystore. A single keystore can hold multiple keys, but the one referenced by # the "key_alias" is the only key that will be used for encrypt opertaions; previously used keys # can still (and should!) be in the keystore and will be used on decrypt operations # (to handle the case of key rotation). The intended method to rotate keys from that comments seems to be: 1. Stop the node 2. Adds a new key to the keystore 3. Changes the “key_alias” to reference the new key 4. Start the node All newly encrypted things would use the new key, existing things would use metadata to get at the previous key they were encrypted with. And then with such a method running “nodetool upgradesstables --all” or doing an offline “upgradesstables” would rewrite things with the new key. > > The tangential topic to this problem is if we are trying to do this > while the whole cluster is fully up and operational or we are ok to > bring that respective node down for a while to rewrite sstables for > it. I consider the "always up" scenario very complex to nail down > correctly and that is not a task I can do on my own with my current > understanding of Cassandra codebase. I think we need to be able to do this with the nodes up and running, per the procedure described above. It can take a long while to decrypt and re-encrypt 2 TB of data, so I would not want to keep a node down long enough to do that.
