As far as i know, In Apache Spark versions 3.5.x, SSL/TLS encryption for RPC (Remote Procedure Call) communication is not available. Instead, Spark utilizes an AES-based encryption mechanism for securing RPC connections. This method relies on a shared secret and requires RPC authentication to be enabled. While functional, this approach is considered less secure compared to SSL/TLS encryption.
downloads.apache.org <https://downloads.apache.org/spark/docs/3.5.0/security.html?utm_source=chatgpt.com> Starting from Spark version 4.0.0, SSL/TLS encryption for RPC communications has been introduced as a preferred method over the legacy AES-based encryption. SSL/TLS is standardized and offers enhanced security. To enable SSL/TLS encryption in Spark 4.0.0 and later, proper configuration of keys and certificates is required. It's important to note that SSL/TLS encryption is not automatically enabled even if spark.ssl.enabled is set; it must be explicitly configured for RPC communications. spark.apache.org <https://spark.apache.org/docs/preview/security.html?utm_source=chatgpt.com> Therefore, if you're using Spark 3.5.x, SSL/TLS encryption for RPC communication is not available. To utilize SSL/TLS encryption, consider upgrading to Spark version 4.0.0 or later and configuring the necessary SSL settings. The latest stable release of Apache Spark is version 3.5.4, released in December 2024. The Apache Spark community has been working on version 4.0, with preview releases made available for testing and feedback. The most recent preview, Spark 4.0.0-preview2, was announced on September 26, 2024. spark.apache.org <https://spark.apache.org/news/spark-4.0.0-preview2.html?utm_source=chatgpt.com> According to the project's planning discussions, the timeline for Spark 4.0 is as follows: - *January 15, 2025*: Code freeze and creation of the branch-4.0. - *February 1, 2025*: Feature freeze, focusing on bug fixes and stability improvements. - *February 15, 2025*: Initiation of the release candidate (RC) process. Given this schedule, the stable release of Spark 4.0 is anticipated in the first quarter of 2025. issues.apache.org <https://issues.apache.org/jira/browse/SPARK-44111?utm_source=chatgpt.com> Therefore, SSL/TLS encryption for RPC communication is expected to be available in the upcoming Spark 4.0 release. In the meantime, for versions 3.5.x, the AES-based encryption mechanism remains the standard method for securing RPC communications. El lun, 3 feb 2025 a las 22:57, Pablo Fernández (<pablof5...@gmail.com>) escribió: > Is SSL configuration being used for RPC communication in 3.5.* versions? > > I am setting up a standalone spark cluster and I am a little bit confused > in the security configuration. > > In the SSL-configuration docs > <https://spark.apache.org/docs/latest/security.html#ssl-configuration> it > says that the SSL settings will be use for all the supported communication > protocols. But this SSL thing is in the web UI section, which makes me > think that SSL is only for the web UI. > > I know that there are spark.network.* configurations that can enable > AES-based encryption for RPC connections, but I want to understand if > having ssl and network settings overwrite one or the other? Or if they are > meant to be used together for better protection? > > Thanks > -- Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links>
