Gabriel39 opened a new issue, #70740: URL: https://github.com/apache/doris/issues/70740
### Search before asking - [x] I searched existing issues. Related TLS/MySQL and Arrow Flight work does not describe the OSS private RPC transport design below. ### Description Provide built-in mutual TLS for the open-source private FE/BE RPC providers. The existing provider interfaces and TLS configuration do not by themselves provide a usable authenticated transport: the OSS clients use plaintext, and private listener activation is unsupported. This issue records the problem, implementation approach and design of #70693. The design baseline is commit [`f842904af888`](https://github.com/apache/doris/commit/f842904af888d3aa3fb8ba32fa8f8ca5e6eca0a5). Target release: Doris 5.0. The implementation PR is open at the time of writing; this issue does not claim a released feature. #### Motivation and the Iceberg PME connection Private RPC carries execution plans, scan ranges, split responses, status reports and data exchanges. Deployments that require confidentiality and authenticated peers need protection on these connections, independently of client-facing MySQL or HTTP TLS. The encrypted Iceberg read design makes the requirement particularly concrete. Parquet Modular Encryption protects file contents. The proposed FE-to-BE reader descriptor, `TFileEncryptionInfo`, contains an **unwrapped file data-encryption key (DEK)** and an AAD prefix. It never contains the KMS master key. If an unwrapped DEK were sent over plaintext RPC, someone able to observe that connection could recover the key and decrypt the corresponding file if they also obtained its ciphertext. KMS is a key-management service; DEK is the key used to encrypt file data. In an envelope-encryption design, an upper-level key managed by KMS protects lower-level keys. Once FE has resolved a usable file DEK, FE-to-BE distribution needs its own protection. Iceberg PME does not itself mandate Doris TLS: this requirement follows from the chosen key-distribution architecture. **Scope boundary:** #70693 supplies a transport prerequisite. It does not implement KMS integration, complete native PME reading, or enable encrypted Iceberg SQL. Existing admission checks remain in place until backend capability negotiation, reader support and end-to-end acceptance are demonstrated. Key redaction and statement-scoped key/cache ownership are separate work, including #70675. ### Solution #### 1. Goals and security boundary - Encrypt the supported internal RPC traffic and require certificates from both sides. - Authenticate the configured cluster CA trust domain; never retry a failed TLS connection as plaintext. - Preserve the registered target hostname across DNS resolution, connection reuse and reopen where hostname verification is supported. - Bound authentication IO, propagate configuration/handshake failures through existing RPC failure contracts, and release resources on partial initialization. - Preserve disabled-TLS behavior and the existing policy of external services and legacy custom providers. TLS protects network traffic. It does not protect against a compromised participating process, privileged host access, or plaintext keys captured from process memory. A dedicated cluster CA is required: possession of a certificate issued by that CA is a cluster RPC credential. Certificate authentication does not introduce a per-node or per-role authorization system. #### 2. Transport coverage | Connection | Implementation | Policy | | --- | --- | --- | | FE to local BE private services | Java gRPC client to native BRPC listener | Mutual TLS when private `brpc` is included | | Local BE to local BE private services | Native BRPC channels, including exchanges and internal streaming paths | Same internal TLS policy | | FE/BE internal Thrift clients and FE, BE backend/heartbeat listeners | Java/native blocking TLS transports | Mutual TLS when `thrift` is included; includes BE-to-FE lazy split fetches and callbacks | | Brokers and external RPC functions | Existing external-purpose clients | Retain the existing OSS plaintext policy | | Independent remote Doris cluster | Isolated remote FE/gRPC/BRPC clients | Retain its external policy; do not inherit local private credentials | | Public MySQL, HTTP, Arrow Flight, BDBJE and other cloud control-plane channels | Their own providers | Outside this transport implementation | Representative local flows: ```mermaid flowchart LR FE["Local FE"] -->|"gRPC to BRPC: mTLS"| BE1["Local BE 1"] FE -->|"Thrift heartbeat: mTLS"| BE1 BE1 -->|"Thrift split fetch and callbacks: mTLS"| FE BE1 <-->|"Private BRPC exchange: mTLS"| BE2["Local BE 2"] ``` #### 3. Reuse the provider boundaries On FE, `PrivateRpcTls` centralizes TLS context construction and configuration validation. `OssBackendServiceClientChannelProvider` configures authenticated gRPC channels. `OssThriftClientTransportProvider`, `TlsThriftSocket` and `TlsAcceptedThriftSocket` implement blocking Thrift client/server authentication, and `OssFeServerStarterProvider` wires the listener. On BE, `util/private_rpc_tls.{h,cpp}` centralizes native BRPC options and Thrift TLS socket factories. The OSS client provider and BE server starter install them on the existing channel/listener paths. The BRPC listener requires TLS rather than accepting both encrypted and plaintext connections. Business RPC methods and service processing remain unchanged. Default provider overloads preserve existing custom-provider dispatch unless those providers opt into the new endpoint-purpose handling. #### 4. Identity, DNS and connection lifetime Connect to the current resolved address while retaining the logical registered hostname as the certificate identity. The native Thrift wrapper separates the connect address from the peer identity; Java transports preserve the original peer name. Reopen authenticates again, including after a DNS address change. FE gRPC and the Java/native Thrift clients verify the target hostname/IP. The pinned native BRPC client authenticates the CA chain but does **not** implement per-node hostname or role allowlists. This difference is part of the trust contract and must not be presented as uniform hostname verification. Native socket ownership retains the TLS factory/context until the socket is destroyed. Failed initialization closes owned sockets and publishes no reusable client. Unstarted and stopped listeners remain safe to destroy; repeated cleanup is handled. #### 5. Bounded handshakes and failure behavior An unlimited application RPC read timeout must not permit an unauthenticated TCP peer to occupy a worker indefinitely. Thrift authentication therefore uses a positive connect/handshake bound, preserves a smaller caller timeout, and restores normal RPC read/write timeouts after authentication. Accepted-socket authentication runs on request workers. BE snapshots and validates `thrift_connect_timeout_seconds`, including overflow-safe conversion to milliseconds. FE uses a positive `thrift_rpc_connect_timeout_ms`. A later invalid mutable timeout or missing/malformed certificate must return a failed acquisition rather than escape a worker as an uncaught construction exception. Thrift cache creation returns the existing failed `Status`; BRPC setup failure returns the cache's existing null-client result. Failed clients/stubs are not cached, and a later valid configuration can recover. Diagnostics use generic classifications rather than embedding PEM input, passwords or provider parser details. #### 6. Separate internal and external endpoint policies Endpoint purpose is explicit; it is not inferred from whether the destination address happens to look local. Thrift pools, gRPC proxies and native BRPC caches keep internal and external clients separate, even when host/port values coincide. Reuse, reconnect and eviction preserve that separation. Remote Doris metadata/transaction RPCs, fragment prepare/start, fetch/cancel, write channels and partition callbacks retain the remote external policy. Both tablet writers propagate the optional `TOlapTableSink.remote_endpoint` flag. The exchange tablet partitioner derives the same policy from the existing explicit remote master-address metadata; cloning preserves it. The new Thrift flag uses optional field 26 and defaults to false. Its identifier was compared against complete master and 4.2/4.1/4.0/3.1 definitions. Existing partition metadata is reused for the exchange path, so it needs no additional wire field. Encrypted file-key descriptors are rejected before serialization to external plaintext fragment channels. Enabling local TLS alone is not evidence that an independent remote cluster can receive those keys safely. #### 7. Cross-cluster query graphs A direct remote-worker plan can make a remote BE initiate exchanges or status reports back into the local cluster. Changing FE's outbound channel cannot give that independent BE the local cluster's TLS identity. When OSS private BRPC or Thrift is enabled, remote Doris reads therefore use the existing default `use_arrow_flight=true` mode. External results are fetched into local workers, and local query exchanges stay inside the TLS cluster. Incompatible `use_arrow_flight=false` worker graphs, including remote-only graphs with reverse callbacks, are rejected before task dispatch with actionable guidance. Custom providers retain their own contract. This PR does not add TLS to the external Arrow Flight connection. Metadata access and remote tablet-write transport from local workers keep their existing external routing; this is not a claim that every remote write SQL scenario has passed end-to-end acceptance. #### 8. Configuration and activation Use the existing settings on every participating FE and BE, with a distinct node certificate/private key and the dedicated cluster CA: ```properties enable_tls=true tls_verify_mode=verify_fail_if_no_peer_cert tls_certificate_path=/path/to/node-certificate.pem tls_private_key_path=/path/to/node-private-key.pem tls_ca_certificate_path=/path/to/cluster-ca.pem tls_cert_refresh_interval_seconds=0 ``` For the private-RPC-only deployment described here: | Process | `tls_excluded_protocols` | Additional requirements | | --- | --- | --- | | FE | `http,mysql,arrowflight,bdbje` | Blocking Thrift server: `THREAD_POOL` or `SIMPLE`; positive `thrift_rpc_connect_timeout_ms` | | BE | `http,arrowflight` | Positive bounded `thrift_connect_timeout_seconds`; unencrypted PEM private key | `enable_tls=false` remains the default. Private TLS requires mutual verification and rejects missing identities, a nonzero certificate-refresh interval and a nonempty `tls_peer_cert_required_san_dns`. BE rejects a nonempty private-key password; FE retains its password setting. Certificates need client/server usage and SANs matching the addresses peers use. Deploy the code before activation, provision identities, then coordinate activation/restart across participating nodes. Plaintext and TLS-only peers cannot communicate; this implementation does not claim a rolling mixed plaintext/TLS transition. Identity/trust changes require restart. Keep the CA signing key off Doris nodes and restrict access to each node's private key. Rollback also requires coordinated configuration and restart. #### 9. Alternatives and design choices - **Make every existing pool use local TLS:** breaks plaintext brokers, RPC functions and remote Doris clusters. Explicit endpoint purpose and isolated caches preserve those contracts. - **Let every BE resolve wrapped keys through KMS:** could avoid FE-to-BE plaintext-DEK distribution, but changes KMS credentials, authorization and key ownership on BEs. It is a separate key-distribution design and would not itself encrypt query exchanges. - **Use an external encrypted network/tunnel:** a deployment can provide a different protected transport boundary. This PR implements native certificate authentication through the existing OSS provider interfaces and does not assume such infrastructure. #### 10. Validation and acceptance | Area | Required coverage | | --- | --- | | Authentication | Real gRPC/BRPC and Thrift listeners; trusted, absent and untrusted client certificates; plaintext rejection | | Identity | DNS-only certificates, wrong host rejection, cache reuse/reopen and changed resolved address | | Failure and cleanup | Invalid timeout, missing/malformed identity files, bounded idle-peer handshake, failed acquisition/recovery, partial listener startup and repeated shutdown | | Compatibility | Disabled TLS, legacy custom providers, broker heartbeat/filesystem consistency, external RPC validation/execution and same-address policy isolation | | Remote routing | Metadata and worker RPC paths, native write/stream transports, partition master retry, exchange partitioner and clone policy | | Default regression | Local TLS FE plus two TLS BEs against an independent plaintext FE/BE; ordered reads and directional broadcast/shuffle joins; direct-worker refusals | | Fixture lifecycle | Generated ephemeral identities/ports, sanitizer resources, owned/foreign PID handling, startup failure/cancellation, key deletion even if log archiving fails | The SQL suite is `rpc_p0/test_remote_doris_private_rpc_tls`, in `p0,nonConcurrent`, selected by the standard NonConcurrent profile. It has no test-specific enable flag or supplied remote endpoint/account settings. `privateRpcTlsPackageHome` optionally locates FE/BE packages and defaults to `DORIS_HOME`; both checkout `output/` and installed `fe/`/`be/` layouts work. Missing packages or failed startup fail the test rather than silently skip it. Diagnostic logs remain in the standard regression log tree; owned processes, keys and data are removed. Previously recorded local validation for the design baseline includes a full ASAN BE package build; 32 related ASAN native tests; nine helper tests executed by the default suite; six SQL result blocks and five expected refusals with zero skips; and trusted/missing/untrusted identity checks on seven private listeners. Both new exchange-partitioner tests fail with the policy initializer removed and pass with it, including a real plaintext `createPartition` RPC under enabled local TLS. The SQL replay used that fresh ASAN BE and a previously built FE package. Creating this issue did not rerun those tests. CI and release packaging/ABI validation remain separate requirements. Full remote auto-partition INSERT SELECT acceptance is not claimed: preexisting metadata/replica-ID write problems also reproduce with TLS disabled and are outside this transport change. Complete Iceberg PME SQL acceptance remains separate from the transport tests. #### References - Implementation: #70693; related key/cache ownership work: #70675. - [Operator guide at the design baseline](https://github.com/apache/doris/blob/f842904af888d3aa3fb8ba32fa8f8ca5e6eca0a5/docs/private-rpc-tls.md). - [Native TLS implementation](https://github.com/apache/doris/blob/f842904af888d3aa3fb8ba32fa8f8ca5e6eca0a5/be/src/util/private_rpc_tls.cpp) and [FE TLS implementation](https://github.com/apache/doris/blob/f842904af888d3aa3fb8ba32fa8f8ca5e6eca0a5/fe/fe-core/src/main/java/org/apache/doris/tls/PrivateRpcTls.java). - [File-key descriptor contract](https://github.com/apache/doris/blob/f842904af888d3aa3fb8ba32fa8f8ca5e6eca0a5/gensrc/thrift/PlanNodes.thrift#L327-L337). - [Default regression](https://github.com/apache/doris/blob/f842904af888d3aa3fb8ba32fa8f8ca5e6eca0a5/regression-test/suites/rpc_p0/test_remote_doris_private_rpc_tls.groovy). - Operator documentation: apache/doris-website#4666. ### Are you willing to submit PR? - [x] Yes. Implementation is proposed in #70693. ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
