BoYang created SPARK-57793:
------------------------------
Summary: [SparkConnect] Support url path in SparkConnect
connection string
Key: SPARK-57793
URL: https://issues.apache.org/jira/browse/SPARK-57793
Project: Spark
Issue Type: New Feature
Components: Connect
Affects Versions: 4.1.0
Reporter: BoYang
See [https://github.com/apache/spark/issues/56816]
Current Spark Connect uses connection string like {{{}sc://host1:15002{}}},
which does not support a path in the url, e.g. not support value like
{{{}sc://host1/path1:15002{}}}.
In Kubernetes, people can set up an Ingress to expose Spark Connect driver
endpoint by a matching url, e.g. {{http://host1/path1}} can be routed to the
driver endpoint. Need to update Spark Connect client to support this scenario.
This could be a Spark Client side change, e.g. in Python Spark Connect client,
connect to server based on the host name, and intercept GPRC call to avoid path
interfering GRPC protocol.
I am working on a prototype, also open to hear feedback / suggestions.
Also from [~viirya] in the Github issue discussion:
For an Ingress-routing-hint prototype, a gRPC metadata header (e.g. an x-
header the client sets from the URL path) is probably less invasive than
changing :path, and Ingress controllers can route on headers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
