Haiyang Sun created SPARK-59435:
-----------------------------------
Summary: Qualify UDF worker protobuf import paths
Key: SPARK-59435
URL: https://issues.apache.org/jira/browse/SPARK-59435
Project: Spark
Issue Type: Sub-task
Components: UDF
Affects Versions: 4.4.0
Reporter: Haiyang Sun
The UDF worker protobuf definitions currently use bare imports such as
`common.proto` and `udf_message.proto`. These filenames are generic and do not
identify the UDF worker protocol namespace.
This works while the protobuf definitions are compiled in isolation, but it can
become ambiguous when Spark protobuf sources are collected, indexed, or
compiled alongside other protocol definitions containing similarly named files.
The import paths should therefore be stable and globally meaningful.
Proposed approach:
Move the UDF worker protobuf definitions under a namespaced directory beneath
the protobuf source root and update intra-proto imports to use fully qualified
paths:
- `udf/worker/proto/common.proto`
- `udf/worker/proto/udf_message.proto`
- `udf/worker/proto/udf_service.proto`
- `udf/worker/proto/worker_spec.proto`
For example, replace `import "common.proto";` with
`import "udf/worker/proto/common.proto";`.
This preserves the protocol schema, generated Java package, and wire format.
It does change protobuf descriptor filenames, so the implementation should
verify that no Spark code depends on the old descriptor names.
Testing:
- Compile the UDF worker protobuf module.
- Compile the UDF worker gRPC module, which consumes the service definition.
- Confirm generated sources and existing UDF worker tests continue to pass.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
