Joey Pereira created FLINK-40404:
------------------------------------
Summary: Split Python wheel releases for connector (e.g. kafka)
Key: FLINK-40404
URL: https://issues.apache.org/jira/browse/FLINK-40404
Project: Flink
Issue Type: Improvement
Components: API / Python, Connectors / Kafka
Reporter: Joey Pereira
Currently, the {{pyflink}} wheel release combines bindings including the
separate https://github.com/apache/flink-connector-kafka repository.
When it comes to making changes, and iterating on them, having the bindings
released directly on {{pyflink}} is a bit of a burden as it makes it more
difficult to release them independently (plus the cross-repo release
management). This has been the case while iterating on various Kafka features
like the dynamic kafka source.
Instead, adopting an approach commonly used with separate releases and python's
{{extras}} would decouple these two things akin to how the Java module releases
already work.
For example, the {{pyflink}} release could have a {{kafka}} extras, with the
kafka connector releasing a package such as {{flink-connector-kafka-python}}.
Any necessary version pins between the two (e.g. which flink versions a given
python library requires, or vice versa) can be encoded in there.
Mind, this is in part self-induced as we've adopted separate release procedures
such as:
- We generate Python wheels with optional {{apache-flink-libraries}}, so as to
not actually bundle and utilizes wheel distributed JARs
- We install JARs from their maven releases directly
- Thus, the {{pyflink}} and it's transitive {{apache-flink-libraries}} will
have separate set of versions across Python and Java
---
Does this seem like a reasonable thing to do for the OSS?
(cc [~shuyichen] / [~suez1224])
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
