Am I reading this correctly that we publish 7 different artifacts just
for python?
What does the release matrix look like?
On 3/16/2021 3:45 AM, Dian Fu wrote:
Hi Xingbo,
Thanks a lot for bringing up this discussion. Actually the size limit already
becomes an issue during releasing 1.11.3 and 1.12.1. It blocks us to publish
PyFlink packages to PyPI during the release as there is no enough space left
(PS: already published the packages after increasing the size limit).
Considering that the total package size are about 1.5GB (220MB * 7) for each
release, it makes sense to split the PyFlink package. It could reduce the total
package size to about 250MB (3MB * 7 + 220 MB) for each release. We don’t need
to increase the size limit any more in the next few years as currently we still
have about 7.5 GB space left.
So +1 from my side.
Regards,
Dian
2021年3月12日 下午2:30，Xingbo Huang <hxbks...@gmail.com> 写道：
Hi everyone,
Since release-1.11, pyflink has introduced cython support and we will
release 7 packages (for different platforms and Python versions) to PyPI
for each release and the size of each package is more than 200MB as we need
to bundle the jar files into the package. The entire project space in PyPI
grows very fast, and we need to apply to PyPI for more project space
frequently. Please refer to [https://github.com/pypa/pypi-support/issues/831]
for more details.
The root cause to this problem is that we bundled the jar files in each
package. This is actually unnecessary if we could extract the jar files
into a separate package which is dedicated to hold the jar files.
I’d like to propose to split the pyflink package into two packages: the
original apache-flink and apache-flink-libraries (Any suggestions for the
name?). The package apache-flink-libraries only contains jar files and
there is only one apache-flink-libraries package for each release. The
package apache-flink depends on apache-flink-libraries and for users, they
still only need to install apache-flink and there is nothing different from
before. We still need to release multiple wheel packages of apache-flink.
However, the size will be very small as it doesn't contain the jar files
any more.
Looking forward to your feedback.
Best,
Xingbo
