Yes, the size of .whl file in PyFlink will also be about 3MB if we split the package. Currently the package is big because we bundled the jar files in it.
> 2021年3月16日 下午8:13，Chesnay Schepler <ches...@apache.org> 写道： > > key difference being that the beam .whl files are 3mb large, aka 60x smaller. > > On 3/16/2021 1:06 PM, Dian Fu wrote: >> Hi Chesnay, >> >> We will publish binary packages separately for: >> 1) Python 3.5 / 3.6 / 3.7 / 3.8 (since 1.12) separately >> 2) Linux / Mac separately >> >> Besides, there is also a source package which is used when none of the above >> binary packages is usable, e.g. for Window users. >> >> PS: publishing multiple binary packages is very common in Python world, e.g. >> Beam published 22 packages in 2.28, Pandas published 16 packages in 1.2.3 >> [2]. We could also publishing more packages if we splitting the packages as >> the cost of adding another package will be very small. >> >> Regards, >> Dian >> >> [1] https://pypi.org/project/apache-beam/#files >> <https://pypi.org/project/apache-beam/#files> >> <https://pypi.org/project/apache-beam/#files >> <https://pypi.org/project/apache-beam/#files>> >> [2] https://pypi.org/project/pandas/#files >> >> >> Hi Xintong, >> >> Yes, you are right that there is 9 packages in 1.12 as we added Python 3.8 >> support in 1.12. >> >> Regards, >> Dian >> >>> 2021年3月16日 下午7:45，Xintong Song <tonysong...@gmail.com> 写道： >>> >>> And it's not only uploaded to PyPI, but the ASF mirrors as well. >>> >>> https://dist.apache.org/repos/dist/release/flink/flink-1.12.2/python/ >>> >>> Thank you~ >>> >>> Xintong Song >>> >>> >>> >>> On Tue, Mar 16, 2021 at 7:41 PM Xintong Song <tonysong...@gmail.com> wrote: >>> >>>> Actually, I think it's 9 packages, not 7. >>>> >>>> Check here for the 1.12.2 packages. >>>> https://pypi.org/project/apache-flink/#files >>>> >>>> Thank you~ >>>> >>>> Xintong Song >>>> >>>> >>>> >>>> On Tue, Mar 16, 2021 at 7:08 PM Chesnay Schepler <ches...@apache.org> >>>> wrote: >>>> >>>>> Am I reading this correctly that we publish 7 different artifacts just >>>>> for python? >>>>> What does the release matrix look like? >>>>> >>>>> On 3/16/2021 3:45 AM, Dian Fu wrote: >>>>>> Hi Xingbo, >>>>>> >>>>>> >>>>>> Thanks a lot for bringing up this discussion. Actually the size limit >>>>> already becomes an issue during releasing 1.11.3 and 1.12.1. It blocks us >>>>> to publish PyFlink packages to PyPI during the release as there is no >>>>> enough space left (PS: already published the packages after increasing the >>>>> size limit). >>>>>> >>>>>> Considering that the total package size are about 1.5GB (220MB * 7) for >>>>> each release, it makes sense to split the PyFlink package. It could reduce >>>>> the total package size to about 250MB (3MB * 7 + 220 MB) for each release. >>>>> We don’t need to increase the size limit any more in the next few years as >>>>> currently we still have about 7.5 GB space left. >>>>>> So +1 from my side. >>>>>> >>>>>> Regards, >>>>>> Dian >>>>>> >>>>>>> 2021年3月12日 下午2:30，Xingbo Huang <hxbks...@gmail.com> 写道： >>>>>>> >>>>>>> Hi everyone, >>>>>>> >>>>>>> Since release-1.11, pyflink has introduced cython support and we will >>>>>>> release 7 packages (for different platforms and Python versions) to >>>>> PyPI >>>>>>> for each release and the size of each package is more than 200MB as we >>>>> need >>>>>>> to bundle the jar files into the package. The entire project space in >>>>> PyPI >>>>>>> grows very fast, and we need to apply to PyPI for more project space >>>>>>> frequently. Please refer to [ >>>>> https://github.com/pypa/pypi-support/issues/831] >>>>>>> for more details. >>>>>>> >>>>>>> The root cause to this problem is that we bundled the jar files in each >>>>>>> package. This is actually unnecessary if we could extract the jar files >>>>>>> into a separate package which is dedicated to hold the jar files. >>>>>>> >>>>>>> I’d like to propose to split the pyflink package into two packages: the >>>>>>> original apache-flink and apache-flink-libraries (Any suggestions for >>>>> the >>>>>>> name?). The package apache-flink-libraries only contains jar files and >>>>>>> there is only one apache-flink-libraries package for each release. The >>>>>>> package apache-flink depends on apache-flink-libraries and for users, >>>>> they >>>>>>> still only need to install apache-flink and there is nothing different >>>>> from >>>>>>> before. We still need to release multiple wheel packages of >>>>> apache-flink. >>>>>>> However, the size will be very small as it doesn't contain the jar >>>>> files >>>>>>> any more. >>>>>>> >>>>>>> Looking forward to your feedback. >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> Xingbo
