Hi Tison & Matteo, I noticed a comment here: https://github.com/apache/pulsar/pull/17733#discussion_r975552686
Currently the Python Functions relies on the Python client. I see there are still some Python Function examples in pulsar-functions/python-examples directory. How should we deal with these examples? And should we still Include the Python Functions in the Docker images? Thanks, Yunze > On Sep 20, 2022, at 23:40, tison <wander4...@gmail.com> wrote: > > One more thing to mention here: > > Currently, Pulsar Docker Image bundles C++ client and Python client, and > from my perspective, the image is mainly used as a server, perhaps we can > remove these clients from bundling. > > Best, > tison. > > > Yunze Xu <y...@streamnative.io.invalid> 于2022年9月20日周二 16:50写道： > >> Hi Tison, >> >> Sorry I just missed that. Thanks for your reminder. >> >> Thanks, >> Yunze >> >> >> >> >>> On Sep 20, 2022, at 16:35, tison <wander4...@gmail.com> wrote: >>> >>> Hi Yunze, >>> >>>> Just wondering if there is a way to retain the git history in the >>> pulsar-client-cpp directory? >>> >>> Matteo's proposal already write: >>> >>>> git filter-repo --subdirectory-filter pulsar-client-cpp >>> >>> So you will retain the git history. >>> >>> Best, >>> tison. >>> >>> >>> Yunze Xu <y...@streamnative.io.invalid> 于2022年9月20日周二 16:27写道： >>> >>>> LGTM. I also listed the related files outside the pulsar-client-cpp >>>> directory recently: >>>> >>>> - pulsar-common/src/main/proto/PulsarApi.proto: the Pulsar binary >>>> proto file >>>> - src/gen-pulsar-version-macro.py: generate the internal version info >>>> - pulsar-client/src/test/proto/*.proto: test the protobuf native >>>> schema feature >>>> >>>> It would not be a complicated job for that. Just wondering if there is >>>> a way to retain the git history in the pulsar-client-cpp directory? >>>> >>>> Thanks, >>>> Yunze >>>> >>>> >>>> >>>> >>>>> On Sep 20, 2022, at 07:25, Matteo Merli <mme...@apache.org> wrote: >>>>> >>>>> https://github.com/apache/pulsar/issues/17724 >>>>> >>>>> >>>>> >>>>> ## Motivation >>>>> >>>>> Pulsar C++ code base is in the same main repository for the Pulsar >>>> project. >>>>> >>>>> While the decision was the right one at the time, there is a >>>>> considerable overhead >>>>> in keeping the C++ client in its current position. >>>>> >>>>> ### Issues with the current approach >>>>> >>>>> The Pulsar repository has grown a lot in size and number of active >>>> developers. >>>>> >>>>> 1. The frequency of changes in various parts of the codebase has >>>> increased to a >>>>> point where the amount of resources dedicated to CI is very >>>> significant. >>>>> >>>>> Every change in Java code will trigger the CI jobs for the C++ >>>>> client and every >>>>> change in the C++ client will do the same. >>>>> >>>>> During a CI job we are building the C++ client multiple times: >>>>> 1. For C++ and Python client tests >>>>> 2. To build Python wheels to be included in the pulsar Docker >>>>> images (for supporting >>>>> Pulsar functions) >>>>> >>>>> 2. The release process for Pulsar has become very complex and >>>>> requires building a >>>>> large number of binaries for C++ and Python clients. This has >>>>> become too much >>>>> of a burden during the course of a Pulsar release. >>>>> >>>>> >>>>> ## Goal >>>>> >>>>> Decouple the development of C++ and Python client libraries from the >>>> development >>>>> of the core components of Pulsar in Java. >>>>> >>>>> >>>>> ## Changes >>>>> >>>>> ### Repositories >>>>> >>>>> 1. Move the C++ client code to a new repository >>>>> `github.com/apache/pulsar-client-c++` >> <http://github.com/apache/pulsar-client-c++> >>>> <http://github.com/apache/pulsar-client-c++> >>>>> 2. Move the Python client code to a new repository >>>>> `github.com/apache/pulsar-client-python` >> <http://github.com/apache/pulsar-client-python> >>>> <http://github.com/apache/pulsar-client-python> >>>>> >>>>> The change will be done without losing any history, extracting a >>>>> sub-directory into >>>>> a new Git repository. >>>>> >>>>> ``` >>>>> git filter-repo --subdirectory-filter pulsar-client-cpp >>>>> ``` >>>>> >>>>> ### Release process >>>>> >>>>> The release process will be split in multiple parts: >>>>> >>>>> 1. the main Pulsar release will only contain the Java parts (server >>>>> distribution >>>>> and Java client library) >>>>> 2. The C++ client will have its own release schedule and versioning >>>>> 3. The Python client will have its own release schedule and versioning >>>>> >>>>> #### Versioning >>>>> >>>>> Both C++ and Python clients will continue with their own individual >>>> versioning. >>>>> >>>>> In order to not break anything or cause more confusion, we would need >> to >>>> use >>>>> a new version that is bigger than the current version (2.11.x). >>>>> >>>>> The suggestion is to start the new releases for both C++ and Python >> from >>>> 3.0.0. >>>>> >>>>> >>>>> #### Existing branches >>>>> >>>>> Existing branches of Pulsar, where the C++ client will still be in the >>>> same main >>>>> the repository and will be receiving bug fixes in their current >> location. >>>>> >>>>> The different location of the new C++ code will make the cherry-picking >>>> process >>>>> slightly more painful in the short term, though it will even out in >> long >>>> term. >>>>> >>>>> >>>>> ### Projects dependencies >>>>> >>>>> #### C++/Python --> Pulsar >>>>> >>>>> Both C++ and Python unit/integration tests are designed to run against >>>>> a standalone >>>>> instance of Pulsar broker. In the current form, they're using the >>>> `master` code >>>>> that is built to run the tests. >>>>> >>>>> After the split, the unit tests will use a Docker image of Pulsar. We >>>>> can use a few >>>>> different images to test for compatibility >>>>> 1. Latest stable (eg: 2.10.1) >>>>> 2. Nightly (Pulsar Docker image published every day from master branch) >>>>> >>>>> #### Pulsar --> Python >>>>> >>>>> To create a Pulsar image, we are now building the Python client wheel >>>>> file and then >>>>> installing it at build time. >>>>> >>>>> Instead, we are going to include a wheel file for a version of the >>>> Python client >>>>> that has been already released. >>>>> >>>>> #### Python --> C++ >>>>> >>>>> The Python client library is just a wrapper on top of the C++ client. >>>>> Today these >>>>> are built together, with Python wrapper code residing in a >>>>> sub-directory of C++ client >>>>> code, and compiled using the same CMake build script. >>>>> >>>>> By separating the Python client into a different repository, we are >>>> going to >>>>> depend on an already released version of the C++ client. >>>>> >>>>> >>>>> #### Automated documentation in the website >>>>> >>>>> On the Pulsar website we are auto-generating C++ documentation with the >>>> Doxygen >>>>> tool and the Python one with Pdoc. >>>>> >>>>> Instead of just fetching the main repo code, the website build job >>>> should be >>>>> also fetching the new repos to run the tooling. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Matteo Merli >>>>> <mme...@apache.org>
