https://github.com/apache/pulsar/issues/17724
## Motivation
Pulsar C++ code base is in the same main repository for the Pulsar project.
While the decision was the right one at the time, there is a
considerable overhead
in keeping the C++ client in its current position.
### Issues with the current approach
The Pulsar repository has grown a lot in size and number of active developers.
1. The frequency of changes in various parts of the codebase has increased to a
point where the amount of resources dedicated to CI is very significant.
Every change in Java code will trigger the CI jobs for the C++
client and every
change in the C++ client will do the same.
During a CI job we are building the C++ client multiple times:
1. For C++ and Python client tests
2. To build Python wheels to be included in the pulsar Docker
images (for supporting
Pulsar functions)
2. The release process for Pulsar has become very complex and
requires building a
large number of binaries for C++ and Python clients. This has
become too much
of a burden during the course of a Pulsar release.
## Goal
Decouple the development of C++ and Python client libraries from the development
of the core components of Pulsar in Java.
## Changes
### Repositories
1. Move the C++ client code to a new repository
`github.com/apache/pulsar-client-c++`
2. Move the Python client code to a new repository
`github.com/apache/pulsar-client-python`
The change will be done without losing any history, extracting a
sub-directory into
a new Git repository.
```
git filter-repo --subdirectory-filter pulsar-client-cpp
```
### Release process
The release process will be split in multiple parts:
1. the main Pulsar release will only contain the Java parts (server
distribution
and Java client library)
2. The C++ client will have its own release schedule and versioning
3. The Python client will have its own release schedule and versioning
#### Versioning
Both C++ and Python clients will continue with their own individual versioning.
In order to not break anything or cause more confusion, we would need to use
a new version that is bigger than the current version (2.11.x).
The suggestion is to start the new releases for both C++ and Python from 3.0.0.
#### Existing branches
Existing branches of Pulsar, where the C++ client will still be in the same main
the repository and will be receiving bug fixes in their current location.
The different location of the new C++ code will make the cherry-picking process
slightly more painful in the short term, though it will even out in long term.
### Projects dependencies
#### C++/Python --> Pulsar
Both C++ and Python unit/integration tests are designed to run against
a standalone
instance of Pulsar broker. In the current form, they're using the `master` code
that is built to run the tests.
After the split, the unit tests will use a Docker image of Pulsar. We
can use a few
different images to test for compatibility
1. Latest stable (eg: 2.10.1)
2. Nightly (Pulsar Docker image published every day from master branch)
#### Pulsar --> Python
To create a Pulsar image, we are now building the Python client wheel
file and then
installing it at build time.
Instead, we are going to include a wheel file for a version of the Python client
that has been already released.
#### Python --> C++
The Python client library is just a wrapper on top of the C++ client.
Today these
are built together, with Python wrapper code residing in a
sub-directory of C++ client
code, and compiled using the same CMake build script.
By separating the Python client into a different repository, we are going to
depend on an already released version of the C++ client.
#### Automated documentation in the website
On the Pulsar website we are auto-generating C++ documentation with the Doxygen
tool and the Python one with Pdoc.
Instead of just fetching the main repo code, the website build job should be
also fetching the new repos to run the tooling.
--
Matteo Merli
<mme...@apache.org>
