https://github.com/apache/pulsar/issues/17724
## Motivation
Pulsar C++ code base is in the same main repository for the Pulsar project.
While the decision was the right one at the time, there is a
considerable overhead
in keeping the C++ client in its current position.
### Issues with the current approach
The Pulsar repository has grown a lot in size and number of active developers.
1. The frequency of changes in various parts of the codebase has
increased to a point where the amount of resources dedicated to CI is
very significant.
Every change in Java code will trigger the CI jobs for the C++
client and every change in the C++ client will do the same.
During a CI job we are building the C++ client multiple times:
1. For C++ and Python client tests
2. To build Python wheels to be included in the pulsar Docker
images (for supporting Pulsar functions)
2. The release process for Pulsar has become very complex and
requires building a
large number of binaries for C++ and Python clients. This has
become too much of a burden during the course of a Pulsar release.
## Goal
Decouple the development of C++ and Python client libraries from the
development of the core components of Pulsar in Java.
## Changes
### Repositories
1. Move the C++ client code to a new repository
`github.com/apache/pulsar-client-c++`
2. Move the Python client code to a new repository
`github.com/apache/pulsar-client-python`
The change will be done without losing any history, extracting a
sub-directory into a new Git repository.
```
git filter-repo --subdirectory-filter pulsar-client-cpp
```
### Release process
The release process will be split in multiple parts:
1. the main Pulsar release will only contain the Java parts (server
distribution and Java client library)
2. The C++ client will have its own release schedule and versioning
3. The Python client will have its own release schedule and versioning
#### Versioning
Both C++ and Python clients will continue with their own individual versioning.
In order to not break anything or cause more confusion, we would need
to use a new version that is bigger than the current version (2.11.x).
The suggestion is to start the new releases for both C++ and Python from 3.0.0.
#### Existing branches
Existing branches of Pulsar, where the C++ client will still be in the
same main the repository and will be receiving bug fixes in their
current location.
The different location of the new C++ code will make the
cherry-picking process slightly more painful in the short term, though
it will even out in long term.
### Projects dependencies
#### C++/Python --> Pulsar
Both C++ and Python unit/integration tests are designed to run against
a standalone instance of Pulsar broker. In the current form, they're
using the `master` code that is built to run the tests.
After the split, the unit tests will use a Docker image of Pulsar. We
can use a few different images to test for compatibility
1. Latest stable (eg: 2.10.1)
2. Nightly (Pulsar Docker image published every day from master branch)
#### Pulsar --> Python
To create a Pulsar image, we are now building the Python client wheel
file and then installing it at build time.
Instead, we are going to include a wheel file for a version of the
Python client that has been already released.
#### Python --> C++
The Python client library is just a wrapper on top of the C++ client.
Today these are built together, with Python wrapper code residing in a
sub-directory of C++ client code, and compiled using the same CMake
build script.
By separating the Python client into a different repository, we are
going to depend on an already released version of the C++ client.
#### Automated documentation in the website
On the Pulsar website we are auto-generating C++ documentation with
the Doxygen tool and the Python one with Pdoc.
Instead of just fetching the main repo code, the website build job
should be also fetching the new repos to run the tooling.
--
Matteo Merli
<mme...@apache.org>
