To enable a faster development cadence I am proposing the following steps to reorg the current pulsar monorepo and test harness. Here is proposed list of milestones.
Milestone 1 Move doc artifacts to an independent repo. We can copy move the site2 folder out of the current repo, we need to communicate with unmerged documentation related pr authors to target the new repository. There is a jenkins builds script that needs to be rewritten and release process that needs to be updated. We can call the new repo apache/pulsar-docs Milestone 2 Move connectors, external integration to another repo. We need to set up we can call the new repo apache/pulsar-ext. This should be most of the connectors, and other integration jars just such storm , flink and spark sources and sinks. The integration test suite needs to be split aslo. This requires new release aggregator script to be be built, that can make a release with artifacts generated from multiple repos. Milestone 3 Move C++/python/go artifacts to an external repo. If the above milestones are met we can also move C++/python/go packages to another repo We can call that apache/pulsar-native-clients. We will just need to update the release aggregator. Milestone 4 Rebuild the broker unit test suite. The current broker test suite is quite brittle an unorganized. about 20 percent of the tests can move to the integration suite about 20 percent will stay about 60 percent need to rewritten to use a common that can provide test isolation and parallelization via namespaces.
