It seems there are no objections to this PIP. I am proceeding to create 3 new repos to kick off the effort.
https://github.com/apache/pulsar-connectors https://github.com/apache/pulsar-adapters https://github.com/apache/pulsar-presto - Sijie On Sat, Apr 11, 2020 at 10:35 AM Sanjeev Kulkarni <sanjee...@gmail.com> wrote: > +1 > > On Thu, Apr 9, 2020 at 11:54 PM Sijie Guo <guosi...@gmail.com> wrote: > > > Sanjeev, Matteo, > > > > > Will these > > > additional repositories build against the latest pulsar release? Or > will > > > they build against say a nightly build of master? > > > > > testing dependents modules against the latest version, > > > > Here is my original proposal - "The three new repositories will run their > > Github Action based CI tests. The CI tests will use the latest build from > > the main pulsar repo. The pulsarbot can be enhanced to add a common > action > > to install Pulsar client dependencies from main repo before running any > > actions." > > > > I was proposing adding a common github action which fetches the latest > code > > from main repo and build the libraries and install them before running > > other actions. > > So it ensures all the three repos in-sync with main repo. This approach > is > > easy to start and we can reuse existing Github workflows when those > > components are moved to separated repos. If the built time for those new > > repos is still a concern, we can find a way to improve later. > > > > Let me know what you think. > > > > - Sijie > > > > > > On Thu, Apr 9, 2020 at 9:42 PM Matteo Merli <matteo.me...@gmail.com> > > wrote: > > > > > +1 > > > > > > I think there are still some unknowns, but they will only be > > > discovered once the effort is started. > > > > > > Regarding, testing dependents modules against the latest version, we > > > could rely on publishing snapshot Jars (which we already do on a daily > > > basis) and snapshot Docker images. It's not perfect, but it would be > > > easier to setup until we can get a more rigorous daily CI pipeline > > > that cascades the artifacts across builds. In any case, we should have > > > a ~daily build against the "latest" master in order to avoid > > > regressions. > > > > > > > > > -- > > > Matteo Merli > > > <matteo.me...@gmail.com> > > > > > > On Thu, Apr 9, 2020 at 1:11 PM Sanjeev Kulkarni <sanjee...@gmail.com> > > > wrote: > > > > > > > > Thanks Sijie for writing this proposal. > > > > I whole-heartedly agree with this proposal. Not only does it reduce > > > Pulsar > > > > build time and make our lives as pulsar developers simple, it will > also > > > > make connector/adaptor contributions less frictionless for > > contributors. > > > No > > > > longer will developers need to know about or care to compile the core > > > > pulsar. This could result in more contributions from non-core > > developers. > > > > I took a look at the document and it doesn't address one point. Will > > > these > > > > additional repositories build against the latest pulsar release? Or > > will > > > > they build against say a nightly build of master? > > > > Thanks! > > > > > > > > On Thu, Apr 9, 2020 at 12:57 PM Sijie Guo <si...@apache.org> wrote: > > > > > > > > > Hi all, > > > > > > > > > > I chatted with some committers a few months ago about the build > time > > > of the > > > > > current main repo. I'd like to raise a discussion of moving > > connectors, > > > > > adapters, and presto-connector to separate repositories so that we > > can > > > > > reduce the build time of the main repo and improve the review > process > > > for > > > > > individual components. > > > > > > > > > > I wrote the proposal in PIP-62. Please review it and let me know > what > > > you > > > > > think. > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/wiki/PIP-62:-Move-connectors,-adapters-and-Pulsar-Presto-to-separate-repositories > > > > > > > > > > Thanks, > > > > > Sijie > > > > > > > > > > >
