Hi Pulsar Community, Here are the meeting notes from today's community meeting. We had some great discussions. I expect many of them will continue on the mailing list in the coming weeks.
Disclaimer: I am the primary author of these notes. I took the notes while participating in the meeting discussions. It is possible that I missed or misunderstood information. If something is misattributed or misrepresented, please send a correction to this list and consider updating the Google doc. Source google doc: https://docs.google.com/document/d/19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE Thanks, Michael 2022/03/31, (8:30 AM PST) - Attendees: - Matteo Merli - Aaron Williams - Christophe Bornet - Lari Hotari - Chris Bartholomew - Enrico Olivelli - Andrey Yegorov - Michael Marshall - Massimiliano Mirelli - Dave Fisher - Bharani Chadalavada - Nicolò Boschi - Discussions - Andrey: BK 4.15.0 RC0 vote is out, pulsar code changes will be needed -- Matteo: should we shoot for 4.15 or 4.14.5 for Pulsar 2.11. Andrey says 4.14.5 will be easier. Andrey says it would be helpful to post on both pulsar and bookkeeper mailing lists to get community consensus. Nicolò says he will send the email. -- Brief discussion about bookkeeper build maven vs gradle (Matteo and Andrey) - Lari: describes the recent CI updates. There is one optimization to ensure that the GitHub account doesn’t store too much data in Azure Cloud. There is also caching of 10 GB per repo (apache/pulsar), but it is an LRU cache, so it wouldn’t work well for all of our concurrent builds. We only cache the maven dependencies. Other artifacts are stored in Azure Blob Storage. Lari has found several issues since merging the original changes. Lari identified a runtime issue with certain versions of bookkeeper because of Rocks DB incompatibility (runtime errors are something like “missing method” or “no such method”). - Matteo: discussing RocksDB version nuances and making sure Pulsar Standalone works on Arm machines (particularly MacOS). Part of the question is whether or not we can do a major upgrade of RocksDB in a minor or patch release of Bookkeeper. Andrey asked whether or not we could upgrade RocksDB on older versions. Matteo thinks we shouldn’t upgrade RocksDB in existing stable versions of Pulsar (2.8 to 2.10). Andrey asks about users running with Arm and wanting to test out old versions, and Matteo mentioned that he would prefer not to introduce a possible regression in the production just to get it to work on MacOS. Andrey agreed. - Discuss moving Pulsar to gradle. Andrey expresses that the bookkeeper work has been hard to troubleshoot. Lari is happy to help, as a former gradle employee. Matteo details several of the reasons that a transition to gradle would be hard, but definitely not impossible. One such issue is the gradle nar nifi plugin. Lari mentions that he is now the maintainer of the plugin https://github.com/lhotari/gradle-nar-plugin. Further discussion would happen on the mailing list. - Michael: what is the advantage of moving to gradle? Matteo: caching is much better, so incremental builds are faster. It would improve developer experience. Lari: you should never use clean when running with gradle. It basically means something is wrong in your build. Matteo: clean in gradle still intelligently relies on a build cache. - Lari: Pulsar SQL. Asks about upgrading to Presto 334 to get it building correctly in the new CI. This would require that we run for Java 11. Matteo: should we upgrade minimum requirement to Java 11 or Java 17 for new releases (starting with 2.11). Lari mentions that Nicolò has already been working on moving to Java 17. Matteo thinks that we should upgrade server side to JVM 17. Michael: Because we’d also build the client with Java 17, should we still test the Java client against older JVM versions to check for compatibility? Matteo and Lari: we can add a test to do different runtime integration tests. Lari: we could possibly refactor tests to make it easy to run the same client/server tests with different JVM versions. - Michael: I will push along the release process discussion on the mailing list. Matteo: yes, please do that. - Michael: what is the specific reason that a Pulsar build would pick up “bad” jars for bookkeeper? Matteo: the bookkeeper build has native libraries that it uses for performance optimizations to run on linux, so if a user built the bookkeeper project, they would have the native MacOS deps. Lari: we could tell maven to use a different cache (not the ~/.m2 repo) cache to ensure that all dependencies are downloaded fresh and concurrent work on the machine would not affect the build. Michael to add that as a recommendation to the release process wiki page. - Dave: CFP is open for ApacheCon, and there is a Pulsar track that is open for tentatively 2 days.
