Hi Pulsar Community, Here are the meeting notes from last Thursday's community meeting. Thanks to all who participated!
Disclaimer: If something is misattributed or misrepresented, please send a correction to this list. Source google doc: https://docs.google.com/document/d/19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE Thanks, Michael 2022/05/26, (8:30 AM PST) - Attendees: - Matteo Merli - Michael Marshall - Heesung Sohn - Dave Fisher - Andrey Yegorov - Chris Bartholomew - Hang Chen - Discussions - Matteo: the thread from Lari on the mailing list. One of the points from Lari’s email is about full consistency, and that would certainly be nice, but the reality is that it is pretty much impossible to achieve that with a distributed cache because it is based on the assumption that each broker can do its own caching. Michael: I believe there is supposed to be an owner for each piece of metadata, so there isn’t the same caching. Matteo: for example, create a namespace on one broker and then call a get operation on another broker, it might not know about the new namespace. This would only work without caching by going back through zookeeper. Dave: I’ve seen this behavior with OMB. What if you return right away but signal eventual consistency? Matteo: either you wait for all brokers to respond, or you don’t. If you do, you have an availability problem. Dave: what if you have a guarantee of about a second? Matteo: everything should be distributed within a second, but there are always edge cases. Michael: I am not able to represent the full argument, so we’ll need to defer to another meeting to discuss further with Lari. Matteo: acknowledges that there is cost to the persistent watches, but believes it is better in the new implementation than having many smaller watches, though it’d be worth doing additional testing to verify the assumption. - Matteo: Heesung is on the call and he will be looking at improving the load manager. Heesung: I’m starting by documenting the current flow, including sequence diagrams. Michael: that sounds great! - PIPs - Michael: Asks for review on PIP 167 https://github.com/apache/pulsar/issues/15597. Discussed with Matteo. There was confusion about how the current subscription level authorization actually works because it is not additive, but rather negative. Matteo proposed changing the current implementation to make subscription permission additive and add a configuration to allow for users that rely on the current implementation to have backwards compatibility. Michael: that would certainly work, too. The main goal is to improve the subscription level auth, and it’d certainly be easier to understand in an additive model. Michael: also note that the current feature is essentially undocumented. Matteo: that likely means that only a handful of users are actually using the feature, which makes it even more reasonable to take the time to change it to the “right” implementation. Matteo to review the PIP, discuss the historical implementation with Rajan, and then respond on the mailing list.
