2020-07-15 10:15:51 UTC - Prashanth YV: @Prashanth YV has joined the channel ---- 2020-07-15 12:33:45 UTC - andrius.antipovas: Hi, why Pulsar SQL cannot query the latest message? ---- 2020-07-15 16:14:22 UTC - Joshua Eric: See more information here on a solution: <https://github.com/apache/pulsar/issues/3828> ---- 2020-07-15 17:19:20 UTC - Raman Gupta: > Does pulsar provide a similar interface to Kafka’s rebalance listener to allow applications to know when a particular consumer has it’s key range that it listens to changed? I don't believe so but those better knowledgeable here can correct me if I'm wrong. What is the use case for this? There might be a different way to achieve the same ends with Pulsar. ---- 2020-07-15 17:21:16 UTC - Pedro Cardoso: The use-case is to propagate topology changes of a messaging system down to a stateful application ---- 2020-07-15 17:21:52 UTC - Raman Gupta: Sure but for what reason? Why are you coupling the app to the messaging system's topology? ---- 2020-07-15 17:23:55 UTC - Pedro Cardoso: It’s a distributed streaming engine, where I partition a set of jobs based on (in pulsar nomenclature) key ranges. These jobs are stateful, if the topology changes (in Kafka it would be topic partitions). I.e: a pulsar consumer running in one of the streaming nodes has it’a assigned keys modified, then I need to change the state of the streaming node ---- 2020-07-15 17:24:26 UTC - Pedro Cardoso: Key range changes may occur if the system needs to scale in/out or if a given node crashes ---- 2020-07-15 17:26:02 UTC - Raman Gupta: Ok. Looks like there is already an issue for this: <https://github.com/apache/pulsar/issues/6555>. +1 : Pedro Cardoso ---- 2020-07-15 20:29:30 UTC - Pedro Cardoso: Good to know! Another very useful thing is to be able to control which keys are assigned to which consumers. To implement kafka’s stickyAssignor-like logic. Despite the disadvantages from Kafka, the rebalancing/partition assignment logic is pretty useful ---- 2020-07-15 20:38:43 UTC - Raman Gupta: Sticky assignor was added in Kafka mostly because rebalancing was expensive and it was seen as a way to reduce the cost of it (ref Motivation in <https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy>). I'm not sure Pulsar really needs all that complexity without the problems introduced by Kafka's partition assignment design. If you need particular consumers to handle particular subsets of messages, they can either consume/ignore and ack the uninteresting messages, or the messages could be split into different topics via functions. ---- 2020-07-15 20:40:31 UTC - Pedro Cardoso: Ignoring uninteresting messages does not scale and has a high network cost, essentially broadcasting a large of messages multiples time until some consumer takes them, which also adds potential out of order consistency issues ---- 2020-07-15 20:41:47 UTC - Pedro Cardoso: Splitting different topics via functions. I do not know how it would work, but I’m not that proficient with pulsar. Either way, it would have to be done in a dynamic manner. Recall that the use-case is scaling in/out and handling downstream stateful application crashes ---- 2020-07-15 20:42:46 UTC - Pedro Cardoso: If pulsar does not have these functionalities in mind, that’s fine. It is simply unsuitable for my particular use-case at this point in time with the architecture we have in place. ---- 2020-07-15 20:43:14 UTC - Pedro Cardoso: As pulsar evolves, if it works on these types of issues then naturally I’ll reevaluate ---- 2020-07-15 20:44:12 UTC - Pedro Cardoso: Because pulsar’s design seems better than Kafka. If these features are incompatible with the design so be it. ---- 2020-07-15 20:44:45 UTC - Raman Gupta: But how does sticky assignor solve that problem? It jus tries to minimize partition assignment movements, right? How does it allow you to control the key assignments? ---- 2020-07-15 20:47:14 UTC - Pedro Cardoso: Because it allows me to specify to which consumer a given partition/key range will be assign to based on some state in the node where the consumer is running ---- 2020-07-15 20:48:14 UTC - Pedro Cardoso: Why do I want this? Because the cost of recomputing something from scratch due to not having state in the node beforehand is too large for my streaming SLAs ---- 2020-07-15 20:49:34 UTC - Raman Gupta: Are we talking about the same thing? I didn't think the StickyAssignor from Kafka allowed you to specify the assignments. Perhaps you mean a custom implementation of `PartitionAssignor`? ---- 2020-07-15 20:50:27 UTC - Raman Gupta: I think the equivalent of that in Pulsar would be a message router to publish into a particular partition of a partitioned topic, based on your dynamic logic: <https://pulsar.apache.org/docs/en/concepts-messaging/#routing-modes> ---- 2020-07-15 20:51:58 UTC - Pedro Cardoso: Yes, my bad for not explaining clearly. It would be an application specific custom built partition assignor that works like Kafka’s StickyAssignor ---- 2020-07-15 20:52:13 UTC - Raman Gupta: Ok yeah, that makes more sense. ---- 2020-07-15 20:52:20 UTC - Raman Gupta: Yeah look into that link I just sent +1 : Pedro Cardoso ---- 2020-07-15 20:52:31 UTC - Pedro Cardoso: I apologize for the confusion :sweat_smile: ok_hand : Raman Gupta ---- 2020-07-15 20:57:07 UTC - Pedro Cardoso: Are there ordering guarantees for CustomPartitions based on key? The documentation does not mention the CustomPartition in the ordering guarantee section. Is it left to the actual implementation to provide any sort of guarantee? ---- 2020-07-15 20:58:56 UTC - Raman Gupta: I believe so. That's how the built-in routers work: <https://pulsar.apache.org/docs/en/concepts-messaging/#ordering-guarantee> ---- 2020-07-15 21:02:43 UTC - Raman Gupta: Ah, see also this which seems to be what you need : <https://github.com/apache/pulsar/issues/4169> +1 : Pedro Cardoso ---- 2020-07-16 01:30:25 UTC - Ali Ahmed: all go based test are failing something may have changes in github <https://github.com/apache/pulsar/pull/7549/checks?check_run_id=875769465> ---- 2020-07-16 03:30:34 UTC - rohit keshwani: @rohit keshwani has joined the channel ---- 2020-07-16 03:55:13 UTC - Raghav: @Raghav has joined the channel ----
