> Additionally it would be good to force triggering the creation of a non > partitioned topic even if the namespace/broker is configured to > automatically create partitioned topics
I think this additional feature request is a bit more controversial than the fail fast logic discussed earlier. To trigger non partitioned topic creation would essentially require that the non-admin client can create topics outside of auto-topic creation. To date, that feature has been limited to the admin client and is not available in the Pulsar protocol. It'd be pretty easy to expand the pulsar protocol so that the Producer command has a "partition_count" field and that field is only applied when the topic is created (and perhaps fails if the count is 0 for an already created partitioned topic). I could see this as a valuable addition for use cases that are sensitive to message ordering, but perhaps it is worth asking whether we should just make topic creation a first class feature in the Pulsar protocol? Thanks, Michael On Thu, Feb 17, 2022 at 5:49 AM Enrico Olivelli <eolive...@gmail.com> wrote: > > Sijie, > > Il Gio 17 Feb 2022, 07:13 Michael Marshall <mmarsh...@apache.org> ha > scritto: > > > > Ideally, people shouldn't be aware of partitioned vs non-partitioned. > > > > Sometimes it is a requirement to have a non-partioned topic and the > application will misbehave if running on a partitioned topic with more than > one partition. > This is because if you have a partitioned topic Pulsar does not provide > guarantees about ordering of all the messages sent by a producer, as it > will depend on the key (partition). > > Unfortunately when you create a Producer you may trigger the creation of a > topic and this topic maybe a partitioned topic. > > The problem is also on the Consumer side, with a partitioned topic you are > not guaranteed to receive the messages in the same order as they were > written, because the order is guaranteed per partition. > > Do you know a way to address my problem in spite of any broker > side/namespace configuration option ? Because the user may not have control > over such configurations or even over the Admin API > > > > > > > I'm not sure I agree that people shouldn't be aware of these details > > considering the two types of topics provide different guarantees > > regarding ordering. Enrico's request is essentially to have a way to > > fail fast when a topic is partitioned. > > > > Yes, this is correct. > Additionally it would be good to force triggering the creation of a non > partitioned topic even if the namespace/broker is configured to > automatically create partitioned topics > > Enrico > > > > > > Note that the logic for whether or not a topic is partitioned is > > already available to the client: that is how it knows whether to > > create a partitioned producer or a non-partitioned producer. > > > > The simplest solution is to add a method to the client named > > "createNonPartitionedProducer". The method would then fail if the > > topic lookup results in a topic with more than 0 partitions. > > > > Alternatively, you could create a custom `MessageRouter` that always > > picks the 0 partition when `choosePartition` is called. However, I > > view this as a bit of a hack and think my first solution is a better > > fit for Enrico's request. > > > > Thanks, > > Michael > > > > On Wed, Feb 16, 2022 at 3:27 PM Sijie Guo <guosi...@gmail.com> wrote: > > > > > > I don't think we should expose this feature flag on the client-side. > > > Ideally, people shouldn't be aware of partitioned vs non-partitioned. > > > > > > - Sijie > > > > > > On Wed, Feb 16, 2022 at 1:37 AM Enrico Olivelli <eolive...@gmail.com> > > wrote: > > > > > > > Ping > > > > > > > > Il Dom 13 Feb 2022, 11:49 Enrico Olivelli <eolive...@gmail.com> ha > > > > scritto: > > > > > > > > > Hello, > > > > > I have a use case in which my application MUST use a non-partitioned > > > > > topic to work properly, but the topic name is (of course) > > configurable > > > > > by the user. > > > > > If I am not using a non-partitioned topic, I do not have string > > > > > guarantees on the ordering of the message (because messages will be > > > > > spread across multiple partitions). > > > > > > > > > > Currently there is no way to require that the topic IS a > > > > > NON-PARTITIONED topic. Especially when the topic does not exist yet, > > > > > you configured the namespace to create partitioned topics by default. > > > > > > > > > > Using a PulsarAdmin preliminary call is not a good workaround because > > > > > it is expensive, and ideally I want to be able to create the Producer > > > > > (or the Consumer) and see that everything works automatically, > > failing > > > > > in case of a partitioned topic. > > > > > > > > > > Thoughts ? > > > > > > > > > > > > > > > Enrico > > > > > > > > > > >
