rkaw92 commented on issue #230: URL: https://github.com/apache/pulsar-client-node/issues/230#issuecomment-1240355308
Aha! https://github.com/apache/pulsar-client-node/blob/master/src/ProducerConfig.cc#L103 Setting `maxPendingMessagesAcrossPartitions` actually calls `pulsar_producer_configuration_set_max_pending_messages`, not `pulsar_producer_configuration_set_max_pending_messages_across_partitions`. So the default stays the same at 50k: https://github.com/apache/pulsar/blob/bff34000385f6faf6dbff4385d0dc562602ac623/pulsar-client-cpp/lib/ProducerConfigurationImpl.h#L36 And indeed, I'm only able to reproduce this error on a partitioned topic. Let's see: * I set maxPendingMessages to 10000 (ten thousand) in code * I set maxPendingMessagesAcrossPartitions to some really high value like 1 million * I'm writing to a topic with 16 partitions So, by my logic, the maximum number I can actually fit between `flush()` calls is 50000 (**the default from C++**), divided by the partition count. 50000 / 16 = 3125. This is regardless of my settings. Let's see if this is true! ``` node perf/perf_producer.js -m 3125 -i 10 -u pulsar://mycluster.url --topic persistent://rkaw-test/benchmark/perf_producer # this completes perfectly ``` and... ``` node perf/perf_producer.js -m 3126 -i 10 -u pulsar://mycluster.url --topic persistent://rkaw-test/benchmark/perf_producer [...] [Error: Failed to send message: ProducerQueueIsFull] ``` So yeah, it's the configuration not making it to the actual client. I think flush() works, after all. Some caveats: * It only works with `batchEnabled: true` - flushing seems to have no effect if you disable batches (by design?), so you can still get ProducerQueueIsFull in non-batch mode * This problem only affects publishing to a partitioned topic * I'm not sure why the Producer is erroring out so easily, i.e. if you exceed the *real* maxPendingMessages by 1. I'd expect the messages to go to different partitions - maybe batching is causing them to bunch up in one underlying Producer for 1 partition. I'll try a local fix that actually propagates `pulsar_producer_configuration_set_max_pending_messages_across_partitions` and see if that alleviates the problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
