rkaw92 opened a new issue, #230: URL: https://github.com/apache/pulsar-client-node/issues/230
Hi, I have a problem with the Producer where I do this in a loop: * `send()` a given number of messages * `await flush()`, so that the send queue is empty * `send()` more messages... (loop) I'd expect the following to hold true: as long as the number of messages I send **between flushes** is lower than `maxPendingMessages`, I should be able to send. But, this assumption is broken: when I send immediately after flush has resolved, I get a `ProducerQueueIsFull` error. I think that, when `flush()` resolves, the messages haven't actually been flushed. This, or there exists another, hidden limit that triggers `ProducerQueueIsFull`. I've patched the benchmark client to demonstrate this: https://gist.github.com/rkaw92/32ac9ecaa4a5a1fa0d80ba4c4dec19bc The only differences from the "normal" perf_producer.js are: * the ProducerConfig (`maxPendingMessages: 10000` - ten thousand - and some other options) * the removal of `delay(1000)` in each iteration. Note that, to run the producer perf test, you may need to *downgrade* hdr-histogram-js - this is currently bugged in this repo: ``` npm install hdr-histogram-js@1 ``` With the no-delay version, it's easy to trigger a `ProducerQueueIsFull` error: ``` $ node perf/perf_producer.js -m 10000 -i 10 -u pulsar://mycluster.url --topic persistent://rkaw-test/benchmark/perf_producer ---------------------- size: 1024 url: pulsar://mycluster.url topic: persistent://rkaw-test/benchmark/perf_producer iteration: 10 messages: 10000 ---------------------- Throughput produced: 84717.558 msg/s --- 661.856 Mbit/s --- Latency: mean: 0 ms - med: 0 - 95pct: 0 - 99pct: 0 - 99.9pct: 0 - 99.99pct: 0 - Max: 0 node:internal/process/promises:265 triggerUncaughtException(err, true /* fromPromise */); ^ [Error: Failed to send message: ProducerQueueIsFull] ``` It's even possible to do this with lower message counts: `m 5000 -i 10` also produces the error `ProducerQueueIsFull`. I believe I'm using `flush()` correctly by awaiting it. Is there a problem with operation ordering between `send()` and `flush()`? This happens regardless of `batchingEnabled` - setting it to false changes nothing. Also, this seems rather troubling: if I change to `blockIfQueueFull`, and re-run perf_producer.js, I get this at the very end: ``` [Error: Failed to send message: AlreadyClosed] ``` Since the code does an orderly `await producer.flush()` at the end of every iteration, and the `producer.close()` only takes place at the end *after the last iteration*, is there an ordering issue between `flush()` and the rest? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
