[
https://issues.apache.org/jira/browse/FLINK-32645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17745916#comment-17745916
]
Vijaya Bhaskar V commented on FLINK-32645:
------------------------------------------
After disabling batching of the messages, message throughput increased little
bit.
This shows that, pulsar writer thread queue lock is causing the issue while
adding more messages to it, also as shown by thread dump.
One more aspect that confirms it is:
Increased the partitions of sink topic and increased the parallelism of sink
equal to partitions. Throughput is increasing.
But following are the problems with this approach
# We have to use more task slots of flink, i.e., more resources unnecessarily
# Inspite of increasing back pressure is still remaining same because pulsar
sink is 100% busy always
# !Screenshot 2023-07-22 at 2.56.55 PM.png!!Screenshot 2023-07-22 at 3.45.21
PM.png!!Screenshot 2023-07-22 at 3.45.21 PM.png! no matter the number of slots
are attached the screen shot
> Flink pulsar sink is having poor performance
> --------------------------------------------
>
> Key: FLINK-32645
> URL: https://issues.apache.org/jira/browse/FLINK-32645
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
> Affects Versions: 1.16.2
> Environment: !Screenshot 2023-07-22 at 1.59.42 PM.png!!Screenshot
> 2023-07-22 at 2.03.53 PM.png!
>
> Reporter: Vijaya Bhaskar V
> Priority: Major
> Attachments: Screenshot 2023-07-22 at 2.03.53 PM.png, Screenshot
> 2023-07-22 at 2.56.55 PM.png, Screenshot 2023-07-22 at 3.45.21 PM-1.png,
> Screenshot 2023-07-22 at 3.45.21 PM.png
>
>
> Found following issue with flink pulsar sink:
>
> Flink pulsar sink is always waiting while enqueueing the message and making
> the task slot busy no matter how many free slots we provide. Attached the
> screen shot of the same
> Just sending messages of less rate 8k msg/sec and stand alone flink job with
> discarding sink is able to receive full rate if 8K msg/sec
> Where as pulsar sink was consuming only upto 2K msg/sec and the sink is
> always busy waiting. Snapshot of thread dump attached.
> Also snap shot of flink stream graph attached
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
