I have made the suggested change and used
ConfluentSchemaRegistryDeserializerProvider
the results are slightly better.. average of 8000 msg/sec
Thank you both for your response and i'll appreciate if you can keep me in
the loop in the planned work with kafka schema or let me know if i can
assist
Thanks for testing this!
It requires some additional investigations, so I created an issue for that:
https://github.com/apache/beam/issues/26262
Feel free to add more details if you have there.
—
Alexey
> On 13 Apr 2023, at 12:45, Sigalit Eliazov wrote:
>
> I have made the suggested change a
Hi Apache Beam team. I have been working on a POC for the company i'm
working for and Im using apache beam kafka connector to read from kafka
topic and write into other kafka topic. The source and target topic have 3
partitions and is compulsory keep ordering by certain message keys.
Regarding it I
Hi Juan,
Under normal usage, Kafka will maintain ordering within a partition without
any extra work by you.
For 2, you can use .commitOffsetsInFinalize to only commit back to the
source topic once the pipeline has persisted the message, at which point it
may not be fully processed, but it is guar
Hi John. Thanks for your response!.
Point 2 is clear now for me. I was reading lot of documentation about it
and i only wanted to confirm with you.
Regarding point 1 I know the drawbacks , we have to avoid hot partitions
(For this purpouse we can define a hash function that evenly distributes
the
Unfortunately, we don't have a feature to publish to a specific partition.
We tried to design with Kafka conventions in mind, and I don't believe we
plan to add this functionality.
On Thu, Apr 13, 2023 at 3:03 PM Juan Romero wrote:
> Hi John. Thanks for your response!.
>
> Point 2 is clear now f
I might be missing something, but it sounds like you want to keep ordering
of events which belong to the same id, and not really about having
hard-coded partition number for writing.
If so, would it be enough to create a keyed PCollection with id as a key +
use KafkaIO.writeRecords() ?
I am conclu
