I might be missing something, but it sounds like you want to keep ordering
of events which belong to the same id, and not really about having
hard-coded partition number for writing.
If so, would it be enough to create a keyed PCollection with id as a key +
use KafkaIO.writeRecords() ?
I am conclu
Unfortunately, we don't have a feature to publish to a specific partition.
We tried to design with Kafka conventions in mind, and I don't believe we
plan to add this functionality.
On Thu, Apr 13, 2023 at 3:03 PM Juan Romero wrote:
> Hi John. Thanks for your response!.
>
> Point 2 is clear now f
Hi John. Thanks for your response!.
Point 2 is clear now for me. I was reading lot of documentation about it
and i only wanted to confirm with you.
Regarding point 1 I know the drawbacks , we have to avoid hot partitions
(For this purpouse we can define a hash function that evenly distributes
the
Hi Juan,
Under normal usage, Kafka will maintain ordering within a partition without
any extra work by you.
For 2, you can use .commitOffsetsInFinalize to only commit back to the
source topic once the pipeline has persisted the message, at which point it
may not be fully processed, but it is guar
