Thanks for the response, but I'm afraid that a simple affinity key doesn't
help in this case. To clarify - I don't just want to ensure that data records
in Ignite are stored in the same node - I want to ensure that the events from
Kafka arrive on the right node too.
Kafka uses a partitoning and assignment scheme very similar to the one Ignite
uses. It also supports event affinity. But the actual paritition & node
assignment will never match up between Kafka and Ignite.
That means that an event e1 might land on a kafka consumer running on Ignite
node n1 - but Ignite may chose to store the data on node n2. For a high
throughput of events, this kills performance and scale.
What I want to do is override the assignment to ensure that the event landing
on node n1 is also stored on node n1. Meaning that the data access will
normally be local and therefore fast.
Second prize would be to ensure that at least the partitions are aligned. So
that the Kafka and Ignite partition number for a given key is the same. So if
k1 & k2 are mapped to partition 1 and k3, k4 are mapped to partition 2 - that
will be true both for Kafka and Ignite. Then at least the node assignment can
be handled on the consumer and doesn't require a change to the producer.
In any case, I think all of this could be achieved in Ignite 2 - but the
question is - can it be achieved also in Ignite 3?
Thanks,Colin.
On Thursday, 26 January 2023 at 11:57:06 GMT, Stephen Darlington
<stephen.darling...@gridgain.com> wrote:
You don’t need to override the RendezvousAffinityFunction to do what you’re
asking. Instead, if you define an affinity key to your table, it will guarantee
that related records will be kept together. However, make sure your groups are
not too coarse or you’ll get poor distribution across your cluster.
On 26 Jan 2023, at 11:37, Colin Cassidy via user <user@ignite.apache.org> wrote:
I would like to override the Ignite AffinityFunction to delegate to Kafka's
partitioning algorithm. Possibly also the node assignment - but definitely the
partitioning
This will allow me to ensure that events are processed local to where the data
is stored.
I think this is possible in Ignite 2.x but there appear to be significant
changes in Ignite 3.x. Could I confirm - if I follow this path, will it be
possible to override the cache partitioning still in 3.x?
Thanks in advance,Colin.
