Some updates below.
After changing the number of contains *from 6 to 4*, we are able to see
changelog offsets moving across the board. Yay!
Now the question is: why?
With 20 input partitions/task instances, the previous setting will allocate
a different number of task instances (4) to some cont
Hi, Yi,
I couldn't find any errors in the log indicating any issue writing to those
particular changelog partitions. I event went ahead and removed all
checkpoint, coordinator and changelog topics and started fresh. This issue
is still manifested itself with offsets of 0:
partition offset
---
Hi, David,
Did you check the log to see whether there is any log lines indicating the
producer issues on the three partitions that you suspect? And could you
also check whether you have auto-commit turned on? If your auto-commit is
on and producer does not report any issue writing to the changelog
Jagadish,
All your description matches my understand.
Here are our settings:
- Our task aggregates user events into user sessions.
- We have one k-v store for each task, which tracks active user sessions
(with sessionId as the key).
- When a user session expires, the session will be removed from
Some context: Each k-v store has a changelog topic. The # of partitions in
that changelog topic is equal to the # of tasks. Each task's K-V store will
be mapped to a particular partition of that changelog topic. This mapping
from taskNames-changeLogPartitionNumber is stored in coordinator stream.
