Hey guys,
We are running a stream application in our production environment. On our
latest restart, the application is consistently moving between these two
states.
>From our logs:
grep "State transition from " application.log | jq -r '.message' | sort |
uniq -c | sort -n -r
40 stream-thread [yyyy-StreamThread-9] State transition from
PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
40 stream-thread [yyyy-StreamThread-8] State transition from
PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
40 stream-thread [yyyy-StreamThread-7] State transition from
PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
40 stream-thread [yyyy-StreamThread-6] State transition from
PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
40 stream-thread [yyyy-StreamThread-5] State transition from
PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
40 stream-thread [yyyy-StreamThread-4] State transition from
PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
40 stream-thread [yyyy-StreamThread-3] State transition from
PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
40 stream-thread [yyyy-StreamThread-2] State transition from
PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
40 stream-thread [yyyy-StreamThread-1] State transition from
PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
40 stream-thread [yyyy-StreamThread-12] State transition from
PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
40 stream-thread [yyyy-StreamThread-11] State transition from
PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
40 stream-thread [yyyy-StreamThread-10] State transition from
PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
39 stream-thread [yyyy-StreamThread-9] State transition from
PARTITIONS_ASSIGNED to PARTITIONS_REVOKED
39 stream-thread [yyyy-StreamThread-8] State transition from
PARTITIONS_ASSIGNED to PARTITIONS_REVOKED
39 stream-thread [yyyy-StreamThread-7] State transition from
PARTITIONS_ASSIGNED to PARTITIONS_REVOKED
39 stream-thread [yyyy-StreamThread-6] State transition from
PARTITIONS_ASSIGNED to PARTITIONS_REVOKED
39 stream-thread [yyyy-StreamThread-5] State transition from
PARTITIONS_ASSIGNED to PARTITIONS_REVOKED
39 stream-thread [yyyy-StreamThread-4] State transition from
PARTITIONS_ASSIGNED to PARTITIONS_REVOKED
39 stream-thread [yyyy-StreamThread-3] State transition from
PARTITIONS_ASSIGNED to PARTITIONS_REVOKED
39 stream-thread [yyyy-StreamThread-2] State transition from
PARTITIONS_ASSIGNED to PARTITIONS_REVOKED
39 stream-thread [yyyy-StreamThread-1] State transition from
PARTITIONS_ASSIGNED to PARTITIONS_REVOKED
39 stream-thread [yyyy-StreamThread-12] State transition from
PARTITIONS_ASSIGNED to PARTITIONS_REVOKED
39 stream-thread [yyyy-StreamThread-11] State transition from
PARTITIONS_ASSIGNED to PARTITIONS_REVOKED
39 stream-thread [yyyy-StreamThread-10] State transition from
PARTITIONS_ASSIGNED to PARTITIONS_REVOKED
As we can see the stream threads are first revoked than again assigned.
Also we can see the logs of resetting of offsets continuously as follows:
Resetting offset for partition xxxx-2 to offset 9166288.
We had actually deleted the consumer group on broker before the restart as
there was considerable lag in the topic and processing of the stale data
was not intended. We had assumed that on deleting the group, the
application will start processing from latest offset as mentioned in the
config auto.offset.reset policy.
On describing the consumer group on broker side, we receive output with the
current offset and lag set as -- (Eg shown below)
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID
xxx 10 - 129822997 -
yyyy-StreamThread-2-consumer-a-b-c-d
Please help us understand why this can be happening and how to solve this.
