Hi John,
Please find the full stack trace with 3 frames from the app removed that
are process/state-store related functions.
{"log":"2022-08-17 15:21:32,890 [ERROR] Util: Failed saving
key=cx_dev_sys:c3d376b6e62f11ecbf8256eb17370719:SG18KW70HN to stateStore
with Exception={}\n\njava.lang.UnsupportedOperationException: this should
not happen: timestamp() is not supported in standby tasks.\n\n\tat
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.throwUnsupportedOperationExceptionIfStandby(ProcessorContextImpl.java:352)\n\n\tat
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.timestamp(ProcessorContextImpl.java:328)\n\n\tat
org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.log(ChangeLoggingKeyValueBytesStore.java:136)\n\n\tat
org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:78)\n\n\tat
org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:32)\n\n\tat
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$put$4(MeteredKeyValueStore.java:197)\n\n\tat
org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)\n\n\tat
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.put(MeteredKeyValueStore.java:197)\n\n\tat
org.apache.kafka.streams.processor.internals.AbstractReadWriteDecorator$KeyValueStoreReadWriteDecorator.put(AbstractReadWriteDecorator.java:120)\n\n\tat
// app-calls to process & save to state store - 3 frames
org.apache.kafka.streams.processor.internals.ProcessorNode.lambda$process$2(ProcessorNode.java:181)\n\n\tat
org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)\n\n\tat
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:181)\n\n\tat
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:273)\n\n\tat
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:252)\n\n\tat
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:219)\n\n\tat
org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:86)\n\n\tat
org.apache.kafka.streams.processor.internals.StreamTask.lambda$process$1(StreamTask.java:703)\n\n\tat
org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)\n\n\tat
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:703)\n\n\tat
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1105)\n\n\tat
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:647)\n\n\tat
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)\n\n\tat
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:512)\n"
If replication factor = 1 and standby replicas=1, is that an issue? Do we
expect that the replication factor should be at least n+1, if standby
replicas=1 (or) there is no relationship?
Another data point is that once I made the standby replicas to 0, the crash
stopped (or) if I run only one pod (one application instance), even then
the crash is not seen.
So, is there something that is not correct in the way we have configured
the Kafka Streams app and/or Kafka is not handling the standby task
correctly (like calling the put from the standby context, when it is not
supposed to)? Is there a way to debug this better by enabling some logs
from Kafka (like set some env var to add more debug logs) to isolate the
issue?
thanks,
-sureshr
On Tue, Aug 23, 2022 at 3:38 PM Suresh Rukmangathan <suresh...@gmail.com>
wrote:
> Hi,
>
> We have a Kafka streams application with "num.standby.replicas" = 1 and
> with replication-factor of 2 for that topic. The application is crashing
> with the following stack trace.
>
> java.lang.UnsupportedOperationException: this should not happen:
> timestamp() is not supported in standby tasks.\n\n\tat
>
>
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.throwUnsupportedOperationExceptionIfStandby(ProcessorContextImpl.java:352)\n\n\tat
>
>
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.timestamp(ProcessorContextImpl.java:328)\n\n\tat
>
>
> org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.log(ChangeLoggingKeyValueBytesStore.java:136)\n\n\tat
>
>
> org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:78)\n\n\tat
>
>
> org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:32)\n\n\tat
>
>
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$put$4(MeteredKeyValueStore.java:197)\n\n\tat
>
>
> org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)\n\n\tat
>
>
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore.put(MeteredKeyValueStore.java:197)\n\n\tat
>
> Key Kafka streams application configuration details are as below:-
>
> {replication.factor=1, num.standby.replicas=1, topology.optimization=all,
>
> producer.partitioner.class=com.hpe.gravity.gravitycommon.utils.Murmur3Partitioner,
> max.request.size=1048576, auto.offset.reset=earliest}
>
> Is this a Kafka library issue or something wrong with the application
> configuration?
>
> thanks,
> -sureshr
>
>
