Is the Kafka producer configuration different than the Samza configuration
of the Samza task that references the store? If not, here is are those
configuration values. The changelog in question is resourceStore-changelog.
# Job
job.factory.class=org.apache.samza.job.local.ThreadJobFactory
job.name=resource_normalizer
job.coordinator.system=kafka
# Task
task.class=com.foo.blazer.resource.normalizer.samza.ResourceNormalizerSamzaTask
task.inputs=kafka.com.foo.iam.indexing.resource.mutation
task.window.ms=10000
task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory
task.checkpoint.system=kafka
# Normally, this would be 3, but we have only one broker.
task.checkpoint.replication.factor=3
task.checkpoint.skip-migration=true
# Serializers
serializers.registry.byte.class=org.apache.samza.serializers.ByteSerdeFactory
serializers.registry.entity.class=com.foo.blazer.resource.normalizer.serde.ResourceEventEntitySerdeFactory
serializers.registry.string.class=org.apache.samza.serializers.StringSerdeFactory
serializers.registry.int.class=org.apache.samza.serializers.IntegerSerdeFactory
serializers.registry.metrics.class=org.apache.samza.serializers.MetricsSnapshotSerdeFactory
# Systems
systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory
systems.kafka.samza.msg.serde=byte
systems.kafka.samza.offset.default=oldest
systems.kafka.consumer.zookeeper.connect=${ZK_NODES}/${ZK_ROOT}
systems.kafka.consumer.auto.offset.reset=smallest
systems.kafka.producer.bootstrap.servers=${KAFKA_NODES}
systems.kafka.producer.max.request.size=52428800
systems.kafka.streams.metrics.samza.msg.serde=metrics
# Metrics
metrics.reporter.snapshot.class=org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactor
metrics.reporter.snapshot.stream=kafka.metrics
metrics.reporter.jmx.class=org.apache.samza.metrics.reporter.JmxReporterFactory
metrics.reporters=snapshot,jmx
# Stores
stores.resourceStore.factory=org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory
stores.resourceStore.changelog=kafka.resourceStore-changelog
stores.resourceStore.changelog.replication.factor=3
stores.resourceStore.key.serde=string
stores.resourceStore.msg.serde=entity
stores.deletedFlagStore.factory=org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory
stores.deletedFlagStore.changelog=kafka.deletedFlagStore-changelog
stores.deletedFlagStore.changelog.replication.factor=3
stores.deletedFlagStore.key.serde=string
stores.deletedFlagStore.msg.serde=int
We do not get a stack trace from the Samza task itself, it just never seems
to fully start. However if we use kafka-console-consumer to try to examine
the changelog we get this:
2016-02-02 22:10:11,252] ERROR Error processing message, terminating
consumer process: (kafka.tools.ConsoleConsumer$)
kafka.common.MessageSizeTooLargeException: Found a message larger than the
maximum fetch size of this consumer on topic resourceStore-changelog
partition 7 at fetch offset 0. Increase the fetch size, or decrease the
maximum message size the broker will allow.
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:90)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
at kafka.consumer.OldConsumer.receive(BaseConsumer.scala:79)
at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:110)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:69)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:47)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
On Fri, Feb 5, 2016 at 3:42 PM, Ramesh Bhojan <rbhojan.soc...@gmail.com>
wrote:
> Jason,
> Can we please share more information about the exact stack trace and the
> job configuration, especially the Kafka producer configuration for the
> changelog system, as requested by Yi Pan?
>
> Regards,
> Ramesh
>
> On Thu, Feb 4, 2016 at 11:56 AM, Ramesh Bhojan <rbhojan.soc...@gmail.com>
> wrote:
>
>> Dear team @ Samza,
>> I would really appreciate some help with the following question posted in
>> Stack Overflow :
>>
>>
>> http://stackoverflow.com/questions/35168641/is-there-a-configuration-setting-to-allow-large-values-in-my-samza-store
>>
>> Thanks,
>> Ramesh
>>
>
>
--
Thanks,
Jason Erickson
