[
https://issues.apache.org/jira/browse/KAFKA-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15317929#comment-15317929
]
Ewen Cheslack-Postava commented on KAFKA-3793:
----------------------------------------------
[~rgo...@threatmetrix.com] Can you provide more details? Given the lack of
reports of similar issues with other clients, I would suspect it might be the
client library itself instead of Kafka itself (which would need to be reported
to the author of the library). Is there any pattern to the message that causes
the problem? Another possibility is that you're passing data to the library
that is getting truncated incorrectly, e.g. maybe if there was a null byte not
correctly encoded for JSON in the middle of your data.
> Kafka Python Consumer library messages gets truncated
> -----------------------------------------------------
>
> Key: KAFKA-3793
> URL: https://issues.apache.org/jira/browse/KAFKA-3793
> Project: Kafka
> Issue Type: Bug
> Reporter: Rahul
>
> Snippet code is below:
> from kafka import KafkaConsumer
> from kafka.client import KafkaClient
> from kafka.consumer import SimpleConsumer
> consumer = KafkaConsumer('eventdetails_ingestion' ,
> group_id='1',bootstrap_servers=‘xxxx:9092',
> max_partition_fetch_bytes=10555555)
> for msg in consumer:
> try:
> jValue = json.loads(str(msg.value))
> except ValueError:
> fileErr.write(str(msg.value)+"\n")
> Steps:
> We send/produce large sets of messages to Kafka of around 20 to 30 KB size
> each messages in JSON format and producing around 200 messages / sec for 1
> hour duration. We have 3 Kafka Brokers running and I am trying to consume the
> messages from these 3 Kafka Brokers from the same topic using the above code.
> The problem is that sometimes some of the messages gets truncated, I am not
> sure why it happen ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
