[
https://issues.apache.org/jira/browse/KAFKA-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105519#comment-14105519
]
Sriharsha Chintalapani commented on KAFKA-1588:
-----------------------------------------------
[~toddpalino]
I gone through kafka-python code .Please correct me if I am doing something
wrong.
I am sending two OffsetRequests
offsets=kafka.send_offset_request([ OffsetRequest("my-topic", 0, -2, 1),
OffsetRequest("my-topic", 0, -1, 1)])
in send_offset_request which calls Protocol.encode_offset_request
it get these above two payloads but convert them into a grouped_payloads which
is a dict
this is what it looks like
payloads
[OffsetRequest(topic='my-topic', partition=0, time=-2, max_offsets=1),
OffsetRequest(topic='my-topic', partition=0, time=-1, max_offsets=1)]
grouped_payloads
defaultdict(<type 'dict'>, {'my-topic': {0: OffsetRequest(topic='my-topic',
partition=0, time=-1, max_offsets=1)}})
Kafka api expects it to be Map with key being topic name
so even though you are sending two requests using client.send_offset_request
if the topic is the same it will be overwritten and only one request is being
sent to kafka.
The two OffsetResponses coming back because in
client._send_broker_aware_request it takes the keys from your two requests in
this case it will form [('my-topic', 0), ('my-topic', 0)] (check original_keys
variable) and the response is constructed by looping over this original_keys
and returns the same OffsetResponse twice
# Order the accumulated responses by the original key order
return (acc[k] for k in original_keys) if acc else ()
[~junrao] [~guozhang]
Please let me know if its ok to support two offsetRequest for the same topic in
a single call. This would also means we might allow duplicate requests for the
same topic,partition, offset. Changing to a list instead of map probably be
easier way but how about the backward compatability. I can follow the version
for protocol changes to support both .
> Offset response does not support two requests for the same topic/partition
> combo
> --------------------------------------------------------------------------------
>
> Key: KAFKA-1588
> URL: https://issues.apache.org/jira/browse/KAFKA-1588
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8.2
> Reporter: Todd Palino
> Assignee: Sriharsha Chintalapani
> Labels: newbie
>
> When performing an OffsetRequest, if you request the same topic and partition
> combination in a single request more than once (for example, if you want to
> get both the head and tail offsets for a partition in the same request), you
> will get a response for both, but they will be the same offset.
> We identified that the problem is that when the offset response is assembled,
> a map is used to store the offset info before it is converted to the response
> format and sent to the client. Therefore, the second request for a
> topic/partition combination will overwrite the offset from the first request.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
