I have a topic with three partitions, to which I send 5 messages (very rapidly after one another) and they get partitioned well ([1,2,2] as it happens). I'm launching three identical high level java customers (kafka 8.2.1), single-threaded, to consume those messages. Once a message is received, the consumers have to do a fair amount of work, which typically takes a few minutes. Right now I'm committing the offset after every message has been completed (so I have auto.commit.enable to false *).
What I am seeing, with this setup, is that the three consumers each work through their first message and then just hang (in the subsequent "it.hasNext()"). The remaining two messages do not arrive. When, on the other hand, I set auto.commit.enable to true, everything goes well and the three consumers churn through the five messages without trouble. This makes me suspect there is some sort of heartbeat issue going on, that is triggered by the auto.commit? We don't want to auto-commit in this case, though, so does anyone know of a different solution that could be tried? My situation sounds somewhat similar to http://grokbase.com/t/kafka/users/14ck70w38q/kafka-consumer-session-timeouts - but I am not sure if anyone got to the bottom of that. As always, any help would be much appreciated! -Joris. (*) These are the consumer properties I have set explicitly: props.put("auto.commit.interval.ms", "1000"); props.put("auto.commit.enable", "false"); // or true. props.put("offsets.storage", "kafka"); Winton Capital Management Limited ("Winton") is a limited company registered in England and Wales with its registered offices at 16 Old Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is authorised and regulated by the Financial Conduct Authority in the United Kingdom, registered as an investment adviser with the US Securities and Exchange Commission, registered with the US Commodity Futures Trading Commission and a member of the National Futures Association in the United States. This communication, including any attachments, is confidential and may be privileged. This email is for use by the intended recipient only. If you receive it in error, please notify the sender and delete it. You should not copy or disclose all or any part of this email. This email does not constitute an offer or solicitation and nothing contained in this email constitutes, and should not be construed as, investment advice. Prospective investors should request offering materials and consult their own advisers with respect to investment decisions and inform themselves as to applicable legal requirements, exchange control regulations and taxes in the countries of their citizenship, residence or domicile. Past performance is not indicative of future results. Winton takes reasonable steps to ensure the accuracy and integrity of its communications, including emails. However Winton accepts no liability for any materials transmitted. Emails are not secure and cannot be guaranteed to be error free.
