Assuming this is a test case with a new topic/consumer groups for each run, do you have auto.offset.reset=smallest? This happens to me constantly in tests because my consumers end up missing the first message since the default is largest (in which case auto commit is a red herring.)
On Wed, Sep 23, 2015 at 3:17 AM Joris Peeters <j.peet...@wintoncapital.com> wrote: > I have a topic with three partitions, to which I send 5 messages (very > rapidly after one another) and they get partitioned well ([1,2,2] as it > happens). > I'm launching three identical high level java customers (kafka 8.2.1), > single-threaded, to consume those messages. Once a message is received, the > consumers have to do a fair amount of work, which typically takes a few > minutes. Right now I'm committing the offset after every message has been > completed (so I have auto.commit.enable to false *). > > What I am seeing, with this setup, is that the three consumers each work > through their first message and then just hang (in the subsequent > "it.hasNext()"). The remaining two messages do not arrive. When, on the > other hand, I set auto.commit.enable to true, everything goes well and the > three consumers churn through the five messages without trouble. > > This makes me suspect there is some sort of heartbeat issue going on, that > is triggered by the auto.commit? We don't want to auto-commit in this case, > though, so does anyone know of a different solution that could be tried? > > My situation sounds somewhat similar to > http://grokbase.com/t/kafka/users/14ck70w38q/kafka-consumer-session-timeouts > - but I am not sure if anyone got to the bottom of that. > > As always, any help would be much appreciated! > -Joris. > > > (*) These are the consumer properties I have set explicitly: > > props.put("auto.commit.interval.ms", "1000"); > props.put("auto.commit.enable", "false"); // or true. > props.put("offsets.storage", "kafka"); > > > > > Winton Capital Management Limited ("Winton") is a limited company > registered in England and Wales with its registered offices at 16 Old > Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is > authorised and regulated by the Financial Conduct Authority in the United > Kingdom, registered as an investment adviser with the US Securities and > Exchange Commission, registered with the US Commodity Futures Trading > Commission and a member of the National Futures Association in the United > States. > > This communication, including any attachments, is confidential and may be > privileged. This email is for use by the intended recipient only. If you > receive it in error, please notify the sender and delete it. You should not > copy or disclose all or any part of this email. > > This email does not constitute an offer or solicitation and nothing > contained in this email constitutes, and should not be construed as, > investment advice. Prospective investors should request offering materials > and consult their own advisers with respect to investment decisions and > inform themselves as to applicable legal requirements, exchange control > regulations and taxes in the countries of their citizenship, residence or > domicile. Past performance is not indicative of future results. > > Winton takes reasonable steps to ensure the accuracy and integrity of its > communications, including emails. However Winton accepts no liability for > any materials transmitted. Emails are not secure and cannot be guaranteed > to be error free. >
