I've found that (especially in tests) I can get a ConsumerIterator, but
that the consumer is not actually connected and settled onto its partitions
yet. Unfortunately, I haven't found a good way to establish that aside from
sending pilot messages through first. smallest is easier :)
On Wed, Sep 23, 2015 at 4:26 AM Joris Peeters <j.peet...@wintoncapital.com>
wrote:
> Looks like you are right, although I am not sure I understand why. :)
>
> The outlined scenario is indeed part of a test suite, which goes through
> the following stages (with some temporal separation between each),
>
> 1/ Create a new topic (with some guid in the name), with three partitions
> 2/ Launch the three consumers (all part of the same consumer group, which
> is fixed at compile time).
> 3/ Fire the five messages
>
> Adding 'props.put("auto.offset.reset", "smallest");' to my list of
> settings seems to resolve the hanging issue I mentioned.
>
> The line
> >>
> If this is set to largest, the consumer may lose some messages when the
> number of partitions, for the topics it subscribes to, changes on the
> broker. To prevent data loss during partition addition, set
> auto.offset.reset to smallest"
> <<
>
> seems to suggest that problems would only occur if the number of
> partitions changes on the broker, but that is set to (and stays at) 3.
> Whilst I'm quite interested in finding out what is going on here, I'm in
> any case quite thrilled that at least it seems to work now. :) Thanks!
> -J
>
> -----Original Message-----
> From: noah [mailto:iamn...@gmail.com]
> Sent: 23 September 2015 09:44
> To: users@kafka.apache.org
> Subject: Re: high level consumer timeout?
>
> Assuming this is a test case with a new topic/consumer groups for each
> run, do you have auto.offset.reset=smallest? This happens to me constantly
> in tests because my consumers end up missing the first message since the
> default is largest (in which case auto commit is a red herring.)
>
> On Wed, Sep 23, 2015 at 3:17 AM Joris Peeters <j.peet...@wintoncapital.com
> >
> wrote:
>
> > I have a topic with three partitions, to which I send 5 messages (very
> > rapidly after one another) and they get partitioned well ([1,2,2] as
> > it happens).
> > I'm launching three identical high level java customers (kafka 8.2.1),
> > single-threaded, to consume those messages. Once a message is
> > received, the consumers have to do a fair amount of work, which
> > typically takes a few minutes. Right now I'm committing the offset
> > after every message has been completed (so I have auto.commit.enable to
> false *).
> >
> > What I am seeing, with this setup, is that the three consumers each
> > work through their first message and then just hang (in the subsequent
> > "it.hasNext()"). The remaining two messages do not arrive. When, on
> > the other hand, I set auto.commit.enable to true, everything goes well
> > and the three consumers churn through the five messages without trouble.
> >
> > This makes me suspect there is some sort of heartbeat issue going on,
> > that is triggered by the auto.commit? We don't want to auto-commit in
> > this case, though, so does anyone know of a different solution that
> could be tried?
> >
> > My situation sounds somewhat similar to
> > http://grokbase.com/t/kafka/users/14ck70w38q/kafka-consumer-session-ti
> > meouts
> > - but I am not sure if anyone got to the bottom of that.
> >
> > As always, any help would be much appreciated!
> > -Joris.
> >
> >
> > (*) These are the consumer properties I have set explicitly:
> >
> > props.put("auto.commit.interval.ms", "1000");
> > props.put("auto.commit.enable", "false"); // or true.
> > props.put("offsets.storage", "kafka");
> >
> >
> >
> >
> > Winton Capital Management Limited ("Winton") is a limited company
> > registered in England and Wales with its registered offices at 16 Old
> > Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is
> > authorised and regulated by the Financial Conduct Authority in the
> > United Kingdom, registered as an investment adviser with the US
> > Securities and Exchange Commission, registered with the US Commodity
> > Futures Trading Commission and a member of the National Futures
> > Association in the United States.
> >
> > This communication, including any attachments, is confidential and may
> > be privileged. This email is for use by the intended recipient only.
> > If you receive it in error, please notify the sender and delete it.
> > You should not copy or disclose all or any part of this email.
> >
> > This email does not constitute an offer or solicitation and nothing
> > contained in this email constitutes, and should not be construed as,
> > investment advice. Prospective investors should request offering
> > materials and consult their own advisers with respect to investment
> > decisions and inform themselves as to applicable legal requirements,
> > exchange control regulations and taxes in the countries of their
> > citizenship, residence or domicile. Past performance is not indicative
> of future results.
> >
> > Winton takes reasonable steps to ensure the accuracy and integrity of
> > its communications, including emails. However Winton accepts no
> > liability for any materials transmitted. Emails are not secure and
> > cannot be guaranteed to be error free.
> >
>
>
> Winton Capital Management Limited (“Winton”) is a limited company
> registered in England and Wales with its registered offices at 16 Old
> Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is
> authorised and regulated by the Financial Conduct Authority in the United
> Kingdom, registered as an investment adviser with the US Securities and
> Exchange Commission, registered with the US Commodity Futures Trading
> Commission and a member of the National Futures Association in the United
> States.
>
> This communication, including any attachments, is confidential and may be
> privileged. This email is for use by the intended recipient only. If you
> receive it in error, please notify the sender and delete it. You should not
> copy or disclose all or any part of this email.
>
> This email does not constitute an offer or solicitation and nothing
> contained in this email constitutes, and should not be construed as,
> investment advice. Prospective investors should request offering materials
> and consult their own advisers with respect to investment decisions and
> inform themselves as to applicable legal requirements, exchange control
> regulations and taxes in the countries of their citizenship, residence or
> domicile. Past performance is not indicative of future results.
>
> Winton takes reasonable steps to ensure the accuracy and integrity of its
> communications, including emails. However Winton accepts no liability for
> any materials transmitted. Emails are not secure and cannot be guaranteed
> to be error free.
>
