For us, group ID is a configuration parameter of the application. So we
store it in configuration files (generally on disk) and maintain it there
through our configuration and deployment infrastructure. As you pointed
out, hard coding the group ID into the application is not usually a good
pattern.
If you want to reset, you have a couple choices. One is that you can just
switch group names and start fresh. Another is that you can shut down the
consumer and delete the existing consumer group, then restart. You could
also stop, edit the offsets to set them to something specific (if you need
to roll back to a specific point, for example), and restart.
-Todd
On Wed, Jun 10, 2015 at 1:20 PM, James Cheng <jch...@tivo.com> wrote:
> Hi,
>
> How are people specifying/persisting/resetting the consumer group
> identifier ("group.id") when using the high-level consumer?
>
> I understand how it works. I specify some string and all consumers that
> use that same string will help consume a topic. The partitions will be
> distributed amongst them for consumption. And when they save their offsets,
> the offsets will be saved according to the consumer group. That all makes
> sense to me.
>
> What I don't understand is the best way to set and persist them, and reset
> them if needed. For example, do I simply hardcode the string in my code? If
> so, then all deployed instances will have the same value (that's good). If
> I want to bring up a test instance of that code, or a new installation,
> though, then it will also share the load (that's bad).
>
> If I pass in a value to my instances, that lets me have different test and
> production instances of the same code (that's good), but then I have to
> persist my consumer group id somewhere outside of the process (on disk, in
> zookeeper, etc). Which then means I need some way to manage *that*
> identifier (that's... just how it is?).
>
> What if I decide that I want my app to start over? In the case of
> log-compacted streams, I want to throw away any processing I did and start
> "from the beginning". Do I change my consumer group, which effective resets
> everything? Or do I delete my saved offsets, and then resume with the same
> consumer group? The latter is functionally equivalent to the former.
>
> Thanks,
> -James
>
>
