You have (at least) two problems there, stuck processing and blocking processing retry indefinitely. To solve first issue, there is no general answer, it depends on what kind of processing you're doing that gets stuck, that will influence how you can terminate/interrupt it. I assume you use Java or at least JVM. It will help with both issues, to split reading a message and processing a message into separate threads, actors, or whatever your favorite concurrent computation abstraction. For second issue, reading/consuming part can wait, not indefinitely, but until a configurable timeout for processing to complete, and on timeout retry processing, giving work to a new worker.
Kind regards, Stevo Slavic. On Sun, Oct 26, 2014 at 6:17 PM, Srinivas Reddy Kancharla < getre...@gmail.com> wrote: > Thanks for your response. I agree on sending messages to other new topic > and thats what I am doing now. The reason of asking deleting of messages > from log : > Say I have a topic "InitialState_topic1" where one consumer group > mygroupID1 is reading and is sending messages to new topic > "InprogressState_topic1" -> which is read by another consumer group > "mygroupID2". Now one of the consumer thread in mygroupID2 is stuck or hung > in reading one of the partition, I was under impression that there should > be expiry rule which I can set on each message so that I can delete such > message from that partition and put it back to "initialState_topic1" topic > so that other idle threads can pick it up again. > > So if a thread of a consumer group is stuck or hung (with one single > message) and it holds a partition, how can I make other idle thread (from > same consumer group) to take control of that partition so that it can read > other pending messages? > > Thanks and regards, > Srini > > On Sun, Oct 26, 2014 at 12:40 AM, Stevo Slavić <ssla...@gmail.com> wrote: > > > Have group 1 act like a filter, publish to a new topic all messages that > > group 2 should process and then have group 2 actually consume only new > > topic. > > > > Kind regards, > > Stevo Slavic > > On Oct 26, 2014 2:36 AM, "Srinivas Reddy Kancharla" <getre...@gmail.com> > > wrote: > > > > > Hi, > > > > > > I have a scenario where I produce messages for a given topic (say > having > > 10 > > > partitions), and I have consumer group ( say mygroupID1) with 10 > threads > > > reading those 10 partitions. After consuming, I would like to delete > > > specific messages from that topic (i.e. from a given partition). > > > > > > How should I restrict other consumer group say mygroupID2 to read > > messages > > > from the same topic? > > > > > > Thanks for the help, > > > Srini > > > > > >
