Hi John, It doesn't seem like you care for the ordering (since you're using multiple "fallback" topics that are processed in parallel if I understood you correctly), but the alternative would be to implement the backoff using the same topic and consumer. We're using the "pausing" feature of the consumer for that purpose ( https://kafka.apache.org/0102/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#pause(java.util.Collection) ) to make sure that we maintain the ordering on the partitions with events that cannot be immediately processed (we're pausing them waiting e.g. for the external service to become available again, which is fine for us), while other partitions can still progress normally.
M. On 9 October 2017 at 22:41, John Walker <johnwalk...@gmail.com> wrote: > I have a pair of services. One dispatches commands to the other for > processing. > > My consumer sometimes fails to execute commands as a result of transient > errors. To deal with this, commands are retried after an exponentially > increasing delay up to a maximum of 4 times. (Delays: 1hr, 2hr, 4hr, 8hr.) > What's the standard way to set up something like this using Kafka? > > The only solution I've found so far is to setup 5 topics (main_topic, > delayed_1hr, delayed_2hr, delayed_4hr,delayed_8hr), and then have pollers > that poll each of these topics, enforce delays, and escalate messages from > one topic to another if errors occur. > > Thanks in advance for any pointers you guys can give me, > > -John >
