Fwd: Consumer service that supports retry with exponential backoff

[John Walker]

I have a pair of services.  One dispatches commands to the other for
processing.

My consumer sometimes fails to execute commands as a result of transient
errors.  To deal with this, commands are retried after an exponentially
increasing delay up to a maximum of 4 times.  (Delays: 1hr, 2hr, 4hr, 8hr.)
  What's the standard way to set up something like this using Kafka?

The only solution I've found so far is to setup 5 topics (main_topic,
delayed_1hr, delayed_2hr, delayed_4hr,delayed_8hr), and then have pollers
that poll each of these topics, enforce delays, and escalate messages from
one topic to another if errors occur.

Thanks in advance for any pointers you guys can give me,

-John

Bundling already-serialized messages

[John Walker]

I want to do something crazy.  Someone talk me off the ledge:

record Bundle {
  string key;
  array msgs;
}

Producers individually serialize a bunch of messages that share a key, then
serialize a bundle and post to a topic.

A generic Flattener service is configured by startup parameters to listen
to 1...n kafka topics containing bundles, then blindly forward the bundled
messages on to configured output topics one at a time.  (Blindly meaning it
takes the bytes from the array and puts them on the wire.)

Use case:

I have services that respond to small operations (update record, delete
record, etc).  At times, I want batches of ops that are gauranteed not to
be interleaved with other ops for the same key.

To accomplish this, my thought was to position a Flattener in front of each
of the services in question.  Normal, one-off commands get stored in 1-item
bundles, true batchs are bundled into bigger ones.

I don't use a specific field type for the inner messages, because I'd like
to be able to re-use Flattener all over the place

Does this make any sense at all?  Potential drawbacks?

With multiple producers to a topic, is there another way to publish a
series of messages that gaurantees they won't be interleaved?  (It seems
like that goes again the whole spirit of Kafka.)

Thanks!

-John