Re: [External] How to read-your-writes in a side effect

[Tauzell, Dave]

When you send a message the returned RecordMetadata has an offset.   If you 
know the name of the Consumer Group that is reading these messages there is an 
API to get the latest offset of the Consumer Group so you could poll that and 
wait for the latest Offset to be greater than the offset of your message.

I don't recall what that API is right now.

-Dave

On 10/13/20, 9:50 AM, "Pedro Teixeira" <pedro.teixe...@gmail.com> wrote:

    I was hoping there was an API for at least knowing the consumer progress..

    —
    Pedro Teixeira
    ✉️ i...@pgte.me

    💻 https://github.com/pgte


    👨🏻‍🎓 https://www.linkedin.com/in/pedroteixeira/


    ✍🏻 https://metaduck.com

    ✍🏻 https://pgte.me

    --- original message ---
    On 13 October 2020, 3:42 PM WEST dave.tauz...@surescripts.com wrote:

    Your consumer will need to put some sort of response message on a different 
topic and the API will have to subscribe and wait for that response message. 
You'll need some sort of key so that it can know which one was for that call. I 
would avoid this type of API, however. Instead I would create an API endpoint 
to get the "status" of that side effect and have clients periodically poll for 
that.

    -Dave

    On 10/13/20, 9:22 AM, "pedro.teixe...@gmail.com" <pedro.teixe...@gmail.com> 
wrote:

    Imagine I have a Kafka cluster with one producer and one consumer, and 
behind it an API server. The API allows clients to write (PUT requests). When a 
client writes, it produces an event into Kafka with that data on a stream. Then 
there is a consumer listening on that stream that will be responsible for 
updating an external database used for indexing. This is a side-effect of 
producing that event.

    My question is:

    Is there a way for the API to be able to respond only once the side-effect 
of the said event has occurred?

    In other words, is it possible for me, at the API level, to wait until a 
certain consumer is up-to-date?

    This e-mail and any files transmitted with it are confidential, may contain 
sensitive information, and are intended solely for the use of the individual or 
entity to whom they are addressed. If you have received this e-mail in error, 
please notify the sender by reply e-mail immediately and destroy all copies of 
the e-mail and any attachments.
    --- end of original message ---