I'd use option 2 (Kafka Connect).
Advantages of #2:
- The code is decoupled from the processing code and easier to refactor in
the future. (same as #4)
- The runtime/uptime/scalability of your Kafka Streams app (processing) is
decoupled from the runtime/uptime/scalability of the data ingestion in
Thank Eno,
Yes, I am aware of that. It indeed looks like a very useful feature.
The result of the processing in kafka streams is only a small amount of
data that is require by our service.
Currently it make more sense for us to update the remote database were we
have more data that our applicatio
Hi Shimi,
Could you tell us more about your scenario? Kafka Streams uses embedded
databases (RocksDb) to store it's state, so often you don't need to write
anything to an external database and you can query your streams state directly
from streams. Have a look at this blog if that matches your
Hi Everyone,
I was wondering about writing data to remote database.
I see 4 possible options:
1. Read from a topic and write to the database.
2. Use kafka connect
3. Write from anywhere in kafka streams.
4. Register a CachedStateStore FlushListener that will send a batch of
records
