Thanks, Hans for the insight. Will use compacted topic.
On Thu, Dec 15, 2016 at 3:53 PM, Hans Jespersen wrote:
> for #2 definitely use a compacted topic. Compaction will remove old
> messages and keep the last update for each key. To use this function you
> will need to publish messages as Key/V
for #2 definitely use a compacted topic. Compaction will remove old
messages and keep the last update for each key. To use this function you
will need to publish messages as Key/Value pairs. Apache Kafka 0.10.1 has
some important fixes to make compacted topics more reliable when scaling to
large nu
Thanks, Kenny for confirming. Message updates I mean to say that for same
document/message there will be updates coming in (for e.g. person details
may change). As you mentioned using the proper key should make that happen
so good on that.
On Thu, Dec 15, 2016 at 1:16 PM, Kenny Gorman wrote:
>
A couple thoughts..
- If you plan on fetching old messages in a non-contiguous manner then this may
not be the best design. For instance, “give me messages from mondays for the
last 3 quarters” is better served with a database. But if you want to say “give
me messages from the last month until
riginal Message-
> From: Susheel Kumar [mailto:susheel2...@gmail.com]
> Sent: Thursday, December 15, 2016 12:00 PM
> To: users@kafka.apache.org
> Subject: Kafka as a database/repository question
>
> Hello Folks,
>
> I am going thru an existing design where Kafka is planned to b
What is the plan for backup and recovery of the kafka data?
-Dave
-Original Message-
From: Susheel Kumar [mailto:susheel2...@gmail.com]
Sent: Thursday, December 15, 2016 12:00 PM
To: users@kafka.apache.org
Subject: Kafka as a database/repository question
Hello Folks,
I am going thru an
Hello Folks,
I am going thru an existing design where Kafka is planned to be utilised in
below manner
1. Messages will pushed to Kafka by producers
2. There will be updates to existing messages on ongoing basis. The
expectation is that all the updates are consolidated in Kafka and the
