Heya, Say I'm building a live auction site, with different products. Different users will bid on different products. And each time they do, I want to update the product's price, so it should always have the latest price in place.
Example: Person 1 bids $3 on Product A, and Person 2 bids $5 on the same product 100 ms later. The second bid arrives first and the price is updated to $5. Then the first bid arrives. I want the price to not be updated in this case, as this bid is older than the one I've already processed. Here's my understanding of how I can achieve this with Kafka Streaming - is my understanding correct? - I have a topic for receiving bids. The topic has N partitions, and I have N replicas of my application which hooks up w/ Kafka Streaming, up and running. - I assume each replica of my app will listen to a different partition of the topic. - A user makes a bid on product A. - This is pushed to the topic with the key bid_a - Another user makes a bid. This is also pushed with the same key (bid_a) - The 2nd bid arrives first, and gets processed. Then the first (older) bid arrives. - Because I'm using a KTable, the timestamp of the messages is extracted, and I'm not shown the older bid because I've already processed the later bid. The older bid is ignored. - All bids on product A go to the same topic partition, and hence the same replica of my app, because they all have the key bid_a. - Because of this, the replica already knows which timestamps it has processed, and is able to ignore the older messages. Is the above understandning correct? Also, what will happen if bid 2 arrived and got processed, and then the particular replica crashed, and was restarted. The restarted replica won't have any memory of which timestamps it has previously processed. So if bid 2 got processed, replica crashed and restarted, and then bid 1 arrived, what would happen in that case? Thanks.
