What does kafka streams groupBy does internally?

2024-01-24 Thread warrior2031
Let's say there's a topic in which chunks of different files are all mixed up represented by a tuple (FileId, Chunk).Chunks of a same file also can be a little out of order.The task is to aggregate all files and store them into some store.The number of files is unbound.In pseudo stream DSL that mig

Kafka Streams LEFT JOIN multiple tables

2024-01-24 Thread Karsten Stöckmann
Hi, we're currently in the process of evaluating Debezium and Kafka as a CDC system for our Postgres database in order to build an indexing solution (i.e. Solr or OpenSearch). Debezium captures changes per table and propagates them into dedicated Kafka topics each. The ingested tables originally

Re: Kafka Streams LEFT JOIN multiple tables

2024-01-24 Thread Dharin Shah
Hi Karsten, Before delving deeper into Kafka Streams, it's worth considering if direct aggregation in the database might be a more straightforward solution, unless there's a compelling reason to avoid it. Aggregating data at the database level often leads to more efficient and maintainable systems

Re: Kafka Streams LEFT JOIN multiple tables

2024-01-24 Thread Karsten Stöckmann
Hi Dharin, thanks so much for getting back and for your suggestions. At the moment I'm not quite sure if aggregation in our database is a viable option. Creating aggregate views seemed like an obvious solution at first, yet Debezium does not support subscribing to publications based on views. We'

Re: Kafka Streams LEFT JOIN multiple tables

2024-01-24 Thread Slathia p
Hi Team, Greetings, Apologies for the delay in reply as I was down with flu. We actually reached out to you for IT/ SAP/ Oracle/ Infor / Microsoft “VOTEC IT SERVICE PARTNERSHIP” “IT SERVICE OUTSOURCING” “ “PARTNER SERVICE SUBCONTRACTING” We have very attractive newly introduce re

Re: Kafka Streams LEFT JOIN multiple tables

2024-01-24 Thread Slathia p
Hi Team, Greetings, Apologies for the delay in reply as I was down with flu. We actually reached out to you for IT/ SAP/ Oracle/ Infor / Microsoft “VOTEC IT SERVICE PARTNERSHIP” “IT SERVICE OUTSOURCING” “ “PARTNER SERVICE SUBCONTRACTING” We have very attractive newly introduce re