Mike, a couple things you might take a look at: Kafka Connect might be useful for writing to your DB. Your consumer->producer->DB flow seems to fit well with Connect.
Transactions might be useful for ensuring that your application runs to completion before committing records. It sounds like your application outputs a json file, perhaps after doing something that takes a long time, and you are worried about sending Kafka messages until the entire json file is written out. With transactions, you might be able to send messages while the application is running and then close the transaction at the end (instead or writing to a file). That said, I've seen scenarios where large files are stored in a data store somewhere, and then Kafka is used to pass around the location of the files instead of the files themselves. If this is your scenario, it can be a good model when files are large, since Kafka is sorta inherently bad at dealing with large messages. For example, this model is common with video processing pipelines. Ryanne On Fri, Dec 7, 2018, 5:55 AM Mikael Petterson <mikael.petter...@ericsson.com wrote: > Hi, > We have one application that produces various data ( dataset1, dataset 2 > ....) at various sites when our application is executed. Currently > dataset1, dataset2 ..... is stored into separate *.json data files for each > execution. > > We don't want to transfer the data when we are running the application. > The reason for it is that > application might not be able to connect to database, for various reasons. > We want the producer of data to notify consumer ( just a central one if > possible) that it has data and where data is located. Then consumer get it. > Or is there a better way..... Consumer will then push the data for one > execution to the db. > > Our application can run at various sites. > > > Site 1 Producer1 -------> Consumer1 ---> db > Site 2 Producer1 -------> Consumer1 ---> db > Site 2 Producer2 -------> Consumer1 ---> db > Site 3 Producer1 -------> Consumer1 ---> db > ... > Site x Producerx --------> Consumer1 ---> db > > I need user input if someone else out there has used Kafka in this way? Is > it recommended? Or is there alternatives? > > Br > > //mike >
