Hi, As of today, our company IT is mainly composed of domain specific software (proprietary and homemade). We could like to migrate them one after another to a microservice architecture with Kafka as the data pipeline. With the system now in place it's quite difficult to have a common data flow because there is numerous and various input types and all of them have their own integration within the IT infrastructure.
According to our readings, Kafka has been built for managing huge data trafic so its functionnalities have been thought this way. Hence, some questions arised from our reflection: - we have a low data traffic compared to your figures: around 30 GB a day. Will it be an issue? - we live in a country with low network bandwith, so partitioning over WAN links is a hight risk and we have multiple access points scattered around the country. Would it be possible to configure Kafka to work with such an environment: during network patition, an isolated node becomes its own master, on partition ending data merges, merge quality would be ensured by ensuring access point only write data that he managed. Or is there a better way to deal with this issue? Also, we would like to know what are the minimal server hardware requirements for using Kafka with our data trafic? For information, the idea for this project came to us after reading this enlightening blog post: http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying Thanks for your time Best regards On Tue, Aug 19, 2014 at 10:18 PM, Daniel Compton <d...@danielcompton.net> wrote: > Hi Justin > > It sounds like Kafka could be a good fit for your environment. Are you able > to tell us more about the kinds of applications you will be running? > > Daniel. > >> On 19/08/2014, at 10:53 am, Justin Maltat <justin.mal...@gmail.com> wrote: >> >> Hello, >> >> I'm managing a study to explore possibilities for migrating a monolith >> architecture IT to a service oriented one. >> That said, the company i'm working for is not a web based company, so >> the number of user (i.e: the load) is not the heart of the issue, but >> data/services diversity is. >> >> I'm truly interested in Kafka as the main data pipeline because it'll >> enable us abstracting storages from services, and as a consequence >> having a better data management. >> I would like to engineer a proof of concept around Kafka and Samza but >> before going further into digging, I would like to have your advice >> regarding this project in order to know weather your software will be >> of use to a more 'common' company IT structure. >> >> Thanks in advance for your insights. >> >> >> Best regards >> >> Justin Maltat
