Thanks Philip and Anand for the hints. I fill more comfortable going further now.
On Wed, Aug 20, 2014 at 6:49 PM, Anand Nalya <anand.na...@gmail.com> wrote: > For operating kafka across multiple data centers have a look at > https://kafka.apache.org/08/ops.html and MirrorMaker ( > https://kafka.apache.org/08/tools.html) > > > On 20 August 2014 04:09, Justin Maltat <justin.mal...@gmail.com> wrote: > >> Hi, >> >> As of today, our company IT is mainly composed of domain specific >> software (proprietary and homemade). We could like to migrate them one >> after another to a microservice architecture with Kafka as the data >> pipeline. With the system now in place it's quite difficult to have a >> common data flow because there is numerous and various input types and >> all of them have their own integration within the IT infrastructure. >> >> According to our readings, Kafka has been built for managing huge data >> trafic so its functionnalities have been thought this way. >> Hence, some questions arised from our reflection: >> - we have a low data traffic compared to your figures: around 30 GB a >> day. Will it be an issue? >> - we live in a country with low network bandwith, so partitioning over >> WAN links is a hight risk and we have multiple access points scattered >> around the country. Would it be possible to configure Kafka to work >> with such an environment: during network patition, an isolated node >> becomes its own master, on partition ending data merges, merge quality >> would be ensured by ensuring access point only write data that he >> managed. Or is there a better way to deal with this issue? >> >> Also, we would like to know what are the minimal server hardware >> requirements for using Kafka with our data trafic? >> >> For information, the idea for this project came to us after reading >> this enlightening blog post: >> >> http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying >> >> Thanks for your time >> >> Best regards >> >> On Tue, Aug 19, 2014 at 10:18 PM, Daniel Compton <d...@danielcompton.net> >> wrote: >> > Hi Justin >> > >> > It sounds like Kafka could be a good fit for your environment. Are you >> able to tell us more about the kinds of applications you will be running? >> > >> > Daniel. >> > >> >> On 19/08/2014, at 10:53 am, Justin Maltat <justin.mal...@gmail.com> >> wrote: >> >> >> >> Hello, >> >> >> >> I'm managing a study to explore possibilities for migrating a monolith >> >> architecture IT to a service oriented one. >> >> That said, the company i'm working for is not a web based company, so >> >> the number of user (i.e: the load) is not the heart of the issue, but >> >> data/services diversity is. >> >> >> >> I'm truly interested in Kafka as the main data pipeline because it'll >> >> enable us abstracting storages from services, and as a consequence >> >> having a better data management. >> >> I would like to engineer a proof of concept around Kafka and Samza but >> >> before going further into digging, I would like to have your advice >> >> regarding this project in order to know weather your software will be >> >> of use to a more 'common' company IT structure. >> >> >> >> Thanks in advance for your insights. >> >> >> >> >> >> Best regards >> >> >> >> Justin Maltat >>
