For operating kafka across multiple data centers have a look at https://kafka.apache.org/08/ops.html and MirrorMaker ( https://kafka.apache.org/08/tools.html)
On 20 August 2014 04:09, Justin Maltat <justin.mal...@gmail.com> wrote: > Hi, > > As of today, our company IT is mainly composed of domain specific > software (proprietary and homemade). We could like to migrate them one > after another to a microservice architecture with Kafka as the data > pipeline. With the system now in place it's quite difficult to have a > common data flow because there is numerous and various input types and > all of them have their own integration within the IT infrastructure. > > According to our readings, Kafka has been built for managing huge data > trafic so its functionnalities have been thought this way. > Hence, some questions arised from our reflection: > - we have a low data traffic compared to your figures: around 30 GB a > day. Will it be an issue? > - we live in a country with low network bandwith, so partitioning over > WAN links is a hight risk and we have multiple access points scattered > around the country. Would it be possible to configure Kafka to work > with such an environment: during network patition, an isolated node > becomes its own master, on partition ending data merges, merge quality > would be ensured by ensuring access point only write data that he > managed. Or is there a better way to deal with this issue? > > Also, we would like to know what are the minimal server hardware > requirements for using Kafka with our data trafic? > > For information, the idea for this project came to us after reading > this enlightening blog post: > > http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying > > Thanks for your time > > Best regards > > On Tue, Aug 19, 2014 at 10:18 PM, Daniel Compton <d...@danielcompton.net> > wrote: > > Hi Justin > > > > It sounds like Kafka could be a good fit for your environment. Are you > able to tell us more about the kinds of applications you will be running? > > > > Daniel. > > > >> On 19/08/2014, at 10:53 am, Justin Maltat <justin.mal...@gmail.com> > wrote: > >> > >> Hello, > >> > >> I'm managing a study to explore possibilities for migrating a monolith > >> architecture IT to a service oriented one. > >> That said, the company i'm working for is not a web based company, so > >> the number of user (i.e: the load) is not the heart of the issue, but > >> data/services diversity is. > >> > >> I'm truly interested in Kafka as the main data pipeline because it'll > >> enable us abstracting storages from services, and as a consequence > >> having a better data management. > >> I would like to engineer a proof of concept around Kafka and Samza but > >> before going further into digging, I would like to have your advice > >> regarding this project in order to know weather your software will be > >> of use to a more 'common' company IT structure. > >> > >> Thanks in advance for your insights. > >> > >> > >> Best regards > >> > >> Justin Maltat >
