Hi Himanshu, We looked at MSK when we moved into AWS. Some of the other aspects to consider is that inter-replica network traffic isn't charged for separately under MSK (whereas traffic between EC2 nodes is charged, and at a higher rate when it's inter-AZ) so if you're running a high replica situation, MSK may offer some savings over your own Kafka deployment. You can also get Prometheus monitoring rather simply with it. Likewise it makes upgrading a cluster rather trivial.
A caveat - in some regions, you can only have a two AZ MSK cluster, not three AZ clusters, I encountered this in ap-southeast-2 (Sydney). This means that losing one AZ could prevent ZooKeeper from reaching quorum. Also, if any of your replicas end up under-replicated or you want to reassign partitions etc. you'll still have to do that manually. MSK doesn't really offer anything in terms of per-node recovery more than EC2 already does. In the end, we decided not to use MSK because our desired failover topology didn't quite fit how MSK works, and because we wanted to retain flexibility - an obvious example being that we could upgrade to 2.5 when we needed it, instead of waiting for AWS to catch up - latest supported version is 2.4.1. I'd suggest trialling MSK alongside an existing cluster to get some ideas of how the costs will stack up for you, to decide if it's the right path. Kind regards, Liam Clarke-Hutchinson On Mon, Sep 7, 2020 at 9:27 PM Himanshu Shukla <himanshushukla...@gmail.com> wrote: > The reasons which I could think are > > > 1. I need not take care of the state of the cluster for scenarios when > the broker goes down or the zookeeper goes down. > 2. They are by default deploying the cluster across multiple > Availability zones. > 3. They have their own mirroring policy across the AZs. > > > > On Mon, Sep 7, 2020 at 2:45 PM Tom Black <tom.v.bl...@gmail.com> wrote: > > > > > > > Himanshu Shukla wrote: > > > We are planning to go for amazon MSK instead of having our own self > > > launched cluster on bare EC2 machines. > > > > > > It would be very helpful, if someone who has used it before or know > about > > > it, can share the feedback/observations regarding the same. > > > > > > My main concerns are, > > > Is it worth using Amazon MSK? > > > > They are expensive. I am curious why you'd consider to use the full > > managed service instead of building your own cluster? > > > > Regards. > > > > > -- > Regards, > Himanshu Shukla >
