As Henry mentions in the KIP, we are seeing a great deal of improvements and efficiency by using the mirroring enhancement proposed in this KIP, and believe it would be equally beneficial to everyone that runs Kafka and Kafka Mirror at scale.
I'm bumping up this thread in case there are additional feedback or comments. Thanks, --Vahid On Sat, Feb 13, 2021, 13:59 Ryanne Dolan <ryannedo...@gmail.com> wrote: > Glad to hear that latency and thruput aren't negatively affected somehow. I > would love to see this KIP move forward. > > Ryanne > > On Sat, Feb 13, 2021, 3:00 PM Henry Cai <h...@pinterest.com> wrote: > > > Ryanne, > > > > Yes, network performance is also important. In our deployment, we are > > bottlenecked on the CPU/memory on the mirror hosts. We are using c5.2x > and > > m5.2x nodes in AWS, before the deployment, CPU would peak to 80% but > there > > is enough network bandwidth left on those hosts. Having said that, we > > maintain the same network throughput before and after the switch. > > > > On Fri, Feb 12, 2021 at 12:20 PM Ryanne Dolan <ryannedo...@gmail.com> > > wrote: > > > >> Hey Henry, great KIP. The performance improvements are impressive! > >> However, often cpu, ram, gc are not the metrics most important to a > >> replication pipeline -- often the network is mostly saturated anyway. Do > >> you know how this change affects latency or thruput? I suspect less GC > >> pressure means slightly less p99 latency, but it would be great to see > that > >> confirmed. I don't think it's necessary that this KIP improves these > >> metrics, but I think it's important to show that they at least aren't > made > >> worse. > >> > >> I suspect any improvement in MM1 would be magnified in MM2, given there > >> is a lot more machinery between consumer and producer in MM2. > >> > >> > >> I'd like to do some performance analysis based on these changes. Looking > >> forward to a PR! > >> > >> Ryanne > >> > >> On Wed, Feb 10, 2021, 3:50 PM Henry Cai <h...@pinterest.com> wrote: > >> > >>> On the question "whether shallow mirror is only applied on mirror maker > >>> v1", the code change is mostly on consumer and producer code path, the > >>> change to mirrormaker v1 is very trivial. We chose to modify the > >>> consumer/producer path (instead of creating a new mirror product) so > other > >>> use cases can use that feature as well. The change to mirror maker v2 > >>> should be straightforward as well but we don't have that environment in > >>> house. I think the community can easily port this change to mirror > maker > >>> v2. > >>> > >>> > >>> > >>> On Wed, Feb 10, 2021 at 12:58 PM Vahid Hashemian < > >>> vahid.hashem...@gmail.com> wrote: > >>> > >>>> Retitled the thread to conform to the common format. > >>>> > >>>> On Fri, Feb 5, 2021 at 4:00 PM Ning Zhang <ning2008w...@gmail.com> > >>>> wrote: > >>>> > >>>> > Hello Henry, > >>>> > > >>>> > This is a very interesting proposal. > >>>> > https://issues.apache.org/jira/browse/KAFKA-10728 reflects the > >>>> similar > >>>> > concern of re-compressing data in mirror maker. > >>>> > > >>>> > Probably one thing may need to clarify is: how "shallow" mirroring > is > >>>> only > >>>> > applied to mirrormaker use case, if the changes need to be made on > >>>> generic > >>>> > consumer and producer (e.g. by adding `fetch.raw.bytes` and > >>>> > `send.raw.bytes` to producer and consumer config) > >>>> > > >>>> > On 2021/02/05 00:59:57, Henry Cai <h...@pinterest.com.INVALID> > wrote: > >>>> > > Dear Community members, > >>>> > > > >>>> > > We are proposing a new feature to improve the performance of Kafka > >>>> mirror > >>>> > > maker: > >>>> > > > >>>> > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-712%3A+Shallow+Mirroring > >>>> > > > >>>> > > The current Kafka MirrorMaker process (with the underlying > Consumer > >>>> and > >>>> > > Producer library) uses significant CPU cycles and memory to > >>>> > > decompress/recompress, deserialize/re-serialize messages and copy > >>>> > multiple > >>>> > > times of messages bytes along the mirroring/replicating stages. > >>>> > > > >>>> > > The KIP proposes a *shallow mirror* feature which brings back the > >>>> shallow > >>>> > > iterator concept to the mirror process and also proposes to skip > the > >>>> > > unnecessary message decompression and recompression steps. We > >>>> argue in > >>>> > > many cases users just want a simple replication pipeline to > >>>> replicate the > >>>> > > message as it is from the source cluster to the destination > >>>> cluster. In > >>>> > > many cases the messages in the source cluster are already > >>>> compressed and > >>>> > > properly batched, users just need an identical copy of the message > >>>> bytes > >>>> > > through the mirroring without any transformation or > repartitioning. > >>>> > > > >>>> > > We have a prototype implementation in house with MirrorMaker v1 > and > >>>> > > observed *CPU usage dropped from 50% to 15%* for some mirror > >>>> pipelines. > >>>> > > > >>>> > > We name this feature: *shallow mirroring* since it has some > >>>> resemblance > >>>> > to > >>>> > > the old Kafka 0.7 namesake feature but the implementations are not > >>>> quite > >>>> > > the same. ‘*Shallow*’ means 1. we *shallowly* iterate > RecordBatches > >>>> > inside > >>>> > > MemoryRecords structure instead of deep iterating records inside > >>>> > > RecordBatch; 2. We *shallowly* copy (share) pointers inside > >>>> ByteBuffer > >>>> > > instead of deep copying and deserializing bytes into objects. > >>>> > > > >>>> > > Please share discussions/feedback along this email thread. > >>>> > > > >>>> > > >>>> > >>>> > >>>> -- > >>>> > >>>> Thanks! > >>>> --Vahid > >>>> > >>> >
