Hi Daniel, I'd like to resurface this KIP in case you're still interested in pursuing it. I know it's been a while since you published it, and it hasn't received much attention, but I'm hoping we can give it a try now and finally put this long-standing bug to rest. To that end, I have some thoughts about the proposal. This isn't a complete review, but I wanted to give enough to get the ball rolling:
1. Some environments with firewalls or strict security policies may not be able to bring up a REST server for each MM2 node. If we decide that we'd like to use the Connect REST API (or even just parts of it) to address this bug with MM2, it does make sense to eventually make the availability of the REST API a hard requirement for running MM2, but it might be a bit too abrupt to do that all in a single release. What do you think about making the REST API optional for now, but noting that it will become required in a later release (probably 4.0.0 or, if that's not enough time, 5.0.0)? We could choose not to bring the REST server for any node whose configuration doesn't explicitly opt into one, and maybe log a warning message on startup if none is configured. In effect, we'd be marking the current mode (no REST server) as deprecated. 2. I'm not sure that we should count out the "Creating an internal-only derivation of the Connect REST API" rejected alternative. Right now, the single source of truth for the configuration of a MM2 cluster (assuming it's being run in dedicated mode, and not as a connector in a vanilla Connect cluster) is the configuration file used for the process. By bringing up the REST API, we'd expose endpoints to modify connector configurations, which would not only add complexity to the operation of a MM2 cluster, but even qualify as an attack vector for malicious entities. Thanks to KIP-507 we have some amount of security around the internal-only endpoints used by the Connect framework, but for any public endpoints, the Connect REST API doesn't come with any security out of the box. 3. Small point, but with support for exactly-once source connectors coming out in 3.3.0, it's also worth noting that that's another feature that won't work properly with multi-node MM2 clusters without adding a REST server for each node (or some substitute that accomplishes the same goal). I don't think this will affect the direction of the design discussion too much, but it does help strengthen the motivation. Cheers, Chris On 2021/02/18 15:57:36 Dániel Urbán wrote: > Hello everyone, > > * Sorry, I meant KIP-710. > > Right now the MirrorMaker cluster is somewhat unreliable, and not > supporting running in a cluster properly. I'd say that fixing this would be > a nice addition. > Does anyone have some input on this? > > Thanks in advance > Daniel > > Dániel Urbán <ur...@gmail.com> ezt írta (időpont: 2021. jan. 26., K, > 15:56): > > > Hello everyone, > > > > I would like to start a discussion on KIP-709, which addresses some > > missing features in MM2 dedicated mode. > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters > > > > Currently, the dedicated mode of MM2 does not fully support running in a > > cluster. The core issue is that the Connect REST Server is not included in > > the dedicated mode, which makes follower->leader communication impossible. > > In some cases, this results in the cluster not being able to react to > > dynamic configuration changes (e.g. dynamic topic filter changes). > > Another smaller detail is that MM2 dedicated mode eagerly resolves config > > provider references in the Connector configurations, which is undesirable > > and a breaking change compared to vanilla Connect. This can cause an issue > > for example when there is an environment variable reference, which contains > > some host-specific information, like a file path. The leader resolves the > > reference eagerly, and the resolved value is propagated to other MM2 nodes > > instead of the reference being resolved locally, separately on each node. > > > > The KIP addresses these by adding the Connect REST Server to the MM2 > > dedicated mode for each replication flow, and postponing the config > > provider reference resolution. > > > > Please discuss, I know this is a major change, but also an important > > feature for MM2 users. > > > > Daniel > > >
