On Wed, Jul 13, 2022 at 2:30 AM Enrico Olivelli <eolive...@gmail.com> wrote:
> Very interesting. Overall I support this proposal > > A couple of questions: > - What about monitoring the status of the migration? > There will be a cluster level state. Once all topics are migrated and deleted, the broker will change cluster's state to migration completed. So, it will not require any additional monitoring. > - Should we block all maintenance operations on the "blue" cluster ? > like deleting/creating stuff > Broker iterates over all topics and first mark them terminated. So, it will eventually block operations on all topics. > - Should we stop ledger trimming and offloading ? > As topics will be terminated, topics will not receive any new writes. so, I don't think there will be any issues in that process. > - What about authentication/authorization between the two clusters ? > should we provide a dedicated set of credentials for the "migration"? > New cluster should have identical policies including auth compare to old cluster, and transition should be seamless for users without any downtime and any user involvement. You can use PIP-136 ( https://github.com/apache/pulsar/issues/16424) to synchronize policies on new cluster before redirecting traffic on the cluster. Thanks, Rajan > > thanks > Enrico > > Il giorno mer 13 lug 2022 alle ore 10:55 Asaf Mesika > <asaf.mes...@gmail.com> ha scritto: > > > > Few questions > > > > "CompletableFuture<Position> asyncMigrate();" > > Does this method only change the status of the managed ledger? > > > > "message ManagedLedgerInfo { > > > > // Flag to check if topic is terminated and migrated to different > cluster > > optional bool migrated = 4; > > > > }" > > > > This flag then is only changed to true when it has finished migration: > i.e. > > no new messages were written, all existing consumers finished reading all > > messages and disconnected and the topic can now be deleted? > > > > "Broker sends topic migration message to client so, producer/consumer at > > client side can handle redirection accordingly" > > > > For producers, the message will be sent the moment the status of the > topic > > has changed, so all messages from there on will be written to the new > > cluster? > > For consumers, the message will be sent when there are no more messages > to > > read? > > > > > > > > On Tue, Jul 12, 2022 at 8:23 PM Rajan Dhabalia <rdhaba...@apache.org> > wrote: > > > > > Hi, > > > > > > We have created PIP-184 which helps users to perform cluster migration > with > > > Apache Pulsar. Cluster migration or Blue-Green cluster deployment is > one of > > > the proven solutions to migrate live traffic from one cluster to > another. > > > One of the examples is applications running on Kubernetes sometimes > require > > > a Kubernetes cluster upgrade which can cause downtime for the entire > > > application during a Kubernetes cluster upgrade. Blue-green deployment > is > > > an application release model that gradually transfers user traffic > from a > > > previous version of an app or microservice to a nearly identical new > > > release—both of which are running in production. > > > > > > The old version can be called the blue environment while the new > version > > > can be known as the green environment. Once production traffic is fully > > > transferred from blue to green, blue can standby in case of rollback > or be > > > pulled from production and updated to become the template upon which > the > > > next update is made. We need such capability in Apache pulsar to > migrate > > > live traffic from the blue cluster to the green cluster so, > eventually, the > > > entire traffic moves from the blue cluster to the green cluster without > > > causing downtime for the topics. > > > > > > PIP: https://github.com/apache/pulsar/issues/16551 > > > > > > Thanks, > > > Rajan > > > >
