On Wed, Jul 13, 2022 at 10:57 AM Joe F <joefranc...@gmail.com> wrote:
> Hi Rajan, > > > > > For consumers, the message will be sent when there are no more messages > > to > > > read? > > > > > Yes. > > > I think it would be helpful to add a bit more detail on the consumer > switch (sp. Shared consumers). There is a window of transition where > "dispatched-but-not-acked messages" might need to be dispatched again. > >> I think I understand the confusion here, This PIP is an enhancement on top of Topic-termination feature which makes sure to dispatch all the messages to subscribers and wait for their ack until they reach the end of topic. And that also handles all types of subscriptions in the system. Unfortunately, I couldn't find the PIP for topic termination so, I have added explanation of how topic termination works in this PIP ( https://github.com/apache/pulsar/issues/16551) with section: "Topic termination" With this PIP, the broker waits for consumer redirection until the subscriber receives and ack all the messages and then only the broker redirects consumers to Green cluster. > Dispatch will then need to move to the new cluster to continue dispatch > (which creates some interesting issues) or hold up the migration of > consumption to the new cluster till the topic is completely consumed on the > old cluster. What is the behavior here? Would be helpful to be clear on > this. > >> Thanks for the question. I didn't add this point earlier but now, updated PIP with this concern at Broker Changes#8: Once, topic is marked as migrated, brokers will start redirecting producers to new cluster to publish new messages but if consumers are not yet redirected then messages at green cluster can be lost as appropriate subscriptions won't be created yet . Therefore, we can apply default retention policy at Green cluster until blue cluster is migrated. So, Green cluster will retain the messages until blue cluster is completely migrated to green cluster. And then the default retention policy can be removed from Green cluster. > The other point is about how replication would move over (Blue -> Red) and > (Green ->Red) To maintain ordering, will Green delay replication until > Blue has finished replication? Is the expectation that this is operator > managed ? > >> I have updated PIP with the replicator flow as well. This PIP gives an ordering guarantee for the replicator. a. Incoming replication messages from other region's replicator producers to Blue cluster => Will not require special handling. b. Outgoing replication messages from Blue cluster's replicator producers to other regions => will have trade-off of topic unavailability to achieve ordering guarantee. Blue cluster will not redirect producers to green cluster until all existing published messages in the blue cluster replicates to remote regions. > There are potentially other dependencies that might be impacted. Not asking > that all these be solved, but clarity would be nice - .a, b, c features > will work through this migration, x,y.z features will not work and are not > supported by this migration (or will need to be manually dealt with) > >> Sure, I have updated PIP in the last section where it lists down the features this PIP supports. I haven't seen any limitation yet with this PIP but we can add if we see any concerns with the PIP. > -joe > > > On Wed, Jul 13, 2022 at 9:31 AM Rajan Dhabalia <rdhaba...@apache.org> > wrote: > > > On Wed, Jul 13, 2022 at 2:30 AM Enrico Olivelli <eolive...@gmail.com> > > wrote: > > > > > Very interesting. Overall I support this proposal > > > > > > A couple of questions: > > > - What about monitoring the status of the migration? > > > > > There will be a cluster level state. Once all topics are migrated and > > deleted, the broker will change cluster's state to migration completed. > So, > > it will not require any additional monitoring. > > > > > > > - Should we block all maintenance operations on the "blue" cluster ? > > > like deleting/creating stuff > > > > > Broker iterates over all topics and first mark them terminated. So, it > will > > eventually block operations on all topics. > > > > > > > - Should we stop ledger trimming and offloading ? > > > > > As topics will be terminated, topics will not receive any new writes. > so, I > > don't think there will be any issues in that process. > > > > > > > - What about authentication/authorization between the two clusters ? > > > should we provide a dedicated set of credentials for the "migration"? > > > > > New cluster should have identical policies including auth compare to old > > cluster, and transition should be seamless for users without any downtime > > and any user involvement. You can use PIP-136 ( > > https://github.com/apache/pulsar/issues/16424) to synchronize policies > on > > new cluster before redirecting traffic on the cluster. > > > > Thanks, > > Rajan > > > > > > > > > > thanks > > > Enrico > > > > > > Il giorno mer 13 lug 2022 alle ore 10:55 Asaf Mesika > > > <asaf.mes...@gmail.com> ha scritto: > > > > > > > > Few questions > > > > > > > > "CompletableFuture<Position> asyncMigrate();" > > > > Does this method only change the status of the managed ledger? > > > > > > > > "message ManagedLedgerInfo { > > > > > > > > // Flag to check if topic is terminated and migrated to different > > > cluster > > > > optional bool migrated = 4; > > > > > > > > }" > > > > > > > > This flag then is only changed to true when it has finished > migration: > > > i.e. > > > > no new messages were written, all existing consumers finished reading > > all > > > > messages and disconnected and the topic can now be deleted? > > > > > > > > "Broker sends topic migration message to client so, producer/consumer > > at > > > > client side can handle redirection accordingly" > > > > > > > > For producers, the message will be sent the moment the status of the > > > topic > > > > has changed, so all messages from there on will be written to the new > > > > cluster? > > > > For consumers, the message will be sent when there are no more > messages > > > to > > > > read? > > > > > > > > > > > > > > > > On Tue, Jul 12, 2022 at 8:23 PM Rajan Dhabalia <rdhaba...@apache.org > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > We have created PIP-184 which helps users to perform cluster > > migration > > > with > > > > > Apache Pulsar. Cluster migration or Blue-Green cluster deployment > is > > > one of > > > > > the proven solutions to migrate live traffic from one cluster to > > > another. > > > > > One of the examples is applications running on Kubernetes sometimes > > > require > > > > > a Kubernetes cluster upgrade which can cause downtime for the > entire > > > > > application during a Kubernetes cluster upgrade. Blue-green > > deployment > > > is > > > > > an application release model that gradually transfers user traffic > > > from a > > > > > previous version of an app or microservice to a nearly identical > new > > > > > release—both of which are running in production. > > > > > > > > > > The old version can be called the blue environment while the new > > > version > > > > > can be known as the green environment. Once production traffic is > > fully > > > > > transferred from blue to green, blue can standby in case of > rollback > > > or be > > > > > pulled from production and updated to become the template upon > which > > > the > > > > > next update is made. We need such capability in Apache pulsar to > > > migrate > > > > > live traffic from the blue cluster to the green cluster so, > > > eventually, the > > > > > entire traffic moves from the blue cluster to the green cluster > > without > > > > > causing downtime for the topics. > > > > > > > > > > PIP: https://github.com/apache/pulsar/issues/16551 > > > > > > > > > > Thanks, > > > > > Rajan > > > > > > > > > > >
