Makes sense, just wondering if the behavior should only be enabled when config.isSchemaLedgerForceRecovery() == true (PIP-327)?
-Lari On 2024/10/09 04:51:23 Rajan Dhabalia wrote: > >> When the schemas of a topic are lost, all of the messages in the topic > can not be consumed successfully, and producers can not publish messages > anymore. > > Well, there are already many issues created related to the lost schema > ledgers so, it is a very well known issue that the topic becomes > unavailable and producers are not able to publish messages anymore just > because of server issues which is extremely critical for mission critical > usecases and not acceptable as well. Because of that brokers must be fault > tolerant to recover in case of missing ledgers. Broker already handles such > issues during manged-ledger or managed-cursor legers but it should also > handle in case of schema ledger as well and make sure that broker won't > impact topic's unavailability. It is also very important to just not rely > on operational effort to fix such unavailability issues manually but broker > should have a mechanism to recover by itself, > Therefore, this PR is important to make sure tenants don't face topic > unavailability due to well known issues of missing schema ledgers. > > Thanks, > Rajan > > On Tue, Oct 8, 2024 at 8:29 PM Yubiao Feng > <yubiao.f...@streamnative.io.invalid> wrote: > > > Hi all > > > > Background: When the schemas of a topic are lost, all of the messages in > > the topic can not be consumed successfully, and producers can not publish > > messages anymore. This mechanism alerts users to try to recover their > > schemas or recreate their topics. > > > > https://github.com/apache/pulsar/pull/23395 added a patch: producers will > > rebuild schemas if the original schemas were lost, which will mix the old > > schema and new schema as the same schema ID. For example: > > - send M1 with schema `Int32`, get schema id: `1` > > - send M2 with schema `String`, get schema id: `2` > > - schemas are lost > > - send M3 with schema `String`, get schema id `1` > > > > The messages `M1` and `M3` use different schemas, but have the same schema > > id, but users assume all things are fine, which is dangerous. So I want to > > revert this PR. > > > > Thanks > > Yubiao Feng > > >
