#17221 <https://github.com/apache/pulsar/issues/17221> describes an
environment when multiple bookie copies are corrupted, or a Ledger has been
deleted. The loss of schema ledger results in new producers and consumers
not even being created and working properly.

At present, if the integrity of the schema is damaged, it cannot be
repaired because this function does not exist at present.
But the current behavior is that even if the scheme is lost, the connected
producers and consumers can work normally.

*So we need to discuss solutions for the schema that has been lost:*


*1. The first is to skip the non-recoverable ledger error.*
- Description in https://github.com/apache/pulsar/pull/18010: If enabled
autoSkipNonRecoverableData, when the schema ledger is lost, the consumer
and producer can add new schemas without compatibility check(because the
original schema definition cannot be found).

- Description in https://github.com/apache/pulsar/pull/22469: Schema should
be recovered if schema ledger is failing to open due to non-recoverable
ledger error.

The second PR has been Merged, which causes producers and consumers who are
already connected may not work properly.
https://github.com/apache/pulsar/pull/22469#issuecomment-2057198666
Compared with #18010 <https://github.com/apache/pulsar/pull/18010>, there
is no configuration to control this behavior. The default behavior is to
automatically skip when the integrity of the schema is destroyed.

*2. If we don't just skip error, we can fix the schema in some way to
maintain the integrity of the schema version. Even if this requires the
user to manually handle the missing schema, and this topic cannot be used
during this period. This is also better than just skipping the error.
Skipping errors will bring more problems.*

- https://github.com/apache/pulsar/pull/20415 (
https://github.com/apache/pulsar/issues/20414)
Currently this PR tries to fix the missing schema.



*I hope you can discuss these two schemes and what to do with the #22469
<https://github.com/apache/pulsar/pull/22469> that has been Merged.*

*If for the second solution +1, we can talk about
https://github.com/apache/pulsar/issues/20414
<https://github.com/apache/pulsar/issues/20414>. The way to manually fix
the missing schema is described in the `Alternatives`.I think we can add
this functions to the `upload schema` admin api
(https://pulsar.apache.org/docs/3.2.x/admin-api-schemas/#upload-a-schema
<https://pulsar.apache.org/docs/3.2.x/admin-api-schemas/#upload-a-schema>)*


Thanks,
sinan

Reply via email to