I would be in favour of changing interfaces only between major versions. Otherwise we risk that existing setups break when upgrading to the latest minor version.
+1 for the outreach. Cheers, Till On Wed, Oct 23, 2019 at 1:11 PM Zili Chen <wander4...@gmail.com> wrote: > Thanks for your clarification Till. > > For terminology pluggable and customizable, I am mainly concerning about > interface audience issue. Pluggable means we have multiple > high-availability > implementation but closed to extend in user scope; customizable means > high-availability interface are stable and user-facing. > > I agree that we should try to keep backward compatibility if possible. I > start > this thread due to being confused how I progress FLINK-10333. Specifically, > how to deal with compatibility things. > > Given that it is an internal interface I think it is reasonable we > evolve it for > supporting leader store based high-availability storage in a minor version > bump. I'm going to start a discuss thread next week for gathering wider > feedbacks beyond the original (maybe limited) JIRA, which also calls of > review among community members. What do you think? > > Best, > tison. > > > Till Rohrmann <trohrm...@apache.org> 于2019年10月21日周一 下午5:21写道： > > > Hi Tison, > > > > I'm not sure whether I fully understand your distinction between > > customizable and pluggable. Maybe you could clarify your ideas a bit > > because you seem to favour support for pluggable implementations. > > > > Maybe let me try to answer some other questions you raised. With the > > HighAvailabilityServices interface and the functionality to load custom > > implementations specified via `high-availability: FQDN` it is indeed > > possible to provide a custom implementation of the HA services. This is, > > however, pretty much a power user feature where users have to implement > > against an internal API. As such, it can be subject to change and we > don't > > give guarantees that these interfaces won't change. Of course, if > possible, > > we should extend/change it in a way that we guarantee backwards > > compatibility. > > > > You are right that at the moment this interface is not stable enough for > > being public API. Once this changes and we are happy with it, then we can > > think about making it a public API and documenting it properly. Afaik, > > there is no documentation how to implement your own HA services at the > > moment. This underlines as well that this interface is an internal API. > > > > Cheers, > > Till > > > > On Fri, Oct 18, 2019 at 5:56 AM Zili Chen <wander4...@gmail.com> wrote: > > > > > Another perspective is that a stable, carefully-designed interface with > > > clear semantic > > > could be safer to customize. > > > > > > Following the discussion in FLINK-10333 our JobGraphStore is actually > > > required > > > performing write operation only with leadership, which is a basic > > > requirement > > > for coordination rather than an implementation detail. > > > Thus it depends on LeaderElectionService(in the design, we narrow the > > > specific interface > > > as LeaderStore). HighAvailabilityServices#getJobGraphStore() infers a > > > implicit field for > > > that which is hard to express the relationship between them. > > > > > > If the interface is unstable(also we introduce a ClientHAService for > > > separate concern and > > > have to keep b/w comp. for customized), we'd better keep it internal > for > > > freely evolution. > > > And when we try to support customized, it would be helpful to start a > > > proposal to revisit > > > the interface to be well-designed and stable. ref[2]. > > > > > > In short, current high-availability services as well as > > > runtime/coordination is still under > > > development and active evolution. It is possibly not a good time for > make > > > it public and > > > customizable. > > > > > > Best, > > > tison. > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-10333 > > > [2] > > > > > > > > > https://github.com/apache/pulsar/wiki/PIP-45%3A-Pluggable-metadata-interface > > > > > > > > > Zili Chen <wander4...@gmail.com> 于2019年10月17日周四 下午8:13写道： > > > > > > > A challenge is how we ensure the support for customized > implementation. > > > > When > > > > we introduce JobGraphStore#releaseJobGraph we actually change quite a > > bit > > > > codepath > > > > in Dispatcher. While we are unable to test arbitrarily customized > > > > implementation our > > > > compatibility promise is actually no more than compilation > compatible. > > > > > > > > Customer should still be required to be familiar with implementation > > > > details to figure > > > > out the fitment when they bump Flink version. This effort requires > also > > > > and no extra > > > > when we support pluggable strategy. In another word, a customized > > support > > > > tends > > > > to hide the challenge when customer want to use their own > > implementation. > > > > > > > > > >
