On Mon, Jan 30, 2023 at 8:51 PM Chesnay Schepler <ches...@apache.org> wrote:
> On 30/01/2023 13:24, Dong Lin wrote: > > it should be OK for > > apache/flink-connector-aws to use non-public API, similar to how classes > > inside Flink can use APIs marked with @Internal. > > > > We can mark related classes with @Internal as appropriate, without > > requiring any new FLIP. > > It's not though, because these APIs don't offer any compatibility > guarantees. > Existing APIs in Flink which are marked with @Internal also don't offer any compatibility guarantee. Yet we are OK to have classes in Flink use them. This is because we (the Flink community developers) have full control of those code in apache/flink and are free to update these code to work with any backward incompatible changes made to those internal API. After all, externalized connectors are Flink project's code, not Flink users' code, and thus can be considered internal code by default I think the same logic apply to those code in apache/flink-connector-aws. Can you explain why it is not OK? > > We want to arrive at a state where connectors are fully independent of > Flink releases. Using non-public APIs runs counter to that. > By "fully independent", I assume you mean that we want to be able to bump up the version of Flink in externalized connectors without having to change those connectors. Strictly speaking, we can not be fully independent because even APIs marked with @Public can still have backward incompatible changes across the major release. Yes we can minimize the changes needed to externalized connectors by using only public PAI. Though it is a nice property, I don't think this is a strict requirement due to the reasons explained above. Maybe we should discuss this in a separate thread to reach an agreement on this topic?
