Hi Mika,
Thanks a lot for bringing up this discussion.
> Currently, the Java APIs use annotations like @PublicEvolving and
> @Experimental to clearly indicate the stability status of API components.
> This is super helpful for understanding which parts of the API are stable for
> production use and which might change in future releases.
This is indeed missing in the Python API. +1 for this. It should be a
good improvement.
> It's possible to add a `deprecated` directive to the class/function
> docstrings to produce a deprecated warning in the API documentation, but it's
> not widely used.
Actually there exists this kind of mechanism, see the following as an
example which is learned from the practices of other Python projects.
However, since Flink 2.0, we have removed most deprecated APIs[1] and
so it seems missing.
```
def register_table(self, name: str, table: Table):
"""
.. note:: Deprecated in 1.10. Use :func:`create_temporary_view` instead.
"""
warnings.warn("Deprecated in 1.10. Use create_temporary_view
instead.", DeprecationWarning)
self._j_tenv.registerTable(name, table._j_table)
```
Another problem is that sometimes the deprecation is not always
consistent between the Java and Python API. When we deprecate an API
in Java, the author may forget to deprecate it in Python sometimes.
For the improvement to make it an annotation, +1 for it, this may be a
good improvement.
> A warning could be emitted at runtime on function use, for example, but I'm
> not entirely sure how nice that would be.
I agree with Timo on this. There is no compiling phase in Python and
so I guess we could not do too much on this.
Regards,
Dian
[1] https://github.com/apache/flink/pull/25398
On Thu, Feb 20, 2025 at 6:30 PM Timo Walther <twal...@apache.org> wrote:
>
> Hi Mika,
>
> thanks for bringing this up! I was actually quite surprised that we
> don't use any kind of marking in our interfaces in the Python API yet.
>
> Your proposal makes total sense. We should reuse the same annotation
> categories that we use and agreed on in Java. Since the Python Table API
> is a thin layer around the Java one, we should use the same stability
> annotations corresponding to each method.
>
> > The only real wrinkle is that the @Deprecated annotation in Java
> > outputs a warning at compile time if it is used, whereas Python
> > doesn't have a compile time. A warning could be emitted at runtime on
> > function use, for example, but I'm not entirely sure how nice that
> > would be.
>
> I think printing a warning at runtime is fine as long as the "runtime"
> is not the Flink runtime but Flink's pre-flight / pre-cluster phase. So
> maybe we should avoid deprecations within UDFs. But deprecating
> functions that construct the Table API program, and are not on the hot
> (i.e. per record) path, should be fine.
>
> +1 to this effort. Personally, I don't think we need a FLIP. We
> basically make the Python API consistent with previous FLIPs around API
> stability. Also, we bring consistency between Java and Python.
>
> What do others think?
>
> Thanks,
> Timo
>
>
> On 19.02.25 15:31, Mika Naylor wrote:
> > Hi everyone,
> >
> > I'd like to start a discussion about adding API stability annotations to
> > PyFlink, similar to those currently used in the Java APIs (@PublicEvolving,
> > @Experimental, etc.).
> >
> > Currently, the Java APIs use annotations like @PublicEvolving and
> > @Experimental to clearly indicate the stability status of API components.
> > This is super helpful for understanding which parts of the API are stable
> > for production use and which might change in future releases.
> >
> > The Python APIs lack this kind of explicit stability indication. It's
> > possible to add a `deprecated` directive to the class/function docstrings
> > to produce a deprecated warning in the API documentation, but it's not
> > widely used.
> >
> > I would like to port the API stability annotations from the Java project
> > into Python equivalents as decorators for Python classes/functions, and to
> > align the stability annotations between the Java and Python APIs. These
> > decorators would enrich the docstrings of the underlying classes/functions
> > to output consistent sections in the API documentation describing the
> > stability of the class/function as PublicEvolving, Experimental,
> > Deprecated, etc. For example:
> >
> > Example usage:
> > ```python
> >
> > @PublicEvolving()
> > class Table(object):
> > ...
> >
> > @Deprecated(since="2.1.0", detail="Use :func:`Table.get_resolved_schema`
> > instead.")
> > def get_schema(self) -> TableSchema:
> > ...
> > ```
> >
> > Would output in the API documentation a notice that the Table class'
> > interface is public but evolving, and that `get_schema` is deprecated since
> > 2.1.0 in favour of `get_resolved_schema`.
> >
> > The only real wrinkle is that the @Deprecated annotation in Java outputs a
> > warning at compile time if it is used, whereas Python doesn't have a
> > compile time. A warning could be emitted at runtime on function use, for
> > example, but I'm not entirely sure how nice that would be.
> >
> > Thanks for reading! I would love any opinions/feedback on whether this
> > would be useful to developers currently using the Python APIs.
> >
> > Thanks,
> > Mika
>
