Hi,
I'm maintaining a company small internal library that implements some
company specific spark native functions. But since the 2nd preview of
the spark 4.0.0, it seems like some changes has been made that lock the
possibility of implementing native functions in a clean manner.
Since the commit https://issues.apache.org/jira/browse/SPARK-49022 a
Column can't be created from an Expression, but now only from a
ColumnNode which is private. The codebase of spark is fairly huge, and
the number of commits is also impressive, so their might have been a
public api that i missed.
If i didn't miss anything my question is thus, is the will to Spark to
stop user/dev from creating native functions and fallback to UDF with
the drawbacks they are carying. Or is it juste a forgotten use-case?
I know that I can force open the package, and that's what I did, but I
don't like resorting to this kind of hack.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
