Hi Community!

If Iceberg provides a capability to plugin procedures, it will be really
helpful for users to plugin their own spark actions to handle their
business logic around Iceberg tables.
So, can we have a mechanism that allows plugging additional implementations
of *org.apache.spark.sql.connector.iceberg.catalog.Procedure *
for all users of SparkCatalog and SparkSessionCatalog by just dropping an
additional jar ?

Without this feature, users can still add their custom procedure by
extending *SparkCatalog* and/or *SparkSessionCatalog* and override
*loadProcedure. *Which requires users to configure the subclasses of
Spark[Session]Catalog in their Spark configuration. This way it is a lot of
work and it is not a clean way to handle this.

Another option is to add these custom procedures as UDF, but UDF is meant
to be column related. It doesn't make sense to have UDF for spark actions.


*So, I want to know what most of you think about having pluggable
procedures in Iceberg? Does this feature solve your problems too?*

Thanks,
Ajantha

Reply via email to