Re: Regarding the support of pluggable procedures in Iceberg

2021-11-10 Thread Ryan Blue
I think that probably the best way to handle this use case is to have people implement the Iceberg `ProcedureCatalog` API. That's what we want to get upstream into Spark and is a really reasonable (and small) addition to Spark. The problem with adding pluggable procedures to Iceberg is that it is

[ANNOUNCE] Apache Iceberg release 0.12.1

2021-11-10 Thread Kyle Bendickson
I'm pleased to announce the release of Apache Iceberg 0.12.1! Apache Iceberg is an open table format for huge analytic datasets. Iceberg delivers high query performance for tables with tens of petabytes of data, along with atomic commits, concurrent writes, and SQL-compatible table evolution. Thi

Re: Iceberg python library sync

2021-11-10 Thread Jun H.
Hi everyone, Thanks for attending today's python sync meeting and here are meeting notes: https://docs.google.com/document/d/1oMKodaZJrOJjPfc8PDVAoTdl02eGQKHlhwuggiw7s9U/edit#heading=h.8sa2bi5d4lms. Please feel free to add additional notes there. Also, please join the iceberg-python-sync

Regarding the support of pluggable procedures in Iceberg

2021-11-10 Thread Ajantha Bhat
Hi Community! If Iceberg provides a capability to plugin procedures, it will be really helpful for users to plugin their own spark actions to handle their business logic around Iceberg tables. So, can we have a mechanism that allows plugging additional implementations of *org.apache.spark.sql.conn