Re: [VOTE] SPIP: Add FunctionCatalog

Tue, 09 Mar 2021 01:12:40 -0800

+1, looks great!
Kent Yao 
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubiis a unified multi-tenant JDBC interface for large-scale data processing and analytics, built on top of Apache Spark.

spark-authorizerA Spark SQL extension which provides SQL Standard Authorization for Apache Spark.
spark-postgres A library for reading data from and transferring data to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
spark-func-extrasA library that brings excellent and useful functions from various modern database management systems to Apache Spark.




On 03/9/2021 17:10，Wenchen Fan<cloud0...@gmail.com> wrote：
+1 (binding)

On Tue, Mar 9, 2021 at 1:47 PM Russell Spitzer <russell.spit...@gmail.com> wrote:
+1 (for what it's worth)

Thanks for making such a robust proposal, i'm excited to see the new work coming from this

On Mar 8, 2021, at 11:44 PM, Dongjoon Hyun <dongjoon.h...@gmail.com> wrote:

+1 (binding)

Thank you, Ryan.

Bests,
Dongjoon.


On Mon, Mar 8, 2021 at 5:20 PM Chao Sun <sunc...@apache.org> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 5:13 PM John Zhuge <jzh...@apache.org> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 4:32 PM Holden Karau <hol...@pigscanfly.ca> wrote:
+1 (binding)

On Mon, Mar 8, 2021 at 3:56 PM Ryan Blue <b...@apache.org> wrote:

Hi everyone, I’d like to start a vote for the FunctionCatalog design proposal (SPIP).

The proposal is to add a FunctionCatalog interface that can be used to load and list functions for Spark to call. There are interfaces for scalar and aggregate functions.

In the discussion we’ve come to consensus and I’ve updated the design doc to match how functions will be called:

In addition to produceResult(InternalRow), which is optional, functions can define produceResult methods with arguments that are Spark’s internal data types, like UTF8String. Spark will prefer these methods when calling the UDF using codgen.

I’ve also updated the AggregateFunction interface and merged it with the partial aggregate interface because Spark doesn’t support non-partial aggregates.

The full SPIP doc is here: https://docs.google.com/document/d/1PLBieHIlxZjmoUB0ERF-VozCRJ0xw2j3qKvUNWpWA2U/edit#heading=h.82w8qxfl2uwl

Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll do a final update of the PR and we can merge the API.

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …

--
Ryan Blue
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
John Zhuge

--------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Reply via email to