+1 On Mon, Sep 11, 2017 at 5:47 PM, Sameer Agarwal <sam...@databricks.com> wrote:
> +1 (non-binding) > > On Thu, Sep 7, 2017 at 9:10 PM, Bryan Cutler <cutl...@gmail.com> wrote: > >> +1 (non-binding) for the goals and non-goals of this SPIP. I think it's >> fine to work out the minor details of the API during review. >> >> Bryan >> >> On Wed, Sep 6, 2017 at 5:17 AM, Takuya UESHIN <ues...@happy-camper.st> >> wrote: >> >>> Hi all, >>> >>> Thank you for voting and suggestions. >>> >>> As Wenchen mentioned and also we're discussing at JIRA, we need to >>> discuss the size hint for the 0-parameter UDF. >>> But I believe we got a consensus about the basic APIs except for the >>> size hint, I'd like to submit a pr based on the current proposal and >>> continue discussing in its review. >>> >>> https://github.com/apache/spark/pull/19147 >>> >>> I'd keep this vote open to wait for more opinions. >>> >>> Thanks. >>> >>> >>> On Wed, Sep 6, 2017 at 9:48 AM, Wenchen Fan <cloud0...@gmail.com> wrote: >>> >>>> +1 on the design and proposed API. >>>> >>>> One detail I'd like to discuss is the 0-parameter UDF, how we can >>>> specify the size hint. This can be done in the PR review though. >>>> >>>> On Sat, Sep 2, 2017 at 2:07 AM, Felix Cheung <felixcheun...@hotmail.com >>>> > wrote: >>>> >>>>> +1 on this and like the suggestion of type in string form. >>>>> >>>>> Would it be correct to assume there will be data type check, for >>>>> example the returned pandas data frame column data types match what are >>>>> specified. We have seen quite a bit of issues/confusions with that in R. >>>>> >>>>> Would it make sense to have a more generic decorator name so that it >>>>> could also be useable for other efficient vectorized format in the future? >>>>> Or do we anticipate the decorator to be format specific and will have more >>>>> in the future? >>>>> >>>>> ------------------------------ >>>>> *From:* Reynold Xin <r...@databricks.com> >>>>> *Sent:* Friday, September 1, 2017 5:16:11 AM >>>>> *To:* Takuya UESHIN >>>>> *Cc:* spark-dev >>>>> *Subject:* Re: [VOTE][SPIP] SPARK-21190: Vectorized UDFs in Python >>>>> >>>>> Ok, thanks. >>>>> >>>>> +1 on the SPIP for scope etc >>>>> >>>>> >>>>> On API details (will deal with in code reviews as well but leaving a >>>>> note here in case I forget) >>>>> >>>>> 1. I would suggest having the API also accept data type specification >>>>> in string form. It is usually simpler to say "long" then "LongType()". >>>>> >>>>> 2. Think about what error message to show when the rows numbers don't >>>>> match at runtime. >>>>> >>>>> >>>>> On Fri, Sep 1, 2017 at 12:29 PM Takuya UESHIN <ues...@happy-camper.st> >>>>> wrote: >>>>> >>>>>> Yes, the aggregation is out of scope for now. >>>>>> I think we should continue discussing the aggregation at JIRA and we >>>>>> will be adding those later separately. >>>>>> >>>>>> Thanks. >>>>>> >>>>>> >>>>>> On Fri, Sep 1, 2017 at 6:52 PM, Reynold Xin <r...@databricks.com> >>>>>> wrote: >>>>>> >>>>>>> Is the idea aggregate is out of scope for the current effort and we >>>>>>> will be adding those later? >>>>>>> >>>>>>> On Fri, Sep 1, 2017 at 8:01 AM Takuya UESHIN <ues...@happy-camper.st> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> We've been discussing to support vectorized UDFs in Python and we >>>>>>>> almost got a consensus about the APIs, so I'd like to summarize >>>>>>>> and call for a vote. >>>>>>>> >>>>>>>> Note that this vote should focus on APIs for vectorized UDFs, not >>>>>>>> APIs for vectorized UDAFs or Window operations. >>>>>>>> >>>>>>>> https://issues.apache.org/jira/browse/SPARK-21190 >>>>>>>> >>>>>>>> >>>>>>>> *Proposed API* >>>>>>>> >>>>>>>> We introduce a @pandas_udf decorator (or annotation) to define >>>>>>>> vectorized UDFs which takes one or more pandas.Series or one >>>>>>>> integer value meaning the length of the input value for 0-parameter >>>>>>>> UDFs. >>>>>>>> The return value should be pandas.Series of the specified type and >>>>>>>> the length of the returned value should be the same as input value. >>>>>>>> >>>>>>>> We can define vectorized UDFs as: >>>>>>>> >>>>>>>> @pandas_udf(DoubleType()) >>>>>>>> def plus(v1, v2): >>>>>>>> return v1 + v2 >>>>>>>> >>>>>>>> or we can define as: >>>>>>>> >>>>>>>> plus = pandas_udf(lambda v1, v2: v1 + v2, DoubleType()) >>>>>>>> >>>>>>>> We can use it similar to row-by-row UDFs: >>>>>>>> >>>>>>>> df.withColumn('sum', plus(df.v1, df.v2)) >>>>>>>> >>>>>>>> As for 0-parameter UDFs, we can define and use as: >>>>>>>> >>>>>>>> @pandas_udf(LongType()) >>>>>>>> def f0(size): >>>>>>>> return pd.Series(1).repeat(size) >>>>>>>> >>>>>>>> df.select(f0()) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> The vote will be up for the next 72 hours. Please reply with your >>>>>>>> vote: >>>>>>>> >>>>>>>> +1: Yeah, let's go forward and implement the SPIP. >>>>>>>> +0: Don't really care. >>>>>>>> -1: I don't think this is a good idea because of the following >>>>>>>> technical >>>>>>>> reasons. >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> -- >>>>>>>> Takuya UESHIN >>>>>>>> Tokyo, Japan >>>>>>>> >>>>>>>> http://twitter.com/ueshin >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Takuya UESHIN >>>>>> Tokyo, Japan >>>>>> >>>>>> http://twitter.com/ueshin >>>>>> >>>>> >>>> >>> >>> >>> -- >>> Takuya UESHIN >>> Tokyo, Japan >>> >>> http://twitter.com/ueshin >>> >> >> > > > -- > Sameer Agarwal > Software Engineer | Databricks Inc. > http://cs.berkeley.edu/~sameerag >
