hi, Yi,
Probably, I miss something though, we cannot just wrap the udf with
`if (isnull(x)) null else udf(knownnotnull(x))`?
On Fri, Mar 13, 2020 at 6:22 PM wuyi wrote:
> Hi all, I'd like to raise a discussion here about null-handling of
> primitive-type of untyped Scala UDF [ udf(f: AnyRef, da
Hi Takeshi, thanks for your reply.
Before the broken, we only do the null check for primitive types and leave
null value of non-primitive type to UDF itself in case it will be handled
specifically, e.g., a UDF may return something else for null String.
--
Sent from: http://apache-spark-develop
Ah, I see now what the "broken' means. Thanks, Yi.
I personally think the option 1 is the best for existing Spark users to
support the usecase you suggested above.
So, I think this decision depends on how difficult it is to implement "get
Scala lambda parameter types by reflection"
and the complexi
I don't think it's possible to get the parameters by reflection
anymore -- they are lambdas now in the JVM. At least, indeed, I recall
a few people couldn't find a solution back when we added 2.12 support.
This isn't 'new' in that it has always been the case for Scala 2.12.
If there is a better ide
Hi, All.
Apache Spark has been suffered from a known consistency issue on `CHAR`
type behavior among its usages and configurations. However, the evolution
direction has been gradually moving forward to be consistent inside Apache
Spark because we don't have `CHAR` offically. The following is the s
Hi Sean
On Fri, Mar 13, 2020 at 6:46 PM Sean Owen wrote:
> Do you really need a new cluster per user? and if so, why specify N
> workers > M machines? I am not seeing a need for that. I don't even
> think 2 workers on the same host makes sense, as they are both
> managing the same resources; it
On Sat, Mar 14, 2020 at 5:56 PM Andrew Melo wrote:
> Sorry, I'm from a completely different field, so I've inherited a completely
> different vocabulary. So thanks for bearing with me :)
>
> I think from reading your response, maybe the confusion is that HTCondor is a
> completely different reso
I don’t understand this change. Wouldn’t this “ban” confuse the hell out of
both new and old users?
For old users, their old code that was working for char(3) would now stop
working.
For new users, depending on whether the underlying metastore char(3) is
either supported but different from ansi S
