[
https://issues.apache.org/jira/browse/SPARK-16253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-16253.
-------------------------------
Resolution: Not A Problem
> make spark sql compatible with hive sql that using python script transform
> like using 'xxx.py'
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-16253
> URL: https://issues.apache.org/jira/browse/SPARK-16253
> Project: Spark
> Issue Type: Task
> Components: SQL
> Affects Versions: 1.6.2
> Reporter: zenglinxi
>
> Some hive sql like:
> {quote}
> add file /tmp/spark_sql_test/test.py;
> select transform(cityname) using 'test.py' as (new_cityname) from
> test.spark2_orc where dt='20160622' limit 5 ;
> {quote}
> can't be executed by spark sql directly, since it will return error like:
> {quote}
> 16/06/26 11:01:28 INFO codegen.GenerateUnsafeProjection: Code generated in
> 19.054534 ms
> 16/06/26 11:01:28 ERROR execution.ScriptTransformationWriterThread:
> /bin/bash: test.py: command not found
> {quote}
> and the sql works fine in hive with MR.
> Lots of ETL can't be moved from hive to spark sql because of this problem.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
