Re: Hi Team, I put a UDF-Utils jar on Google Cloud Storage, but I can't run it

2021-12-12 Thread bo zhao
Update. My Spark version is 2.3. bo zhao 于2021年12月13日周一 11:24写道: > Hi Team, > I'm doing migration from onPerm to Google Cloud Platform. I have a > UDF-Uitls jar including several udf functions. when code on Perm running > by Spark just need "add jar hdfs://x/home/pa

Hi Team, I put a UDF-Utils jar on Google Cloud Storage, but I can't run it

2021-12-12 Thread bo zhao
Hi Team, I'm doing migration from onPerm to Google Cloud Platform. I have a UDF-Uitls jar including several udf functions. when code on Perm running by Spark just need "add jar hdfs://x/home/path/UDF-Uitls.jar" with SparkSQL. Now I put this jar on cloud storage. when I run SparkSQL with "add jar g

Have you ever meet issue like this?

2021-10-25 Thread bo zhao
Hi Team, Have you ever meeti Exception like this? My production: dataFrame write to hdfs(hive external table path) with parquet type then "msck repair table x" After job finising, I select data by using Spark-SQL, it works fine. However, when I use Hive to select it showed: "Failed with ex

About Spark executs sqlscript

2021-05-13 Thread bo zhao
Hi Team, I've followed Spark community for several years. This is my first time for asking help. I hope you guys can give some experience. I want to develop a spark application with processing a sqlscript file. The data is on BigQuery. For example, the sqlscript is: delete from tableA; insert in