RE: how to use cluster sparkSession like localSession

2018-11-04 Thread Sun, Keith
Hello, I think you can try with below , the reason is only yarn-cllient mode is supported for your scenario. master("yarn-client") Thanks very much. Keith From: 张万新 Sent: Thursday, November 1, 2018 11:36 PM To: 崔苗(数据与人工智能产品开发部) <0049003...@znv.com> Cc: user Subject: Re: how to use cluster

How to find the temporary views' DDL

2017-10-01 Thread Sun, Keith
Hello, Is there a way to find the DDL of the “temporary” view created in current session with spark sql: For example : create or replace temporary view tmp_v as select c1 from table table_x; “Show create table “ does not work for this case as it is not a table . “Describe” could show the c

RE: A bug in spark or hadoop RPC with kerberos authentication?

2017-08-23 Thread Sun, Keith
.getOrCreate(); Thanks very much. Keith From: 周康 [mailto:zhoukang199...@gmail.com] Sent: 2017年8月22日 20:22 To: Sun, Keith Cc: user@spark.apache.org Subject: Re: A bug in spark or hadoop RPC with kerberos authentication? you can checkout Hadoop**credential class in spark yarn。During spark submit,it will use config on the classpath. I wonder how do you reference your own config?

RE: A bug in spark or hadoop RPC with kerberos authentication?

2017-08-23 Thread Sun, Keith
Finally find the root cause and raise a bug issue in https://issues.apache.org/jira/browse/SPARK-21819 Thanks very much. Keith From: Sun, Keith Sent: 2017年8月22日 8:48 To: user@spark.apache.org Subject: A bug in spark or hadoop RPC with kerberos authentication? Hello , I met this very weird

A bug in spark or hadoop RPC with kerberos authentication?

2017-08-22 Thread Sun, Keith
Hello , I met this very weird issue, while easy to reproduce, and stuck me for more than 1 day .I suspect this may be an issue/bug related to the class loader. Can you help confirm the root cause ? I want to specify a customized Hadoop configuration set instead of those on the class path(we hav