How to reissue a delegated token after max lifetime passes for a spark streaming application on a Kerberized cluster

2019-01-02 Thread Ali Nazemian
Hi, We are using a headless keytab to run our long-running spark streaming application. The token is renewed automatically every 1 day until it hits the max life limit. The problem is token is expired after max life (7 days) and we need to restart the job. Is there any way we can re-issue the toke

Re: Powered By Spark

2019-01-02 Thread Mann Du
Doesn't this suggestion work for you? -- "To add yourself to the list, please email d...@spark.apache.org with your organization name, URL, a list of which Spark components you are using, and a short description of your use case." On Sat, Dec 22, 2018 at 12:13 AM Ascot Moss wrote: > Hi, > > We u

[spark-ml] How to write a Spark Application correctly?

2019-01-02 Thread Pola Yao
Hello Spark Community, I have a dataset of size 20G, 20 columns. Each column is categorical, so I applied string-indexer and one-hot-encoding on every column. After, I applied vector-assembler on all the newly derived columns to form a feature vector for each record, and then feed the feature vect

Re: Python

2019-01-02 Thread Gourav Sengupta
Cool, thanks! On Wed, Jan 2, 2019 at 10:00 AM Hyukjin Kwon wrote: > Yup, it's supported. > > On Wed, 2 Jan 2019, 3:35 pm Gourav Sengupta wrote: > >> Hi, >> Can I please confirm which version of Python 3.x is supported by Spark >> 2.4? >> >> Regards, >> Gourav >> >

Re: Python

2019-01-02 Thread Hyukjin Kwon
Yup, it's supported. On Wed, 2 Jan 2019, 3:35 pm Gourav Sengupta Hi, > Can I please confirm which version of Python 3.x is supported by Spark 2.4? > > Regards, > Gourav >