Re: Pyspark access to scala/java libraries

2018-07-14 Thread Mohit Jaggi
Trying again…anyone know how to make this work? > On Jul 9, 2018, at 3:45 PM, Mohit Jaggi wrote: > > Folks, > I am writing some Scala/Java code and want it to be usable from pyspark. > > For example: > class MyStuff(addend: Int) { > def myMapFunction(x: Int) = x + addend > } > > I want

Re: Review notification bot

2018-07-14 Thread Holden Karau
Hearing no objections (and in a shout out to @ Nicholas Chammas who initially suggested mention-bot back in 2016) I've set up a copy of mention bot and run it against my own repo (looks like https://github.com/holdenk/spark-testing-base/pull/253 ). If no one objects I’ll ask infra to turn this on

Spark Shortcut

2018-07-14 Thread Deepu Raj
Hi Team, Using Spark 2.3 :paste -raw not working. Do ctrl+D after pasting the code get message //Exiting paste mode, now interpreting. Nothing happens. Please help. Thanks, Deepu

Future of SparkSQL - enterprise grade roadmap?

2018-07-14 Thread t4
Are there any plans to make Sparksql have the necessary enterprise grade features to join the ranks of Presto/HiveServer2? Some key things missing from SparkSQL: Role Based Access control authorization (either Sql standard or Apache Ranger) LDAP user/group filters Data masking plugin DoAs imperson