While that's easy for some users, we basically want to load up some functions by default into all session catalogues regardless of who made them. We do this with certain rules and strategies using the SparkExtensions, so all apps that run through our submit scripts get a config parameter added and it's transparent to the user. I think we'll probably have to do some forks (at least for the CliDriver), the thriftserver has a bunch of code which doesn't run under "startWithContext" so we may have an issue there as well.
On Wed, Sep 26, 2018, 6:21 PM Mark Hamstra <m...@clearstorydata.com> wrote: > You're talking about users starting Thriftserver or SqlShell from the > command line, right? It's much easier if you are starting a Thriftserver > programmatically so that you can register functions when initializing a > SparkContext and then HiveThriftServer2.startWithContext using that > context. > > On Wed, Sep 26, 2018 at 3:30 PM Russell Spitzer <russell.spit...@gmail.com> > wrote: > >> I've been looking recently on possible avenues to load new functions into >> the Thriftserver and SqlShell at launch time. I basically want to preload a >> set of functions in addition to those already present in the Spark Code. >> I'm not sure there is at present a way to do this and I was wondering if >> anyone had any ideas. >> >> I would basically want to make it so that any user launching either of >> these tools would automatically have access to some custom functions. In >> the SparkShell I can do this by adding additional lines to the init section >> but I think It would be nice if we could pass in a parameter which would >> point to a class with a list of additional functions to add to all new >> session states. >> >> An interface like Spark Sessions Extensions but instead of running during >> Session Init, it would run after session init has completed. >> >> Thanks for your time and I would be glad to hear any opinions or ideas on >> this, >> >
