Re: Optimizing Spark interpreter startup

2023-05-03 Thread Jeff Zhang
Hi Vladimir, Have you compared it with spark shell? I think it is similar as spark shell On Wed, May 3, 2023 at 10:12 PM Vladimir Prus wrote: > Hi, > > I was profiling the startup time of Spark Interpreter in our environment, > and it looks like > a total of 5 seconds is spent at this line in >

Optimizing Spark interpreter startup

2023-05-03 Thread Vladimir Prus
Hi, I was profiling the startup time of Spark Interpreter in our environment, and it looks like a total of 5 seconds is spent at this line in SparkScala212Interpreter.scala: sparkILoop.initializeSynchronous() That line, eventually, calls nsc.Global constructor, which spends 5 seconds creatin