spark driver with OOM due to org.apache.spark.status.ElementTrackingStore

He there, We have spark driver running 24x7, and we are continiously getting OOM in spark driver every 10 days. I found org.apache.spark.status.ElementTrackingStore keep 85% of heap usage after analyzing heap dump like this image: [image: image.png] i found these parameter would be the root cause

Is spark fair scheduler is for kubernete?

the official doc, https://spark.apache.org/docs/latest/job-scheduling.html, didn't mention that its working for kubernete cluster? Can anyone quickly answer this? TIA. Jason

sharing class between NonClosableMutableURLClassLoader and MutableURLClassLoader

Hi there, I'm tweaking hive thrift server and spart session to provide custom sql capabilities, and i came across java.lang.ClassNotFoundException to load customer session builder. What i found is that the customer session builder is tried to load by MutableURLClassLoader. I have no idea about w

java.lang.ClassNotFoundException for custom hive authentication

Hi there, I'm leveraging thriftserver to provide sql service, and using custom hive authentication: -- hive.server2.custom.authentication.class com.abc.ABCAuthenticationProvider I've got this error when logging into thrift server. class path was set using --jar option. I guess this

Re: How to convert InternalRow to Row.

And the new API is : > > val encoder = RowEncoder(schema) > > val row = encoder.createDeserializer().apply(internalRow) > > > > Thanks, > > Jia Ke > > > > *From:* Wenchen Fan > *Sent:* Friday, November 27, 2020 9:32 PM > *To:* Jason Jun > *C

How to convert InternalRow to Row.

Hi dev, i'm working on generating custom pipeline on the fly, which means I generate SparkPlan along with each node in my pipeline. So, my pipeline end up with PipeLineRelation extending BaseRelation like: case class PipeLineRelation(schema: StructType, pipeLinePlan: LogicalPlan)(@transient over