How to calculate the spark.kryoserializer.buffer.max?

during the runtime? Appreciate your any suggestions. BR. Arthur Li - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

How to estimate the executor memory size according by the data

memory size by the data size. Any options I appreciated. Arthur Li - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Question about relationship between number of files and initial tasks(partitions)

Hi Sparkers, I noticed that in my spark application, the number of tasks in the first stage is equal to the number of files read by the application(at least for Avro) if the number of cpu cores is less than the number of files. Though If cpu cores are more than number of files, it's usually equal