value col is not a member of org.apache.spark.rdd.RDD

Hi I am trying to generate a hierarchy table using Spark GraphX but during runtime i am getting following error. *error: value col is not a member of org.apache.spark.rdd.RDD[(Any, (Int, Any, String, Int, Int))] val empHirearchyDF = empHirearchyExtDF.join(empDF , empDF.col("emp_id") === e

Error while getting RDD partitions for a parquet dataframe in Spark 3

Hi, When migrating to Spark 3, I'm getting a NoSuchElementException exception when getting partitions for a parquet dataframe - The code I'm trying to execute is - val df = sparkSession.read.parquet(inputFilePath) val partitions = df.rdd.partitions and the spark session is created l