Hi everyone, I am trying to filter my features based on the spark.mllib ChiSqSelector.
filteredData = vectorizedTestPar.map(lambda lp: LabeledPoint(lp.label, model.transform(lp.features))) However when I do the following I get the error below. Is there any other way to filter my data to avoid this error? filteredDataDF=filteredData.toDF() Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063. I would directly use the spark.ml ChiSqSelector and work with dataframes, but I am on spark 1.4 and using pyspark. So spark.ml's ChiSqSelector is not available to me. filteredData is of type piplelineRDD, if that helps. It is not a regular RDD. I think that may part of why calling toDF() is not working. Thanks, Tobi
