Re: toDebugString - RDD Logical Plan

2019-04-23 Thread kanchan tewary
Hello Dylan, Thank you for help. The result do look formatted after making the change. However, from the following code, I was expecting RDD types like MappedRDD and filteredRDD to be present in the lineage. However, I can only see PythonRDD and parallelCollectionRDD in the lineage [I am running i

Re: toDebugString - RDD Logical Plan

2019-04-20 Thread Dylan Guedes
Kanchan, the `toDebugString` looks unformatted because in some scenarios you need to parse it before (can't remember the reason, though). I suggest you to print the RDD Lineage using `print(rdd.toDebugString().decode("utf-8"))` instead (obs: this only occurs in Pyspark). About the other question,