How to print plan of Structured Streaming DataFrame

Hi Guys I modified StructuredNetworkWordCount to see what the executed plan is, here are my codes: val wordCounts = words.groupBy("value").count() // Start running the query that prints the running counts to the console val query = wordCounts.writeStream .outputMode("complete") .format("cons

Re: Faster and Lower memory implementation toPandas

I have used a very similar script, I think there might be some extra steps that are needed before it could be as robust as toPandas. If you look at _to_corrected_pandas_type in the toPandas (https://github.com/apache/spark/blob/master/python/pyspark/sql/dataframe.py#L1869), this would have to be im

Re: How to print plan of Structured Streaming DataFrame

wordCounts.explain() -> query.explain()? Chang Chen wrote > Hi Guys > > I modified StructuredNetworkWordCount to see what the executed plan is, > here are my codes: > > val wordCounts = words.groupBy("value").count() > > // Start running the query that prints the running counts to the console