Re: Structured Streaming - not showing records on console

2022-02-01 Thread karan alang
Hi Mich, thnx, seems 'complete' mode is supported only if there are streaming aggregations. I get this error on changing the output mode. pyspark.sql.utils.AnalysisException: Complete output mode not supported when there are no streaming aggregations on streaming DataFrames/Datasets; Project [va

Re: Structured Streaming - not showing records on console

2022-02-01 Thread Mich Talebzadeh
hm. I am trying to recall if I am correct so you should try outpudeMode('complete') with format('console') result = resultMF. \ writeStream. \ outputMode('complete'). \ option("numRows", 1000). \ opti

Structured Streaming - not showing records on console

2022-02-01 Thread karan alang
Hello Spark Experts, I've a simple Structured Streaming program, which reads data from Kafka, and writes on the console. This is working in batch mode (i.e spark.read or df.write), not not working in streaming mode. Details are in the stackoverflow https://stackoverflow.com/questions/70948967/st