print DataFrame.show(100) to text file at HDFS
Use .limit on the dataframe followed by .write
On Apr 14, 2019, at 5:10 AM, Chetan Khatri mailto:chetan.opensou...@gmail.com> > wrote:
Nuthan,
Thank you for reply. the solution proposed will give everything. for me is like
one Datafram
Use .limit on the dataframe followed by .write
On Apr 14, 2019, 5:10 AM, at 5:10 AM, Chetan Khatri
wrote:
>Nuthan,
>
>Thank you for reply. the solution proposed will give everything. for me
>is
>like one Dataframe show(100) in 3000 lines of Scala Spark code.
>However, yarn logs --applicationId
Nuthan,
Thank you for reply. the solution proposed will give everything. for me is
like one Dataframe show(100) in 3000 lines of Scala Spark code.
However, yarn logs --applicationId > 1.log also gives all
stdout and stderr.
Thanks
On Sun, Apr 14, 2019 at 10:30 AM Nuthan Reddy
wrote:
> Hi Chet
Hi Chetan,
You can use
spark-submit showDF.py | hadoop fs -put - showDF.txt
showDF.py:
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("Write stdout").getOrCreate()
spark.sparkContext.setLogLevel("OFF")
spark.table("").show(100,truncate=false)
But is there any sp
