Windows function examples in pyspark

Hi to all, Im trying to use some windows functions (ntile and percentRank) for a Dataframe but i dont know how to use them. Does anyone can help me with this please? in the Python API documentation there are no examples about it. In specific, im trying to get quantiles of a numeric field in my

Pyspark not working on yarn-cluster mode

Hi to all, Is there any way to run pyspark scripts with yarn-cluster mode without using the spark-submit script? I need it in this way because i will integrate this code into a django web app. When i try to run any script in yarn-cluster mode i got the following error : org.apache.spark.SparkExc

Remote spark-submit not working with YARN

I'm trying to submit a spark job from a different server outside of my Spark Cluster (running spark 1.4.0, hadoop 2.4.0 and YARN) using the spark-submit script : spark/bin/spark-submit --master yarn-client --executor-memory 4G myjobScript.py The think is that my application never pass from the ac