Hi,
In answer to your question I did some tests using broadly your approach.
With regard to your questions:
"but it does not work well because it does not give a temperature average
as you can see in the attached pic.
Why is the average not calculated on temperature?
How can I view data in each w
This is an interesting one.
I have never tried to add --files ...
spark-submit --master yarn --deploy-mode client --files
/etc/hive/conf/hive-site.xml,/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml
Rather, under $SPARK_HOME/conf, I create soft links to the needed XML files
as belo
Thanks everyone. I was able to resolve this.
Here is what I did. Just passed conf file using —files option.
Mistake that I did was reading the json conf file before creating spark session
. Reading if after creating spark session helped it. Thanks once again for your
valuable suggestions
Tha
Hi
I am trying to stream a Kafka topic using createDirectStream(). The Kafka
cluster is SSL enabled. The code for the same is:
***
import findspark
findspark.init('/u01/idp/spark')
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pys
If code running on the executors need some local file like a config file,
then it does have to be passed this way. That much is normal.
On Sat, May 15, 2021 at 1:41 AM Gourav Sengupta
wrote:
> Hi,
>
> once again lets start with the requirement. Why are you trying to pass xml
> and json files to
