Can't seem to figure this out. I've tried several different approaches without
success. For example, I've tried setting spark.executor.extraJavaOptions in the
spark-default.conf (prior to starting the spark-shell) but this seems to have
no effect.
Outside of spark-shell (within a java application I wrote), I successfully do
the following:
// Set environment variables for the executors
conf.setExecutorEnv("AWS_ACCESS_KEY_ID", System.getenv("AWS_ACCESS_KEY_ID"));
conf.setExecutorEnv("AWS_SECRET_ACCESS_KEY",
System.getenv("AWS_SECRET_ACCESS_KEY"));
But, because my SparkContext already exists within spark-shell, this really
isn't an option (unless I'm missing something).
Thanks.
Darin.
