Turns out one of the other developers wrapped the jobs in script and did a cd to another folder in the script before executing spark-submit.
On 12 June 2015 at 14:20, Matthew Jones <mle...@gmail.com> wrote: > Hmm either spark-submit isn't picking up the relative path or Chronos is > not setting your working directory to your sandbox. Try using "cd > $MESOS_SANDBOX && spark-submit --properties-file props.properties" > > On Fri, Jun 12, 2015 at 12:32 PM Gary Ogden <gog...@gmail.com> wrote: > >> That's a great idea. I did what you suggested and added the url to the >> props file in the uri of the json. The properties file now shows up in the >> sandbox. But when it goes to run spark-submit with "--properties-file >> props.properties" it fails to find it: >> >> Exception in thread "main" java.lang.IllegalArgumentException: requirement >> failed: Properties file props.properties does not exist >> >> >> On 11 June 2015 at 22:17, Matthew Jones <mle...@gmail.com> wrote: >> >>> If you are using chronos you can just put the url in the task json and >>> chronos will download it into your sandbox. Then just use spark-submit >>> --properties-file app.properties. >>> >>> On Thu, 11 Jun 2015 15:52 Marcelo Vanzin <van...@cloudera.com> wrote: >>> >>>> That's not supported. You could use wget / curl to download the file to >>>> a temp location before running spark-submit, though. >>>> >>>> On Thu, Jun 11, 2015 at 12:48 PM, Gary Ogden <gog...@gmail.com> wrote: >>>> >>>>> I have a properties file that is hosted at a url. I would like to be >>>>> able to use the url in the --properties-file parameter when submitting a >>>>> job to mesos using spark-submit via chronos >>>>> >>>>> I would rather do this than use a file on the local server. >>>>> >>>>> This doesn't seem to work though when submitting from chronos: >>>>> >>>>> bin/spark-submit --properties-file >>>>> http://server01/props/app.properties >>>>> >>>>> Inside the properties file: >>>>> spark.executor.memory=256M >>>>> spark.cores.max=1 >>>>> spark.shuffle.consolidateFiles=true >>>>> spark.task.cpus=1 >>>>> spark.deploy.defaultCores=1 >>>>> spark.driver.cores=1 >>>>> spark.scheduler.mode=FAIR >>>>> >>>>> So how do I specify a properties file in a url? >>>>> >>>> >>>> >>>> >>>> -- >>>> Marcelo >>>> >>> >>
