You should specify cluster manager (--master) and deploy mode
(--deploy-mode) in the spark-submit arguments, specifying this through
SparkConf is too late to switch to yarn cluster mode.
On Fri, Oct 7, 2016 at 5:20 PM, Aditya <aditya.calangut...@augmentiq.co.in>
wrote:
> Hi Saurav,
>
> Please share spark-submit command which you used.
>
>
> On Friday 07 October 2016 02:41 PM, Saurav Sinha wrote:
>
> I am submitting job by spark-submit but still it is giving message.
>
> Please use spark-submit.
>
> Can any one give me resone for this error.
>
> Thanks,
> Saurav Sinha
>
> On Thu, Oct 6, 2016 at 3:38 PM, Saurav Sinha <sauravsinh...@gmail.com>
> wrote:
>
>> I did not get you I am submitting job by spark-submit but still it is
>> giving message.
>>
>> Please use spark-submit.
>>
>> On Thu, Oct 6, 2016 at 3:35 PM, Sean Owen < <so...@cloudera.com>
>> so...@cloudera.com> wrote:
>>
>>> Did you search on the internet / read the docs? the error here tells you
>>> the problem exactly. Your other question is easily answerable by looking up
>>> PermGen.
>>>
>>> On Thu, Oct 6, 2016 at 10:51 AM Saurav Sinha < <sauravsinh...@gmail.com>
>>> sauravsinh...@gmail.com> wrote:
>>>
>>>> Hi User,
>>>>
>>>> I am trying to run spark streaming job in yarn-cluster mode.
>>>>
>>>> It is failing with code
>>>>
>>>> val conf = new SparkConf().setAppName("XXX").
>>>> conf.setMaster("yarn-cluster")
>>>>
>>>> val ssc = new StreamingContext(conf, Seconds(properties.getProperty
>>>> ("batchDurationInSeconds").toInt));
>>>>
>>>>
>>>> org.apache.spark.SparkException: Detected yarn-cluster mode, but isn't
>>>> running on a cluster. Deployment to YARN is not supported directly by
>>>> SparkContext. Please use spark-submit.
>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
>>>> at org.apache.spark.streaming.StreamingContext$.createNewSparkC
>>>> ontext(StreamingContext.scala:847)
>>>> at org.apache.spark.streaming.StreamingContext.<init>(Streaming
>>>> Context.scala:81)
>>>> at com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
>>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.s
>>>> cala:215)
>>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.s
>>>> cala:215)
>>>> at scala.Option.getOrElse(Option.scala:120)
>>>> at org.apache.spark.streaming.StreamingContext$.getOrCreate(Str
>>>> eamingContext.scala:837)
>>>> at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
>>>> at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>>> ssorImpl.java:62)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>> thodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
>>>> $SparkSubmit$$runMain(SparkSubmit.scala:672)
>>>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit
>>>> .scala:180)
>>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>> Exception in thread "main" org.apache.spark.SparkException: Detected
>>>> yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is
>>>> not supported directly by SparkContext. Please use spark-submit.
>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
>>>> at org.apache.spark.streaming.StreamingContext$.createNewSparkC
>>>> ontext(StreamingContext.scala:847)
>>>> at org.apache.spark.streaming.StreamingContext.<init>(Streaming
>>>> Context.scala:81)
>>>> at com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
>>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.s
>>>> cala:215)
>>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.s
>>>> cala:215)
>>>> at scala.Option.getOrElse(Option.scala:120)
>>>> at org.apache.spark.streaming.StreamingContext$.getOrCreate(Str
>>>> eamingContext.scala:837)
>>>> at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
>>>> at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>>> ssorImpl.java:62)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>> thodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
>>>> $SparkSubmit$$runMain(SparkSubmit.scala:672)
>>>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit
>>>> .scala:180)
>>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>
>>>>
>>>> Please help.
>>>> --
>>>> Thanks and Regards,
>>>>
>>>> Saurav Sinha
>>>>
>>>> Contact: 9742879062
>>>>
>>>
>>
>>
>> --
>> Thanks and Regards,
>>
>> Saurav Sinha
>>
>> Contact: 9742879062
>>
>
>
>
> --
> Thanks and Regards,
>
> Saurav Sinha
>
> Contact: 9742879062
>
>
>
>
