I am submitting job by spark-submit but still it is giving message.
Please use spark-submit.
Can any one give me resone for this error.
Thanks,
Saurav Sinha
On Thu, Oct 6, 2016 at 3:38 PM, Saurav Sinha <sauravsinh...@gmail.com>
wrote:
> I did not get you I am submitting job by spark-submit but still it is
> giving message.
>
> Please use spark-submit.
>
> On Thu, Oct 6, 2016 at 3:35 PM, Sean Owen <so...@cloudera.com> wrote:
>
>> Did you search on the internet / read the docs? the error here tells you
>> the problem exactly. Your other question is easily answerable by looking up
>> PermGen.
>>
>> On Thu, Oct 6, 2016 at 10:51 AM Saurav Sinha <sauravsinh...@gmail.com>
>> wrote:
>>
>>> Hi User,
>>>
>>> I am trying to run spark streaming job in yarn-cluster mode.
>>>
>>> It is failing with code
>>>
>>> val conf = new SparkConf().setAppName("XXX").
>>> conf.setMaster("yarn-cluster")
>>>
>>> val ssc = new StreamingContext(conf, Seconds(properties.getProperty
>>> ("batchDurationInSeconds").toInt));
>>>
>>>
>>> org.apache.spark.SparkException: Detected yarn-cluster mode, but isn't
>>> running on a cluster. Deployment to YARN is not supported directly by
>>> SparkContext. Please use spark-submit.
>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
>>> at org.apache.spark.streaming.StreamingContext$.createNewSparkC
>>> ontext(StreamingContext.scala:847)
>>> at org.apache.spark.streaming.StreamingContext.<init>(Streaming
>>> Context.scala:81)
>>> at com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
>>> at scala.Option.getOrElse(Option.scala:120)
>>> at org.apache.spark.streaming.StreamingContext$.getOrCreate(Str
>>> eamingContext.scala:837)
>>> at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
>>> at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
>>> $SparkSubmit$$runMain(SparkSubmit.scala:672)
>>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit
>>> .scala:180)
>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>> Exception in thread "main" org.apache.spark.SparkException: Detected
>>> yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is
>>> not supported directly by SparkContext. Please use spark-submit.
>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
>>> at org.apache.spark.streaming.StreamingContext$.createNewSparkC
>>> ontext(StreamingContext.scala:847)
>>> at org.apache.spark.streaming.StreamingContext.<init>(Streaming
>>> Context.scala:81)
>>> at com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
>>> at scala.Option.getOrElse(Option.scala:120)
>>> at org.apache.spark.streaming.StreamingContext$.getOrCreate(Str
>>> eamingContext.scala:837)
>>> at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
>>> at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
>>> $SparkSubmit$$runMain(SparkSubmit.scala:672)
>>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit
>>> .scala:180)
>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>
>>>
>>> Please help.
>>> --
>>> Thanks and Regards,
>>>
>>> Saurav Sinha
>>>
>>> Contact: 9742879062
>>>
>>
>
>
> --
> Thanks and Regards,
>
> Saurav Sinha
>
> Contact: 9742879062
>
--
Thanks and Regards,
Saurav Sinha
Contact: 9742879062
