Re: Detected yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.

Aditya Fri, 07 Oct 2016 02:21:48 -0700

Hi Saurav,

Please share spark-submit command which you used.


On Friday 07 October 2016 02:41 PM, Saurav Sinha wrote:

I am submitting job by spark-submit but still it is giving message.

Please use spark-submit.

Can any one give me resone for this error.

Thanks,
Saurav Sinha

On Thu, Oct 6, 2016 at 3:38 PM, Saurav Sinha <sauravsinh...@gmail.com<mailto:sauravsinh...@gmail.com>> wrote:


    I did not get you I am submitting job by spark-submit but still it
    is giving message.

    Please use spark-submit.

    On Thu, Oct 6, 2016 at 3:35 PM, Sean Owen <so...@cloudera.com
    <mailto:so...@cloudera.com>> wrote:

        Did you search on the internet / read the docs? the error here
        tells you the problem exactly. Your other question is easily
        answerable by looking up PermGen.

        On Thu, Oct 6, 2016 at 10:51 AM Saurav Sinha
        <sauravsinh...@gmail.com <mailto:sauravsinh...@gmail.com>> wrote:

            Hi User,

            I am trying to run spark streaming job in yarn-cluster mode.

            It is failing with code

            val conf = new
            SparkConf().setAppName("XXX"). conf.setMaster("yarn-cluster")

            val ssc = new StreamingContext(conf,
            Seconds(properties.getProperty("batchDurationInSeconds").toInt));


            org.apache.spark.SparkException: Detected yarn-cluster
            mode, but isn't running on a cluster. Deployment to YARN
            is not supported directly by SparkContext. Please use
            spark-submit.
            at
            org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
            at
            
org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:847)
            at
            
org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
            at
            com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
            at
            
com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
            at
            
com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
            at scala.Option.getOrElse(Option.scala:120)
            at
            
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:837)
            at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
            at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at
            
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at
            
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at
            
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
            at
            
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
            at
            org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
            at
            org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
            at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
            Exception in thread "main"
            org.apache.spark.SparkException: Detected yarn-cluster
            mode, but isn't running on a cluster. Deployment to YARN
            is not supported directly by SparkContext. Please use
            spark-submit.
            at
            org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
            at
            
org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:847)
            at
            
org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
            at
            com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
            at
            
com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
            at
            
com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
            at scala.Option.getOrElse(Option.scala:120)
            at
            
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:837)
            at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
            at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at
            
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at
            
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at
            
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
            at
            
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
            at
            org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
            at
            org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
            at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


            Please help.

--Thanks and Regards,


            Saurav Sinha

            Contact: 9742879062

--Thanks and Regards,


    Saurav Sinha

    Contact: 9742879062




--
Thanks and Regards,

Saurav Sinha

Contact: 9742879062

Re: Detected yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.

Reply via email to