The only way I've found to make it work now is by using the current spark
context and changing its configuration using spark-shell options. Which is
really different from pyspark where you can't instantiate a new one, initialize
it etc..
> On Apr 4, 2016, at 18:16, Cyril Scetbon <cyril.scet...@free.fr> wrote:
>
> It doesn't as you can see : http://pastebin.com/nKcMCtGb
> <http://pastebin.com/nKcMCtGb>
>
> I don't need to set the master as I'm using Yarn and I'm on one of the yarn
> nodes. When I instantiate the Spark Streaming Context with the spark conf, it
> tries to create a new Spark Context but even with
> .set("spark.driver.allowMultipleContexts", "true") it doesn't work and
> complains at line 956 that the Spark Context created by spark-shell was not
> initialized with allowMultipleContexts ...
>
>
>> On Apr 4, 2016, at 16:29, Mich Talebzadeh <mich.talebza...@gmail.com
>> <mailto:mich.talebza...@gmail.com>> wrote:
>>
>> Hi Cyril,
>>
>> You can connect to Spark shell from any node. The connection is made to
>> master through --master IP Address like below:
>>
>> spark-shell --master spark://50.140.197.217:7077
>> <http://50.140.197.217:7077/>
>>
>> Now in the Scala code you can specify something like below:
>>
>> val sparkConf = new SparkConf().
>> setAppName("StreamTest").
>> setMaster("local").
>> set("spark.cores.max", "2").
>> set("spark.driver.allowMultipleContexts", "true").
>> set("spark.hadoop.validateOutputSpecs", "false")
>>
>> And that will work
>>
>> Have you tried it?
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>> LinkedIn
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>>
>> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
>>
>>
>> On 4 April 2016 at 21:11, Cyril Scetbon <cyril.scet...@free.fr
>> <mailto:cyril.scet...@free.fr>> wrote:
>> I suppose it doesn't work using spark-shell too ? If you can confirm
>>
>> Thanks
>>
>>> On Apr 3, 2016, at 03:39, Mich Talebzadeh <mich.talebza...@gmail.com
>>> <mailto:mich.talebza...@gmail.com>> wrote:
>>>
>>> This works fine for me
>>>
>>> val sparkConf = new SparkConf().
>>> setAppName("StreamTest").
>>> setMaster("yarn-client").
>>> set("spark.cores.max", "12").
>>> set("spark.driver.allowMultipleContexts", "true").
>>> set("spark.hadoop.validateOutputSpecs", "false")
>>>
>>> Time: 1459669805000 ms
>>> -------------------------------------------
>>> -------------------------------------------
>>> Time: 1459669860000 ms
>>> -------------------------------------------
>>> (Sun Apr 3 08:35:01 BST 2016 ======= Sending messages from rhes5)
>>>
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>> LinkedIn
>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>>>
>>> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
>>>
>>>
>>> On 3 April 2016 at 03:34, Cyril Scetbon <cyril.scet...@free.fr
>>> <mailto:cyril.scet...@free.fr>> wrote:
>>> Nobody has any idea ?
>>>
>>> > On Mar 31, 2016, at 23:22, Cyril Scetbon <cyril.scet...@free.fr
>>> > <mailto:cyril.scet...@free.fr>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I'm having issues to create a StreamingContext with Scala using
>>> > spark-shell. It tries to access the localhost interface and the
>>> > Application Master is not running on this interface :
>>> >
>>> > ERROR ApplicationMaster: Failed to connect to driver at localhost:47257,
>>> > retrying ...
>>> >
>>> > I don't have the issue with Python and pyspark which works fine (you can
>>> > see it uses the ip address) :
>>> >
>>> > ApplicationMaster: Driver now available: 192.168.10.100:43290
>>> > <http://192.168.10.100:43290/>
>>> >
>>> > I use similar codes though :
>>> >
>>> > test.scala :
>>> > --------------
>>> >
>>> > import org.apache.spark._
>>> > import org.apache.spark.streaming._
>>> > val app = "test-scala"
>>> > val conf = new SparkConf().setAppName(app).setMaster("yarn-client")
>>> > val ssc = new StreamingContext(conf, Seconds(3))
>>> >
>>> > command used : spark-shell -i test.scala
>>> >
>>> > test.py :
>>> > -----------
>>> >
>>> > from pyspark import SparkConf, SparkContext
>>> > from pyspark.streaming import StreamingContext
>>> > app = "test-python"
>>> > conf = SparkConf().setAppName(app).setMaster("yarn-client")
>>> > sc = SparkContext(conf=conf)
>>> > ssc = StreamingContext(sc, 3)
>>> >
>>> > command used : pyspark test.py
>>> >
>>> > Any idea why scala can't instantiate it ? I thought python was barely
>>> > using scala under the hood, but it seems there are differences. Are there
>>> > any parameters set using Scala but not Python ?
>>> >
>>> > Thanks
>>> > --
>>> > Cyril SCETBON
>>> >
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> > <mailto:user-unsubscr...@spark.apache.org>
>>> > For additional commands, e-mail: user-h...@spark.apache.org
>>> > <mailto:user-h...@spark.apache.org>
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> <mailto:user-unsubscr...@spark.apache.org>
>>> For additional commands, e-mail: user-h...@spark.apache.org
>>> <mailto:user-h...@spark.apache.org>
>>>
>>>
>>
>>
>
