local[3] spawns 3 threads on 1 core :) Thanks Best Regards
On Fri, Feb 20, 2015 at 12:50 PM, bit1...@163.com <bit1...@163.com> wrote: > Thanks Akhil, you are right. > I checked and find that I have only 1 core allocated to the program.... > I am running on a visual machine,and only allocate one processor to it(1 > core per processor), so even if I have specified --total-executor-cores 3 > in the submit script, the application will still only be allocated one > processor. > > This leads to me another question: > Although I have only one core, If I have specified the master and executor > as --master local[3] --executor-memory 512M --total-executor-cores 3. Since > I have only one core, why does this work? > > ------------------------------ > bit1...@163.com > > > *From:* Akhil Das <ak...@sigmoidanalytics.com> > *Date:* 2015-02-20 15:13 > *To:* bit1...@163.com > *CC:* user <user@spark.apache.org> > *Subject:* Re: Spark streaming doesn't print output when working with > standalone master > While running the program go to your clusters webUI (that runs on 8080, > prolly at hadoop.master:8080) and see how many cores are allocated to the > program, it should be >= 2 for the stream to get processed. > > > [image: Inline image 1] > > > > Thanks > Best Regards > > On Fri, Feb 20, 2015 at 9:29 AM, bit1...@163.com <bit1...@163.com> wrote: > >> Hi, >> I am trying the spark streaming log analysis reference application >> provided by Databricks at >> https://github.com/databricks/reference-apps/tree/master/logs_analyzer >> When I deploy the code to the standalone cluster, there is no output at >> will with the following shell script.Which means, the windowDStream has 0 >> RDDs >> ./spark-submit --deploy-mode client --name LogAnalyzerStreaming --master >> spark://hadoop.master:7077 --executor-memory 512M --total-executor-cores 3 >> --class >> spark.examples.databricks.reference.apps.loganalysis.LogAnalyzerStreaming >> LogApp.jar >> >> But, when I change --master to be --master local[3], the program starts >> to work fine. Can anyone have some advice? Thanks! >> ./spark-submit --deploy-mode client --name LogAnalyzerStreaming --master >> local[3] --executor-memory 512M --total-executor-cores 3 --class >> spark.examples.databricks.reference.apps.loganalysis.LogAnalyzerStreaming >> LogApp.jar >> >> >> object LogAnalyzerStreaming { >> >> val WINDOW_LENGTH = new Duration(12 * 1000) >> val SLIDE_INTERVAL = new Duration(6 * 1000) >> >> def main(args: Array[String]) { >> val sparkConf = new SparkConf().setAppName("Log Analyzer Streaming in >> Scala") >> val sc = new SparkContext(sparkConf) >> val streamingContext = new StreamingContext(sc, SLIDE_INTERVAL) >> >> val logLinesDStream = streamingContext.socketTextStream("localhost", >> 9999) >> >> val accessLogsDStream = >> logLinesDStream.map(ApacheAccessLog.parseLogLine).cache() >> val windowDStream = accessLogsDStream.window(WINDOW_LENGTH, >> SLIDE_INTERVAL) >> >> windowDStream.foreachRDD(accessLogs => { >> if (accessLogs.count() == 0) { >> println("No access com.databricks.app.logs received in this time >> interval") >> } else { >> // Calculate statistics based on the content size. >> val contentSizes = accessLogs.map(log => log.contentSize).cache() >> println("Content Size Avg: %s, Min: %s, Max: %s".format( >> contentSizes.reduce(_ + _) / contentSizes.count, >> contentSizes.min, >> contentSizes.max >> )) >> >> streamingContext.start() >> streamingContext.awaitTermination() >> } >> } >> ------------------------------ >> >> > [image: 提示图标] 邮件带有附件预览链接，若您转发或回复此邮件时不希望对方预览附件，建议您手动删除链接。 > 共有 *1* 个附件 > image.png(13K) 极速下载 > <http://preview.mail.163.com/xdownload?filename=image.png&mid=xtbBERqmgVD%2Ba54h8AAAsm&part=3&sign=6ae34eba8ee23b742de1031cde09ee34&time=1424414893&uid=bit1129%40163.com> > 在线预览 > <http://preview.mail.163.com/preview?mid=xtbBERqmgVD%2Ba54h8AAAsm&part=3&sign=6ae34eba8ee23b742de1031cde09ee34&time=1424414893&uid=bit1129%40163.com> > >
