We know that the JobScheduler have the function to assign the straggle task to another node. - only if you enable and configure spark.speculation
On 20 October 2015 at 15:20, Triones,Deng(vip.com) <[email protected] > wrote: > Hi All > > We run an application with version 1.4.1 standalone mode. We saw two tasks > in one stage which runs very slow seems it is hang. We know that the > JobScheduler have the function to assign the straggle task to another node. > But what we saw it does not reassign. So we want to know is there anyone > know that is there ant para to open this function. > What's more, we saw that these two straggle tasks hangs at socket read. Is > it because it cannot be interrupted so the reassign function does not work, > the Thread stack as below: > > > java.net.SocketInputStream.socketRead0(Native Method) > java.net.SocketInputStream.read(SocketInputStream.java:152) > java.net.SocketInputStream.read(SocketInputStream.java:122) > java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > java.io.BufferedInputStream.read(BufferedInputStream.java:254) > org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) > org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) > > org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) > > org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) > > org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) > > org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) > > org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) > > org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) > > org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > com.vip.logview.tsdb.HttpClientUtils.postHandler(HttpClientUtils.java:80) > > com.vip.logview.function.DomainHostStatusSaveFunction$1.call(DomainHostStatusSaveFunction.java:75) > > com.vip.logview.function.DomainHostStatusSaveFunction$1.call(DomainHostStatusSaveFunction.java:30) > > org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219) > > org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219) > > org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:878) > > org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:878) > > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767) > > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767) > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) > org.apache.spark.scheduler.Task.run(Task.scala:70) > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > > Thanks > > > > 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ > This communication is intended only for the addressee(s) and may contain > information that is privileged and confidential. You are hereby notified > that, if you are not an intended recipient listed above, or an authorized > employee or agent of an addressee of this communication responsible for > delivering e-mail messages to an intended recipient, any dissemination, > distribution or reproduction of this communication (including any > attachments hereto) is strictly prohibited. If you have received this > communication in error, please notify us immediately by a reply e-mail > addressed to the sender and permanently delete the original e-mail > communication and any attachments from all storage devices without making > or otherwise retaining a copy. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
