Hi All We run an application with version 1.4.1 standalone mode. We saw two tasks in one stage which runs very slow seems it is hang. We know that the JobScheduler have the function to assign the straggle task to another node. But what we saw it does not reassign. So we want to know is there anyone know that is there ant para to open this function. What's more, we saw that these two straggle tasks hangs at socket read. Is it because it cannot be interrupted so the reassign function does not work, the Thread stack as below:
java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.read(SocketInputStream.java:152) java.net.SocketInputStream.read(SocketInputStream.java:122) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) com.vip.logview.tsdb.HttpClientUtils.postHandler(HttpClientUtils.java:80) com.vip.logview.function.DomainHostStatusSaveFunction$1.call(DomainHostStatusSaveFunction.java:75) com.vip.logview.function.DomainHostStatusSaveFunction$1.call(DomainHostStatusSaveFunction.java:30) org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219) org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219) org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:878) org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:878) org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767) org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767) org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) org.apache.spark.scheduler.Task.run(Task.scala:70) org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thanks 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ This communication is intended only for the addressee(s) and may contain information that is privileged and confidential. You are hereby notified that, if you are not an intended recipient listed above, or an authorized employee or agent of an addressee of this communication responsible for delivering e-mail messages to an intended recipient, any dissemination, distribution or reproduction of this communication (including any attachments hereto) is strictly prohibited. If you have received this communication in error, please notify us immediately by a reply e-mail addressed to the sender and permanently delete the original e-mail communication and any attachments from all storage devices without making or otherwise retaining a copy. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
