[
https://issues.apache.org/jira/browse/HIVE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13792468#comment-13792468
]
Pere Ferrera Bertran commented on HIVE-5235:
--------------------------------------------
One thing I would like to emphasize, is that the error is not deterministic
with the input data. The same input may produce a corrupted file, or it may
not. If I leave it working for enough hours, at some point there's a high
chance I will produce a corrupted ORC file. This is all I know so far.
So I'm not sure if it will be valuable to reproduce this again with some
determined input data, as you won't be able to reproduce it anyway. Some
strange conditions are met into this cluster which causes a random corruption
to happen. Maybe some packet gets corrupted in the network under high load and
the underlying implementation doesn't recover from that.
I will however turn off compression and try to generate a corrupted file, so at
least we know something else.
> Infinite loop with ORC file and Hive 0.11
> -----------------------------------------
>
> Key: HIVE-5235
> URL: https://issues.apache.org/jira/browse/HIVE-5235
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.11.0
> Environment: Gentoo linux with Hortonworks Hadoop
> hadoop-1.1.2.23.tar.gz and Apache Hive 0.11d
> Reporter: Iván de Prado
> Priority: Blocker
> Attachments: gendata.py
>
>
> We are using Hive 0.11 with ORC file format and we get some tasks blocked in
> some kind of infinite loop. They keep working indefinitely when we set a huge
> task expiry timeout. If we the expiry time to 600 second, the taks fail
> because of not reporting progress, and finally, the Job fails.
> That is not consistent, and some times between jobs executions the behavior
> changes. It happen for different queries.
> We are using Hive 0.11 with Hadoop hadoop-1.1.2.23 from Hortonworks. The taks
> that is blocked keeps consuming 100% of CPU usage, and the stack trace is
> always the same consistently. Everything points to some kind of infinite
> loop. My guessing is that it has some relation to the ORC file. Maybe some
> pointer is not right when writing generating some kind of infinite loop when
> reading. Or maybe there is a bug in the reading stage.
> More information below. The stack trace:
> {noformat}
> "main" prio=10 tid=0x00007f20a000a800 nid=0x1ed2 runnable [0x00007f20a8136000]
> java.lang.Thread.State: RUNNABLE
> at java.util.zip.Inflater.inflateBytes(Native Method)
> at java.util.zip.Inflater.inflate(Inflater.java:256)
> - locked <0x00000000f42a6ca0> (a java.util.zip.ZStreamRef)
> at
> org.apache.hadoop.hive.ql.io.orc.ZlibCodec.decompress(ZlibCodec.java:64)
> at
> org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.readHeader(InStream.java:128)
> at
> org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.read(InStream.java:143)
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readVulong(SerializationUtils.java:54)
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readVslong(SerializationUtils.java:65)
> at
> org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReader.readValues(RunLengthIntegerReader.java:66)
> at
> org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReader.next(RunLengthIntegerReader.java:81)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$IntTreeReader.next(RecordReaderImpl.java:332)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:802)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1214)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:71)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:46)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:274)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:101)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:41)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:108)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:300)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:218)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
> - eliminated <0x00000000e1459700> (a
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
> - locked <0x00000000e1459700> (a
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1178)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
> We have seen the same stack trace repeatedly for several executions of jstack.
> The log file for this kind of task is the following:
> {noformat}
> 2013-09-04 23:12:34,332 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded
> the native-hadoop library
> 2013-09-04 23:12:34,681 INFO org.apache.hadoop.mapred.TaskRunner: Creating
> symlink:
> /hd/hd6/mapred/local/taskTracker/distcache/1758137359311570022_-1138434677_1812098585/master/tmp/hive-datasalt/hive_2013-09-04_22-45-13_829_2202639092470021957/-mr-10004/22027cf8-f583-41d7-adb8-e7e74922d113
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/HIVE_PLAN22027cf8-f583-41d7-adb8-e7e74922d113
> 2013-09-04 23:12:34,718 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/com
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/com
> 2013-09-04 23:12:34,733 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/javolution
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/javolution
> 2013-09-04 23:12:34,743 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/org
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/org
> 2013-09-04 23:12:34,756 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/job.jar
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/job.jar
> 2013-09-04 23:12:34,768 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/javax
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/javax
> 2013-09-04 23:12:34,779 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/hive-exec-log4j.properties
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/hive-exec-log4j.properties
> 2013-09-04 23:12:34,790 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/META-INF
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/META-INF
> 2013-09-04 23:12:34,796 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/javaewah
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/javaewah
> 2013-09-04 23:12:34,813 INFO
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
> /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/.job.jar.crc
> <-
> /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/.job.jar.crc
> 2013-09-04 23:12:35,127 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-09-04 23:12:35,412 INFO org.apache.hadoop.util.ProcessTree: setsid
> exited with exit code 0
> 2013-09-04 23:12:35,473 INFO org.apache.hadoop.mapred.Task: Using
> ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5d0fa836
> 2013-09-04 23:12:36,927 INFO
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader: Processing file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-28/000798_0
> 2013-09-04 23:12:36,927 INFO org.apache.hadoop.mapred.MapTask:
> numReduceTasks: 72
> 2013-09-04 23:12:36,936 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
> 100
> 2013-09-04 23:12:37,028 INFO org.apache.hadoop.mapred.MapTask: data buffer =
> 79691776/99614720
> 2013-09-04 23:12:37,028 INFO org.apache.hadoop.mapred.MapTask: record buffer
> = 262144/327680
> 2013-09-04 23:12:37,036 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy:
> Snappy native library is available
> 2013-09-04 23:12:37,036 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy:
> Snappy native library loaded
> 2013-09-04 23:12:37,046 INFO ExecMapper: maximum memory = 559284224
> 2013-09-04 23:12:37,046 INFO ExecMapper: conf classpath =
> [file:/home/datasalt/sw/hadoop-1.1.2.23/conf/,
> file:/opt/oracle-jdk-bin-1.7.0.17/lib/tools.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/,
> file:/home/datasalt/sw/hadoop-1.1.2.23/hadoop-core-1.1.2.23.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/asm-3.2.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjrt-1.6.11.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjtools-1.6.11.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-1.7.0.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-core-1.8.0.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-cli-1.2.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-codec-1.4.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-collections-3.2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-configuration-1.6.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-daemon-1.0.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-digester-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-el-1.0.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-httpclient-3.0.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-io-2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-lang-2.4.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-1.1.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-api-1.0.4.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-math-2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-net-3.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/core-3.1.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-capacity-scheduler-1.1.2.23.jar,
>
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-fairscheduler-1.1.2.23.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-thriftfs-1.1.2.23.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hsqldb-1.8.0.10.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-core-asl-1.8.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-mapper-asl-1.8.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-compiler-5.5.12.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-runtime-5.5.12.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jdeb-0.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-core-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-json-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-server-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jets3t-0.6.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-6.1.26.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-util-6.1.26.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsch-0.1.42.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/junit-4.5.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/kfs-0.2.2.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/log4j-1.2.15.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/mockito-all-1.8.5.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/oro-2.0.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/servlet-api-2.5-20081211.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-api-1.4.3.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-log4j12-1.4.3.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/xmlenc-0.52.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-api-2.1.jar,
> file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/classes,
>
> file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/,
>
> file:/hd/hd1/mapred/local/taskTracker/datasalt/distcache/-1837517190973317952_-652944854_1812099894/master/tmp/hadoop-datasalt/mapred/staging/datasalt/.staging/job_201309040511_0003/libjars/derby-10.4.2.0.jar/,
>
> file:/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/]
> 2013-09-04 23:12:37,047 INFO ExecMapper: thread classpath =
> [file:/home/datasalt/sw/hadoop-1.1.2.23/conf/,
> file:/opt/oracle-jdk-bin-1.7.0.17/lib/tools.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/,
> file:/home/datasalt/sw/hadoop-1.1.2.23/hadoop-core-1.1.2.23.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/asm-3.2.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjrt-1.6.11.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjtools-1.6.11.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-1.7.0.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-core-1.8.0.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-cli-1.2.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-codec-1.4.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-collections-3.2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-configuration-1.6.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-daemon-1.0.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-digester-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-el-1.0.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-httpclient-3.0.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-io-2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-lang-2.4.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-1.1.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-api-1.0.4.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-math-2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-net-3.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/core-3.1.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-capacity-scheduler-1.1.2.23.jar,
>
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-fairscheduler-1.1.2.23.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-thriftfs-1.1.2.23.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hsqldb-1.8.0.10.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-core-asl-1.8.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-mapper-asl-1.8.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-compiler-5.5.12.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-runtime-5.5.12.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jdeb-0.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-core-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-json-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-server-1.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jets3t-0.6.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-6.1.26.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-util-6.1.26.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsch-0.1.42.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/junit-4.5.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/kfs-0.2.2.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/log4j-1.2.15.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/mockito-all-1.8.5.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/oro-2.0.8.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/servlet-api-2.5-20081211.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-api-1.4.3.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-log4j12-1.4.3.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/xmlenc-0.52.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-2.1.jar,
> file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-api-2.1.jar,
> file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/classes,
>
> file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/,
>
> file:/hd/hd1/mapred/local/taskTracker/datasalt/distcache/-1837517190973317952_-652944854_1812099894/master/tmp/hadoop-datasalt/mapred/staging/datasalt/.staging/job_201309040511_0003/libjars/derby-10.4.2.0.jar/,
>
> file:/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/]
> 2013-09-04 23:12:37,085 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Adding alias hits to work list for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-28
> 2013-09-04 23:12:37,089 INFO org.apache.hadoop.hive.ql.exec.MapOperator: dump
> TS
> struct<hour:int,user_ip:string,country_iso:string,user_language:string,num_hits:int,site_hostname:string,idsite:int,num_ads:int,num_ads_wanted:int,category:int,idcampaign:int,idvariation:int,sub_id:int,cpm_value:double,publisher_share:int,publisher_currency:string,advertiser_currency:string,idzone:int,idagent:int,idadvertiser:int,idpublisher:int,ddate:string>
> 2013-09-04 23:12:37,091 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Adding alias hits to work list for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-29
> 2013-09-04 23:12:37,092 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Adding alias hits to work list for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-30
> 2013-09-04 23:12:37,093 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Adding alias hits to work list for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-31
> 2013-09-04 23:12:37,094 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Adding alias hits to work list for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-09-01
> 2013-09-04 23:12:37,094 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Adding alias hits to work list for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-09-02
> 2013-09-04 23:12:37,095 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Adding alias hits to work list for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-09-03
> 2013-09-04 23:12:37,096 INFO ExecMapper:
> <MAP>Id =7
> <Children>
> <TS>Id =3
> <Children>
> <SEL>Id =2
> <Children>
> <GBY>Id =1
> <Children>
> <RS>Id =0
> <Parent>Id = 1 null<\Parent>
> <\RS>
> <\Children>
> <Parent>Id = 2 null<\Parent>
> <\GBY>
> <\Children>
> <Parent>Id = 3 null<\Parent>
> <\SEL>
> <\Children>
> <Parent>Id = 7 null<\Parent>
> <\TS>
> <\Children>
> <\MAP>
> 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Initializing Self 7 MAP
> 2013-09-04 23:12:37,100 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing Self 3 TS
> 2013-09-04 23:12:37,100 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Operator 3 TS initialized
> 2013-09-04 23:12:37,100 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing children of 3
> TS
> 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
> Initializing child 2 SEL
> 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
> Initializing Self 2 SEL
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
> SELECT
> struct<hour:int,user_ip:string,country_iso:string,user_language:string,num_hits:int,site_hostname:string,idsite:int,num_ads:int,num_ads_wanted:int,category:int,idcampaign:int,idvariation:int,sub_id:int,cpm_value:double,publisher_share:int,publisher_currency:string,advertiser_currency:string,idzone:int,idagent:int,idadvertiser:int,idpublisher:int,ddate:string>
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
> Operator 2 SEL initialized
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
> Initializing children of 2 SEL
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator:
> Initializing child 1 GBY
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator:
> Initializing Self 1 GBY
> 2013-09-04 23:12:37,116 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator:
> Operator 1 GBY initialized
> 2013-09-04 23:12:37,116 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator:
> Initializing children of 1 GBY
> 2013-09-04 23:12:37,116 INFO
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initializing child 0 RS
> 2013-09-04 23:12:37,116 INFO
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initializing Self 0 RS
> 2013-09-04 23:12:37,116 INFO
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Using tag = -1
> 2013-09-04 23:12:37,139 INFO
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Operator 0 RS initialized
> 2013-09-04 23:12:37,139 INFO
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initialization Done 0 RS
> 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator:
> Initialization Done 1 GBY
> 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
> Initialization Done 2 SEL
> 2013-09-04 23:12:37,139 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initialization Done 3 TS
> 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Initialization Done 7 MAP
> 2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
> Processing alias hits for file
> hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-28
> 2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
> forwarding 1 rows
> 2013-09-04 23:12:37,165 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 1 rows
> 2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
> forwarding 1 rows
> 2013-09-04 23:12:37,168 INFO ExecMapper: ExecMapper: processing 1 rows: used
> memory = 151151936
> 2013-09-04 23:12:37,169 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
> forwarding 10 rows
> 2013-09-04 23:12:37,169 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 10 rows
> 2013-09-04 23:12:37,169 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
> forwarding 10 rows
> 2013-09-04 23:12:37,169 INFO ExecMapper: ExecMapper: processing 10 rows: used
> memory = 151151936
> 2013-09-04 23:12:37,175 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
> forwarding 100 rows
> 2013-09-04 23:12:37,175 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 100 rows
> 2013-09-04 23:12:37,175 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
> forwarding 100 rows
> 2013-09-04 23:12:37,175 INFO ExecMapper: ExecMapper: processing 100 rows:
> used memory = 151151936
> 2013-09-04 23:12:37,247 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
> forwarding 1000 rows
> 2013-09-04 23:12:37,250 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 1000 rows
> 2013-09-04 23:12:37,250 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
> forwarding 1000 rows
> 2013-09-04 23:12:37,251 INFO ExecMapper: ExecMapper: processing 1000 rows:
> used memory = 151151936
> 2013-09-04 23:12:37,642 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
> forwarding 10000 rows
> 2013-09-04 23:12:37,642 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 10000 rows
> 2013-09-04 23:12:37,644 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
> forwarding 10000 rows
> 2013-09-04 23:12:37,644 INFO ExecMapper: ExecMapper: processing 10000 rows:
> used memory = 153032176
> 2013-09-04 23:12:38,467 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
> forwarding 100000 rows
> 2013-09-04 23:12:38,468 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 100000 rows
> 2013-09-04 23:12:38,468 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
> forwarding 100000 rows
> 2013-09-04 23:12:38,469 INFO ExecMapper: ExecMapper: processing 100000 rows:
> used memory = 125346272
> 2013-09-04 23:12:40,626 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
> forwarding 1000000 rows
> 2013-09-04 23:12:40,627 INFO
> org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 1000000 rows
> 2013-09-04 23:12:40,627 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
> forwarding 1000000 rows
> 2013-09-04 23:12:40,627 INFO ExecMapper: ExecMapper: processing 1000000 rows:
> used memory = 123383952
> {noformat}
> For this particular example, the task was killed 6 hours after the last log
> line.
> The strace of the process when it is in the infinite loop has the following
> looking:
> {noformat}
> Process 7887 attached with 24 threads
> [pid 7930] futex(0x7f20a06df28c, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
> [pid 7922] restart_syscall(<... resuming interrupted call ...> <unfinished
> ...>
> [pid 7920] restart_syscall(<... resuming interrupted call ...> <unfinished
> ...>
> [pid 8333] accept(61, <unfinished ...>
> [pid 7908] restart_syscall(<... resuming interrupted call ...> <unfinished
> ...>
> [pid 7907] restart_syscall(<... resuming interrupted call ...> <unfinished
> ...>
> [pid 7906] restart_syscall(<... resuming interrupted call ...> <unfinished
> ...>
> [pid 7905] futex(0x7f20a00fa554, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished ...>
> [pid 7903] futex(0x7f20a00f5254, FUTEX_WAIT_PRIVATE, 417, NULL <unfinished
> ...>
> [pid 7904] futex(0x7f20a00f7c54, FUTEX_WAIT_PRIVATE, 387, NULL <unfinished
> ...>
> [pid 7902] futex(0x7f20a7513dc0, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
> [pid 7901] futex(0x7f20a00a6f54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished ...>
> [pid 7899] restart_syscall(<... resuming interrupted call ...> <unfinished
> ...>
> [pid 7898] futex(0x7f20a0026554, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished
> ...>
> [pid 7900] futex(0x7f20a00a4b54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished ...>
> [pid 7897] futex(0x7f20a0024754, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished
> ...>
> [pid 7896] futex(0x7f20a0022a54, FUTEX_WAIT_PRIVATE, 41, NULL <unfinished
> ...>
> [pid 7895] futex(0x7f20a0020c54, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished
> ...>
> [pid 7894] futex(0x7f20a001ef54, FUTEX_WAIT_PRIVATE, 37, NULL <unfinished
> ...>
> [pid 7893] futex(0x7f20a001d154, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished
> ...>
> [pid 7887] futex(0x7f20a81389d0, FUTEX_WAIT, 7890, NULL <unfinished ...>
> [pid 7892] futex(0x7f20a001b454, FUTEX_WAIT_PRIVATE, 35, NULL <unfinished
> ...>
> [pid 7891] futex(0x7f20a0019654, FUTEX_WAIT_PRIVATE, 37, NULL <unfinished
> ...>
> [pid 7906] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed
> out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373836, 993325000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 43598000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 93858000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 144101000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 194344000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 244659000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 294924000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 345198000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 395462000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 445768000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 495983000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 546226000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 596489000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 646798000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 697063000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 747370000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 797688000},
> ffffffff <unfinished ...>
> [pid 7899] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed
> out)
> [pid 7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7899] futex(0x7f20a009d054,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 782077000},
> ffffffff <unfinished ...>
> [pid 7906] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 848014000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 898283000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 948489000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 998751000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 49086000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 99351000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 149589000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 199852000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 250180000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 300489000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 350762000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 401078000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 451258000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 501527000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 551767000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 601970000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 652161000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 702290000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 752523000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 802739000},
> ffffffff <unfinished ...>
> [pid 7899] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7899] futex(0x7f20a009d054,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 782336000},
> ffffffff <unfinished ...>
> [pid 7922] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed
> out)
> [pid 7922] futex(0x7f20a06d3a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7922] futex(0x7f20a0550554, FUTEX_WAKE_OP_PRIVATE, 1, 1,
> 0x7f20a0550550, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid 7907] <... restart_syscall resumed> ) = 0
> [pid 7907] futex(0x7f20a0550528, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7922] write(56, "\0\0\0\335\0\0\2.\0\4ping\0\0\0\2\0&org.apache.h"...,
> 225 <unfinished ...>
> [pid 7907] read(56, <unfinished ...>
> [pid 7922] <... write resumed> ) = 225
> [pid 7907] <... read resumed> 0x7f2098000b70, 8192) = -1 EAGAIN (Resource
> temporarily unavailable)
> [pid 7922] futex(0x7f20a009d054, FUTEX_WAKE_OP_PRIVATE, 1, 1,
> 0x7f20a009d050, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid 7899] <... futex resumed> ) = 0
> [pid 7922] futex(0x7f20a06d3e54, FUTEX_WAIT_PRIVATE, 53, NULL <unfinished
> ...>
> [pid 7907] epoll_ctl(59, EPOLL_CTL_ADD, 56, {...} <unfinished ...>
> [pid 7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 7907] <... epoll_ctl resumed> ) = 0
> [pid 7899] <... futex resumed> ) = 0
> [pid 7907] epoll_wait(59, {?} 0x7f20a0544000, 4096, 60000) = 1
> [pid 7899] mprotect(0x7f20a8147000, 4096, PROT_READ <unfinished ...>
> [pid 7907] futex(0x7f20a0551e54, FUTEX_WAIT_PRIVATE, 11, NULL <unfinished
> ...>
> [pid 7899] <... mprotect resumed> ) = 0
> [pid 7890] futex(0x7f20a000b854, FUTEX_WAIT_PRIVATE, 349, NULL <unfinished
> ...>
> [pid 7899] mprotect(0x7f20a8147000, 4096, PROT_READ|PROT_WRITE) = 0
> [pid 7899] mprotect(0x7f20a8148000, 4096, PROT_NONE) = 0
> [pid 7899] mprotect(0x7f20a8148000, 4096, PROT_READ) = 0
> [pid 7899] futex(0x7f20a0551e54, FUTEX_WAKE_OP_PRIVATE, 1, 1,
> 0x7f20a0551e50, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid 7899] futex(0x7f20a06d3e54, FUTEX_WAKE_OP_PRIVATE, 1, 1,
> 0x7f20a06d3e50, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} <unfinished ...>
> [pid 7922] <... futex resumed> ) = 0
> [pid 7907] <... futex resumed> ) = 0
> [pid 7922] futex(0x7f20a06d3e28, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
> [pid 7899] <... futex resumed> ) = 1
> [pid 7907] futex(0x7f20a0551e28, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 7899] futex(0x7f20a06d3e28, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 7922] <... futex resumed> ) = 0
> [pid 7907] <... futex resumed> ) = 0
> [pid 7922] futex(0x7f20a06d3e28, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 7899] <... futex resumed> ) = 1
> [pid 7922] <... futex resumed> ) = 0
> [pid 7899] futex(0x7f20a009d054,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 3, {1378373839, 794194000},
> ffffffff <unfinished ...>
> [pid 7922] futex(0x7f20a06d3a54, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
> [pid 7907] futex(0x7f20a000b854, FUTEX_WAKE_OP_PRIVATE, 1, 1,
> 0x7f20a000b850, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid 7890] <... futex resumed> ) = 0
> [pid 7890] futex(0x7f20a000b828, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 7907] epoll_ctl(59, EPOLL_CTL_DEL, 56, {...} <unfinished ...>
> [pid 7890] <... futex resumed> ) = 0
> [pid 7907] <... epoll_ctl resumed> ) = 0
> [pid 7907] epoll_wait(59, {}, 4096, 0) = 0
> [pid 7907] read(56, "\0\0\2.\0\0\0\0\0\7boolean\1", 8192) = 18
> [pid 7907] futex(0x7f20a06d3a54, FUTEX_WAKE_OP_PRIVATE, 1, 1,
> 0x7f20a06d3a50, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid 7922] <... futex resumed> ) = 0
> [pid 7907] futex(0x7f20a0550554,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1107, {1378373848,
> 793674000}, ffffffff <unfinished ...>
> [pid 7922] futex(0x7f20a06d3a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7922] futex(0x7f20a06d3a54,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 3, {1378373841, 794758000},
> ffffffff <unfinished ...>
> [pid 7906] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 852867000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 903052000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 953226000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 3436000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 53695000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 103966000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 154232000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 204466000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 254802000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 305067000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 355318000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 405623000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 455817000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 506130000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 556395000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 606661000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 656898000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 7906] futex(0x7f20a0104954,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 707177000},
> ffffffff) = -1 ETIMEDOUT (Connection timed out)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
