Hi,
Just check the logs of datanode, it looks like this:
*
2015-05-20 11:42:14,605 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.9.0.48:50676, dest: /10.9.0.17:50010, bytes: 134217728, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_804680172_54, offset: 0, srvID:
39fb78d5-828a-4319-8303-c704fab526e3, blockid:
BP-436159032-10.9.0.16-1431330007172:blk_1073742096_1273, duration:
16994466261
2015-05-20 11:42:14,606 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder:
BP-436159032-10.9.0.16-1431330007172:blk_1073742096_1273,
type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2015-05-20 11:42:17,788 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.9.0.17:49046, dest: /10.9.0.17:50010, bytes: 134217728, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_102926009_54, offset: 0, srvID:
39fb78d5-828a-4319-8303-c704fab526e3, blockid:
BP-436159032-10.9.0.16-1431330007172:blk_1073742099_1276, duration:
17829554438
2015-05-20 11:42:17,788 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder:
BP-436159032-10.9.0.16-1431330007172:blk_1073742099_1276,
type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2015-05-20 11:42:17,904 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving
BP-436159032-10.9.0.16-1431330007172:blk_1073742103_1280 src:
/10.9.0.17:49049 dest: /10.9.0.17:50010
2015-05-20 11:42:17,904 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
BlockReceiver constructor. Cause is
2015-05-20 11:42:17,904 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock
BP-436159032-10.9.0.16-1431330007172:blk_1073742103_1280 received exception
org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException: Out of space:
The volume with the most available space (=114409472 B) is less than the
block size (=134217728 B).
2015-05-20 11:42:17,905 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
HadoopV26Slave1:50010:DataXceiver error processing WRITE_BLOCK operation
src: /10.9.0.17:49049 dst: /10.9.0.17:50010
org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException: Out of space:
The volume with the most available space (=114409472 B) is less than the
block size (=134217728 B).
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy.chooseVolume(RoundRobinVolumeChoosingPolicy.java:67)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.getNextVolume(FsVolumeList.java:69)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1084)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:114)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:183)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
at java.lang.Thread.run(Thread.java:745)
2015-05-20 11:43:59,669 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification
succeeded for BP-436159032-10.9.0.16-1431330007172:blk_1073741999_1176
2015-05-20 11:46:10,214 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification
succeeded for BP-436159032-10.9.0.16-1431330007172:blk_1073742000_1177
2015-05-20 11:48:35,445 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification
succeeded for BP-436159032-10.9.0.16-1431330007172:blk_1073741990_1167
2015-05-20 11:50:04,043 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073742080_1257 file
/tmp/hadoop-hduser/dfs/data/current/BP-436159032-10.9.0.16-1431330007172/current/finalized/subdir0/subdir1/blk_1073742080
for deletion
2015-05-20 11:50:04,136 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073742081_1258 file
/tmp/hadoop-hduser/dfs/data/current/BP-436159032-10.9.0.16-1431330007172/current/finalized/subdir0/subdir1/blk_1073742081
for deletion
2015-05-20 11:50:04,136 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073742082_1259 file
/tmp/hadoop-hduser/dfs/data/current/BP-436159032-10.9.0.16-1431330007172/current/finalized/subdir0/subdir1/blk_1073742082
for deletion
2015-05-20 11:50:04,136 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073742083_1260 file
/tmp/hadoop-hduser/dfs/data/current/BP-436159032-10.9.0.16-1431330007172/current/finalized/subdir0/subdir1/blk_1073742083
for deletion
2015-05-20 11:50:04,136 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073742084_1261 file
/tmp/hadoop-hduser/dfs/data/current/BP-436159032-10.9.0.16-1431330007172/current/finalized/subdir0/subdir1/blk_1073742084
for deletion*
But why this is a hdfs issue, because I think spark broadcast the variable
in memory.
BTW, the datanode logs seems like I don't have any space to save the
storage. So the question comes back, originally I have 50GB for HDFS, why I
broadcast a variable and then that variable will dominate the whole hdfs?
P.S. when I reduce the data size to only 10 lines (200kb), it works without
problem.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/java-program-got-Stuck-at-broadcasting-tp22953p22972.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
