Hi, I think there the bottleneck might be HDFS. With 300 operators with parallelism 6 you will have 1800 concurrent writes (i.e. connections) to HDFS, which might be to much for the master node and the worker nodes.
This is the same problem that you had on the local filesystem but now in the distributed filesystem. Best, Aljoscha > On 28. Apr 2017, at 22:15, mclendenin <marcusc...@gmail.com> wrote: > > There are only 3 nodes in the HDFS cluster and when running fsck it shows the > filesystem as healthy. > > $ hdfs fsck > /user/hadoop/flink/checkpoints/dc2aee563bebce76e420029525c37892/chk-43/ > 17/04/28 16:24:59 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Connecting to namenode via > http://localhost/fsck?ugi=hadoop&path=%2Fuser%2Fhadoop%2Fflink%2Fcheckpoints%2Fdc2aee563bebce76e420029525c37892%2Fchk-43 > FSCK started by hadoop (auth:SIMPLE) from / for path > /user/hadoop/flink/checkpoints/dc2aee563bebce76e420029525c37892/chk-43 at > Fri Apr 28 16:25:00 EDT 2017 > .....Status: HEALTHY > Total size: 33197 B > Total dirs: 1 > Total files: 5 > Total symlinks: 0 (Files currently being written: 460) > Total blocks (validated): 5 (avg. block size 6639 B) > Minimally replicated blocks: 5 (100.0 %) > Over-replicated blocks: 0 (0.0 %) > Under-replicated blocks: 0 (0.0 %) > Mis-replicated blocks: 0 (0.0 %) > Default replication factor: 2 > Average block replication: 3.0 > Corrupt blocks: 0 > Missing replicas: 0 (0.0 %) > Number of data-nodes: 3 > Number of racks: 1 > FSCK ended at Fri Apr 28 16:25:00 EDT 2017 in 13 milliseconds > > > The filesystem under path > '/user/hadoop/flink/checkpoints/dc2aee563bebce76e420029525c37892/chk-43' is > HEALTHY > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/RocksDB-error-with-flink-1-2-0-tp12897p12909.html > Sent from the Apache Flink User Mailing List archive. mailing list archive at > Nabble.com.
