Hi Bernd, thanks for the report. I tried to reproduce it locally but both a telnet connection to the BlobServer as well as the BLOB download by the TaskManagers work for me. Can you share your configuration that is causing the problem? You could also try increasing the log level to DEBUG and see if there is something more in the logs (the exception thrown in StaticFileServerHandler looks suspicious but is not related to the BlobServer). Apparently, the TaskManager resolves flink-jobmanager to 10.104.5.130. Is that the correct address and can the TaskManager talk to this IP? (may a firewall block this?)
Did you, by any chance, set up SSL, too? There was a recent thread on the mailing list [1] where a had some problems with "security.ssl.verify-hostname" being set to true which may be related. Nico [1] https://lists.apache.org/thread.html/879d072bfd6761947b4bd703324489db50e8b14c328992118af875d8@%3Cuser.flink.apache.org%3E On 04/12/17 10:03, bernd.winterst...@dev.helaba.de wrote: > Hi > Since we switched to Release 1.4 the taskmanagers are unable to download > blobs from the jobmanager. > The taskmanager registration still works. > Netstat on jobmanager shows open ports at 6123 and 50000. But a telnet > connection from taskmanager to jobmanager on port 50000 times out. > > Any ideas are welcome. > > Regards > > Bernd > > Jobmanager log: > > 2017-12-04 08:48:30,167 INFO > org.apache.flink.runtime.jobmanager.JobManager - Starting > JobManager actor > 2017-12-04 08:48:30,197 INFO > org.apache.flink.runtime.blob.BlobServer - Created > BLOB server storage directory > /tmp/blobStore-81cd12c7-394e-4777-85a1-98389b72dd08 > 2017-12-04 08:48:30,205 INFO > org.apache.flink.runtime.blob.BlobServer - Started > BLOB server at 0.0.0.0:50000 - max concurrent requests: 50 - max > backlog: 1000 > 2017-12-04 08:48:30,608 INFO > org.apache.flink.runtime.jobmanager.JobManager - Starting > JobManager at akka.tcp://flink@flink-jobmanager:6123/user/jobmanager. > 2017-12-04 08:48:30,628 INFO > org.apache.flink.runtime.jobmanager.MemoryArchivist - Started > memory archivist akka://flink/user/archive > 2017-12-04 08:48:30,676 INFO > org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager > > - Trying to associate with JobManager leader > akka.tcp://flink@flink-jobmanager:6123/user/jobmanager > 2017-12-04 08:48:30,692 INFO > org.apache.flink.runtime.jobmanager.JobManager - > JobManager akka.tcp://flink@flink-jobmanager:6123/user/jobmanager was > granted leadership with leader session ID > Some(00000000-0000-0000-0000-000000000000). > 2017-12-04 08:48:30,700 INFO > org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager > > - Resource Manager associating with leading JobManager > Actor[akka://flink/user/jobmanager#886586058] - leader session > 00000000-0000-0000-0000-000000000000 > 2017-12-04 08:53:50,635 INFO > org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager > > - TaskManager 627338086a766c140909ba45f2e717d0 has started. > 2017-12-04 08:53:50,638 INFO > org.apache.flink.runtime.instance.InstanceManager - > Registered TaskManager at flink-taskmanager-65cf757d9b-hj65d > (akka.tcp://flink@flink-taskmanager-65cf757d9b-hj65d:45932/user/taskmanager) > as f9d2843d0223b15d8fce52aea8231cc6. Current number of registered hosts > is 1. Current number of alive task slots is 8. > 2017-12-04 08:53:50,658 WARN > akka.serialization.Serialization(akka://flink) - Using > the default Java serializer for class > [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage] > which is not recommended because of performance implications. Use > another serializer or disable this warning using the setting > 'akka.actor.warn-about-java-serializer-usage' > 2017-12-04 08:53:55,714 INFO > org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager > > - TaskManager 08c3e6f7c765e2ab88e2ea645049cb9d has started. > 2017-12-04 08:53:55,714 INFO > org.apache.flink.runtime.instance.InstanceManager - > Registered TaskManager at flink-taskmanager-65cf757d9b-jtzw5 > (akka.tcp://flink@flink-taskmanager-65cf757d9b-jtzw5:41710/user/taskmanager) > as da8a8da3650ce53f460784c54938a071. Current number of registered hosts > is 2. Current number of alive task slots is 16. > 2017-12-04 09:04:08,850 ERROR > org.apache.flink.runtime.rest.handler.legacy.files.StaticFileServerHandler > - Caught exception > java.io.IOException: Operation timed out > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:192) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) > > > Taskmanager log: > 2017-12-04 08:53:55,511 INFO > org.apache.flink.runtime.taskmanager.TaskManager - Starting > TaskManager actor at akka://flink/user/taskmanager#-977591027. > 2017-12-04 08:53:55,511 INFO > org.apache.flink.runtime.taskmanager.TaskManager - > TaskManager data connection information: > 08c3e6f7c765e2ab88e2ea645049cb9d @ flink-taskmanager-65cf757d9b-jtzw5 > (dataPort=42142) > 2017-12-04 08:53:55,512 INFO > org.apache.flink.runtime.taskmanager.TaskManager - > TaskManager has 8 task slot(s). > 2017-12-04 08:53:55,513 INFO > org.apache.flink.runtime.taskmanager.TaskManager - Memory > usage stats: [HEAP: 131/981/981 MB, NON HEAP: 34/35/-1 MB > (used/committed/max)] > 2017-12-04 08:53:55,518 INFO > org.apache.flink.runtime.taskmanager.TaskManager - Trying > to register at JobManager > akka.tcp://flink@flink-jobmanager:6123/user/jobmanager (attempt 1, > timeout: 500 milliseconds) > 2017-12-04 08:53:55,671 WARN > akka.serialization.Serialization(akka://flink) - Using > the default Java serializer for class > [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage] > which is not recommended because of performance implications. Use > another serializer or disable this warning using the setting > 'akka.actor.warn-about-java-serializer-usage' > 2017-12-04 08:53:55,737 INFO > org.apache.flink.runtime.taskmanager.TaskManager - > Successful registration at JobManager > (akka.tcp://flink@flink-jobmanager:6123/user/jobmanager), starting > network stack and library cache. > 2017-12-04 08:53:55,742 INFO > org.apache.flink.runtime.taskmanager.TaskManager - > Determined BLOB server address to be > flink-jobmanager/10.104.5.130:50000. Starting BLOB cache. > 2017-12-04 08:53:55,752 INFO > org.apache.flink.runtime.blob.PermanentBlobCache - Created > BLOB cache storage directory > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5 > 2017-12-04 08:53:55,776 INFO > org.apache.flink.runtime.blob.TransientBlobCache - Created > BLOB cache storage directory > /tmp/blobStore-8d7c8660-4455-43e9-93be-dadeb622b9e1 > 2017-12-04 08:54:00,796 WARN > akka.serialization.Serialization(akka://flink) - Using > the default Java serializer for class > [org.apache.flink.runtime.messages.TaskManagerMessages$Heartbeat] which > is not recommended because of performance implications. Use another > serializer or disable this warning using the setting > 'akka.actor.warn-about-java-serializer-usage' > 2017-12-04 09:10:38,726 WARN > akka.serialization.Serialization(akka://flink) - Using > the default Java serializer for class > [org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricSerializationResult] > which is not recommended because of performance implications. Use > another serializer or disable this warning using the setting > 'akka.actor.warn-about-java-serializer-usage' > 2017-12-04 09:11:35,403 INFO > org.apache.flink.runtime.taskmanager.TaskManager - Received > task Source: PACS008 Generator -> Double Submission Generator -> Invalid > IBAN Generator -> Sink: IP-PREPROCESSOR-INBOUND (1/1) > 2017-12-04 09:11:35,404 INFO > org.apache.flink.runtime.taskmanager.Task - Source: > PACS008 Generator -> Double Submission Generator -> Invalid IBAN > Generator -> Sink: IP-PREPROCESSOR-INBOUND (1/1) > (797de317bebf24df087f2da63cf5118e) switched from CREATED to DEPLOYING. > 2017-12-04 09:11:35,404 INFO > org.apache.flink.runtime.taskmanager.Task - Creating > FileSystem stream leak safety net for task Source: PACS008 Generator -> > Double Submission Generator -> Invalid IBAN Generator -> Sink: > IP-PREPROCESSOR-INBOUND (1/1) (797de317bebf24df087f2da63cf5118e) [DEPLOYING] > 2017-12-04 09:11:35,408 WARN > akka.serialization.Serialization(akka://flink) - Using > the default Java serializer for class > [org.apache.flink.runtime.messages.Acknowledge] which is not recommended > because of performance implications. Use another serializer or disable > this warning using the setting 'akka.actor.warn-about-java-serializer-usage' > 2017-12-04 09:11:35,410 INFO > org.apache.flink.runtime.taskmanager.Task - Loading > JAR files for task Source: PACS008 Generator -> Double Submission > Generator -> Invalid IBAN Generator -> Sink: IP-PREPROCESSOR-INBOUND > (1/1) (797de317bebf24df087f2da63cf5118e) [DEPLOYING]. > 2017-12-04 09:11:35,440 INFO > org.apache.flink.runtime.blob.BlobClient - > Downloading > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120d2b1b-392b55cacd4ed70b48e620f131a8bc61 > from flink-jobmanager/10.104.5.130:50000 > 2017-12-04 09:13:42,663 ERROR > org.apache.flink.runtime.blob.BlobClient - Failed > to fetch BLOB > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120d2b1b-392b55cacd4ed70b48e620f131a8bc61 > from flink-jobmanager/10.104.5.130:50000 and store it under > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-00000000 > Retrying... > 2017-12-04 09:13:42,663 INFO > org.apache.flink.runtime.blob.BlobClient - > Downloading > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120d2b1b-392b55cacd4ed70b48e620f131a8bc61 > from flink-jobmanager/10.104.5.130:50000 (retry 1) > 2017-12-04 09:15:49,895 ERROR > org.apache.flink.runtime.blob.BlobClient - Failed > to fetch BLOB > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120d2b1b-392b55cacd4ed70b48e620f131a8bc61 > from flink-jobmanager/10.104.5.130:50000 and store it under > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-00000000 > Retrying... > 2017-12-04 09:15:49,895 INFO > org.apache.flink.runtime.blob.BlobClient - > Downloading > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120d2b1b-392b55cacd4ed70b48e620f131a8bc61 > from flink-jobmanager/10.104.5.130:50000 (retry 2) > 2017-12-04 09:17:57,127 ERROR > org.apache.flink.runtime.blob.BlobClient - Failed > to fetch BLOB > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120d2b1b-392b55cacd4ed70b48e620f131a8bc61 > from flink-jobmanager/10.104.5.130:50000 and store it under > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-00000000 > Retrying... > 2017-12-04 09:17:57,127 INFO > org.apache.flink.runtime.blob.BlobClient - > Downloading > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120d2b1b-392b55cacd4ed70b48e620f131a8bc61 > from flink-jobmanager/10.104.5.130:50000 (retry 3) > 2017-12-04 09:20:04,359 ERROR > org.apache.flink.runtime.blob.BlobClient - Failed > to fetch BLOB > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120d2b1b-392b55cacd4ed70b48e620f131a8bc61 > from flink-jobmanager/10.104.5.130:50000 and store it under > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-00000000 > Retrying... > 2017-12-04 09:20:04,359 INFO > org.apache.flink.runtime.blob.BlobClient - > Downloading > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120d2b1b-392b55cacd4ed70b48e620f131a8bc61 > from flink-jobmanager/10.104.5.130:50000 (retry 4) > 2017-12-04 09:22:11,590 ERROR > org.apache.flink.runtime.blob.BlobClient - Failed > to fetch BLOB > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120d2b1b-392b55cacd4ed70b48e620f131a8bc61 > from flink-jobmanager/10.104.5.130:50000 and store it under > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-00000000 > Retrying... > 2017-12-04 09:22:11,591 INFO > org.apache.flink.runtime.blob.BlobClient - > Downloading > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120d2b1b-392b55cacd4ed70b48e620f131a8bc61 > from flink-jobmanager/10.104.5.130:50000 (retry 5) > 2017-12-04 09:24:18,823 ERROR > org.apache.flink.runtime.blob.BlobClient - Failed > to fetch BLOB > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120d2b1b-392b55cacd4ed70b48e620f131a8bc61 > from flink-jobmanager/10.104.5.130:50000 and store it under > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-00000000 > No retries left. > java.io.IOException: Could not connect to BlobServer at address > flink-jobmanager/10.104.5.130:50000 > > > _ ________________________________ _ > > > Landesbank Hessen-Thueringen Girozentrale > Anstalt des oeffentlichen Rechts > Sitz: Frankfurt am Main / Erfurt > Amtsgericht Frankfurt am Main, HRA 29821 / Amtsgericht Jena, HRA 102181 > > Bitte nutzen Sie die E-Mail-Verbindung mit uns ausschliesslich zum > Informationsaustausch. Wir koennen auf diesem Wege keine > rechtsgeschaeftlichen Erklaerungen (Auftraege etc.) entgegennehmen. > > Der Inhalt dieser Nachricht ist vertraulich und nur fuer den angegebenen > Empfaenger bestimmt. Jede Form der Kenntnisnahme oder Weitergabe durch > Dritte ist unzulaessig. Sollte diese Nachricht nicht fur Sie bestimmt > sein, so bitten wir Sie, sich mit uns per E-Mail oder telefonisch in > Verbindung zu setzen. > > Please use your E-mail connection with us exclusively for the exchange > of information. We do not accept legally binding declarations (orders, > etc.) by this means of communication. > > The contents of this message is confidential and intended only for the > recipient indicated. Taking notice of this message or disclosure by > third parties is not > permitted. In the event that this message is not intended for you, > please contact us via E-mail or phone.
signature.asc
Description: OpenPGP digital signature
