Hi Bernd, just to make sure, is your setup working now as expected?
Thanks, Fabian 2017-12-06 8:45 GMT+01:00 <bernd.winterst...@dev.helaba.de>: > Hi Nico > I think there were changes in the default port fort the BLOB server. I > missed the fact that the Kubernetes configuration was still exposing 6124 > for the JobManager BLOB server. > Thanks > > Bernd > > -----Ursprüngliche Nachricht----- > Von: Nico Kruber [mailto:n...@data-artisans.com] > Gesendet: Montag, 4. Dezember 2017 14:17 > An: Winterstein, Bernd; user@flink.apache.org > Betreff: Re: Blob server not working with 1.4.0.RC2 > > Hi Bernd, > thanks for the report. I tried to reproduce it locally but both a telnet > connection to the BlobServer as well as the BLOB download by the > TaskManagers work for me. Can you share your configuration that is causing > the problem? You could also try increasing the log level to DEBUG and see > if there is something more in the logs (the exception thrown in > StaticFileServerHandler looks suspicious but is not related to the > BlobServer). > Apparently, the TaskManager resolves flink-jobmanager to 10.104.5.130. > Is that the correct address and can the TaskManager talk to this IP? > (may a firewall block this?) > > Did you, by any chance, set up SSL, too? There was a recent thread on the > mailing list [1] where a had some problems with > "security.ssl.verify-hostname" being set to true which may be related. > > > Nico > > [1] > https://lists.apache.org/thread.html/879d072bfd6761947b4bd703324489 > db50e8b14c328992118af875d8@%3Cuser.flink.apache.org%3E > > On 04/12/17 10:03, bernd.winterst...@dev.helaba.de wrote: > > Hi > > Since we switched to Release 1.4 the taskmanagers are unable to > > download blobs from the jobmanager. > > The taskmanager registration still works. > > Netstat on jobmanager shows open ports at 6123 and 50000. But a telnet > > connection from taskmanager to jobmanager on port 50000 times out. > > > > Any ideas are welcome. > > > > Regards > > > > Bernd > > > > Jobmanager log: > > > > 2017-12-04 08:48:30,167 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > Starting JobManager actor > > 2017-12-04 08:48:30,197 INFO > > org.apache.flink.runtime.blob.BlobServer - > > Created BLOB server storage directory > > /tmp/blobStore-81cd12c7-394e-4777-85a1-98389b72dd08 > > 2017-12-04 08:48:30,205 INFO > > org.apache.flink.runtime.blob.BlobServer - > > Started BLOB server at 0.0.0.0:50000 - max concurrent requests: 50 - > > max > > backlog: 1000 > > 2017-12-04 08:48:30,608 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > Starting JobManager at akka.tcp://flink@flink-jobmanager:6123/user/ > jobmanager. > > 2017-12-04 08:48:30,628 INFO > > org.apache.flink.runtime.jobmanager.MemoryArchivist - > > Started memory archivist akka://flink/user/archive > > 2017-12-04 08:48:30,676 INFO > > org.apache.flink.runtime.clusterframework.standalone.StandaloneResourc > > eManager > > - Trying to associate with JobManager leader > > akka.tcp://flink@flink-jobmanager:6123/user/jobmanager > > 2017-12-04 08:48:30,692 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > JobManager akka.tcp://flink@flink-jobmanager:6123/user/jobmanager was > > granted leadership with leader session ID > > Some(00000000-0000-0000-0000-000000000000). > > 2017-12-04 08:48:30,700 INFO > > org.apache.flink.runtime.clusterframework.standalone.StandaloneResourc > > eManager > > - Resource Manager associating with leading JobManager > > Actor[akka://flink/user/jobmanager#886586058] - leader session > > 00000000-0000-0000-0000-000000000000 > > 2017-12-04 08:53:50,635 INFO > > org.apache.flink.runtime.clusterframework.standalone.StandaloneResourc > > eManager > > - TaskManager 627338086a766c140909ba45f2e717d0 has started. > > 2017-12-04 08:53:50,638 INFO > > org.apache.flink.runtime.instance.InstanceManager - > > Registered TaskManager at flink-taskmanager-65cf757d9b-hj65d > > (akka.tcp://flink@flink-taskmanager-65cf757d9b-hj65d:45932/user/taskma > > nager) as f9d2843d0223b15d8fce52aea8231cc6. Current number of > > registered hosts is 1. Current number of alive task slots is 8. > > 2017-12-04 08:53:50,658 WARN > > akka.serialization.Serialization(akka://flink) - Using > > the default Java serializer for class > > [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMes > > sage] which is not recommended because of performance implications. > > Use another serializer or disable this warning using the setting > > 'akka.actor.warn-about-java-serializer-usage' > > 2017-12-04 08:53:55,714 INFO > > org.apache.flink.runtime.clusterframework.standalone.StandaloneResourc > > eManager > > - TaskManager 08c3e6f7c765e2ab88e2ea645049cb9d has started. > > 2017-12-04 08:53:55,714 INFO > > org.apache.flink.runtime.instance.InstanceManager - > > Registered TaskManager at flink-taskmanager-65cf757d9b-jtzw5 > > (akka.tcp://flink@flink-taskmanager-65cf757d9b-jtzw5:41710/user/taskma > > nager) as da8a8da3650ce53f460784c54938a071. Current number of > > registered hosts is 2. Current number of alive task slots is 16. > > 2017-12-04 09:04:08,850 ERROR > > org.apache.flink.runtime.rest.handler.legacy.files.StaticFileServerHan > > dler > > - Caught exception > > java.io.IOException: Operation timed out > > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > > at sun.nio.ch.IOUtil.read(IOUtil.java:192) > > at > > sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) > > > > > > Taskmanager log: > > 2017-12-04 08:53:55,511 INFO > > org.apache.flink.runtime.taskmanager.TaskManager - > > Starting TaskManager actor at akka://flink/user/taskmanager#-977591027. > > 2017-12-04 08:53:55,511 INFO > > org.apache.flink.runtime.taskmanager.TaskManager - > > TaskManager data connection information: > > 08c3e6f7c765e2ab88e2ea645049cb9d @ flink-taskmanager-65cf757d9b-jtzw5 > > (dataPort=42142) > > 2017-12-04 08:53:55,512 INFO > > org.apache.flink.runtime.taskmanager.TaskManager - > > TaskManager has 8 task slot(s). > > 2017-12-04 08:53:55,513 INFO > > org.apache.flink.runtime.taskmanager.TaskManager - Memory > > usage stats: [HEAP: 131/981/981 MB, NON HEAP: 34/35/-1 MB > > (used/committed/max)] > > 2017-12-04 08:53:55,518 INFO > > org.apache.flink.runtime.taskmanager.TaskManager - Trying > > to register at JobManager > > akka.tcp://flink@flink-jobmanager:6123/user/jobmanager (attempt 1, > > timeout: 500 milliseconds) > > 2017-12-04 08:53:55,671 WARN > > akka.serialization.Serialization(akka://flink) - Using > > the default Java serializer for class > > [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMes > > sage] which is not recommended because of performance implications. > > Use another serializer or disable this warning using the setting > > 'akka.actor.warn-about-java-serializer-usage' > > 2017-12-04 08:53:55,737 INFO > > org.apache.flink.runtime.taskmanager.TaskManager - > > Successful registration at JobManager > > (akka.tcp://flink@flink-jobmanager:6123/user/jobmanager), starting > > network stack and library cache. > > 2017-12-04 08:53:55,742 INFO > > org.apache.flink.runtime.taskmanager.TaskManager - > > Determined BLOB server address to be > > flink-jobmanager/10.104.5.130:50000. Starting BLOB cache. > > 2017-12-04 08:53:55,752 INFO > > org.apache.flink.runtime.blob.PermanentBlobCache - > > Created BLOB cache storage directory > > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5 > > 2017-12-04 08:53:55,776 INFO > > org.apache.flink.runtime.blob.TransientBlobCache - > > Created BLOB cache storage directory > > /tmp/blobStore-8d7c8660-4455-43e9-93be-dadeb622b9e1 > > 2017-12-04 08:54:00,796 WARN > > akka.serialization.Serialization(akka://flink) - Using > > the default Java serializer for class > > [org.apache.flink.runtime.messages.TaskManagerMessages$Heartbeat] > > which is not recommended because of performance implications. Use > > another serializer or disable this warning using the setting > > 'akka.actor.warn-about-java-serializer-usage' > > 2017-12-04 09:10:38,726 WARN > > akka.serialization.Serialization(akka://flink) - Using > > the default Java serializer for class > > [org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricS > > erializationResult] which is not recommended because of performance > > implications. Use another serializer or disable this warning using the > > setting 'akka.actor.warn-about-java-serializer-usage' > > 2017-12-04 09:11:35,403 INFO > > org.apache.flink.runtime.taskmanager.TaskManager - > > Received task Source: PACS008 Generator -> Double Submission Generator > > -> Invalid IBAN Generator -> Sink: IP-PREPROCESSOR-INBOUND (1/1) > > 2017-12-04 09:11:35,404 INFO > > org.apache.flink.runtime.taskmanager.Task - Source: > > PACS008 Generator -> Double Submission Generator -> Invalid IBAN > > Generator -> Sink: IP-PREPROCESSOR-INBOUND (1/1) > > (797de317bebf24df087f2da63cf5118e) switched from CREATED to DEPLOYING. > > 2017-12-04 09:11:35,404 INFO > > org.apache.flink.runtime.taskmanager.Task - > > Creating FileSystem stream leak safety net for task Source: PACS008 > > Generator -> Double Submission Generator -> Invalid IBAN Generator -> > Sink: > > IP-PREPROCESSOR-INBOUND (1/1) (797de317bebf24df087f2da63cf5118e) > > [DEPLOYING] > > 2017-12-04 09:11:35,408 WARN > > akka.serialization.Serialization(akka://flink) - Using > > the default Java serializer for class > > [org.apache.flink.runtime.messages.Acknowledge] which is not > > recommended because of performance implications. Use another > > serializer or disable this warning using the setting > 'akka.actor.warn-about-java-serializer-usage' > > 2017-12-04 09:11:35,410 INFO > > org.apache.flink.runtime.taskmanager.Task - > > Loading JAR files for task Source: PACS008 Generator -> Double > > Submission Generator -> Invalid IBAN Generator -> Sink: > > IP-PREPROCESSOR-INBOUND > > (1/1) (797de317bebf24df087f2da63cf5118e) [DEPLOYING]. > > 2017-12-04 09:11:35,440 INFO > > org.apache.flink.runtime.blob.BlobClient - > > Downloading > > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120 > > d2b1b-392b55cacd4ed70b48e620f131a8bc61 > > from flink-jobmanager/10.104.5.130:50000 > > 2017-12-04 09:13:42,663 ERROR > > org.apache.flink.runtime.blob.BlobClient - Failed > > to fetch BLOB > > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120 > > d2b1b-392b55cacd4ed70b48e620f131a8bc61 > > from flink-jobmanager/10.104.5.130:50000 and store it under > > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-0000 > > 0000 > > Retrying... > > 2017-12-04 09:13:42,663 INFO > > org.apache.flink.runtime.blob.BlobClient - > > Downloading > > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120 > > d2b1b-392b55cacd4ed70b48e620f131a8bc61 > > from flink-jobmanager/10.104.5.130:50000 (retry 1) > > 2017-12-04 09:15:49,895 ERROR > > org.apache.flink.runtime.blob.BlobClient - Failed > > to fetch BLOB > > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120 > > d2b1b-392b55cacd4ed70b48e620f131a8bc61 > > from flink-jobmanager/10.104.5.130:50000 and store it under > > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-0000 > > 0000 > > Retrying... > > 2017-12-04 09:15:49,895 INFO > > org.apache.flink.runtime.blob.BlobClient - > > Downloading > > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120 > > d2b1b-392b55cacd4ed70b48e620f131a8bc61 > > from flink-jobmanager/10.104.5.130:50000 (retry 2) > > 2017-12-04 09:17:57,127 ERROR > > org.apache.flink.runtime.blob.BlobClient - Failed > > to fetch BLOB > > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120 > > d2b1b-392b55cacd4ed70b48e620f131a8bc61 > > from flink-jobmanager/10.104.5.130:50000 and store it under > > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-0000 > > 0000 > > Retrying... > > 2017-12-04 09:17:57,127 INFO > > org.apache.flink.runtime.blob.BlobClient - > > Downloading > > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120 > > d2b1b-392b55cacd4ed70b48e620f131a8bc61 > > from flink-jobmanager/10.104.5.130:50000 (retry 3) > > 2017-12-04 09:20:04,359 ERROR > > org.apache.flink.runtime.blob.BlobClient - Failed > > to fetch BLOB > > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120 > > d2b1b-392b55cacd4ed70b48e620f131a8bc61 > > from flink-jobmanager/10.104.5.130:50000 and store it under > > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-0000 > > 0000 > > Retrying... > > 2017-12-04 09:20:04,359 INFO > > org.apache.flink.runtime.blob.BlobClient - > > Downloading > > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120 > > d2b1b-392b55cacd4ed70b48e620f131a8bc61 > > from flink-jobmanager/10.104.5.130:50000 (retry 4) > > 2017-12-04 09:22:11,590 ERROR > > org.apache.flink.runtime.blob.BlobClient - Failed > > to fetch BLOB > > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120 > > d2b1b-392b55cacd4ed70b48e620f131a8bc61 > > from flink-jobmanager/10.104.5.130:50000 and store it under > > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-0000 > > 0000 > > Retrying... > > 2017-12-04 09:22:11,591 INFO > > org.apache.flink.runtime.blob.BlobClient - > > Downloading > > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120 > > d2b1b-392b55cacd4ed70b48e620f131a8bc61 > > from flink-jobmanager/10.104.5.130:50000 (retry 5) > > 2017-12-04 09:24:18,823 ERROR > > org.apache.flink.runtime.blob.BlobClient - Failed > > to fetch BLOB > > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120 > > d2b1b-392b55cacd4ed70b48e620f131a8bc61 > > from flink-jobmanager/10.104.5.130:50000 and store it under > > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-0000 > > 0000 > > No retries left. > > java.io.IOException: Could not connect to BlobServer at address > > flink-jobmanager/10.104.5.130:50000 > > > > > > _ ________________________________ _ > > > > > > Landesbank Hessen-Thueringen Girozentrale Anstalt des oeffentlichen > > Rechts > > Sitz: Frankfurt am Main / Erfurt > > Amtsgericht Frankfurt am Main, HRA 29821 / Amtsgericht Jena, HRA > > 102181 > > > > Bitte nutzen Sie die E-Mail-Verbindung mit uns ausschliesslich zum > > Informationsaustausch. Wir koennen auf diesem Wege keine > > rechtsgeschaeftlichen Erklaerungen (Auftraege etc.) entgegennehmen. > > > > Der Inhalt dieser Nachricht ist vertraulich und nur fuer den > > angegebenen Empfaenger bestimmt. Jede Form der Kenntnisnahme oder > > Weitergabe durch Dritte ist unzulaessig. Sollte diese Nachricht nicht > > fur Sie bestimmt sein, so bitten wir Sie, sich mit uns per E-Mail oder > > telefonisch in Verbindung zu setzen. > > > > Please use your E-mail connection with us exclusively for the exchange > > of information. We do not accept legally binding declarations (orders, > > etc.) by this means of communication. > > > > The contents of this message is confidential and intended only for the > > recipient indicated. Taking notice of this message or disclosure by > > third parties is not permitted. In the event that this message is not > > intended for you, please contact us via E-mail or phone. > > > ________________________________ > > > Landesbank Hessen-Thueringen Girozentrale > Anstalt des oeffentlichen Rechts > Sitz: Frankfurt am Main / Erfurt > Amtsgericht Frankfurt am Main, HRA 29821 / Amtsgericht Jena, HRA 102181 > > Bitte nutzen Sie die E-Mail-Verbindung mit uns ausschliesslich zum > Informationsaustausch. Wir koennen auf diesem Wege keine > rechtsgeschaeftlichen Erklaerungen (Auftraege etc.) entgegennehmen. > > Der Inhalt dieser Nachricht ist vertraulich und nur fuer den angegebenen > Empfaenger bestimmt. Jede Form der Kenntnisnahme oder Weitergabe durch > Dritte ist unzulaessig. Sollte diese Nachricht nicht fur Sie bestimmt sein, > so bitten wir Sie, sich mit uns per E-Mail oder telefonisch in Verbindung > zu setzen. > > Please use your E-mail connection with us exclusively for the exchange of > information. We do not accept legally binding declarations (orders, etc.) > by this means of communication. > > The contents of this message is confidential and intended only for the > recipient indicated. Taking notice of this message or disclosure by third > parties is not > permitted. In the event that this message is not intended for you, please > contact us via E-mail or phone. >
