bash-4.1$ ls -l /local/scratch/flink_historyserver_tmpdir/ total 8 drwxrwxr-x 3 p2epdlsuat p2epdlsuat 4096 Apr 21 10:43 flink-web-history-7fbb97cc-9f38-4844-9bcf-6272fe6828e9 drwxrwxr-x 3 p2epdlsuat p2epdlsuat 4096 Apr 21 10:22 flink-web-history-95b3f928-c60f-4351-9926-766c6ad3ee76
There are just two directories in here. I don't see cache directories from my attempts today, which is interesting. Looking a little deeper into them: bash-4.1$ ls -lr /local/scratch/flink_historyserver_tmpdir/flink-web-history-7fbb97cc-9f38-4844-9bcf-6272fe6828e9 total 1756 drwxrwxr-x 2 p2epdlsuat p2epdlsuat 1789952 Apr 21 10:44 jobs bash-4.1$ ls -lr /local/scratch/flink_historyserver_tmpdir/flink-web-history-7fbb97cc-9f38-4844-9bcf-6272fe6828e9/jobs total 0 -rw-rw-r-- 1 p2epdlsuat p2epdlsuat 0 Apr 21 10:43 overview.json There are indeed archives already in HDFS - I've included some in my initial mail, but here they are again just for reference: -bash-4.1$ hdfs dfs -ls /user/p2epda/lake/delp_qa/flink_hs Found 44282 items -rw-r----- 3 delp datalake_admin_dev 50569 2020-03-21 23:17 /user/p2epda/lake/delp_qa/flink_hs/000144dba9dc0f235768a46b2f26e936 -rw-r----- 3 delp datalake_admin_dev 49578 2020-03-03 08:45 /user/p2epda/lake/delp_qa/flink_hs/000347625d8128ee3fd0b672018e38a5 -rw-r----- 3 delp datalake_admin_dev 50842 2020-03-24 15:19 /user/p2epda/lake/delp_qa/flink_hs/0004be6ce01ba9677d1eb619ad0fa757 ... // ah From: Chesnay Schepler <ches...@apache.org> Sent: Monday, April 27, 2020 10:28 AM To: Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com>; user@flink.apache.org Subject: Re: History Server Not Showing Any Jobs - File Not Found? If historyserver.web.tmpdir is not set then java.io.tmpdir is used, so that should be fine. What are the contents of /local/scratch/flink_historyserver_tmpdir? I assume there are already archives in HDFS? On 27/04/2020 16:02, Hailu, Andreas wrote: My machine's /tmp directory is not large enough to support the archived files, so I changed my java.io.tmpdir to be in some other location which is significantly larger. I hadn't set anything for historyserver.web.tmpdir, so I suspect it was still pointing at /tmp. I just tried setting historyserver.web.tmpdir to the same location as my java.io.tmpdir location, but I'm afraid I'm still seeing the following issue: 2020-04-27 09:37:42,904 [nioEventLoopGroup-3-4] DEBUG HistoryServerStaticFileServerHandler - Unable to load requested file /overview.json from classloader 2020-04-27 09:37:42,906 [nioEventLoopGroup-3-6] DEBUG HistoryServerStaticFileServerHandler - Unable to load requested file /jobs/overview.json from classloader flink-conf.yaml for reference: jobmanager.archive.fs.dir: hdfs:///user/p2epda/lake/delp_qa/flink_hs/ historyserver.archive.fs.dir: hdfs:///user/p2epda/lake/delp_qa/flink_hs/ historyserver.web.tmpdir: /local/scratch/flink_historyserver_tmpdir/ Did you have anything else in mind when you said pointing somewhere funny? // ah From: Chesnay Schepler <ches...@apache.org><mailto:ches...@apache.org> Sent: Monday, April 27, 2020 5:56 AM To: Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com><mailto:andreas.ha...@ny.email.gs.com>; user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: History Server Not Showing Any Jobs - File Not Found? overview.json is a generated file that is placed in the local directory controlled by historyserver.web.tmpdir. Have you configured this option to point to some non-local filesystem? (Or if not, is the java.io.tmpdir property pointing somewhere funny?) On 24/04/2020 18:24, Hailu, Andreas wrote: I'm having a further look at the code in HistoryServerStaticFileServerHandler - is there an assumption about where overview.json is supposed to be located? // ah From: Hailu, Andreas [Engineering] Sent: Wednesday, April 22, 2020 1:32 PM To: 'Chesnay Schepler' <ches...@apache.org><mailto:ches...@apache.org>; Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com><mailto:andreas.ha...@ny.email.gs.com>; user@flink.apache.org<mailto:user@flink.apache.org> Subject: RE: History Server Not Showing Any Jobs - File Not Found? Hi Chesnay, thanks for responding. We're using Flink 1.9.1. I enabled DEBUG level logging and this is something relevant I see: 2020-04-22 13:25:52,566 [Flink-HistoryServer-ArchiveFetcher-thread-1] DEBUG DFSInputStream - Connecting to datanode 10.79.252.101:1019 2020-04-22 13:25:52,567 [Flink-HistoryServer-ArchiveFetcher-thread-1] DEBUG SaslDataTransferClient - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2020-04-22 13:25:52,567 [Flink-HistoryServer-ArchiveFetcher-thread-1] DEBUG SaslDataTransferClient - SASL client skipping handshake in secured configuration with privileged port for addr = /10.79.252.101, datanodeId = DatanodeI nfoWithStorage[10.79.252.101:1019,DS-7f4ec55d-7c5f-4a0e-b817-d9e635480b21,DISK] 2020-04-22 13:25:52,571 [Flink-HistoryServer-ArchiveFetcher-thread-1] DEBUG DFSInputStream - DFSInputStream has been closed already 2020-04-22 13:25:52,573 [nioEventLoopGroup-3-6] DEBUG HistoryServerStaticFileServerHandler - Unable to load requested file /jobs/overview.json from classloader 2020-04-22 13:25:52,576 [IPC Parameter Sending Thread #0] DEBUG Client$Connection$3 - IPC Client (1578587450) connection to d279536-002.dc.gs.com/10.59.61.87:8020 from d...@gs.com<mailto:d...@gs.com> sending #1391 Aside from that, it looks like a lot of logging around datanodes and block location metadata. Did I miss something in my classpath, perhaps? If so, do you have a suggestion on what I could try? // ah From: Chesnay Schepler <ches...@apache.org<mailto:ches...@apache.org>> Sent: Wednesday, April 22, 2020 2:16 AM To: Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com<mailto:andreas.ha...@ny.email.gs.com>>; user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: History Server Not Showing Any Jobs - File Not Found? Which Flink version are you using? Have you checked the history server logs after enabling debug logging? On 21/04/2020 17:16, Hailu, Andreas [Engineering] wrote: Hi, I'm trying to set up the History Server, but none of my applications are showing up in the Web UI. Looking at the console, I see that all of the calls to /overview return the following 404 response: {"errors":["File not found."]}. I've set up my configuration as follows: JobManager Archive directory: jobmanager.archive.fs.dir: hdfs:///user/p2epda/lake/delp_qa/flink_hs/ -bash-4.1$ hdfs dfs -ls /user/p2epda/lake/delp_qa/flink_hs Found 44282 items -rw-r----- 3 delp datalake_admin_dev 50569 2020-03-21 23:17 /user/p2epda/lake/delp_qa/flink_hs/000144dba9dc0f235768a46b2f26e936 -rw-r----- 3 delp datalake_admin_dev 49578 2020-03-03 08:45 /user/p2epda/lake/delp_qa/flink_hs/000347625d8128ee3fd0b672018e38a5 -rw-r----- 3 delp datalake_admin_dev 50842 2020-03-24 15:19 /user/p2epda/lake/delp_qa/flink_hs/0004be6ce01ba9677d1eb619ad0fa757 ... ... History Server will fetch the archived jobs from the same location: historyserver.archive.fs.dir: hdfs:///user/p2epda/lake/delp_qa/flink_hs/ So I'm able to confirm that there are indeed archived applications that I should be able to view in the histserver. I'm not able to find out what file the overview service is looking for from the repository - any suggestions as to what I could look into next? Best, Andreas ________________________________ Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices> ________________________________ Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices> ________________________________ Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices> ________________________________ Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>
