The Ops guy would probably be fired if he doesn't make sure the container runtime is up 24/7. 🙂 ________________________________ From: Mich Talebzadeh <mich.talebza...@gmail.com> Sent: Wednesday, December 8, 2021 12:33 PM Cc: user @spark <user@spark.apache.org> Subject: Re: start-history-server.sh doesn't survive system reboot. Recommendation?
Well that is just relying on docker daemon to start after reboot. It may not docker ps Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? systemctl start docker docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES So your mileage varies. A proper way would be to have two instances of start-history-server.sh running on different hosts listening to the same Spark, thus avoiding a single point of failure. HTH [https://docs.google.com/uc?export=download&id=1-q7RFGRfLMObPuQPWSd9sl_H1UPNFaIZ&revid=0B1BiUVX33unjMWtVUWpINWFCd0ZQTlhTRHpGckh4Wlg4RG80PQ] view my Linkedin profile<https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!DvJyiiI1!bR1iEMzC4cyfUQYQRuFn4GzLtS2Qq2vn7gux-j3NU7HJmDM9jWzQgrP4D2HaDw$> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Wed, 8 Dec 2021 at 19:45, James Yu <ja...@ispot.tv<mailto:ja...@ispot.tv>> wrote: Just thought about another possibility which is to containerize the history server and run the container with proper restart policy. This may be the approach we will be taking because the deployment of such HS would be more flexible. Thanks! ________________________________ From: Sean Owen <sro...@gmail.com<mailto:sro...@gmail.com>> Sent: Tuesday, December 7, 2021 1:29 PM To: James Yu <ja...@ispot.tv<mailto:ja...@ispot.tv>> Cc: user @spark <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Re: start-history-server.sh doesn't survive system reboot. Recommendation? The scripts just launch the processes. To make any process restart on system restart, you would need to set it up as a system service (i.e. controlled by 'service'). That's easy enough but out of scope for Spark itself. On Tue, Dec 7, 2021 at 3:23 PM James Yu <ja...@ispot.tv<mailto:ja...@ispot.tv>> wrote: Hi Users, We found that the history server launched by using the "start-history-server.sh" command does not survive system reboot. Any recommendation of making it always up even after reboot? Thanks, James
