Re: Running Spark master/slave instances in non Daemon mode

2016-10-03 Thread Jakob Odersky
Hi Mike, I can imagine the trouble that daemonization is causing and I think that having non-forking start script is a good idea. A simple, non-intrusive, fix could be to change the "spark-daemon.sh" script to conditionally omit the "nohup &". Personally, I think the semantically correct approach w

Re: Running Spark master/slave instances in non Daemon mode

2016-09-29 Thread Mike Ihbe
Our particular use case is for Nomad, using the "exec" configuration described here: https://www.nomadproject.io/docs/drivers/exec.html. It's not exactly a container, just a cgroup. It performs a simple fork/exec of a command and binds to the output fds from that process, so daemonizing is causing

Re: Running Spark master/slave instances in non Daemon mode

2016-09-29 Thread Jakob Odersky
I'm curious, what kind of container solutions require foreground processes? Most init systems work fine with "starter" processes that run other processes. IIRC systemd and start-stop-daemon have an option called "fork", that will expect the main process to run another one in the background and only