When the task launches, a sandbox directory is created, in the above e-mail it was /tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781. Is there anything else in that directory? Please post anything logs you find in there as they may provide useful clues.
Possibly unrelated, your slave work directory is /tmp. I have observed that some environments a temp cleaner process runs that automatically deletes things under here. This may not fix the issue at hand here, but i suggest you move this outside /tmp as it is critical state for the slave. -=Bill On Fri, Mar 13, 2015 at 9:24 AM, Oskar Stenman <oskar.sten...@magine.com> wrote: > I wasn't subscribed to the mailing-list (I'm subscribed now though) so > i'm sorry if this reply ends up in the wrong place.. > > >From the mesos slave log it looks like the executor is failing. Most > likely the issue is your image doesn't have the native libraries needed to > run it. The next step would be to look in the sandbox for a failed run > (you can find the path in the slave logs) and look at the stderr log for > errors. > > Everything's included in the image, that's the image i'm running > mesos-master, aurora, and mesos-slave from. > > Stdout was included in the last email: > > WARNING: Logging before InitGoogleLogging() is written to STDERR > I0313 13:39:45.608578 258 fetcher.cpp:76] Fetching URI > '/usr/local/bin/thermos_executor.pex' > I0313 13:39:45.609048 258 fetcher.cpp:179] Copying resource from > '/usr/local/bin/thermos_executor.pex' to > > '/tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781' > > Also, when relaunching the container manually: > > > 1. root@s1:/var/log/upstart# docker start -i > mesos-cdf5e59f-c2be-47ba-b30a-2a690657e248 > 2. twitter.common.app debug: Initializing: twitter.common.log > (Logging subsystem.) > 3. Writing log files to disk in /mnt/mesos/sandbox > 4. I0313 14:50:36.273181 5 exec.cpp:132] Version: 0.21.1 > 5. I0313 14:50:36.278043 29 exec.cpp:379] Executor asked to shutdown > 6. Killed > > > It seems to be really tricky to troubleshoot from my point of view as > i don't have any output at all. > Is the executor giving up immediately? is it even trying to connect to > the slave? > Parameters to the executor missing? (like job-config or something) > Directory doesn't contain what it wants? > Network config wrong? > > /Oskar > > > On Fri, Mar 13, 2015 at 3:08 PM, Oskar Stenman <oskar.sten...@magine.com> > wrote: > > > Hi! > > > > I'm investigating aurora + mesos + docker and i'm stuck. > > > > I can create the hello world docker-task in aurora, it gets assigned a > > slave, the docker-container is launched but the executor immediately > > terminates and it ends up in "task lost" state. > > > > Can anyone make any sense of this or tell me how to troubleshoot further? > > > > > > -- > > [image: MagineTV] > > > > *Oskar Stenman* > > Network Architect > > > > *Magine TV* > > oskar.sten...@magine.com | Mob: +46 70 565 21 52 > > Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com > > <http://www.magine.com/> > > > > Privileged and/or Confidential Information may be contained in this > > message. If you are not the addressee indicated in this message > > (or responsible for delivery of the message to such a person), you may > not > > copy or deliver this message to anyone. In such case, > > you should destroy this message and kindly notify the sender by reply > > email. > > > > > > -- > [image: MagineTV] > > *Oskar Stenman* > Network Architect > > *Magine TV* > oskar.sten...@magine.com | Mob: +46 70 565 21 52 > Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com > <http://www.magine.com/> > > Privileged and/or Confidential Information may be contained in this > message. If you are not the addressee indicated in this message > (or responsible for delivery of the message to such a person), you may not > copy or deliver this message to anyone. In such case, > you should destroy this message and kindly notify the sender by reply > email. >
