I am using a `JupyterHub` deployment with `SwarmSpawner`. When I try to
login, `JupyterHub` fails spawning the new container. I have the debug mode
in all the configuration but the logs doesn't give me enoght information.
I am using the following components:
- Docker version 18.09.2
- JupyterHub version 0.9.4
- Jupyter Notebook image: datascience-notebook:59b402ce701d
- dockerspawner version 0.10.0 (swarmspawner)
- dummyauthenticator
This is my configuration:
*Dockerfile*:
ARG JUPYTERHUB_VERSION=0.9.4
FROM jupyterhub/jupyterhub:${JUPYTERHUB_VERSION}
RUN pip install --upgrade pip
RUN pip install --no-cache \
jupyterhub-dummyauthenticator \
dockerspawner==0.10.0
COPY jupyterhub_config.py /srv/jupyterhub/jupyterhub_config.py
*docker-compose.yml*:
version: "3.7"
services:
jupyterhub:
image: hubimport os
notebook = {
'dir' : os.environ.get('DOCKER_NOTEBOOK_DIR') or '~',
'image' : os.environ.get('DOCKER_NOTEBOOK_IMAGE') or
'jupyter/datascience-notebook:59b402ce701d',
'network' : os.environ.get('DOCKER_NETWORK_NAME') or 'jupyterhub_net',
}
# Spawner
c.Spawner.cmd = ['jupyterhub-singleuser']
c.Spawner.start_timeout = 60
c.Spawner.http_timeout = 30
# Notebook Resources
c.Spawner.mem_limit =
'300M'
c.Spawner.cpu_limit =
0.20
# SwarmSpawner
c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'
c.SwarmSpawner.image = notebook['image']
c.JupyterHub.ip = '0.0.0.0'
c.JupyterHub.port = 8000
c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.hub_connect_ip = 'mussol.cpl.upc.edu'
#
Authentification
c.JupyterHub.authenticator_class = 'dummyauthenticator.DummyAuthenticator'
c.DummyAuthenticator.password = "passwd"
c.Authenticator.whitelist = {'joan'}
c.Authenticator.admin_users = {'joan'}
#
Debug
c.Application.log_level = 1
c.JupyterHub.log_level = 'DEBUG'
c.Spawner.debug = True
c.SwarmSpawner.debug = True
from docker.types import Mount,DriverConfig
mounts = [
Mount(type='volume',
source='jupyterhub-user-{username}',
target=notebook['dir'],
driver_config=DriverConfig('vieux/sshfs'),
),
Mount(type='bind',
source='/var/run/docker.sock',
target='/var/run/docker.sock',
),
]
c.SwarmSpawner.extra_container_spec = {
'mounts' : mounts
}
c.SwarmSpawner.network_name = notebook['network']
c.SwarmSpawner.extra_host_config = { 'network_mode' : notebook['network'] }
hostname: jupyterhub_mussol
deploy:
replicas: 1
restart_policy:
condition: on-failure
# assegurem execució al swarm manager
placement:
constraints:
- node.role == manager
ports:
- "8000:8000"
- "8081:8081"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
networks:# pels notebooks (standalone containers)
- jupyterhub_net
environment:
DOCKER_NOTEBOOK_DIR: /home/jovyan/work
DOCKER_NOTEBOOK_IMAGE: jupyter/datascience-notebook:59b402ce701d
DOCKER_NETWORK_NAME: jupyterhub_net
networks:
jupyterhub_net:
name: jupyterhub_net
driver: overlay
attachable: true
*jupyterhub_config.py*:
import os
notebook = {
'dir' : os.environ.get('DOCKER_NOTEBOOK_DIR') or '~',
'image' : os.environ.get('DOCKER_NOTEBOOK_IMAGE') or
'jupyter/datascience-notebook:59b402ce701d',
'network' : os.environ.get('DOCKER_NETWORK_NAME') or 'jupyterhub_net',
}
# Spawner
c.Spawner.cmd = ['jupyterhub-singleuser']
c.Spawner.start_timeout = 60
c.Spawner.http_timeout = 30
# Notebook Resources
c.Spawner.mem_limit = '300M'
c.Spawner.cpu_limit = 0.20
# SwarmSpawner
c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'
c.SwarmSpawner.image = notebook['image']
c.JupyterHub.ip = '0.0.0.0'
c.JupyterHub.port = 8000
c.JupyterHub.hub_ip = '0.0.*jupyterhub_config.py*:0.0'
c.JupyterHub.hub_connect_ip = 'mussol.cpl.upc.edu'
This is my configuration:
# Authentification
c.JupyterHub.authenticator_class = 'dummyauthenticator.DummyAuthenticator'
c.DummyAuthenticator.password = "passwd"
c.Authenticator.whitelist = {'joan'}
c.Authenticator.admin_users = {'joan'}
# Debug
c.Application.log_level = 1
c.JupyterHub.log_level = 'DEBUG'
c.Spawner.debug = True
c.SwarmSpawner.debug = True
from docker.types import Mount,DriverConfig
mounts = [
Mount(type='volume',
source='jupyterhub-user-{username}',
target=notebook['dir'],
driver_config=DriverConfig('vieux/sshfs'),
),
Mount(type='bind',
source='/var/run/docker.sock',
target='/var/run/docker.sock',
),
]
c.SwarmSpawner.extra_container_spec = {
'mounts' : mounts
}
c.SwarmSpawner.network_name = notebook['network']
c.SwarmSpawner.extra_host_config = { 'network_mode' : notebook['network'] }
Here the *log* trying to login to the server:
[I 2019-05-07 16:05:52.273 JupyterHub app:1673] Using Authenticator:
dummyauthenticator.dummyauthenticator.DummyAuthenticator
[I 2019-05-07 16:05:52.274 JupyterHub app:1673] Using Spawner:
dockerspawner.swarmspawner.SwarmSpawner-0.10.0
[D 2019-05-07 16:05:52.278 JupyterHub app:1050] Generating new cookie_secret
[I 2019-05-07 16:05:52.279 JupyterHub app:1055] Writing cookie_secret to
/srv/jupyterhub/jupyterhub_cookie_secret
[D 2019-05-07 16:05:52.280 JupyterHub app:1071] Connecting to db:
sqlite:///jupyterhub.sqlite
[D 2019-05-07 16:05:52.303 JupyterHub orm:656] Stamping empty database with
alembic revision 896818069c98
[I 2019-05-07 16:05:52.310 alembic.runtime.migration migration:117] Context
impl SQLiteImpl.
[I 2019-05-07 16:05:52.310 alembic.runtime.migration migration:122] Will
assume non-transactional DDL.
[I 2019-05-07 16:05:52.423 alembic.runtime.migration migration:327] Running
stamp_revision -> 896818069c98
[D 2019-05-07 16:05:52.424 alembic.runtime.migration migration:516] new
branch insert 896818069c98
[I 2019-05-07 16:05:53.463 JupyterHub proxy:431] Generating new
CONFIGPROXY_AUTH_TOKEN
[D 2019-05-07 16:05:53.582 JupyterHub app:1473] Loading state for joan from
db
[D 2019-05-07 16:05:53.585 JupyterHub app:1489] Loaded users:
joan admin
[I 2019-05-07 16:05:53.599 JupyterHub app:1855] Hub API listening on
http://0.0.0.0:8081/hub/
[W 2019-05-07 16:05:53.600 JupyterHub proxy:565] Running JupyterHub without
SSL. I hope there is SSL termination happening somewhere else...
[I 2019-05-07 16:05:53.600 JupyterHub proxy:567] Starting proxy @
http://0.0.0.0:8000/
[D 2019-05-07 16:05:53.600 JupyterHub proxy:568] Proxy cmd:
['configurable-http-proxy', '--ip', '0.0.0.0', '--port', '8000',
'--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target',
'http://0.0.0.0:8081/hub/error']
[D 2019-05-07 16:05:53.608 JupyterHub proxy:517] Writing proxy pid file:
jupyterhub-proxy.pid
16:05:54.141 - info: [ConfigProxy] Proxying http://0.0.0.0:8000 to (no
default)
16:05:54.144 - info: [ConfigProxy] Proxy API at
http://127.0.0.1:8001/api/routes
[D 2019-05-07 16:05:54.189 JupyterHub proxy:603] Proxy started and appears
to be up
[D 2019-05-07 16:05:54.190 JupyterHub proxy:296] Fetching routes to check
[D 2019-05-07 16:05:54.192 JupyterHub proxy:686] Proxy: Fetching GET
http://127.0.0.1:8001/api/routes
16:05:54.205 - info: [ConfigProxy] 200 GET /api/routes
[I 2019-05-07 16:05:54.206 JupyterHub proxy:301] Checking routes
[I 2019-05-07 16:05:54.206 JupyterHub proxy:370] Adding default route for
Hub: / => http://0.0.0.0:8081
[D 2019-05-07 16:05:54.206 JupyterHub proxy:686] Proxy: Fetching POST
http://127.0.0.1:8001/api/routes/
16:05:54.209 - info: [ConfigProxy] Adding route / -> http://0.0.0.0:8081
16:05:54.210 - info: [ConfigProxy] 201 POST /api/routes/
[I 2019-05-07 16:05:54.210 Jupyt```erHub*log* app:1912] JupyterHub is now
running at http://0.0.0.0:8000/
[I 2019-05-07 16:06:07.027 JupyterHub log:158] 302 GET / -> /hub
(@10.255.0.2) 1.71ms
[W 2019-05-07 16:06:07.058 JupyterHub base:242] Invalid or expired cookie
token
[I 2019-05-07 16:06:07.058 JupyterHub log:158] 302 GET /hub -> /hub/
(@10.255.0.2) 1.01ms
[W 2019-05-07 16:06:07.070 JupyterHub base:242] Invalid or expired cookie
token
[I 2019-05-07 16:06:07.071 JupyterHub log:158] 302 GET /hub/ -> /hub/login
(@10.255.0.2) 1.64ms
[I 2019-05-07 16:06:07.132 JupyterHub log:158] 200 GET /hub/login
(@10.255.0.2) 52.47ms
[D 2019-05-07 16:06:10.255 JupyterHub base:362] Setting cookie for joan:
jupyterhub-hub-login
[D 2019-05-07 16:06:10.255 JupyterHub base:357] Setting cookie
jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2019-05-07 16:06:10.255 JupyterHub base:499] User logged in: joan
[I 2019-05-07 16:06:10.256 JupyterHub log:158] 302 POST /hub/login?next= ->
/user/joan/ ([email protected]) 13.01ms
[I 2019-05-07 16:06:10.270 JupyterHub log:158] 302 GET /user/joan/ ->
/hub/user/joan/ (@10.255.0.2) 0.95ms
[D 2019-05-07 16:06:10.371 JupyterHub user:190] Creating <class
'dockerspawner.swarmspawner.SwarmSpawner'> for joan:
[D 2019-05-07 16:06:10.442 JupyterHub base:603] Initiating spawn for joan
[D 2019-05-07 16:06:10.442 JupyterHub base:609] 0/100 concurrent spawns
[D 2019-05-07 16:06:10.442 JupyterHub base:612] 0 active servers
[D 2019-05-07 16:06:10.738 JupyterHub dockerspawner:557] Getting container
'jupyter-joan'
[I 2019-05-07 16:06:10.751 JupyterHub dockerspawner:564] Service
'jupyter-joan' is gone
[I 2019-05-07 16:06:10.792 JupyterHub dockerspawner:698] Created service
jupyter-joan (id: nnguhuf) from image
jupyter/datascience-notebook:59b402ce701d
[I 2019-05-07 16:06:10.793 JupyterHub dockerspawner:721] Starting service
jupyter-joan (id: nnguhuf)
[D 2019-05-07 16:06:10.904 JupyterHub spawner:851] Polling subprocess every
30s
[D 2019-05-07 16:06:20.444 JupyterHub swarmspawner:137] Getting task of
service 'jupyter-joan'
[D 2019-05-07 16:06:20.444 JupyterHub dockerspawner:557] Getting container
'jupyter-joan'
[W 2019-05-07 16:06:20.455 JupyterHub swarmspawner:121] Service
jupyter-joan not found
[W 2019-05-07 16:06:20.456 JupyterHub web:1667] 500 GET /hub/user/joan/
(10.255.0.2): Spawner failed to start [status=0]. The logs for joan may
contain details.
[D 2019-05-07 16:06:20.456 JupyterHub base:880] No template for 500
[E 2019-05-07 16:06:20.467 JupyterHub log:150] {
"X-Forwarded-Host": "mussol.cpl.upc.edu:8000",
"X-Forwarded-Proto": "http",
"X-Forwarded-Port": "8000",
"X-Forwarded-For": "10.255.0.2",
"Upgrade-Insecure-Requests": "1",
"Connection": "close",
"Cookie":
"jupyterhub-hub-login=\"2|1:0|10:1557245170|20:jupyterhub-hub-login|44:NDg0ZTE0ZDgzMjliNDU2Yjk2ODRkYWI4NzlmYmI2OGQ=|6dadad2e54ab049a2f3fcde00eb7e12561ff07f8405f179674ac25f52e8bf568\";
_ga=GA1.2.917524424.1530000626;
jupyterhub-session-id=7fe2aa90931b4454bb52c9cf25ae9de8",
"Referer": "http://mussol.cpl.upc.edu:8000/hub/login";,
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "en-US,en;q=0.5",
"Accept":
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:60.0)
Gecko/20100101 Firefox/60.0",
"Host": "mussol.cpl.upc.edu:8000"
}
[E 2019-05-07 16:06:20.467 JupyterHub log:158] 500 GET /hub/user/joan/
([email protected]) 10190.43ms
--
You received this message because you are subscribed to the Google Groups
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jupyter/225655c4-d37c-4876-b0c4-1b1ec3b12204%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
