Thank you so much David for such a detailed and clear response! I've just checked and it doesn't seem to be enough to simply log out to have the storage cleaned up. Apparently I had to manually stop and restart the server from the Hub's Control Panel to do so.
I've updated the documentation and cited your email for future reference. I hope that's OK :) https://wikitech.wikimedia.org/w/index.php?title=PAWS/Python_with_Pip&diff=prev&oldid=2363465 Thanks, Diego On Wed, 19 Nov 2025 at 18:56, Bryan Davis <[email protected]> wrote: > On Wed, Nov 19, 2025 at 9:56 AM Diego de la Hera <[email protected]> > wrote: > > I see in the documentation that it is possible to install additional > Python libraries in PAWS using pip like so: > > > > import sys > > !{sys.executable} -m pip install [PACKAGE] > > > > I see libraries are installed to /srv/paws/lib/python3.12/site-packages. > I assume this is not shared across PAWS users. > > Correct. It looks like a shared location, but what may not be obvious > is that the changes are happening in an instance of a Docker container > that is specific to your user. Each PAWS session is a separate Pod > running a "singleuser" container[0] in a Kubernetes cluster. The pip > install command is an ephemeral change in the runtime environment of > that container. It will not be seen across users or sessions. > > > By the way, does this count toward the 5 GB storage available? > > I don't think it will because the storage used is ephemeral > (temporary) storage in the active Docker container that is running the > notebook. > > > Is there a recommended way to create and use Python virtual environments > in PAWS notebooks? I think that would make it easier to identify which > packages and dependencies have been installed for which notebooks, and to > remove them when no longer needed. > > Your local management system seems reasonable, but it does not feel > needed in PAWS, at least for cleanup, because that is handled by > terminating the Kubernetes Pod that is running your session. When you > want a clean slate just log out and then log back in. You will also > get a new session when you connect after your prior session has been > deleted for being idle. > > One way to think of this is that it is very similar to creating a new > venv each time you run a notebook locally and then deleting it > afterwards. PAWS is not exactly the same because within one session > you can run multiple notebooks, but it is at least pretty similar. > > > [0]: https://github.com/toolforge/paws/tree/main/images/singleuser > > Bryan > -- > Bryan Davis Wikimedia Foundation > Principal Software Engineer Boise, ID USA > [[m:User:BDavis (WMF)]] irc: bd808 > _______________________________________________ > Cloud mailing list -- [email protected] > List information: > https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ >
_______________________________________________ Cloud mailing list -- [email protected] List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
