On 6 June 2024 18:06:53 BST, Matthias Koeppe <matthiaskoe...@gmail.com> wrote:
>On Thursday, June 6, 2024 at 3:56:15 AM UTC-7 Dima Pasechnik wrote:
>
>What's described here is a very well known fact: "pip install" (or "sage
>--pip install") might mess up your venv, anyone who works with large
>collections of python packages is well aware of it.
>
>
>That's right.
>People are aware of it when they use pip _manually_.
>And when something goes wrong, people tend to delete their venvs and start
>over.
>
>Talking about it as something "underdeveloped in Sage" is a bit strange,
>these are just well-known features and deficiencies of pip.
>
>
>You are missing the point: We cannot build something that is expected to be
>robust on top of this automated use of "pip", to install and update
>packages one by one, in the Sage distribution.
Why? E.g. scipy/numpy and sympy do not hoard compilers, do not ship a vendored
Jupyter, or pytest, or Sphinx, or matplotlib, or OpenBLAS - yet are popular
and successful - and thus robust, I presume.
It seems that you are missing the point of how to build a successful project,
by demanding that we don't follow the normal Python practices, as these are
"not robust".
>
>Yet, pytest, a pip package, is installed and used rather regularly in Sage,
>and nobody gets hurt. It is found to be safe to install this particular
>package this way ("pip install pytest" does not affect anything in the Sage
>venv, that's why).
>
>
>That so far there has not been breakage (other than that our use of pytest
>itself was entirely broken) just means that we have been lucky.
No, this means that pytest folks do a good job, and we use results of their
hard work wisely, without vendoring and version micromanagement.
>There is simply no mechanism that is keeping it safe, even retroactively (=
>for the same published version of Sage).
>Even pinning the pytest version would not be enough because any of its
>_dependencies_ (unpinned, and undeclared in the Sage distribution), when a
>future version of that is published to PyPI, can add an incompatible
>dependency itself.
>By converting packages to "pip" packages, these interdependencies --
>exactly through the random packages that nobody wants to know or care about
>-- are hidden complexity, which only becomes visible when something breaks.
"something breaks" for you means "stops installing".
But it can well be broken underneath, even though it installs. E.g. we are
shipping rather broken symbolic routines - sure, they install, but they often
produce incorrect results, and leak memory. Also, our graphics routines have a
huge room for improvement. Etc.
>
>There is no reason not to promote it to standard and keep it a pip package.
>
>(Same applies to any other well-isolated from the rest of Sage bunch of
>packages, e.g. Sphinx+Jupyter, i.e. the vast majority of Python packages
>
>
>No, that's just wishful thinking. There's absolutely no mechanism that
>isolates them.
>Common dependencies can become incompatible, and then by the action of the
>resolver, things will break.
No, it's mostly because you want to micromanage each and every PyPI package
involved in Sage. But we don't need this, if instead we use a few well tested
largish components - because their providers actually test things, not only in
isolation.
Surely the latest Jupyter and Sphinx and pytest and matplotlib all work
together rather well, almost all the time. And one can take a snapshot of
versions and keep it if something (unlikely) breaks in an update.
>
>The extra tax we pay by not using pip
>
>packages more has started to outweight the benefits long ago. People
>
>don't want to spend extra time and other resources
>to maintain and build often somewhat outdated versions of packages which
>are available from PyPI in a ready way.
>
>
>Not sure which "people" you are attempting to speak for here, and on what
>basis.
On the basis of previous discussions on this topic.
I don't understand why you assume that the majority of the developers wants to
follow you and build a mini-clone of something like Conda.
>But most of the packages that you are talking about are "wheel" packages.
>They are not "built", they are only installed.
>And that packages may be "outdated" is outweighed by actually having
>already been tested in this combination, so that users can actually be
>users and not perpetual beta-testers.
>
>Pinning versions works, as everyone knows.
>
>The precaution of not using "pip install" in the Sage venv is akin to the
>same precaution done by most, if not all, Linux distros, which just forbid
>installing packages with pip into the main python environment. (one needs a
>venv to use pip install).
>
>
>Good observation. Yes, this is part of best practices for exactly the same
>reasons.
>(It has recently been standardized via the EXTERNALLY-MANAGED mechanism;
>see https://github.com/sagemath/sage/pull/37250, needs review)
>
>
>Sage is a white crow among other python packages, in its way how it demands
>hundreds of particular versions
>of extra python packages to come along.
>
>
>No, Sage does not "demand" particular versions.
"demand" or "provide" are essentially synonyms here.
>The Sage distribution "provides" a set of particular versions that have
>been tested to work well together.
We heard this already, and "work well" is not true. Because only sagelib is
tested. The rest, most of the shipped packages, are not being tested. The fact
that all these 300 packages install does not mean that they work well. We
don't provide tests for Jupyter, for instance.
In fact, we trust upstreams, e.g., Jupyter project, to provide working
notebooks, etc. pytest to provide non+broken testing tool, etc.
In fact, they bring out new versions to fix bugs. By not closely following what
they provide, i.e. by vendoring their packages, we ship more broken versions.
So it's the opposite of what you advertise as "working well". You would like to
trust them less, and I would like, following examples of successful Python
projects, to trust them more.
>
>That's how software distributions work. That's the whole point.
>
And thus you are dragging us into becoming a full-fledged distribution, no?
I am sure most Sage developers do not want to work primarily on yet another
software distribution project. The project you outline does not work like a
Python project, it's a distribution project, largely single-handedly designed.
A project with a very low bus factor, among other problems.
Successful Python projects work by trusting other packages to do the good job,
and using the fruits of these jobs, not by hoarding often outdated versions and
vendoring them, in hope that mere installability implies that things work as
advertised.
Dima
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sage-devel/133F861C-E5AD-4548-A6FE-70EBC5A68C46%40gmail.com.
