Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 08.05.2014 23:22, Donald Stufft wrote: > >> On a personal note, I'm uncomfortable with the way this change is >> perceived as a case of *pip* enforcing a behaviour that the pip >> developers feel should be required. I actually don't like this change >> particularly. So having pip implement the behaviour required by that >> PEP is to me simply a case of compliance with the agreed standard. But >> now, as a pip developer, being held responsible for the resulting user >> pain, and being expected to defend it, does not make me happy. > > I think the pain is being overrepresented and the positives are being ignored. > The problem is the benefits of this PEP are much like the benefits of TLS > too. For the vast majority of people they don’t notice anything different > except > installing things is faster and more reliable. They don’t attribute that to > the > PEP or this decision, they just internalize it as the new norm. However the > people who this does affect will seek out why it broke and raise an issue > citing that thing specifically. This creates a perception of lots of pain for > no > gain when the reality is not that. Donald: I don't think anyone is arguing that hosting packages on PyPI is a bad thing and PyPI as a service has gotten a lot better than it was a few years ago. However, I find it troubling that we as Python developers are *forcing* the whole Python world to put their code into PyPI. There are plenty good reasons not to do this, and sometimes it's even impossible if you want to stay legal (see the PEP for details). Accordingly, we should respect those reasons make it possible for Python packages to live elsewhere, without having our tools put those packages into a bad light or making it harder for Python users to install such packages than needed. With the checksum uploaded to PyPI, the only argument against fetching packages from other places on the Internet is reliability, not security. PyPI is not the only reliable file hosting system on the Internet, so this argument is rather weak. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 09 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 9 May 2014 05:34, Donald Stufft wrote: > On May 8, 2014, at 5:22 PM, Donald Stufft wrote: > >>> Socially, this change does not seem to be having the effect of >>> persuading more package developers to host on PyPI. The stick doesn't >>> appear to have worked, maybe we should be trying to find a carrot? >> >> Do you have any data to point to that says it hasn’t worked? Just to see >> what impact it has had, I’m running my scripts again that I ran a year >> ago to see what has changed, already I can see they are processing >> MUCH faster than last year. > > The data has finished processing, it represents a time diff of approximately > one year. The pip release that caused all of this was released about 4-5 > months > ago. > > Overall PyPI has seen a 50% growth in installable projects in that time. If > the > change would have had no effect we'd expect to see a ~50% increase across the > board. However what we've seen is a a 60% (+10% of expected) increase in > projects that can only be installed from PyPI and a 12% decrease in projects > that have any unsafe files (-62% of expected). Donald, Thanks for taking the time to get those figures. It does appear that there are less cases that would be affected than the number of complaints would imply. The only concern I have about this type of analysis is that it doesn't "weight" projects. It may be (and again, I have no data to back this up) that the projects that are affected detrimentally by this change are unusually popular or otherwise significant. There's obviously no way to assess this sensibly other than by making a judgement on the level of complaints. But arguing numbers was never my intention here, so let's just say that I concede that the change has had a positive effect, which is great. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Donald Stufft wrote: > I?m unsure if you?re being willfully dense or if you?re just not understanding > what I mean when I say ?almost?. Of course there are going to be a few > outliers > where people do bother to do that, but it?s not going to be common place at > all. I suggest that you confine that style of discussion to distutils-sig. Stefan Krah ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 4:12 AM, M.-A. Lemburg wrote: > On 08.05.2014 23:22, Donald Stufft wrote: >> >>> On a personal note, I'm uncomfortable with the way this change is >>> perceived as a case of *pip* enforcing a behaviour that the pip >>> developers feel should be required. I actually don't like this change >>> particularly. So having pip implement the behaviour required by that >>> PEP is to me simply a case of compliance with the agreed standard. But >>> now, as a pip developer, being held responsible for the resulting user >>> pain, and being expected to defend it, does not make me happy. >> >> I think the pain is being overrepresented and the positives are being >> ignored. >> The problem is the benefits of this PEP are much like the benefits of TLS >> too. For the vast majority of people they don’t notice anything different >> except >> installing things is faster and more reliable. They don’t attribute that to >> the >> PEP or this decision, they just internalize it as the new norm. However the >> people who this does affect will seek out why it broke and raise an issue >> citing that thing specifically. This creates a perception of lots of pain >> for no >> gain when the reality is not that. > > Donald: I don't think anyone is arguing that hosting packages on > PyPI is a bad thing and PyPI as a service has gotten a lot better > than it was a few years ago. Didn’t mean to imply that I thought it was otherwise. > > However, I find it troubling that we as Python developers are *forcing* > the whole Python world to put their code into PyPI. Forcing is a strong word. As of right now we’re "forcing" you to put it onto PyPI if you want to install it without *any* flags to pip. You're more then capable of hosting it externally if you wish, and pip will even tell the people who try to install it what they need to enable in order to install it. It even allows people to say "I don't care about any of this crap, just make all the external stuff work". Even if pip removed the external link handling all together and required you to do something like: $ pip install --extra-index-url https://example.com/our-packages/ foo $ pip install --find-links https://example.com/our-packages/ foo We still wouldn't be forcing anyone to upload things to PyPI. We are, however, discouraging people from not hosting on PyPI and providing incentives to doing that. > > There are plenty good reasons not to do this, and sometimes it's > even impossible if you want to stay legal (see the PEP for details). I re-read the reasons, I'm not sure I really agree with most (or all?) of them however if people really want to do it, then there is nothing stopping them. It's their choice and people on the *other* side of that who are installing these packages also get to make a choice if they want to allow it or not. > > Accordingly, we should respect those reasons make it possible for > Python packages to live elsewhere, without having our tools put > those packages into a bad light or making it harder for Python > users to install such packages than needed. I'm not sure what "put it into a bad light" is supposed to mean here. Is it about the warning? I've already removed that completely. As far as making it harder than it "needs" to be, that's a hard line to draw. I personally know people for whom things that are hosted externally have caused them a lot of pain to the point where they have to automatically spider PyPI for their own dependencies every hour or so in order to isolate themselves from the failure of random dependencies suddenly no longer being installable. This breakage is going to happen for basically any project that installs their dependencies with any frequency. One of the big projects that I'm aware of that has had this problem is Openstack who, in their CI, installs things several hundred times a day. If a service they depend on goes down that causes significant disruption for them. Now they've solved it by the above solution of hosting their own mirror of PyPI that includes spidering for externally installable things. They have one or two packages left that don't host directly on PyPI at which point they'll no longer need that afaik. I don't think telling every downstream of us of any size that in order to get reliable installs they are going to have to work around PyPI, and in some cases even develop custom software in order to do so effectively, is a very user friendly position. > > With the checksum uploaded to PyPI, the only argument against > fetching packages from other places on the Internet is reliability, > not security. I've never said differently. There are some minor privacy things in there but primarily it's about reliability and that people should generally be aware where their packages are coming from. > > PyPI is not the only reliable file hosting system on the Internet, > so this argument is rather weak. It actually doesn't matter how reliable the other systems are. Reliabi
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 9 May 2014 12:44, Donald Stufft wrote: > We still wouldn't be forcing anyone to upload things to PyPI. We are, however, > discouraging people from not hosting on PyPI and providing incentives to doing > that. But you're doing so by inflicting pain on people using pip to install those packages. Those users complain about *pip*, not about the packages. Better to directly impact the package maintainers, rather than their users (who are innocent victims). Better still of course to encourage people to improve things, not to punish them for not doing so. > I think it's important to point out that one of the driving factors that > caused > me to finally push for changes and what lead to PEP438 being created was that > Mercurial's external hosted was being extremely flaky. I can't remember the > exact details but I want to say that over the span of a week or two I was > getting massive numbers of users complaining that ``pip install Mercurial`` > was suddenly failing. This isn't to knock on the Mercurial folks or anything > but to simply point out that these problems aren't things that just happen to > (under|un)maintained software nor are they hypothetical. This PEP was born of > the frustration that was being relayed to me by end users of PyPI/pip. So now "pip install Mercurial" always fails? And adding a flag allows it to work as well as before, but no better? How did that fix the issue? Seriously - I'm missing something here. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 5:01 AM, Paul Moore wrote: > On 9 May 2014 05:34, Donald Stufft wrote: >> On May 8, 2014, at 5:22 PM, Donald Stufft wrote: >> Socially, this change does not seem to be having the effect of persuading more package developers to host on PyPI. The stick doesn't appear to have worked, maybe we should be trying to find a carrot? >>> >>> Do you have any data to point to that says it hasn’t worked? Just to see >>> what impact it has had, I’m running my scripts again that I ran a year >>> ago to see what has changed, already I can see they are processing >>> MUCH faster than last year. >> >> The data has finished processing, it represents a time diff of approximately >> one year. The pip release that caused all of this was released about 4-5 >> months >> ago. >> >> Overall PyPI has seen a 50% growth in installable projects in that time. If >> the >> change would have had no effect we'd expect to see a ~50% increase across the >> board. However what we've seen is a a 60% (+10% of expected) increase in >> projects that can only be installed from PyPI and a 12% decrease in projects >> that have any unsafe files (-62% of expected). > > Donald, > Thanks for taking the time to get those figures. It does appear that > there are less cases that would be affected than the number of > complaints would imply. Of course, I don’t like making claims without backing them up if I can :) > > The only concern I have about this type of analysis is that it doesn't > "weight" projects. It may be (and again, I have no data to back this > up) that the projects that are affected detrimentally by this change > are unusually popular or otherwise significant. There's obviously no > way to assess this sensibly other than by making a judgement on the > level of complaints. Yea, I don’t have a good way to weight those projects in any way. Normally I could get some sort of estimate by looking at the download numbers from PyPI but well ;) For the record, here’s the list of projects that are hosted *only* safely externally or that have *any* safely externally hosted files: https://gist.github.com/dstufft/1b16c305f97fff6cef2f Most of these don’t stand out to me at all. The only ones that do are: * pyOpenSSL which has one older release that is hosted that way * argparse which has the latest release hosted this way but has older releases hosted on PyPI * new relic which only hosts older releases externally * beautifulsoup4 which hosts things safely externally *and* on PyPI * Paste which has one “external” thing which is actually only external because it used a cheeseshop.python.org link instead of a pypi.python.org link. * ipython which has one older release hosted safely externally but the latest is on PyPI * netifaces which has one older release hosted safely externally but the rest are on PyPI > > But arguing numbers was never my intention here, so let's just say > that I concede that the change has had a positive effect, which is > great. > Paul I didn’t mean to try to imply that it was :) I just wanted to make sure that *my* claims were true, or if they weren’t I wanted to be able to say that I was wrong. Since I had the numbers computed already it didn’t make any sense not to share them here. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 9 May 2014 21:55, Paul Moore wrote: > On 9 May 2014 12:44, Donald Stufft wrote: >> I think it's important to point out that one of the driving factors that >> caused >> me to finally push for changes and what lead to PEP438 being created was that >> Mercurial's external hosted was being extremely flaky. I can't remember the >> exact details but I want to say that over the span of a week or two I was >> getting massive numbers of users complaining that ``pip install Mercurial`` >> was suddenly failing. This isn't to knock on the Mercurial folks or anything >> but to simply point out that these problems aren't things that just happen to >> (under|un)maintained software nor are they hypothetical. This PEP was born of >> the frustration that was being relayed to me by end users of PyPI/pip. > > So now "pip install Mercurial" always fails? And adding a flag allows > it to work as well as before, but no better? How did that fix the > issue? Seriously - I'm missing something here. Mercurial downloads are now available directly from PyPI. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 7:55 AM, Paul Moore wrote: > On 9 May 2014 12:44, Donald Stufft wrote: >> We still wouldn't be forcing anyone to upload things to PyPI. We are, >> however, >> discouraging people from not hosting on PyPI and providing incentives to >> doing >> that. > > But you're doing so by inflicting pain on people using pip to install > those packages. Those users complain about *pip*, not about the > packages. Better to directly impact the package maintainers, rather > than their users (who are innocent victims). Better still of course to > encourage people to improve things, not to punish them for not doing > so. We can’t directly impact the package maintainers and the vast bulk of people who have had a problem who have complained about it to pip also need to add the —allow-unverifiable flag and would not simply be able to be fixed by allowing safely externally hosted files. Looking at the numbers and what packages are hosted externally, allowing safely externally hosted files would have practically no benefit to pip’s end users. The only case that I can see with a quick scan would be it would allow the latest version of argparse. TBH I think the biggest source of confusion reduction would be to remove the “safely externally hosted’ category all together and just make it hosted on PyPI -> Install by default, hosted off PyPI -> requires a per package flag. However I’m sure the vocal minority would have a problem with that. > >> I think it's important to point out that one of the driving factors that >> caused >> me to finally push for changes and what lead to PEP438 being created was that >> Mercurial's external hosted was being extremely flaky. I can't remember the >> exact details but I want to say that over the span of a week or two I was >> getting massive numbers of users complaining that ``pip install Mercurial`` >> was suddenly failing. This isn't to knock on the Mercurial folks or anything >> but to simply point out that these problems aren't things that just happen to >> (under|un)maintained software nor are they hypothetical. This PEP was born of >> the frustration that was being relayed to me by end users of PyPI/pip. > > So now "pip install Mercurial" always fails? And adding a flag allows > it to work as well as before, but no better? How did that fix the > issue? Seriously - I'm missing something here. No, This caused Mercurial to upload their packages to PyPI. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 9 May 2014 13:06, Donald Stufft wrote: >>> I think it's important to point out that one of the driving factors that >>> caused >>> me to finally push for changes and what lead to PEP438 being created was >>> that >>> Mercurial's external hosted was being extremely flaky. I can't remember the >>> exact details but I want to say that over the span of a week or two I was >>> getting massive numbers of users complaining that ``pip install Mercurial`` >>> was suddenly failing. This isn't to knock on the Mercurial folks or anything >>> but to simply point out that these problems aren't things that just happen >>> to >>> (under|un)maintained software nor are they hypothetical. This PEP was born >>> of >>> the frustration that was being relayed to me by end users of PyPI/pip. >> >> So now "pip install Mercurial" always fails? And adding a flag allows >> it to work as well as before, but no better? How did that fix the >> issue? Seriously - I'm missing something here. > > No, This caused Mercurial to upload their packages to PyPI. You're claiming that Mercurial moved to hosting on PyPI solely because users suddenly needed to add a flag to install from pip? As opposed to because PyPI gave them a more reliable hosting platform, for example? OK. I certainly can't give any evidence to dispute that claim, although I'm surprised. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 8:21 AM, Paul Moore wrote: > On 9 May 2014 13:06, Donald Stufft wrote: I think it's important to point out that one of the driving factors that caused me to finally push for changes and what lead to PEP438 being created was that Mercurial's external hosted was being extremely flaky. I can't remember the exact details but I want to say that over the span of a week or two I was getting massive numbers of users complaining that ``pip install Mercurial`` was suddenly failing. This isn't to knock on the Mercurial folks or anything but to simply point out that these problems aren't things that just happen to (under|un)maintained software nor are they hypothetical. This PEP was born of the frustration that was being relayed to me by end users of PyPI/pip. >>> >>> So now "pip install Mercurial" always fails? And adding a flag allows >>> it to work as well as before, but no better? How did that fix the >>> issue? Seriously - I'm missing something here. >> >> No, This caused Mercurial to upload their packages to PyPI. > > You're claiming that Mercurial moved to hosting on PyPI solely because > users suddenly needed to add a flag to install from pip? As opposed to > because PyPI gave them a more reliable hosting platform, for example? > OK. I certainly can't give any evidence to dispute that claim, > although I'm surprised. > > Paul I don’t know that for a fact but If my memory is correct that’s a reasonable assumption based on the timeline. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Paul Moore wrote: > You're claiming that Mercurial moved to hosting on PyPI solely because > users suddenly needed to add a flag to install from pip? As opposed to > because PyPI gave them a more reliable hosting platform, for example? > OK. I certainly can't give any evidence to dispute that claim, > although I'm surprised. That is my understanding of the posted statistics, too. PyPI was overloaded, the improved infrastructure fixed that, so more projects now host on PyPI. Stefan Krah ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 9 May 2014 13:25, Donald Stufft wrote: >> You're claiming that Mercurial moved to hosting on PyPI solely because >> users suddenly needed to add a flag to install from pip? As opposed to >> because PyPI gave them a more reliable hosting platform, for example? >> OK. I certainly can't give any evidence to dispute that claim, >> although I'm surprised. > > I don’t know that for a fact but If my memory is correct that’s a reasonable > assumption based on the timeline. As Stefan pointed out, improved PyPI infrastructure such as the CDN seems like a much more likely reason for Mercurial to have switched. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On Fri, 9 May 2014 13:47:49 +0100 Paul Moore wrote: > On 9 May 2014 13:25, Donald Stufft wrote: > >> You're claiming that Mercurial moved to hosting on PyPI solely because > >> users suddenly needed to add a flag to install from pip? As opposed to > >> because PyPI gave them a more reliable hosting platform, for example? > >> OK. I certainly can't give any evidence to dispute that claim, > >> although I'm surprised. > > > > I don’t know that for a fact but If my memory is correct that’s a reasonable > > assumption based on the timeline. > > As Stefan pointed out, improved PyPI infrastructure such as the CDN > seems like a much more likely reason for Mercurial to have switched. Here is the only reference I could find: http://www.selenic.com/pipermail/mercurial-devel/2013-July/052129.html Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 05/08/2014 02:02 PM, Paul Moore wrote: Socially, this change does not seem to be having the effect of persuading more package developers to host on PyPI. The stick doesn't appear to have worked, maybe we should be trying to find a carrot? Or maybe we have to accept that some developers have sound reasons for not hosting on PyPI and work with them to find an acceptable compromise? Has anyone checked what Stefan's reasons are for not hosting cdecimal on PyPI? Do they represent a use case that the PEP hasn't considered? Well, I do host a small handful of modules on PyPI, but I can say that some of my pain points are: - getting a good name: the obvious ones are taken, so the search begins to find a name that is not taken and yet still feels at least somewhat appropriate: my OO path module ended up being called strpath (dumb name); eventually changed to antipathy my script param line parser is called scription (okay name) my enum backport is called enum34 (blah) my dbf package is called dbf (lucky lucky lucky!) - reputation of PyPI is not that great: anybody can upload packages, no curating is done, once there the name is taken for all time, no standard version numbering, . . . (granted, these are my impressions and may not be entirely valid) Anyway, I know it's a volunteer effort I don't want to criticize those actually running it, but these are some of the problems that I see with the service itself. -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 09.05.2014 13:44, Donald Stufft wrote: > > On May 9, 2014, at 4:12 AM, M.-A. Lemburg wrote: >> Donald: I don't think anyone is arguing that hosting packages on >> PyPI is a bad thing and PyPI as a service has gotten a lot better >> than it was a few years ago. > > Didn’t mean to imply that I thought it was otherwise. > >> >> However, I find it troubling that we as Python developers are *forcing* >> the whole Python world to put their code into PyPI. > > Forcing is a strong word. As of right now we’re "forcing" you to put it onto > PyPI if you want to install it without *any* flags to pip. Which is what most people expect to be able to do. > You're more then > capable of hosting it externally if you wish, and pip will even tell the > people > who try to install it what they need to enable in order to install it. It even > allows people to say "I don't care about any of this crap, just make all the > external stuff work". > > Even if pip removed the external link handling all together and required you > to do something like: > > $ pip install --extra-index-url https://example.com/our-packages/ foo > $ pip install --find-links https://example.com/our-packages/ foo > > We still wouldn't be forcing anyone to upload things to PyPI. We are, however, > discouraging people from not hosting on PyPI and providing incentives to doing > that. This is all true, but in reality, you are making externally hosted packages second class citizens in Python land by requiring extra options even for package listings that are perfectly safe to download from other servers. As mentioned before: I can understand that you want to make downloads safe for users, but if a package is hosted on some other reliable servers and pip can check that it's a valid package, there's little to argue for not allowing such downloads. >> There are plenty good reasons not to do this, and sometimes it's >> even impossible if you want to stay legal (see the PEP for details). > > I re-read the reasons, I'm not sure I really agree with most (or all?) of them > however if people really want to do it, then there is nothing stopping them. > It's their choice and people on the *other* side of that who are installing > these packages also get to make a choice if they want to allow it or not. You don't have to agree with those reasons. Fact is, they exist, prevent package authors from uploading to PyPI and we as Python developers should respect those reasons. With the new infrastructure, it's far more attractive to host packages on PyPI than it was before, so people who do not host on PyPI will have carefully thought about this. >> Accordingly, we should respect those reasons make it possible for >> Python packages to live elsewhere, without having our tools put >> those packages into a bad light or making it harder for Python >> users to install such packages than needed. > > I'm not sure what "put it into a bad light" is supposed to mean here. Is it > about the warning? I've already removed that completely. It's using the same unfortunate strategy that setuptools used by requiring an option called --single-version-externally-managed to be able to install a package without all the .pth and egg logic (an option that pip now uses per default, BTW ;-)). I snipped the rest of the discussion and reliability, using unmaintained packages and projects using their own mirrors (which should really be the standard, not an exceptional case), because it's not really leading anywhere: At the time we discussed the PEP, security was the main concern, not reliability. Now that there is a secure way to reference external distribution files, I think we should revisit the defaults decision in the light of the new possibilities. BTW: The PEP mentions re-hosting tools to upload their packages to PyPI and also says "This re-hosting tool MUST be available before automated hosting-mode changes are announced to package maintainers." I am not aware of such tools. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 09 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On Fri, May 09, 2014 at 06:09:28AM -0700, Ethan Furman wrote: > On 05/08/2014 02:02 PM, Paul Moore wrote: > Well, I do host a small handful of modules on PyPI, but I can say that some > of my pain points are: > > - getting a good name: the obvious ones are taken, so the search > begins to find a name that is not taken and yet still feels at > least somewhat appropriate: > > my OO path module ended up being called strpath (dumb name); > eventually changed to antipathy > > my script param line parser is called scription (okay name) > > my enum backport is called enum34 (blah) > > my dbf package is called dbf (lucky lucky lucky!) For many reasons I avoid github/gitorious/bitbucket, but there is one thing they do right -- two-levels namespaces. "Namespaces are one honking great idea -- let's do more of those!" Oleg. -- Oleg Broytmanhttp://phdru.name/[email protected] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 9:58 AM, M.-A. Lemburg wrote: > On 09.05.2014 13:44, Donald Stufft wrote: >> >> On May 9, 2014, at 4:12 AM, M.-A. Lemburg wrote: >>> Donald: I don't think anyone is arguing that hosting packages on >>> PyPI is a bad thing and PyPI as a service has gotten a lot better >>> than it was a few years ago. >> >> Didn’t mean to imply that I thought it was otherwise. >> >>> >>> However, I find it troubling that we as Python developers are *forcing* >>> the whole Python world to put their code into PyPI. >> >> Forcing is a strong word. As of right now we’re "forcing" you to put it onto >> PyPI if you want to install it without *any* flags to pip. > > Which is what most people expect to be able to do. Sure, but sometimes it’s better to make an informed choice about things being installed instead of having it be installed by default and sometimes it’s better to disallow doing something at all. Further most people don’t expect an install to touch any server host other than PyPI. As far as I’m aware none of the other package repositories feature this, and even with the fact we support it barely anyone even cares to utilize it. > >> You're more then >> capable of hosting it externally if you wish, and pip will even tell the >> people >> who try to install it what they need to enable in order to install it. It >> even >> allows people to say "I don't care about any of this crap, just make all the >> external stuff work". >> >> Even if pip removed the external link handling all together and required you >> to do something like: >> >>$ pip install --extra-index-url https://example.com/our-packages/ foo >>$ pip install --find-links https://example.com/our-packages/ foo >> >> We still wouldn't be forcing anyone to upload things to PyPI. We are, >> however, >> discouraging people from not hosting on PyPI and providing incentives to >> doing >> that. > > This is all true, but in reality, you are making externally hosted > packages second class citizens in Python land by requiring > extra options even for package listings that are perfectly safe > to download from other servers. > > As mentioned before: I can understand that you want to make downloads > safe for users, but if a package is hosted on some other reliable > servers and pip can check that it's a valid package, there's little > to argue for not allowing such downloads. Except the fact that the only people I’ve ever seen *happy* that people can host packages externally are a handful (<5) of people (tbh primarily you and Stefan) and the feedback I get from every other person around the web in unequivocally yes please get rid of externally hosted files, they’ve done nothing but cause me pain. It’s not reasonable to allow a minority of users to negatively impact the majority. > >>> There are plenty good reasons not to do this, and sometimes it's >>> even impossible if you want to stay legal (see the PEP for details). >> >> I re-read the reasons, I'm not sure I really agree with most (or all?) of >> them >> however if people really want to do it, then there is nothing stopping them. >> It's their choice and people on the *other* side of that who are installing >> these packages also get to make a choice if they want to allow it or not. > > You don't have to agree with those reasons. Fact is, they exist, > prevent package authors from uploading to PyPI and we as > Python developers should respect those reasons. No I disagree that they are reasons at all. Half of the ones you enumerated are nonsensical or irrelevant, the other half can be fixed by adding features to or fixing PyPI. One or two read like reasons why someone might actually decide not to upload something to PyPI but which that reason isn’t really all that reasonable and finally a grand total of one or two of them look like an actual legit reasons and it only applies to Crypto software. I mean your reasoning included things like: > PyPI doesn’t let you upload two files with the same name, you gave the > example of UC2 vs UC4 The thing being if PyPI did allow you to do that then we’d have no way to determine which one is the right one and half of the people would just get randomly failing installs because we guessed the wrong one. > They don’t know it’s possible to upload to PyPI I don’t even know how to response to this one. > PyPI used to be unreliable but isn’t now and they don’t know about it Nor this one. > Not wanting people to download a package at all and instead check out > from a VCS repo This’ll randomly fail for people depending on if they happen to have that VCS installed. Also I don’t see any evidence that people are actually doing this outside of supporting a foo==dev install which pip won’t install by default either. > Not wanting to add latency for yourself for downloading from PyPI Instead you’re going to inflict extra latency on everyone else when you could just run a mirror or upload the packages to your own serve
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2014-05-02 - 2014-05-09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open4611 ( +6) closed 28624 (+42) total 33235 (+48) Open issues with patches: 2119 Issues opened (35) == #18604: Consolidate gui available checks in test.support http://bugs.python.org/issue18604 reopened by ned.deily #21416: argparse should accept bytes arguments as originally passed http://bugs.python.org/issue21416 opened by underrun #21417: Compression level for zipfile http://bugs.python.org/issue21417 opened by Sworddragon #21418: Segv during call to super_init in application embedding Python http://bugs.python.org/issue21418 opened by snoeberger #21419: Use calloc() instead of malloc() for int << int (lshift) http://bugs.python.org/issue21419 opened by haypo #21420: Optimize 2 ** n: implement it as 1 << n http://bugs.python.org/issue21420 opened by haypo #21422: int << 0: return the number unmodified http://bugs.python.org/issue21422 opened by haypo #21423: concurrent.futures.ThreadPoolExecutor should accept an initial http://bugs.python.org/issue21423 opened by andreasvc #21424: Simplify and speed-up heaqp.nlargest() http://bugs.python.org/issue21424 opened by rhettinger #21425: Python 3 pipe handling breaks python mode in emacs on Windows http://bugs.python.org/issue21425 opened by marczellm #21427: installer not working http://bugs.python.org/issue21427 opened by ellipso #21428: Python suddenly cares about EOLs formats on windows http://bugs.python.org/issue21428 opened by pygnol #21429: Input.output error with multiprocessing http://bugs.python.org/issue21429 opened by mikaela #21430: Document ssl.pending() http://bugs.python.org/issue21430 opened by shevek #21431: 3.4.1rc1 test_pydoc fails: pydoc_data.topics.topics values are http://bugs.python.org/issue21431 opened by ned.deily #21434: python -3 documentation is outdated http://bugs.python.org/issue21434 opened by berker.peksag #21436: Consider leaving importlib.abc.Loader.load_module() http://bugs.python.org/issue21436 opened by srittau #21437: document that asyncio.ProactorEventLoop doesn't support SSL http://bugs.python.org/issue21437 opened by akira #21439: Numerous minor issues in Language Reference http://bugs.python.org/issue21439 opened by zach.ware #21441: Buffer Protocol Documentation Error http://bugs.python.org/issue21441 opened by Jake.Vanderplas #21443: asyncio logging documentation clarifications http://bugs.python.org/issue21443 opened by hardkrash #21445: Some asserts in test_filecmp have the wrong messages http://bugs.python.org/issue21445 opened by Steven.Barker #21446: Update reload fixer to use importlib instead of imp http://bugs.python.org/issue21446 opened by berker.peksag #21447: Intermittent asyncio.open_connection / futures.InvalidStateErr http://bugs.python.org/issue21447 opened by ryder.lewis #21448: Email Parser use 100% CPU http://bugs.python.org/issue21448 opened by jader.fabiano #21449: Replace _PyUnicode_CompareWithId with _PyUnicode_CompareWithId http://bugs.python.org/issue21449 opened by josh.rosenberg #21452: make_buildinfo.exe with VS2013 fails due ill-formed IntDir pat http://bugs.python.org/issue21452 opened by mloskot #21453: Support of RPM subpackages in distutils http://bugs.python.org/issue21453 opened by vitalyisaev2 #21454: asyncio's loop.connect_read_pipe makes pipes non-blocking cont http://bugs.python.org/issue21454 opened by John Isidore #21455: add default backlog to socket.listen() http://bugs.python.org/issue21455 opened by neologix #21456: skip 2 tests in test_urllib2net.py if _ssl module not present http://bugs.python.org/issue21456 opened by rpointel #21457: NetBSD curses support improvements http://bugs.python.org/issue21457 opened by wiz #21461: Recognize -pthread http://bugs.python.org/issue21461 opened by wiz #21462: PEP 466: upgrade OpenSSL in the Python 2.7 Windows builds http://bugs.python.org/issue21462 opened by ncoghlan #21463: RuntimeError when URLopener.ftpcache is full http://bugs.python.org/issue21463 opened by erik.bray Most recent 15 issues with no replies (15) == #21463: RuntimeError when URLopener.ftpcache is full http://bugs.python.org/issue21463 #21461: Recognize -pthread http://bugs.python.org/issue21461 #21453: Support of RPM subpackages in distutils http://bugs.python.org/issue21453 #21449: Replace _PyUnicode_CompareWithId with _PyUnicode_CompareWithId http://bugs.python.org/issue21449 #21445: Some asserts in test_filecmp have the wrong messages http://bugs.python.org/issue21445 #21443: asyncio logging documentation clarifications http://bugs.python.org/issue21443 #21441: Buffer Protocol Documentation Error http://bugs.python.org/issue21441 #21439: Numerous minor issues in Language Reference http://b
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Donald, I'm out of his discussion. I have one last request: please don't gossip about core devs in public as long as you have commit privs: https://botbot.me/freenode/python-requests/ Stefan Krah ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 09.05.2014 17:39, Donald Stufft wrote: > > On May 9, 2014, at 9:58 AM, M.-A. Lemburg wrote: > >> On 09.05.2014 13:44, Donald Stufft wrote: >>> >>> On May 9, 2014, at 4:12 AM, M.-A. Lemburg wrote: Donald: I don't think anyone is arguing that hosting packages on PyPI is a bad thing and PyPI as a service has gotten a lot better than it was a few years ago. >>> >>> Didn’t mean to imply that I thought it was otherwise. >>> However, I find it troubling that we as Python developers are *forcing* the whole Python world to put their code into PyPI. >>> >>> Forcing is a strong word. As of right now we’re "forcing" you to put it onto >>> PyPI if you want to install it without *any* flags to pip. >> >> Which is what most people expect to be able to do. > > Sure, but sometimes it’s better to make an informed choice about things being > installed instead of having it be installed by default and sometimes it’s > better > to disallow doing something at all. > > Further most people don’t expect an install to touch any server host other > than PyPI. For some idea of "most people" that's probably true. I'd argue that most people probably don't really care all that much where their packages are coming from as long as they are getting the packages registered with PyPI, the Python package index, and can be sure that no one has tampered with them. But both arguments don't really count much, since it's all speculation. > As far as I’m aware none of the other package repositories > feature this, and even with the fact we support it barely anyone even cares to > utilize it. Most Linux repos comes with a list of standard repos to include. In Python land, Plone uses zc.buildout with a similar default list of repos. >>> You're more then >>> capable of hosting it externally if you wish, and pip will even tell the >>> people >>> who try to install it what they need to enable in order to install it. It >>> even >>> allows people to say "I don't care about any of this crap, just make all the >>> external stuff work". >>> >>> Even if pip removed the external link handling all together and required you >>> to do something like: >>> >>>$ pip install --extra-index-url https://example.com/our-packages/ foo >>>$ pip install --find-links https://example.com/our-packages/ foo >>> >>> We still wouldn't be forcing anyone to upload things to PyPI. We are, >>> however, >>> discouraging people from not hosting on PyPI and providing incentives to >>> doing >>> that. >> >> This is all true, but in reality, you are making externally hosted >> packages second class citizens in Python land by requiring >> extra options even for package listings that are perfectly safe >> to download from other servers. >> >> As mentioned before: I can understand that you want to make downloads >> safe for users, but if a package is hosted on some other reliable >> servers and pip can check that it's a valid package, there's little >> to argue for not allowing such downloads. > > Except the fact that the only people I’ve ever seen *happy* that people can > host packages externally are a handful (<5) of people (tbh primarily you and > Stefan) and the feedback I get from every other person around the web > in unequivocally yes please get rid of externally hosted files, they’ve done > nothing but cause me pain. > > It’s not reasonable to allow a minority of users to negatively impact the > majority. I think you're getting this wrong: There *are* package authors who would like to host PyPI indexed packages external to PyPI and they all have good reasons to do so. It may well be a minority, but that's fine. Changing the default to allow secure external downloads is *not* impacting any majority. At worst, it's impacting the users of those packages, but that's really within the realm and responsibility of the package authors, not PyPI or the infrastructure team, so I don't understand why you are strongly objecting this. There are plenty good reasons not to do this, and sometimes it's even impossible if you want to stay legal (see the PEP for details). >>> >>> I re-read the reasons, I'm not sure I really agree with most (or all?) of >>> them >>> however if people really want to do it, then there is nothing stopping them. >>> It's their choice and people on the *other* side of that who are installing >>> these packages also get to make a choice if they want to allow it or not. >> >> You don't have to agree with those reasons. Fact is, they exist, >> prevent package authors from uploading to PyPI and we as >> Python developers should respect those reasons. > > No I disagree that they are reasons at all. Half of the ones you enumerated > are > nonsensical or irrelevant, the other half can be fixed by adding features to > or > fixing PyPI. One or two read like reasons why someone might actually decide > not to upload something to PyPI but which that reason isn’t really all that > reasonable and finally a
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 12:11 PM, Stefan Krah wrote: > Donald, I'm out of his discussion. I have one last request: please don't > gossip about core devs in public as long as you have commit privs: > > https://botbot.me/freenode/python-requests/ I don’t really know how to respond to this. I was talking to some people I know and I gave them a summary of what was happening. I stand by everything I said there and I don’t think any of it was wrong or even gossip at all. If people feel that was inappropriate then go ahead and take my commit privs. I have them to make it easier for me to write PEPs and to update ensurepip. If they’re going to be used as an excuse to attempt to censor me then I’d rather not have them as I generally always speak my mind and I won’t stop doing so. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Stefan, If the only way you can think of to invalidate Donald's (vastly superior) arguments is to accuse of him of "gossip", you should probably reconsider your arguments. Looking at the conversation you didn't actually link to (https://botbot.me/freenode/python-requests/msg/14389415/) there is no gossip. There are no insinuations about your character. All that is there is a succinct description of the topic of this conversation. Cheers, Ian On Fri, May 9, 2014 at 11:23 AM, Donald Stufft wrote: > > On May 9, 2014, at 12:11 PM, Stefan Krah wrote: > >> Donald, I'm out of his discussion. I have one last request: please don't >> gossip about core devs in public as long as you have commit privs: >> >> https://botbot.me/freenode/python-requests/ > > I don’t really know how to respond to this. I was talking to some people I > know > and I gave them a summary of what was happening. I stand by everything I > said there and I don’t think any of it was wrong or even gossip at all. > > If people feel that was inappropriate then go ahead and take my commit privs. > I > have them to make it easier for me to write PEPs and to update ensurepip. If > they’re going to be used as an excuse to attempt to censor me then I’d rather > not have them as I generally always speak my mind and I won’t stop doing so. > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/graffatcolmingov%40gmail.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tix version needed to build 2.7 Windows installer?
On Thu, May 8, 2014 at 4:20 PM, Zachary Ware wrote: > I updated the 2.7 buildbot scripts to pull in Tcl/Tk 8.5.15 a couple > of weeks ago (see http://bugs.python.org/issue21303), but hadn't > gotten anything done with Tix yet. It should just need python.mak > updated to point to Tcl/Tk 8.5.15; it's on my list to get fixed as > soon as I can. Tix for 2.7 is now http://svn.python.org/projects/external/tix-8.4.3.5. You can build it with this monster of a command, run from tix-8.4.3.5\win: nmake -f python.mak DEBUG=0 MACHINE=IX86 COMPILERFLAGS=-DWINVER=0x0500 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk all Use "install" instead of "all" after building to install it to ..\..\tcltk. Set DEBUG and MACHINE as needed; DEBUG does not need to be set if you're building Release, but MACHINE always has to be set so that Tix uses the right build dir for Tk (IX86 for 32-bit, AMD64 for 64-bit). -- Zach ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On Fri, 09 May 2014 11:39:02 -0400, Donald Stufft wrote: > > On May 9, 2014, at 9:58 AM, M.-A. Lemburg wrote: > > On 09.05.2014 13:44, Donald Stufft wrote: > >> On May 9, 2014, at 4:12 AM, M.-A. Lemburg wrote: > > I snipped the rest of the discussion and reliability, using > > unmaintained packages and projects using their own mirrors (which > > should really be the standard, not an exceptional case), > > because it's not really leading anywhere: > > Using your own mirror shouldnât be the standard if all youâre doing > is automatically updating that mirror. Itâs a hack to get around > unreliability and it should be seen of as a sign of a failure to provide > a service that people can rely on and thatâs how I see it. People > depend on this service and itâs irresponsible to not treat it as a > critical piece of infrastructure. I don't understand this. Why it is our responsibility to provide a free service for a large project to repeatedly download a set of files they need? Why does it not make more sense for them to download them once, and only update their local copies when they change? That's almost completely orthogonal to making the service we do provide reliable. For perspective, Gentoo requests that people only do an emerge sync at most once a day, and if they have multiple machines to update, that they only do one pull, and they update the rest of their infrastructure from their local copy. As another point of information for comparison, Gentoo downloads files from wherever they are hosted first, and only if that fails falls back to a Gentoo provided mirror (if I remember correctly...I think the Gentoo mirror copy doesn't always exist?). --David ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 1:28 PM, R. David Murray wrote: > On Fri, 09 May 2014 11:39:02 -0400, Donald Stufft wrote: >> >> On May 9, 2014, at 9:58 AM, M.-A. Lemburg wrote: >>> On 09.05.2014 13:44, Donald Stufft wrote: On May 9, 2014, at 4:12 AM, M.-A. Lemburg wrote: >>> I snipped the rest of the discussion and reliability, using >>> unmaintained packages and projects using their own mirrors (which >>> should really be the standard, not an exceptional case), >>> because it's not really leading anywhere: >> >> Using your own mirror shouldn’t be the standard if all you’re doing >> is automatically updating that mirror. It’s a hack to get around >> unreliability and it should be seen of as a sign of a failure to provide >> a service that people can rely on and that’s how I see it. People >> depend on this service and it’s irresponsible to not treat it as a >> critical piece of infrastructure. > > I don't understand this. Why it is our responsibility to provide a > free service for a large project to repeatedly download a set of files > they need? Why does it not make more sense for them to download them > once, and only update their local copies when they change? That's almost > completely orthogonal to making the service we do provide reliable. Well here’s the thing right. The large projects repeatedly downloading the same set of files is a canary. If any particular project goes uninstallable on PyPI (or if PyPI itself goes down) then nobody can install it, the people installing things over and over every day or the people who just happened to be installing it during that downtime. However intermittent failures and general insatiability is going to be noticed by the projects who install things over and over again quicker and easier and thus it becomes a lot easier to use them as a general gauge for what the average “uptime” is. IOW if PyPI goes unavailable for 10 minutes 5 times a day, you might get a handful of “small” installers (e.g. not the big projects) in each downtime, but a different set who are likely to shrug it off and just call treat it as the norm even though it’s very disruptive to what they’re doing. However the big project is highly likely to hit every single one of those downtimes and be able to say “wow PyPI is flaky as hell”. To expand further on that if we assume that we want ``pip install `` to be reliable and not work sometimes and work at other times then we’re aiming for as high as uptime as possible. PyPI gets enough traffic that any single large project isn’t a noticeable drop in our bucket and due to the way our caching works it actually helps us to be faster and more reliable to have people constantly hitting packages because they’ll be in cache and able to be served without hitting the Origin servers. Just for the record, PyPI gets roughly 350 req/s basically 24/7, in the month of April we served 71.4TB of data with 877.4 million requests of which 80.5% never made it to the actual servers that run PyPI and were served directly out of the geo distributed CDN that sits in front of PyPI. We are vastly better positioned to maintain a reliable infrastructure than ask that every large project that uses Python to do the same. The reason that it’s our responsibility for providing it is because we chose to provide it. There isn’t a moral imperative to run PyPI, but running PyPI badly seems like a crummy thing to do. > > For perspective, Gentoo requests that people only do an emerge sync at > most once a day, and if they have multiple machines to update, that they > only do one pull, and they update the rest of their infrastructure from > their local copy. To be clear, there are other reasons to run a local mirror but I don’t think that it’s reasonable to expect anyone who wants a reliable install using pip to stand up their own infrastructure. Further to this point here I’m currently working on adding caching by default for pip so that we minimize how often different people hit PyPI and we do it automatically and in a way that doesn’t generally require people to think about it and that also doesn’t require them to stand up their own infra. > > As another point of information for comparison, Gentoo downloads files > from wherever they are hosted first, and only if that fails falls back to > a Gentoo provided mirror (if I remember correctly...I think the Gentoo > mirror copy doesn't always exist?). > > --David > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-de
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On 5/9/2014 2:12 PM, Donald Stufft wrote: On May 9, 2014, at 1:28 PM, R. David Murray wrote: I don't understand this. Why it is our responsibility to provide a free service for a large project to repeatedly download a set of files they need? Why does it not make more sense for them to download them once, and only update their local copies when they change? That's almost completely orthogonal to making the service we do provide reliable. Well here’s the thing right. The large projects repeatedly downloading the same set of files is a canary. If any particular project goes uninstallable on PyPI (or if PyPI itself goes down) then nobody can install it, the people installing things over and over every day or the people who just happened to be installing it during that downtime. However intermittent failures and general insatiability is going to be noticed by the projects who install things over and over again quicker and easier and thus it becomes a lot easier to use them as a general gauge for what the average “uptime” is. I have had the same question as David, so I also appreciate your answer. IOW if PyPI goes unavailable for 10 minutes 5 times a day, you might get a handful of “small” installers (e.g. not the big projects) in each downtime, but a different set who are likely to shrug it off and just call treat it as the norm even though it’s very disruptive to what they’re doing. However the big project is highly likely to hit every single one of those downtimes and be able to say “wow PyPI is flaky as hell”. To expand further on that if we assume that we want ``pip install `` to be reliable and not work sometimes and work at other times then we’re aiming for as high as uptime as possible. PyPI gets enough traffic that any single large project isn’t a noticeable drop in our bucket and due to the way our caching works it actually helps us to be faster and more reliable to have people constantly hitting packages because they’ll be in cache and able to be served without hitting the Origin servers. Just for the record, PyPI gets roughly 350 req/s basically 24/7, in the month of April we served 71.4TB of data with 877.4 million requests of which 80.5% never made it to the actual servers that run PyPI and were served directly out of the geo distributed CDN that sits in front of PyPI. We are vastly better positioned to maintain a reliable infrastructure than ask that every large project that uses Python to do the same. The reason that it’s our responsibility for providing it is because we chose to provide it. There isn’t a moral imperative to run PyPI, but running PyPI badly seems like a crummy thing to do. Agreed. For perspective, Gentoo requests that people only do an emerge sync at most once a day, and if they have multiple machines to update, that they only do one pull, and they update the rest of their infrastructure from their local copy. To be clear, there are other reasons to run a local mirror but I don’t think that it’s reasonable to expect anyone who wants a reliable install using pip to stand up their own infrastructure. Ok, you are not saying that caching is bad, but that having everyone reinvent caching, and possibly doing it badly, or at least not in thebest way, is bad. Further to this point here I’m currently working on adding caching by default for pip so that we minimize how often different people hit PyPI and we do it automatically and in a way that doesn’t generally require people to think about it and that also doesn’t require them to stand up their own infra. This seems like the right solution. It would sort of make each machine a micro-CDN node. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
On May 9, 2014, at 4:20 PM, Terry Reedy wrote: > On 5/9/2014 2:12 PM, Donald Stufft wrote: >> >> On May 9, 2014, at 1:28 PM, R. David Murray wrote: > >>> I don't understand this. Why it is our responsibility to provide a >>> free service for a large project to repeatedly download a set of files >>> they need? Why does it not make more sense for them to download them >>> once, and only update their local copies when they change? That's almost >>> completely orthogonal to making the service we do provide reliable. >> >> Well here’s the thing right. The large projects repeatedly downloading the >> same set of files is a canary. If any particular project goes uninstallable >> on >> PyPI (or if PyPI itself goes down) then nobody can install it, the people >> installing things over and over every day or the people who just happened >> to be installing it during that downtime. However intermittent failures and >> general insatiability is going to be noticed by the projects who install >> things >> over and over again quicker and easier and thus it becomes a lot easier >> to use them as a general gauge for what the average “uptime” is. > > I have had the same question as David, so I also appreciate your answer. > >> IOW if PyPI goes unavailable for 10 minutes 5 times a day, you might get >> a handful of “small” installers (e.g. not the big projects) in each downtime, >> but a different set who are likely to shrug it off and just call treat it as >> the >> norm even though it’s very disruptive to what they’re doing. However the >> big project is highly likely to hit every single one of those downtimes and >> be able to say “wow PyPI is flaky as hell”. >> >> To expand further on that if we assume that we want ``pip install `` >> to be reliable and not work sometimes and work at other times then we’re >> aiming for as high as uptime as possible. PyPI gets enough traffic that >> any single large project isn’t a noticeable drop in our bucket and due to the >> way our caching works it actually helps us to be faster and more reliable >> to have people constantly hitting packages because they’ll be in cache >> and able to be served without hitting the Origin servers. >> >> Just for the record, PyPI gets roughly 350 req/s basically 24/7, in the >> month of April we served 71.4TB of data with 877.4 million requests of >> which 80.5% never made it to the actual servers that run PyPI and were >> served directly out of the geo distributed CDN that sits in front of PyPI. We >> are vastly better positioned to maintain a reliable infrastructure than ask >> that every large project that uses Python to do the same. > >> The reason that it’s our responsibility for providing it is because we chose >> to provide it. There isn’t a moral imperative to run PyPI, but running PyPI >> badly seems like a crummy thing to do. > > Agreed. > >>> For perspective, Gentoo requests that people only do an emerge sync at >>> most once a day, and if they have multiple machines to update, that they >>> only do one pull, and they update the rest of their infrastructure from >>> their local copy. >> >> To be clear, there are other reasons to run a local mirror but I don’t think >> that >> it’s reasonable to expect anyone who wants a reliable install using pip to >> stand up their own infrastructure. > > Ok, you are not saying that caching is bad, but that having everyone reinvent > caching, and possibly doing it badly, or at least not in thebest way, is bad. Yea, caching isn’t in general a bad thing, and actually PyPI uses it heavily. All access to /simple/ and /packages/ is cached for 24 hours by our CDN unless someone uploads or deletes a file, in which case we selectively purge those URLs from the CDN cache so that they (nearly) instantly get the updated results. Warehouse (aka PyPI 2.0) is designed to utilize our CDN cache even further and I’m hoping to get our cache rate even higher using it. > >> Further to this point here I’m currently working on adding caching by default >> for pip so that we minimize how often different people hit PyPI and we do it >> automatically and in a way that doesn’t generally require people to think >> about >> it and that also doesn’t require them to stand up their own infra. > > This seems like the right solution. It would sort of make each machine a > micro-CDN node. Yes, it’s bog standard HTTP stuff just like a browser does it. The major difference is we’re limiting the maximum lifetime of a cache item in the client (pip) for the index pages but we are not doing that for the package files themselves. This is done to prevent a misconfigured server from causing pip to not see new versions for hours/days/weeks/years/whatever. Additionally this change also includes making pip smarter about HTTP requests in that if we have a stale item in the cache which as a Last-Modified or an ETag header we’ll do a conditional GET which will hopefully be returned with an HTTP 304 Not Modified so
[Python-Dev] should tests be thread-safe?
Hi, May tests expect that unless they themselves start a thread then there are no threads to worry about? I see that some old tests are not thread-safe and I have not found it to be explicitly mentioned in the devguide. I've written a test for http://bugs.python.org/issue21332 that is known to be non-thread-safe. Is it acceptable for new tests? -- akira ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] pushing 2.7.7 back a week
I'm pushing the release schedule for 2.7.7 back a week. That means the release candidate will be next weekend (May 17). This will hopefully save some trouble for the people who build our binaries, since 3.4.1 final is happening next weekend, too. Regards, Benjamin ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
