[Python-ideas] Re: dict method to retrieve a key from a value

[Dom Grigonis]

> On 2 Jul 2023, at 10:12, Paul Moore  wrote:
> 
> On Sun, 2 Jul 2023 at 06:11, Christopher Barker  > wrote:
> 
> The OP of this thread is not alone -- folks want an authoritative source
> ...
> Unfortunately, too much of this discussion is framed as “someone should”, or 
> “it would be good if”. No-one is saying “I will”. Naming groups, like “the 
> PyPA should” doesn’t help either - groups don’t do things, people do. Who in 
> the PyPA? Me? Nope, sorry, I don’t have the time or interest - I’d *use* a 
> curated index, I sure as heck couldn’t *create* one.

If someone took this on, I co-lunteer.___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/DOSZCAQEMG6KMM5QEIG2USMYVOBAKB5Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] "Curated" package repo?

[Christopher Barker]

Stating a new thread with a correct title.

On 2 Jul 2023, at 10:12, Paul Moore  wrote:

Unfortunately, too much of this discussion is framed as “someone should”,
> or “it would be good if”. No-one is saying “I will”. Naming groups, like
> “the PyPA should” doesn’t help either - groups don’t do things, people do.
> Who in the PyPA? Me? Nope, sorry, I don’t have the time or interest - I’d
> *use* a curated index, I sure as heck couldn’t *create* one.


Well, I started this topic, and I don't *think* I ever wrote "someone
should", and I certainly didn't write "PyPa should".

But whatever I or anyone else wrote, my intention was to discuss what might
be done to address what I think is a real problem/limitation in the
discoverability of useful packages for Python.

And I think of it not so much as "someone should" but as "it would be nice
to have".

Of course, any of these ideas would take a lot of work to implement -- and
even though there are a lot of folks, on this list  and elsewhere, that
would like to help, I don't think any substantial open-source project has
gotten anywhere without a concerted effort by a very small group (often
just 1) of people doing a lot of work to get it to a useful state before a
larger group can contribute. So I"m fully aware that nothings going to
happen unless *someone* really puts the work in up front. That someone
*might* be me, but I'm really good at over-committing myself, and not so
great at keeping my nose to the grindstone, so 

And I think this particular problem calls for a solution that would have to
be pretty well established before reaching critical mass to actually be
useful -- after all, we already have PyPi -- why go anywhere else that is
less comprehensive?

All that being said, it's still worth having a conversation about what a
good solution might look like -- there are a lot of options, and hashing
out some of the ideas might inspire someone to rise to the occasion.

The :problem", as I see it.

 - The Python standard library is not, and will never be fully
comprehensive -- most projects require *some* third party packages.
 - There are a LOT of packages available on PyPi -- with a very wide range
of usefulness, quality and maintenance -- everything from widely used
packages with a huge community (e.g. numpy) to packages that are release
0.0.1, and never seen an update, and may not even work.

So the odds that there's a package that does what you need are good, but it
can be pretty hard to find them sometimes -- and can be a fair bit of work
to sift through to find the good ones -- and many folks don't feel
qualified to do so.

This can result in two opposite consequences:

1) People using a package that really isn't reliable or maintained (or not
supported on all platforms, or ..) and getting stuck with it (I've had that
on some of my projects -- I'm pretty careful, but not everyone on my team
is)

2) People writing their own code - wasting time, and maybe not getting a
very good solution either. I've also had that problem on my projects...

To avoid this -- SOME way for folks to find packages that have at least has
some level of vetting would be good -- exactly what level of vetting, is a
very open question, but I think "even a little" could be very helpful.

A few ideas that have come up in the previous thread -- more or less in
order of level of effort.

1) A "clearing house" of package reviews, for lack of a better word -- a
single web site that would point to various resources on the internet --
blog posts, etc, that would help people select packages. So one might go to
the "JSON: section, and find links to some of the comparisons of JSON
parsers, to help you choose. The authors of this site could even
solicit folks to write reviews, etc that they could then point to.
Chris A: this is my interpretation of your decentralized idea -- please do
correct me if I got it wrong.

2) A Python package review site. This could be setup and managed with, e.g.
a gitHub repo, so that there could be a small number of editors, but anyone
could contribute a review via PR. The ultimiate goal would be
reviews/recommendations of all the major package categories on PyPi. If
well structured and searchable, this could be very useful.
 - This was proposed by someone on the previous thread -- again, correct me
if I'm wrong.

3) A rating system built into PyPi -- This could be a combination of two
things:
  A - Automated analysis -- download stats, dependency stats, release
frequency, etc, etc, etc.
  B - Community ratings -- upvotes. stars, whatever.

If done well, that could be very useful -- search on PyPi listed by rating.
However -- :done well" ios a huge challenge -- I don't think there's a way
to do the automated system right, and community scoring can be abused
pretty easily. But maybe folks smarter than me could make it work with one
or both of these approaches.

4)  A self contained repository of packages that you could point pip to --
it would contain only the pac

[Python-ideas] Re: "Curated" package repo?

[Chris Angelico]

On Wed, 5 Jul 2023 at 10:26, Christopher Barker  wrote:
> The :problem", as I see it.
>
>  - The Python standard library is not, and will never be fully comprehensive 
> -- most projects require *some* third party packages.
>  - There are a LOT of packages available on PyPi -- with a very wide range of 
> usefulness, quality and maintenance -- everything from widely used packages 
> with a huge community (e.g. numpy) to packages that are release 0.0.1, and 
> never seen an update, and may not even work.
>

Remember though that this problem is also Python's strength here. The
standard library does not NEED to be comprehensive, and publishing on
PyPI is deliberately easy. The barrier to entry for the stdlib is
high; the barrier to entry for PyPI is low.

> 1) A "clearing house" of package reviews, for lack of a better word -- a 
> single web site that would point to various resources on the internet -- blog 
> posts, etc, that would help people select packages. So one might go to the 
> "JSON: section, and find links to some of the comparisons of JSON parsers, to 
> help you choose. The authors of this site could even solicit folks to write 
> reviews, etc that they could then point to.
> Chris A: this is my interpretation of your decentralized idea -- please do 
> correct me if I got it wrong.
>

That is a correct interpretation of what I said, yes. I'll take it a
bit further though and add that this "single web site" would be
ideally editable by multiple people. In fact, Python has a place for
that sort of thing:

https://wiki.python.org/moin/FrontPage

Imagine a page like
https://wiki.python.org/moin/Python_Package_Recommendations (doesn't
currently exist) that would be managed the same way as other
recommendation pages like
https://wiki.python.org/moin/IntroductoryBooks - anyone can add a link
to their blog post about packages, and if they've focused on a
specific category (eg "web frameworks"), that can be right there in
the wiki page so people can search for it.

That way, it's decentralized for editing, but has a central "hub" that
people can easily find.

> 2) A Python package review site. This could be setup and managed with, e.g. a 
> gitHub repo, so that there could be a small number of editors, but anyone 
> could contribute a review via PR. The ultimiate goal would be 
> reviews/recommendations of all the major package categories on PyPi. If well 
> structured and searchable, this could be very useful.
>  - This was proposed by someone on the previous thread -- again, correct me 
> if I'm wrong.
>

I suspect this would end up being broadly equivalent to the first
option, but with more effort by a core group of people (or a single
maintainer), and in return, would have a more consistent look and
feel.

> 3) A rating system built into PyPi -- This could be a combination of two 
> things:
>   A - Automated analysis -- download stats, dependency stats, release 
> frequency, etc, etc, etc.
>   B - Community ratings -- upvotes. stars, whatever.
>
> If done well, that could be very useful -- search on PyPi listed by rating. 
> However -- :done well" ios a huge challenge -- I don't think there's a way to 
> do the automated system right, and community scoring can be abused pretty 
> easily. But maybe folks smarter than me could make it work with one or both 
> of these approaches.
>

Huge challenge indeed. Possibly insurmountable. Popularity contests
(purely based on download stats and such) have their value, but do not
tell you what's best, only what's the most used. Community ratings, as
you say, can be gamed all too easily, plus they STILL tend to favour
those that are heavily used rather than niche things that are
potentially better. Neither of them adequately answers questions like
"which is right *for this use-case*", which will leave a lot of
packages out in the cold.

> 4)  A self contained repository of packages that you could point pip to -- it 
> would contain only the packages that had met some sort of "vetting" criteria. 
> In theory, anyone could run it, but a stamp of approval from the PSF would 
> make it far more acceptable to people. This would be a LOT of work to get set 
> up, and still a lot of work to maintain.
>

Definitely possible; how would this compare to Conda? What about OS
package managers like the Debian repositories?

> Personally, I think (4) is the best end result, but probably the most work as 
> well[*], so ???
>
> (1) and (2) have the advantage that they could be useful even without being 
> comprehensive -- tthey's need to have some critical mass to get any traction, 
> but maybe not THAT much.

Right, and notably, they can be useful without covering every topic.
You could write a blog post about database ORM packages, and that's
useful right there, without worrying about whether there's any review
of internet protocol packages.

Hmm. Once that sort of collection gets some traction, someone might
say "Hey, anyone know about grammar parsers?" and add that to a "help

[Python-ideas] Re: "Curated" package repo?

[Dom Grigonis]

To have 4) would be great. 2) could be a temporary test pilot to get some ideas 
when/if 4) was to be implemented.

Starting point for 2) could be simple (?). E.g. 4 parts:
  1. Features & integration
  2. Performance - set-up automatic benchmarking via CI
  3. Status (stack hits, user base, open issues, lines of code, update 
frequency & similar)
  4. User rating (wander if up/down voting or star rating can be somehow 
implemented in github)

Author collects inputs from this e-mail group or via opened issue and collates 
results. Once the comparison tables are there package authors can then issue PR 
themselves or next interested user does the update.

Could be a simple way to gauge variables before committing to 3) or 4).

> On 5 Jul 2023, at 03:21, Christopher Barker  wrote:
> 
> Stating a new thread with a correct title.
> 
> On 2 Jul 2023, at 10:12, Paul Moore  > wrote:
> 
> Unfortunately, too much of this discussion is framed as “someone should”, or 
> “it would be good if”. No-one is saying “I will”. Naming groups, like “the 
> PyPA should” doesn’t help either - groups don’t do things, people do. Who in 
> the PyPA? Me? Nope, sorry, I don’t have the time or interest - I’d *use* a 
> curated index, I sure as heck couldn’t *create* one.
> 
> Well, I started this topic, and I don't *think* I ever wrote "someone 
> should", and I certainly didn't write "PyPa should".
> 
> But whatever I or anyone else wrote, my intention was to discuss what might 
> be done to address what I think is a real problem/limitation in the 
> discoverability of useful packages for Python.
> 
> And I think of it not so much as "someone should" but as "it would be nice to 
> have".
> 
> Of course, any of these ideas would take a lot of work to implement -- and  
> even though there are a lot of folks, on this list  and elsewhere, that would 
> like to help, I don't think any substantial open-source project has gotten 
> anywhere without a concerted effort by a very small group (often just 1) of 
> people doing a lot of work to get it to a useful state before a larger group 
> can contribute. So I"m fully aware that nothings going to happen unless 
> *someone* really puts the work in up front. That someone *might* be me, but 
> I'm really good at over-committing myself, and not so great at keeping my 
> nose to the grindstone, so 
> 
> And I think this particular problem calls for a solution that would have to 
> be pretty well established before reaching critical mass to actually be 
> useful -- after all, we already have PyPi -- why go anywhere else that is 
> less comprehensive?
> 
> All that being said, it's still worth having a conversation about what a good 
> solution might look like -- there are a lot of options, and hashing out some 
> of the ideas might inspire someone to rise to the occasion.
> 
> The :problem", as I see it.
> 
>  - The Python standard library is not, and will never be fully comprehensive 
> -- most projects require *some* third party packages.
>  - There are a LOT of packages available on PyPi -- with a very wide range of 
> usefulness, quality and maintenance -- everything from widely used packages 
> with a huge community (e.g. numpy) to packages that are release 0.0.1, and 
> never seen an update, and may not even work.
> 
> So the odds that there's a package that does what you need are good, but it 
> can be pretty hard to find them sometimes -- and can be a fair bit of work to 
> sift through to find the good ones -- and many folks don't feel qualified to 
> do so.
> 
> This can result in two opposite consequences:
> 
> 1) People using a package that really isn't reliable or maintained (or not 
> supported on all platforms, or ..) and getting stuck with it (I've had that 
> on some of my projects -- I'm pretty careful, but not everyone on my team is) 
> 
> 2) People writing their own code - wasting time, and maybe not getting a very 
> good solution either. I've also had that problem on my projects...
> 
> To avoid this -- SOME way for folks to find packages that have at least has 
> some level of vetting would be good -- exactly what level of vetting, is a 
> very open question, but I think "even a little" could be very helpful.
> 
> A few ideas that have come up in the previous thread -- more or less in order 
> of level of effort.
> 
> 1) A "clearing house" of package reviews, for lack of a better word -- a 
> single web site that would point to various resources on the internet -- blog 
> posts, etc, that would help people select packages. So one might go to the 
> "JSON: section, and find links to some of the comparisons of JSON parsers, to 
> help you choose. The authors of this site could even solicit folks to write 
> reviews, etc that they could then point to.
> Chris A: this is my interpretation of your decentralized idea -- please do 
> correct me if I got it wrong.
> 
> 2) A Python package review site. This could be setup and managed with, e.g. a 
> git

[Python-ideas] Re: "Curated" package repo?

[Jonathan Crall]

I like the idea of a vetted package index that pip can point to. The more I
think about it, the more I think that it needs some sort of peer review
system as the barrier to entry, and my thoughts to to establishing some
DeSci DAO that could distribute the peer review of packages amongst a set
of trusted maintainers while also being a mechanism to add new trusted
maintainers to the peer-review pool. Peer reviewers could be funded via a
fee to submit a package for publishing. There are a lot of open questions
of how this would be done correctly - or even if this is necessary - but I
think to achieve a scalable, funded, decentralized, and trustworthy package
index a DAO makes some amount of sense.



On Tue, Jul 4, 2023 at 8:52 PM Dom Grigonis  wrote:

> To have 4) would be great. 2) could be a temporary test pilot to get some
> ideas when/if 4) was to be implemented.
>
> Starting point for 2) could be simple (?). E.g. 4 parts:
>   1. Features & integration
>   2. Performance - set-up automatic benchmarking via CI
>   3. Status (stack hits, user base, open issues, lines of code, update
> frequency & similar)
>   4. User rating (wander if up/down voting or star rating can be somehow
> implemented in github)
>
> Author collects inputs from this e-mail group or via opened issue and
> collates results. Once the comparison tables are there package authors can
> then issue PR themselves or next interested user does the update.
>
> Could be a simple way to gauge variables before committing to 3) or 4).
>
> On 5 Jul 2023, at 03:21, Christopher Barker  wrote:
>
> Stating a new thread with a correct title.
>
> On 2 Jul 2023, at 10:12, Paul Moore  wrote:
>
> Unfortunately, too much of this discussion is framed as “someone should”,
>> or “it would be good if”. No-one is saying “I will”. Naming groups, like
>> “the PyPA should” doesn’t help either - groups don’t do things, people do.
>> Who in the PyPA? Me? Nope, sorry, I don’t have the time or interest - I’d
>> *use* a curated index, I sure as heck couldn’t *create* one.
>
>
> Well, I started this topic, and I don't *think* I ever wrote "someone
> should", and I certainly didn't write "PyPa should".
>
> But whatever I or anyone else wrote, my intention was to discuss what
> might be done to address what I think is a real problem/limitation in the
> discoverability of useful packages for Python.
>
> And I think of it not so much as "someone should" but as "it would be nice
> to have".
>
> Of course, any of these ideas would take a lot of work to implement --
> and  even though there are a lot of folks, on this list  and elsewhere,
> that would like to help, I don't think any substantial open-source project
> has gotten anywhere without a concerted effort by a very small group (often
> just 1) of people doing a lot of work to get it to a useful state before a
> larger group can contribute. So I"m fully aware that nothings going to
> happen unless *someone* really puts the work in up front. That someone
> *might* be me, but I'm really good at over-committing myself, and not so
> great at keeping my nose to the grindstone, so 
>
> And I think this particular problem calls for a solution that would have
> to be pretty well established before reaching critical mass to actually be
> useful -- after all, we already have PyPi -- why go anywhere else that is
> less comprehensive?
>
> All that being said, it's still worth having a conversation about what a
> good solution might look like -- there are a lot of options, and hashing
> out some of the ideas might inspire someone to rise to the occasion.
>
> The :problem", as I see it.
>
>  - The Python standard library is not, and will never be fully
> comprehensive -- most projects require *some* third party packages.
>  - There are a LOT of packages available on PyPi -- with a very wide range
> of usefulness, quality and maintenance -- everything from widely used
> packages with a huge community (e.g. numpy) to packages that are release
> 0.0.1, and never seen an update, and may not even work.
>
> So the odds that there's a package that does what you need are good, but
> it can be pretty hard to find them sometimes -- and can be a fair bit
> of work to sift through to find the good ones -- and many folks don't feel
> qualified to do so.
>
> This can result in two opposite consequences:
>
> 1) People using a package that really isn't reliable or maintained (or not
> supported on all platforms, or ..) and getting stuck with it (I've had that
> on some of my projects -- I'm pretty careful, but not everyone on my team
> is)
>
> 2) People writing their own code - wasting time, and maybe not getting a
> very good solution either. I've also had that problem on my projects...
>
> To avoid this -- SOME way for folks to find packages that have at least
> has some level of vetting would be good -- exactly what level of vetting,
> is a very open question, but I think "even a little" could be very helpful.
>
> A few ideas that have come up in

[Python-ideas] Re: "Curated" package repo?

[Chris Angelico]

On Wed, 5 Jul 2023 at 15:07, Jonathan Crall  wrote:
>
> I like the idea of a vetted package index that pip can point to. The more I 
> think about it, the more I think that it needs some sort of peer review 
> system as the barrier to entry, and my thoughts to to establishing some DeSci 
> DAO that could distribute the peer review of packages amongst a set of 
> trusted maintainers while also being a mechanism to add new trusted 
> maintainers to the peer-review pool. Peer reviewers could be funded via a fee 
> to submit a package for publishing. There are a lot of open questions of how 
> this would be done correctly - or even if this is necessary - but I think to 
> achieve a scalable, funded, decentralized, and trustworthy package index a 
> DAO makes some amount of sense.
>

This adds up to a HUGE barrier to entry for new packages. For your
proposal to be successful, a good number of users would need to point
pip to the vetted index and NOT to the normal one (otherwise there's
no benefit - you're just getting non-vetted packages), and in order
for a new package to become relevant, it needs to:

1. Pay money. Even if it's not a huge dollar amount, that is already a
very significant barrier for casual users.
2. Get reviewed. This is going to take time.
3. Pass the review. If the review system is at all meaningful, it has
to knock some packages back, otherwise all you have is "pay to be
visible" which is a horrible system.
4. OR, instead of those steps: Convince your users to switch to the
"untrusted" package repository.

That makes for a very closed-off ecosystem, where an incumbent has a
dramatic advantage over anything that comes along. It would almost
certainly require that vetted packages not depend on non-vetted
packages (otherwise you'd need some bizarre mechanic whereby "pip
install package-name" looks at one repository, but it resolves
dependencies by looking at a different one), so nothing would have a
chance to be seen in the walled garden until you get it vetted AND
recognized.

So, sure, this would make life easier for those who want to randomly
download packages without thinking about them, but at the cost of
making the package repository extremely minimalist and insular. It'd
make the Python packaging ecosystem look as unfriendly as iPhone app
publishing.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/2WLUMPDRTDVWIHWIJ7KINPF6LD4KRDCY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: "Curated" package repo?

[Christopher Barker]

On Tue, Jul 4, 2023 at 10:06 PM Jonathan Crall  wrote:

> I like the idea of a vetted package index that pip can point to. The more
> I think about it, the more I think that it needs some sort of peer review
> system as the barrier to entry, and my thoughts to to establishing some
> DeSci DAO that could distribute the peer review of packages amongst a set
> of trusted maintainers while also being a mechanism to add new trusted
> maintainers to the peer-review pool.
>


> Peer reviewers could be funded via a fee to submit a package for
> publishing.
>

I agree until this point -- we REALLY don't want to have a pay to play
system.

Yes, it needs to be funded somehow, but some sort of donation / non profit
/ etc funding mechanism would be best -- but I don't think peer
reviewers should be paid. Peer review in academic journals isn't cash
compensated either.


> I think to achieve a scalable, funded, decentralized, and trustworthy
> package index a DAO makes some amount of sense.
>

I had to look that up: "Decentralized autonomous organization (DAO)"

So, yes.

-CHB



-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/C2W5F6U5Y55SBGQZ5S3O3ZETSVYCOQYG/
Code of Conduct: http://python.org/psf/codeofconduct/