Re: Python 2, Python 3, Stretch & Buster
On Mon, Apr 20, 2015 at 08:39:35AM -0400, Scott Kitterman wrote: > I think the porting team would do most of the work, we'd just need you to > review and integrate changes as they come along. That works for me. Enrico -- GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150423124541.ga8...@enricozini.org
Re: Python 2, Python 3, Stretch & Buster
On Mon, Apr 20, 2015 at 11:14:28AM -0400, Paul Tagliamonte wrote: > So, round one of all of this is getting the critical path *under* each > of our services ready, so that when we need to migrate, we don't need > to scramble. There is another constraint that I forgot: the production environment is Debian Stable, which will soon mean Jessie. If a python2-only module gets ported to python3 in jessie+1, to use it for debian.org services the python3 version also needs to be maintained in backports. For example, I am about to add python-git as a dependency to nm.debian.org, as I need it to parse the stream of changes that I get via the keyring.debian.org git changelog, and python-git is python2-only in Jessie[1]. Having python3-git in jessie+1 won't be enough to make nm.debian.org deployable with python3: python3-git will also need to be in jessie-backports, and with a commitment to keep it up to date with security uploads, because I'm going to run it every night to parse git logs that are obtained via potentially insecure network connections. Enrico [1] I've recently been really sad to figure out that python-git is python2-only, and so is dulwich. As a personal policy, if I start a new python project now I start it with python 3.4. I had to start a new project with python2 because of that, and I felt dirty. -- GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini signature.asc Description: Digital signature
Re: Python 2, Python 3, Stretch & Buster
On Mon, Apr 20, 2015 at 01:33:37PM -0400, Barry Warsaw wrote: > Personally, I think the use of dict.iteritems is way overused. The > performance hit is theoretical unless you've actually measured it, and even > then I think it probably doesn't help much unless you've got huge dicts. I tend not to take other people's personal preferences as useful data points when choosing my coding style, so I tend to get easily irritated by comments like yours, just as I was irritated by people on that lwn thread saying that all the incompatibilities introduced by python 3.5 where "obscure". I generally pick my coding style according to the official documentation. If iteritems is documented to iterate a dictionary with an O(1) of memory usage, and items() is documented as creating a copy of the dictionary into a list and therefore having an O(n) memory complexity, I will use iteritems() unless I need to have a list as a result. If iteritems() turns out to be less performing, I consider it a problem in the way things are documented, and I won't be held accountable for not making measurements. It is my habit to carefully follow the documentation trying to pick the tool that has built precisely with the intention of addressing what I am doing. I would rather hear people saying "I'm sorry that things took a different direction than expected", rather than hear people moaning about the use of obscure features, or the neglect of doing measurements. > >Also, if I implement new features I am not necessarily going to test > >them on python3 unless it somehow happens automatically. > > One word: tox :) Nice word. It is a tool that I have never used, and that is not available on debian stable where the services are currently deployed. > >Running the test suite twice is not much of an option, though, because it > >already takes a lot of time to run, and doubling the time it takes just means > >that code is going to get tested less. Unless of course someone reviews my > >test code and finds a way to make it run super fast: I'd certainly welcome > >that! > > The nice thing is you can run `tox -e py27` for your normal cases, and then > `tox` before you push updates to just be sure it works for all supported > versions. The unfortunate thing is that I would have to budget several hours of my time to learn a new tool and integrate it in my workflow, and I have more urgent things on top of my todo list than "redo what currently works fine using a new tool". I would be fine if someone sent me patches to make the code *also* work with tox, but I would not personally use it myself until I have allocated time that I currently don't have to master it, because I refuse to be the primary maintainer of code that uses tools that I do not know well[1]. Knowledge transfer and migrations may be fun for some, but for others *they are a cost*, as shown by the mere existence of this thread. Enrico [1] also, given the volatility of a lot of new tools in the python ecosystem, I have adopted the safety practice of making sure that a tool has been /widely/ adopted for at least a year or two before even bothering to look at it. I like that practice: I saved myself the burden of rewriting setup.py files countless times because of that. -- GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini signature.asc Description: Digital signature
Re: Python 2, Python 3, Stretch & Buster
On Thursday, April 23, 2015 02:52:33 PM Enrico Zini wrote: > On Mon, Apr 20, 2015 at 11:14:28AM -0400, Paul Tagliamonte wrote: > > So, round one of all of this is getting the critical path *under* each > > of our services ready, so that when we need to migrate, we don't need > > to scramble. > > There is another constraint that I forgot: the production environment is > Debian Stable, which will soon mean Jessie. If a python2-only module > gets ported to python3 in jessie+1, to use it for debian.org services > the python3 version also needs to be maintained in backports. > > For example, I am about to add python-git as a dependency to > nm.debian.org, as I need it to parse the stream of changes that I get > via the keyring.debian.org git changelog, and python-git is python2-only > in Jessie[1]. Having python3-git in jessie+1 won't be enough to make > nm.debian.org deployable with python3: python3-git will also need to be > in jessie-backports, and with a commitment to keep it up to date with > security uploads, because I'm going to run it every night to parse git > logs that are obtained via potentially insecure network connections. > > > Enrico > > [1] I've recently been really sad to figure out that python-git is > python2-only, and so is dulwich. As a personal policy, if I start a > new python project now I start it with python 3.4. I had to start a > new project with python2 because of that, and I felt dirty. Where we started this was with a goal to make sure that 100% of needed depends will be available in Stretch, so that by the time Buster is released, services can be migrated. It'd be nice to migrate other things sooner, but getting the depenencies complete is the essential task for the Stretch cycle for exactly this reason. Scott K signature.asc Description: This is a digitally signed message part.
Re: Re: Python 2, Python 3, Stretch & Buster
On Mon, Apr 20, 2015 at 09:58:31PM +0200, Karsten Hilbert wrote: > > Also, if I implement new features I am not necessarily going to test > > them on python3 unless it somehow happens automatically. Running the > > test suite twice is not much of an option, though, because it already > > takes a lot of time to run, and doubling the time it takes just means > > that code is going to get tested less. > > Would it be an option to > > - work on the code > - run the py2 test suite > - deploy > - run the py3 test suite > > ? It is an option. I propose another option: * I work on the code. * I run the py2 test suite. * I deploy. * /Someone else/ runs the py3 test suite and if needed sends patches. At least until the py3 version becomes the production version, which may or may not happen until 2019, or until the python core devs make a commitment of supporting a python3 release at least as long as they are supporting python 2.7. Enrico -- GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini signature.asc Description: Digital signature
Re: Python 2, Python 3, Stretch & Buster
On Apr 23, 2015, at 03:19 PM, Enrico Zini wrote: > * I work on the code. > * I run the py2 test suite. > * I deploy. > * /Someone else/ runs the py3 test suite and if needed sends patches. I suggest using tox to ensure the tests run in all supported Python versions. Actually, I suggest tox even if you only support one Python version. tox has become for me an essential tool in the Python testing bag. Cheers, -Barry pgpSD4QTJlmQZ.pgp Description: OpenPGP digital signature
Re: Python 2, Python 3, Stretch & Buster
On Apr 23, 2015, at 03:15 PM, Enrico Zini wrote: >[1] also, given the volatility of a lot of new tools in the python ecosystem, >I have adopted the safety practice of making sure that a tool has been >/widely/ adopted for at least a year or two before even bothering to look at >it. I like that practice: I saved myself the burden of rewriting setup.py >files countless times because of that. We're all volunteers here so we each make our own decision about how and where we spend our time. There *are* however proven, stable tools that improve the Python coding and maintenance experience and for me, tox is one of those. There are others, such as nose2, that I won't even start a new project without adopting from the first commit. I'm happy to help people who are interesting in learning more about these tools, and welcome a discussion with others about what tools and techniques they find must-have for Python development. Cheers, -Barry pgpIreu1smAL5.pgp Description: OpenPGP digital signature
Re: Python 2, Python 3, Stretch & Buster
On Thu, Apr 23, 2015 at 09:36:50AM -0400, Barry Warsaw wrote: > There *are* however proven, stable tools that improve the Python coding and > maintenance experience and for me, tox is one of those. There are others, > such as nose2, that I won't even start a new project without adopting from the > first commit. I'm happy to help people who are interesting in learning more > about these tools, and welcome a discussion with others about what tools and > techniques they find must-have for Python development. I had a quick look at tox. It seems to maintain different virtualenv setup, autoinstalling dependencies via pip, and then running the test suite in all of them. Since tox uses pip, which installs software to be run as my own user skipping the usual Debian trust chain, does it give any guarantee that I won't be running untrusted, unverified code as my user in my machine? Is https://github.com/pypa/pip/issues/425 still relevant? This said, in code that will be deployed in production systems I normally target debian stable and not arbitrary virtualenv setups[1], so for testing dual-python code I would rather be interested in a tool that runs the test suite in the main system, but runs it once per locally installed interpreter. Is there anything that beats a script like this for this purpose? #!/bin/sh nose2-2.7 "$@" nose2-3.4 "$@" Enrico [1] http://www.enricozini.org/2014/debian/debops/ -- GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini signature.asc Description: Digital signature
Re: Python 2, Python 3, Stretch & Buster
On Thu, Apr 23, 2015 at 02:52:33PM +0200, Enrico Zini wrote: > On Mon, Apr 20, 2015 at 11:14:28AM -0400, Paul Tagliamonte wrote: > > > So, round one of all of this is getting the critical path *under* each > > of our services ready, so that when we need to migrate, we don't need > > to scramble. > > There is another constraint that I forgot: the production environment is > Debian Stable, which will soon mean Jessie. If a python2-only module > gets ported to python3 in jessie+1, to use it for debian.org services > the python3 version also needs to be maintained in backports. Yep! So, my plan that I outlined on d-d-a was to basically attack that -- the depends chains -- first. Since we are doing this before strictly needed (that is, nearly two cycles out), we can ensure everything is working and migrated a cycle before we are forced to. This gives us a relaxing pace and also lets us target things pragmatically. You're right, and I'd be happy to help with backporting stuff for services porting to Python 3 as needed, but there's no urgent rush to move everything over today. That's the idea behind porting now, so that everything's totally tight. > For example, I am about to add python-git as a dependency to > nm.debian.org, as I need it to parse the stream of changes that I get > via the keyring.debian.org git changelog, and python-git is python2-only > in Jessie[1]. Having python3-git in jessie+1 won't be enough to make > nm.debian.org deployable with python3: python3-git will also need to be > in jessie-backports, and with a commitment to keep it up to date with > security uploads, because I'm going to run it every night to parse git > logs that are obtained via potentially insecure network connections. Aye. Yep. > Enrico > > [1] I've recently been really sad to figure out that python-git is > python2-only, and so is dulwich. As a personal policy, if I start a > new python project now I start it with python 3.4. I had to start a > new project with python2 because of that, and I felt dirty. Yeah, ditto. I talked with Jelmer about this, there was a patch for Py3, but it fell too far behind development's trunk. If anyone on the py3porters wants to do that, it'd seriously help me out too. Cheers, Paul -- .''`. Paul Tagliamonte | Proud Debian Developer : :' : 4096R / 8F04 9AD8 2C92 066C 7352 D28A 7B58 5B30 807C 2A87 `. `'` http://people.debian.org/~paultag `- http://people.debian.org/~paultag/conduct-statement.txt signature.asc Description: Digital signature
Re: Python 2, Python 3, Stretch & Buster
On Thu, Apr 23, 2015 at 09:17:38AM -0400, Scott Kitterman wrote: > Where we started this was with a goal to make sure that 100% of needed > depends > will be available in Stretch, so that by the time Buster is released, > services > can be migrated. It'd be nice to migrate other things sooner, but getting > the depenencies complete is the essential task for the Stretch cycle for > exactly this reason. I like that, I think it's a really sane and worthwile plan. I look forward to it! <3 Enrico -- GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini signature.asc Description: Digital signature
nose2 vs pytest (was: Python 2, Python 3, Stretch & Buster)
On Apr 23 2015, Barry Warsaw wrote: > There *are* however proven, stable tools that improve the Python coding and > maintenance experience and for me, tox is one of those. There are others, > such as nose2, that I won't even start a new project without adopting from the > first commit. Are you by any chance also familiar with pytest and could same a few word about pros/cons? I've chosen pytest over nose1 because of the richer feature set for writing tests, but I think in several cases pytest is suffering from its own complexity (eg. https://bitbucket.org/pytest-dev/pytest/issue/635/). Reading your comment I was wondering if nose2 might be a better choice, but the nose2 manual wasn't very informative at all (I actually did not found any information about how to write tests). Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« signature.asc Description: PGP signature
Re: Python 2, Python 3, Stretch & Buster
On Thu, Apr 23, 2015 at 10:55:26AM -0400, Paul Tagliamonte wrote: > Yep! So, my plan that I outlined on d-d-a was to basically attack that -- > the depends chains -- first. Since we are doing this before strictly > needed (that is, nearly two cycles out), we can ensure everything is > working and migrated a cycle before we are forced to. > > This gives us a relaxing pace and also lets us target things > pragmatically. You're right, and I'd be happy to help with backporting > stuff for services porting to Python 3 as needed, but there's no urgent > rush to move everything over today. > > That's the idea behind porting now, so that everything's totally tight. I like this plan. I think I've been too acid on this list, and I'm sorry for that. I got defensive and wanted to set clear boundaries on expectations of work on my part that I perceived as currently unreasonable, and I'm afraid of having been a party breaker. I hope that it will succeed, and I hope that it will be possible to have a Stretch release goal that says that all python2 modules in the distribution must have a corresponding python3 version. Enrico -- GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini signature.asc Description: Digital signature
Re: nose2 vs pytest (was: Python 2, Python 3, Stretch & Buster)
On Apr 23, 2015, at 08:05 AM, Nikolaus Rath wrote: >Are you by any chance also familiar with pytest and could same a few >word about pros/cons? Of course this is all personal opinion, so take with a grain of salt. I know a lot of people really like pytest but I don't use it. I don't like the use of assert, and I think the output is a bit too verbose for my tastes. I use stdlib unittest with its plethora of .assert*() methods. Some folks think there's too many of those and they're hard to remember. Usually I find it easy enough to remember the 80% I use all the time, and then I just look up some of the more obscure ones (e.g. assertMultiLineEqual() when I can't remember if the 'L' is capitalized or not ;). What I like about nose2 is its plugin and event architecture for extensibility, which I think is a big improvement over nose (it's not maintenance-only precursor). There nose2 plugins for doing zope.testrunner style layers, which is a resource management technique. There are definitely other options for that (fixtures, etc). I usually carry around a my own plugins to do zope.testrunner style test narrowing, system-wide startups, and per-test diagnostics. E.g. I can do something like: .tox/py34/bin/python -m nose2 -vv -P test_some_specific_thing The thing after the -P (of which multiple can be given) can be any Python regex, and it can match a test method, test class, test module, package path, doctest, or any combination. It really helps me quickly narrow down to any regressions or other failures without wasting time running the full test suite every time. I can also do setups and teardowns (e.g. of a D-Bus service) for the entire test suite, and I can interpose at many levels for doing additional work (e.g. logging) when individual tests start and stop. nose2's plugin/event architecture makes all this really really easy to write. TL;DR: I use stdlib unittest for writing tests, and nose2 as my test runner. >I've chosen pytest over nose1 because of the richer feature set for >writing tests, but I think in several cases pytest is suffering from its >own complexity (eg. https://bitbucket.org/pytest-dev/pytest/issue/635/). >Reading your comment I was wondering if nose2 might be a better choice, >but the nose2 manual wasn't very informative at all (I actually did not >found any information about how to write tests). Right, because you already know how... unittest! :) Nothing against pytest, it just doesn't fit my brain. Cheers, -Barry pgpy1EEGsqJig.pgp Description: OpenPGP digital signature
Re: Python 2, Python 3, Stretch & Buster
On Apr 23, 2015, at 04:18 PM, Enrico Zini wrote: >Since tox uses pip, which installs software to be run as my own user >skipping the usual Debian trust chain, does it give any guarantee that I >won't be running untrusted, unverified code as my user in my machine? In the tox.ini, you should be able to set sitepackages=True and indexserver to point to a nonexistent url (e.g. default=http://missing.example.com). That should force tox to use only system installed packages, avoid pip installing from PyPI, and of course fail if one or more aren't available. You could probably also use the d/rules and pybuild trick of setting http_proxy and https_proxy to the localhost discard port, but that can break some tests (e.g. I have tests that connect to a local test HTTP/S server). Cheers, -Barry pgpf5VjLXdZjf.pgp Description: OpenPGP digital signature
Re: [py3porters-devel] Python 2, Python 3, Stretch & Buster
Hi, I haven't been following this thread too closely, and I'm not familiar with debian packaging related stuff. But I can help write code :) >From what I see, we need people to help port python-git? >From a search on debian packages, this is the package we're talking about: https://packages.debian.org/wheezy/python-git The homepage no longer exist (https://gitorious.org/projects/git-python/), and I think it got rename as GitPython, which exists here ( https://pypi.python.org/pypi/GitPython/1.0.0) I'm not sure how we should proceed, either we can port python-git, or package gitpython up (a search for it on packages.debian.org turned up empty) and change the dependency. Best, Zhi An On Thu, Apr 23, 2015 at 10:55 AM, Paul Tagliamonte wrote: > On Thu, Apr 23, 2015 at 02:52:33PM +0200, Enrico Zini wrote: > > On Mon, Apr 20, 2015 at 11:14:28AM -0400, Paul Tagliamonte wrote: > > > > > So, round one of all of this is getting the critical path *under* each > > > of our services ready, so that when we need to migrate, we don't need > > > to scramble. > > > > There is another constraint that I forgot: the production environment is > > Debian Stable, which will soon mean Jessie. If a python2-only module > > gets ported to python3 in jessie+1, to use it for debian.org services > > the python3 version also needs to be maintained in backports. > > Yep! So, my plan that I outlined on d-d-a was to basically attack that -- > the depends chains -- first. Since we are doing this before strictly > needed (that is, nearly two cycles out), we can ensure everything is > working and migrated a cycle before we are forced to. > > This gives us a relaxing pace and also lets us target things > pragmatically. You're right, and I'd be happy to help with backporting > stuff for services porting to Python 3 as needed, but there's no urgent > rush to move everything over today. > > That's the idea behind porting now, so that everything's totally tight. > > > For example, I am about to add python-git as a dependency to > > nm.debian.org, as I need it to parse the stream of changes that I get > > via the keyring.debian.org git changelog, and python-git is python2-only > > in Jessie[1]. Having python3-git in jessie+1 won't be enough to make > > nm.debian.org deployable with python3: python3-git will also need to be > > in jessie-backports, and with a commitment to keep it up to date with > > security uploads, because I'm going to run it every night to parse git > > logs that are obtained via potentially insecure network connections. > > Aye. Yep. > > > Enrico > > > > [1] I've recently been really sad to figure out that python-git is > > python2-only, and so is dulwich. As a personal policy, if I start a > > new python project now I start it with python 3.4. I had to start a > > new project with python2 because of that, and I felt dirty. > > Yeah, ditto. I talked with Jelmer about this, there was a patch for Py3, > but it fell too far behind development's trunk. If anyone on the > py3porters wants to do that, it'd seriously help me out too. > > Cheers, > Paul > > -- > .''`. Paul Tagliamonte | Proud Debian Developer > : :' : 4096R / 8F04 9AD8 2C92 066C 7352 D28A 7B58 5B30 807C 2A87 > `. `'` http://people.debian.org/~paultag > `- http://people.debian.org/~paultag/conduct-statement.txt > > ___ > py3porters-devel mailing list > py3porters-de...@lists.alioth.debian.org > https://lists.alioth.debian.org/mailman/listinfo/py3porters-devel > >
