Re: [Python-Dev] Automated testing of patches from bugs.python.org
On Tue, May 19, 2015 at 11:37 PM, Terry Reedy wrote: > On 5/19/2015 11:02 AM, Kushal Das wrote: >> >> Hi, >> >> With the help of CentOS project I am happy to announce an automated >> system [1] to test patches from bugs.python.org. This can be fully >> automated >> to test the patches whenever someone uploads a patch in the roundup, but >> for now it accepts IRC commands on #python-dev channel. I worked on a >> docker based prototype during sprints in PyCon. >> >> How to use it? >> --- >> >> 1. Join #python-dev on irc.freenode.net. >> 2. Ask for test privilege from any one of kushal,Taggnostr,bitdancer >> 3. They will issue a simple command. #add: YOUR_NICK_NAME >> 4. You can then test by issuing the following command in the channel: >> >> #test: BUGNUMBER >> like #test: 21271 > > > What if there are multiple patches on the issue? Pick the latest? > This is not correct if someone follows up a patch with a 2.7 backport, or if > there are competing patches. Here is the code that checks how much outstanding patches a certain module has by downloading all patches from open issues, parsing them and comparing the paths. It is possible to reuse the parser to check paths in patch against paths present in certain Python versions, or add different heuristics. https://bitbucket.org/techtonik/python-stdlib All this is pure Python and should work cross-platform too. This was intended to add status for bugs.python.org, but the work on Roundup had stalled due to uncertainty and despair on how to handle utf-8 (internal to Roundup) vs unicode (internal to Jinja2) in this issue: http://issues.roundup-tracker.org/issue2550811 The root of the problem is that Python 2.7 uses 'ascii' and not 'utf-8' internally, so Jinja2 engine fails with 'ascii' ordinal not in range somewhere in the way. Need an expert advice how to handle that, because my brain power is not enough to process it. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 484 (Type Hints) -- penultimate(?) draft
I'm happy to present a much updated PEP 484 for your review. It will (hopefully) appear on python.org within the next hour. I'm also working on an implementation ( https://github.com/ambv/typehinting/tree/master/prototyping) which I hope will be good enough to include in beta 1, assuming the BDFL-Delegate (Mark Shannon) approves. --Guido PEP: 484 Title: Type Hints Version: $Revision$ Last-Modified: $Date$ Author: Guido van Rossum , Jukka Lehtosalo < [email protected]>, Łukasz Langa BDFL-Delegate: Mark Shannon Discussions-To: Python-Dev Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 29-Sep-2014 Post-History: 16-Jan-2015,20-Mar-2015,17-Apr-2015,20-May-2015 Resolution: Abstract PEP 3107 introduced syntax for function annotations, but the semantics were deliberately left undefined. There has now been enough 3rd party usage for static type analysis that the community would benefit from a standard vocabulary and baseline tools within the standard library. This PEP introduces a provisional module to provide these standard definitions and tools, along with some conventions for situations where annotations are not available. Note that this PEP still explicitly does NOT prevent other uses of annotations, nor does it require (or forbid) any particular processing of annotations, even when they conform to this specification. It simply enables better coordination, as PEP 333 did for web frameworks. For example, here is a simple function whose argument and return type are declared in the annotations:: def greeting(name: str) -> str: return 'Hello ' + name While these annotations are available at runtime through the usual ``__annotations__`` attribute, *no type checking happens at runtime*. Instead, the proposal assumes the existence of a separate off-line type checker which users can run over their source code voluntarily. Essentially, such a type checker acts as a very powerful linter. (While it would of course be possible for individual users to employ a similar checker at run time for Design By Contract enforcement or JIT optimization, those tools are not yet as mature.) The proposal is strongly inspired by mypy [mypy]_. For example, the type "sequence of integers" can be written as ``Sequence[int]``. The square brackets mean that no new syntax needs to be added to the language. The example here uses a custom type ``Sequence``, imported from a pure-Python module ``typing``. The ``Sequence[int]`` notation works at runtime by implementing ``__getitem__()`` in the metaclass (but its significance is primarily to an offline type checker). The type system supports unions, generic types, and a special type named ``Any`` which is consistent with (i.e. assignable to and from) all types. This latter feature is taken from the idea of gradual typing. Gradual typing and the full type system are explained in PEP 483. Other approaches from which we have borrowed or to which ours can be compared and contrasted are described in PEP 482. Rationale and Goals === PEP 3107 added support for arbitrary annotations on parts of a function definition. Although no meaning was assigned to annotations then, there has always been an implicit goal to use them for type hinting [gvr-artima]_, which is listed as the first possible use case in said PEP. This PEP aims to provide a standard syntax for type annotations, opening up Python code to easier static analysis and refactoring, potential runtime type checking, and (perhaps, in some contexts) code generation utilizing type information. Of these goals, static analysis is the most important. This includes support for off-line type checkers such as mypy, as well as providing a standard notation that can be used by IDEs for code completion and refactoring. Non-goals - While the proposed typing module will contain some building blocks for runtime type checking -- in particular a useful ``isinstance()`` implementation -- third party packages would have to be developed to implement specific runtime type checking functionality, for example using decorators or metaclasses. Using type hints for performance optimizations is left as an exercise for the reader. It should also be emphasized that **Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.** The meaning of annotations == Any function without annotations should be treated as having the most general type possible, or ignored, by any type checker. Functions with the ``@no_type_check`` decorator or with a ``# type: ignore`` comment should be treated as having no annotations. It is recommended but not required that checked functions have annotations for all arguments and the return type. For a checked function, the default annotation for arguments and for the return type is ``Any``. An exception is that the first argument of instance and class met
Re: [Python-Dev] Automated testing of patches from bugs.python.org
On 5/20/2015 3:44 AM, anatoly techtonik wrote: This was intended to add status for bugs.python.org, but the work on Roundup had stalled due to uncertainty and despair on how to handle utf-8 (internal to Roundup) vs unicode (internal to Jinja2) in this issue: http://issues.roundup-tracker.org/issue2550811 The root of the problem is that Python 2.7 uses 'ascii' and not 'utf-8' internally, so Jinja2 engine fails with 'ascii' ordinal not in range somewhere in the way. Need an expert advice how to handle that, because my brain power is not enough to process it. In my view, the root of the problem is using Python 2 and working with encoded bytes. The fix is to upgrade to Python 3.3+, with an improved text model (unicode str class), and work with text. Follow the standard process: decode encoded bytes to text when received from browsers, work internally with text, and encode output just before sending to browsers. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Automated testing of patches from bugs.python.org
On 5/19/15, Terry Reedy wrote: > On 5/19/2015 11:02 AM, Kushal Das wrote: >> Hi, >> Hi ! I'm not very familiar with python-dev development workflows . Nonetheless I just wanted to mention something that proved to be useful for me in the past . >> With the help of CentOS project I am happy to announce an automated >> system [1] to test patches from bugs.python.org. This can be fully >> automated >> to test the patches whenever someone uploads a patch in the roundup, but >> for now it accepts IRC commands on #python-dev channel. I worked on a >> docker based prototype during sprints in PyCon. >> >> How to use it? >> --- >> >> 1. Join #python-dev on irc.freenode.net. >> 2. Ask for test privilege from any one of kushal,Taggnostr,bitdancer >> 3. They will issue a simple command. #add: YOUR_NICK_NAME >> 4. You can then test by issuing the following command in the channel: >> >> #test: BUGNUMBER >> like #test: 21271 > > What if there are multiple patches on the issue? Pick the latest? > This is not correct if someone follows up a patch with a 2.7 backport, > or if there are competing patches. > [...] It is a fact that running automated tests for patches is a really useful feature . Nevertheless , IMHO for this to succeed at large scale there is a need to manage the content of patches themselves , the base version they were built upon , as well as their order should they be stacked . My suggestion for you therefore is to use Hg patch repositories [1]_ as the starting point for your patch CI system . Some of the benefits I could mention : - triggering (patch) builds on commit via web hooks - CI infrastructure needed turns out to be very similar to the one setup for the main project - Commands to work on patch queue repositories are easy to learn - The possibility of editing series file is also useful for ignoring some patches without removing their contents . - halt if patch cannot be applied upon latest version * ... but still be able to see it in action by checking out the right version of the code base used to build it in first place . - try the patch against different versions of the code base as it evolves - fuzzy refresh - version control for patches - multiple branches * which may be bound to tickets in many ways e.g. naming conventions * ... particularly useful for competing patches . There are a few black spots too . Patch repositories deal with the diff of a diff , hence some operations applied upon patches (e.g. merging) might be quite messy , Most of the time this is no big deal though . The following are repositories I used while developing Apache Bloodhound , during incubation and after it became a TLP . I'm including them to illustrate branching and naming conventions (I used) to keep track of tickets . https://bitbucket.org/olemis/bloodhound-incubator-mq/ https://bitbucket.org/olemis/bloodhound-mq HTH , since this way the workflow would be tightly integrated with Mercurial , as highlighted by Berker Peksağ in previous messages . .. [1] http://mercurial.selenic.com/wiki/MqTutorial -- Regards, Olemis - @olemislc Apache™ Bloodhound contributor http://issues.apache.org/bloodhound http://blood-hound.net Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Enable access to the AST for Python code
Hi Python devs,
Enabling access to the AST for compiled code would make some cool
things possible (C# LINQ-style ORMs, for example), and not knowing too
much about this part of Python internals, I'm wondering how possible
and practical this would be.
Context: PonyORM (http://ponyorm.com/) allows you to write regular
Python generator expressions like this:
select(c for c in Customer if sum(c.orders.price) > 1000)
which compile into and run SQL like this:
SELECT "c"."id"
FROM "Customer" "c"
LEFT JOIN "Order" "order-1" ON "c"."id" = "order-1"."customer"
GROUP BY "c"."id"
HAVING coalesce(SUM("order-1"."total_price"), 0) > 1000
I think the Pythonic syntax here is beautiful. But the tricks PonyORM
has to go to get it are ... not quite so beautiful. Because the AST is
not available, PonyORM decompiles Python bytecode into an AST first,
and then converts that to SQL. (More details on all that from author's
EuroPython talk at http://pyvideo.org/video/2968)
I believe PonyORM needs the AST just for generator expressions and
lambda functions, but obviously if this kind of AST access feature
were in Python it'd probably be more general.
I believe C#'s LINQ provides something similar, where if you're
developing a LINQ converter library (say LINQ to SQL), you essentially
get the AST of the code ("expression tree") and the library can do
what it wants with that.
What would it take to enable this kind of AST access in Python? Is it
possible? Is it a good idea?
-Ben
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable access to the AST for Python code
Hey Ben, this is probably a better topic for python-ideas. I'll warn you
that a hurdle for ideas like this is that ideally you don't want to support
this just for CPython. It's definitely cool though! (Using movie poster
style quotes you can turn this into a ringing endorsement: "definitely
cool" -- The BDFL. :-)
On Wed, May 20, 2015 at 7:26 PM, Ben Hoyt wrote:
> Hi Python devs,
>
> Enabling access to the AST for compiled code would make some cool
> things possible (C# LINQ-style ORMs, for example), and not knowing too
> much about this part of Python internals, I'm wondering how possible
> and practical this would be.
>
> Context: PonyORM (http://ponyorm.com/) allows you to write regular
> Python generator expressions like this:
>
> select(c for c in Customer if sum(c.orders.price) > 1000)
>
> which compile into and run SQL like this:
>
> SELECT "c"."id"
> FROM "Customer" "c"
> LEFT JOIN "Order" "order-1" ON "c"."id" = "order-1"."customer"
> GROUP BY "c"."id"
> HAVING coalesce(SUM("order-1"."total_price"), 0) > 1000
>
> I think the Pythonic syntax here is beautiful. But the tricks PonyORM
> has to go to get it are ... not quite so beautiful. Because the AST is
> not available, PonyORM decompiles Python bytecode into an AST first,
> and then converts that to SQL. (More details on all that from author's
> EuroPython talk at http://pyvideo.org/video/2968)
>
> I believe PonyORM needs the AST just for generator expressions and
> lambda functions, but obviously if this kind of AST access feature
> were in Python it'd probably be more general.
>
> I believe C#'s LINQ provides something similar, where if you're
> developing a LINQ converter library (say LINQ to SQL), you essentially
> get the AST of the code ("expression tree") and the library can do
> what it wants with that.
>
> What would it take to enable this kind of AST access in Python? Is it
> possible? Is it a good idea?
>
> -Ben
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable access to the AST for Python code
On 21 May 2015 at 12:31, Guido van Rossum wrote: > Hey Ben, this is probably a better topic for python-ideas. I'll warn you > that a hurdle for ideas like this is that ideally you don't want to support > this just for CPython. It's definitely cool though! (Using movie poster > style quotes you can turn this into a ringing endorsement: "definitely cool" > -- The BDFL. :-) Agreed this is python-ideas territory, but yes there's definitely interest in being able to mark a section of code as being compiled to an AST object at compile time, and then further processed at runtime (essentially having syntax to switch on PyCF_ONLY_AST for a subexpression and/or entire statement). At the moment, you can do this all through the ast module and the PyCF_ONLY_AST flag to compile(), but you need to pass the code to be compiled around as strings, which tends to somewhat user (and IDE!) unfriendly. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable access to the AST for Python code
Guido van Rossum wrote: Hey Ben, this is probably a better topic for python-ideas. I'll warn you that a hurdle for ideas like this is that ideally you don't want to support this just for CPython. It's definitely cool though! This would effectively be a macro system. I thought your position on macros was that they're uncool? If you've changed your mind about this, that's cool too -- just checking. -- Greg ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
