[Python-Dev] Cipher implementation (such as AES) in standard library
Dear all, Have we tried cipher implementation includes AES as a standard library in the past? https://docs.python.org/3.6/library/crypto.html if possible I want to try to implement AES because famous 3rd party library is not maintained and general cipher programs should be used for multiple purpose.Though the implementation is tough, I believe this should be worth to it. In my case, I want to use AES implementation for zipfile module. Thanks and Regards, --- Takahiro Ono ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] AES cipher implementation in standard library
Dear all, Have we tried cipher implementation includes AES as a standard library in the past? https://docs.python.org/3.6/library/crypto.html if possible I want to try to implement AES because famous 3rd party library is not maintained and general cipher programs should be used for multiple purpose.Though the implementation is tough, I believe this should be worth to it. In my case, I want to use AES implementation for zipfile module. Thanks and Regards, --- Takahiro Ono ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Heap-allocated StructSequences
PEP-384 talks about defining a Stable ABI by making PyTypeObject opaque. Thus,
banning the use of static PyTypeObjects.
Specifically, I’d like to focus on those static PyTypeObjects that are
initialized as StructSequences. As of today, there are 14 instances of these
types (in timemodule.c, posixmodule.c, etc.) within cpython/Modules. These are
all initialized through PyStructSequence_InitType2. This is very problematic
when trying to make these types conform to PEP-384 as they are used through
static PyTypeObjects.
Problems:
* PyStructSequence_InitType2 overrides the PyTypeObject:
This C-API does a direct memcpy from a “prototype” structure. This effectively
overrides anything set within the PyTypeObject. For example, if we were to
initialize a heap allocated PyTypeObject and pass it on to this function, the
C-API would just get rid of the Py_TPFLAG_HEAPTYPE flag, causing issues with
the GC.
* PyStructSequence_InitType2 does not work with heap allocated
PyTypeObjects:
Even if the function is fixed to preserve the state of the PyTypeObject and
only overriding the specific slots (i.e. tp_new, tp_repr, etc.), it is expected
that PyStructSequence_InitType2 will call PyType_Ready on the object. That
means that the incoming object shouldn’t be initialized by a function such as
PyType_FromSpec, as that would have already called PyType_Ready on it.
Therefore, PyStructSequence_InitType2 will now have the responsibility of
setting all the slots and properties of the PyHeapTypeObject, which is not
feasible.
* PyStructSequence_NewType is non-functional:
This function was meant to be used as a way of creating a heap-allocated
PyTypeObject that be passed to PyStructSequence_InitType2, effectively
returning a heap allocated PyTypeObject. The current implementation doesn’t
work in practice. Given that this struct is created in the heap, the GC has
control over it. Thus, when the GC tries to traverse the type it complains
with: “Error: type_traverse() called for non-heap type”, since it doesn’t have
the Py_TPFLAG_HEAPTYPE flag. If we add the flag, we run into bullet point 1, if
we are able to preserve the flag then we will still run into the problem of
bullet point 2. Extra note: This C-API is not being used anywhere within
CPython itself.
Solution:
* Fix the implementation of PyStructSequence_NewType:
The best solution would be to fix the implementation of this function. This can
easily be done by dynamically creating a PyType_Spec and calling PyType_FromSpec
```
PyObject*
PyStructSequence_NewType(PyStructSequence_Desc *desc)
{
// …
PyType_Spec* spec = PyMem_NEW(PyType_Spec, 1);
spec->name = desc->name;
spec->basicsize = sizeof(PyStructSequence) - sizeof(PyObject *);
spec->itemsize = sizeof(PyObject *);
spec->flags = Py_TPFLAGS_DEFAULT;
spec->slots = PyMem_NEW(PyType_Slot, 6);
spec->slots[0].slot = Py_tp_dealloc;
spec->slots[0].pfunc = (destructor)structseq_dealloc;
// …
bases = PyTuple_Pack(1, &PyTuple_Type);
type = PyType_FromSpecWithBases(spec, bases);
// …
```
This will cleanly create a heap allocated PyStructSequence which can be used
just like any stack allocated PyTypeObject initialized through
PyStructSequence_InitType2. This means that any C-Extension should be using
PyStructSequence_NewType and only built-in types should be calling
PyStructSequence_InitType2. This will enable these types to comply with PEP-384
As an extra, I already have patches for this proposal. They can be found here:
Branch: https://github.com/eduardo-elizondo/cpython/tree/heap-structseq
Reimplement PyStructSequence_NewType:
https://github.com/eduardo-elizondo/cpython/commit/413f8ca5bc008d84b3397ca1c9565c604d54b661
Patch timemodule with NewType:
https://github.com/eduardo-elizondo/cpython/commit/0a35ea263a531cb03c06be9efc9e96d68162b308
Thoughts?
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use of Cython
Hi Stefan,
On Sat, Sep 1, 2018 at 6:12 PM Stefan Behnel wrote:
>
> Yury,
>
> given that people are starting to quote enthusiastically the comments you
> made below, let me set a couple of things straight.
To everyone reading this thread please keep in mind that I'm not in
position to "defend" mypyc or to "promote" it, and I'm not affiliated
with the project at all. I am just excited about yet another tool to
statically compile Python and I'm discussing it only from a
theoretical standpoint.
>
> Yury Selivanov schrieb am 07.08.2018 um 19:34:
> > On Mon, Aug 6, 2018 at 11:49 AM Ronald Oussoren via Python-Dev wrote:
> >
> >> I have no strong opinion on using Cython for tests or in the stdlib, other
> >> than that it is a fairly large dependency. I do think that adding a
> >> “Cython-lite” tool the CPython distribution would be less ideal, creating
> >> and maintaining that tool would be a lot of work without clear benefits
> >> over just using Cython.
> >
> > Speaking of which, Dropbox is working on a new compiler they call "mypyc".
> >
> > mypyc will compile type-annotated Python code to an optimized C.
>
> That's their plan. Saying that "it will" is a bit premature at this point.
> The list of failed attempts at writing static Python compilers is rather
> long, even if you only count those that compile the usual "easy subset" of
> Python.
>
> I wish them the best of luck and endurance, but they have a long way to go.
I fully agree with you here.
>
>
> > The
> > first goal is to compile mypy with it to make it faster, so I hope
> > that the project will be completed.
>
> That's not "the first goal". It's the /only/ goal. The only intention of
> mypyc is to be able to compile and optimise enough of Python to speed up
> the kind or style of code that mypy uses.
>
>
> > Essentially, mypyc will be similar
> > to Cython, but mypyc is a *subset of Python*, not a superset.
>
> Which is bad, right? It means that there will be many things that simply
> don't work, and that you need to change your code in order to make it
> compile at all. Cython is way beyond that point by now. Even RPython will
> probably continue to be way better than mypyc for quite a while, maybe
> forever, who knows.
To be clear I'm not involved with mypyc, but my understanding is that
the entire Python syntax will be supported, except some dynamic
features like patching `globals()`, `locals()`, or classes, or
__class__. IMO this is *good* and in general Python programs don't do
that anyways.
>
>
> > Interfacing with C libraries can be easily achieved with cffi.
>
> Except that it will be fairly slow. cffi is not designed for static
> analysis but for runtime operations.
Could you please clarify this point? My current understanding is that
you can build a static compiler with a knowledge about cffi so that it
can compile calls like `ffi.new("something_t[]", 80)` to pure C.
> You can obviously also use cffi from
> Cython – but then, why would you, if you can get much faster code much more
> easily without using cffi?
The "much more easily" part is debatable here and is highly
subjective. For me using Cython is also easier *at this point*
because I've spent so much time working with it. Although getting
there wasn't easy for me :(
>
> That being said, if someone wants to write a static cffi optimiser for
> Cython, why not, I'd be happy to help with my advice. The cool thing is
> that this can be improved gradually, because compiling the cffi code
> probably already works out of the box. It's just not (much) faster than
> when interpreted.
Yeah, statically compiling cffi-enabled code is probably the way to go
for mypyc and Cython.
>
>
> > Being a
> > strict subset of Python means that mypyc code will execute just fine
> > in PyPy.
>
> So does normal (non-subset) Python code. You can run it in PyPy, have
> CPython interpret it, or compile it with Cython if you want it to run
> faster in CPython, all without having to limit yourself to a subset of
> Python. Seriously, you make this sound like requiring users to rewrite
> their code to make it compilable with mypyc was a good thing.
But that's the point: unless you add Cython types to your Python code
it gets only moderate speedups. Using Cython/C types usually means
that you need to use pxd/pyx files which means that the code isn't
Python anymore. I know that Cython has a mode to use decorators in
pure Python code to annotate types, but they are less intuitive than
using typing annotations in 3.6+.
[..]
> > I'd be more willing to start using mypyc+cffi in CPython stdlib
> > *eventually*, than Cython now. Cython is a relatively complex and
> > still poorly documented language.
>
> You are free to improve the documentation or otherwise help us find and
> discuss concrete problems with it.
Fair point.
> Calling Cython a "poorly documented
> language" could easily feel offensive towards those who have put a lot of
> work into the documentation, wiki, tutorials, trainings and
Re: [Python-Dev] AES cipher implementation in standard library
On 2018-09-04 16:37, 大野隆弘 wrote: > Dear all, > > Have we tried cipher implementation includes AES as a standard library > in the past? > https://docs.python.org/3.6/library/crypto.html > > if possible I want to try to implement AES because famous 3rd party > library is not maintained and general cipher programs should be used for > multiple purpose.Though the implementation is tough, I believe this > should be worth to it. > In my case, I want to use AES implementation for zipfile module. strong -1 The Python standard library doesn't contain any encryption, signing, and other cryptographic algorithms for multiple reasons. The only exception from the rule are hashing algorithms and HMAC construct. There are legal implications like export restrictions. Crypto is just too hard to get right and we don't want to give the user additional rope. We already had a very lengthy and exhausting discussion for the secrets module. That module just provides a user-friendly interface to CPRNG. By the way, AES by itself is a useless to borderline dangerous algorithm. It must be embedded within additional layers like block mode, authenticated encryption / MAC, and more. There isn't a single correct answer for block mode and AD algorithm, too. It highly depends on the problem space. While GCM AEAD mode is good choice for network communication, it can be a pretty bad idea for persistent storage. There is one excellent Python library with high level and low level cryptographic algorithms: http://cryptography.readthedocs.io/ . It's t Regards, Christian ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use of Cython
Yury Selivanov schrieb am 04.09.2018 um 18:19:
> On Sat, Sep 1, 2018 at 6:12 PM Stefan Behnel wrote:
>> Yury Selivanov schrieb am 07.08.2018 um 19:34:
>>> The first goal is to compile mypy with it to make it faster, so I hope
>>> that the project will be completed.
>>
>> That's not "the first goal". It's the /only/ goal. The only intention of
>> mypyc is to be able to compile and optimise enough of Python to speed up
>> the kind or style of code that mypy uses.
>>
>>> Essentially, mypyc will be similar
>>> to Cython, but mypyc is a *subset of Python*, not a superset.
>>
>> Which is bad, right? It means that there will be many things that simply
>> don't work, and that you need to change your code in order to make it
>> compile at all. Cython is way beyond that point by now. Even RPython will
>> probably continue to be way better than mypyc for quite a while, maybe
>> forever, who knows.
>
> To be clear I'm not involved with mypyc, but my understanding is that
> the entire Python syntax will be supported, except some dynamic
> features like patching `globals()`, `locals()`, or classes, or
> __class__.
No, that's not the goal, at least from what I understood from my
discussions with Jukka. The goal is to make it compile mypy, be it by
supporting Python features in mypyc or by avoiding Python features in mypy.
I'm sure they will take any shortcut they can in order to avoid having to
make mypyc too capable, because mypyc is not more than a means to an end.
For example, they may easily get away without supporting generators and
closures, which are quite difficult to implement in C. But finding a
non-trivial piece of Python code out there that uses neither of the two is
probably not easy.
I'm also sure they will avoid Python semantics wherever they can, because
implementing them in the same way as CPython and Cython would mean that
certain constructs cannot safely be statically reasoned about, and thus
cannot be optimised. Avoiding (full) Python semantics relieves you from
these restrictions, and if you control both sides, the compiler and the
code that it compiles, then it becomes much easier to apply arbitrary
optimisations at will.
IMHO, what they are implementing is much closer to ShedSkin than to Cython.
>>> Interfacing with C libraries can be easily achieved with cffi.
>>
>> Except that it will be fairly slow. cffi is not designed for static
>> analysis but for runtime operations.
>
> Could you please clarify this point? My current understanding is that
> you can build a static compiler with a knowledge about cffi so that it
> can compile calls like `ffi.new("something_t[]", 80)` to pure C.
I'm sure there is a relatively large subset of cffi's API that could be
compiled statically, as long as the declartions and their usage are kept
simple and fully visible to the compiler. What that subset is remains to be
seen once someone actually tries to do it.
> Yeah, statically compiling cffi-enabled code is probably the way to go
> for mypyc and Cython.
I doubt it, given the expected restrictions and verbosity. But debating
this is useless as long as no-one attempts to actually write a static
compiler for cffi(-like) code.
> Using Cython/C types usually means
> that you need to use pxd/pyx files which means that the code isn't
> Python anymore.
I'm aware that this is a very common misconception that is difficult to get
out of people's heads. You probably got this idea from wrapping a native
library, in which case the only choice you have in order to declare an
external C-API is really to use Cython's special syntax. However, this
would not apply to most use cases in the CPython project context, and it
also does not necessarily apply to most of the code in a Cython module even
if it uses external libraries.
Cython has four ways to provide type declarations: cdef statements in
Cython code, external .pxd files for Python or Cython files, special
decorators and declaration functions, and PEP-484/526 type annotations.
All four have their use cases (e.g. syntax support in different Python
versions, efficiency of expression, readability for people with different
backgrounds, etc.), and all but the first allow users to keep their module
code in Python syntax. As long as you do not call into external native
code, it's your choice which of these you prefer for your code base,
project context and developer background. You can even mix them at will, if
you feel like it.
> I know that Cython has a mode to use decorators in
> pure Python code to annotate types, but they are less intuitive than
> using typing annotations in 3.6+.
You can use PEP-484/526 type annotations to declare Cython types in Python
code that you intend to compile. It's entirely up to you, and it's an
entirely subjective measure which "is better". Many people prefer Cython's
non-Python syntax because it allows them to apply their existing C
knowledge. For them, PEP-484 annotations may easily be non-intuitive in
comparison.
> For CPython
Re: [Python-Dev] Use of Cython
On 09/04/2018 11:55 AM, Stefan Behnel wrote: Adding the right language would lower the bar, IMHO. Cython is Python. It allows users with a Python background to implement C things without having to thoroughly learn C/and/ the CPython C-API first. So, the way I see it, rather than/adding/ a "third" language to the mix, it substantially lowers the entry level from the current two and a half languages (Python + C + C-API) to one and a half (Python + Cython). As somebody who only has light exposure to C, I would very much like to have Cython be an option. -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use of Cython
On Tue, Sep 4, 2018 at 2:58 PM Stefan Behnel wrote: [..] > Cython has four ways to provide type declarations: cdef statements in > Cython code, external .pxd files for Python or Cython files, special > decorators and declaration functions, and PEP-484/526 type annotations. Great to hear that PEP 484 type annotations are supported. Here's a link to the docs: https://cython.readthedocs.io/en/latest/src/tutorial/pure.html#static-typing [..] > > I know that Cython has a mode to use decorators in > > pure Python code to annotate types, but they are less intuitive than > > using typing annotations in 3.6+. > > You can use PEP-484/526 type annotations to declare Cython types in Python > code that you intend to compile. It's entirely up to you, and it's an > entirely subjective measure which "is better". Many people prefer Cython's > non-Python syntax because it allows them to apply their existing C > knowledge. For them, PEP-484 annotations may easily be non-intuitive in > comparison. Yeah, but if we decide to use Cython in CPython we probably need to come up with something like PEP 7 to recommend one particular style and have an overall guideline. Using PEP 484 annotations means that we have pure Python code that PyPy and other interpreters can still run. [..] > > I'd be +0.5 on using Cython (optionally?) to compile some pure Python > > code to make it 30-50% faster. asyncio, for instance, would certainly > > benefit from that. > > Since most of this (stdlib) Python code doesn't need to stay syntax > compatible with Python < 3.6 (actually 3.8) anymore, you can probably get > much higher speedups than that by statically typing some variables and > functions here and there. I recently tried that with difflib, makes a big > difference. I'd be willing to try this in asyncio if we start using Cython. Yury ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
