Re: [Twisted-Python] Request for help with Twisted bindings in M2Crypto

[Daniel Sank]

Glyph,

>> 2. I have no idea what a "task" is. I realize this is python and yay
duck-typing but not
>> specifying the expected behavior of an argument seems like a big
omission.

> Did you miss the part where it said "type: 0-argument callable" in the
documentation?

Yes :(

I can only guess that I missed it because the type is defined near the end
of the description line, whereas I'm used to seeing

def foo(x, y):
"""Do something.

Args:
x (int): blah blah
y (banana): yadda yadda
"""

FWIW, now that I look at the code, the type specification is way more
visually apparent there than it is in the generated HTML.

tl,dr: I take it all back and thanks for pointing out the obvious.

> An interface is a very simple concept - an abstract description of what
an object is expected
> to provide if you're going to do something useful with it.

Indeed, a general understanding of interfaces is not the problem.

> In my view, interface definition is the primary activity of software
development

Agreed 100%.

> The fact that so many people seem to find either the basic idea of an
abstract type, or the concrete
> instantiation of that idea in the Zope Interface library, so horribly
confusing, makes me despair of
> ever communicating the actually *hard* stuff that Twisted gets up to in
its internals.

> I would very much like to understand *what* is so confusing about
"interfaces".  Is it, as Cory posited,
> just that the documentation is not properly linked?  Or is it that the
average Python developer needs
> a gentle introduction to the entire idea of abstract rather than concrete
types?  If they do - is it really
> Twisted's responsibility to provide it to them?  Should Zope Interface
just have a snazzier website?

Some years ago when I tried to understand Twisted's use of interfaces via
Twisted's own documentation (which included something about hair dryers and
voltage standards) I was puzzled by the fact that the examples didn't
really show me how to solve a useful problem (or I was too stupid to
understand that the examples did in fact do that) *despite the fact that I
knew what an interface was in general terms*. It was a case of
understanding the intent but none of the examples. A brief look at the zope
documentation just now makes me think the situation has improved.

The other problem was that interfaces were sprinkled somewhat haphazardly
around the code I was trying to understand (perspective broker) and it was
just plain hard to keep navigating around the code files to understand who
was implementing what interfaces. This could have been my own fault for not
having a editor set up. I don't know.

> Should Zope Interface just have a snazzier website?

I think the real issue is the need for compelling and simple examples.

- Daniel


P.S. Everything below here is completely off topic of this thread and I
probably shouldn't have written it.

> Given that Twisted is often translating network protocol data into Python
method calls, one needs both
> a working domain knowledge of the protocol involved and a robust
understanding of Python
> metaprogramming constructs.  It sounds here like where you fell down was
mostly in the "Python
> metaprogramming" area, where PB is especially intense.

> But it's also not really specific to Twisted either.  This is another
case where I'm not sure what to do
> except to refer people to the language reference and tell them to work
through it slowly.

> Unfortunately, framework code just looks like that

I spent a considerable amount of time reading the PB code, reproducing
parts of it myself, and talking to people on IRC and the mailing list to
understand a particularly weird issue in PB. See here for the bug I was
trying to fix (note in particular my first comment to the one existing
answer):

http://stackoverflow.com/questions/23421423/why-are-dummy-objects-created-in-twisteds-pb-system

I distinctly recall that near the end of my efforts you (Glyph) or someone
else more or less told me that the PB code was old, horrible, and that the
issues I was trying to understand were probably incidental complexity due
to poor design etc. You guys were joking around on IRC about how ridiculous
all the dummy object construction is. So, I think this *particular*
incident was
less due a lack of understanding of python metaprogramming and more due to
PB having some bizarre warts.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Request for help with Twisted bindings in M2Crypto

[Glyph Lefkowitz]

> On Jul 25, 2016, at 02:03, Matěj Cepl  wrote:
> 
> 3) Moreover, I would like to know how much interest there is in
> maintaining the M2Crypto module for Twisted. I got some hope from
> http://twistedmatrix.com/trac/wiki/TransportLayerSecurity 
>  which seems
> like there is an interest in more complete OpenSSL bindings,

That is a very old wiki page.  I will delete it to avoid confusing people in 
the future!  Not only has pyOpenSSL had complete enough bindings to implement 
the feature described on that wiki page for several years now (a Twisted 
developer, Jean-Paul Calderone, actually took over maintenance of pyOpenSSL 
expressly for the purpose of adding those APIs), we actually implemented the 
TLS implementation based on those APIs 
> in 
2011, and fully finished transitioning to that new API in 2014 

 when we deleted the older, SSLSocket based API.

> but OTOH I see on the list that Twisted now seems to use more and more of
> Cryptography (why in the world somebody made such confusing name of
> their project ...).

The choice of name is intentional: it is designed to convey a sense of 
authoritativeness.  I.e. if you need cryptography in Python, you should 'import 
cryptography', and ignore everything else.  The Cryptography project 
specifically calls out M2Crypto, PyCrypto, and PyOpenSSL as having problems and 
lacking maintenance: 
>.  
pyOpenSSL is now just a thin wrapper over Cryptography itself, and PyCrypto's 
maintainer now generally suggests Cryptography 
>.  So it 
is 2/3 of the way to achieving its goal of eliminating these libraries which 
duplicate so much effort - M2Crypto is all that remains :).

> Obviously the most simple way for me to be cutting
> Twisted module from M2Crypto and let it be (although I am afraid I have
> still some legacy users who would like to see it maintained, and given
> that the legacy support is still the most important reason for
> maintaining M2Crypto, I don't want to give up lightly).

I do not want to denigrate the work you've done maintaining a legacy library.  
I think it's noble to take on this kind of work.  But if you don't have any 
particular reason for needing to maintain this library beyond "legacy support", 
and it is not different from Cryptography in any meaningful way, the best thing 
that you could do for its existing users would be to do the same thing that was 
done with pyOpenSSL: make it a thin wrapper over the bindings layer in 
Cryptography, get rid of all of the SWIG code in M2Crypto, and start gently 
directing users in the direction of Cryptography for any new code.  This would 
get everyone onto a supported base platform for their security primitives, 
allow them to share code with other parts of large systems that already use 
Cryptography or pyOpenSSL, and provide a well-maintained path forward.

M2Crypto's main claim to superiority in past years was its higher degree of 
completeness of OpenSSL APIs, but Cryptography has since far surpassed it.  
Even if there are some APIs that Cryptography's bindings layer doesn't export, 
it's easier and safer to add more bindings there than in M2Crypto.

Even if you're not going to get rid of M2Crypto entirely, M2Crypto's 
implementation of Twisted TLS copies the terrible no-security defaults it 
inherits from OpenSSL, the same ones that Twisted had 5 years ago, and Twisted 
has moved on to have actual security (certificate verification, service 
identity, and trust root configuration).  We also continue to improve that 
security regularly.  Given all that, this is a rare case where I would not mind 
seeing Twisted support disappear from a library.  Unless I were to get some new 
information I would have to very strongly discourage anyone who wanted to use 
the SSL backend in M2Crypto.  I realize you have users, but possibly the best 
thing you could do for their own good would be to force them to move to 
Twisted's much better native TLS support, which thankfully is easy to adopt at 
this point.

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] RFC: IPv6 multicast join/ticket 6597

[Glyph Lefkowitz]

> On Jul 25, 2016, at 9:43 PM, Jason Litzinger  wrote:
> 
> 
> Hello,
> 
> I'm looking at making the changes to support IPv6 multicast groups as 
> described
> in ticket 6597 but wanted to get some feedback (and get a feel whether this is
> even desirable) before formally submitting any patches.

Thanks for taking this up!

> Specifically:
> 
> 1.  I've read [1] and it alludes to udp hopefully disappearing, is that 
> something in the
>works?  Is there a new approach to solving this problem I should look at?  
> A
>branch in the works where this (conceptual) change belongs?
> 
>Note:  The addressFamily attribute referenced already exists and is set
>   properly for IPv6.

'twisted.internet.udp', as an importable module; not 'udp' as a feature of 
Twisted (or of the Internet, for that matter).  

> 2.  The attached change has the side effect that calls to 
> ReactorBase.resolve()
>with IPv6 literals will now likely succeed where they may have failed in 
> the
>past.  That means clients counting on resolve raising an exception for an
>IPv6 literal will break.  Not sure whether this is considered a
>compatibility issue, but I wanted to raise it.

We have explicitly avoided adding IPv6 name resolution to the reactor because 
the reactor's API for name resolution is fundamentally the wrong shape for 
IPv6.  If you want to add the ability to resolve IPv6 names to the reactor 
itself, please see this ticket: https://twistedmatrix.com/trac/ticket/4362 


For the purposes of this ticket alone, you should probably just skip resolution 
in _joinAddr1 if resolution is 

> 3.  One alternative to the above would be a complete API separation, via
>something like joinIPv6Group(), and a new resolve.  Is that more appealing
>in this case?

Given what we've done with connectTCP et. al., it makes sense to leave 
'joinGroup' as the API for doing this.  But we probably want to leave 
'.resolve' alone.

> Caveats:
> 
> 1.  I have not finished all of the documentation related to developers, I will
>do so prior to formal submission.

I think we can do the narrative docs in a separate PR, as the interface looks 
like the straightforward expansion of the IPv4 interface.  You should clean up 
the reference documentation (i.e. docstrings) to ensure they're accurate of 
course.

> 2.  I know I need tests and docs and will submit them with the final changes.

Testing multicast is ... challenging.  I barely have any idea how to set up a 
test environment for IPv4, and no idea what to do for IPv6.  If you can speak 
to this in your tests (and hopefully docs as well) that would be super helpful.

> On to the patches.  With these changes, I can use the joinGroup API to add
> myself to an IPv6 multicast group on Linux (verified via /proc/net/igmp6).
> Additionally, trial reports the same two failures before and after these 
> changes
> (twisted.python.test.test_release.APIBuilderTests.test_build and
> test_buildWithPolicy).  These changes struck me as the obvious approach, but
> given the changes to resolve, not necessarily the best.
> 
> 
> diff --git a/twisted/internet/base.py b/twisted/internet/base.py
> index 4f2c862..e813741 100644
> --- a/twisted/internet/base.py
> +++ b/twisted/internet/base.py
> @@ -567,6 +567,8 @@ class ReactorBase(object):
> return defer.succeed('0.0.0.0')
> if abstract.isIPAddress(name):
> return defer.succeed(name)
> +elif abstract.isIPv6Address(name):
> +return defer.succeed(name)
> return self.resolver.getHostByName(name, timeout)
> 
> # Installation.
> diff --git a/twisted/internet/udp.py b/twisted/internet/udp.py
> index b5a5322..210b079 100644
> --- a/twisted/internet/udp.py
> +++ b/twisted/internet/udp.py
> @@ -485,7 +485,10 @@ class MulticastMixin:
> 
> 
> def _joinAddr1(self, addr, interface, join):
> -return self.reactor.resolve(interface).addCallback(self._joinAddr2, 
> addr, join)
> +if self.addressFamily == socket.AF_INET:
> +return 
> self.reactor.resolve(interface).addCallback(self._joinAddr2, addr, join)
> +else:
> +return 
> self.reactor.resolve(interface).addCallback(self._joinAddrIPv6, addr, join)
> 
> 
> def _joinAddr2(self, interface, addr, join):
> @@ -500,6 +503,18 @@ class MulticastMixin:
> except socket.error as e:
> return failure.Failure(error.MulticastJoinError(addr, interface, 
> *e.args))
> 
> +def _joinAddrIPv6(self, interface, addr, join):
> +addr = socket.inet_pton(socket.AF_INET6, addr)
> +interface = socket.inet_pton(socket.AF_INET6, interface)
> +if join:
> +cmd = socket.IPV6_JOIN_GROUP
> +else:
> +cmd = socket.IPV6_LEAVE_GROUP
> +try:
> +self.socket.setsockopt(socket.IPPROTO_IPV6, cmd, addr + 
> interface)
> +except socket.error as e:
> +return failur

Re: [Twisted-Python] Request for help with Twisted bindings in M2Crypto

[Glyph Lefkowitz]

> On Jul 26, 2016, at 12:26 AM, Daniel Sank  wrote:
> 
> Glyph,
> 
> >> 2. I have no idea what a "task" is. I realize this is python and yay 
> >> duck-typing but not
> >> specifying the expected behavior of an argument seems like a big omission.
> 
> > Did you miss the part where it said "type: 0-argument callable" in the 
> > documentation?
> 
> Yes :(
> 
> I can only guess that I missed it because the type is defined near the end of 
> the description line, whereas I'm used to seeing
> 
> def foo(x, y):
> """Do something.
> 
> Args:
> x (int): blah blah
> y (banana): yadda yadda
> """
> 
> FWIW, now that I look at the code, the type specification is way more 
> visually apparent there than it is in the generated HTML.
> 
> tl,dr: I take it all back and thanks for pointing out the obvious.

This does at least point to a real problem with pydoctor in the way it presents 
types.  It should probably put them in their own colored box, not use the 
string 'type' or parentheses to offset them, and put the type closer to (rather 
than farther from) the parameter name.  Would you mind filing a bug on 
pydoctor?  Or commenting on one if it already exists? :)

> > An interface is a very simple concept - an abstract description of what an 
> > object is expected
> > to provide if you're going to do something useful with it.
> 
> Indeed, a general understanding of interfaces is not the problem.
> 
> > In my view, interface definition is the primary activity of software 
> > development
> 
> Agreed 100%.

OK.  Glad to hear it.

> > The fact that so many people seem to find either the basic idea of an 
> > abstract type, or the concrete
> > instantiation of that idea in the Zope Interface library, so horribly 
> > confusing, makes me despair of
> > ever communicating the actually hard stuff that Twisted gets up to in its 
> > internals.
> 
> > I would very much like to understand *what* is so confusing about 
> > "interfaces".  Is it, as Cory posited,
> > just that the documentation is not properly linked?  Or is it that the 
> > average Python developer needs
> > a gentle introduction to the entire idea of abstract rather than concrete 
> > types?  If they do - is it really
> > Twisted's responsibility to provide it to them?  Should Zope Interface just 
> > have a snazzier website?
> 
> Some years ago when I tried to understand Twisted's use of interfaces via 
> Twisted's own documentation (which included something about hair dryers and 
> voltage standards) I was puzzled by the fact that the examples didn't really 
> show me how to solve a useful problem (or I was too stupid to understand that 
> the examples did in fact do that) despite the fact that I knew what an 
> interface was in general terms. It was a case of understanding the intent but 
> none of the examples.

OK... it's a fair cop.  That documentation is not the best.  Among other 
things, it's mainly trying to explain adaptation, which sort of puts the cart 
before the horse, and automatic adaptation is increasingly considered spooky 
action-at-a-distance within Twisted code.  You can see it here: 
>.

You're the perfect person to submit patches against this doc, by the way, since 
you have a firm grasp of the whole "abstract interface" thing but also found it 
confusing.  Personally, I find the examples very clear - I say the 
documentation is "not the best" because I could see how it could confuse 
somebody _else_, but it doesn't confuse _me_ at all, so it's a bit hard for me 
to improve it (especially incrementally).

> A brief look at the zope documentation just now makes me think the situation 
> has improved.

Well that's good, at least.  perhaps we should link to it more prominently.

> The other problem was that interfaces were sprinkled somewhat haphazardly 
> around the code I was trying to understand (perspective broker) and it was 
> just plain hard to keep navigating around the code files to understand who 
> was implementing what interfaces. This could have been my own fault for not 
> having a editor set up. I don't know.

Setting up your editor to have a 'jump to definition' key definitely helps; but 
then, it generally helps with any large codebase.  So, hard to say.

> > Should Zope Interface just have a snazzier website?
> 
> I think the real issue is the need for compelling and simple examples.

Do you think it would be better to put things in terms of a concrete Twisted 
interface, like "IProtocol"?  I am pretty sure these docs were trying to stay 
away from anything "real" because this is a highly abstract concept that could 
apply to anything, and when we drag a concrete example in 

> P.S. Everything below here is completely off topic of this thread and I 
> probably shouldn't have written it.

In for a penny...

(We may want to spin out into a different thread for t

Re: [Twisted-Python] Request for help with Twisted bindings in M2Crypto

[Glyph Lefkowitz]

> On Jul 26, 2016, at 1:18 AM, Glyph Lefkowitz  wrote:
> 
>> I spent a considerable amount of time reading the PB code, reproducing parts 
>> of it myself, and talking to people on IRC and the mailing list to 
>> understand a particularly weird issue in PB. See here for the bug I was 
>> trying to fix (note in particular my first comment to the one existing 
>> answer):
>> 
>> http://stackoverflow.com/questions/23421423/why-are-dummy-objects-created-in-twisteds-pb-system
>>  
>> 
> 

BTW, since this discussion raised this question again, and since I now 
understand better what I think you were _actually_ asking, I put a new answer 
on that question. Hopefully it resolves the mystery for you :).

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Request for help with Twisted bindings in M2Crypto

[Matěj Cepl]

On 2016-07-26, 06:05 GMT, Glyph Lefkowitz wrote:

An interface is a very simple concept


Actually I found 
https://twistedmatrix.com/documents/current/core/howto/components.html 
to be a very good description. Yes, the concept is not that 
complicated, but it is very uncommon in the Pythonic world and 
the experience with M2Crypto and your previous reply seems to 
indicate that even you in the end prefer hard-wiring 
Cryptography to Twisted instead of using your own interfaces.  
Isn’t simple better than complex?


Whatever,

Matěj

--
https://matej.ceplovi.cz/blog/, Jabber: mc...@ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5  BC1D 7920 5802 880B C9D8

http://xkcd.com/743/ … enough said.



pgpSbMxli3J1I.pgp
Description: PGP signature
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Request for help with Twisted bindings in M2Crypto

[Matěj Cepl]

On 2016-07-26, 07:41 GMT, Glyph Lefkowitz wrote:
I do not want to denigrate the work you've done maintaining 
a legacy library.  I think it's noble to take on this kind of 
work.


I was watching “A Special Day” (1977) yesterday so I have 
somewhat lesser tolerance for the pompous superiority complex, 
but I will think about your reply and the support for Twisted 
will probably go.


Matěj

--
https://matej.ceplovi.cz/blog/, Jabber: mc...@ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5  BC1D 7920 5802 880B C9D8

http://xkcd.com/743/ … enough said.



pgpzFJMxi_VoB.pgp
Description: PGP signature
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Request for help with Twisted bindings in M2Crypto

[Matěj Cepl]

On 2016-07-25, 09:55 GMT, Craig Rodrigues wrote:
First of all, thank you very much for all help you gave to 
M2Crypto. However, ...



I call shenanigans on you.


Nothing of which I have been accused of has anything to do with 
what I meant. When I said „opaque“, I didn’t mean to offend 
Twisted. Just to say, that while I was marking whole library 
with PEP-484 type hints, I usually dealt with bytes, str, ints, 
and very few rather simple objects. With Twisted I get object


   @implementer(ITLSTransport)
   class TLSProtocolWrapper(ProtocolWrapper):
   def __init__(self, factory, wrappedProtocol, 
startPassThrough, client,

contextFactory, postConnectionCheck):
   # type: policies.WrappingFactory, object, int, int, object, Checker

(and with those two objects, I don't even dare to guess what 
types these are, and all that covered in some weird decorator 
from Zope (?)).


That's nothing wrong with Twisted, just that it is really 
difficult for idiot like me to understand what's going on.


If you are unfamiliar with Twisted's code and data types, and 
don't have the energy to dig in, then be honest about that, 
but don't accuse Twisted of being "opaque", because it isn't.


I don't think what's dishonest on saying that Twisted API is 
quite complicated and I am stupid enough not understanding 
what's going on.



Regarding your code example which is failing,
your code is failing because you are intermixing bytes and strings which is
a big no-no for Python 3.


Of course I know that (it is not the first place where I have to 
deal with bytes × str dichotomy in py3k), but in order to 
understand what's going, I have to first understand where these 
values come from and where they run to, i.e., to decipher 
Twisted. Thus I was asking for help.



If I look at this line for example:
https://gitlab.com/m2crypto/m2crypto/blob/master/M2Crypto/SSL/TwistedProtocolWrapper.py#L357

I see the code is doing stuff like:
   data = ''
   encryptedData = ''

Those are of type str, and need to be of type bytes:

   data = b''
   encryptedData = b''

You need to clean stuff like that up in your code so that you are only
using bytes.


I believe I have fixed all I can do without actually 
understanding Twisted in 
https://gitlab.com/mcepl/m2crypto/commit/6cd5f87b31e50016ebb7e44f3f2ae46610bc24e0.  
So now, if Twisted is so transparent and perfectly 
understandable, could you please suggest, what I do wrong, that 
the test ends in the endless loop 
(https://travis-ci.org/mcepl/M2Crypto/builds/147175901)?


Thank you,

Matěj Cepl

--
https://matej.ceplovi.cz/blog/, Jabber: mc...@ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5  BC1D 7920 5802 880B C9D8

Never ascribe to malice that which is adequately explained by
stupidity.
   -- Napoleon Bonaparte (or many other people to whom this
  quote is ascribed)



pgpkgiJOMX1e9.pgp
Description: PGP signature
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Pydoc parameter formatting and explaining interfaces

[Daniel Sank]

This is a branch from the thread with subject "Request for help with
Twisted bindings in M2Crypt".

Regarding my inability to read documentation:

> This does at least point to a real problem with pydoctor in the way it
presents types.
> It should probably put them in their own colored box, not use the string
'type' or
> parentheses to offset them, and put the type closer to (rather than
farther from) the
> parameter name.  Would you mind filing a bug on pydoctor?  Or commenting
on
> one if it already exists? :)

Done: https://github.com/twisted/pydoctor/issues/121

>> Some years ago when I tried to understand Twisted's use of interfaces
via Twisted's
>> own documentation (which included something about hair dryers and
voltage standards)
>> I was puzzled by the fact that the examples didn't really show me how to
solve a useful
>> problem (or I was too stupid to understand that the examples did in fact
do that) despite
>> the fact that I knew what an interface was in general terms. It was a
case of
>> understanding the intent but none of the examples.

> OK... it's a fair cop.

I'm unfamiliar with that term.

> Among other things, it's mainly trying to explain adaptation, which sort
of puts the cart before
> the horse

Yes! That is definitely a big part of the problem. When I think "interface"
I think "methods and their signatures an object promises to provided".
Adaptation is a detail, so to speak. It's also somewhat confusing that the
discussion begins with shapes as it goes over the basic idea of interfaces,
and then switches to hair dryers when it comes time for an example. It
would be easier to read if the examples were more consistent.

> and automatic adaptation is increasingly considered spooky
action-at-a-distance within
> Twisted code.

All the more reason to not use adaptation as the in-your-face example.

> You're the perfect person to submit patches against this doc, by the way,
since you have a
> firm grasp of the whole "abstract interface" thing but also found it
confusing.

Perhaps. On the other hand I think it might be better to replace Twisted's
own documentation with a link to zope's, or at least put the link at the
top and say "read this before reading our examples about adaptation." We'll
see if such a patch receives any love.

--
Daniel Sank
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Request for help with Twisted bindings in M2Crypto

[Glyph Lefkowitz]

> On Jul 26, 2016, at 8:43 AM, Matěj Cepl  wrote:
> 
> On 2016-07-26, 07:41 GMT, Glyph Lefkowitz wrote:
>> I do not want to denigrate the work you've done maintaining a legacy 
>> library.  I think it's noble to take on this kind of work.
> 
> I was watching “A Special Day” (1977) yesterday so I have somewhat lesser 
> tolerance for the pompous superiority complex,

Despite your quite rude introduction to the mailing list (twisted is "opaque 
and complicated", "deep magic", learning it is "horribly complicated"), several 
people (including myself) invested quite a bit of time to try to answer your 
questions in detail.  And, for taking the trouble to be diplomatic, you have 
now, as I understand it, made a veiled allegation that I'm a fascist?  If you 
are not _trying_ to be heinously offensive, perhaps you should stop posting 
here.

-glyph
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Pydoc parameter formatting and explaining interfaces

[Glyph Lefkowitz]

> On Jul 26, 2016, at 9:37 AM, Daniel Sank  wrote:
> 
> This is a branch from the thread with subject "Request for help with Twisted 
> bindings in M2Crypt".
> 
> Regarding my inability to read documentation:
> 
> > This does at least point to a real problem with pydoctor in the way it 
> > presents types.
> > It should probably put them in their own colored box, not use the string 
> > 'type' or
> > parentheses to offset them, and put the type closer to (rather than farther 
> > from) the
> > parameter name.  Would you mind filing a bug on pydoctor?  Or commenting on
> > one if it already exists? :)
> 
> Done: https://github.com/twisted/pydoctor/issues/121 
> 

Thanks!  I especially appreciate the screen shot :).

> >> Some years ago when I tried to understand Twisted's use of interfaces via 
> >> Twisted's
> >> own documentation (which included something about hair dryers and voltage 
> >> standards)
> >> I was puzzled by the fact that the examples didn't really show me how to 
> >> solve a useful
> >> problem (or I was too stupid to understand that the examples did in fact 
> >> do that) despite
> >> the fact that I knew what an interface was in general terms. It was a case 
> >> of
> >> understanding the intent but none of the examples.
> 
> > OK... it's a fair cop.
> 
> I'm unfamiliar with that term.

For me it's a monty python reference, but I suspect for most speakers of 
British English, it's just an idiom :).  It roughly means "mea culpa", 
although, more specifically, I believe it means "you've caught me doing 
something bad".

> > Among other things, it's mainly trying to explain adaptation, which sort of 
> > puts the cart before
> > the horse
> 
> Yes! That is definitely a big part of the problem. When I think "interface" I 
> think "methods and their signatures an object promises to provided". 
> Adaptation is a detail, so to speak. It's also somewhat confusing that the 
> discussion begins with shapes as it goes over the basic idea of interfaces, 
> and then switches to hair dryers when it comes time for an example. It would 
> be easier to read if the examples were more consistent.
> 
> > and automatic adaptation is increasingly considered spooky 
> > action-at-a-distance within
> > Twisted code.
> 
> All the more reason to not use adaptation as the in-your-face example.
> 
> > You're the perfect person to submit patches against this doc, by the way, 
> > since you have a
> > firm grasp of the whole "abstract interface" thing but also found it 
> > confusing.
> 
> Perhaps. On the other hand I think it might be better to replace Twisted's 
> own documentation with a link to zope's, or at least put the link at the top 
> and say "read this before reading our examples about adaptation." We'll see 
> if such a patch receives any love.

Please direct my attention to it when one exists :).

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] txtorcon 0.15.0

[meejah]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

I'm happy to announce txtorcon 0.15.0:

 * added support for NULL control-port-authentication which is often
   appropriate when used with a UNIX domain socket
 * switched to https://docs.python.org/3/library/ipaddress.html instead
   of
   Google's ipaddr; the API should be the same from a user
   perspective but **packagers and tutorials** will want to change
   their instructions slightly (``pip install ipaddress`` or ``apt-get
   install python-ipaddress`` are the new ways).
 * support the new ADD_ONION and DEL_ONION "ephemeral hidden services"
   commands in TorConfig
 * a first stealth-authentication implementation (for "normal" hidden
   services, not ephemeral)
 * bug-fix from https://github.com/david415 to raise
   ConnectionRefusedError instead of StopIteration when running out of
   SOCKS ports.
 * new feature from https://github.com/david415 adding a
   ``build_timeout_circuit`` method which provides a Deferred that
   callbacks only when the circuit is completely built and errbacks if
   the provided timeout expires. This is useful because
   :doc:`TorState.build_circuit` callbacks as soon as a Circuit
   instance can be provided (and then you'd use
   :doc:`Circuit.when_built` to find out when it's done building).
 * new feature from https://github.com/coffeemakr
   falling back to password authentication if cookie authentication
   isn't available (or fails, e.g. because the file isn't readable).
 * both TorState and TorConfig now have a ``.from_protocol``
   class-method.
 * spec-compliant string-un-escaping from https://github.com/coffeemakr
 * fix https://github.com/meejah/txtorcon/issues/176

You can download the release from PyPI or GitHub (or of
course "pip install txtorcon"):

   https://pypi.python.org/pypi/txtorcon/0.15.0
   https://github.com/meejah/txtorcon/releases/tag/v0.15.0

Releases are also available from the hidden service:

   http://timaq4ygg2iegci7.onion/txtorcon-0.15.0.tar.gz
   http://timaq4ygg2iegci7.onion/txtorcon-0.15.0.tar.gz.asc

You can verify the sha256sum of both by running the following 4 lines
in a shell wherever you have the files downloaded:

cat 

Re: [Twisted-Python] Pydoc parameter formatting and explaining interfaces

[Clayton Daley]

Speaking just from my own experience, I don't think it's a problem with the
docs per se.  Rather, I think there's often a disconnect between the
answers people are looking for and the nature of Twisted.  It's hard to
explain, but this is the best I can manage:

   - If you're used to using frameworks, interfaces reflect your motives
   when interacting with a library.  To use an audio-video (A/V) analogy, you
   have interfaces like IVolume, IPlayback (play, pause, stop, ff, rew), and
   IChannel. Documentation just "makes sense".
   - Twisted is like a box of electronics parts.  The docs are enough to
   see why IDigital and IAnalog are used by DigitalToAnalogConverter, but
   that's a long way from understanding when you'd even need/care to consume
   something using an IAnalog.
   - There are several layers of architecture between a LineReader and a
   REST library.  If you don't realize these layers exist, it's not obvious
   that you'd need to find/create them in a Twisted app.
   - The (excellent) krondo tutorial is like a small kit that shows you how
   to build a model piece of electronics.  If you want to build a complete A/V
   device, where do you go next?

To make it more concrete for twisted, here's a example of that next step:

   - Start with the chat server example (
   https://twistedmatrix.com/documents/current/_downloads/chatserver.py)
   - To add rooms, you need to introduce a layer of commands (minimally
   JOIN and SEND).  This requires a complete rewrite of the Protocol and I'm
   not aware of any tutorial that helps you make this architectural leap.
   - Maybe Twisted has a class for this so one might search the docs, spend
   lots of time, get really confused, and (tentatively) conclude it's not
   there.  Figuring out that something doesn't exist is especially hard.
   - The authentication example (
   https://twistedmatrix.com/documents/current/core/howto/cred.html)
   actually has a command layer, but key parts of it (i.e. the lineReceived
   call) are buried in the ellipses.  How do you intuit the missing code if
   you don't even realize that you need a commands layer?
   - Of course, to combine auth with chat, you also need to figure out how
   to rewrite the business logic as an IMailbox implementing avatars.  That's
   a LOT of moving pieces to get right at the same time.

When I decided to try out Zend Framework (PHP), their full-stack skeleton
application (
https://docs.zendframework.com/tutorials/getting-started/skeleton-application/)
showed you how everything fit together. Maybe twisted would benefit from
something similar... especially if it emphasized the various architectural
layers by putting the protocol, command handler, and actual business logic
into separate classes.

Clayton Daley

On Tue, Jul 26, 2016 at 12:58 PM, Glyph Lefkowitz 
wrote:

>
> On Jul 26, 2016, at 9:37 AM, Daniel Sank  wrote:
>
> This is a branch from the thread with subject "Request for help with
> Twisted bindings in M2Crypt".
>
> Regarding my inability to read documentation:
>
> > This does at least point to a real problem with pydoctor in the way it
> presents types.
> > It should probably put them in their own colored box, not use the string
> 'type' or
> > parentheses to offset them, and put the type closer to (rather than
> farther from) the
> > parameter name.  Would you mind filing a bug on pydoctor?  Or commenting
> on
> > one if it already exists? :)
>
> Done: https://github.com/twisted/pydoctor/issues/121
>
>
> Thanks!  I especially appreciate the screen shot :).
>
> >> Some years ago when I tried to understand Twisted's use of interfaces
> via Twisted's
> >> own documentation (which included something about hair dryers and
> voltage standards)
> >> I was puzzled by the fact that the examples didn't really show me how
> to solve a useful
> >> problem (or I was too stupid to understand that the examples did in
> fact do that) despite
> >> the fact that I knew what an interface was in general terms. It was a
> case of
> >> understanding the intent but none of the examples.
>
> > OK... it's a fair cop.
>
> I'm unfamiliar with that term.
>
>
> For *me* it's a monty python reference, but I suspect for most speakers
> of British English, it's just an idiom :).  It roughly means "mea culpa",
> although, more specifically, I believe it means "you've caught me doing
> something bad".
>
> > Among other things, it's mainly trying to explain adaptation, which sort
> of puts the cart before
> > the horse
>
> Yes! That is definitely a big part of the problem. When I think
> "interface" I think "methods and their signatures an object promises to
> provided". Adaptation is a detail, so to speak. It's also somewhat
> confusing that the discussion begins with shapes as it goes over the basic
> idea of interfaces, and then switches to hair dryers when it comes time for
> an example. It would be easier to read if the examples were more consistent.
>
> > and automatic adaptation is increasingly considere

[Twisted-Python] Fixes for Twisted bindings in M2Crypto

[Craig Rodrigues]

On Tue, Jul 26, 2016 at 8:47 AM, Matěj Cepl  wrote:

>
> I believe I have fixed all I can do without actually understanding Twisted
> in
> https://gitlab.com/mcepl/m2crypto/commit/6cd5f87b31e50016ebb7e44f3f2ae46610bc24e0.
> So now, if Twisted is so transparent and perfectly understandable, could
> you please suggest, what I do wrong, that the test ends in the endless loop
> (https://travis-ci.org/mcepl/M2Crypto/builds/147175901)?
>
>

In your initial e-mail, it would have been useful if you could have
provided reproduction steps
for your problem.

Can you provide reproduction steps?

Since I wasn't sure, I took a guess.

I did the following inside a Python 3.6 virtual environment under OS X:

git clone https://gitlab.com/mcepl/m2crypto.git m2crypto_test
cd m2crypto_test
git checkout python3
python setup.py build --openssl=/usr/local/opt/openssl
python setup.py bdist
python setup.py develop

py.test -v -s -k test_twisted_wrapper tests/test_ssl.py

I was able to reproduce the problem with something looping around:

DEBUG:_encrypt:self.data = "b'GET / HTTP/1.0\n\n\r\n'"
DEBUG:_decrypt:self.encrypted = "b''"
DEBUG:_encrypt:self.data = "b'GET / HTTP/1.0\n\n\r\n'"
DEBUG:_decrypt:self.encrypted = "b''"
DEBUG:_encrypt:self.data = "b'GET / HTTP/1.0\n\n\r\n'"
DEBUG:_decrypt:self.encrypted = "b''"
DEBUG:_encrypt:self.data = "b'GET / HTTP/1.0\n\n\r\n'"
DEBUG:_decrypt:self.encrypted = "b''"

I took a look in the M2Crypto code which interfaces with
Twisted, and found that the dataReceived() loop was never terminating
because dataReceived() takes bytes, but the code was
comparing to an emptry str.  So, the loop was never terminating.

I fixed this and other problems, and now the test_twisted_wrapper
passes on Python 3.  I submitted this merge request:

https://gitlab.com/mcepl/m2crypto/merge_requests/3/commits


While I was looking at the code, I also saw that the make_certs.py
utility does not work on Python 3.

I fixed that, and submitted this merge request:

https://gitlab.com/mcepl/m2crypto/merge_requests/4/commits

--
Craig
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] RFC: IPv6 multicast join/ticket 6597

[Jason Litzinger]

> 
> Thanks for taking this up!
No problem, do I need to reflect anything in the Ticket to indicate I'm
looking at it?
> 
> 'twisted.internet.udp', as an importable module; not 'udp' as a feature of 
> Twisted (or of the Internet, for that matter).  
My question was poorly phrased, I assumed that it was the module that
was problematic, and wondered if anyone was working on an alternative
where this is better suited.

> 
> We have explicitly avoided adding IPv6 name resolution to the reactor because 
> the reactor's API for name resolution is fundamentally the wrong shape for 
> IPv6.  If you want to add the ability to resolve IPv6 names to the reactor 
> itself, please see this ticket: https://twistedmatrix.com/trac/ticket/4362 
> 
> 
> For the purposes of this ticket alone, you should probably just skip 
> resolution in _joinAddr1 if resolution is 

I assume you mean skip resolution in joinGroup as well?  That's the only
way to avoid resolve completely.

Additionally, any objections to me updating setTTL in this patch?  It's
pretty common to set the hop limit when doing a multicast.  Not
required, but common.
> 
> Testing multicast is ... challenging.  I barely have any idea how to set up a 
> test environment for IPv4, and no idea what to do for IPv6.  If you can speak 
> to this in your tests (and hopefully docs as well) that would be super 
> helpful.
> 
Indeed.  I have an interest beyond the scope of this change so I'll see
what I can do/find.

Thanks!
-Jason Litzinger

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python