Re: [Twisted-Python] twisted.python.dist as a public API

[Adi Roiban]

On 7 August 2016 at 00:50, Glyph Lefkowitz  wrote:

[snip]


> You always have to follow the compatibility process - if it's not a
> security bug, we would have to change the process first :-).  But if you
> want to follow the exception path rather than the deprecation path (this
> message doesn't qualify, because the subject is incorrectly formatted) I
> certainly don't have any objection.
>
>
As pointed by Craig in the PR, dist3.py is used by
https://github.com/habnabit/twisted-depgraph. I will revert the dist3
change.

With this email, I just wanted to get an initial feedback and try to
understand why we have code split between setup.py and dist.py

I will come back with an email following the compatibility process.

Thanks!
-- 
Adi Roiban
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.dist renamed to twisted.python._setup

[Adi Roiban]

Hi,

I would like to remove twisted.python.dist as a public API without
following the 1 year deprecation processes.

The module has a comment which informs people not to use it outside of
Twisted, but I think it should be private module instead.

The ticket is here https://twistedmatrix.com/trac/ticket/8622
The PR is here https://github.com/twisted/twisted/pull/467
The branch is here
https://github.com/twisted/twisted/tree/8622-setup-coverage

This is part of the setup.py cleanup effort for having most of the setup.py
code outside of setup.py itself so that we can test it as part of the
automated test suite.

twisted.python.dist3 was kept as a private API as it is already used  by
https://github.com/habnabit/twisted-depgraph

Please let me know if you are using twisted.python.dist outside of Twisted
and are against this change or if you approve it.

Thanks!

-- 
Adi Roiban
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted.python.dist as a public API

[Glyph Lefkowitz]

> On Aug 7, 2016, at 4:03 AM, Adi Roiban  wrote:
> 
> 
> 
> On 7 August 2016 at 00:50, Glyph Lefkowitz  > wrote:
> 
> [snip]
>  
> You always have to follow the compatibility process - if it's not a security 
> bug, we would have to change the process first :-).  But if you want to 
> follow the exception path rather than the deprecation path (this message 
> doesn't qualify, because the subject is incorrectly formatted) I certainly 
> don't have any objection.
> 
> 
> As pointed by Craig in the PR, dist3.py is used by 
> https://github.com/habnabit/twisted-depgraph 
> . I will revert the dist3 
> change.

I should point out that the entire point of twisted-depgraph is to help Twisted 
developers - basically, you and Craig - understand the dependencies remaining 
and port more things to py3.  So I'm pretty sure Aaron would not mind at all if 
he had to update it :).  It's not really a user-facing application, just a core 
developer tool.

> With this email, I just wanted to get an initial feedback and try to 
> understand why we have code split between setup.py and dist.py 
> 
> I will come back with an email following the compatibility process.
> 
> Thanks!

No problem!  I have a little tool that imports dist3 myself, and it  is _super_ 
encouraging to see the percentages of ported code these days.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.dist renamed to twisted.python._setup

[Glyph Lefkowitz]

> On Aug 7, 2016, at 4:19 AM, Adi Roiban  wrote:
> 
> Hi,
> 
> I would like to remove twisted.python.dist as a public API without following 
> the 1 year deprecation processes.
> 
> The module has a comment which informs people not to use it outside of 
> Twisted, but I think it should be private module instead.
> 
> The ticket is here https://twistedmatrix.com/trac/ticket/8622 
> 
> The PR is here https://github.com/twisted/twisted/pull/467 
> 
> The branch is here 
> https://github.com/twisted/twisted/tree/8622-setup-coverage 
> 
> 
> This is part of the setup.py cleanup effort for having most of the setup.py 
> code outside of setup.py itself so that we can test it as part of the 
> automated test suite.
> 
> twisted.python.dist3 was kept as a private API as it is already used  by 
> https://github.com/habnabit/twisted-depgraph 
> 
> 
> Please let me know if you are using twisted.python.dist outside of Twisted 
> and are against this change or if you approve it.
> 
> Thanks!

+1 from me!

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.dist renamed to twisted.python._setup

[Craig Rodrigues]

On Sun, Aug 7, 2016 at 4:15 PM, Glyph Lefkowitz 
wrote:

>
> On Aug 7, 2016, at 4:19 AM, Adi Roiban  wrote:
>
> Hi,
>
> I would like to remove twisted.python.dist as a public API without
> following the 1 year deprecation processes.
>
> The module has a comment which informs people not to use it outside of
> Twisted, but I think it should be private module instead.
>
> The ticket is here https://twistedmatrix.com/trac/ticket/8622
> The PR is here https://github.com/twisted/twisted/pull/467
> The branch is here https://github.com/twisted/twisted/tree/8622-
> setup-coverage
>
> This is part of the setup.py cleanup effort for having most of the
> setup.py code outside of setup.py itself so that we can test it as part of
> the automated test suite.
>
> twisted.python.dist3 was kept as a private API as it is already used  by
> https://github.com/habnabit/twisted-depgraph
>
> Please let me know if you are using twisted.python.dist outside of Twisted
> and are against this change or if you approve it.
>
> Thanks!
>
>
> +1 from me!
>
>
>

-1 from me.  dist.py probably should have been a private API from the
beginning, but it slipped through the cracks.
However, what has been done is done.  This change seems gratuitous and
pointless with
no benefits to anybody.

--
Craig
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.dist renamed to twisted.python._setup

[Amber "Hawkie" Brown]

> On 8 Aug 2016, at 08:15, Craig Rodrigues  wrote:
> 
> 
> 
> On Sun, Aug 7, 2016 at 4:15 PM, Glyph Lefkowitz  > wrote:
> 
>> On Aug 7, 2016, at 4:19 AM, Adi Roiban > > wrote:
>> 
>> Hi,
>> 
>> I would like to remove twisted.python.dist as a public API without following 
>> the 1 year deprecation processes.
>> 
>> The module has a comment which informs people not to use it outside of 
>> Twisted, but I think it should be private module instead.
>> 
>> The ticket is here https://twistedmatrix.com/trac/ticket/8622 
>> 
>> The PR is here https://github.com/twisted/twisted/pull/467 
>> 
>> The branch is here 
>> https://github.com/twisted/twisted/tree/8622-setup-coverage 
>> 
>> 
>> This is part of the setup.py cleanup effort for having most of the setup.py 
>> code outside of setup.py itself so that we can test it as part of the 
>> automated test suite.
>> 
>> twisted.python.dist3 was kept as a private API as it is already used  by 
>> https://github.com/habnabit/twisted-depgraph 
>> 
>> 
>> Please let me know if you are using twisted.python.dist outside of Twisted 
>> and are against this change or if you approve it.
>> 
>> Thanks!
> 
> +1 from me!
> 
> 
> 
> 
> -1 from me.  dist.py probably should have been a private API from the 
> beginning, but it slipped through the cracks.
> However, what has been done is done.  This change seems gratuitous and 
> pointless with
> no benefits to anybody.
> 
> --
> Craig

I believe there's a point to it -- that is, we make it clear what is and what 
isn't user API. It's always said it in the header, but we should codify that 
it's really not user API. Plus it being private means we can make what changes 
we need for future setup.py changes, without deprecating things that nobody but 
us should ever use.

+1 from me.

- Amber



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.dist renamed to twisted.python._setup

[Craig Rodrigues]

On Sun, Aug 7, 2016 at 5:25 PM, Amber "Hawkie" Brown <
hawk...@atleastfornow.net> wrote:

>
> On 8 Aug 2016, at 08:15, Craig Rodrigues  wrote:
>
>
>
> On Sun, Aug 7, 2016 at 4:15 PM, Glyph Lefkowitz 
> wrote:
>
>>
>> On Aug 7, 2016, at 4:19 AM, Adi Roiban  wrote:
>>
>> Hi,
>>
>> I would like to remove twisted.python.dist as a public API without
>> following the 1 year deprecation processes.
>>
>> The module has a comment which informs people not to use it outside of
>> Twisted, but I think it should be private module instead.
>>
>> The ticket is here https://twistedmatrix.com/trac/ticket/8622
>> The PR is here https://github.com/twisted/twisted/pull/467
>> The branch is here https://github.com/twisted/twisted/tree/8622-setup-
>> coverage
>>
>> This is part of the setup.py cleanup effort for having most of the
>> setup.py code outside of setup.py itself so that we can test it as part of
>> the automated test suite.
>>
>> twisted.python.dist3 was kept as a private API as it is already used  by
>> https://github.com/habnabit/twisted-depgraph
>>
>> Please let me know if you are using twisted.python.dist outside of
>> Twisted and are against this change or if you approve it.
>>
>> Thanks!
>>
>>
>> +1 from me!
>>
>>
>>
>
> -1 from me.  dist.py probably should have been a private API from the
> beginning, but it slipped through the cracks.
> However, what has been done is done.  This change seems gratuitous and
> pointless with
> no benefits to anybody.
>
> --
> Craig
>
>
> I believe there's a point to it -- that is, we make it clear what is and
> what isn't user API. It's always said it in the header, but we should
> codify that it's really not user API. Plus it being private means we can
> make what changes we need for future setup.py changes, without deprecating
> things that nobody but us should ever use.
>
> +1 from me.
>
>
I understand the motivation, but the time to have done this was when dist.py
was originally reviewed.

There are real problems in the Twisted code, and in the Twisted
infrastructure.
Focusing on something like this instead of those problems seems pointless,
gratuitous and of little benefit.

--
Craig
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.dist renamed to twisted.python._setup

[Amber Brown]

t.p.dist dates from 2005, and as far as I can tell, predates Twisted's code
review system. It's an early mistake, but we can fix many mistakes in
parallel.

- Amber

On 8 Aug 2016 08:36, "Craig Rodrigues"  wrote:

>
>
> On Sun, Aug 7, 2016 at 5:25 PM, Amber "Hawkie" Brown <
> hawk...@atleastfornow.net> wrote:
>
>>
>> On 8 Aug 2016, at 08:15, Craig Rodrigues  wrote:
>>
>>
>>
>> On Sun, Aug 7, 2016 at 4:15 PM, Glyph Lefkowitz 
>>  wrote:
>>
>>>
>>> On Aug 7, 2016, at 4:19 AM, Adi Roiban  wrote:
>>>
>>> Hi,
>>>
>>> I would like to remove twisted.python.dist as a public API without
>>> following the 1 year deprecation processes.
>>>
>>> The module has a comment which informs people not to use it outside of
>>> Twisted, but I think it should be private module instead.
>>>
>>> The ticket is here https://twistedmatrix.com/trac/ticket/8622
>>> The PR is here https://github.com/twisted/twisted/pull/467
>>> The branch is here https://github.com/twiste
>>> d/twisted/tree/8622-setup-coverage
>>>
>>> This is part of the setup.py cleanup effort for having most of the
>>> setup.py code outside of setup.py itself so that we can test it as part of
>>> the automated test suite.
>>>
>>> twisted.python.dist3 was kept as a private API as it is already used  by
>>>  https://github.com/habnabit/twisted-depgraph
>>>
>>> Please let me know if you are using twisted.python.dist outside of
>>> Twisted and are against this change or if you approve it.
>>>
>>> Thanks!
>>>
>>>
>>> +1 from me!
>>>
>>>
>>>
>>
>> -1 from me.  dist.py probably should have been a private API from the
>> beginning, but it slipped through the cracks.
>> However, what has been done is done.  This change seems gratuitous and
>> pointless with
>> no benefits to anybody.
>>
>> --
>> Craig
>>
>>
>> I believe there's a point to it -- that is, we make it clear what is and
>> what isn't user API. It's always said it in the header, but we should
>> codify that it's really not user API. Plus it being private means we can
>> make what changes we need for future setup.py changes, without deprecating
>> things that nobody but us should ever use.
>>
>> +1 from me.
>>
>>
> I understand the motivation, but the time to have done this was when
> dist.py
> was originally reviewed.
>
> There are real problems in the Twisted code, and in the Twisted
> infrastructure.
> Focusing on something like this instead of those problems seems pointless,
> gratuitous and of little benefit.
>
> --
> Craig
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.dist renamed to twisted.python._setup

[Tristan Seligmann]

+1 from me

On Mon, 8 Aug 2016 at 02:35 Craig Rodrigues  wrote:

>
> I understand the motivation, but the time to have done this was when
> dist.py
> was originally reviewed.
>
> There are real problems in the Twisted code, and in the Twisted
> infrastructure.
> Focusing on something like this instead of those problems seems pointless,
> gratuitous and of little benefit.
>

The problem is that if we defer this change until later, it becomes a lot
more work then. If we do it now, while nobody is depending on the module,
then we save ourselves that work in future by doing a trivial amount of
work now.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.dist renamed to twisted.python._setup

[Chris Wolfe]

+1 from me as well. Depending directly on how twisted installs itself seems
problematic. In my opinion, the more we discourage this practice the better.

On Sun, Aug 7, 2016 at 8:00 PM Tristan Seligmann 
wrote:

> +1 from me
>
> On Mon, 8 Aug 2016 at 02:35 Craig Rodrigues 
> wrote:
>
>>
>> I understand the motivation, but the time to have done this was when
>> dist.py
>> was originally reviewed.
>>
>> There are real problems in the Twisted code, and in the Twisted
>> infrastructure.
>> Focusing on something like this instead of those problems seems pointless,
>> gratuitous and of little benefit.
>>
>
> The problem is that if we defer this change until later, it becomes a lot
> more work then. If we do it now, while nobody is depending on the module,
> then we save ourselves that work in future by doing a trivial amount of
> work now.
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-- 
- Chris
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.dist renamed to twisted.python._setup

[Glyph Lefkowitz]

> On Aug 7, 2016, at 5:35 PM, Craig Rodrigues  wrote:
> 
> I understand the motivation, but the time to have done this was when dist.py
> was originally reviewed.

It certainly would have been better :-).

> There are real problems in the Twisted code, and in the Twisted 
> infrastructure.

This is also addressing a real problem.  One of the major issues with Twisted 
is documentation noise.  Developers trying to use Twisted to build things 
frequently complain about this: they look for an API to do XYZ, and they search 
the reference documentation, which produces dozens of confusing, irrelevant 
hits.  There are multiple issues that interact here (see, for example, 
 which is a big part of the 
problem) but one significant one is that our apparently-public API surface is 
just too big.  Removing things like this, which are really _totally_ useless to 
anyone, is useful in and of itself and also amplifies the benefit of any other 
work on docs and tooling to properly segregate public and private API 
documentation.

For a long time we were extremely conservative about doing any 
compatibility-breaking thing, even hypothetically breaking, even within the 
policy we'd set forth, and I think the result of that is a sprawling API that's 
incredibly hard to navigate, full of old junk that nobody should ever use.  The 
slightly more aggressive attitude that has been exemplified by e.g. Amber has 
been a breath of fresh air when it comes to unburdening ourselves from literal 
garbage.

So I think that "overburdened public API surface" is one of the largest 
problems facing Twisted and every little bit we can chip away at it is a real 
win.  We do have to be super careful of course, we don't want to be chopping 
off obscure but useful pieces of API that programs were actually using, but 
that's what the compat policy (and judicious judgement) are for.

> Focusing on something like this instead of those problems seems pointless, 
> gratuitous and of little benefit.

In addition to underplaying the usefulness of maintenance like this, I think 
this comment belies acceptance of a superficially true falsehood - one that 
I've often fallen prey to myself - and that is, the idea that volunteer time is 
fungible.  It isn't.  If the members of a project have problems A, B, C, X, Y, 
Z in very clear ranked order of importance, and a volunteer ("V") shows up to 
work on Z, the impulse of the existing members of the project is "but you 
should really just work on 'A'!".  V doesn't care about A, doesn't want to work 
on A, and the likely result of telling them they should be working on A is that 
they will feel bad about not working on A and their fix for Z will go 
unfinished.

I'm not saying that necessarily applies in its full force here; after all Adi 
is a member of the project as well.  But one of the corollaries of this 
realization is that if A were really so important someone would already be 
working on it :).

We should totally have the ABCXYZ list in case anyone shows up just wondering 
what would be helpful to work on; but if someone shows up with a solution to Z, 
we should always try to incorporate it and help them feel motivated about 
continuing, hopefully up the list towards A of course, but maybe they are going 
to contribute stuff for M, N, and O that we didn't even realize was a problem.

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.dist renamed to twisted.python._setup

[Craig Rodrigues]

On Sun, Aug 7, 2016 at 11:01 PM, Glyph Lefkowitz 
wrote:

>
> This is also addressing a real problem.  One of the *major* issues with
> Twisted is *documentation noise*.  Developers trying to use Twisted to
> build things frequently complain about this: they look for an API to do
> XYZ, and they search the reference documentation, which produces dozens of
> confusing, irrelevant hits.  There are multiple issues that interact here
> (see, for example,  which
> is a big part of the problem) but one significant one is that our
> apparently-public API surface is just too big.  Removing things like this,
> which are really _totally_ useless to anyone, is useful in and of itself
> and also amplifies the benefit of any other work on docs and tooling to
> properly segregate public and private API documentation.
>
>

Making dist.py private seems to fall under the category of "code
cleanliness", and doesn't
seem to solve any specific pressing issue that I see.  Reducing
documentation noise is a good goal,
but if dist.py was left the way it is, things would probably be OK with
Twisted, and I think people
could figure things out in Twisted with respect to the documentation.

Since you mention pydoctor, I will mention two issues, that I feel are more
important to the project
that moving around dist.py:


   - pydoctor produces false errors, specifically in h2 code:
   
https://buildbot.twistedmatrix.com/builders/documentation/builds/1291/steps/api-documentation/logs/pydoctor%20errors
   - pydoctor doesn't work on Python 3:
   https://github.com/twisted/pydoctor/issues/96

I understand that Twisted is a volunteer project, and people can work on
whatever they want.
I also see on this list, that a few people support the dist.py change, so
as long as
Adi's change meets the Twisted coding standards, gets code review approval,
and can pass all the buildbots,
it can move forward, despite my objections.

--
Craig
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python