[Twisted-Python] Implement HTTP request acception / rejection as defined in RFC 2616 - 8.2.3

[Adi Roiban]

Hi,

As requested during review I am moving this to mailing list for discussions.

The initial ticket is here : https://twistedmatrix.com/trac/ticket/6928

---

The current Twisted implementation of RFC 2616 - 8.2.3 is very simple and
Twisted will always reply with continue to Accept: 100-continue requests.

I am working at web application based on Twisted web so I did a quick
security check for current twisted web code.

The major problem is that when reading POST data, requests under 1M are
kept in memory, the other being stored in temporary file... but for
form-mulipart request the whole data is loaded again in memory for parsing.
Now, if the client initially reports that content is less than 1M, but then
continue to send more than 1M, twisted.web will continue to keep all data
in memory.

To address the problem of unwanted big post requests I first start with
implementing support for well behaved HTTP/1.1 clients.

In current code, Resource objects are called only after whole request data
was received and processed so a resource can not decide whether or not it
accepts the request, just by looking at headers.

Also the Request object is called after headers are received but with calls
like parseCookie or gotLength. So there is no API for Request to accept or
reject a request based on headers.



To solve this problem, I add a new `headersReceived` API call on Request
and Resource so that Requests and Resources can decided if they want to
accept the request.

The biggest change required to implement this API is that URL traversal can
now begin before a resource has received all its content.
If a resource needs full request content before making a traversal
decistion, then this API does not work.

This is why this is an optional API not enforced on all resources.

For now the ticket only has demo code as I wanted to highlight the API
changes.

I have already implemented this code in my project and wrote full test for
these changes.

If this kind of API is accepted in Twisted I can port the code and tests,
otherwise I can try to work on a different implementation.

Thanks!

-- 
Adi Roiban
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Contributing documentation inside CONTRIBUTING repo file

[Adi Roiban]

On 20 February 2014 17:13,  wrote:

> On 02:55 pm, a...@roiban.ro wrote:
>
>> Hi.
>>
>> Right now CONTRIBUTING file from twisted repo sends developers to:
>> https://twistedmatrix.com/trac/wiki/ContributingToTwistedLabs
>>
>> What do you think about having basic contributing documentation inside the
>> repo?
>> I am talking about these 2 pages:
>>
>> https://twistedmatrix.com/trac/wiki/BasicGuideToContributingCode
>> https://twistedmatrix.com/trac/wiki/TwistedDevelopment
>>
>> Commiters Check list can be left on the wiki page:
>> https://twistedmatrix.com/trac/wiki/CommitterCheckList
>>
>> 
>>
>> I think that those pages are very important to assist new developers in
>> creating a successful patch (ie one that is merged without to much
>> ping-pong)
>>
>> Even if I now have wiki edit page, I don't have the courage to change the
>> wiki page (for example to add information about twistedchecker or how to
>> build the new docs).
>>
>> Changing a file and submitting for review feels more comfortable.
>>
>> Also, developers could always refer to that file from inside their
>> preferred text editor.
>>
>
> There is currently an effort to re-organize the contributor documentation
> underway.  Given its fragile state, I wouldn't want to disturb it sending
> work off in a different direction.  On the other hand, if you'd like to
> take over and complete it, feel free to take whatever direction you like.
> :)  Either way I suggest you try to coordinate with the folks working on
> that.
>

Thanks for the reply. I am happy to coordinate with folks working on
contributor documentation re-organization, but I have idea how exactly I am
supposed to do that and who is working on this task.

Thanks!
-- 
Adi Roiban
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Status of twistedchecker

[Glyph Lefkowitz]

On Feb 20, 2014, at 6:00 AM, Adi Roiban  wrote:

> While submitting a patch on Twisted I got a review response that my code is 
> not complaint with `twistedchecker`.
> 
> My first reaction was: Holy code, what is twistedchecker!

As new tools are added to our development process to make things easier (the 
things twistedchecker checks for used to be entirely manual) it's an ongoing 
struggle to keep our documentation up to date.

> Now that I got some info about twistedchecker  and wiki write access I would 
> like to update the https://twistedmatrix.com/trac/wiki/TwistedDevelopment 
> wiki page with information about how a Twisted developer is expected to use 
> twistedchecker.

Please do!

> I run twistedcheck on trunk and got a lot of errors.

Yes.

Because the things twistedchecker checks for used to be manual, we used to fail 
to check for a lot of them.  And many of the things it checks for were not even 
thought of when Twisted was first developed.  This means that the codebase is 
largely not compliant with twistedchecker.

> Am I suppose to look through all those error and filter the ones generated by 
> my changes?

Yes and no.

Yes, in that you should only care about the new ones that your changes created. 
 We aim to have the number of twistedchecker warnings decreasing over time; but 
as long as trunk still produces hundreds and hundreds of warnings we're never 
going to have a big project to eliminate them in one fell swoop.

No, in that you don't have to do this yourself, the buildbot will do it for 
you.  For example, if on the page for this build 
 you 
click on "new twistedchecker errors", you will see 

 a very small number of warnings.  If there are no new twistedchecker errors, 
the builds should succeed.  (This mechanism isn't perfect, however; sometimes 
it fails, although less now than it used to.)

> Since twistedchecker is only about twisted, shouldn't it be included as a 
> dependency to twisted-dev-tools so that as I developer I would only need to 
> install one extra package and it will pull all development/testing 
> depenencies?

Sounds like a good idea to me.

> Thanks!

Thank you for pointing this out!

-glyph




smime.p7s
Description: S/MIME cryptographic signature
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Contributing documentation inside CONTRIBUTING repo file

[Glyph Lefkowitz]

On Feb 21, 2014, at 8:03 AM, Adi Roiban  wrote:

> There is currently an effort to re-organize the contributor documentation 
> underway.  Given its fragile state, I wouldn't want to disturb it sending 
> work off in a different direction.  On the other hand, if you'd like to take 
> over and complete it, feel free to take whatever direction you like. :)  
> Either way I suggest you try to coordinate with the folks working on that.
> 
> Thanks for the reply. I am happy to coordinate with folks working on 
> contributor documentation re-organization, but I have idea how exactly I am 
> supposed to do that and who is working on this task.
> 

Richard Wall and Ashwini Oruganti are two of the people working on this, but to 
the best of my knowledge this effort is unfortunately mostly stalled right now. 
 If neither of them respond promptly to this thread, I think it would be best 
to consider that effort defunct and proceed with whatever changes you want to 
make.

The timeline report on wiki changes 

 suggests that no work has been done on this in over 90 days, which seems like 
an appropriate window beyond which superseding it is fair game :).

-glyph



smime.p7s
Description: S/MIME cryptographic signature
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python