Re: [Twisted-Python] Agent and "Cannot assign requested address"

[akira]

"Jason J. W. Williams"  writes:

> Actually, I think the TIME_WAIT is the problem. It's what I see in
> netstat, and the Agent requests are fired sequentially via yield
> inside a for loop (inlineCallbacks). So they shouldn't be running in
> parallel. 

`yield` returns before TIME_WAIT expires otherwise it would require ~1
minute per request.

>
> The use case here is loading a Riak server with keys to prepare for a
> test. There's not a real way to get around sending one POST per key.
>
> How would I set the timeout value in Twisted? Or do I have to modify
> the timeout/keepalive systemwide in /proc?

In addition to net.ipv4.tcp_fin_timeout you could increase the ephemeral
port range (net.ipv4.ip_local_port_range sysctl parameter).

Each connection can be identified using 4-tuple (server IP, server port,
client IP, client port) Everything except client port is fixed in your
case so there could be at most
~ net.ipv4.ip_local_port_range/net.ipv4.tcp_fin_timeout connections per
second (even less in practice due to other applications and other
settings taking preference such as fs.file-max). For example:

  net.ipv4.ip_local_port_range = 3276861000
  net.ipv4.tcp_fin_timeout = 30

There could be ~900 connections per second that might be good enough.

Reusing a local port via SO_REUSEADDR or better yet reusing a tcp
connection via HTTP keep-alive aren't available with twisted as I
understand it.


--
akira


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [Twisted-web] Question regarding callLater() and service creation

[Andrew Bennetts]

This thread belongs on the twisted-python@ list, rather than
twisted-web@, as it has nothing to do with twisted.web.

Eric Chamberlain wrote:
> I'm creating a service using Twisted.  It has nothing to do with
> networking (I know that Twisted is an event-driven networking
> engine... stay with me here :).  It's simply polling a database every
> 30 seconds.  I'm using callLater() once the work has been completed.
> My question is does the reactor spawn a new thread once the
> callLater() timeout has been reached?  If so, this would mean that the
> work being done may be effected by the GIL, correct?

Incorrect.  callLater spawns no threads.

There's nothing wrong or even particularly strange with using Twisted
for projects unrelated to networking.  An event loop with good
facilities for spawning and interacting with subprocesses and calling
functions at certain times has plenty of uses, even if you don't also
need the capacity to handle network IO.

That doesn't mean that Twisted is necessarily the best tool for your
task, but I wouldn't rule it out automatically either.

-Andrew.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [Twisted-web] Question regarding callLater() and service creation

[Jason Rennie]

On Thu, Mar 10, 2011 at 6:43 AM, Andrew Bennetts wrote:

> There's nothing wrong or even particularly strange with using Twisted
> for projects unrelated to networking.  An event loop with good
> facilities for spawning and interacting with subprocesses and calling
> functions at certain times has plenty of uses, even if you don't also
> need the capacity to handle network IO.
>

+1  Twisted is my preferred tool for managing subprocesses.  I've even found
it to be superior for the seemingly simple task of combining output from
multiple processes.

Jason
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Agent and "Cannot assign requested address"

[Glyph Lefkowitz]

On Mar 10, 2011, at 5:31 AM, akira wrote:

> Reusing a local port via SO_REUSEADDR or better yet reusing a tcp
> connection via HTTP keep-alive aren't available with twisted as I
> understand it.

Reusing a local connection-oriented port with SO_REUSEADDR is potentially a bad 
idea; there's a reason that your TCP stack gives you this error.  That option 
is practically only for listening ports.

Keep-alive is a work in progress, previously mentioned in this thread: 
.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Agent and "Cannot assign requested address"

[Jason J. W. Williams]

I ended up getting around the problem by increasing my Riak cluster
size and putting a load balancer in front for the test.  But
connection pooling would be really helpful, both here and in the
CouchDB client. I've refactored both txRiak and Paisley in the past
couple of months to use Agent in the hopes ticket 3420 gets completed.
:)

-J

On Thu, Mar 10, 2011 at 3:31 AM, akira <4kir4...@gmail.com> wrote:
> "Jason J. W. Williams"  writes:
>
>> Actually, I think the TIME_WAIT is the problem. It's what I see in
>> netstat, and the Agent requests are fired sequentially via yield
>> inside a for loop (inlineCallbacks). So they shouldn't be running in
>> parallel.
>
> `yield` returns before TIME_WAIT expires otherwise it would require ~1
> minute per request.
>
>>
>> The use case here is loading a Riak server with keys to prepare for a
>> test. There's not a real way to get around sending one POST per key.
>>
>> How would I set the timeout value in Twisted? Or do I have to modify
>> the timeout/keepalive systemwide in /proc?
>
> In addition to net.ipv4.tcp_fin_timeout you could increase the ephemeral
> port range (net.ipv4.ip_local_port_range sysctl parameter).
>
> Each connection can be identified using 4-tuple (server IP, server port,
> client IP, client port) Everything except client port is fixed in your
> case so there could be at most
> ~ net.ipv4.ip_local_port_range/net.ipv4.tcp_fin_timeout connections per
> second (even less in practice due to other applications and other
> settings taking preference such as fs.file-max). For example:
>
>  net.ipv4.ip_local_port_range = 32768    61000
>  net.ipv4.tcp_fin_timeout = 30
>
> There could be ~900 connections per second that might be good enough.
>
> Reusing a local port via SO_REUSEADDR or better yet reusing a tcp
> connection via HTTP keep-alive aren't available with twisted as I
> understand it.
>
>
> --
> akira
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Failing tests in trunk

[Facundo Batista]

On Tue, Mar 8, 2011 at 7:04 AM, Facundo Batista
 wrote:

>> None of these are known failures:
>>
>>    http://buildbot.twistedmatrix.com/boxes-supported
>>
>> Just looking at the list of failing tests, I would make a small bet that
>> the failures are caused by your locale setting somehow.
>
> I thought of that, but why would the test pass when running alone?
>
> I'll debug this.

Ok, found it.

See one example of the test that failed,
twisted.mail.test.test_imap.NewFetchTestCase.testFetchInternalDate:

twisted.trial.unittest.FailTest: not equal:
a = {0: {'INTERNALDATE': '02-nov-2003 21:25:10 +'},
 1: {'INTERNALDATE': '29-dic-2013 11:31:52 -0500'},
 2: {'INTERNALDATE': '10-mar-1992 02:44:30 -0600'},
 3: {'INTERNALDATE': '11-ene-2000 14:40:24 -0800'}}
b = {0: {'INTERNALDATE': '02-Nov-2003 21:25:10 +'},
 1: {'INTERNALDATE': '29-Dec-2013 11:31:52 -0500'},
 2: {'INTERNALDATE': '10-Mar-1992 02:44:30 -0600'},
 3: {'INTERNALDATE': '11-Jan-2000 14:40:24 -0800'}}

See the difference? "nov" instead of "Nov", etc. "dic" instead of
"Dec". Yes, in Spanish December is 'diciembre'.

So, funny 'locale' issue, confirmed by showing getlocale() in the
test... when running the whole suite, the locale is ('es_AR', 'UTF8'),
and when running twisted.mail the locale is (None, None).

Weird, specially taking into account that 'locale' is not used in the
whole twisted project.

But I tracked it down.

In twisted/manhole/ui/test/test_gtk2manhole.py, gtk is imported, and
that changes the locale [0].

So, how this can be fixed?

Is ok for the IMAP4Server to spew the message internal date according
to the locale in which it's being run? Or it should always format the
dates in an specific way, because some definition somewhere? (I don't
know much about IMAP)

If it's ok for the server to format the dates according to the locale,
I should make the test more robust. If not, the server should set the
locale to (None, None) for the date formatting, or just set it at
__init__ and leave it there, or use other formatting (not
time.strftime, I prefer this solution).

Ideas? Opinions?

Thanks!

(BTW, I'm already at PyCon :D )


[0] Which I easily tested in a clean Python interpreter:

>>> import locale
>>> locale.getlocale()
(None, None)
>>> import gtk
>>> locale.getlocale()
('es_AR', 'UTF8')

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Authentication & Access Control system for web services

[Allen Bierbaum]

I have been looking into this further and decided on an API that works
as follows:

- Use HTTPS for all requests
- POST to /session to create a new session token
  - pass in username and password as parameters
  - returns token string to be used for all further communication
- All further requests must have the token string which is used to
lookup the user/session
  - on the server, the token will map to a user object to give me
information about their access rights, etc.

Now the question is how does this fit into twisted's view of the
world.  The twisted web in 60 seconds tutorials [1] seem focused on
using HTTP Auth for credential checking and a internal cookie
(TWISTED_SESSION) for session management.  Is there an easy way to
adapt these to my needs or do I need to roll my own code for this type
of twisted.web usage?

-Allen


[1] http://twistedmatrix.com/documents/current/web/howto/web-in-60/index.html

On Mon, Mar 7, 2011 at 9:19 AM, Allen Bierbaum  wrote:
> I have a REST service I have implemented using twisted.web.  Based
> upon a new requirement I need to put role-based access control
> security on the service and am trying to find the most twisted way to
> do it.
>
> I would like to have:
> - Username / password login that is checked against a backend database
> - Roles and associated privileges associated with each user
> - Administration interface to edit users, roles, and privileges
> - "Simple" way to configure the access control requirements on the
> services. (ex: which services need which roles)
>
> Before I role my own code I wanted to check and see if there are any
> addons for this or if anyone else had attacked this problem with
> twisted and had some open source code I could look at.
>
> I have found a couple of projects for WSGI that I may try to pull
> ideas from, but I haven't yet found anything that uses the twisted
> resource model.  (http://authkit.org/,
> http://docs.repoze.org/who/2.0/)
>
> Any pointers to twisted projects I could leverage?
>
> -Allen
>

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Authentication & Access Control system for web services

[George Pauly]

Allen,

In my very limited experience with Twisted,

On Thu, 2011-03-10 at 14:01 -0600, Allen Bierbaum wrote:
> I have been looking into this further and decided on an API that works
> as follows:
> 
> - Use HTTPS for all requests
> - POST to /session to create a new session token
>   - pass in username and password as parameters
>   - returns token string to be used for all further communication

In the non-https case, roll a salt and other items (ip address, user
agent, etc) into a secondary session key on the server.

> - All further requests must have the token string which is used to
> lookup the user/session
>   - on the server, the token will map to a user object to give me
> information about their access rights, etc.
> 

that's all I've ever needed: use the session key (token) to access an
object array - the accessed object has all the twisty magic.

> Now the question is how does this fit into twisted's view of the
> world.  The twisted web in 60 seconds tutorials [1] seem focused on
> using HTTP Auth for credential checking and a internal cookie
> (TWISTED_SESSION) for session management.  Is there an easy way to
> adapt these to my needs or do I need to roll my own code for this type
> of twisted.web usage?

Now you've gone back to credentials - this is outside of my experience
with Twisted.  Sessions are simple enough with Python alone in a twisted
app.  I'll need to use credentials soon so I hope you get an answer. 

Anybody using OpenID or webID instead of login/password?  Could be
better...

> 
> -Allen
> 

George
-- 
George Pauly
Ring Development
www.ringdevelopment.com


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] twisted.web.error.Error & BaseException.message deprecation

[Jason J. W. Williams]

Hi Guys,

Been seeing this error for a long time and finally getting off my bum
to see about fixing it:

twisted/web/error.py:53: DeprecationWarning: BaseException.message has
been deprecated as of Python 2.6

It looks like the issue is the "self.message = message" assignment in
__init__: https://gist.github.com/865097

Before I go about getting rid of the warning, does anyone have a
reason Error.message should stick around?

-J

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Authentication & Access Control system for web services

[Jason J. W. Williams]

I believe this implements OAuth 2 for Twisted using Twisted Cred:

https://github.com/simplegeo/txoauth

-J

On Thu, Mar 10, 2011 at 2:16 PM, George Pauly
 wrote:
> Allen,
>
> In my very limited experience with Twisted,
>
> On Thu, 2011-03-10 at 14:01 -0600, Allen Bierbaum wrote:
>> I have been looking into this further and decided on an API that works
>> as follows:
>>
>> - Use HTTPS for all requests
>> - POST to /session to create a new session token
>>   - pass in username and password as parameters
>>   - returns token string to be used for all further communication
>
> In the non-https case, roll a salt and other items (ip address, user
> agent, etc) into a secondary session key on the server.
>
>> - All further requests must have the token string which is used to
>> lookup the user/session
>>   - on the server, the token will map to a user object to give me
>> information about their access rights, etc.
>>
>
> that's all I've ever needed: use the session key (token) to access an
> object array - the accessed object has all the twisty magic.
>
>> Now the question is how does this fit into twisted's view of the
>> world.  The twisted web in 60 seconds tutorials [1] seem focused on
>> using HTTP Auth for credential checking and a internal cookie
>> (TWISTED_SESSION) for session management.  Is there an easy way to
>> adapt these to my needs or do I need to roll my own code for this type
>> of twisted.web usage?
>
> Now you've gone back to credentials - this is outside of my experience
> with Twisted.  Sessions are simple enough with Python alone in a twisted
> app.  I'll need to use credentials soon so I hope you get an answer.
>
> Anybody using OpenID or webID instead of login/password?  Could be
> better...
>
>>
>> -Allen
>>
>
> George
> --
> George Pauly
> Ring Development
> www.ringdevelopment.com
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Best strategies for pb Referenceables running long methods from callRemote

[Charles Solar]

I am using PB to run remote methods in a testing system at my company.  The
code works very well but breaks down when I start running multiple tests at
once.  I have tracked this down to overflowing the thread pool on the remote
machines.  I am wondering if anyone might have better suggestions for
running long methods from a remote method.

I coded up a sample of what I am seeing here: http://pastebin.com/rBPp20Ms

Basically I have 1 server that calls remote_execute on many clients on a
remote server.  This remote_execute method starts a new method using
threads.deferToThread and returns the defer to make the server's callRemote
defer wait until the remote long method end.
What I do in those methods is run test code that waits, blocks, sleeps, and
all sorts of nasty things that make the thread take a while.  In the example
code I simply sleep for 20 seconds.

The problem I see with this code specifically is that I run out of threads
on the pool and even though I wanted all execute methods to run at the same
time, I see 10 run, then 10 more, then 10 more.. etc.  The testing depends
on all these methods being run at the same time as they run mechanisms that
depend on each other and need everyone running.  When I overflow the thread
pool some methods do not run until other methods stop, which makes the whole
test fail.

I am not holding the GIL or blocking the reactor, which was the first thing
I checked.

Setting reactor.suggestThreadPoolSize(50) does help, but I do not think its
the best solution, and does not work very well on our slow and older
machines.

Any feedback is appreciated
Charles
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Failing tests in trunk

[exarkun]

On 10 Mar, 07:39 pm, facundobati...@gmail.com wrote:

On Tue, Mar 8, 2011 at 7:04 AM, Facundo Batista
 wrote:

None of these are known failures:

� �http://buildbot.twistedmatrix.com/boxes-supported

Just looking at the list of failing tests, I would make a small bet 
that

the failures are caused by your locale setting somehow.


I thought of that, but why would the test pass when running alone?

I'll debug this.


Ok, found it.

See one example of the test that failed,
twisted.mail.test.test_imap.NewFetchTestCase.testFetchInternalDate:

   twisted.trial.unittest.FailTest: not equal:
   a = {0: {'INTERNALDATE': '02-nov-2003 21:25:10 +'},
1: {'INTERNALDATE': '29-dic-2013 11:31:52 -0500'},
2: {'INTERNALDATE': '10-mar-1992 02:44:30 -0600'},
3: {'INTERNALDATE': '11-ene-2000 14:40:24 -0800'}}
   b = {0: {'INTERNALDATE': '02-Nov-2003 21:25:10 +'},
1: {'INTERNALDATE': '29-Dec-2013 11:31:52 -0500'},
2: {'INTERNALDATE': '10-Mar-1992 02:44:30 -0600'},
3: {'INTERNALDATE': '11-Jan-2000 14:40:24 -0800'}}

See the difference? "nov" instead of "Nov", etc. "dic" instead of
"Dec". Yes, in Spanish December is 'diciembre'.

So, funny 'locale' issue, confirmed by showing getlocale() in the
test... when running the whole suite, the locale is ('es_AR', 'UTF8'),
and when running twisted.mail the locale is (None, None).

Weird, specially taking into account that 'locale' is not used in the
whole twisted project.

But I tracked it down.

In twisted/manhole/ui/test/test_gtk2manhole.py, gtk is imported, and
that changes the locale [0].


Aha, nicely done.  I figured the locale change would be in Python 
somewhere, I didn't consider it might be hidden in Gtk.

So, how this can be fixed?

Is ok for the IMAP4Server to spew the message internal date according
to the locale in which it's being run? Or it should always format the
dates in an specific way, because some definition somewhere? (I don't
know much about IMAP)


IMAP specifies the strings in the unit test.  So the implementation 
really needs to change to not depend on the locale.

If it's ok for the server to format the dates according to the locale,
I should make the test more robust. If not, the server should set the
locale to (None, None) for the date formatting, or just set it at
__init__ and leave it there, or use other formatting (not
time.strftime, I prefer this solution).


I agree with your preference, switching away from time.strftime is 
probably the right thing to do.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted.web.error.Error & BaseException.message deprecation

[Glyph Lefkowitz]

On Mar 10, 2011, at 5:45 PM, Jason J. W. Williams wrote:

> Hi Guys,
> 
> Been seeing this error for a long time and finally getting off my bum
> to see about fixing it:
> 
> twisted/web/error.py:53: DeprecationWarning: BaseException.message has
> been deprecated as of Python 2.6
> 
> It looks like the issue is the "self.message = message" assignment in
> __init__: https://gist.github.com/865097
> 
> Before I go about getting rid of the warning, does anyone have a
> reason Error.message should stick around?

This is the attribute used (in some cases) to relay the protocol-level error 
message printed in the status line area of the HTTP response.  So yes, we need 
to keep it; it doesn't mean the same thing as Python's earlier 'message' 
attribute on Exception.  If we can simply squash the warning that would be best.



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted.web.error.Error & BaseException.message deprecation

[Jason J. W. Williams]

Setting it as a class level attribute seems to suppress it:

>>> class TestError(Exception):
... message = ""
...
... def __init__(self, msg):
...self.message = msg

Since it's a string and passed by value I think this would work. I'll
open up a ticket if one's not already.

-J


On Thu, Mar 10, 2011 at 10:12 PM, Glyph Lefkowitz
 wrote:
> On Mar 10, 2011, at 5:45 PM, Jason J. W. Williams wrote:
>
>> Hi Guys,
>>
>> Been seeing this error for a long time and finally getting off my bum
>> to see about fixing it:
>>
>> twisted/web/error.py:53: DeprecationWarning: BaseException.message has
>> been deprecated as of Python 2.6
>>
>> It looks like the issue is the "self.message = message" assignment in
>> __init__: https://gist.github.com/865097
>>
>> Before I go about getting rid of the warning, does anyone have a
>> reason Error.message should stick around?
>
> This is the attribute used (in some cases) to relay the protocol-level error 
> message printed in the status line area of the HTTP response.  So yes, we 
> need to keep it; it doesn't mean the same thing as Python's earlier 'message' 
> attribute on Exception.  If we can simply squash the warning that would be 
> best.
>
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python