Re: [Twisted-Python] Setting socket options before connect

[Phil Mayers]

On 17/07/13 18:44, Glyph wrote:


Anyhow, I think this certainly means a ticket should be filed for
introducing some API - but it seems that a little more discussion
about what the API should be will still be necessary.


Agreed.  But then, it's been a couple days and nobody else has
contributed to this thread :-).


Well, since I started it...

Some kind of endpoint argument might be problematic for some use-cases. 
In particular, Twisted would have to know how to convert the argument 
into the value to pass into setsockopt() and possibly in a 
platform-specific way.


The API as presented also omits the SOL.

I guess it might be ok if there was a way to reliably inject unknown 
options with arbitrary payloads, but I'm struggling to see a clean way 
to do this with a "parse a string"-style API.


So, vote me +0.5 for a "pre-connect" function.

But but but...

It might be possible to sidestep this entire issue by providing a clean 
way for an app to inject itself "above" the socket. I can think of a few 
use-cases for this, most notably things like cmsg/IP_HDRINCL which 
Twisted doesn't know about, and thus can't handle.


So maybe the correct way to handle this is "implement your own FD object"?

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Setting socket options before connect

[exarkun]

On 10:27 am, p.may...@imperial.ac.uk wrote:

On 17/07/13 18:44, Glyph wrote:

Anyhow, I think this certainly means a ticket should be filed for
introducing some API - but it seems that a little more discussion
about what the API should be will still be necessary.


Agreed.  But then, it's been a couple days and nobody else has
contributed to this thread :-).


Well, since I started it...

Some kind of endpoint argument might be problematic for some use-cases. 
In particular, Twisted would have to know how to convert the argument 
into the value to pass into setsockopt() and possibly in a platform- 
specific way.


The API as presented also omits the SOL.


It wasn't so much an API as a "Hey, I have an idea... endpoints... here 
is an example I can think of in 10 seconds." :)
I guess it might be ok if there was a way to reliably inject unknown 
options with arbitrary payloads, but I'm struggling to see a clean way 
to do this with a "parse a string"-style API.


I think you're talking about the fact that "sockopts" are random 
integers associated with other big piles of random integers.  Some of 
them are flags you turn on, but some come with random payloads of 
basically no possible known shape.


It sounds like you're trying to think of an API that will support any 
and all socket options without understanding them.  This API exists 
already.  It is `setsockopt`.  There's no reason to re-invent it.


I prefer the approach taken elsewhere in Twisted, where a particular 
option is given some consideration and an API that understands the 
option is introduced.  This approach certainly has its shortcomings - 
for example, it doesn't support arbitrary options. :)  Do people really 
like using `setsockopt` though?


So, vote me +0.5 for a "pre-connect" function.

But but but...

It might be possible to sidestep this entire issue by providing a clean 
way for an app to inject itself "above" the socket. I can think of a 
few use- cases for this, most notably things like cmsg/IP_HDRINCL which 
Twisted doesn't know about, and thus can't handle.


So maybe the correct way to handle this is "implement your own FD 
object"?


Nothing stops anyone from doing this already, today.  Except that it's a 
lot of work and no one seems to want any of these features badly enough 
to do it.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Setting socket options before connect

[Phil Mayers]

On 18/07/13 13:10, exar...@twistedmatrix.com wrote:


It wasn't so much an API as a "Hey, I have an idea... endpoints... here
is an example I can think of in 10 seconds." :)


Understood!


I guess it might be ok if there was a way to reliably inject unknown
options with arbitrary payloads, but I'm struggling to see a clean way
to do this with a "parse a string"-style API.


I think you're talking about the fact that "sockopts" are random
integers associated with other big piles of random integers.  Some of
them are flags you turn on, but some come with random payloads of
basically no possible known shape.


Yep.


It sounds like you're trying to think of an API that will support any
and all socket options without understanding them.  This API exists
already.  It is `setsockopt`.  There's no reason to re-invent it.


Sure.


I prefer the approach taken elsewhere in Twisted, where a particular
option is given some consideration and an API that understands the
option is introduced.  This approach certainly has its shortcomings -
for example, it doesn't support arbitrary options. :)  Do people really
like using `setsockopt` though?


No-one "likes" it, but if you need to set a socket option, you don't 
really have much choice (absent a higher-layer API).



So maybe the correct way to handle this is "implement your own FD
object"?


Nothing stops anyone from doing this already, today.  Except that it's a
lot of work and no one seems to want any of these features badly enough
to do it.


Well, exactly. It's currently easier for me to sub-class a Twisted 
internal/private class. But if that goes away, that option is 
unavailable and a replacement would be nice - even if it's a convenience 
API like:


connectExistingSocket(...)

Isn't there code to do this as part of the systemd activation?

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Best way to trigger a future connection with data

[Nick Johnson]

Hi,

I'm porting some code to twisted and got a little stuck. The current
(non-twisted) code connects to a server, gives it some data (a job to
do), gets a job-id and then terminates the connection. Sometime later,
it reconnects and gets the output using the job-id.

I've a factory and protocol set up, with buildProtocol over-riden in the
Factory class to supply some extra data to each Protocol instance when
it's created. This is fine when I submit a job as I can call a factory
function to generate the required data (ie the job to do). My problem is
how to initiate a connection at some point in the future AND pass it
some specific info (ie the job-id) for it to pass to the server.

I start the connections using task.callingLoop and reactor.connectTCP so
my first guess was to add: task.callLater(time, reactor.connectTCP, ...)
to the connectionLost method of my Protocol. However, I cannot see how
to pass in any extra information (ie the job-id) this way? I know I can
store state in the Factory, so I guess I just need to know how to pass
this to a scheduled invocation of the Protocol.

Is anyone able to point me in the right direction here?

Cheers,
-Nick.
-- 
Nick Johnson, Applications Developer, EPCC
2407 JCMB, King's Buildings, Mayfield Road, Edinburgh, EH9 3JF
e: nick.john...@ed.ac.uk t: 0131 651 3388

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Setting socket options before connect

[exarkun]

On 01:09 pm, p.may...@imperial.ac.uk wrote:

On 18/07/13 13:10, exar...@twistedmatrix.com wrote:

I prefer the approach taken elsewhere in Twisted, where a particular
option is given some consideration and an API that understands the
option is introduced.  This approach certainly has its shortcomings -
for example, it doesn't support arbitrary options. :)  Do people 
really

like using `setsockopt` though?


No-one "likes" it, but if you need to set a socket option, you don't 
really have much choice (absent a higher-layer API).


Glad to hear it is universally reviled. ;)

To be a bit more explicit, I think it's worth considering particular 
socket options individually and introducing high-level APIs for dealing 
with them.  They don't get requested too often, so even though it's kind 
of labor intensive it still might not be overwhelming.

So maybe the correct way to handle this is "implement your own FD
object"?


Nothing stops anyone from doing this already, today.  Except that it's 
a
lot of work and no one seems to want any of these features badly 
enough

to do it.


Well, exactly. It's currently easier for me to sub-class a Twisted 
internal/private class. But if that goes away, that option is 
unavailable and a replacement would be nice - even if it's a 
convenience API like:


connectExistingSocket(...)

Isn't there code to do this as part of the systemd activation?


I really wanted to suggest something along those lines.  The current 
code has the limitation that it will only accept an *already* connected 
socket.  There isn't something that will accept a socket and then do the 
connection part for you.


Maybe `connectExistingSocket` could still leverage this code somehow to 
simplify its implementation, though.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] SQL ORM for Twisted & PostgreSQL?

[Glyph]

On Jul 17, 2013, at 5:04 PM, Jeffrey Ollie  wrote:

> I know that this has been asked before, but it's been a while and I'm
> hoping for some good news.  Is there a SQL ORM that works well with
> Twisted and PostgreSQL?  In particular I'm hoping to find something
> that works with txpostgres as that's the library I prefer to access
> the database with.

There's bits of one in Calendar Server: 

  It doesn't use txpostgres, but that's because it relies on adbapi2 (also in 
calendar server, hopefully to be moved into Twisted at some point).

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Best way to trigger a future connection with data

[Laurens Van Houtven]

Hi Nick,

You're pretty much there already. Instantiate a ClientFactory that holds
all the necessary state. By default, your protocol will have access to that
state through its factory attribute (unless you override the Factory's
buildProtocol method).

cheers
lvh
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] SQL ORM for Twisted & PostgreSQL

[Matt Haggard]

> On Jul 17, 2013, at 5:04 PM, Jeffrey Ollie  wrote:

>
> > I know that this has been asked before, but it's been a while and I'm
> > hoping for some good news.  Is there a SQL ORM that works well with
> > Twisted and PostgreSQL?  In particular I'm hoping to find something
> > that works with txpostgres as that's the library I prefer to access
> > the database with.
>
> There's bits of one in Calendar Server: 
> 
>   It doesn't use txpostgres, but that's because it relies on adbapi2 (also in 
> calendar server, hopefully to be moved into Twisted at some point).
>
> -glyph

I've made a thing that has a similar API to txpostgres and will use
txpostgres underneath if it's installed:

https://github.com/iffy/norm

I welcome feedback/patches.

- Matt Haggard

(I wasn't on the mailing list when the original message was sent, so
this "reply" may not appear quite correctly)

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python