[Twisted-Python] Logging and SpawnProcess

[Robert DiFalco]

Can anyone point me to some sample code for handling logging with
SpawnProcess? Right now I'm using an EchoProtocol that takes stdout and
resends it to the log. But it's kludgy because some sub-processes use
stdin/stderr for marshaling data. What I'd most like to do is separate data
transfer from log output in a clean way.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Logging and SpawnProcess

[Tom Most]

I'm not entirely following the question (is the log output intermixed with 
other data on stdout?) but I think that you might find the implementation of 
twisted.internet.endpoints.ProcessEndpoint a useful example. It relays stderr 
to the logging stream: 
https://github.com/twisted/twisted/blob/c8064075a207af1fc9ce19d8f885a986bc5ab00c/src/twisted/internet/endpoints.py#L336-L479

---Tom

On Wed, Dec 30, 2020, at 11:38 AM, Robert DiFalco wrote:
> Can anyone point me to some sample code for handling logging with 
> SpawnProcess? Right now I'm using an EchoProtocol that takes stdout and 
> resends it to the log. But it's kludgy because some sub-processes use 
> stdin/stderr for marshaling data. What I'd most like to do is separate data 
> transfer from log output in a clean way. 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Logging and SpawnProcess

[Glyph]

One minor point, if this is within your power: don't marshal data over stdout; 
sometimes libraries scribble on it.  Open an arbitrary additional file 
descriptor - say, 7 - and put your marshalled data there.

Definitely never use stderr for marshaling anything but lines of text - you're 
not gonna get away with that :-).  Even well-behaved libraries sometimes dump 
random text to stderr, that's what it's there for, so you don't do it with 
stdout!

Subunit has some really interesting ideas here - it's not Twisted code, but it 
does deal with communicating structured data over stdio streams in such a way 
that random overzealous SSH MOTD messages won't ruin your day:

https://github.com/testing-cabal/subunit/tree/0e9f67b9683bf2c8c82edb4e511d9b2c7708d900#version-2
 


Hope this is helpful,

-g

> On Dec 30, 2020, at 9:12 PM, Tom Most  wrote:
> 
> I'm not entirely following the question (is the log output intermixed with 
> other data on stdout?) but I think that you might find the implementation of 
> twisted.internet.endpoints.ProcessEndpoint a useful example. It relays stderr 
> to the logging stream: 
> https://github.com/twisted/twisted/blob/c8064075a207af1fc9ce19d8f885a986bc5ab00c/src/twisted/internet/endpoints.py#L336-L479
>  
> 
> 
> ---Tom
> 
> On Wed, Dec 30, 2020, at 11:38 AM, Robert DiFalco wrote:
>> Can anyone point me to some sample code for handling logging with 
>> SpawnProcess? Right now I'm using an EchoProtocol that takes stdout and 
>> resends it to the log. But it's kludgy because some sub-processes use 
>> stdin/stderr for marshaling data. What I'd most like to do is separate data 
>> transfer from log output in a clean way. 
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com 
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python 
>> 
>> 
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com 
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python 
> 
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Logging and SpawnProcess

[Robert DiFalco]

This was very helpful thanks!

On Wed, Dec 30, 2020 at 10:08 PM Glyph  wrote:

> One minor point, if this is within your power: don't marshal data over
> stdout; sometimes libraries scribble on it.  Open an arbitrary additional
> file descriptor - say, 7 - and put your marshalled data there.
>
> Definitely *never* use stderr for marshaling anything but lines of text -
> you're not gonna get away with that :-).  Even well-behaved libraries
> sometimes dump random text to stderr, that's what it's there for, so you
> don't do it with stdout!
>
> Subunit has some really interesting ideas here - it's not Twisted code,
> but it does deal with communicating structured data over stdio streams in
> such a way that random overzealous SSH MOTD messages won't ruin your day:
>
>
> https://github.com/testing-cabal/subunit/tree/0e9f67b9683bf2c8c82edb4e511d9b2c7708d900#version-2
>
> Hope this is helpful,
>
> -g
>
>
> On Dec 30, 2020, at 9:12 PM, Tom Most  wrote:
>
> I'm not entirely following the question (is the log output intermixed with
> other data on stdout?) but I think that you might find the implementation
> of twisted.internet.endpoints.ProcessEndpoint a useful example. It relays
> stderr to the logging stream:
> https://github.com/twisted/twisted/blob/c8064075a207af1fc9ce19d8f885a986bc5ab00c/src/twisted/internet/endpoints.py#L336-L479
>
> ---Tom
>
> On Wed, Dec 30, 2020, at 11:38 AM, Robert DiFalco wrote:
>
> Can anyone point me to some sample code for handling logging with
> SpawnProcess? Right now I'm using an EchoProtocol that takes stdout and
> resends it to the log. But it's kludgy because some sub-processes use
> stdin/stderr for marshaling data. What I'd most like to do is separate data
> transfer from log output in a clean way.
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python