[Twisted-Python] Twisted and Thread, thread not running when twisted app is running as service

[Sereysethy TOUCH]

Hello,

I am developing a twisted app, and it runs as a service using twistd -y to
start the app.
I am having a problem of running a thread. I know it is not recommended to
use thread, but the library that I use, the object created is running in a
thread.

Here is the problem:

1) if I start reactor by running reactor.run() directly, thread is running
fine
2) if I run it as a service using twisted, the thread is not running, it
runs but it seems to be blocked, because I tried to write something to file
using time.sleep(), but file is empty.

Is there something that I miss? How can I debug this?

Thank you,

TS
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted and Thread, thread not running when twisted app is running as service

[Jean-Paul Calderone]

On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH <
touch.sereyse...@gmail.com> wrote:

> Hello,
>
> I am developing a twisted app, and it runs as a service using twistd -y to
> start the app.
> I am having a problem of running a thread. I know it is not recommended to
> use thread, but the library that I use, the object created is running in a
> thread.
>
> Here is the problem:
>
> 1) if I start reactor by running reactor.run() directly, thread is running
> fine
> 2) if I run it as a service using twisted, the thread is not running, it
> runs but it seems to be blocked, because I tried to write something to file
> using time.sleep(), but file is empty.
>
> Is there something that I miss? How can I debug this?
>

A good next step would be to create at Short, Self Contained, Correct
(Compilable), Example  and share it.

Jean-Paul


>
> Thank you,
>
> TS
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted and Thread, thread not running when twisted app is running as service

[Sereysethy TOUCH]

Hi Jean-Paul,

I found the error after working on a short, self contained, correcte
example. It is not about twisted and normal thread in general, but
something else.

After more tests, I come to realise that whenever I instantiate this class
PolicyClient from this library
https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py in
my protocol or just anywhere, twisted app either freezes or terminates, and
this *only* happens when the app runs as a daemon, created using a service
(.tac file).

I hope you can take a look at the class PolicyClient, why it causes problem
to reactor main loop. Please point me to where the idea might come from.

I can give you the example but not sure if it is enough, as the client
needs to connect to a server.

Thank you,
TS


On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone <
exar...@twistedmatrix.com> wrote:

> On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH <
> touch.sereyse...@gmail.com> wrote:
>
>> Hello,
>>
>> I am developing a twisted app, and it runs as a service using twistd -y
>> to start the app.
>> I am having a problem of running a thread. I know it is not recommended
>> to use thread, but the library that I use, the object created is running in
>> a thread.
>>
>> Here is the problem:
>>
>> 1) if I start reactor by running reactor.run() directly, thread is
>> running fine
>> 2) if I run it as a service using twisted, the thread is not running, it
>> runs but it seems to be blocked, because I tried to write something to file
>> using time.sleep(), but file is empty.
>>
>> Is there something that I miss? How can I debug this?
>>
>
> A good next step would be to create at Short, Self Contained, Correct
> (Compilable), Example  and share it.
>
> Jean-Paul
>
>
>>
>> Thank you,
>>
>> TS
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
import time
import threading

from twisted.internet import reactor
from twisted.application import service
from twisted.python import failure, log
from twisted.internet.protocol import Factory, Protocol
from twisted.internet.endpoints import TCP4ServerEndpoint
import logging

# from ray.rllib.env.policy_client import PolicyClient

logging.basicConfig(filename='rllib.log')
logger = logging.getLogger(__name__)
logger.setLevel("INFO")

class MyThread(threading.Thread):
def __init__(self,stop=False):
threading.Thread.__init__(self)
self.stop = stop

def run(self):
while not self.stop:
time.sleep(2)
logger.info('thread is alive')

class Echo(Protocol):
def __init__(self):
self.t = MyThread()
self.t.daemon = True
self.t.start()
# this will freeze everything
# self.policyClient = PolicyClient("http://localhost:9900";)

# def connectionMade(self):
# self.policyClient = PolicyClient("http://localhost:9900";)

def dataReceived(self, data):
self.transport.write(data)

def connectionLost(self, reason):
self.t.stop = True

class EchoFactory(Factory):
def buildProtocol(self, addr):
return Echo()

class EchoService(service.Service):

def startService(self):
self._port = reactor.listenTCP(8080, EchoFactory())

def stopService(self):
return self._port.stopListening()

if __name__ == "__main__":
endpoint = TCP4ServerEndpoint(reactor, 8080)
endpoint.listen(EchoFactory())
reactor.run()

service.tac
Description: Binary data
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted and Thread, thread not running when twisted app is running as service

[Jean-Paul Calderone]

On Thu, May 21, 2020 at 3:56 PM Sereysethy TOUCH 
wrote:

> Hi Jean-Paul,
>
> I found the error after working on a short, self contained, correcte
> example. It is not about twisted and normal thread in general, but
> something else.
>
> After more tests, I come to realise that whenever I instantiate this class
> PolicyClient from this library
> https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py in
> my protocol or just anywhere, twisted app either freezes or terminates, and
> this *only* happens when the app runs as a daemon, created using a
> service (.tac file).
>
> I hope you can take a look at the class PolicyClient, why it causes
> problem to reactor main loop. Please point me to where the idea might come
> from.
>
> I can give you the example but not sure if it is enough, as the client
> needs to connect to a server.
>

It looks like PolicyClient does a blocking HTTP request in __init__.  Since
PolicyClient is instantiated in the reactor thread, this will block the
reactor.  Perhaps this call is hanging for some reason when the process has
daemonized?  This would be unusual but not unheard of.  For example, there
are macOS environments where a process does not have access to the network
if it is not associated with a windowing session.

What platform do you observe the problem on, and what more can you learn
about what the process does on its way to hanging (eg can you run strace on
it)?

Jean-Paul


>
> Thank you,
> TS
>
>
> On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
>
>> On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH <
>> touch.sereyse...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I am developing a twisted app, and it runs as a service using twistd -y
>>> to start the app.
>>> I am having a problem of running a thread. I know it is not recommended
>>> to use thread, but the library that I use, the object created is running in
>>> a thread.
>>>
>>> Here is the problem:
>>>
>>> 1) if I start reactor by running reactor.run() directly, thread is
>>> running fine
>>> 2) if I run it as a service using twisted, the thread is not running, it
>>> runs but it seems to be blocked, because I tried to write something to file
>>> using time.sleep(), but file is empty.
>>>
>>> Is there something that I miss? How can I debug this?
>>>
>>
>> A good next step would be to create at Short, Self Contained, Correct
>> (Compilable), Example  and share it.
>>
>> Jean-Paul
>>
>>
>>>
>>> Thank you,
>>>
>>> TS
>>> ___
>>> Twisted-Python mailing list
>>> Twisted-Python@twistedmatrix.com
>>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted and Thread, thread not running when twisted app is running as service

[Sereysethy TOUCH]

I am on MacOS, it is a development phase, but deployment will be on Linux.

It is hard to run dtrace/dtruss on MacOS due to the "system integrity
protection" on MacOS. I got some output but very limited information.

So if there is a workaround you might suggest?

TS

On Thu, May 21, 2020 at 10:20 PM Jean-Paul Calderone <
exar...@twistedmatrix.com> wrote:

> On Thu, May 21, 2020 at 3:56 PM Sereysethy TOUCH <
> touch.sereyse...@gmail.com> wrote:
>
>> Hi Jean-Paul,
>>
>> I found the error after working on a short, self contained, correcte
>> example. It is not about twisted and normal thread in general, but
>> something else.
>>
>> After more tests, I come to realise that whenever I instantiate this
>> class PolicyClient from this library
>> https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py in
>> my protocol or just anywhere, twisted app either freezes or terminates, and
>> this *only* happens when the app runs as a daemon, created using a
>> service (.tac file).
>>
>> I hope you can take a look at the class PolicyClient, why it causes
>> problem to reactor main loop. Please point me to where the idea might come
>> from.
>>
>> I can give you the example but not sure if it is enough, as the client
>> needs to connect to a server.
>>
>
> It looks like PolicyClient does a blocking HTTP request in __init__.
> Since PolicyClient is instantiated in the reactor thread, this will block
> the reactor.  Perhaps this call is hanging for some reason when the process
> has daemonized?  This would be unusual but not unheard of.  For example,
> there are macOS environments where a process does not have access to the
> network if it is not associated with a windowing session.
>
> What platform do you observe the problem on, and what more can you learn
> about what the process does on its way to hanging (eg can you run strace on
> it)?
>
> Jean-Paul
>
>
>>
>> Thank you,
>> TS
>>
>>
>> On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone <
>> exar...@twistedmatrix.com> wrote:
>>
>>> On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH <
>>> touch.sereyse...@gmail.com> wrote:
>>>
 Hello,

 I am developing a twisted app, and it runs as a service using twistd -y
 to start the app.
 I am having a problem of running a thread. I know it is not recommended
 to use thread, but the library that I use, the object created is running in
 a thread.

 Here is the problem:

 1) if I start reactor by running reactor.run() directly, thread is
 running fine
 2) if I run it as a service using twisted, the thread is not running,
 it runs but it seems to be blocked, because I tried to write something to
 file using time.sleep(), but file is empty.

 Is there something that I miss? How can I debug this?

>>>
>>> A good next step would be to create at Short, Self Contained, Correct
>>> (Compilable), Example  and share it.
>>>
>>> Jean-Paul
>>>
>>>

 Thank you,

 TS
 ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com
 https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

>>> ___
>>> Twisted-Python mailing list
>>> Twisted-Python@twistedmatrix.com
>>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python