Hi Lidi, Thanks for looking into it. I've now submitted an issue on Github: https://github.com/grpc/grpc/issues/18626
Thanks, Alex. On Tue, 2 Apr 2019 at 19:04, Lidi Zheng <[email protected]> wrote: > Hi Alex, > > I'm glad to see the problem is finally going away. Sorry that I can't > track down this issue in Windows. > And there are many possibilities to cause that behavior, might related to > underlying file descriptors, the garbage collections, fork+exec issue. > I don't have an answer. > > In addition, the details you posted here is more than enough to file a bug > report. > It would be great if you can submit an issue in Github where we can > revisit this issue when we got resources. > https://github.com/grpc/grpc/issues/new > > Thanks, > Lidi Zheng > > > > On Tue, Apr 2, 2019 at 3:38 AM Alejandro Villagrán <[email protected]> > wrote: > >> Hi, >> >> The subprocess is finishing OK, almost immediately as it's not doing >> anything apart from printing a message to the console. "subprocess.call" >> returns a status code 0. So everything should be fine on that side. >> Not 100% sure what you meant with tracing the lifecycle of the channel. I >> called grpc.Channel.subscribe to print the different states of the channel >> used: >> >> *def stateChange(conn):* >> * print conn* >> >> *class GRPCStub(object):* >> * def __init__(self):* >> * self.channel = grpc.insecure_channel('127.0.0.1:50051 >> <http://127.0.0.1:50051>')* >> * self.channel.subscribe(stateChange)* >> * self.stub = Test_pb2_grpc.GRPCTestStub(self.channel)* >> >> With that call to subscribe, the exception is no longer thrown. The >> channel starts IDLE and then changes to READY as soon as the first grpc >> call is made and stays like that for the rest of the program. >> >> It'd be interesting if you could reproduce it on your end to see what's >> really happening under the bonnet. Why would that call to >> grpc.Channel.subscribe will prevent the exception from being raised? >> >> Thanks, >> Alex. >> >> On Mon, 1 Apr 2019 at 22:32, Lidi Zheng <[email protected]> wrote: >> >>> Hi Alex, >>> >>> From the log, I found the error is located at client-side. The gRPC >>> client sends reset stream frame with error code 2 indicating internal >>> error, and its channel state somehow becomes SHUTDOWN. >>> Also, I tried your code in Linux, it works fine without any error. I had >>> another failed attempt to build the VS solution with 400+ errors... >>> >>> Although the root cause still remained unclear, the scope is reduced to >>> the gRPC client behavior on Windows. To dig deeper into the bug, I think >>> one have to trace if the "subprocess" finishes and the life cycle of >>> channel. >>> >>> Lidi Zheng >>> >>> On Mon, Apr 1, 2019 at 7:55 AM Alejandro Villagrán <[email protected]> >>> wrote: >>> >>>> Hi Lidi, >>>> >>>> I've set those environment variables you mentioned and I've attached >>>> the log files (serverlog.txt and clientlog.txt). >>>> Since I reported this issue, I've upgrade the Python version of gRPC, >>>> so the attached log files were created with grpcio 1.19.0 >>>> >>>> Do they give you any hint as to what could be wrong? >>>> >>>> Thanks, >>>> Alex. >>>> >>>> On Thu, 28 Mar 2019 at 17:07, Alejandro Villagrán <[email protected]> >>>> wrote: >>>> >>>>> Hi Lidi, >>>>> >>>>> In the last version of the code I attached, I don't use CherryPy >>>>> anymore so the issue cannot come from there. Eric said that fork handlers >>>>> are not registered on Windows so he ruled out that option too. >>>>> Unfortunately I haven't got access to a Linux machine and I'm not >>>>> familiar enough with Linux to be able to recreate the issue there. >>>>> >>>>> I did use turn on the debug trace a few weeks ago but I couldn't see >>>>> anything obvious. I'll find time to enable them again and share the logs >>>>> with you. >>>>> >>>>> Thanks, >>>>> Alex. >>>>> >>>>> On Thu, 28 Mar 2019 at 16:55, Lidi Zheng <[email protected]> wrote: >>>>> >>>>>> My apologies Alex. There are other stuff keep come up and consumed my >>>>>> time. I'm not a Windows expert, it will take me a long time to setup the >>>>>> compilation environment for gRPC in Windows with debugger. And I failed >>>>>> to >>>>>> find a Windows expert to debug your issue. >>>>>> Eric has mentioned that the breakage can be caused by either fork >>>>>> handlers registration or CherryPy. Do you think you can migrate the >>>>>> reproduce case to Linux, if the root cause is the software? It would be >>>>>> much easier to debug. >>>>>> Also, have you tried to turn on the debug trace in gRPC by setting >>>>>> environmental >>>>>> variables >>>>>> <https://github.com/grpc/grpc/blob/master/doc/environment_variables.md> >>>>>> "*GRPC_VERBOSITY*" to "*DEBUG*", and "*GRPC_TRACE*" to " >>>>>> *api,channel,connectivity_state*"? They might produce useful >>>>>> information for us to identify the problem. >>>>>> >>>>>> Thanks, >>>>>> Lidi Zheng >>>>>> >>>>>> On Thu, Mar 28, 2019 at 3:47 AM Alejandro Villagrán < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Lidi, >>>>>>> >>>>>>> Did you manage to reproduce the issue? >>>>>>> >>>>>>> Thanks, >>>>>>> Alex. >>>>>>> >>>>>>> On Tue, 19 Mar 2019 at 01:06, Lidi Zheng <[email protected]> wrote: >>>>>>> >>>>>>>> Hi Alex, >>>>>>>> >>>>>>>> Thank you for providing the reproduce code. I will spin up a >>>>>>>> Windows machine to investigate this error. >>>>>>>> If I'm able to find something useful, I'll let you know. >>>>>>>> >>>>>>>> Lidi Zheng >>>>>>>> >>>>>>>> On Mon, Mar 18, 2019 at 4:09 AM Alejandro Villagrán < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Eric/Lidi, >>>>>>>>> >>>>>>>>> Yes, I'm running on Windows. I have now removed the CherryPy code >>>>>>>>> and I still get the exception. >>>>>>>>> >>>>>>>>> Please follow these steps to reproduce the issue: >>>>>>>>> - Unzip ReproduceGRPCIssue.zip >>>>>>>>> - Go to the BreakGRPC folder and compile BreakGRPC.sln. Make sure >>>>>>>>> BreakGRPC.exe is saved in BreakGRPC/x64/Release. >>>>>>>>> - Go to the Services/src folder and open two command prompts there. >>>>>>>>> - Run "python GRPCserver.py" in one command prompt. >>>>>>>>> - Run "python GRPCclient.py" in the other command prompt. >>>>>>>>> >>>>>>>>> You should see the exception on the client command prompt. >>>>>>>>> >>>>>>>>> Please let me know if you are still unable to reproduce the issue >>>>>>>>> with this version of the code. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Alex. >>>>>>>>> >>>>>>>>> On Mon, 4 Mar 2019 at 17:04, Eric Gribkoff < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> +Lidi Zheng <[email protected]>, who will be available for any >>>>>>>>>> follow-up questions (it will be easier for him to notice your >>>>>>>>>> questions if >>>>>>>>>> you include his email address on the "to:" line) >>>>>>>>>> >>>>>>>>>> Hi Alex, >>>>>>>>>> >>>>>>>>>> Sorry for the delay. I was not able to reproduce the problem; it >>>>>>>>>> looks like you are running on Windows, in which case gRPC's fork >>>>>>>>>> handlers >>>>>>>>>> are not registered/run, so those shouldn't be the cause here . Since >>>>>>>>>> the >>>>>>>>>> reproduction example also uses CherryPy websockets, it's quite >>>>>>>>>> possible the >>>>>>>>>> issue stems from that software rather than the gRPC stack - we'd >>>>>>>>>> likely >>>>>>>>>> need a reproduction case that only uses gRPC, without the >>>>>>>>>> websockets, to be >>>>>>>>>> able to help debug this further. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Eric >>>>>>>>>> >>>>>>>>>> On Mon, Mar 4, 2019 at 2:36 AM Alex <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Eric, >>>>>>>>>>> >>>>>>>>>>> Just wondering if you had time to run my attached example and >>>>>>>>>>> managed to reproduce the problem? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Alex. >>>>>>>>>>> >>>>>>>>>>> On Wednesday, February 20, 2019 at 7:04:51 PM UTC, Eric Gribkoff >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Can you post the code you're using to reproduce this error? If >>>>>>>>>>>> you're using subprocess.Popen (or otherwise using fork+exec) to >>>>>>>>>>>> start the >>>>>>>>>>>> C++ grpc client process, the C++ client itself cannot be >>>>>>>>>>>> interfering with >>>>>>>>>>>> the Python process. Something could be going wrong in the gRPC >>>>>>>>>>>> core fork >>>>>>>>>>>> handlers, however - you can try running with the environment >>>>>>>>>>>> variable >>>>>>>>>>>> `GRPC_ENABLE_FORK_SUPPORT=0` to disable this feature and see if it >>>>>>>>>>>> fixes >>>>>>>>>>>> the issue. >>>>>>>>>>>> >>>>>>>>>>>> Also, in your step 5 you note that the C++ client isn't >>>>>>>>>>>> communicating with the server. If you remove the fork+exec of a C++ >>>>>>>>>>>> subprocess altogether, do you still see this intermittent >>>>>>>>>>>> exception in the >>>>>>>>>>>> Python client? >>>>>>>>>>>> >>>>>>>>>>>> Eric >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Feb 20, 2019 at 6:57 AM Alex <[email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I should add that the Python client application which owns the >>>>>>>>>>>>> Python grpc client is the one that runs the C++ grpc client as a >>>>>>>>>>>>> subprocess >>>>>>>>>>>>> in case that makes a difference. >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "grpc.io" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>>>>>> Visit this group at https://groups.google.com/group/grpc-io. >>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>> https://groups.google.com/d/msgid/grpc-io/b323fac3-978b-47c1-b1fa-555c2f62b544%40googlegroups.com >>>>>>>>>>>>> <https://groups.google.com/d/msgid/grpc-io/b323fac3-978b-47c1-b1fa-555c2f62b544%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>> . >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "grpc.io" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>>>> Visit this group at https://groups.google.com/group/grpc-io. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/grpc-io/af4d55fa-a5e0-4e3f-a5ad-9cb62378703d%40googlegroups.com >>>>>>>>>>> <https://groups.google.com/d/msgid/grpc-io/af4d55fa-a5e0-4e3f-a5ad-9cb62378703d%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAKQGxBD0v9Q5Mt_%3DTUNtsdnP5z%3DwEwFgidZ6d_Fuap93J5WxmQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
