Nginx version: 1.13.6.1
1) In our use case, the Nginx is reloaded constantly. you will see lots
worker process hanging at "nginx: worker process is shutting down" after
couple days:
58 root 0:00 nginx: master process ./openresty/nginx/sbin/nginx
-p /opt/applicatio
1029 nobody 0:22 nginx: worker process is shutting down
1030 nobody 0:27 nginx: worker process is shutting down
1041 nobody 0:54 nginx: worker process is shutting down
1054 nobody 0:37 nginx: worker process is shutting down
1131 nobody 0:02 nginx: worker process is shutting down
1132 nobody 0:02 nginx: worker process is shutting down
1215 nobody 0:50 nginx: worker process is shutting down
1216 nobody 0:53 nginx: worker process is shutting down
1515 nobody 0:23 nginx: worker process is shutting down
1516 nobody 0:47 nginx: worker process is shutting down
1533 nobody 0:03 nginx: worker process is shutting down
1534 nobody 0:16 nginx: worker process is shutting down
1598 nobody 0:00 nginx: worker process
1599 nobody 0:00 nginx: worker process
2) And if you now check some listen port on the host using netstat, you will
see it is owned by the worker process:
[root@paas-controller-177-1-1-137:~]$ netstat -anp |grep 10080
tcp 0 0 0.0.0.0:10080 0.0.0.0:* LISTEN
6288/nginx: worker
tcp 0 0 10.47.205.136:10080 10.47.205.137:50827
ESTABLISHED 6296/nginx: worker
tcp 0 0 10.47.205.136:10080 10.47.205.137:50833
ESTABLISHED 6300/nginx: worker
tcp 0 0 10.47.205.136:10080 10.47.205.137:49411
ESTABLISHED 6296/nginx: worker
tcp 0 0 10.47.205.136:10080 10.40.157.154:54074
ESTABLISHED 6296/nginx: worker
tcp 0 0 10.47.205.136:10080 10.47.205.137:49715
ESTABLISHED 6299/nginx: worker
tcp6 0 0 :::10080 :::* LISTEN
6288/nginx: worker
3) So far, you would say it is not a big deal as long as the worker could
serve the request correctly. but it does NOT. suppose the listen port 10080
above would proxy the request to the upstream server A initially (and for
sure this port was listening by the master process), days later, it has been
changed to the server B and of course the Nginx was been reloaded at the
same time (and perhaps some worker was left in " is shutting down" state at
that time). well now, I see this port is listening by a worker process, and
it would proxy the request to the old server A.
I suspected this could be caused by the "shutting down" worker processes, so
I "kill -9" all these ones and try again, but nothing changed. even
reloading the Nginx did no help. I shut it down and rerun the binary,
finally this worked and the request can be proxyed to the server B.
I guess maybe some cache in Nginx was doing the bad, any clue?
Thanks,
Allen
Posted at Nginx Forum:
https://forum.nginx.org/read.php?2,283700,283856#msg-283856
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
