Hi all, I was hoping someone might have an idea here.. I have a number of nginx doing load balancing sitting behind AWS's network load balancers (TCP) - which seem to only support TCP checks.
Recently a few have stopped working / frozen - they still seem to accept a tcp connection from the NLB - which leads the health check not to fail. But they cannot internally process the request and you cannot even ssh into the machine. A reboot is required and that takes longer than normal. I think the failure is related to a disk issue since the only error in the entire logs where regarding the disk. (error logs below) Ideally if nginx or the O/S fails it would be better if the port just closed. I've considered writing a small daemon that monitors via http locally and keeps a port open if everything is ok. These machines have been running for months now without any issues until now. Anyone have an idea? Thanks! ---- [4161960.544106] INFO: task jbd2/xvda1-8:271 blocked for more than 120 seconds. [4161960.551035] Not tainted 4.4.0-1022-aws #31-Ubuntu [4161960.556118] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [4161960.562846] INFO: task monit:13224 blocked for more than 120 seconds. [4161960.567394] Not tainted 4.4.0-1022-aws #31-Ubuntu [4161960.571120] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [4162080.576076] INFO: task dhclient:696 blocked for more than 120 seconds. [4162080.579596] Not tainted 4.4.0-1022-aws #31-Ubuntu [4162080.582355] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [4162080.586470] INFO: task monit:13224 blocked for more than 120 seconds. [4162080.589847] Not tainted 4.4.0-1022-aws #31-Ubuntu [4162080.592654] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [4162200.596100] INFO: task jbd2/xvda1-8:271 blocked for more than 120 seconds. [4162200.599646] Not tainted 4.4.0-1022-aws #31-Ubuntu [4162200.602422] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [4162200.606423] INFO: task dhclient:696 blocked for more than 120 seconds. [4162200.610118] Not tainted 4.4.0-1022-aws #31-Ubuntu [4162200.613093] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [4162200.617889] INFO: task monit:13224 blocked for more than 120 seconds. [4162200.621641] Not tainted 4.4.0-1022-aws #31-Ubuntu [4162200.624506] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [4162244.551431] systemd[1]: Failed to start Journal Service. [4162320.628099] INFO: task jbd2/xvda1-8:271 blocked for more than 120 seconds. [4162320.631942] Not tainted 4.4.0-1022-aws #31-Ubuntu [4162320.635012] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [4162320.639647] INFO: task dhclient:696 blocked for more than 120 seconds. [4162320.643241] Not tainted 4.4.0-1022-aws #31-Ubuntu [4162320.646233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [4162320.650712] INFO: task monit:13224 blocked for more than 120 seconds. [4162320.654190] Not tainted 4.4.0-1022-aws #31-Ubuntu [4162320.657183] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [4162334.801390] systemd[1]: Failed to start Journal Service. [4162425.051503] systemd[1]: Failed to start Journal Service. [4162515.301393] systemd[1]: Failed to start Journal Service.
_______________________________________________ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
