Em 07-03-2014 20:40, Michael Friedrich escreveu: > On 07.03.2014 20:48, Carlos Cesario wrote: >> I think that I found a important information in the debug log >> >> [2014-03-07 13:02:04 -0300] <Q #0x7f932233bf18 W #0x7f932233c078> >> warning/icinga: Exception occured during check for service >> 'Node-1-Master!ping4': ../../../lib/base/process-unix.cpp(124): Throw >> in function icinga::ProcessResult icinga::Process::Run() >> Dynamic exception type: >> boost::exception_detail::clone_impl<icinga::posix_error> >> std::exception::what: std::exception >> [icinga::StackTrace*] = >> (0) libbase.so: void >> boost::throw_exception<icinga::posix_error>(icinga::posix_error >> const&) (+0xa6) [0x7f93219d8066] (RVA: 548966) >> (1) libbase.so: void >> boost::exception_detail::throw_exception_<icinga::posix_error>(icinga::posix_error >> const&, char const*, char const*, int) (+0x3c) [0x7f93219d810c] (RVA: >> 549132) >> (2) libbase.so: icinga::Process::Run() (+0x1332) >> [0x7f9321a0edb2] (RVA: 773554) >> (3) libmethods.so: >> icinga::PluginCheckTask::ScriptFunc(boost::shared_ptr<icinga::Service> >> const&) >> (+0x78e) [0x7f931ea5b19e] (RVA: 65950) >> (4) libmethods.so: icinga::Value >> icinga::ScriptFunctionWrapperR<boost::shared_ptr<icinga::CheckResult>, >> boost::shared_ptr<icinga::Service> >> const&>(boost::shared_ptr<icinga::CheckResult> >> (*)(boost::shared_ptr<icinga::Service> const&), >> std::vector<icinga::Value, std::allocator<icinga::Value> > const&) >> (+0x64) [0x7f931ea5a274] (RVA: 62068) >> (5) libmethods.so: >> boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<icinga::Value, >> icinga::Value (*)(boost::shared_ptr<icinga::CheckResult> >> (*)(boost::shared_ptr<icinga::Service> const&), >> std::vector<icinga::Value, std::allocator<icinga::Value> > const&), >> boost::_bi::list2<boost::_bi::value<boost::shared_ptr<icinga::CheckResult> >> (*)(boost::shared_ptr<icinga::Service> const&)>, boost::arg<1> > >, >> icinga::Value, std::vector<icinga::Value, >> std::allocator<icinga::Value> > >> const&>::invoke(boost::detail::function::function_buffer&, >> std::vector<icinga::Value, std::allocator<icinga::Value> > const&) >> (+0xd) [0x7f931ea57b0d] (RVA: 51981) >> (6) libbase.so: >> icinga::ScriptFunction::Invoke(std::vector<icinga::Value, >> std::allocator<icinga::Value> > const&) (+0x1c) [0x7f9321a14d7c] >> (RVA: 798076) >> (7) libbase.so: >> icinga::DynamicObject::InvokeMethod(icinga::String const&, >> std::vector<icinga::Value, std::allocator<icinga::Value> > const&) >> (+0x121) [0x7f93219f34b1] (RVA: 660657) >> (8) libicinga.so: >> icinga::CheckCommand::Execute(boost::shared_ptr<icinga::Service> >> const&) (+0x16d) [0x7f931ed15bed] (RVA: 711661) >> (9) libicinga.so: icinga::Service::ExecuteCheck() (+0x265) >> [0x7f931edaa885] (RVA: 1321093) >> (10) libchecker.so: >> icinga::CheckerComponent::ExecuteCheckHelper(boost::shared_ptr<icinga::Service> >> const&) (+0x19) [0x7f931b7dffc9] (RVA: 217033) >> (11) libbase.so: >> icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) >> (+0x39c) [0x7f9321a4025c] (RVA: 975452) >> (12) libboost_thread.so.1.46.1: thread_proxy (+0x69) >> [0x7f93209eece9] (RVA: 52457) >> (13) libpthread.so.0: <unknown function> (+0x7e9a) >> [0x7f9321f2be9a] (RVA: 32410) >> (14) libc.so.6: clone (+0x6d) [0x7f9320ef33fd] (RVA: 1000445) >> >> >> [icinga::ContextTrace*] = >> (0) Executing service check for service 'ping4' on host >> 'Node-1-Master' >> >> [boost::errinfo_errno_*] = 11, "Resource temporarily unavailable" >> [boost::errinfo_api_function_*] = fork >> >> Context: >> (0) Executing service check for service 'ping4' on host >> 'Node-1-Master' > > # ulimit -a root@localhost:/var/src/icinga2-next# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 23938 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 23938 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
> > # lsof -c icinga2 > root@localhost:/var/src/icinga2-next# lsof -c icinga COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME icinga2 14952 nagios cwd DIR 252,0 4096 2 / icinga2 14952 nagios rtd DIR 252,0 4096 2 / icinga2 14952 nagios txt REG 252,0 224992 1336867 /usr/sbin/icinga2 icinga2 14952 nagios mem REG 252,0 881128 1325171 /usr/lib/x86_64-linux-gnu/icinga2/libcluster.so icinga2 14952 nagios mem REG 252,0 384424 1336855 /usr/lib/x86_64-linux-gnu/icinga2/libchecker.so icinga2 14952 nagios mem REG 252,0 18282384 1310657 /usr/lib/libicudata.so.48.1.1 icinga2 14952 nagios mem REG 252,0 1866528 1310660 /usr/lib/libicui18n.so.48.1.1 icinga2 14952 nagios mem REG 252,0 1465096 1310661 /usr/lib/libicuuc.so.48.1.1 icinga2 14952 nagios mem REG 252,0 1054456 1319990 /usr/lib/libboost_regex.so.1.46.1 icinga2 14952 nagios mem REG 252,0 1118984 1324836 /usr/lib/x86_64-linux-gnu/icinga2/liblivestatus.so icinga2 14952 nagios mem REG 252,0 1040640 1336853 /usr/lib/x86_64-linux-gnu/icinga2/libdb_ido.so icinga2 14952 nagios mem REG 252,0 3412232 1325338 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0 icinga2 14952 nagios mem REG 252,0 343776 1312811 /usr/lib/x86_64-linux-gnu/icinga2/libdb_ido_mysql.so icinga2 14952 nagios mem REG 252,0 265352 1325194 /usr/lib/x86_64-linux-gnu/icinga2/libnotification.so icinga2 14952 nagios mem REG 252,0 733848 1325192 /usr/lib/x86_64-linux-gnu/icinga2/libcompat.so icinga2 14952 nagios mem REG 252,0 117712 1325191 /usr/lib/x86_64-linux-gnu/icinga2/libmethods.so icinga2 14952 nagios mem REG 252,0 1995440 1324903 /usr/lib/x86_64-linux-gnu/icinga2/libicinga.so icinga2 14952 nagios mem REG 252,0 52120 134550 /lib/x86_64-linux-gnu/libnss_files-2.15.so icinga2 14952 nagios mem REG 252,0 47680 134554 /lib/x86_64-linux-gnu/libnss_nis-2.15.so icinga2 14952 nagios mem REG 252,0 97248 134566 /lib/x86_64-linux-gnu/libnsl-2.15.so icinga2 14952 nagios mem REG 252,0 35680 134548 /lib/x86_64-linux-gnu/libnss_compat-2.15.so icinga2 14952 nagios mem REG 252,0 92720 131044 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 icinga2 14952 nagios mem REG 252,0 1030512 134557 /lib/x86_64-linux-gnu/libm-2.15.so icinga2 14952 nagios mem REG 252,0 6120 1336825 /usr/lib/x86_64-linux-gnu/icinga2/libexecvpe.so icinga2 14952 nagios mem REG 252,0 5992 1336863 /usr/lib/x86_64-linux-gnu/icinga2/libmmatch.so icinga2 14952 nagios mem REG 252,0 22560 1336857 /usr/lib/x86_64-linux-gnu/icinga2/libcJSON.so icinga2 14952 nagios mem REG 252,0 1930616 130851 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 icinga2 14952 nagios mem REG 252,0 382896 134575 /lib/x86_64-linux-gnu/libssl.so.1.0.0 icinga2 14952 nagios mem REG 252,0 101304 1319768 /usr/lib/libboost_thread.so.1.46.1 icinga2 14952 nagios mem REG 252,0 14768 134561 /lib/x86_64-linux-gnu/libdl-2.15.so icinga2 14952 nagios mem REG 252,0 1815224 130841 /lib/x86_64-linux-gnu/libc-2.15.so icinga2 14952 nagios mem REG 252,0 88384 130860 /lib/x86_64-linux-gnu/libgcc_s.so.1 icinga2 14952 nagios mem REG 252,0 962656 1313155 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16 icinga2 14952 nagios mem REG 252,0 511704 1324738 /usr/lib/x86_64-linux-gnu/icinga2/libconfig.so icinga2 14952 nagios mem REG 252,0 1356864 1324739 /usr/lib/x86_64-linux-gnu/icinga2/libbase.so icinga2 14952 nagios mem REG 252,0 414280 1319971 /usr/lib/libboost_program_options.so.1.46.1 icinga2 14952 nagios mem REG 252,0 135366 134555 /lib/x86_64-linux-gnu/libpthread-2.15.so icinga2 14952 nagios mem REG 252,0 149280 134558 /lib/x86_64-linux-gnu/ld-2.15.so icinga2 14952 nagios 0u CHR 1,3 0t0 1029 /dev/null icinga2 14952 nagios 1u CHR 1,3 0t0 1029 /dev/null icinga2 14952 nagios 2w REG 252,0 38 1046568 /var/log/icinga2/icinga2.err icinga2 14952 nagios 4w REG 252,0 3725186477 1046641 /var/log/icinga2/debug.log icinga2 14952 nagios 5w REG 252,0 5856 1073775 /var/log/icinga2/compat/icinga.log icinga2 14952 nagios 6u unix 0xffff8800ba093a80 0t0 3244825 /var/run/icinga2/cmd/livestatus icinga2 14952 nagios 8w REG 252,0 32223081 1048732 /var/log/icinga2/icinga2.log icinga2 14952 nagios 9w REG 252,0 817552 1199375 /var/lib/icinga2/cluster/log/current icinga2 14952 nagios 10u IPv6 3244827 0t0 TCP *:8888 (LISTEN) icinga2 14952 nagios 11u IPv6 3223398 0t0 TCP 192.168.15.233:8888->192.168.15.232:43046 (ESTABLISHED) icinga2 14952 nagios 12wW REG 0,15 5 3223399 /run/icinga2/icinga2.pid icinga2 14952 nagios 13r FIFO 0,8 0t0 3223400 pipe icinga2 14952 nagios 14u unix 0xffff8800b9903740 0t0 3223403 socket icinga2 14952 nagios 15r FIFO 0,8 0t0 3244829 pipe icinga2 14952 nagios 16r FIFO 0,8 0t0 3244830 pipe icinga2 14952 nagios 17r FIFO 0,8 0t0 3244831 pipe icinga2 14952 nagios 18r FIFO 0,8 0t0 3244832 pipe icinga2 14952 nagios 19r FIFO 0,8 0t0 3244833 pipe icinga2 14952 nagios 20r FIFO 0,8 0t0 3244834 pipe icinga2 14952 nagios 21r FIFO 0,8 0t0 3244835 pipe icinga2 14952 nagios 22r FIFO 0,8 0t0 3244836 pipe .... root@localhost:/var/src/icinga2-next# lsof -c icinga2 | wc -l 16419 >> >> >> >> >> Em 07-03-2014 16:42, Carlos Cesario escreveu: >>> After 2 hours >>> >>> ....... >>> 2736 ? Z 0:00 [sh] <defunct> >>> 32738 ? Z 0:00 [sh] <defunct> >>> 32740 ? Z 0:00 [sh] <defunct> >>> 32742 ? Z 0:00 [sh] <defunct> >>> 32744 ? Z 0:00 [sh] <defunct> >>> 32746 ? Z 0:00 [sh] <defunct> >>> 32748 ? Z 0:00 [sh] <defunct> >>> 32750 ? Z 0:00 [sh] <defunct> >>> 32752 ? Z 0:00 [sh] <defunct> >>> 32754 ? Z 0:00 [sh] <defunct> >>> 32756 ? Z 0:00 [sh] <defunct> >>> 32758 ? Z 0:00 [sh] <defunct> >>> 32760 ? Z 0:00 [sh] <defunct> >>> 32762 ? Z 0:00 [sh] <defunct> >>> 32764 ? Z 0:00 [sh] <defunct> >>> 32766 ? Z 0:00 [sh] <defunct> >>> root@localhost:/var/src/icinga2-next# ps ax | grep "\\[sh\\] >>> <defunct>" | wc -l >>> *16299* >>> root@localhost:/var/src/icinga2-next# >>> >>> root@localhost:/var/src/icinga2-next# ps ax | grep -i icinga >>> *10895* ? Ssl 1:19 /usr/sbin/icinga2 -c >>> /etc/icinga2/icinga2.conf -u nagios -g nagios -d -e >>> /var/log/icinga2/icinga2.err >>> root@pandoranew:/var/src/icinga2-next# pstree -a *10895* | more >>> icinga2 -c /etc/icinga2/icinga2.conf -u nagios -g nagios -d -e >>> /var/log/icinga2/icinga2.err >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> |-(sh) >>> ......... >>> >>> Em 07-03-2014 08:15, Carlos Cesario escreveu: >>>> Hello, after the last upgrade form icinga2 (by >>>> icinga-precise-snapshots) >>>> I'm getting a lot of sh process. >>>> >>>> >>>> 23951 ? Ssl 0:00 /usr/sbin/icinga2 -c >>>> /etc/icinga2/icinga2.conf -u nagios -g nagios -d -e >>>> /var/log/icinga2/icinga2.err >>>> 24003 ? Z 0:00 [sh] <defunct> >>>> 24005 ? Z 0:00 [sh] <defunct> >>>> 24007 ? Z 0:00 [sh] <defunct> >>>> 24009 ? Z 0:00 [sh] <defunct> >>>> 24011 ? Z 0:00 [sh] <defunct> >>>> 24013 ? Z 0:00 [sh] <defunct> >>>> 24015 ? Z 0:00 [sh] <defunct> >>>> 24017 ? Z 0:00 [sh] <defunct> >>>> 24019 ? Z 0:00 [sh] <defunct> >>>> 24021 ? Z 0:00 [sh] <defunct> >>>> 24023 ? Z 0:00 [sh] <defunct> >>>> 24025 ? Z 0:00 [sh] <defunct> >>>> 24027 ? Z 0:00 [sh] <defunct> >>>> 24029 ? Z 0:00 [sh] <defunct> >>>> 24031 ? Z 0:00 [sh] <defunct> >>>> 24033 ? Z 0:00 [sh] <defunct> >>>> ...... >>>> >>>> >>>> At around 600 processes. >>>> >>>> But I don't nothing in log or debug files. >>>> >>>> Somebody has any idea ? >>>> >>>> >>>> # icinga2 -V >>>> icinga2 - The Icinga 2 network monitoring daemon. (Version: >>>> v0.0.7-48-g3b8c1a0) >>>> >>>> >>>> >>>> Greats >>>> >>>> -- Carlos >>>> >>>> >>>> _______________________________________________ >>>> icinga-users mailing list >>>> icinga-users@lists.icinga.org >>>> https://lists.icinga.org/mailman/listinfo/icinga-users >>>> >>> >>> >>> -- >>> >>> >>> >>> _______________________________________________ >>> icinga-users mailing list >>> icinga-users@lists.icinga.org >>> https://lists.icinga.org/mailman/listinfo/icinga-users >> >> >> -- >> >> >> >> _______________________________________________ >> icinga-users mailing list >> icinga-users@lists.icinga.org >> https://lists.icinga.org/mailman/listinfo/icinga-users > > -- _______________________________________________ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users
