I guess it works because journal always throws messages away if it cannot deliver them quickly. Luke a very short timeout+drop queue config in rsyslog.
Rainer Sent from phone, thus brief. David Lang <da...@lang.hm> schrieb am Do., 21. Sept. 2023, 08:23: > now you have journald acting as a queue, so all messages from journald > will end > up delayed when your script cannot keep up. You haven't solved the problem > of > the slow script, you've just added another layer of buffer to fill up > before you > notice. > > with rsyslog you can set the queue size to whatever you want, and you can > spill > logs to disk when your queue fills up. > > but no matter what you do, if you have something that is processing logs > slower > than they are being generated, eventually you will run out of queue space > (in > memory or on disk) and have to stop accepting new messages, or start > throwing > away messages you haven't processed yet > > David Lang > > On Thu, 21 Sep 2023, TG Servers via rsyslog wrote: > > > the only way I was able to fix this was to use a dedicated socket > > created via systemd and passed via systemd to rsyslog > > since then it is working without any issues. > > although I implemented a queue, too, this did not fix the problem as > > long as the socket was handled by rsyslog itself > > so this is "fixed" from my point of view, I know for the future now > > > > On 18/09/2023 21:53, TG Servers via rsyslog wrote: > >> I don't know what this is... I implemented a complete queue solution > >> and it occasionally happens when there is no request but one in sight, > >> and this one gets a 111 then, nothing in nginx debug log, no error to > >> be seen in rsyslog log > >> but one thing I realized, after a restart the first log message > >> always, reproducable gets a 111 > >> the socket is not connected, nor listening, only after the first > >> request is logged/or not logged (which is logged with 111 in nginx) > >> the socket is connected and listening, so restarting rsyslog via > >> systemd does not connect/listen to/on the socket > >> > >> the rsyslog debug log just tells us this : > >> 6289.088037540:main thread : imuxsock.c: imuxsock: Opened UNIX > >> socket '/run/logmat' (fd 6). > >> > >> [root@xxx rsyslog.d]# systemctl restart rsyslog > >> [root@xxx rsyslog.d]# ss -x | grep logmat > >> [root@xxx rsyslog.d]# lsof /run/logmat > >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME > >> rsyslogd 2097140 root 6u unix 0x0000000000000000 0t0 25300317 > >> /run/logmat type=DGRAM (UNCONNECTED) > >> > >> make a request from browser or curl > >> > >> [root@xxx rsyslog.d]# lsof /run/logmat > >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME > >> rsyslogd 2097140 root 6u unix 0x0000000000000000 0t0 25300317 > >> /run/logmat type=DGRAM (CONNECTED) > >> [root@xxx rsyslog.d]# ss -x | grep logmat > >> u_dgr ESTAB 0 0 /run/logmat 25300317 * 0 > >> > >> On 18/09/2023 16:34, TG Servers via rsyslog wrote: > >>> I just wanted to add that in a further message as it came to my mind. > >>> you were faster... > >>> the script is definitely "slow", this is what I know for sure as it > >>> does quite a lot of processing/analytics in the background, so even > >>> if you trigger it from command line it can take half a sec or so.... > >>> I can't change that, it needs to do what it does, I didn't write it > >>> though it can handle manual fast F5 triggers in the browser without > >>> issue and then it 111s when there are 2 requests incoming... > >>> I thought rsyslog might handle that just well via the queue... > >>> but then this might eventually really be the issue, and if it is, is > >>> there anything to mitigate this from rsyslog side (in terms of own > >>> queue for that socket or something in that direction)? > >>> ok, will enable impstats, too when I switch back > >>> > >>> Thanks, > >>> Tom > >>> > >>> On 18/09/2023 16:17, Rainer Gerhards wrote: > >>>>> so far not a single 111 today, I let this run the until late evening, > >>>>> and if there is stil no 111 I will put back the python script in > order > >>>>> because right now there are 2 possibilities, I moved the socket as > >>>>> said, > >>>>> and I skipped the script and just appended the message to a file > >>>>> if either of the 2 things are responsible in the end I won't > >>>>> understand > >>>>> it either :) > >>>> I don't know what the script does. But if it is slow, it may push back > >>>> to the main queue, making rsyslog unresponsive. > >>>> > >>>> This is David's concern. Tomorrow, if you re-enable, you should also > >>>> enable impstats as David suggested. > >>>> > >>>> Rainer > >>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> https://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > >>> POST if you DON'T LIKE THAT. > >> > >> _______________________________________________ > >> rsyslog mailing list > >> https://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > >> if you DON'T LIKE THAT. > > > > _______________________________________________ > > rsyslog mailing list > > https://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > > LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
