HI Rainer/David/Marisuz, Could you please help me with creating one input rule with a queue as Marisuz suggested. I was failing to create a rule.
If I change anything in my 90-inputs.conf file do I need to make any changes in 30-rules.conf??? @Rainer : I upgraded to the latest version of rsyslog and tested. But still my rsyslog restarts continuously. It was not stable. Thanks & Regards Vijay Kumar Kanukula On Fri, 17 Jun 2022 at 12:55, Rainer Gerhards <rgerha...@hq.adiscon.com> wrote: > it would be good to find the root cause. The best would be to use the > current 8.2206.0 version and see if it works. If so, all is fine. If > not, we should try to debug the issue. > > Rainer > > El jue, 16 jun 2022 a las 16:22, vijay kumar via rsyslog > (<rsyslog@lists.adiscon.com>) escribió: > > > > Hi Team, > > > > My rsyslog service is getting restarted very frequently and we understand > > it is due to race between the various threads, which causes one thread to > > free a message field while another tries to read/write it. > > > > log: > > === > > ==3035157==ERROR: AddressSanitizer: heap-buffer-overflow on address > > 0x61e000000ac2 at pc 0x7f6085ac62fd bp 0x7f6079d755e0 sp 0x7f6079d74d88 > > READ of size 2627 at 0x61e000000ac2 thread T3 (in:imjournal) > > #0 0x7f6085ac62fc (/lib64/libasan.so.5+0xb92fc) > > #1 0x7f607d5066a5 in readJSONfromJournalMsg imjournal.c:288 > > #2 0x7f607d5066a5 in readjournal imjournal.c:497 > > #3 0x55ae0523366e in thrdStarter ../threads.c:243 > > #4 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce) > > #5 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82) > > > > 0x61e000000ac2 is located 0 bytes to the right of 2626-byte region > > [0x61e000000080,0x61e000000ac2) > > allocated by thread T3 (in:imjournal) here: > > #0 0x7f6085afcfe8 in __interceptor_realloc > (/lib64/libasan.so.5+0xeffe8) > > #1 0x7f6084af1a8b (/lib64/libsystemd.so.0+0x82a8b) > > > > Thread T3 (in:imjournal) created by T0 here: > > #0 0x7f6085a5fea3 in __interceptor_pthread_create > > (/lib64/libasan.so.5+0x52ea3) > > #1 0x55ae05234470 in thrdCreate ../threads.c:289 > > > > SUMMARY: AddressSanitizer: heap-buffer-overflow > > (/lib64/libasan.so.5+0xb92fc) > > Shadow bytes around the buggy address: > > 0x0c3c7fff8100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > > 0x0c3c7fff8110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > > 0x0c3c7fff8120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > > 0x0c3c7fff8130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > > 0x0c3c7fff8140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > > =>0x0c3c7fff8150: 00 00 00 00 00 00 00 00[02]fa fa fa fa fa fa fa > > 0x0c3c7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > > 0x0c3c7fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > > 0x0c3c7fff8180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > > 0x0c3c7fff8190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > > 0x0c3c7fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > > Shadow byte legend (one shadow byte represents 8 application bytes): > > Addressable: 00 > > Partially addressable: 01 02 03 04 05 06 07 > > Heap left redzone: fa > > Freed heap region: fd > > Stack left redzone: f1 > > Stack mid redzone: f2 > > Stack right redzone: f3 > > Stack after return: f5 > > Stack use after scope: f8 > > Global redzone: f9 > > Global init order: f6 > > Poisoned by user: f7 > > Container overflow: fc > > Array cookie: ac > > Intra object redzone: bb > > ASan internal: fe > > Left alloca redzone: ca > > Right alloca redzone: cb > > ================================================================= > > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address > > 0x60c0005d85c0 at pc 0x7f6085a4db27 bp 0x7f6070621e50 sp 0x7f60706215f8 > > READ of size 128 at 0x60c0005d85c0 thread T9 (rs:main Q:Reg) > > #0 0x7f6085a4db26 (/lib64/libasan.so.5+0x40b26) > > #1 0x55ae0516d061 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x390061) > > #2 0x55ae0516e522 in MsgDup msg.c:1129 > > #3 0x55ae05205d58 in execCall ruleset.c:290 > > #4 0x55ae05205d58 in scriptExec ruleset.c:608 > > #5 0x55ae05206532 in execIf ruleset.c:313 > > #6 0x55ae05206532 in scriptExec ruleset.c:614 > > #7 0x55ae05208868 in processBatch ruleset.c:660 > > #8 0x55ae050ae50c in msgConsumer rsyslogd.c:694 > > #9 0x55ae051f100d in ConsumerReg queue.c:2145 > > #10 0x55ae051e0804 in wtiWorker wti.c:428 > > #11 0x55ae051d9dd5 in wtpWorker wtp.c:435 > > #12 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce) > > #13 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82) > > > > 0x60c0005d85c0 is located 0 bytes inside of 128-byte region > > [0x60c0005d85c0,0x60c0005d8640) > > freed by thread T6 (imudp(w1)) here: > > #0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0) > > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471 > > > > previously allocated by thread T6 (imudp(w1)) here: > > #0 0x7f6085afcba8 in __interceptor_malloc > (/lib64/libasan.so.5+0xefba8) > > #1 0x55ae0516cff4 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x38fff4) > > > > Thread T9 (rs:main Q:Reg) created by T3 (in:imjournal) here: > > #0 0x7f6085a5fea3 in __interceptor_pthread_create > > (/lib64/libasan.so.5+0x52ea3) > > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497 > > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570 > > > > Thread T6 (imudp(w1)) created by T0 here: > > #0 0x7f6085a5fea3 in __interceptor_pthread_create > > (/lib64/libasan.so.5+0x52ea3) > > #1 0x55ae05234470 in thrdCreate ../threads.c:289 > > > > SUMMARY: AddressSanitizer: heap-use-after-free > (/lib64/libasan.so.5+0x40b26) > > Shadow bytes around the buggy address: > > 0x0c18800b3060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa > > 0x0c18800b3070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > > 0x0c18800b3080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd > > 0x0c18800b3090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa > > 0x0c18800b30a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > > =>0x0c18800b30b0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd > > 0x0c18800b30c0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa > > 0x0c18800b30d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > > 0x0c18800b30e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > > 0x0c18800b30f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > > 0x0c18800b3100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > > Shadow byte legend (one shadow byte represents 8 application bytes): > > Addressable: 00 > > Partially addressable: 01 02 03 04 05 06 07 > > Heap left redzone: fa > > Freed heap region: fd > > Stack left redzone: f1 > > Stack mid redzone: f2 > > Stack right redzone: f3 > > Stack after return: f5 > > Stack use after scope: f8 > > Global redzone: f9 > > Global init order: f6 > > Poisoned by user: f7 > > Container overflow: fc > > Array cookie: ac > > Intra object redzone: bb > > ASan internal: fe > > Left alloca redzone: ca > > Right alloca redzone: cb > > ================================================================= > > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address > > 0x6040000285a0 at pc 0x55ae0520a723 bp 0x7f60669659a0 sp 0x7f6066965990 > > WRITE of size 4 at 0x6040000285a0 thread T16 (rs:qradar.local) > > #0 0x55ae0520a722 in propDestruct prop.c:63 > > #1 0x55ae05170299 in MsgSetRcvFromIPWithoutAddRef msg.c:457 > > #2 0x55ae05170299 in resolveDNS msg.c:522 > > #3 0x55ae0517064a in getRcvFromIP msg.c:558 > > #4 0x55ae05179317 in MsgGetProp (/usr/sbin/rsyslogd+0x39c317) > > #5 0x55ae0524aece in tplToString ../template.c:207 > > #6 0x55ae0522a4ab in prepareDoActionParams ../action.c:1114 > > #7 0x55ae0522a4ab in processMsgMain ../action.c:1648 > > #8 0x55ae0522c279 in doSubmitToActionQ ../action.c:1825 > > #9 0x55ae05205826 in execAct ruleset.c:209 > > #10 0x55ae05205826 in scriptExec ruleset.c:599 > > #11 0x55ae05208868 in processBatch ruleset.c:660 > > #12 0x55ae050ae50c in msgConsumer rsyslogd.c:694 > > #13 0x55ae051f100d in ConsumerReg queue.c:2145 > > #14 0x55ae051e0804 in wtiWorker wti.c:428 > > #15 0x55ae051d9dd5 in wtpWorker wtp.c:435 > > #16 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce) > > #17 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82) > > > > 0x6040000285a0 is located 16 bytes inside of 48-byte region > > [0x604000028590,0x6040000285c0) > > freed by thread T16 (rs:qradar.local) here: > > #0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0) > > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471 > > > > previously allocated by thread T16 (rs:qradar.local) here: > > #0 0x7f6085afcdb0 in calloc (/lib64/libasan.so.5+0xefdb0) > > #1 0x55ae0520a439 in propConstruct prop.c:56 > > > > Thread T16 (rs:qradar.local) created by T9 (rs:main Q:Reg) here: > > #0 0x7f6085a5fea3 in __interceptor_pthread_create > > (/lib64/libasan.so.5+0x52ea3) > > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497 > > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570 > > > > SUMMARY: AddressSanitizer: heap-use-after-free prop.c:63 in propDestruct > > Shadow bytes around the buggy address: > > 0x0c087fffd060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > > 0x0c087fffd070: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa > > 0x0c087fffd080: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fd > > 0x0c087fffd090: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa > > 0x0c087fffd0a0: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa > > =>0x0c087fffd0b0: fa fa fd fd[fd]fd fd fd fa fa fa fa fa fa fa fa > > 0x0c087fffd0c0: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa > > 0x0c087fffd0d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd > > 0x0c087fffd0e0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa > > 0x0c087fffd0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa > > 0x0c087fffd100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd > > Shadow byte legend (one shadow byte represents 8 application bytes): > > Addressable: 00 > > Partially addressable: 01 02 03 04 05 06 07 > > Heap left redzone: fa > > Freed heap region: fd > > Stack left redzone: f1 > > Stack mid redzone: f2 > > Stack right redzone: f3 > > Stack after return: f5 > > Stack use after scope: f8 > > Global redzone: f9 > > Global init order: f6 > > Poisoned by user: f7 > > Container overflow: fc > > Array cookie: ac > > Intra object redzone: bb > > ASan internal: fe > > Left alloca redzone: ca > > Right alloca redzone: cb > > ==3035157==ABORTING > > > > ================================= > > what is the possible solution ??? > > > > would be to have multiple rsyslog instances, which is possible if the > > traffic is split between ports. If yes could you please suggest how to > > configure?? > > > > Thanks & Regards > > Vijay Kumar Kanukula > > _______________________________________________ > > rsyslog mailing list > > https://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
