can you please post rsyslog -v output as well as the current ASAN report? Thanks, Rainer
El vie, 17 jun 2022 a las 10:12, vijay kumar (<vijay.kanuk...@gmail.com>) escribió: > > HI Rainer/David/Marisuz, > > Could you please help me with creating one input rule with a queue as Marisuz > suggested. I was failing to create a rule. > > If I change anything in my 90-inputs.conf file do I need to make any changes > in 30-rules.conf??? > > @Rainer : I upgraded to the latest version of rsyslog and tested. But still > my rsyslog restarts continuously. It was not stable. > > Thanks & Regards > Vijay Kumar Kanukula > > > > > > On Fri, 17 Jun 2022 at 12:55, Rainer Gerhards <rgerha...@hq.adiscon.com> > wrote: >> >> it would be good to find the root cause. The best would be to use the >> current 8.2206.0 version and see if it works. If so, all is fine. If >> not, we should try to debug the issue. >> >> Rainer >> >> El jue, 16 jun 2022 a las 16:22, vijay kumar via rsyslog >> (<rsyslog@lists.adiscon.com>) escribió: >> > >> > Hi Team, >> > >> > My rsyslog service is getting restarted very frequently and we understand >> > it is due to race between the various threads, which causes one thread to >> > free a message field while another tries to read/write it. >> > >> > log: >> > === >> > ==3035157==ERROR: AddressSanitizer: heap-buffer-overflow on address >> > 0x61e000000ac2 at pc 0x7f6085ac62fd bp 0x7f6079d755e0 sp 0x7f6079d74d88 >> > READ of size 2627 at 0x61e000000ac2 thread T3 (in:imjournal) >> > #0 0x7f6085ac62fc (/lib64/libasan.so.5+0xb92fc) >> > #1 0x7f607d5066a5 in readJSONfromJournalMsg imjournal.c:288 >> > #2 0x7f607d5066a5 in readjournal imjournal.c:497 >> > #3 0x55ae0523366e in thrdStarter ../threads.c:243 >> > #4 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce) >> > #5 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82) >> > >> > 0x61e000000ac2 is located 0 bytes to the right of 2626-byte region >> > [0x61e000000080,0x61e000000ac2) >> > allocated by thread T3 (in:imjournal) here: >> > #0 0x7f6085afcfe8 in __interceptor_realloc >> > (/lib64/libasan.so.5+0xeffe8) >> > #1 0x7f6084af1a8b (/lib64/libsystemd.so.0+0x82a8b) >> > >> > Thread T3 (in:imjournal) created by T0 here: >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create >> > (/lib64/libasan.so.5+0x52ea3) >> > #1 0x55ae05234470 in thrdCreate ../threads.c:289 >> > >> > SUMMARY: AddressSanitizer: heap-buffer-overflow >> > (/lib64/libasan.so.5+0xb92fc) >> > Shadow bytes around the buggy address: >> > 0x0c3c7fff8100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> > 0x0c3c7fff8110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> > 0x0c3c7fff8120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> > 0x0c3c7fff8130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> > 0x0c3c7fff8140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> > =>0x0c3c7fff8150: 00 00 00 00 00 00 00 00[02]fa fa fa fa fa fa fa >> > 0x0c3c7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> > 0x0c3c7fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> > 0x0c3c7fff8180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> > 0x0c3c7fff8190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> > 0x0c3c7fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> > Shadow byte legend (one shadow byte represents 8 application bytes): >> > Addressable: 00 >> > Partially addressable: 01 02 03 04 05 06 07 >> > Heap left redzone: fa >> > Freed heap region: fd >> > Stack left redzone: f1 >> > Stack mid redzone: f2 >> > Stack right redzone: f3 >> > Stack after return: f5 >> > Stack use after scope: f8 >> > Global redzone: f9 >> > Global init order: f6 >> > Poisoned by user: f7 >> > Container overflow: fc >> > Array cookie: ac >> > Intra object redzone: bb >> > ASan internal: fe >> > Left alloca redzone: ca >> > Right alloca redzone: cb >> > ================================================================= >> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address >> > 0x60c0005d85c0 at pc 0x7f6085a4db27 bp 0x7f6070621e50 sp 0x7f60706215f8 >> > READ of size 128 at 0x60c0005d85c0 thread T9 (rs:main Q:Reg) >> > #0 0x7f6085a4db26 (/lib64/libasan.so.5+0x40b26) >> > #1 0x55ae0516d061 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x390061) >> > #2 0x55ae0516e522 in MsgDup msg.c:1129 >> > #3 0x55ae05205d58 in execCall ruleset.c:290 >> > #4 0x55ae05205d58 in scriptExec ruleset.c:608 >> > #5 0x55ae05206532 in execIf ruleset.c:313 >> > #6 0x55ae05206532 in scriptExec ruleset.c:614 >> > #7 0x55ae05208868 in processBatch ruleset.c:660 >> > #8 0x55ae050ae50c in msgConsumer rsyslogd.c:694 >> > #9 0x55ae051f100d in ConsumerReg queue.c:2145 >> > #10 0x55ae051e0804 in wtiWorker wti.c:428 >> > #11 0x55ae051d9dd5 in wtpWorker wtp.c:435 >> > #12 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce) >> > #13 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82) >> > >> > 0x60c0005d85c0 is located 0 bytes inside of 128-byte region >> > [0x60c0005d85c0,0x60c0005d8640) >> > freed by thread T6 (imudp(w1)) here: >> > #0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0) >> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471 >> > >> > previously allocated by thread T6 (imudp(w1)) here: >> > #0 0x7f6085afcba8 in __interceptor_malloc (/lib64/libasan.so.5+0xefba8) >> > #1 0x55ae0516cff4 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x38fff4) >> > >> > Thread T9 (rs:main Q:Reg) created by T3 (in:imjournal) here: >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create >> > (/lib64/libasan.so.5+0x52ea3) >> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497 >> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570 >> > >> > Thread T6 (imudp(w1)) created by T0 here: >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create >> > (/lib64/libasan.so.5+0x52ea3) >> > #1 0x55ae05234470 in thrdCreate ../threads.c:289 >> > >> > SUMMARY: AddressSanitizer: heap-use-after-free >> > (/lib64/libasan.so.5+0x40b26) >> > Shadow bytes around the buggy address: >> > 0x0c18800b3060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa >> > 0x0c18800b3070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd >> > 0x0c18800b3080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd >> > 0x0c18800b3090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa >> > 0x0c18800b30a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd >> > =>0x0c18800b30b0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd >> > 0x0c18800b30c0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa >> > 0x0c18800b30d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd >> > 0x0c18800b30e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> > 0x0c18800b30f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> > 0x0c18800b3100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd >> > Shadow byte legend (one shadow byte represents 8 application bytes): >> > Addressable: 00 >> > Partially addressable: 01 02 03 04 05 06 07 >> > Heap left redzone: fa >> > Freed heap region: fd >> > Stack left redzone: f1 >> > Stack mid redzone: f2 >> > Stack right redzone: f3 >> > Stack after return: f5 >> > Stack use after scope: f8 >> > Global redzone: f9 >> > Global init order: f6 >> > Poisoned by user: f7 >> > Container overflow: fc >> > Array cookie: ac >> > Intra object redzone: bb >> > ASan internal: fe >> > Left alloca redzone: ca >> > Right alloca redzone: cb >> > ================================================================= >> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address >> > 0x6040000285a0 at pc 0x55ae0520a723 bp 0x7f60669659a0 sp 0x7f6066965990 >> > WRITE of size 4 at 0x6040000285a0 thread T16 (rs:qradar.local) >> > #0 0x55ae0520a722 in propDestruct prop.c:63 >> > #1 0x55ae05170299 in MsgSetRcvFromIPWithoutAddRef msg.c:457 >> > #2 0x55ae05170299 in resolveDNS msg.c:522 >> > #3 0x55ae0517064a in getRcvFromIP msg.c:558 >> > #4 0x55ae05179317 in MsgGetProp (/usr/sbin/rsyslogd+0x39c317) >> > #5 0x55ae0524aece in tplToString ../template.c:207 >> > #6 0x55ae0522a4ab in prepareDoActionParams ../action.c:1114 >> > #7 0x55ae0522a4ab in processMsgMain ../action.c:1648 >> > #8 0x55ae0522c279 in doSubmitToActionQ ../action.c:1825 >> > #9 0x55ae05205826 in execAct ruleset.c:209 >> > #10 0x55ae05205826 in scriptExec ruleset.c:599 >> > #11 0x55ae05208868 in processBatch ruleset.c:660 >> > #12 0x55ae050ae50c in msgConsumer rsyslogd.c:694 >> > #13 0x55ae051f100d in ConsumerReg queue.c:2145 >> > #14 0x55ae051e0804 in wtiWorker wti.c:428 >> > #15 0x55ae051d9dd5 in wtpWorker wtp.c:435 >> > #16 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce) >> > #17 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82) >> > >> > 0x6040000285a0 is located 16 bytes inside of 48-byte region >> > [0x604000028590,0x6040000285c0) >> > freed by thread T16 (rs:qradar.local) here: >> > #0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0) >> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471 >> > >> > previously allocated by thread T16 (rs:qradar.local) here: >> > #0 0x7f6085afcdb0 in calloc (/lib64/libasan.so.5+0xefdb0) >> > #1 0x55ae0520a439 in propConstruct prop.c:56 >> > >> > Thread T16 (rs:qradar.local) created by T9 (rs:main Q:Reg) here: >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create >> > (/lib64/libasan.so.5+0x52ea3) >> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497 >> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570 >> > >> > SUMMARY: AddressSanitizer: heap-use-after-free prop.c:63 in propDestruct >> > Shadow bytes around the buggy address: >> > 0x0c087fffd060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> > 0x0c087fffd070: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa >> > 0x0c087fffd080: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fd >> > 0x0c087fffd090: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa >> > 0x0c087fffd0a0: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa >> > =>0x0c087fffd0b0: fa fa fd fd[fd]fd fd fd fa fa fa fa fa fa fa fa >> > 0x0c087fffd0c0: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa >> > 0x0c087fffd0d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd >> > 0x0c087fffd0e0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa >> > 0x0c087fffd0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa >> > 0x0c087fffd100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd >> > Shadow byte legend (one shadow byte represents 8 application bytes): >> > Addressable: 00 >> > Partially addressable: 01 02 03 04 05 06 07 >> > Heap left redzone: fa >> > Freed heap region: fd >> > Stack left redzone: f1 >> > Stack mid redzone: f2 >> > Stack right redzone: f3 >> > Stack after return: f5 >> > Stack use after scope: f8 >> > Global redzone: f9 >> > Global init order: f6 >> > Poisoned by user: f7 >> > Container overflow: fc >> > Array cookie: ac >> > Intra object redzone: bb >> > ASan internal: fe >> > Left alloca redzone: ca >> > Right alloca redzone: cb >> > ==3035157==ABORTING >> > >> > ================================= >> > what is the possible solution ??? >> > >> > would be to have multiple rsyslog instances, which is possible if the >> > traffic is split between ports. If yes could you please suggest how to >> > configure?? >> > >> > Thanks & Regards >> > Vijay Kumar Kanukula >> > _______________________________________________ >> > rsyslog mailing list >> > https://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
