Hi Rainer, Do you have any luck with asan logs???
Thanks & Regards Vijay Kumar Kanukula On Fri, 17 Jun 2022 at 14:11, vijay kumar <vijay.kanuk...@gmail.com> wrote: > Hello Rainer, > > When we installed ASAN related RPMS to capture the logs rsyslog restart > for every 2 to 3 mins, it was purely unstable so we downgraded immediately. > Attaching ASAN logs captured day before yesterday. > > Thanks & Regards > Vijay Kumar Kanukula > > On Fri, 17 Jun 2022 at 13:44, Rainer Gerhards <rgerha...@hq.adiscon.com> > wrote: > >> can you please post rsyslog -v output as well as the current ASAN report? >> >> Thanks, >> Rainer >> >> El vie, 17 jun 2022 a las 10:12, vijay kumar >> (<vijay.kanuk...@gmail.com>) escribió: >> > >> > HI Rainer/David/Marisuz, >> > >> > Could you please help me with creating one input rule with a queue as >> Marisuz suggested. I was failing to create a rule. >> > >> > If I change anything in my 90-inputs.conf file do I need to make any >> changes in 30-rules.conf??? >> > >> > @Rainer : I upgraded to the latest version of rsyslog and tested. But >> still my rsyslog restarts continuously. It was not stable. >> > >> > Thanks & Regards >> > Vijay Kumar Kanukula >> > >> > >> > >> > >> > >> > On Fri, 17 Jun 2022 at 12:55, Rainer Gerhards <rgerha...@hq.adiscon.com> >> wrote: >> >> >> >> it would be good to find the root cause. The best would be to use the >> >> current 8.2206.0 version and see if it works. If so, all is fine. If >> >> not, we should try to debug the issue. >> >> >> >> Rainer >> >> >> >> El jue, 16 jun 2022 a las 16:22, vijay kumar via rsyslog >> >> (<rsyslog@lists.adiscon.com>) escribió: >> >> > >> >> > Hi Team, >> >> > >> >> > My rsyslog service is getting restarted very frequently and we >> understand >> >> > it is due to race between the various threads, which causes one >> thread to >> >> > free a message field while another tries to read/write it. >> >> > >> >> > log: >> >> > === >> >> > ==3035157==ERROR: AddressSanitizer: heap-buffer-overflow on address >> >> > 0x61e000000ac2 at pc 0x7f6085ac62fd bp 0x7f6079d755e0 sp >> 0x7f6079d74d88 >> >> > READ of size 2627 at 0x61e000000ac2 thread T3 (in:imjournal) >> >> > #0 0x7f6085ac62fc (/lib64/libasan.so.5+0xb92fc) >> >> > #1 0x7f607d5066a5 in readJSONfromJournalMsg imjournal.c:288 >> >> > #2 0x7f607d5066a5 in readjournal imjournal.c:497 >> >> > #3 0x55ae0523366e in thrdStarter ../threads.c:243 >> >> > #4 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce) >> >> > #5 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82) >> >> > >> >> > 0x61e000000ac2 is located 0 bytes to the right of 2626-byte region >> >> > [0x61e000000080,0x61e000000ac2) >> >> > allocated by thread T3 (in:imjournal) here: >> >> > #0 0x7f6085afcfe8 in __interceptor_realloc >> (/lib64/libasan.so.5+0xeffe8) >> >> > #1 0x7f6084af1a8b (/lib64/libsystemd.so.0+0x82a8b) >> >> > >> >> > Thread T3 (in:imjournal) created by T0 here: >> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create >> >> > (/lib64/libasan.so.5+0x52ea3) >> >> > #1 0x55ae05234470 in thrdCreate ../threads.c:289 >> >> > >> >> > SUMMARY: AddressSanitizer: heap-buffer-overflow >> >> > (/lib64/libasan.so.5+0xb92fc) >> >> > Shadow bytes around the buggy address: >> >> > 0x0c3c7fff8100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> >> > 0x0c3c7fff8110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> >> > 0x0c3c7fff8120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> >> > 0x0c3c7fff8130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> >> > 0x0c3c7fff8140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> >> > =>0x0c3c7fff8150: 00 00 00 00 00 00 00 00[02]fa fa fa fa fa fa fa >> >> > 0x0c3c7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> >> > 0x0c3c7fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> >> > 0x0c3c7fff8180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> >> > 0x0c3c7fff8190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> >> > 0x0c3c7fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> >> > Shadow byte legend (one shadow byte represents 8 application bytes): >> >> > Addressable: 00 >> >> > Partially addressable: 01 02 03 04 05 06 07 >> >> > Heap left redzone: fa >> >> > Freed heap region: fd >> >> > Stack left redzone: f1 >> >> > Stack mid redzone: f2 >> >> > Stack right redzone: f3 >> >> > Stack after return: f5 >> >> > Stack use after scope: f8 >> >> > Global redzone: f9 >> >> > Global init order: f6 >> >> > Poisoned by user: f7 >> >> > Container overflow: fc >> >> > Array cookie: ac >> >> > Intra object redzone: bb >> >> > ASan internal: fe >> >> > Left alloca redzone: ca >> >> > Right alloca redzone: cb >> >> > ================================================================= >> >> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address >> >> > 0x60c0005d85c0 at pc 0x7f6085a4db27 bp 0x7f6070621e50 sp >> 0x7f60706215f8 >> >> > READ of size 128 at 0x60c0005d85c0 thread T9 (rs:main Q:Reg) >> >> > #0 0x7f6085a4db26 (/lib64/libasan.so.5+0x40b26) >> >> > #1 0x55ae0516d061 in msgSetFromSockinfo >> (/usr/sbin/rsyslogd+0x390061) >> >> > #2 0x55ae0516e522 in MsgDup msg.c:1129 >> >> > #3 0x55ae05205d58 in execCall ruleset.c:290 >> >> > #4 0x55ae05205d58 in scriptExec ruleset.c:608 >> >> > #5 0x55ae05206532 in execIf ruleset.c:313 >> >> > #6 0x55ae05206532 in scriptExec ruleset.c:614 >> >> > #7 0x55ae05208868 in processBatch ruleset.c:660 >> >> > #8 0x55ae050ae50c in msgConsumer rsyslogd.c:694 >> >> > #9 0x55ae051f100d in ConsumerReg queue.c:2145 >> >> > #10 0x55ae051e0804 in wtiWorker wti.c:428 >> >> > #11 0x55ae051d9dd5 in wtpWorker wtp.c:435 >> >> > #12 0x7f60855dd1ce in start_thread >> (/lib64/libpthread.so.0+0x81ce) >> >> > #13 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82) >> >> > >> >> > 0x60c0005d85c0 is located 0 bytes inside of 128-byte region >> >> > [0x60c0005d85c0,0x60c0005d8640) >> >> > freed by thread T6 (imudp(w1)) here: >> >> > #0 0x7f6085afc7e0 in __interceptor_free >> (/lib64/libasan.so.5+0xef7e0) >> >> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471 >> >> > >> >> > previously allocated by thread T6 (imudp(w1)) here: >> >> > #0 0x7f6085afcba8 in __interceptor_malloc >> (/lib64/libasan.so.5+0xefba8) >> >> > #1 0x55ae0516cff4 in msgSetFromSockinfo >> (/usr/sbin/rsyslogd+0x38fff4) >> >> > >> >> > Thread T9 (rs:main Q:Reg) created by T3 (in:imjournal) here: >> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create >> >> > (/lib64/libasan.so.5+0x52ea3) >> >> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497 >> >> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570 >> >> > >> >> > Thread T6 (imudp(w1)) created by T0 here: >> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create >> >> > (/lib64/libasan.so.5+0x52ea3) >> >> > #1 0x55ae05234470 in thrdCreate ../threads.c:289 >> >> > >> >> > SUMMARY: AddressSanitizer: heap-use-after-free >> (/lib64/libasan.so.5+0x40b26) >> >> > Shadow bytes around the buggy address: >> >> > 0x0c18800b3060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa >> >> > 0x0c18800b3070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd >> >> > 0x0c18800b3080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd >> >> > 0x0c18800b3090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa >> >> > 0x0c18800b30a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd >> >> > =>0x0c18800b30b0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd >> >> > 0x0c18800b30c0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa >> >> > 0x0c18800b30d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd >> >> > 0x0c18800b30e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> >> > 0x0c18800b30f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> >> > 0x0c18800b3100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd >> >> > Shadow byte legend (one shadow byte represents 8 application bytes): >> >> > Addressable: 00 >> >> > Partially addressable: 01 02 03 04 05 06 07 >> >> > Heap left redzone: fa >> >> > Freed heap region: fd >> >> > Stack left redzone: f1 >> >> > Stack mid redzone: f2 >> >> > Stack right redzone: f3 >> >> > Stack after return: f5 >> >> > Stack use after scope: f8 >> >> > Global redzone: f9 >> >> > Global init order: f6 >> >> > Poisoned by user: f7 >> >> > Container overflow: fc >> >> > Array cookie: ac >> >> > Intra object redzone: bb >> >> > ASan internal: fe >> >> > Left alloca redzone: ca >> >> > Right alloca redzone: cb >> >> > ================================================================= >> >> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address >> >> > 0x6040000285a0 at pc 0x55ae0520a723 bp 0x7f60669659a0 sp >> 0x7f6066965990 >> >> > WRITE of size 4 at 0x6040000285a0 thread T16 (rs:qradar.local) >> >> > #0 0x55ae0520a722 in propDestruct prop.c:63 >> >> > #1 0x55ae05170299 in MsgSetRcvFromIPWithoutAddRef msg.c:457 >> >> > #2 0x55ae05170299 in resolveDNS msg.c:522 >> >> > #3 0x55ae0517064a in getRcvFromIP msg.c:558 >> >> > #4 0x55ae05179317 in MsgGetProp (/usr/sbin/rsyslogd+0x39c317) >> >> > #5 0x55ae0524aece in tplToString ../template.c:207 >> >> > #6 0x55ae0522a4ab in prepareDoActionParams ../action.c:1114 >> >> > #7 0x55ae0522a4ab in processMsgMain ../action.c:1648 >> >> > #8 0x55ae0522c279 in doSubmitToActionQ ../action.c:1825 >> >> > #9 0x55ae05205826 in execAct ruleset.c:209 >> >> > #10 0x55ae05205826 in scriptExec ruleset.c:599 >> >> > #11 0x55ae05208868 in processBatch ruleset.c:660 >> >> > #12 0x55ae050ae50c in msgConsumer rsyslogd.c:694 >> >> > #13 0x55ae051f100d in ConsumerReg queue.c:2145 >> >> > #14 0x55ae051e0804 in wtiWorker wti.c:428 >> >> > #15 0x55ae051d9dd5 in wtpWorker wtp.c:435 >> >> > #16 0x7f60855dd1ce in start_thread >> (/lib64/libpthread.so.0+0x81ce) >> >> > #17 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82) >> >> > >> >> > 0x6040000285a0 is located 16 bytes inside of 48-byte region >> >> > [0x604000028590,0x6040000285c0) >> >> > freed by thread T16 (rs:qradar.local) here: >> >> > #0 0x7f6085afc7e0 in __interceptor_free >> (/lib64/libasan.so.5+0xef7e0) >> >> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471 >> >> > >> >> > previously allocated by thread T16 (rs:qradar.local) here: >> >> > #0 0x7f6085afcdb0 in calloc (/lib64/libasan.so.5+0xefdb0) >> >> > #1 0x55ae0520a439 in propConstruct prop.c:56 >> >> > >> >> > Thread T16 (rs:qradar.local) created by T9 (rs:main Q:Reg) here: >> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create >> >> > (/lib64/libasan.so.5+0x52ea3) >> >> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497 >> >> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570 >> >> > >> >> > SUMMARY: AddressSanitizer: heap-use-after-free prop.c:63 in >> propDestruct >> >> > Shadow bytes around the buggy address: >> >> > 0x0c087fffd060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa >> >> > 0x0c087fffd070: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa >> >> > 0x0c087fffd080: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fd >> >> > 0x0c087fffd090: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa >> >> > 0x0c087fffd0a0: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa >> >> > =>0x0c087fffd0b0: fa fa fd fd[fd]fd fd fd fa fa fa fa fa fa fa fa >> >> > 0x0c087fffd0c0: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa >> >> > 0x0c087fffd0d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd >> >> > 0x0c087fffd0e0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa >> >> > 0x0c087fffd0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa >> >> > 0x0c087fffd100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd >> >> > Shadow byte legend (one shadow byte represents 8 application bytes): >> >> > Addressable: 00 >> >> > Partially addressable: 01 02 03 04 05 06 07 >> >> > Heap left redzone: fa >> >> > Freed heap region: fd >> >> > Stack left redzone: f1 >> >> > Stack mid redzone: f2 >> >> > Stack right redzone: f3 >> >> > Stack after return: f5 >> >> > Stack use after scope: f8 >> >> > Global redzone: f9 >> >> > Global init order: f6 >> >> > Poisoned by user: f7 >> >> > Container overflow: fc >> >> > Array cookie: ac >> >> > Intra object redzone: bb >> >> > ASan internal: fe >> >> > Left alloca redzone: ca >> >> > Right alloca redzone: cb >> >> > ==3035157==ABORTING >> >> > >> >> > ================================= >> >> > what is the possible solution ??? >> >> > >> >> > would be to have multiple rsyslog instances, which is possible if >> the >> >> > traffic is split between ports. If yes could you please suggest how >> to >> >> > configure?? >> >> > >> >> > Thanks & Regards >> >> > Vijay Kumar Kanukula >> >> > _______________________________________________ >> >> > rsyslog mailing list >> >> > https://lists.adiscon.net/mailman/listinfo/rsyslog >> >> > http://www.rsyslog.com/professional-services/ >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you DON'T LIKE THAT. >> > _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
