Hi, sorry I did not get you, asan logs???
On Mon, Jun 20, 2022 at 1:24 PM vijay kumar via rsyslog < rsyslog@lists.adiscon.com> wrote: > Hi Rainer, > > Do you have any luck with asan logs??? > > Thanks & Regards > Vijay Kumar Kanukula > > On Fri, 17 Jun 2022 at 14:11, vijay kumar <vijay.kanuk...@gmail.com> > wrote: > > > Hello Rainer, > > > > When we installed ASAN related RPMS to capture the logs rsyslog restart > > for every 2 to 3 mins, it was purely unstable so we downgraded > immediately. > > Attaching ASAN logs captured day before yesterday. > > > > Thanks & Regards > > Vijay Kumar Kanukula > > > > On Fri, 17 Jun 2022 at 13:44, Rainer Gerhards <rgerha...@hq.adiscon.com> > > wrote: > > > >> can you please post rsyslog -v output as well as the current ASAN > report? > >> > >> Thanks, > >> Rainer > >> > >> El vie, 17 jun 2022 a las 10:12, vijay kumar > >> (<vijay.kanuk...@gmail.com>) escribió: > >> > > >> > HI Rainer/David/Marisuz, > >> > > >> > Could you please help me with creating one input rule with a queue as > >> Marisuz suggested. I was failing to create a rule. > >> > > >> > If I change anything in my 90-inputs.conf file do I need to make any > >> changes in 30-rules.conf??? > >> > > >> > @Rainer : I upgraded to the latest version of rsyslog and tested. But > >> still my rsyslog restarts continuously. It was not stable. > >> > > >> > Thanks & Regards > >> > Vijay Kumar Kanukula > >> > > >> > > >> > > >> > > >> > > >> > On Fri, 17 Jun 2022 at 12:55, Rainer Gerhards < > rgerha...@hq.adiscon.com> > >> wrote: > >> >> > >> >> it would be good to find the root cause. The best would be to use the > >> >> current 8.2206.0 version and see if it works. If so, all is fine. If > >> >> not, we should try to debug the issue. > >> >> > >> >> Rainer > >> >> > >> >> El jue, 16 jun 2022 a las 16:22, vijay kumar via rsyslog > >> >> (<rsyslog@lists.adiscon.com>) escribió: > >> >> > > >> >> > Hi Team, > >> >> > > >> >> > My rsyslog service is getting restarted very frequently and we > >> understand > >> >> > it is due to race between the various threads, which causes one > >> thread to > >> >> > free a message field while another tries to read/write it. > >> >> > > >> >> > log: > >> >> > === > >> >> > ==3035157==ERROR: AddressSanitizer: heap-buffer-overflow on address > >> >> > 0x61e000000ac2 at pc 0x7f6085ac62fd bp 0x7f6079d755e0 sp > >> 0x7f6079d74d88 > >> >> > READ of size 2627 at 0x61e000000ac2 thread T3 (in:imjournal) > >> >> > #0 0x7f6085ac62fc (/lib64/libasan.so.5+0xb92fc) > >> >> > #1 0x7f607d5066a5 in readJSONfromJournalMsg imjournal.c:288 > >> >> > #2 0x7f607d5066a5 in readjournal imjournal.c:497 > >> >> > #3 0x55ae0523366e in thrdStarter ../threads.c:243 > >> >> > #4 0x7f60855dd1ce in start_thread > (/lib64/libpthread.so.0+0x81ce) > >> >> > #5 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82) > >> >> > > >> >> > 0x61e000000ac2 is located 0 bytes to the right of 2626-byte region > >> >> > [0x61e000000080,0x61e000000ac2) > >> >> > allocated by thread T3 (in:imjournal) here: > >> >> > #0 0x7f6085afcfe8 in __interceptor_realloc > >> (/lib64/libasan.so.5+0xeffe8) > >> >> > #1 0x7f6084af1a8b (/lib64/libsystemd.so.0+0x82a8b) > >> >> > > >> >> > Thread T3 (in:imjournal) created by T0 here: > >> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create > >> >> > (/lib64/libasan.so.5+0x52ea3) > >> >> > #1 0x55ae05234470 in thrdCreate ../threads.c:289 > >> >> > > >> >> > SUMMARY: AddressSanitizer: heap-buffer-overflow > >> >> > (/lib64/libasan.so.5+0xb92fc) > >> >> > Shadow bytes around the buggy address: > >> >> > 0x0c3c7fff8100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > >> >> > 0x0c3c7fff8110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > >> >> > 0x0c3c7fff8120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > >> >> > 0x0c3c7fff8130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > >> >> > 0x0c3c7fff8140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > >> >> > =>0x0c3c7fff8150: 00 00 00 00 00 00 00 00[02]fa fa fa fa fa fa fa > >> >> > 0x0c3c7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > >> >> > 0x0c3c7fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > >> >> > 0x0c3c7fff8180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > >> >> > 0x0c3c7fff8190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > >> >> > 0x0c3c7fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > >> >> > Shadow byte legend (one shadow byte represents 8 application > bytes): > >> >> > Addressable: 00 > >> >> > Partially addressable: 01 02 03 04 05 06 07 > >> >> > Heap left redzone: fa > >> >> > Freed heap region: fd > >> >> > Stack left redzone: f1 > >> >> > Stack mid redzone: f2 > >> >> > Stack right redzone: f3 > >> >> > Stack after return: f5 > >> >> > Stack use after scope: f8 > >> >> > Global redzone: f9 > >> >> > Global init order: f6 > >> >> > Poisoned by user: f7 > >> >> > Container overflow: fc > >> >> > Array cookie: ac > >> >> > Intra object redzone: bb > >> >> > ASan internal: fe > >> >> > Left alloca redzone: ca > >> >> > Right alloca redzone: cb > >> >> > ================================================================= > >> >> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address > >> >> > 0x60c0005d85c0 at pc 0x7f6085a4db27 bp 0x7f6070621e50 sp > >> 0x7f60706215f8 > >> >> > READ of size 128 at 0x60c0005d85c0 thread T9 (rs:main Q:Reg) > >> >> > #0 0x7f6085a4db26 (/lib64/libasan.so.5+0x40b26) > >> >> > #1 0x55ae0516d061 in msgSetFromSockinfo > >> (/usr/sbin/rsyslogd+0x390061) > >> >> > #2 0x55ae0516e522 in MsgDup msg.c:1129 > >> >> > #3 0x55ae05205d58 in execCall ruleset.c:290 > >> >> > #4 0x55ae05205d58 in scriptExec ruleset.c:608 > >> >> > #5 0x55ae05206532 in execIf ruleset.c:313 > >> >> > #6 0x55ae05206532 in scriptExec ruleset.c:614 > >> >> > #7 0x55ae05208868 in processBatch ruleset.c:660 > >> >> > #8 0x55ae050ae50c in msgConsumer rsyslogd.c:694 > >> >> > #9 0x55ae051f100d in ConsumerReg queue.c:2145 > >> >> > #10 0x55ae051e0804 in wtiWorker wti.c:428 > >> >> > #11 0x55ae051d9dd5 in wtpWorker wtp.c:435 > >> >> > #12 0x7f60855dd1ce in start_thread > >> (/lib64/libpthread.so.0+0x81ce) > >> >> > #13 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82) > >> >> > > >> >> > 0x60c0005d85c0 is located 0 bytes inside of 128-byte region > >> >> > [0x60c0005d85c0,0x60c0005d8640) > >> >> > freed by thread T6 (imudp(w1)) here: > >> >> > #0 0x7f6085afc7e0 in __interceptor_free > >> (/lib64/libasan.so.5+0xef7e0) > >> >> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471 > >> >> > > >> >> > previously allocated by thread T6 (imudp(w1)) here: > >> >> > #0 0x7f6085afcba8 in __interceptor_malloc > >> (/lib64/libasan.so.5+0xefba8) > >> >> > #1 0x55ae0516cff4 in msgSetFromSockinfo > >> (/usr/sbin/rsyslogd+0x38fff4) > >> >> > > >> >> > Thread T9 (rs:main Q:Reg) created by T3 (in:imjournal) here: > >> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create > >> >> > (/lib64/libasan.so.5+0x52ea3) > >> >> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497 > >> >> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570 > >> >> > > >> >> > Thread T6 (imudp(w1)) created by T0 here: > >> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create > >> >> > (/lib64/libasan.so.5+0x52ea3) > >> >> > #1 0x55ae05234470 in thrdCreate ../threads.c:289 > >> >> > > >> >> > SUMMARY: AddressSanitizer: heap-use-after-free > >> (/lib64/libasan.so.5+0x40b26) > >> >> > Shadow bytes around the buggy address: > >> >> > 0x0c18800b3060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa > >> >> > 0x0c18800b3070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > >> >> > 0x0c18800b3080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd > >> >> > 0x0c18800b3090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa > >> >> > 0x0c18800b30a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > >> >> > =>0x0c18800b30b0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd > >> >> > 0x0c18800b30c0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa > >> >> > 0x0c18800b30d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > >> >> > 0x0c18800b30e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > >> >> > 0x0c18800b30f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > >> >> > 0x0c18800b3100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > >> >> > Shadow byte legend (one shadow byte represents 8 application > bytes): > >> >> > Addressable: 00 > >> >> > Partially addressable: 01 02 03 04 05 06 07 > >> >> > Heap left redzone: fa > >> >> > Freed heap region: fd > >> >> > Stack left redzone: f1 > >> >> > Stack mid redzone: f2 > >> >> > Stack right redzone: f3 > >> >> > Stack after return: f5 > >> >> > Stack use after scope: f8 > >> >> > Global redzone: f9 > >> >> > Global init order: f6 > >> >> > Poisoned by user: f7 > >> >> > Container overflow: fc > >> >> > Array cookie: ac > >> >> > Intra object redzone: bb > >> >> > ASan internal: fe > >> >> > Left alloca redzone: ca > >> >> > Right alloca redzone: cb > >> >> > ================================================================= > >> >> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address > >> >> > 0x6040000285a0 at pc 0x55ae0520a723 bp 0x7f60669659a0 sp > >> 0x7f6066965990 > >> >> > WRITE of size 4 at 0x6040000285a0 thread T16 (rs:qradar.local) > >> >> > #0 0x55ae0520a722 in propDestruct prop.c:63 > >> >> > #1 0x55ae05170299 in MsgSetRcvFromIPWithoutAddRef msg.c:457 > >> >> > #2 0x55ae05170299 in resolveDNS msg.c:522 > >> >> > #3 0x55ae0517064a in getRcvFromIP msg.c:558 > >> >> > #4 0x55ae05179317 in MsgGetProp (/usr/sbin/rsyslogd+0x39c317) > >> >> > #5 0x55ae0524aece in tplToString ../template.c:207 > >> >> > #6 0x55ae0522a4ab in prepareDoActionParams ../action.c:1114 > >> >> > #7 0x55ae0522a4ab in processMsgMain ../action.c:1648 > >> >> > #8 0x55ae0522c279 in doSubmitToActionQ ../action.c:1825 > >> >> > #9 0x55ae05205826 in execAct ruleset.c:209 > >> >> > #10 0x55ae05205826 in scriptExec ruleset.c:599 > >> >> > #11 0x55ae05208868 in processBatch ruleset.c:660 > >> >> > #12 0x55ae050ae50c in msgConsumer rsyslogd.c:694 > >> >> > #13 0x55ae051f100d in ConsumerReg queue.c:2145 > >> >> > #14 0x55ae051e0804 in wtiWorker wti.c:428 > >> >> > #15 0x55ae051d9dd5 in wtpWorker wtp.c:435 > >> >> > #16 0x7f60855dd1ce in start_thread > >> (/lib64/libpthread.so.0+0x81ce) > >> >> > #17 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82) > >> >> > > >> >> > 0x6040000285a0 is located 16 bytes inside of 48-byte region > >> >> > [0x604000028590,0x6040000285c0) > >> >> > freed by thread T16 (rs:qradar.local) here: > >> >> > #0 0x7f6085afc7e0 in __interceptor_free > >> (/lib64/libasan.so.5+0xef7e0) > >> >> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471 > >> >> > > >> >> > previously allocated by thread T16 (rs:qradar.local) here: > >> >> > #0 0x7f6085afcdb0 in calloc (/lib64/libasan.so.5+0xefdb0) > >> >> > #1 0x55ae0520a439 in propConstruct prop.c:56 > >> >> > > >> >> > Thread T16 (rs:qradar.local) created by T9 (rs:main Q:Reg) here: > >> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create > >> >> > (/lib64/libasan.so.5+0x52ea3) > >> >> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497 > >> >> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570 > >> >> > > >> >> > SUMMARY: AddressSanitizer: heap-use-after-free prop.c:63 in > >> propDestruct > >> >> > Shadow bytes around the buggy address: > >> >> > 0x0c087fffd060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > >> >> > 0x0c087fffd070: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa > >> >> > 0x0c087fffd080: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fd > >> >> > 0x0c087fffd090: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa > >> >> > 0x0c087fffd0a0: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa > >> >> > =>0x0c087fffd0b0: fa fa fd fd[fd]fd fd fd fa fa fa fa fa fa fa fa > >> >> > 0x0c087fffd0c0: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa > >> >> > 0x0c087fffd0d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd > >> >> > 0x0c087fffd0e0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa > >> >> > 0x0c087fffd0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa > >> >> > 0x0c087fffd100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd > >> >> > Shadow byte legend (one shadow byte represents 8 application > bytes): > >> >> > Addressable: 00 > >> >> > Partially addressable: 01 02 03 04 05 06 07 > >> >> > Heap left redzone: fa > >> >> > Freed heap region: fd > >> >> > Stack left redzone: f1 > >> >> > Stack mid redzone: f2 > >> >> > Stack right redzone: f3 > >> >> > Stack after return: f5 > >> >> > Stack use after scope: f8 > >> >> > Global redzone: f9 > >> >> > Global init order: f6 > >> >> > Poisoned by user: f7 > >> >> > Container overflow: fc > >> >> > Array cookie: ac > >> >> > Intra object redzone: bb > >> >> > ASan internal: fe > >> >> > Left alloca redzone: ca > >> >> > Right alloca redzone: cb > >> >> > ==3035157==ABORTING > >> >> > > >> >> > ================================= > >> >> > what is the possible solution ??? > >> >> > > >> >> > would be to have multiple rsyslog instances, which is possible if > >> the > >> >> > traffic is split between ports. If yes could you please suggest how > >> to > >> >> > configure?? > >> >> > > >> >> > Thanks & Regards > >> >> > Vijay Kumar Kanukula > >> >> > _______________________________________________ > >> >> > rsyslog mailing list > >> >> > https://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> > http://www.rsyslog.com/professional-services/ > >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > >> you DON'T LIKE THAT. > >> > > > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
