Hi Team,
My rsyslog service is getting restarted very frequently and we understand
it is due to race between the various threads, which causes one thread to
free a message field while another tries to read/write it.
log:
===
==3035157==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x61e000000ac2 at pc 0x7f6085ac62fd bp 0x7f6079d755e0 sp 0x7f6079d74d88
READ of size 2627 at 0x61e000000ac2 thread T3 (in:imjournal)
#0 0x7f6085ac62fc (/lib64/libasan.so.5+0xb92fc)
#1 0x7f607d5066a5 in readJSONfromJournalMsg imjournal.c:288
#2 0x7f607d5066a5 in readjournal imjournal.c:497
#3 0x55ae0523366e in thrdStarter ../threads.c:243
#4 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
#5 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
0x61e000000ac2 is located 0 bytes to the right of 2626-byte region
[0x61e000000080,0x61e000000ac2)
allocated by thread T3 (in:imjournal) here:
#0 0x7f6085afcfe8 in __interceptor_realloc (/lib64/libasan.so.5+0xeffe8)
#1 0x7f6084af1a8b (/lib64/libsystemd.so.0+0x82a8b)
Thread T3 (in:imjournal) created by T0 here:
#0 0x7f6085a5fea3 in __interceptor_pthread_create
(/lib64/libasan.so.5+0x52ea3)
#1 0x55ae05234470 in thrdCreate ../threads.c:289
SUMMARY: AddressSanitizer: heap-buffer-overflow
(/lib64/libasan.so.5+0xb92fc)
Shadow bytes around the buggy address:
0x0c3c7fff8100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c3c7fff8110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c3c7fff8120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c3c7fff8130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c3c7fff8140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c3c7fff8150: 00 00 00 00 00 00 00 00[02]fa fa fa fa fa fa fa
0x0c3c7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c3c7fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c3c7fff8180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c3c7fff8190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c3c7fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
=================================================================
==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
0x60c0005d85c0 at pc 0x7f6085a4db27 bp 0x7f6070621e50 sp 0x7f60706215f8
READ of size 128 at 0x60c0005d85c0 thread T9 (rs:main Q:Reg)
#0 0x7f6085a4db26 (/lib64/libasan.so.5+0x40b26)
#1 0x55ae0516d061 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x390061)
#2 0x55ae0516e522 in MsgDup msg.c:1129
#3 0x55ae05205d58 in execCall ruleset.c:290
#4 0x55ae05205d58 in scriptExec ruleset.c:608
#5 0x55ae05206532 in execIf ruleset.c:313
#6 0x55ae05206532 in scriptExec ruleset.c:614
#7 0x55ae05208868 in processBatch ruleset.c:660
#8 0x55ae050ae50c in msgConsumer rsyslogd.c:694
#9 0x55ae051f100d in ConsumerReg queue.c:2145
#10 0x55ae051e0804 in wtiWorker wti.c:428
#11 0x55ae051d9dd5 in wtpWorker wtp.c:435
#12 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
#13 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
0x60c0005d85c0 is located 0 bytes inside of 128-byte region
[0x60c0005d85c0,0x60c0005d8640)
freed by thread T6 (imudp(w1)) here:
#0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
#1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
previously allocated by thread T6 (imudp(w1)) here:
#0 0x7f6085afcba8 in __interceptor_malloc (/lib64/libasan.so.5+0xefba8)
#1 0x55ae0516cff4 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x38fff4)
Thread T9 (rs:main Q:Reg) created by T3 (in:imjournal) here:
#0 0x7f6085a5fea3 in __interceptor_pthread_create
(/lib64/libasan.so.5+0x52ea3)
#1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
#2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
Thread T6 (imudp(w1)) created by T0 here:
#0 0x7f6085a5fea3 in __interceptor_pthread_create
(/lib64/libasan.so.5+0x52ea3)
#1 0x55ae05234470 in thrdCreate ../threads.c:289
SUMMARY: AddressSanitizer: heap-use-after-free (/lib64/libasan.so.5+0x40b26)
Shadow bytes around the buggy address:
0x0c18800b3060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
0x0c18800b3070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c18800b3080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c18800b3090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
0x0c18800b30a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c18800b30b0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd
0x0c18800b30c0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
0x0c18800b30d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c18800b30e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c18800b30f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c18800b3100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
=================================================================
==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
0x6040000285a0 at pc 0x55ae0520a723 bp 0x7f60669659a0 sp 0x7f6066965990
WRITE of size 4 at 0x6040000285a0 thread T16 (rs:qradar.local)
#0 0x55ae0520a722 in propDestruct prop.c:63
#1 0x55ae05170299 in MsgSetRcvFromIPWithoutAddRef msg.c:457
#2 0x55ae05170299 in resolveDNS msg.c:522
#3 0x55ae0517064a in getRcvFromIP msg.c:558
#4 0x55ae05179317 in MsgGetProp (/usr/sbin/rsyslogd+0x39c317)
#5 0x55ae0524aece in tplToString ../template.c:207
#6 0x55ae0522a4ab in prepareDoActionParams ../action.c:1114
#7 0x55ae0522a4ab in processMsgMain ../action.c:1648
#8 0x55ae0522c279 in doSubmitToActionQ ../action.c:1825
#9 0x55ae05205826 in execAct ruleset.c:209
#10 0x55ae05205826 in scriptExec ruleset.c:599
#11 0x55ae05208868 in processBatch ruleset.c:660
#12 0x55ae050ae50c in msgConsumer rsyslogd.c:694
#13 0x55ae051f100d in ConsumerReg queue.c:2145
#14 0x55ae051e0804 in wtiWorker wti.c:428
#15 0x55ae051d9dd5 in wtpWorker wtp.c:435
#16 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
#17 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
0x6040000285a0 is located 16 bytes inside of 48-byte region
[0x604000028590,0x6040000285c0)
freed by thread T16 (rs:qradar.local) here:
#0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
#1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
previously allocated by thread T16 (rs:qradar.local) here:
#0 0x7f6085afcdb0 in calloc (/lib64/libasan.so.5+0xefdb0)
#1 0x55ae0520a439 in propConstruct prop.c:56
Thread T16 (rs:qradar.local) created by T9 (rs:main Q:Reg) here:
#0 0x7f6085a5fea3 in __interceptor_pthread_create
(/lib64/libasan.so.5+0x52ea3)
#1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
#2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
SUMMARY: AddressSanitizer: heap-use-after-free prop.c:63 in propDestruct
Shadow bytes around the buggy address:
0x0c087fffd060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c087fffd070: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
0x0c087fffd080: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fd
0x0c087fffd090: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa
0x0c087fffd0a0: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
=>0x0c087fffd0b0: fa fa fd fd[fd]fd fd fd fa fa fa fa fa fa fa fa
0x0c087fffd0c0: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa
0x0c087fffd0d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
0x0c087fffd0e0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
0x0c087fffd0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
0x0c087fffd100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==3035157==ABORTING
=================================
what is the possible solution ???
would be to have multiple rsyslog instances, which is possible if the
traffic is split between ports. If yes could you please suggest how to
configure??
Thanks & Regards
Vijay Kumar Kanukula
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
