I will adjust the MTU of the Linux instance. That's a good point. -----Original Message----- From: David Lang <da...@lang.hm> Sent: Tuesday, August 13, 2024 6:46 PM To: David Lang <da...@lang.hm> Cc: Drumm, Daniel <daniel.dr...@dob.texas.gov>; rsyslog-users <rsyslog@lists.adiscon.com> Subject: RE: [rsyslog] Formatting CEF to log.
On Tue, 13 Aug 2024, David Lang wrote: > by default, ethernet packets are limited to 1500 bytes, most OSs will > fragment UDP messages >1500 bytes across multiple packets. As I noted > before, this means that if one of the packets of a message get lost, > the entore message is lost > > Rsyslog supports this, but it sounds like something else in your > environment is not allowing it. > > the fact that you are seeing the messages via tcpdump still confuses > me. But I'm glad you got things working going back and looking at the tcpdump you provided, that's the 1st 1500 byte packet of the message, if the 2nd fragment is getting dropped by the OS (possible, fragment assembly is a common way to bypass firewall filters, so blocking additional packets that claim to be fragments of a message you already received is a valid security thing to do if you don't need longer UDP messages) and as such, rsyslog would never see them note that you can increase the MTU to up to 9000 bytes (search for jumbo packets) if everything on your network supports it, and you will handle larger messages and in general your network will be more efficient. David Lang > David Lang > > On Tue, 13 Aug 2024, Drumm, Daniel wrote: > >> Date: Tue, 13 Aug 2024 23:19:42 +0000 >> From: "Drumm, Daniel" <daniel.dr...@dob.texas.gov> >> To: David Lang <da...@lang.hm> >> Cc: rsyslog-users <rsyslog@lists.adiscon.com> >> Subject: RE: [rsyslog] Formatting CEF to log. >> >> David - >> >> After experimenting and speaking with a Microsoft engineer, the >> underlying issue is known to Microsoft. Rsyslog will truncate or drop >> long messages in CEF format unless TCP is used. UDP cannot be used. >> This was in a side note in their documentation I missed, early docs >> made it appear as they were agnostic about the syslog transport, but this is >> not the case. >> >> Once I did this, and altered rsyslog.conf to listen with tcp: >> >> # /etc/rsyslog.conf configuration file for rsyslog # # For more >> information install rsyslog-doc and see # >> /usr/share/doc/rsyslog-doc/html/configuration/index.html >> # >> # Default logging rules can be found in >> /etc/rsyslog.d/50-default.conf >> >> >> ################# >> #### MODULES #### >> ################# >> module(load="imudp") # needs to be done just once input(type="imudp" >> port="514") >> module(load="imtcp") # load TCP listener input(type="imtcp" >> port="514") # start TCP listener on port 514 $ModLoad imuxsock # >> needs to be done just once >> >> # $AllowedSender - specifies which remote systems are allowed to send >> syslog messages to rsyslogd # -------------- $AllowedSender TCP, UDP, >> 127.0.0.1, 10.40.0.0/16, 10.30.0.0/23 >> >> $SystemLogSocketFlowControl on # enable flow control (use if needed) >> $ActionFileEnableSync off >> >> ... and so on... >> >> Then, I would see the entirety of the CEF message when that format >> was sent in the log files. TCP had to be used. >> >> >> 2024-08-13 23:14:38 DOB-FW-HA-1.OCI Ashburn FD1 CEF:0|Palo Alto >> Networks|PAN-OS|11.2.0|end|TRAFFIC|1|rt=Aug 13 2024 23:14:43 GMT >> deviceExternalId=007959000479054 src=178.128.104.233 dst=10.40.1.97 >> sourceTranslatedAddress=178.128.104.233 >> destinationTranslatedAddress=10.50.0.166 cs1Label=Rule cs1=Inbound to >> Untrust NAT Mapping suser= duser= app=web-browsing cs3Label=Virtual >> System >> cs3=vsys1 cs4Label=Source Zone cs4=Untrust cs5Label=Destination Zone >> cs5=Trust deviceInboundInterface=ethernet1/2 >> deviceOutboundInterface=ethernet1/1 cs6Label=LogProfile cs6=OCI >> Syslog Server Profile cn1Label=SessionID cn1=129748 cnt=1 spt=52452 >> dpt=443 >> sourceTranslatedPort=52452 destinationTranslatedPort=443 >> flexString1Label=Flags flexString1=0x1400010 proto=tcp act=allow >> flexNumber1Label=Total bytes flexNumber1=4857 in=997 out=3860 >> cn2Label=Packets cn2=14 PanOSPacketsReceived=5 PanOSPacketsSent=9 >> start=Aug >> 13 2024 23:14:43 GMT cn3Label=Elapsed time in seconds cn3=1 >> cs2Label=URL Category cs2=any externalId > =7392297676068849671 reason=threat PanOSDGl1=0 PanOSDGl2=0 PanOSDGl3=0 > PanOSDGl4=0 PanOSVsysName= dvchost=DOB-FW-HA-1 cat=from-policy > PanOSActionFlags=0x0 PanOSSrcUUID= PanOSDstUUID= PanOSTunnelID=0 > PanOSMonitorTag= PanOSParentSessionID=0 PanOSParentStartTime= > PanOSTunnelType=N/A PanOSSCTPAssocID=0 PanOSSCTPChunks=0 > PanOSSCTPChunkSent=0 > PanOSSCTPChunksRcv=0 > PanOSRuleUUID=940d959b-9f14-4032-acc5-5a59fa146d4b > PanOSHTTP2Con=0 PanLinkChange=0 PanPolicyID= PanLinkDetail= > PanSDWANCluster= PanSDWANDevice= PanSDWANClustype= PanSDWANSite= > PanDynamicUsrgrp= PanXFFIP= PanSrcDeviceCat= PanSrcDeviceProf= > PanSrcDeviceModel= PanSrcDeviceVendor= PanSrcDeviceOS= > PanSrcDeviceOSv= PanSrcHostname= PanSrcMac= PanDstDeviceCat= > PanDstDeviceProf= PanDstDeviceModel= PanDstDeviceVendor= > PanDstDeviceOS= PanDstDeviceOSv= PanDstHostname= PanDstMac= > PanContainerName= PanPODNamespace= PanPODName= PanSrcEDL= PanDstEDL= > PanGPHostID= PanEPSerial= PanSrcDAG= PanDstDAG= PanHASessionOwner= > PanTimeHighRes=2024-08-13T18: > 14:44.446-05:00 PanASServiceType= PanASServiceDiff= >> >> -----Original Message----- >> From: David Lang <da...@lang.hm> >> Sent: Tuesday, August 13, 2024 3:43 PM >> To: David Lang <da...@lang.hm> >> Cc: Drumm, Daniel <daniel.dr...@dob.texas.gov>; rsyslog-users >> <rsyslog@lists.adiscon.com> >> Subject: RE: [rsyslog] Formatting CEF to log. >> >> I will also point out that templates in rsyslog are for output only, >> they have no effect at all on parsing input. >> >> David Lang >> >> On Tue, 13 Aug 2024, David Lang wrote: >> >>> Date: Tue, 13 Aug 2024 13:42:38 -0700 (PDT) >>> From: David Lang <da...@lang.hm> >>> To: "Drumm, Daniel" <daniel.dr...@dob.texas.gov> >>> Cc: David Lang <da...@lang.hm>, rsyslog-users >>> <rsyslog@lists.adiscon.com> >>> Subject: RE: [rsyslog] Formatting CEF to log. >>> >>> Drumm, Daniel wrote: >>> >>>> David - >>>> >>>> If you want me to instance a brand-new Ubuntu 22.04 VM in the >>>> cloud, get rsyslog on it, and move the 514/udp Palo flow to it, I can do >>>> that. >>>> >>>> Conversely, if you have access to a PAN-OS instance or old physical >>>> firewall, page 8-10 of this guide will set the CEF parameters and, >>>> if I am guessing correctly, will cause rsyslogd not to log the >>>> messages anywhere even with a *.* directive. >>>> https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fd >>>> oc%2F&data=05%7C02%7CDaniel.Drumm%40dob.texas.gov%7Ca2b6d75072b64b5 >>>> 9df2c08dcbbf20adc%7C32a53eeb27594ef4af2858fd990c7d67%7C1%7C0%7C6385 >>>> 91895466024754%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi >>>> V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=ukwr%2FcJ >>>> mwCdWF6ShebjTWq7vXKqo2qXMCFFQvvQuHsE%3D&reserved=0 >>>> s.paloaltonetworks.com%2Fcontent%2Fdam%2Ftechdocs%2Fen_US%2Fpdf%2Fc >>>> ef >>>> %2Fpan-os-10-0-cef-configuration-guide.pdf&data=05%7C02%7CDaniel.Dr >>>> um >>>> m%40dob.texas.gov%7Cd9cb543b675741ac397e08dcbbd89612%7C32a53eeb2759 >>>> 4e >>>> f4af2858fd990c7d67%7C1%7C0%7C638591786114830361%7CUnknown%7CTWFpbGZ >>>> sb >>>> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0% >>>> 3D >>>> %7C0%7C%7C%7C&sdata=fvo7n1H%2Bpm2q1BY4vG2sTRGRP7oGGSCSGuI00k9tYnU%3 >>>> D& >>>> reserved=0 >>>> >>>> If, as you state, the behavior of rsyslog is best effort to write >>>> to log even if the fields are misaligned, then I cannot explain the >>>> behavior I see. Per the discussion on tcpdump, I do not believe >>>> these UDP packets to be corrupt or excessively fragmented beyond >>>> the MSS. I cannot explain why the basic *.* is not logging the >>>> messages when the CEF formatting is applied to the firewall. Again, >>>> looking at the packets using tcpdump -v, I know they are arriving, >>>> and are about 50% longer than the base log messages. >>> >>> it's beyond that, rsyslog does not parse the fields out of the log >>> as it receives them. >>> >>> I don't have a pan firewall handy, so please run this test and show >>> the results. >>> >>> David Lang >>> >>>> >>>> Daniel Drumm, CISSP >>>> Information Security Officer >>>> Texas Department of Banking >>>> 512-475-1328 >>>> daniel.dr...@dob.texas.gov >>>> >>>> >>>> -----Original Message----- >>>> From: David Lang <da...@lang.hm> >>>> Sent: Tuesday, August 13, 2024 2:20 PM >>>> To: Drumm, Daniel <daniel.dr...@dob.texas.gov> >>>> Cc: David Lang <da...@lang.hm>; rsyslog-users >>>> <rsyslog@lists.adiscon.com> >>>> Subject: RE: [rsyslog] Formatting CEF to log. >>>> >>>> Drumm, Daniel wrote: >>>> >>>>> Is there a rsyslog Template that exists to take the data in this >>>>> format and log it? As you state, when rsyslogd gets a malformed >>>>> message, or one with fields additional to that it understands or >>>>> knows, it will not log it at all, will not write it to disk at >>>>> all, not even a partial malformed log. >>>> >>>> No, that is the opposite of what I stated. >>>> >>>> When Rsyslog recieves a message, it attempts to parse the >>>> timestamp, fromhost, and syslog tag out of it, and everything else >>>> is the string 'msg'. If the log is not properly formatted, it may >>>> put the wrong things in these fields, but it will put SOMETHING >>>> there (best effort) It also populates the filed 'rawmsg' with whatever >>>> arrived. >>>> >>>> it does not care if the msg field is CEF:whatever or 'the quick >>>> brown fox...' >>>> >>>> you can then parse fields out of the message if you care to, but >>>> the msg and rawmsg fields will be there no matter what. >>>> >>>> If the message is longer than your configured maxmessagelength, you >>>> will get a log message that is truncated at that length (and then, >>>> depending on the config, either throw away the rest of the message, >>>> or split it at that point into a second, malformed message >>>> containing the next maxmessagelength characters that are sent) >>>> >>>> but it will not just throw them away because it doesn't understand them. >>>> >>>> I have seen the OS IP stack throw away UDP messages when they are >>>> extremely long and get fragmented into multiple packets and some of >>>> the packets never arrive due to congestion problems on the network, >>>> but that is an intermittent problem, you would have some get >>>> through and others not (and it's been a long time since I saw this >>>> because networks are now much faster than they used to be, so it's >>>> far less common for a network to be that congested, but is still >>>> possible) >>>> >>>> but the fact that you are seeing them via tcpdump would make it >>>> unlikely that this is the problem. >>>> >>>> Can you setup a test where the Palo Alto is only sending the >>>> traffic messages to a test machine and setup a very trivial rsyslog >>>> config just writing it to a file? we may end up wanting to get a >>>> debug log if it still fails there. >>>> >>>> David Lang >>>> >>>> >>> >> > _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
