There is no ODBC connector or anything of that sort. The Azure Sentinel tool automatically adds a file to /etc/rsyslog.d. Here it what it does:
root@syslog-server-vnic-primary:/etc/rsyslog.d# more 10-azuremonitoragent-omfwd.conf # Azure Monitor Agent configuration: forward logs to azuremonitoragent template(name="AMA_RSYSLOG_TraditionalForwardFormat" type="string" string="<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%") # queue.workerThreads sets the maximum worker threads, it will scale back to 0 if there is no activity # Forwarding all events through TCP port *.* action(type="omfwd" template="AMA_RSYSLOG_TraditionalForwardFormat" queue.type="LinkedList" queue.filename="omfwd-azuremonitoragent" queue.maxFileSize="32m" action.resumeRetryCount="-1" action.resumeInterval="5" action.reportSuspension="on" action.reportSuspensionContinuation="on" queue.size="25000" queue.workerThreads="100" queue.dequeueBatchSize="2048" queue.saveonshutdown="on" target="127.0.0.1" Port="28330" Protocol="tcp") That file was created by Microsoft engineers specifically for rsyslog, not by me. Since it is a 10 file, it is processed first. I just removed it in case it was causing any issues. So, only the file I created is working which logs all traffic from the firewall IP to a file. No other connection other than 50-default, which logs the local facilities. When I adjust the firewall logging from default to CEF, rsyslogd will not log it at all. Here is the template you assign to the Palo Alto for CEF formattng: -- Traffic - CEF:0|Palo Alto Networks|PAN-OS|$sender_sw_version|$subtype|$type|1|rt=$cef-formatted-receive_time deviceExternalId=$serial src=$src dst=$dst sourceTranslatedAddress=$natsrc destinationTranslatedAddress=$natdst cs1Label=Rule cs1=$rule suser=$srcuser duser=$dstuser app=$app cs3Label=Virtual System cs3=$vsys cs4Label=Source Zone cs4=$from cs5Label=Destination Zone cs5=$to deviceInboundInterface=$inbound_if deviceOutboundInterface=$outbound_if cs6Label=LogProfile cs6=$logset cn1Label=SessionID cn1=$sessionid cnt=$repeatcnt spt=$sport dpt=$dport sourceTranslatedPort=$natsport destinationTranslatedPort=$natdport flexString1Label=Flags flexString1=$flags proto=$proto act=$action flexNumber1Label=Total bytes flexNumber1=$bytes in=$bytes_sent out=$bytes_received cn2Label=Packets cn2=$packets PanOSPacketsReceived=$pkts_received PanOSPacketsSent=$pkts_sent start=$cef-formatted-time_generated cn3Label=Elapsed time in seconds cn3=$elapsed cs2Label=URL Category cs2=$category externalId=$seqno reason=$session_end_reason PanOSDGl1=$dg_hier_level_1 PanOSDGl2=$dg_hier_level_2 PanOSDGl3=$dg_hier_level_3 PanOSDGl4=$dg_hier_level_4 PanOSVsysName=$vsys_name dvchost=$device_name cat=$action_source PanOSActionFlags=$actionflags PanOSSrcUUID=$src_uuid PanOSDstUUID=$dst_uuid PanOSTunnelID=$tunnelid PanOSMonitorTag=$monitortag PanOSParentSessionID=$parent_session_id PanOSParentStartTime=$parent_start_time PanOSTunnelType=$tunnel PanOSSCTPAssocID=$assoc_id PanOSSCTPChunks=$chunks PanOSSCTPChunkSent=$chunks_sent PanOSSCTPChunksRcv=$chunks_received PanOSRuleUUID=$rule_uuid PanOSHTTP2Con=$http2_connection PanLinkChange=$link_change_count PanPolicyID=$policy_id PanLinkDetail=$link_switches PanSDWANCluster=$sdwan_cluster PanSDWANDevice=$sdwan_device_type PanSDWANClustype=$sdwan_cluster_type PanSDWANSite=$sdwan_site PanDynamicUsrgrp=$dynusergroup_name PanXFFIP=$xff_ip PanSrcDeviceCat=$src_category PanSrcDeviceProf=$src_profile PanSrcDeviceModel=$src_model PanSrcDeviceVendor=$src_vendor PanSrcDeviceOS=$src_osfamily PanSrcDeviceOSv=$src_osversion PanSrcHostname=$src_host PanSrcMac=$src_mac PanDstDeviceCat=$dst_category PanDstDeviceProf=$dst_profile PanDstDeviceModel=$dst_model PanDstDeviceVendor=$dst_vendor PanDstDeviceOS=$dst_osfamily PanDstDeviceOSv=$dst_osversion PanDstHostname=$dst_host PanDstMac=$dst_mac PanContainerName=$container_id PanPODNamespace=$pod_namespace PanPODName=$pod_name PanSrcEDL=$src_edl PanDstEDL=$dst_edl PanGPHostID=$hostid PanEPSerial=$serialnumber PanSrcDAG=$src_dag PanDstDAG=$dst_dag PanHASessionOwner=$session_owner PanTimeHighRes=$high_res_timestamp PanASServiceType=$nssai_sst PanASServiceDiff=$nssai_sd -- Is there a rsyslog Template that exists to take the data in this format and log it? As you state, when rsyslogd gets a malformed message, or one with fields additional to that it understands or knows, it will not log it at all, will not write it to disk at all, not even a partial malformed log. -----Original Message----- From: David Lang <da...@lang.hm> Sent: Tuesday, August 13, 2024 12:50 PM To: Drumm, Daniel <daniel.dr...@dob.texas.gov> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang <da...@lang.hm> Subject: RE: [rsyslog] Formatting CEF to log. Drumm, Daniel wrote: > David - > > That appears to be it. Rsyslogd is not understanding the CEF formatting being > passed in and dropping it. When I go into the Palo and revert to the Default > syslog format, I start seeing the TRAFFIC messages logging into the catch-all > rule file. this is why I was having you write with the RSYSLOG_DebugFormat, it will show everything, including exactly what is received (the rawmeg field) when rsyslog receives things, it doesn't reject anything based on the formatting, it parses what it can out of the message, making a best-effort result out of malformed messages. The body of the message is just a text string. If the message is too long (see maxmessagelength) it will truncate/split the message. This could make it so that your database is not parsing the message correctly, but rsyslog would still recevie it and write it to a local file. try sending it to a copy of rsyslog that is not sending it to a database, just to a file, if your database is not accepting the message, that could be blocking other processing of the message (I don't expect this to be the case, but trying to work through the various possibilities) David Lang > So now the issue is, what must be done configuration wise to rsyslog template > to allow the CEF based fields to log? I can't pass to the Azure agent the > syslog in this format, it is expecting CEF. > > root@syslog-server-vnic-primary:/var/log/rsyslog# tail -f > all-the-stuff.log > FROMHOST: '10.40.0.210', fromhost-ip: '10.40.0.210', HOSTNAME: > 'DOB-FW-HA-1.OCI', PRI: 134, syslogtag 'Ashburn', programname: > 'Ashburn', APP-NAME: 'Ashburn', PROCID: '-', MSGID: '-', > TIMESTAMP: 'Aug 13 10:14:41', STRUCTURED-DATA: '-', > msg: ' FD1 1,2024/08/13 10:14:39,007959000479054,TRAFFIC,drop,2818,2024/08/13 > 10:14:39,184.168.122.26,10.40.1.98,0.0.0.0,0.0.0.0,Inbound Drop Logging > Rule,,,not-applicable,vsys1,Untrust,Trust,ethernet1/2,,OCI Syslog Server > Profile,2024/08/13 > 10:14:41,0,1,55585,3389,0,0,0x0,tcp,drop,56,56,0,1,2024/08/13 > 10:14:39,0,any,,7392297676068843279,0x0,Singapore,10.0.0.0-10.255.255.255,,1,0,policy-deny,0,0,0,0,,DOB-FW-HA-1,from-policy,,,0,,0,,N/A,0,0,0,0,d6ca6ff0-71e9-4f09-8e8d-2204deb98205,0,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,2024-08-13T10:14:41.917-05:00,,,unknown,unknown,unknown,1,,,not-applicable,no,no,0,NonProxyTraffic,' > escaped msg: ' FD1 1,2024/08/13 > 10:14:39,007959000479054,TRAFFIC,drop,2818,2024/08/13 > 10:14:39,184.168.122.26,10.40.1.98,0.0.0.0,0.0.0.0,Inbound Drop Logging > Rule,,,not-applicable,vsys1,Untrust,Trust,ethernet1/2,,OCI Syslog Server > Profile,2024/08/13 > 10:14:41,0,1,55585,3389,0,0,0x0,tcp,drop,56,56,0,1,2024/08/13 > 10:14:39,0,any,,7392297676068843279,0x0,Singapore,10.0.0.0-10.255.255.255,,1,0,policy-deny,0,0,0,0,,DOB-FW-HA-1,from-policy,,,0,,0,,N/A,0,0,0,0,d6ca6ff0-71e9-4f09-8e8d-2204deb98205,0,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,2024-08-13T10:14:41.917-05:00,,,unknown,unknown,unknown,1,,,not-applicable,no,no,0,NonProxyTraffic,' > inputname: imudp rawmsg: '<134>Aug 13 10:14:41 DOB-FW-HA-1.OCI Ashburn FD1 > 1,2024/08/13 10:14:39,007959000479054,TRAFFIC,drop,2818,2024/08/13 > 10:14:39,184.168.122.26,10.40.1.98,0.0.0.0,0.0.0.0,Inbound Drop Logging > Rule,,,not-applicable,vsys1,Untrust,Trust,ethernet1/2,,OCI Syslog Server > Profile,2024/08/13 > 10:14:41,0,1,55585,3389,0,0,0x0,tcp,drop,56,56,0,1,2024/08/13 > 10:14:39,0,any,,7392297676068843279,0x0,Singapore,10.0.0.0-10.255.255.255,,1,0,policy-deny,0,0,0,0,,DOB-FW-HA-1,from-policy,,,0,,0,,N/A,0,0,0,0,d6ca6ff0-71e9-4f09-8e8d-2204deb98205,0,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,2024-08-13T10:14:41.917-05:00,,,unknown,unknown,unknown,1,,,not-applicable,no,no,0,NonProxyTraffic,' > $!: > $.: > $/: > > > -----Original Message----- > From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of Drumm, > Daniel via rsyslog > Sent: Tuesday, August 13, 2024 9:51 AM > To: David Lang <da...@lang.hm> > Cc: Drumm, Daniel <daniel.dr...@dob.texas.gov>; Drumm, Daniel via > rsyslog <rsyslog@lists.adiscon.com> > Subject: Re: [rsyslog] Formatting CEF to log. > > David - > > Here is tcpdump -v, showing that I get both the SYSTEM and TRAFFIC messages > from the Palo Alto. What you see is that the TRAFFIC messages have been set > to pass in all the variables that conform to CEF specification. > > CEF Specification: (Page 8) calls for a number of extended variables getting > passed, and this is what Palo Alto recommends: > https://docs/ > .paloaltonetworks.com%2Fcontent%2Fdam%2Ftechdocs%2Fen_US%2Fpdf%2Fcef%2 > Fpan-os-10-0-cef-configuration-guide.pdf&data=05%7C02%7CDaniel.Drumm%4 > 0dob.texas.gov%7C9980901584484dd612f508dcbbc05920%7C32a53eeb27594ef4af > 2858fd990c7d67%7C1%7C0%7C638591682922746424%7CUnknown%7CTWFpbGZsb3d8ey > JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7 > C%7C%7C&sdata=Zv4h1XxlX1jqWnWxNqR6foCMHMaD5%2FfJkVOIEqFzpkE%3D&reserve > d=0 > > So you can see that a SYSTEM message is parsed by rsyslog, but a TRAFFIC > message is not. Is this perhaps because the rsyslogd does not know how, or > requires some additional configuration, to pass in CEF messages? None of the > guides I have read that speak about this say this. The goal here is to use > rsyslog to pass these messages up to Azure Sentinel. > > To that end, I have installed Microsoft's Azure Forwarder: > root@syslog-server-vnic-primary:/home/ubuntu# sudo netstat -tlnp | grep 28330 > tcp 0 0 127.0.0.1:28330 0.0.0.0:* LISTEN > 1762/mdsd > > Then I have a directive to forward all messages to this port. But also to > write all messages to local file log. > > But right now, I cannot get a TRAFFIC message to log, even to *.*. > > I can go into the firewall and reduce or alter the message, to make it > contain fewer variables, but right now it appears to me as if the rsyslogd > simply discards it, perhaps due to its formatting. I am unsure and simply > want to get the TRAFFIC messages to write to disk in any manner. Both SYSTEM > and TRAFFIC are local0.sev6 messages. > > > root@syslog-server-vnic-primary:/home/ubuntu# tcpdump -v "port 514" > tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size > 262144 bytes > > -- SYSTEM Message > > 14:38:47.045722 IP (tos 0x0, ttl 63, id 32310, offset 0, flags [DF], proto > UDP (17), length 639) > 10.40.0.210.45138 > 10.30.0.18.syslog: SYSLOG, length: 611 > Facility local0 (16), Severity info (6) > Msg: Aug 13 09:38:52 DOB-FW-HA-1.OCI Ashburn FD1 CEF:0|Palo > Alto\0x0aNetworks|PAN-OS|11.2.0|userid|SYSTEM|1|rt=Aug 13 2024 > 14:38:51 GMT\0x0adeviceExternalId=007959000479054 cs3Label=Virtual > System cs3= fname= flexString2Label=Module\0x0aflexString2=general > msg="gRPC connection to > identity.services-edge.paloaltonetworks.com:443 is established, > 10.40.0.210:43800 -> 34.136.155.117:443 time: 2024-08-13 09:38:52" > externalId=7392297676068401058 cat=cuid-conn > PanOSDGl1=0\0x0aPanOSDGl2=0 PanOSDGl3=0 PanOSDGl4=0\0x0aPanOSVsysName= > dvchost=DOB-FW-HA-1 > PanOSActionFlags=0x0\0x0aanOSTimeGeneratedHighResolution=2024-08-13T09 > :38:52.345-05:00\0x0a > > --- TRAFFIC Message (which you can see is much longer, passing all the > CEF parameters) > > 14:38:51.433733 IP (tos 0x0, ttl 63, id 34341, offset 0, flags [+], proto UDP > (17), length 1500) > 10.40.0.210.40933 > 10.30.0.18.syslog: SYSLOG, length: 1472 > Facility local0 (16), Severity info (6) > Msg: Aug 13 09:38:56 DOB-FW-HA-1.OCI Ashburn FD1 CEF:0|Palo > Alto Networks|PAN-OS|11.2.0|drop|TRAFFIC|1|rt=Aug 13 2024 14:38:52 > GMT\0x0adeviceExternalId=007959000479054 src=165.154.12.82 > dst=10.40.1.97 > sourceTranslatedAddress=0.0.0.0\0x0adestinationTranslatedAddress=0.0.0 > .0 cs1Label=Rule cs1=Inbound Drop Logging Rule suser= duser= > app=not-applicable\0x0acs3Label=Virtual System cs3=vsys1 > cs4Label=Source Zone cs4=Untrust cs5Label=Destination Zone > cs5=Trust\0x0adeviceInboundInterface=ethernet1/2 > deviceOutboundInterface= cs6Label=LogProfile cs6=OCI Syslog Server > Profile\0x0acn1Label=SessionID cn1=0 cnt=1 spt=59748 dpt=8085 > sourceTranslatedPort=0\0x0adestinationTranslatedPort=0 > flexString1Label=Flags flexString1=0x0 proto=tcp > act=drop\0x0aflexNumber1Label=Total bytes flexNumber1=58 in=58 out=0 > cn2Label=Packets\0x0acn2=1 PanOSPacketsReceived=0 > PanOSPacketsSent=1\0x0astart=Aug 13 2024 14:38:52 GMT cn3Label=Elapsed > time in seconds cn3=0 cs2Label=URL Category\0x0acs2=any externalId=73 9 > 2297676068843056 reason=policy-deny PanOSDGl1=0\0x0aPanOSDGl2=0 > PanOSDGl3=0 PanOSDGl4=0\0x0aPanOSVsysName= dvchost=DOB-FW-HA-1 > cat=from-policy PanOSActionFlags=0x0\0x0aPanOSSrcUUID= PanOSDstUUID= > PanOSTunnelID=0 PanOSMonitorTag=\0x0aPanOSParentSessionID=0 > PanOSParentStartTime= PanOSTunnelType=N/A\0x0aPanOSSCTPAssocID=0 > PanOSSCTPChunks=0 PanOSSCTPChunkSent=0\0x0aPanOSSCTPChunksRcv=0 > PanOSRuleUUID=d6ca6ff0-71e9-4f09-8e8d-2204deb98205 > PanOSHTTP2Con=0\0x0aPanLinkChange=0 PanPolicyID= > PanLinkDetail=\0x0aPanSDWANCluster= > PanSDWANDevice=\0x0aPanSDWANClustype= > > > -----Original Message----- > From: David Lang <da...@lang.hm> > Sent: Monday, August 12, 2024 6:37 PM > To: Drumm, Daniel <daniel.dr...@dob.texas.gov> > Cc: David Lang <da...@lang.hm>; Drumm, Daniel via rsyslog > <rsyslog@lists.adiscon.com> > Subject: RE: [rsyslog] Formatting CEF to log. > > On Mon, 12 Aug 2024, Drumm, Daniel wrote: > >> Messages roll into the rsyslog.debug file from SYSTEM without issue and are >> processed: >> >> Debug line with all properties: >> FROMHOST: '10.40.0.210', fromhost-ip: '10.40.0.210', HOSTNAME: >> 'DOB-FW-HA-1.OCI', PRI: 131, syslogtag 'Ashburn', programname: >> 'Ashburn', APP-NAME: 'Ashburn', PROCID: '-', MSGID: '-', >> TIMESTAMP: 'Aug 12 18:23:59', STRUCTURED-DATA: '-', >> msg: ' FD1 CEF:0|Palo Alto#012Networks|PAN-OS|11.2.0|userid|SYSTEM|4|rt=Aug >> 12 2024 23:23:58 GMT#012deviceExternalId=007959000479054 cs3Label=Virtual >> System cs3= fname= flexString2Label=Module#012flexString2=general msg="gRPC >> connection to identity.services-edge.paloaltonetworks.com:443 is broken, >> error: rpc error: code = Unknown desc = [UploadCUID] [007959000479054] >> tenantId is empty time: 2024-08-12 18:23:59" externalId=7392297676068400080 >> cat=cuid-conn PanOSDGl1=0#012PanOSDGl2=0 PanOSDGl3=0 >> PanOSDGl4=0#012PanOSVsysName= dvchost=DOB-FW-HA-1 >> PanOSActionFlags=0x0#012anOSTimeGeneratedHighResolution=2024-08-12T18:23:59.046-05:00' >> escaped msg: ' FD1 CEF:0|Palo >> Alto#012Networks|PAN-OS|11.2.0|userid|SYSTEM|4|rt=Aug 12 2024 23:23:58 >> GMT#012deviceExternalId=007959000479054 cs3Label=Virtual System cs3= fname= >> flexString2Label=Module#012flexString2=general msg="gRPC connection to >> identity.services-edge.paloaltonetworks.com:443 is broken, error: rpc error: >> code = Unknown desc = [UploadCUID] [007959000479054] tenantId is empty time: >> 2024-08-12 18:23:59" externalId=7392297676068400080 cat=cuid-conn >> PanOSDGl1=0#012PanOSDGl2=0 PanOSDGl3=0 PanOSDGl4=0#012PanOSVsysName= >> dvchost=DOB-FW-HA-1 >> PanOSActionFlags=0x0#012anOSTimeGeneratedHighResolution=2024-08-12T18:23:59.046-05:00' >> inputname: imudp rawmsg: '<131>Aug 12 18:23:59 DOB-FW-HA-1.OCI Ashburn FD1 >> CEF:0|Palo Alto#012Networks|PAN-OS|11.2.0|userid|SYSTEM|4|rt=Aug 12 2024 >> 23:23:58 GMT#012deviceExternalId=007959000479054 cs3Label=Virtual System >> cs3= fname= flexString2Label=Module#012flexString2=general msg="gRPC >> connection to identity.services-edge.paloaltonetworks.com:443 is broken, >> error: rpc error: code = Unknown desc = [UploadCUID] [007959000479054] >> tenantId is empty time: 2024-08-12 18:23:59" externalId=7392297676068400080 >> cat=cuid-conn PanOSDGl1=0#012PanOSDGl2=0 PanOSDGl3=0 >> PanOSDGl4=0#012PanOSVsysName= dvchost=DOB-FW-HA-1 >> PanOSActionFlags=0x0#012anOSTimeGeneratedHighResolution=2024-08-12T18:23:59.046-05:00' >> $!: >> $.: >> $/: >> >> TRAFFIC messages are not processed despite hitting the VNIC. > > did you get this by logging *.* to a file? or by logging things from > 10.40.0.210, or ?? > > can you show a tcpdump of a message getting through from the device vs the > traffic messages that are failing? > > David Lang > > >> -----Original Message----- >> From: David Lang <da...@lang.hm> >> Sent: Monday, August 12, 2024 6:15 PM >> To: Drumm, Daniel <daniel.dr...@dob.texas.gov> >> Cc: David Lang <da...@lang.hm>; Drumm, Daniel via rsyslog >> <rsyslog@lists.adiscon.com> >> Subject: RE: [rsyslog] Formatting CEF to log. >> >> try logging all traffic with the template RSYSLOG_DebugFormat for a >> short time (long enough that it should have some of these TRAFFIC >> messages) and find the messages in there it could be that the >> messages is not showing up as you would expect >> >> David Lang >> >> On Mon, 12 Aug 2024, Drumm, Daniel wrote: >> >>> Date: Mon, 12 Aug 2024 23:11:15 +0000 >>> From: "Drumm, Daniel" <daniel.dr...@dob.texas.gov> >>> To: David Lang <da...@lang.hm> >>> Cc: "Drumm, Daniel via rsyslog" <rsyslog@lists.adiscon.com> >>> Subject: RE: [rsyslog] Formatting CEF to log. >>> >>> I removed the 'stop' from every single directive, as this is confusing the >>> matter. >>> >>> It looks as such, and yet the TRAFFIC messages do not log anywhere. >>> >>> -- >>> >>> root@syslog-server-vnic-primary:/etc/rsyslog.d# more >>> 11-paloalto-cef.conf # Use the template for logs coming from your >>> Palo Alto firewall if $fromhost-ip == '10.40.0.210' then { >>> *.* /var/log/rsyslog/palo-alto-cef.log >>> } >>> root@syslog-server-vnic-primary:/etc/rsyslog.d# more >>> 10-paloalto-traffic.conf # Define a template for Palo Alto TRAFFIC >>> logs $template PaloAltoTraffic,"/var/log/rsyslog/paloalto/traffic.log" >>> >>> # Filter and direct TRAFFIC logs to the specified file if >>> $fromhost-ip == '10.40.0.210' and $msg contains 'TRAFFIC' then { >>> action(type="omfile" dynaFile="PaloAltoTraffic") } >>> root@syslog-server-vnic-primary:/etc/rsyslog.d# more 10- >>> 10-azuremonitoragent-omfwd.conf 10-paloalto-traffic.conf >>> root@syslog-server-vnic-primary:/etc/rsyslog.d# more >>> 10-azuremonitoragent-omfwd.conf # Azure Monitor Agent configuration: >>> forward logs to azuremonitoragent >>> >>> template(name="AMA_RSYSLOG_TraditionalForwardFormat" type="string" >>> string="<%PRI%>%TIMESTAMP% %HOSTNAME% >>> %syslogtag%%msg:::sp-if-no-1st-sp%%msg%") >>> # queue.workerThreads sets the maximum worker threads, it will scale >>> back to 0 if there is no activity # Forwarding all events through >>> TCP port >>> *.* action(type="omfwd" >>> template="AMA_RSYSLOG_TraditionalForwardFormat" >>> queue.type="LinkedList" >>> queue.filename="omfwd-azuremonitoragent" >>> queue.maxFileSize="32m" >>> action.resumeRetryCount="-1" >>> action.resumeInterval="5" >>> action.reportSuspension="on" >>> action.reportSuspensionContinuation="on" >>> queue.size="25000" >>> queue.workerThreads="100" >>> queue.dequeueBatchSize="2048" >>> queue.saveonshutdown="on" >>> target="127.0.0.1" Port="28330" Protocol="tcp") >>> >>> -----Original Message----- >>> From: David Lang <da...@lang.hm> >>> Sent: Monday, August 12, 2024 5:58 PM >>> To: Drumm, Daniel <daniel.dr...@dob.texas.gov> >>> Cc: David Lang <da...@lang.hm>; Drumm, Daniel via rsyslog >>> <rsyslog@lists.adiscon.com> >>> Subject: RE: [rsyslog] Formatting CEF to log. >>> >>> On Mon, 12 Aug 2024, Drumm, Daniel wrote: >>> >>>> Date: Mon, 12 Aug 2024 22:41:21 +0000 >>>> From: "Drumm, Daniel" <daniel.dr...@dob.texas.gov> >>>> To: David Lang <da...@lang.hm> >>>> Cc: "Drumm, Daniel via rsyslog" <rsyslog@lists.adiscon.com> >>>> Subject: RE: [rsyslog] Formatting CEF to log. >>>> >>>> Here is that file with the -o flag: >>>> >>>> root@syslog-server-vnic-primary:/tmp# more rsyslog_combined.conf ## >>>> full conf created by rsyslog version 8.2001.0 at 2024-08-12 >>>> 22:38:01 ## >>>> >>>> ##### BEGIN CONFIG: /etc/rsyslog.conf # /etc/rsyslog.conf >>>> configuration file for rsyslog # # For more information install >>>> rsyslog-doc and see # >>>> /usr/share/doc/rsyslog-doc/html/configuration/index.html >>>> # >>>> # Default logging rules can be found in >>>> /etc/rsyslog.d/50-default.conf >>>> >>>> >>>> ################# >>>> #### MODULES #### >>>> ################# >>>> module(load="imudp") # needs to be done just once input(type="imudp" >>>> port="514") >>>> >>>> $ModLoad imuxsock # needs to be done just once >>>> >>>> # $AllowedSender - specifies which remote systems are allowed to >>>> send syslog messages to rsyslogd # -------------- $AllowedSender >>>> UDP, 127.0.0.1, 10.40.0.0/16, 10.30.0.0/23 >>>> >>>> $SystemLogSocketFlowControl on # enable flow control (use if >>>> needed) $ActionFileEnableSync off >>>> >>>> # A template that resambles traditional syslogd file output: >>>> $template TraditionalFormat,"%timegenerated% %HOSTNAME% >>>> %syslogtag%%msg:::drop-last-lf%\n" >>>> # A more verbose template: >>>> $template >>>> precise,"%syslogpriority%,%syslogfacility%,%timegenerated::fulltime%,%HOSTNAME%,%syslogtag%,%msg%\n" >>>> # A template that resembles RFC 3164 on-the-wire format: >>>> # (yes, there is NO space betwen syslogtag and msg! that's >>>> important!) $template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% >>>> %syslogtag%%msg%" >>>> # a template resembling traditional wallmessage format: >>>> $template wallmsg,"\r\n\7Message from syslogd@%HOSTNAME% at >>>> %timegenerated% ...\r\n %syslogtag%%msg%\n\r" >>>> # The template below emulates winsyslog format, but we need to >>>> check the time # stamps used. It is also a good sampleof the property >>>> replacer in action. >>>> $template >>>> WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,%timegen >>>> e >>>> r >>>> a >>>> ted:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,%timegene >>>> r a t ed:12:19:date-rfc3339%,%syslogfacility%,%syslo >>>> gpriority%,%syslogtag%%msg%\n" >>>> # A template used for database writing (notice it *is* an actual # >>>> sql-statement): >>>> $template dbFormat,"insert into SystemEvents (Message, >>>> Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, >>>> InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, >>>> '%HOSTNAME%',%syslogpr iority%, '%timereported:::date-mysql%', >>>> '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",sql >>>> >>>> >>>> # Turn off message reception via local log socket; # local messages >>>> are retrieved through imjournal now. >>>> # $OmitLocalLogging on >>>> >>>> ########################### >>>> #### GLOBAL DIRECTIVES #### >>>> ########################### >>>> $DebugLevel 2 >>>> $DebugFile /var/log/rsyslog/rsyslog.debug # # Use traditional >>>> timestamp format. >>>> # To enable high precision timestamps, comment out the following line. >>>> # >>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >>>> >>>> # Filter duplicated messages >>>> $RepeatedMsgReduction on >>>> >>>> # >>>> # Set the default permissions for all log files. >>>> # >>>> $FileOwner syslog >>>> $FileGroup syslog >>>> $FileCreateMode 0660 >>>> $DirCreateMode 0755 >>>> $Umask 0022 >>>> $PrivDropToUser syslog >>>> $PrivDropToGroup syslog >>>> >>>> # >>>> # Where to place spool and state files # $WorkDirectory >>>> /var/log/rsyslog >>>> >>>> # Include all config files in /etc/rsyslog.d/ >>>> $IncludeConfig /etc/rsyslog.d/ # whole directory (must >>>> contain the final slash)/etc/rsyslog.d/ >>>> ##### BEGIN CONFIG: /etc/rsyslog.d/50-default.conf >>>> >>>> ##### BEGIN CONFIG: /etc/rsyslog.d/21-cloudinit.conf >>>> >>>> ##### BEGIN CONFIG: /etc/rsyslog.d/20-ufw.conf >>>> >>>> ##### BEGIN CONFIG: /etc/rsyslog.d/12-azuremonitoragent-omfwd.conf >>>> >>>> ##### BEGIN CONFIG: /etc/rsyslog.d/11-paloalto-cef.conf >>>> >>>> ##### BEGIN CONFIG: /etc/rsyslog.d/10-paloalto-traffic.conf >>>> >>>> ##### BEGIN CONFIG: /etc/rsyslog.d/10-azuremonitoragent-omfwd.conf >>>> # Azure Monitor Agent configuration: forward logs to >>>> azuremonitoragent >>>> >>>> template(name="AMA_RSYSLOG_TraditionalForwardFormat" type="string" >>>> string="<%PRI%>%TIMESTAMP% %HOSTNAME% >>>> %syslogtag%%msg:::sp-if-no-1st-sp%%msg%") >>>> # queue.workerThreads sets the maximum worker threads, it will >>>> scale back to 0 if there is no activity # Forwarding all events >>>> through TCP port >>>> *.* action(type="omfwd" >>>> template="AMA_RSYSLOG_TraditionalForwardFormat" >>>> queue.type="LinkedList" >>>> queue.filename="omfwd-azuremonitoragent" >>>> queue.maxFileSize="32m" >>>> action.resumeRetryCount="-1" >>>> action.resumeInterval="5" >>>> action.reportSuspension="on" >>>> action.reportSuspensionContinuation="on" >>>> queue.size="25000" >>>> queue.workerThreads="100" >>>> queue.dequeueBatchSize="2048" >>>> queue.saveonshutdown="on" >>>> target="127.0.0.1" Port="28330" Protocol="tcp") >>>> >>>> ##### END CONFIG: /etc/rsyslog.d/10-azuremonitoragent-omfwd.conf >>>> # Define a template for Palo Alto TRAFFIC logs $template >>>> PaloAltoTraffic,"/var/log/rsyslog/paloalto/traffic.log" >>>> >>>> # Filter and direct TRAFFIC logs to the specified file if >>>> $fromhost-ip == '10.40.0.210' and $msg contains 'TRAFFIC' then { >>>> action(type="omfile" dynaFile="PaloAltoTraffic") >>>> stop >>>> } >>> >>> because you have stop here, your TRAFFIC logs will not be processed >>> past this point. That is why they don't appear in the >>> palo-alto-cef.log file below >>> >>>> ##### END CONFIG: /etc/rsyslog.d/10-paloalto-traffic.conf >>>> # Define a template for CEF logs >>>> $template PaloAltoCEF,"/var/log/rsyslog/e%.log" >>>> >>>> # Use the template for logs coming from your Palo Alto firewall if >>>> $fromhost-ip == '10.40.0.210' then { >>>> *.* /var/log/rsyslog/palo-alto-cef.log >>>> stop >>>> } >>>> >>>> ##### END CONFIG: /etc/rsyslog.d/11-paloalto-cef.conf >>>> # Azure Monitor Agent configuration: forward logs to >>>> azuremonitoragent >>>> >>>> # Azure Monitor Agent configuration: forward logs to >>>> azuremonitoragent template(name="AMA_RSYSLOG_TraditionalForwardFormat" >>>> type="string" >>>> string="<%PRI%>%TIMESTAMP% %HOSTNAME% >>>> %syslogtag%%msg:::sp-if-no-1st-sp%%msg%") >>>> >>>> *.* action(type="omfwd" >>>> template="AMA_RSYSLOG_TraditionalForwardFormat" >>>> queue.type="LinkedList" >>>> queue.filename="omfwd-azuremonitoragent" >>>> queue.maxFileSize="32m" >>>> action.resumeRetryCount="-1" >>>> action.resumeInterval="30" >>>> action.reportSuspension="on" >>>> action.reportSuspensionContinuation="on" >>>> queue.size="100000" >>>> queue.discardmark="97500" >>>> queue.discardseverity="0" >>>> queue.checkpointInterval="100" >>>> queue.workerThreads="4" >>>> queue.timeoutEnqueue="10000" >>>> queue.timeoutWorkerthreadShutdown="60000" >>>> queue.saveOnShutdown="on" >>>> target="127.0.0.1" Port="28330" Protocol="tcp" >>>> ) >>>> >>>> ##### END CONFIG: /etc/rsyslog.d/12-azuremonitoragent-omfwd.conf >>>> # Log kernel generated UFW log messages to file :msg,contains,"[UFW " >>>> /var/log/ufw.log >>>> >>>> # Uncomment the following to stop logging anything that matches the last >>>> rule. >>>> # Doing this will stop logging kernel generated UFW log messages to >>>> the file # normally containing kern.* messages (eg, >>>> /var/log/kern.log) #& stop >>>> >>>> ##### END CONFIG: /etc/rsyslog.d/20-ufw.conf >>>> # Log cloudinit generated log messages to file :syslogtag, isequal, >>>> "[CLOUDINIT]" /var/log/cloud-init.log >>>> >>>> # comment out the following line to allow CLOUDINIT messages through. >>>> # Doing so means you'll also get CLOUDINIT messages in >>>> /var/log/syslog & stop >>>> >>>> ##### END CONFIG: /etc/rsyslog.d/21-cloudinit.conf >>>> # Default rules for rsyslog. >>>> # >>>> # For more information see rsyslog.conf(5) and >>>> /etc/rsyslog.conf >>>> >>>> # >>>> # First some standard log files. Log by facility. >>>> # >>>> auth,authpriv.* /var/log/rsyslog/auth.log >>>> cron.* /var/log/rsyslog/cron.log >>>> daemon.* /var/log/rsyslog/daemon.log >>>> kern.* /var/log/rsyslog/kern.log >>>> lpr.* /var/log/rsyslog/lpr.log >>>> mail.* /var/log/rsyslog/mail.log >>>> user.* /var/log/rsyslog/user.log >>>> local0.info /var/log/rsyslog/local0_info.log >>>> #*.* >>>> /var/log/rsyslog/traditionalfile.log;TraditionalFormat # log to a >>>> file in the traditional format >>>> *.* /var/log/rsyslog/all-the-stuff.log >>>> >>>> # >>>> # Logging for the mail system. Split it up so that # it is easy to >>>> write scripts to parse these files. >>>> # >>>> # mail.info -/var/log/rsyslog/mail.info >>>> # mail.warn -/var/log/rsyslog/mail.warn >>>> # mail.err /var/log/rsyslog/mail.err >>>> >>>> # >>>> # Some "catch-all" log files. >>>> # >>>> *.=debug;auth,authpriv.none;news.none;mail.none /var/log/debug >>>> PreprocFileLineNumber(33) >>>> *.=info;*.=notice;*.=warn;auth,authpriv.none;cron,daemon.none;mail,news.none >>>> /var/log/messages >>>> PreprocFileLineNumber(37) >>>> >>>> # >>>> # Emergencies are sent to everybody logged in. >>>> # >>>> *.emerg :omusrmsg:* >>>> >>>> # >>>> # I like to have messages displayed on the console, but only on a >>>> virtual # console I usually leave idle. >>>> # >>>> #daemon,mail.*;# news.=crit;news.=err;news.=notice;# >>>> *.=debug;*.=info;# *.=notice;*.=warn /dev/tty8 >>>> >>>> ##### END CONFIG: /etc/rsyslog.d/50-default.conf >>>> # whole directory (must contain the final slash) >>>> >>>> ##### END CONFIG: /etc/rsyslog.conf >>>> >>>> -----Original Message----- >>>> From: David Lang <da...@lang.hm> >>>> Sent: Monday, August 12, 2024 5:32 PM >>>> To: Drumm, Daniel <daniel.dr...@dob.texas.gov> >>>> Cc: David Lang <da...@lang.hm>; Drumm, Daniel via rsyslog >>>> <rsyslog@lists.adiscon.com> >>>> Subject: RE: [rsyslog] Formatting CEF to log. >>>> >>>> start rsyslog with the command line option -o /path/to/file >>>> >>>> that will write the combined config files to that file as rsyslog >>>> sees it. Post that and we can look for other interactions that may >>>> cause some logs to be thrown away first >>>> >>>> look especially for othe stop commands earlier than the clause >>>> looking at the fromhost-ip >>>> >>>> David Lang >>>> >>>> On Mon, 12 Aug 2024, Drumm, Daniel wrote: >>>> >>>>> Date: Mon, 12 Aug 2024 22:18:02 +0000 >>>>> From: "Drumm, Daniel" <daniel.dr...@dob.texas.gov> >>>>> To: David Lang <da...@lang.hm>, >>>>> "Drumm, Daniel via rsyslog" <rsyslog@lists.adiscon.com> >>>>> Subject: RE: [rsyslog] Formatting CEF to log. >>>>> >>>>> Thank you, I should have corrected that as I iterated. >>>>> >>>>> What I see with this directive: >>>>> >>>>> root@syslog-server-vnic-primary:/etc/rsyslog.d# more >>>>> 10-paloalto-cef.conf # Define a template for CEF logs $template >>>>> PaloAltoCEF,"/var/log/rsyslog/DOB-FW-HA-1.OCI/%programname%.log" >>>>> >>>>> # Use the template for logs coming from your Palo Alto firewall if >>>>> $fromhost-ip == '10.40.0.210' then { >>>>> *.* /var/log/rsyslog/palo-alto-cef.log >>>>> stop >>>>> } >>>>> >>>>> Is that the log has SYSTEM messages, but none of the Traffic messages: >>>>> >>>>> root@syslog-server-vnic-primary:/var/log/rsyslog# tail -f >>>>> palo-alto-cef.log Aug 12 17:07:12 DOB-FW-HA-1.OCI Ashburn FD1 >>>>> CEF:0|Palo >>>>> Alto#012Networks|PAN-OS|11.2.0|url-filtering|SYSTEM|1|rt=Aug 12 >>>>> 2024 >>>>> 22:07:12 GMT#012deviceExternalId=007959000479054 cs3Label=Virtual >>>>> System cs3= fname= flexString2Label=Module#012flexString2=general >>>>> msg="PAN-DB was upgraded to version 20240812.20332." >>>>> externalId=7392297676068399997 cat=upgrade-url-database-success >>>>> PanOSDGl1=0#012PanOSDGl2=0 PanOSDGl3=0 >>>>> PanOSDGl4=0#012PanOSVsysName= >>>>> dvchost=DOB-FW-HA-1 >>>>> PanOSActionFlags=0x0#012anOSTimeGeneratedHighResolution=2024-08-12T17: >>>>> 07:12.563-05:00 Aug 12 17:08:17 DOB-FW-HA-1.OCI Ashburn FD1 >>>>> CEF:0|Palo >>>>> Alto#012Networks|PAN-OS|11.2.0|url-filtering|SYSTEM|1|rt=Aug 12 >>>>> 2024 >>>>> 22:08:17 GMT#012deviceExternalId=007959000479054 cs3Label=Virtual >>>>> System cs3= fname= flexString2Label=Module#012flexString2=general >>>>> msg="PAN-DB was upgraded to version 20240812.20333." >>>>> externalId=7392297676068399998 cat=upgrade-url-database-success >>>>> PanOSDGl1=0#012PanOSDGl2=0 PanOSDGl3=0 >>>>> PanOSDGl4=0#012PanOSVsysName= >>>>> dvchost=DOB-FW-HA-1 >>>>> PanOSActionFlags=0x0#012anOSTimeGeneratedHighResolution=2024-08-12T17: >>>>> 08:17.737-05:00 Aug 12 17:08:50 DOB-FW-HA-1.OCI Ashburn FD1 >>>>> CEF:0|Palo Alto#012Networks|PAN-OS|11.2.0|userid|SYSTEM|1|rt=Aug >>>>> 12 >>>>> 2024 22:08:49 >>>>> GMT#012deviceExternalId=007959000479054 cs3Label=Virtual System >>>>> cs3= fname= flexString2Label=Module#012flexString2=general >>>>> msg="gRPC connection to >>>>> identity.services-edge.paloaltonetworks.com:443 is established, >>>>> 10.40.0.210:46846 -> 34.136.155.117:443 time: >>>>> 2024-08-12 17:08:50" externalId=7392297676068399999 cat=cuid-conn >>>>> PanOSDGl1=0#012PanOSDGl2=0 PanOSDGl3=0 >>>>> PanOSDGl4=0#012PanOSVsysName= >>>>> dvchost=DOB-FW-HA-1 >>>>> PanOSActionFlags=0x0#012anOSTimeGeneratedHighResolution=2024-08-12T17: >>>>> 08:50.292-05:00 Aug 12 17:08:54 DOB-FW-HA-1.OCI Ashburn FD1 >>>>> CEF:0|Palo Alto#012Networks|PAN-OS|11.2.0|userid|SYSTEM|4|rt=Aug >>>>> 12 >>>>> 2024 22:08:53 >>>>> GMT#012deviceExternalId=007959000479054 cs3Label=Virtual System >>>>> cs3= fname= flexString2Label=Module#012flexString2=general >>>>> msg="gRPC connection to >>>>> identity.services-edge.paloaltonetworks.com:443 is broken, error: >>>>> rpc error: code = Unknown desc = [UploadCUID] [007959000479054] tenantId >>>>> is empty time: 2024-08-12 17:08:54" >>>>> externalId=7392297676068400000 cat=cuid-conn >>>>> PanOSDGl1=0#012PanOSDGl2=0 PanOSDGl3=0 >>>>> PanOSDGl4=0#012PanOSVsysName= >>>>> dvchost=DOB-FW-HA-1 >>>>> PanOSActionFlags=0x0#012anOSTimeGeneratedHighResolution=2024-08-12T17: >>>>> 08:54.050-05:00 >>>>> >>>>> Yet I know for a fact that the TRAFFIC messages are hitting the NIC of >>>>> the syslog server: >>>>> >>>>> root@syslog-server-vnic-primary:/etc/rsyslog.d# tcpdump -v "port 514" >>>>> tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture >>>>> size >>>>> 262144 bytes >>>>> 22:16:32.237523 IP (tos 0x0, ttl 63, id 30088, offset 0, flags [+], proto >>>>> UDP (17), length 1500) >>>>> 10.40.0.210.40933 > 10.30.0.18.syslog: SYSLOG, length: 1472 >>>>> Facility local0 (16), Severity info (6) >>>>> Msg: Aug 12 17:16:36 DOB-FW-HA-1.OCI Ashburn FD1 CEF:0|Palo >>>>> Alto Networks|PAN-OS|11.2.0|drop|TRAFFIC|1|rt=Aug 12 2024 22:16:31 >>>>> GMT\0x0adeviceExternalId=007959000479054 src=162.216.149.73 >>>>> dst=10.40.1.97 >>>>> sourceTranslatedAddress=0.0.0.0\0x0adestinationTranslatedAddress=0.0. >>>>> 0 >>>>> .0 cs1Label=Rule cs1=Inbound Drop Logging Rule suser= duser= >>>>> app=not-applicable\0x0acs3Label=Virtual System cs3=vsys1 >>>>> cs4Label=Source Zone cs4=Untrust cs5Label=Destination Zone >>>>> cs5=Trust\0x0adeviceInboundInterface=ethernet1/2 >>>>> deviceOutboundInterface= cs6Label=LogProfile cs6=OCI Syslog Server >>>>> Profile\0x0acn1Label=SessionID cn1=0 cnt=1 spt=54491 dpt=9408 >>>>> sourceTranslatedPort=0\0x0adestinationTranslatedPort=0 >>>>> flexString1Label=Flags flexString1=0x0 proto=tcp >>>>> act=drop\0x0aflexNumber1Label=Total bytes flexNumber1=58 in=58 >>>>> out=0 >>>>> cn2Label=Packets\0x0acn2=1 PanOSPacketsReceived=0 >>>>> PanOSPacketsSent=1\0x0astart=Aug 12 2024 22:16:31 GMT >>>>> cn3Label=Elapsed time in seconds cn3=0 cs2Label=URL >>>>> Category\0x0acs2=any externalId=7 >>>> 392297676068829896 reason=policy-deny PanOSDGl1=0\0x0aPanOSDGl2=0 >>>> PanOSDGl3=0 PanOSDGl4=0\0x0aPanOSVsysName= dvchost=DOB-FW-HA-1 >>>> cat=from-policy PanOSActionFlags=0x0\0x0aPanOSSrcUUID= >>>> PanOSDstUUID= >>>> PanOSTunnelID=0 PanOSMonitorTag=\0x0aPanOSParentSessionID=0 >>>> PanOSParentStartTime= PanOSTunnelType=N/A\0x0aPanOSSCTPAssocID=0 >>>> PanOSSCTPChunks=0 PanOSSCTPChunkSent=0\0x0aPanOSSCTPChunksRcv=0 >>>> PanOSRuleUUID=d6ca6ff0-71e9-4f09-8e8d-2204deb98205 >>>> PanOSHTTP2Con=0\0x0aPanLinkChange=0 PanPolicyID= >>>> PanLinkDetail=\0x0aPanSDWANCluster= >>>> PanSDWANDevice=\0x0aPanSDWANClustype >>>>> 22:16:32.237613 IP (tos 0x0, ttl 63, id 30089, offset 0, flags >>>>> [+], proto UDP (17), length 1500) >>>>> >>>>> So why are they not logging? >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: David Lang <da...@lang.hm> >>>>> Sent: Monday, August 12, 2024 3:51 PM >>>>> To: Drumm, Daniel via rsyslog <rsyslog@lists.adiscon.com> >>>>> Cc: Drumm, Daniel <daniel.dr...@dob.texas.gov> >>>>> Subject: Re: [rsyslog] Formatting CEF to log. >>>>> >>>>> On Mon, 12 Aug 2024, Drumm, Daniel via rsyslog wrote: >>>>> >>>>>> When I perform a "tcpdump -v" I see that the Palo Alto is sending the >>>>>> rsyslog server the correct logs for traffic flows. But the only logs >>>>>> coming from the firewall that are logging are the system messages. I had >>>>>> to convert the traffic flow messages coming in from the Palo Alto on >>>>>> 514/udp to the CEF (Command Event Format) and I do not see them anywhere >>>>>> in logs: >>>>>> >>>>>> Here is how I know they are arriving to the Ubuntu server: >>>>>> >>>>>> >>>>>> >>>>>> root@syslog-server-vnic-primary:/var/log/rsyslog/DOB-FW-HA-1.OCI#<mailto:root@syslog-server-vnic-primary:/var/log/rsyslog/DOB-FW-HA-1.OCI#> >>>>>> tcpdump -v "port 514" >>>>>> >>>>>> tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture >>>>>> size >>>>>> 262144 bytes >>>>>> >>>>>> 20:30:06.780950 IP (tos 0x0, ttl 63, id 25948, offset 0, flags >>>>>> [+], proto UDP (17), length 1500) >>>>>> >>>>>> 10.40.0.210.40933 > 10.30.0.18.syslog: SYSLOG, length: 1472 >>>>>> >>>>>> Facility local0 (16), Severity info (6) >>>>>> >>>>>> Msg: Aug 12 15:30:11 DOB-FW-HA-1.OCI Ashburn FD1 >>>>>> CEF:0|Palo Alto Networks|PAN-OS|11.2.0|drop|TRAFFIC|1|rt=Aug 12 >>>>>> 2024 20:30:08 >>>>>> GMT\0x0adeviceExternalId=007959000479054 src=193.163.125.224 >>>>>> dst=10.40.1.98 >>>>>> sourceTranslatedAddress=0.0.0.0\0x0adestinationTranslatedAddress=0.0. >>>>>> 0 >>>>>> .0 cs1Label=Rule cs1=Inbound Drop Logging Rule suser= duser= >>>>>> app=not-applicable\0x0acs3Label=Virtual System cs3=vsys1 >>>>>> cs4Label=Source Zone cs4=Untrust cs5Label=Destination Zone >>>>>> cs5=Trust\0x0adeviceInboundInterface=ethernet1/2 >>>>>> deviceOutboundInterface= cs6Label=LogProfile cs6=OCI Syslog >>>>>> Server Profile\0x0acn1Label=SessionID cn1=0 cnt=1 spt=42495 >>>>>> dpt=2650 >>>>>> sourceTranslatedPort=0\0x0adestinationTranslatedPort=0 >>>>>> flexString1Label=Flags flexString1=0x0 proto=tcp >>>>>> act=drop\0x0aflexNumber1Label=Total bytes flexNumber1=58 in=58 >>>>>> out=0 >>>>>> cn2Label=Packets\0x0acn2=1 PanOSPacketsReceived=0 >>>>>> PanOSPacketsSent=1\0x0astart=Aug 12 2024 20:30:08 GMT >>>>>> cn3Label=Elapsed time in seconds cn3=0 cs2Label=URL >>>>>> Category\0x0acs2=any externalId= >>>>> 7 >>>>>> 392297676068828287 reason=policy-deny PanOSDGl1=0\0x0aPanOSDGl2=0 >>>>>> PanOSDGl3=0 PanOSDGl4=0\0x0aPanOSVsysName= dvchost=DOB-FW-HA-1 >>>>>> cat=from-policy PanOSActionFlags=0x0\0x0aPanOSSrcUUID= >>>>>> PanOSDstUUID= >>>>>> PanOSTunnelID=0 PanOSMonitorTag=\0x0aPanOSParentSessionID=0 >>>>>> PanOSParentStartTime= PanOSTunnelType=N/A\0x0aPanOSSCTPAssocID=0 >>>>>> PanOSSCTPChunks=0 PanOSSCTPChunkSent=0\0x0aPanOSSCTPChunksRcv=0 >>>>>> PanOSRuleUUID=d6ca6ff0-71e9-4f09-8e8d-2204deb98205 >>>>>> PanOSHTTP2Con=0\0x0aPanLinkChange=0 PanPolicyID= >>>>>> PanLinkDetail=\0x0aPanSDWANCluster= >>>>>> PanSDWANDevice=\0x0aPanSDWANClustyp >>>>>> >>>>>> 20:30:06.780979 IP (tos 0x0, ttl 63, id 25949, offset 0, flags >>>>>> [+], proto UDP (17), length 1500) >>>>>> >>>>>> 10.40.0.210.40933 > 10.30.0.18.syslog: SYSLOG, length: 1472 >>>>>> >>>>>> Here is my rsyslog.conf file: >>>>>> >>>>>> root@syslog-server-vnic-primary:/etc# more rsyslog.conf >>>>>> >>>>>> # /etc/rsyslog.conf configuration file for rsyslog >>>>>> >>>>>> # >>>>>> >>>>>> # For more information install rsyslog-doc and see >>>>>> >>>>>> # /usr/share/doc/rsyslog-doc/html/configuration/index.html >>>>>> >>>>>> # >>>>>> >>>>>> # Default logging rules can be found in >>>>>> /etc/rsyslog.d/50-default.conf >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ################# >>>>>> >>>>>> #### MODULES #### >>>>>> >>>>>> ################# >>>>>> >>>>>> module(load="imudp") # needs to be done just once >>>>>> >>>>>> input(type="imudp" port="514") >>>>>> >>>>>> >>>>>> >>>>>> $ModLoad imuxsock # needs to be done just once >>>>>> >>>>>> >>>>>> >>>>>> # $AllowedSender - specifies which remote systems are allowed to >>>>>> send syslog messages to rsyslogd >>>>>> >>>>>> # -------------- >>>>>> >>>>>> $AllowedSender UDP, 127.0.0.1, 10.40.0.0/16, 10.30.0.0/23 >>>>> >>>>> this is why the legacy format is depriciated, you should not mix >>>>> new style >>>>> input() module() with old style $foo directives >>>>> >>>>>> >>>>>> >>>>>> $SystemLogSocketFlowControl on # enable flow control (use if >>>>>> needed) >>>>>> >>>>>> $ActionFileEnableSync off >>>>>> >>>>>> >>>>>> >>>>>> # A template that resambles traditional syslogd file output: >>>>>> >>>>>> $template TraditionalFormat,"%timegenerated% %HOSTNAME% >>>>>> %syslogtag%%msg:::drop-last-lf%\n" >>>>>> >>>>>> # A more verbose template: >>>>>> >>>>>> $template >>>>>> precise,"%syslogpriority%,%syslogfacility%,%timegenerated::fulltime%,%HOSTNAME%,%syslogtag%,%msg%\n" >>>>>> >>>>>> # A template that resembles RFC 3164 on-the-wire format: >>>>>> >>>>>> # (yes, there is NO space betwen syslogtag and msg! that's >>>>>> important!) >>>>>> >>>>>> $template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%" >>>>>> >>>>>> # a template resembling traditional wallmessage format: >>>>>> >>>>>> $template wallmsg,"\r\n\7Message from syslogd@%HOSTNAME% at >>>>>> %timegenerated% ...\r\n %syslogtag%%msg%\n\r" >>>>>> >>>>>> # The template below emulates winsyslog format, but we need to >>>>>> check the time >>>>>> >>>>>> # stamps used. It is also a good sampleof the property replacer in >>>>>> action. >>>>>> >>>>>> $template >>>>>> WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,%timeg >>>>>> e >>>>>> n >>>>>> e >>>>>> r >>>>>> a >>>>>> ted:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,%timege >>>>>> n e r a t ed:12:19:date-rfc3339%,%syslogfacility%,%syslo >>>>>> >>>>>> gpriority%,%syslogtag%%msg%\n" >>>>>> >>>>>> # A template used for database writing (notice it *is* an actual >>>>>> >>>>>> # sql-statement): >>>>>> >>>>>> $template dbFormat,"insert into SystemEvents (Message, >>>>>> Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, >>>>>> InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, >>>>>> '%HOSTNAME%',%syslogpr >>>>>> >>>>>> iority%, '%timereported:::date-mysql%', >>>>>> '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",sql >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> # Turn off message reception via local log socket; >>>>>> >>>>>> # local messages are retrieved through imjournal now. >>>>>> >>>>>> # $OmitLocalLogging on >>>>>> >>>>>> >>>>>> >>>>>> ########################### >>>>>> >>>>>> #### GLOBAL DIRECTIVES #### >>>>>> >>>>>> ########################### >>>>>> >>>>>> $DebugLevel 2 >>>>>> >>>>>> $DebugFile /var/log/rsyslog/rsyslog.debug >>>>>> >>>>>> # >>>>>> >>>>>> # Use traditional timestamp format. >>>>>> >>>>>> # To enable high precision timestamps, comment out the following line. >>>>>> >>>>>> # >>>>>> >>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >>>>>> >>>>>> >>>>>> >>>>>> # Filter duplicated messages >>>>>> >>>>>> $RepeatedMsgReduction on >>>>>> >>>>>> >>>>>> >>>>>> # >>>>>> >>>>>> # Set the default permissions for all log files. >>>>>> >>>>>> # >>>>>> >>>>>> $FileOwner syslog >>>>>> >>>>>> $FileGroup syslog >>>>>> >>>>>> $FileCreateMode 0660 >>>>>> >>>>>> $DirCreateMode 0755 >>>>>> >>>>>> $Umask 0022 >>>>>> >>>>>> $PrivDropToUser syslog >>>>>> >>>>>> $PrivDropToGroup syslog >>>>>> >>>>>> >>>>>> >>>>>> # >>>>>> >>>>>> # Where to place spool and state files >>>>>> >>>>>> # >>>>>> >>>>>> $WorkDirectory /var/log/rsyslog >>>>>> >>>>>> >>>>>> >>>>>> # Include all config files in /etc/rsyslog.d/ >>>>>> >>>>>> $IncludeConfig /etc/rsyslog.d/ # whole directory (must >>>>>> contain the final slash) >>>>>> >>>>>> >>>>>> >>>>>> Here is my rsyslog.conf file for Palo Alto directive: >>>>>> >>>>>> root@syslog-server-vnic-primary:/etc/rsyslog.d#<mailto:root@syslo >>>>>> g >>>>>> - s e r ver-vnic-primary:/etc/rsyslog.d#> more >>>>>> 10-paloalto-cef.conf >>>>>> >>>>>> # Define a template for CEF logs >>>>>> >>>>>> $template >>>>>> PaloAltoCEF,"/var/log/rsyslog/DOB-FW-HA-1.OCI/%programname%.log" >>>>>> >>>>>> >>>>>> >>>>>> # Use the template for logs coming from your Palo Alto firewall >>>>>> >>>>>> if $fromhost-ip == '10.40.0.210' then { >>>>>> >>>>>> *.* ?PaloAltoCEF >>>>>> >>>>>> stop >>>>>> >>>>>> } >>>>>> >>>>>> Here is my rsyslog.conf file for Catch All directive: >>>>>> >>>>>> root@syslog-server-vnic-primary:/etc/rsyslog.d#<mailto:root@syslo >>>>>> g >>>>>> - s e r ver-vnic-primary:/etc/rsyslog.d#> more 50-default.conf >>>>>> >>>>>> # Default rules for rsyslog. >>>>>> >>>>>> # >>>>>> >>>>>> # For more information see rsyslog.conf(5) and >>>>>> /etc/rsyslog.conf >>>>>> >>>>>> >>>>>> >>>>>> # >>>>>> >>>>>> # First some standard log files. Log by facility. >>>>>> >>>>>> # >>>>>> >>>>>> auth,authpriv.* /var/log/rsyslog/auth.log >>>>>> >>>>>> cron.* /var/log/rsyslog/cron.log >>>>>> >>>>>> daemon.* /var/log/rsyslog/daemon.log >>>>>> >>>>>> kern.* /var/log/rsyslog/kern.log >>>>>> >>>>>> lpr.* /var/log/rsyslog/lpr.log >>>>>> >>>>>> mail.* /var/log/rsyslog/mail.log >>>>>> >>>>>> user.* /var/log/rsyslog/user.log >>>>>> >>>>>> local0.info /var/log/rsyslog/local0_info.log >>>>>> >>>>>> #*.* >>>>>> /var/log/rsyslog/traditionalfile.log;TraditionalFormat # log to a >>>>>> file in the traditional format >>>>>> >>>>>> *.* /var/log/rsyslog/all-the-stuff.log >>>>>> >>>>>> >>>>>> >>>>>> # >>>>>> >>>>>> # Logging for the mail system. Split it up so that >>>>>> >>>>>> # it is easy to write scripts to parse these files. >>>>>> >>>>>> # >>>>>> >>>>>> # mail.info -/var/log/rsyslog/mail.info >>>>>> >>>>>> # mail.warn -/var/log/rsyslog/mail.warn >>>>>> >>>>>> # mail.err /var/log/rsyslog/mail.err >>>>>> >>>>>> >>>>>> >>>>>> # >>>>>> >>>>>> # Some "catch-all" log files. >>>>>> >>>>>> # >>>>>> >>>>>> *.=debug;\ >>>>>> >>>>>> auth,authpriv.none;\ >>>>>> >>>>>> news.none;mail.none /var/log/debug >>>>>> >>>>>> *.=info;*.=notice;*.=warn;\ >>>>>> >>>>>> auth,authpriv.none;\ >>>>>> >>>>>> cron,daemon.none;\ >>>>>> >>>>>> mail,news.none /var/log/messages >>>>>> >>>>>> >>>>>> >>>>>> # >>>>>> >>>>>> # Emergencies are sent to everybody logged in. >>>>>> >>>>>> # >>>>>> >>>>>> *.emerg :omusrmsg:* >>>>>> >>>>>> >>>>>> >>>>>> # >>>>>> >>>>>> # I like to have messages displayed on the console, but only on a >>>>>> virtual >>>>>> >>>>>> # console I usually leave idle. >>>>>> >>>>>> # >>>>>> >>>>>> #daemon,mail.*;\ >>>>>> >>>>>> # news.=crit;news.=err;news.=notice;\ >>>>>> >>>>>> # *.=debug;*.=info;\ >>>>>> >>>>>> # *.=notice;*.=warn /dev/tty8 >>>>>> >>>>>> >>>>>> >>>>>> I would think they would go to the all-the-stuff log file, but no. They >>>>>> are not being logged anywhere and I do not know why not since the PAN-OS >>>>>> system logs are being logged. >>>>> >>>>> well, you have anything arriving from that IP address being written to >>>>> /var/log/rsyslog/DOB-FW-HA-1.OCI/%programname%.log and then you throw >>>>> away the log, so it would never get down to the action to write it to the >>>>> all-the-stuff file. >>>>> >>>>> David Lang >>>>> >>>>>> >>>>>> >>>>>> Dan. >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> https://list/ >>>>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C02%7CDaniel. >>>>>> D >>>>>> r >>>>>> u >>>>>> mm%40dob.texas.gov%7Ca5b7d1daeca54918bec408dcbb106d71%7C32a53eeb2 >>>>>> 7 >>>>>> 5 >>>>>> 9 >>>>>> 4 >>>>>> e >>>>>> f4af2858fd990c7d67%7C1%7C0%7C638590926965494771%7CUnknown%7CTWFpb >>>>>> G >>>>>> Z >>>>>> s >>>>>> b >>>>>> 3 >>>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0 >>>>>> % >>>>>> 3 >>>>>> D >>>>>> % >>>>>> 7 >>>>>> C60000%7C%7C%7C&sdata=jbqBlxisDwsVCMwjaT90N%2FKimmaD8t4q2sz482zP% >>>>>> 2 >>>>>> F >>>>>> F >>>>>> I >>>>>> % >>>>>> 3D&reserved=0 >>>>>> http://www/. >>>>>> r%2F&data=05%7C02%7CDaniel.Drumm%40dob.texas.gov%7C41a72e59343143 >>>>>> 4 >>>>>> 8 >>>>>> 8 >>>>>> 7 >>>>>> 5e08dcbb1e8a28%7C32a53eeb27594ef4af2858fd990c7d67%7C1%7C0%7C63859 >>>>>> 0 >>>>>> 9 >>>>>> 8 >>>>>> 7 >>>>>> 056813333%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2l >>>>>> u >>>>>> M >>>>>> z >>>>>> I >>>>>> iLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C40000%7C%7C%7C&sdata=0VDHHDdJX2 >>>>>> r >>>>>> 3 >>>>>> y >>>>>> i >>>>>> XUWqNO%2FD%2BEEbpPFWRRAaPIoTcHQ%2Bw%3D&reserved=0 >>>>>> syslog.com%2Fprofessional-services%2F&data=05%7C02%7CDaniel.Drumm >>>>>> % >>>>>> 4 >>>>>> 0 >>>>>> d >>>>>> o >>>>>> b.texas.gov%7Ca5b7d1daeca54918bec408dcbb106d71%7C32a53eeb27594ef4 >>>>>> a >>>>>> f >>>>>> 2 >>>>>> 8 >>>>>> 5 >>>>>> 8fd990c7d67%7C1%7C0%7C638590926965505555%7CUnknown%7CTWFpbGZsb3d8 >>>>>> e >>>>>> y >>>>>> J >>>>>> W >>>>>> I >>>>>> joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C6 >>>>>> 0 >>>>>> 0 >>>>>> 0 >>>>>> 0 >>>>>> % >>>>>> 7C%7C%7C&sdata=Y1xISoLoG4QqSzyfShnzV1At6WqTfmphvcwX27vYX%2Bo%3D&r >>>>>> e >>>>>> s >>>>>> e >>>>>> r >>>>>> v >>>>>> ed=0 What's up with rsyslog? Follow https://twit/ >>>>>> ter.com%2Frgerhards&data=05%7C02%7CDaniel.Drumm%40dob.texas.gov%7 >>>>>> C >>>>>> a >>>>>> 5 >>>>>> b >>>>>> 7 >>>>>> d1daeca54918bec408dcbb106d71%7C32a53eeb27594ef4af2858fd990c7d67%7 >>>>>> C >>>>>> 1 >>>>>> % >>>>>> 7 >>>>>> C >>>>>> 0%7C638590926965512721%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD >>>>>> A >>>>>> i >>>>>> L >>>>>> C >>>>>> J >>>>>> QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C60000%7C%7C%7C&sdat >>>>>> a >>>>>> = >>>>>> v >>>>>> V >>>>>> v >>>>>> 5i4UP326ZNb2mtr1KEfnc4i9ZP58nSzWV5zmzPiw%3D&reserved=0 >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > _______________________________________________ > rsyslog mailing list > https://list/ > s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C02%7CDaniel.Dru > mm%40dob.texas.gov%7C9980901584484dd612f508dcbbc05920%7C32a53eeb27594e > f4af2858fd990c7d67%7C1%7C0%7C638591682922758541%7CUnknown%7CTWFpbGZsb3 > d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7 > C0%7C%7C%7C&sdata=ssGsy8Pldc7LARAwyds3lRjRRwrIxttV9bh05t8xt%2BU%3D&res > erved=0 > http://www.r/ > syslog.com%2Fprofessional-services%2F&data=05%7C02%7CDaniel.Drumm%40do > b.texas.gov%7C9980901584484dd612f508dcbbc05920%7C32a53eeb27594ef4af285 > 8fd990c7d67%7C1%7C0%7C638591682922767249%7CUnknown%7CTWFpbGZsb3d8eyJWI > joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7 > C%7C&sdata=mG6%2FKd%2B5SFsq5sKIdtMkrld9F7HbGzbtjrhAavSMwVI%3D&reserved > =0 What's up with rsyslog? Follow > https://twit/ > ter.com%2Frgerhards&data=05%7C02%7CDaniel.Drumm%40dob.texas.gov%7C9980 > 901584484dd612f508dcbbc05920%7C32a53eeb27594ef4af2858fd990c7d67%7C1%7C > 0%7C638591682922771853%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ > QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1n90wV% > 2B50gDz5srH%2BUUw2Y6HNxtLxoZy5Mt6R9D%2Bd5s%3D&reserved=0 > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > > _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
