Messages roll into the rsyslog.debug file from SYSTEM without issue and are
processed:
Debug line with all properties:
FROMHOST: '10.40.0.210', fromhost-ip: '10.40.0.210', HOSTNAME:
'DOB-FW-HA-1.OCI', PRI: 131,
syslogtag 'Ashburn', programname: 'Ashburn', APP-NAME: 'Ashburn', PROCID: '-',
MSGID: '-',
TIMESTAMP: 'Aug 12 18:23:59', STRUCTURED-DATA: '-',
msg: ' FD1 CEF:0|Palo Alto#012Networks|PAN-OS|11.2.0|userid|SYSTEM|4|rt=Aug 12
2024 23:23:58 GMT#012deviceExternalId=007959000479054 cs3Label=Virtual System
cs3= fname= flexString2Label=Module#012flexString2=general msg="gRPC connection
to identity.services-edge.paloaltonetworks.com:443 is broken, error: rpc error:
code = Unknown desc = [UploadCUID] [007959000479054] tenantId is empty time:
2024-08-12 18:23:59" externalId=7392297676068400080 cat=cuid-conn
PanOSDGl1=0#012PanOSDGl2=0 PanOSDGl3=0 PanOSDGl4=0#012PanOSVsysName=
dvchost=DOB-FW-HA-1
PanOSActionFlags=0x0#012anOSTimeGeneratedHighResolution=2024-08-12T18:23:59.046-05:00'
escaped msg: ' FD1 CEF:0|Palo
Alto#012Networks|PAN-OS|11.2.0|userid|SYSTEM|4|rt=Aug 12 2024 23:23:58
GMT#012deviceExternalId=007959000479054 cs3Label=Virtual System cs3= fname=
flexString2Label=Module#012flexString2=general msg="gRPC connection to
identity.services-edge.paloaltonetworks.com:443 is broken, error: rpc error:
code = Unknown desc = [UploadCUID] [007959000479054] tenantId is empty time:
2024-08-12 18:23:59" externalId=7392297676068400080 cat=cuid-conn
PanOSDGl1=0#012PanOSDGl2=0 PanOSDGl3=0 PanOSDGl4=0#012PanOSVsysName=
dvchost=DOB-FW-HA-1
PanOSActionFlags=0x0#012anOSTimeGeneratedHighResolution=2024-08-12T18:23:59.046-05:00'
inputname: imudp rawmsg: '<131>Aug 12 18:23:59 DOB-FW-HA-1.OCI Ashburn FD1
CEF:0|Palo Alto#012Networks|PAN-OS|11.2.0|userid|SYSTEM|4|rt=Aug 12 2024
23:23:58 GMT#012deviceExternalId=007959000479054 cs3Label=Virtual System cs3=
fname= flexString2Label=Module#012flexString2=general msg="gRPC connection to
identity.services-edge.paloaltonetworks.com:443 is broken, error: rpc error:
code = Unknown desc = [UploadCUID] [007959000479054] tenantId is empty time:
2024-08-12 18:23:59" externalId=7392297676068400080 cat=cuid-conn
PanOSDGl1=0#012PanOSDGl2=0 PanOSDGl3=0 PanOSDGl4=0#012PanOSVsysName=
dvchost=DOB-FW-HA-1
PanOSActionFlags=0x0#012anOSTimeGeneratedHighResolution=2024-08-12T18:23:59.046-05:00'
$!:
$.:
$/:
TRAFFIC messages are not processed despite hitting the VNIC.
-----Original Message-----
From: David Lang <da...@lang.hm>
Sent: Monday, August 12, 2024 6:15 PM
To: Drumm, Daniel <daniel.dr...@dob.texas.gov>
Cc: David Lang <da...@lang.hm>; Drumm, Daniel via rsyslog
<rsyslog@lists.adiscon.com>
Subject: RE: [rsyslog] Formatting CEF to log.
try logging all traffic with the template RSYSLOG_DebugFormat for a short time
(long enough that it should have some of these TRAFFIC messages) and find the
messages in there it could be that the messages is not showing up as you would
expect
David Lang
On Mon, 12 Aug 2024, Drumm, Daniel wrote:
> Date: Mon, 12 Aug 2024 23:11:15 +0000
> From: "Drumm, Daniel" <daniel.dr...@dob.texas.gov>
> To: David Lang <da...@lang.hm>
> Cc: "Drumm, Daniel via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] Formatting CEF to log.
>
> I removed the 'stop' from every single directive, as this is confusing the
> matter.
>
> It looks as such, and yet the TRAFFIC messages do not log anywhere.
>
> --
>
> root@syslog-server-vnic-primary:/etc/rsyslog.d# more
> 11-paloalto-cef.conf # Use the template for logs coming from your Palo
> Alto firewall if $fromhost-ip == '10.40.0.210' then {
> *.* /var/log/rsyslog/palo-alto-cef.log
> }
> root@syslog-server-vnic-primary:/etc/rsyslog.d# more
> 10-paloalto-traffic.conf # Define a template for Palo Alto TRAFFIC
> logs $template PaloAltoTraffic,"/var/log/rsyslog/paloalto/traffic.log"
>
> # Filter and direct TRAFFIC logs to the specified file if $fromhost-ip
> == '10.40.0.210' and $msg contains 'TRAFFIC' then {
> action(type="omfile" dynaFile="PaloAltoTraffic") }
> root@syslog-server-vnic-primary:/etc/rsyslog.d# more 10-
> 10-azuremonitoragent-omfwd.conf 10-paloalto-traffic.conf
> root@syslog-server-vnic-primary:/etc/rsyslog.d# more
> 10-azuremonitoragent-omfwd.conf # Azure Monitor Agent configuration:
> forward logs to azuremonitoragent
>
> template(name="AMA_RSYSLOG_TraditionalForwardFormat" type="string"
> string="<%PRI%>%TIMESTAMP% %HOSTNAME%
> %syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
> # queue.workerThreads sets the maximum worker threads, it will scale
> back to 0 if there is no activity # Forwarding all events through TCP
> port
> *.* action(type="omfwd"
> template="AMA_RSYSLOG_TraditionalForwardFormat"
> queue.type="LinkedList"
> queue.filename="omfwd-azuremonitoragent"
> queue.maxFileSize="32m"
> action.resumeRetryCount="-1"
> action.resumeInterval="5"
> action.reportSuspension="on"
> action.reportSuspensionContinuation="on"
> queue.size="25000"
> queue.workerThreads="100"
> queue.dequeueBatchSize="2048"
> queue.saveonshutdown="on"
> target="127.0.0.1" Port="28330" Protocol="tcp")
>
> -----Original Message-----
> From: David Lang <da...@lang.hm>
> Sent: Monday, August 12, 2024 5:58 PM
> To: Drumm, Daniel <daniel.dr...@dob.texas.gov>
> Cc: David Lang <da...@lang.hm>; Drumm, Daniel via rsyslog
> <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] Formatting CEF to log.
>
> On Mon, 12 Aug 2024, Drumm, Daniel wrote:
>
>> Date: Mon, 12 Aug 2024 22:41:21 +0000
>> From: "Drumm, Daniel" <daniel.dr...@dob.texas.gov>
>> To: David Lang <da...@lang.hm>
>> Cc: "Drumm, Daniel via rsyslog" <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] Formatting CEF to log.
>>
>> Here is that file with the -o flag:
>>
>> root@syslog-server-vnic-primary:/tmp# more rsyslog_combined.conf ##
>> full conf created by rsyslog version 8.2001.0 at 2024-08-12 22:38:01
>> ##
>>
>> ##### BEGIN CONFIG: /etc/rsyslog.conf # /etc/rsyslog.conf
>> configuration file for rsyslog # # For more information install
>> rsyslog-doc and see #
>> /usr/share/doc/rsyslog-doc/html/configuration/index.html
>> #
>> # Default logging rules can be found in
>> /etc/rsyslog.d/50-default.conf
>>
>>
>> #################
>> #### MODULES ####
>> #################
>> module(load="imudp") # needs to be done just once input(type="imudp"
>> port="514")
>>
>> $ModLoad imuxsock # needs to be done just once
>>
>> # $AllowedSender - specifies which remote systems are allowed to send
>> syslog messages to rsyslogd # -------------- $AllowedSender UDP,
>> 127.0.0.1, 10.40.0.0/16, 10.30.0.0/23
>>
>> $SystemLogSocketFlowControl on # enable flow control (use if needed)
>> $ActionFileEnableSync off
>>
>> # A template that resambles traditional syslogd file output:
>> $template TraditionalFormat,"%timegenerated% %HOSTNAME%
>> %syslogtag%%msg:::drop-last-lf%\n"
>> # A more verbose template:
>> $template
>> precise,"%syslogpriority%,%syslogfacility%,%timegenerated::fulltime%,%HOSTNAME%,%syslogtag%,%msg%\n"
>> # A template that resembles RFC 3164 on-the-wire format:
>> # (yes, there is NO space betwen syslogtag and msg! that's
>> important!) $template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME%
>> %syslogtag%%msg%"
>> # a template resembling traditional wallmessage format:
>> $template wallmsg,"\r\n\7Message from syslogd@%HOSTNAME% at %timegenerated%
>> ...\r\n %syslogtag%%msg%\n\r"
>> # The template below emulates winsyslog format, but we need to check
>> the time # stamps used. It is also a good sampleof the property replacer in
>> action.
>> $template
>> WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,%timegener
>> a
>> ted:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,%timegenera
>> t ed:12:19:date-rfc3339%,%syslogfacility%,%syslo
>> gpriority%,%syslogtag%%msg%\n"
>> # A template used for database writing (notice it *is* an actual #
>> sql-statement):
>> $template dbFormat,"insert into SystemEvents (Message,
>> Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt,
>> InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
>> '%HOSTNAME%',%syslogpr iority%, '%timereported:::date-mysql%',
>> '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",sql
>>
>>
>> # Turn off message reception via local log socket; # local messages
>> are retrieved through imjournal now.
>> # $OmitLocalLogging on
>>
>> ###########################
>> #### GLOBAL DIRECTIVES ####
>> ###########################
>> $DebugLevel 2
>> $DebugFile /var/log/rsyslog/rsyslog.debug # # Use traditional
>> timestamp format.
>> # To enable high precision timestamps, comment out the following line.
>> #
>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>
>> # Filter duplicated messages
>> $RepeatedMsgReduction on
>>
>> #
>> # Set the default permissions for all log files.
>> #
>> $FileOwner syslog
>> $FileGroup syslog
>> $FileCreateMode 0660
>> $DirCreateMode 0755
>> $Umask 0022
>> $PrivDropToUser syslog
>> $PrivDropToGroup syslog
>>
>> #
>> # Where to place spool and state files # $WorkDirectory
>> /var/log/rsyslog
>>
>> # Include all config files in /etc/rsyslog.d/
>> $IncludeConfig /etc/rsyslog.d/ # whole directory (must
>> contain the final slash)/etc/rsyslog.d/
>> ##### BEGIN CONFIG: /etc/rsyslog.d/50-default.conf
>>
>> ##### BEGIN CONFIG: /etc/rsyslog.d/21-cloudinit.conf
>>
>> ##### BEGIN CONFIG: /etc/rsyslog.d/20-ufw.conf
>>
>> ##### BEGIN CONFIG: /etc/rsyslog.d/12-azuremonitoragent-omfwd.conf
>>
>> ##### BEGIN CONFIG: /etc/rsyslog.d/11-paloalto-cef.conf
>>
>> ##### BEGIN CONFIG: /etc/rsyslog.d/10-paloalto-traffic.conf
>>
>> ##### BEGIN CONFIG: /etc/rsyslog.d/10-azuremonitoragent-omfwd.conf
>> # Azure Monitor Agent configuration: forward logs to
>> azuremonitoragent
>>
>> template(name="AMA_RSYSLOG_TraditionalForwardFormat" type="string"
>> string="<%PRI%>%TIMESTAMP% %HOSTNAME%
>> %syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
>> # queue.workerThreads sets the maximum worker threads, it will scale
>> back to 0 if there is no activity # Forwarding all events through TCP
>> port
>> *.* action(type="omfwd"
>> template="AMA_RSYSLOG_TraditionalForwardFormat"
>> queue.type="LinkedList"
>> queue.filename="omfwd-azuremonitoragent"
>> queue.maxFileSize="32m"
>> action.resumeRetryCount="-1"
>> action.resumeInterval="5"
>> action.reportSuspension="on"
>> action.reportSuspensionContinuation="on"
>> queue.size="25000"
>> queue.workerThreads="100"
>> queue.dequeueBatchSize="2048"
>> queue.saveonshutdown="on"
>> target="127.0.0.1" Port="28330" Protocol="tcp")
>>
>> ##### END CONFIG: /etc/rsyslog.d/10-azuremonitoragent-omfwd.conf
>> # Define a template for Palo Alto TRAFFIC logs $template
>> PaloAltoTraffic,"/var/log/rsyslog/paloalto/traffic.log"
>>
>> # Filter and direct TRAFFIC logs to the specified file if
>> $fromhost-ip == '10.40.0.210' and $msg contains 'TRAFFIC' then {
>> action(type="omfile" dynaFile="PaloAltoTraffic")
>> stop
>> }
>
> because you have stop here, your TRAFFIC logs will not be processed
> past this point. That is why they don't appear in the
> palo-alto-cef.log file below
>
>> ##### END CONFIG: /etc/rsyslog.d/10-paloalto-traffic.conf
>> # Define a template for CEF logs
>> $template PaloAltoCEF,"/var/log/rsyslog/e%.log"
>>
>> # Use the template for logs coming from your Palo Alto firewall if
>> $fromhost-ip == '10.40.0.210' then {
>> *.* /var/log/rsyslog/palo-alto-cef.log
>> stop
>> }
>>
>> ##### END CONFIG: /etc/rsyslog.d/11-paloalto-cef.conf
>> # Azure Monitor Agent configuration: forward logs to
>> azuremonitoragent
>>
>> # Azure Monitor Agent configuration: forward logs to
>> azuremonitoragent template(name="AMA_RSYSLOG_TraditionalForwardFormat"
>> type="string"
>> string="<%PRI%>%TIMESTAMP% %HOSTNAME%
>> %syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
>>
>> *.* action(type="omfwd"
>> template="AMA_RSYSLOG_TraditionalForwardFormat"
>> queue.type="LinkedList"
>> queue.filename="omfwd-azuremonitoragent"
>> queue.maxFileSize="32m"
>> action.resumeRetryCount="-1"
>> action.resumeInterval="30"
>> action.reportSuspension="on"
>> action.reportSuspensionContinuation="on"
>> queue.size="100000"
>> queue.discardmark="97500"
>> queue.discardseverity="0"
>> queue.checkpointInterval="100"
>> queue.workerThreads="4"
>> queue.timeoutEnqueue="10000"
>> queue.timeoutWorkerthreadShutdown="60000"
>> queue.saveOnShutdown="on"
>> target="127.0.0.1" Port="28330" Protocol="tcp"
>> )
>>
>> ##### END CONFIG: /etc/rsyslog.d/12-azuremonitoragent-omfwd.conf
>> # Log kernel generated UFW log messages to file :msg,contains,"[UFW "
>> /var/log/ufw.log
>>
>> # Uncomment the following to stop logging anything that matches the last
>> rule.
>> # Doing this will stop logging kernel generated UFW log messages to
>> the file # normally containing kern.* messages (eg,
>> /var/log/kern.log) #& stop
>>
>> ##### END CONFIG: /etc/rsyslog.d/20-ufw.conf
>> # Log cloudinit generated log messages to file :syslogtag, isequal,
>> "[CLOUDINIT]" /var/log/cloud-init.log
>>
>> # comment out the following line to allow CLOUDINIT messages through.
>> # Doing so means you'll also get CLOUDINIT messages in
>> /var/log/syslog & stop
>>
>> ##### END CONFIG: /etc/rsyslog.d/21-cloudinit.conf
>> # Default rules for rsyslog.
>> #
>> # For more information see rsyslog.conf(5) and
>> /etc/rsyslog.conf
>>
>> #
>> # First some standard log files. Log by facility.
>> #
>> auth,authpriv.* /var/log/rsyslog/auth.log
>> cron.* /var/log/rsyslog/cron.log
>> daemon.* /var/log/rsyslog/daemon.log
>> kern.* /var/log/rsyslog/kern.log
>> lpr.* /var/log/rsyslog/lpr.log
>> mail.* /var/log/rsyslog/mail.log
>> user.* /var/log/rsyslog/user.log
>> local0.info /var/log/rsyslog/local0_info.log
>> #*.*
>> /var/log/rsyslog/traditionalfile.log;TraditionalFormat # log to a file
>> in the traditional format
>> *.* /var/log/rsyslog/all-the-stuff.log
>>
>> #
>> # Logging for the mail system. Split it up so that # it is easy to
>> write scripts to parse these files.
>> #
>> # mail.info -/var/log/rsyslog/mail.info
>> # mail.warn -/var/log/rsyslog/mail.warn
>> # mail.err /var/log/rsyslog/mail.err
>>
>> #
>> # Some "catch-all" log files.
>> #
>> *.=debug;auth,authpriv.none;news.none;mail.none /var/log/debug
>> PreprocFileLineNumber(33)
>> *.=info;*.=notice;*.=warn;auth,authpriv.none;cron,daemon.none;mail,news.none
>> /var/log/messages
>> PreprocFileLineNumber(37)
>>
>> #
>> # Emergencies are sent to everybody logged in.
>> #
>> *.emerg :omusrmsg:*
>>
>> #
>> # I like to have messages displayed on the console, but only on a
>> virtual # console I usually leave idle.
>> #
>> #daemon,mail.*;# news.=crit;news.=err;news.=notice;#
>> *.=debug;*.=info;# *.=notice;*.=warn /dev/tty8
>>
>> ##### END CONFIG: /etc/rsyslog.d/50-default.conf
>> # whole directory (must contain the final slash)
>>
>> ##### END CONFIG: /etc/rsyslog.conf
>>
>> -----Original Message-----
>> From: David Lang <da...@lang.hm>
>> Sent: Monday, August 12, 2024 5:32 PM
>> To: Drumm, Daniel <daniel.dr...@dob.texas.gov>
>> Cc: David Lang <da...@lang.hm>; Drumm, Daniel via rsyslog
>> <rsyslog@lists.adiscon.com>
>> Subject: RE: [rsyslog] Formatting CEF to log.
>>
>> start rsyslog with the command line option -o /path/to/file
>>
>> that will write the combined config files to that file as rsyslog
>> sees it. Post that and we can look for other interactions that may
>> cause some logs to be thrown away first
>>
>> look especially for othe stop commands earlier than the clause
>> looking at the fromhost-ip
>>
>> David Lang
>>
>> On Mon, 12 Aug 2024, Drumm, Daniel wrote:
>>
>>> Date: Mon, 12 Aug 2024 22:18:02 +0000
>>> From: "Drumm, Daniel" <daniel.dr...@dob.texas.gov>
>>> To: David Lang <da...@lang.hm>,
>>> "Drumm, Daniel via rsyslog" <rsyslog@lists.adiscon.com>
>>> Subject: RE: [rsyslog] Formatting CEF to log.
>>>
>>> Thank you, I should have corrected that as I iterated.
>>>
>>> What I see with this directive:
>>>
>>> root@syslog-server-vnic-primary:/etc/rsyslog.d# more
>>> 10-paloalto-cef.conf # Define a template for CEF logs $template
>>> PaloAltoCEF,"/var/log/rsyslog/DOB-FW-HA-1.OCI/%programname%.log"
>>>
>>> # Use the template for logs coming from your Palo Alto firewall if
>>> $fromhost-ip == '10.40.0.210' then {
>>> *.* /var/log/rsyslog/palo-alto-cef.log
>>> stop
>>> }
>>>
>>> Is that the log has SYSTEM messages, but none of the Traffic messages:
>>>
>>> root@syslog-server-vnic-primary:/var/log/rsyslog# tail -f
>>> palo-alto-cef.log Aug 12 17:07:12 DOB-FW-HA-1.OCI Ashburn FD1
>>> CEF:0|Palo
>>> Alto#012Networks|PAN-OS|11.2.0|url-filtering|SYSTEM|1|rt=Aug 12 2024
>>> 22:07:12 GMT#012deviceExternalId=007959000479054 cs3Label=Virtual
>>> System cs3= fname= flexString2Label=Module#012flexString2=general
>>> msg="PAN-DB was upgraded to version 20240812.20332."
>>> externalId=7392297676068399997 cat=upgrade-url-database-success
>>> PanOSDGl1=0#012PanOSDGl2=0 PanOSDGl3=0 PanOSDGl4=0#012PanOSVsysName=
>>> dvchost=DOB-FW-HA-1
>>> PanOSActionFlags=0x0#012anOSTimeGeneratedHighResolution=2024-08-12T17:
>>> 07:12.563-05:00 Aug 12 17:08:17 DOB-FW-HA-1.OCI Ashburn FD1
>>> CEF:0|Palo
>>> Alto#012Networks|PAN-OS|11.2.0|url-filtering|SYSTEM|1|rt=Aug 12 2024
>>> 22:08:17 GMT#012deviceExternalId=007959000479054 cs3Label=Virtual
>>> System cs3= fname= flexString2Label=Module#012flexString2=general
>>> msg="PAN-DB was upgraded to version 20240812.20333."
>>> externalId=7392297676068399998 cat=upgrade-url-database-success
>>> PanOSDGl1=0#012PanOSDGl2=0 PanOSDGl3=0 PanOSDGl4=0#012PanOSVsysName=
>>> dvchost=DOB-FW-HA-1
>>> PanOSActionFlags=0x0#012anOSTimeGeneratedHighResolution=2024-08-12T17:
>>> 08:17.737-05:00 Aug 12 17:08:50 DOB-FW-HA-1.OCI Ashburn FD1
>>> CEF:0|Palo Alto#012Networks|PAN-OS|11.2.0|userid|SYSTEM|1|rt=Aug 12
>>> 2024 22:08:49
>>> GMT#012deviceExternalId=007959000479054 cs3Label=Virtual System cs3=
>>> fname= flexString2Label=Module#012flexString2=general msg="gRPC
>>> connection to identity.services-edge.paloaltonetworks.com:443 is
>>> established, 10.40.0.210:46846 -> 34.136.155.117:443 time:
>>> 2024-08-12 17:08:50" externalId=7392297676068399999 cat=cuid-conn
>>> PanOSDGl1=0#012PanOSDGl2=0 PanOSDGl3=0 PanOSDGl4=0#012PanOSVsysName=
>>> dvchost=DOB-FW-HA-1
>>> PanOSActionFlags=0x0#012anOSTimeGeneratedHighResolution=2024-08-12T17:
>>> 08:50.292-05:00 Aug 12 17:08:54 DOB-FW-HA-1.OCI Ashburn FD1
>>> CEF:0|Palo Alto#012Networks|PAN-OS|11.2.0|userid|SYSTEM|4|rt=Aug 12
>>> 2024 22:08:53
>>> GMT#012deviceExternalId=007959000479054 cs3Label=Virtual System cs3=
>>> fname= flexString2Label=Module#012flexString2=general msg="gRPC
>>> connection to identity.services-edge.paloaltonetworks.com:443 is
>>> broken, error: rpc error: code = Unknown desc = [UploadCUID]
>>> [007959000479054] tenantId is empty time: 2024-08-12 17:08:54"
>>> externalId=7392297676068400000 cat=cuid-conn
>>> PanOSDGl1=0#012PanOSDGl2=0 PanOSDGl3=0 PanOSDGl4=0#012PanOSVsysName=
>>> dvchost=DOB-FW-HA-1
>>> PanOSActionFlags=0x0#012anOSTimeGeneratedHighResolution=2024-08-12T17:
>>> 08:54.050-05:00
>>>
>>> Yet I know for a fact that the TRAFFIC messages are hitting the NIC of the
>>> syslog server:
>>>
>>> root@syslog-server-vnic-primary:/etc/rsyslog.d# tcpdump -v "port 514"
>>> tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture
>>> size
>>> 262144 bytes
>>> 22:16:32.237523 IP (tos 0x0, ttl 63, id 30088, offset 0, flags [+], proto
>>> UDP (17), length 1500)
>>> 10.40.0.210.40933 > 10.30.0.18.syslog: SYSLOG, length: 1472
>>> Facility local0 (16), Severity info (6)
>>> Msg: Aug 12 17:16:36 DOB-FW-HA-1.OCI Ashburn FD1 CEF:0|Palo
>>> Alto Networks|PAN-OS|11.2.0|drop|TRAFFIC|1|rt=Aug 12 2024 22:16:31
>>> GMT\0x0adeviceExternalId=007959000479054 src=162.216.149.73
>>> dst=10.40.1.97
>>> sourceTranslatedAddress=0.0.0.0\0x0adestinationTranslatedAddress=0.0.
>>> 0
>>> .0 cs1Label=Rule cs1=Inbound Drop Logging Rule suser= duser=
>>> app=not-applicable\0x0acs3Label=Virtual System cs3=vsys1
>>> cs4Label=Source Zone cs4=Untrust cs5Label=Destination Zone
>>> cs5=Trust\0x0adeviceInboundInterface=ethernet1/2
>>> deviceOutboundInterface= cs6Label=LogProfile cs6=OCI Syslog Server
>>> Profile\0x0acn1Label=SessionID cn1=0 cnt=1 spt=54491 dpt=9408
>>> sourceTranslatedPort=0\0x0adestinationTranslatedPort=0
>>> flexString1Label=Flags flexString1=0x0 proto=tcp
>>> act=drop\0x0aflexNumber1Label=Total bytes flexNumber1=58 in=58 out=0
>>> cn2Label=Packets\0x0acn2=1 PanOSPacketsReceived=0
>>> PanOSPacketsSent=1\0x0astart=Aug 12 2024 22:16:31 GMT
>>> cn3Label=Elapsed time in seconds cn3=0 cs2Label=URL
>>> Category\0x0acs2=any externalId=7
>> 392297676068829896 reason=policy-deny PanOSDGl1=0\0x0aPanOSDGl2=0
>> PanOSDGl3=0 PanOSDGl4=0\0x0aPanOSVsysName= dvchost=DOB-FW-HA-1
>> cat=from-policy PanOSActionFlags=0x0\0x0aPanOSSrcUUID= PanOSDstUUID=
>> PanOSTunnelID=0 PanOSMonitorTag=\0x0aPanOSParentSessionID=0
>> PanOSParentStartTime= PanOSTunnelType=N/A\0x0aPanOSSCTPAssocID=0
>> PanOSSCTPChunks=0 PanOSSCTPChunkSent=0\0x0aPanOSSCTPChunksRcv=0
>> PanOSRuleUUID=d6ca6ff0-71e9-4f09-8e8d-2204deb98205
>> PanOSHTTP2Con=0\0x0aPanLinkChange=0 PanPolicyID=
>> PanLinkDetail=\0x0aPanSDWANCluster=
>> PanSDWANDevice=\0x0aPanSDWANClustype
>>> 22:16:32.237613 IP (tos 0x0, ttl 63, id 30089, offset 0, flags [+],
>>> proto UDP (17), length 1500)
>>>
>>> So why are they not logging?
>>>
>>>
>>> -----Original Message-----
>>> From: David Lang <da...@lang.hm>
>>> Sent: Monday, August 12, 2024 3:51 PM
>>> To: Drumm, Daniel via rsyslog <rsyslog@lists.adiscon.com>
>>> Cc: Drumm, Daniel <daniel.dr...@dob.texas.gov>
>>> Subject: Re: [rsyslog] Formatting CEF to log.
>>>
>>> On Mon, 12 Aug 2024, Drumm, Daniel via rsyslog wrote:
>>>
>>>> When I perform a "tcpdump -v" I see that the Palo Alto is sending the
>>>> rsyslog server the correct logs for traffic flows. But the only logs
>>>> coming from the firewall that are logging are the system messages. I had
>>>> to convert the traffic flow messages coming in from the Palo Alto on
>>>> 514/udp to the CEF (Command Event Format) and I do not see them anywhere
>>>> in logs:
>>>>
>>>> Here is how I know they are arriving to the Ubuntu server:
>>>>
>>>>
>>>>
>>>> root@syslog-server-vnic-primary:/var/log/rsyslog/DOB-FW-HA-1.OCI#<mailto:root@syslog-server-vnic-primary:/var/log/rsyslog/DOB-FW-HA-1.OCI#>
>>>> tcpdump -v "port 514"
>>>>
>>>> tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture
>>>> size
>>>> 262144 bytes
>>>>
>>>> 20:30:06.780950 IP (tos 0x0, ttl 63, id 25948, offset 0, flags [+],
>>>> proto UDP (17), length 1500)
>>>>
>>>> 10.40.0.210.40933 > 10.30.0.18.syslog: SYSLOG, length: 1472
>>>>
>>>> Facility local0 (16), Severity info (6)
>>>>
>>>> Msg: Aug 12 15:30:11 DOB-FW-HA-1.OCI Ashburn FD1 CEF:0|Palo
>>>> Alto Networks|PAN-OS|11.2.0|drop|TRAFFIC|1|rt=Aug 12 2024 20:30:08
>>>> GMT\0x0adeviceExternalId=007959000479054 src=193.163.125.224
>>>> dst=10.40.1.98
>>>> sourceTranslatedAddress=0.0.0.0\0x0adestinationTranslatedAddress=0.0.
>>>> 0
>>>> .0 cs1Label=Rule cs1=Inbound Drop Logging Rule suser= duser=
>>>> app=not-applicable\0x0acs3Label=Virtual System cs3=vsys1
>>>> cs4Label=Source Zone cs4=Untrust cs5Label=Destination Zone
>>>> cs5=Trust\0x0adeviceInboundInterface=ethernet1/2
>>>> deviceOutboundInterface= cs6Label=LogProfile cs6=OCI Syslog Server
>>>> Profile\0x0acn1Label=SessionID cn1=0 cnt=1 spt=42495 dpt=2650
>>>> sourceTranslatedPort=0\0x0adestinationTranslatedPort=0
>>>> flexString1Label=Flags flexString1=0x0 proto=tcp
>>>> act=drop\0x0aflexNumber1Label=Total bytes flexNumber1=58 in=58
>>>> out=0
>>>> cn2Label=Packets\0x0acn2=1 PanOSPacketsReceived=0
>>>> PanOSPacketsSent=1\0x0astart=Aug 12 2024 20:30:08 GMT
>>>> cn3Label=Elapsed time in seconds cn3=0 cs2Label=URL
>>>> Category\0x0acs2=any externalId=
>>> 7
>>>> 392297676068828287 reason=policy-deny PanOSDGl1=0\0x0aPanOSDGl2=0
>>>> PanOSDGl3=0 PanOSDGl4=0\0x0aPanOSVsysName= dvchost=DOB-FW-HA-1
>>>> cat=from-policy PanOSActionFlags=0x0\0x0aPanOSSrcUUID=
>>>> PanOSDstUUID=
>>>> PanOSTunnelID=0 PanOSMonitorTag=\0x0aPanOSParentSessionID=0
>>>> PanOSParentStartTime= PanOSTunnelType=N/A\0x0aPanOSSCTPAssocID=0
>>>> PanOSSCTPChunks=0 PanOSSCTPChunkSent=0\0x0aPanOSSCTPChunksRcv=0
>>>> PanOSRuleUUID=d6ca6ff0-71e9-4f09-8e8d-2204deb98205
>>>> PanOSHTTP2Con=0\0x0aPanLinkChange=0 PanPolicyID=
>>>> PanLinkDetail=\0x0aPanSDWANCluster=
>>>> PanSDWANDevice=\0x0aPanSDWANClustyp
>>>>
>>>> 20:30:06.780979 IP (tos 0x0, ttl 63, id 25949, offset 0, flags [+],
>>>> proto UDP (17), length 1500)
>>>>
>>>> 10.40.0.210.40933 > 10.30.0.18.syslog: SYSLOG, length: 1472
>>>>
>>>> Here is my rsyslog.conf file:
>>>>
>>>> root@syslog-server-vnic-primary:/etc# more rsyslog.conf
>>>>
>>>> # /etc/rsyslog.conf configuration file for rsyslog
>>>>
>>>> #
>>>>
>>>> # For more information install rsyslog-doc and see
>>>>
>>>> # /usr/share/doc/rsyslog-doc/html/configuration/index.html
>>>>
>>>> #
>>>>
>>>> # Default logging rules can be found in
>>>> /etc/rsyslog.d/50-default.conf
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> #################
>>>>
>>>> #### MODULES ####
>>>>
>>>> #################
>>>>
>>>> module(load="imudp") # needs to be done just once
>>>>
>>>> input(type="imudp" port="514")
>>>>
>>>>
>>>>
>>>> $ModLoad imuxsock # needs to be done just once
>>>>
>>>>
>>>>
>>>> # $AllowedSender - specifies which remote systems are allowed to
>>>> send syslog messages to rsyslogd
>>>>
>>>> # --------------
>>>>
>>>> $AllowedSender UDP, 127.0.0.1, 10.40.0.0/16, 10.30.0.0/23
>>>
>>> this is why the legacy format is depriciated, you should not mix new
>>> style
>>> input() module() with old style $foo directives
>>>
>>>>
>>>>
>>>> $SystemLogSocketFlowControl on # enable flow control (use if
>>>> needed)
>>>>
>>>> $ActionFileEnableSync off
>>>>
>>>>
>>>>
>>>> # A template that resambles traditional syslogd file output:
>>>>
>>>> $template TraditionalFormat,"%timegenerated% %HOSTNAME%
>>>> %syslogtag%%msg:::drop-last-lf%\n"
>>>>
>>>> # A more verbose template:
>>>>
>>>> $template
>>>> precise,"%syslogpriority%,%syslogfacility%,%timegenerated::fulltime%,%HOSTNAME%,%syslogtag%,%msg%\n"
>>>>
>>>> # A template that resembles RFC 3164 on-the-wire format:
>>>>
>>>> # (yes, there is NO space betwen syslogtag and msg! that's
>>>> important!)
>>>>
>>>> $template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%"
>>>>
>>>> # a template resembling traditional wallmessage format:
>>>>
>>>> $template wallmsg,"\r\n\7Message from syslogd@%HOSTNAME% at
>>>> %timegenerated% ...\r\n %syslogtag%%msg%\n\r"
>>>>
>>>> # The template below emulates winsyslog format, but we need to
>>>> check the time
>>>>
>>>> # stamps used. It is also a good sampleof the property replacer in action.
>>>>
>>>> $template
>>>> WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,%timegen
>>>> e
>>>> r
>>>> a
>>>> ted:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,%timegene
>>>> r a t ed:12:19:date-rfc3339%,%syslogfacility%,%syslo
>>>>
>>>> gpriority%,%syslogtag%%msg%\n"
>>>>
>>>> # A template used for database writing (notice it *is* an actual
>>>>
>>>> # sql-statement):
>>>>
>>>> $template dbFormat,"insert into SystemEvents (Message,
>>>> Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt,
>>>> InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
>>>> '%HOSTNAME%',%syslogpr
>>>>
>>>> iority%, '%timereported:::date-mysql%',
>>>> '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",sql
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> # Turn off message reception via local log socket;
>>>>
>>>> # local messages are retrieved through imjournal now.
>>>>
>>>> # $OmitLocalLogging on
>>>>
>>>>
>>>>
>>>> ###########################
>>>>
>>>> #### GLOBAL DIRECTIVES ####
>>>>
>>>> ###########################
>>>>
>>>> $DebugLevel 2
>>>>
>>>> $DebugFile /var/log/rsyslog/rsyslog.debug
>>>>
>>>> #
>>>>
>>>> # Use traditional timestamp format.
>>>>
>>>> # To enable high precision timestamps, comment out the following line.
>>>>
>>>> #
>>>>
>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>
>>>>
>>>>
>>>> # Filter duplicated messages
>>>>
>>>> $RepeatedMsgReduction on
>>>>
>>>>
>>>>
>>>> #
>>>>
>>>> # Set the default permissions for all log files.
>>>>
>>>> #
>>>>
>>>> $FileOwner syslog
>>>>
>>>> $FileGroup syslog
>>>>
>>>> $FileCreateMode 0660
>>>>
>>>> $DirCreateMode 0755
>>>>
>>>> $Umask 0022
>>>>
>>>> $PrivDropToUser syslog
>>>>
>>>> $PrivDropToGroup syslog
>>>>
>>>>
>>>>
>>>> #
>>>>
>>>> # Where to place spool and state files
>>>>
>>>> #
>>>>
>>>> $WorkDirectory /var/log/rsyslog
>>>>
>>>>
>>>>
>>>> # Include all config files in /etc/rsyslog.d/
>>>>
>>>> $IncludeConfig /etc/rsyslog.d/ # whole directory (must
>>>> contain the final slash)
>>>>
>>>>
>>>>
>>>> Here is my rsyslog.conf file for Palo Alto directive:
>>>>
>>>> root@syslog-server-vnic-primary:/etc/rsyslog.d#<mailto:root@syslog-
>>>> s e r ver-vnic-primary:/etc/rsyslog.d#> more 10-paloalto-cef.conf
>>>>
>>>> # Define a template for CEF logs
>>>>
>>>> $template PaloAltoCEF,"/var/log/rsyslog/DOB-FW-HA-1.OCI/%programname%.log"
>>>>
>>>>
>>>>
>>>> # Use the template for logs coming from your Palo Alto firewall
>>>>
>>>> if $fromhost-ip == '10.40.0.210' then {
>>>>
>>>> *.* ?PaloAltoCEF
>>>>
>>>> stop
>>>>
>>>> }
>>>>
>>>> Here is my rsyslog.conf file for Catch All directive:
>>>>
>>>> root@syslog-server-vnic-primary:/etc/rsyslog.d#<mailto:root@syslog-
>>>> s e r ver-vnic-primary:/etc/rsyslog.d#> more 50-default.conf
>>>>
>>>> # Default rules for rsyslog.
>>>>
>>>> #
>>>>
>>>> # For more information see rsyslog.conf(5) and
>>>> /etc/rsyslog.conf
>>>>
>>>>
>>>>
>>>> #
>>>>
>>>> # First some standard log files. Log by facility.
>>>>
>>>> #
>>>>
>>>> auth,authpriv.* /var/log/rsyslog/auth.log
>>>>
>>>> cron.* /var/log/rsyslog/cron.log
>>>>
>>>> daemon.* /var/log/rsyslog/daemon.log
>>>>
>>>> kern.* /var/log/rsyslog/kern.log
>>>>
>>>> lpr.* /var/log/rsyslog/lpr.log
>>>>
>>>> mail.* /var/log/rsyslog/mail.log
>>>>
>>>> user.* /var/log/rsyslog/user.log
>>>>
>>>> local0.info /var/log/rsyslog/local0_info.log
>>>>
>>>> #*.*
>>>> /var/log/rsyslog/traditionalfile.log;TraditionalFormat # log to a
>>>> file in the traditional format
>>>>
>>>> *.* /var/log/rsyslog/all-the-stuff.log
>>>>
>>>>
>>>>
>>>> #
>>>>
>>>> # Logging for the mail system. Split it up so that
>>>>
>>>> # it is easy to write scripts to parse these files.
>>>>
>>>> #
>>>>
>>>> # mail.info -/var/log/rsyslog/mail.info
>>>>
>>>> # mail.warn -/var/log/rsyslog/mail.warn
>>>>
>>>> # mail.err /var/log/rsyslog/mail.err
>>>>
>>>>
>>>>
>>>> #
>>>>
>>>> # Some "catch-all" log files.
>>>>
>>>> #
>>>>
>>>> *.=debug;\
>>>>
>>>> auth,authpriv.none;\
>>>>
>>>> news.none;mail.none /var/log/debug
>>>>
>>>> *.=info;*.=notice;*.=warn;\
>>>>
>>>> auth,authpriv.none;\
>>>>
>>>> cron,daemon.none;\
>>>>
>>>> mail,news.none /var/log/messages
>>>>
>>>>
>>>>
>>>> #
>>>>
>>>> # Emergencies are sent to everybody logged in.
>>>>
>>>> #
>>>>
>>>> *.emerg :omusrmsg:*
>>>>
>>>>
>>>>
>>>> #
>>>>
>>>> # I like to have messages displayed on the console, but only on a
>>>> virtual
>>>>
>>>> # console I usually leave idle.
>>>>
>>>> #
>>>>
>>>> #daemon,mail.*;\
>>>>
>>>> # news.=crit;news.=err;news.=notice;\
>>>>
>>>> # *.=debug;*.=info;\
>>>>
>>>> # *.=notice;*.=warn /dev/tty8
>>>>
>>>>
>>>>
>>>> I would think they would go to the all-the-stuff log file, but no. They
>>>> are not being logged anywhere and I do not know why not since the PAN-OS
>>>> system logs are being logged.
>>>
>>> well, you have anything arriving from that IP address being written to
>>> /var/log/rsyslog/DOB-FW-HA-1.OCI/%programname%.log and then you throw away
>>> the log, so it would never get down to the action to write it to the
>>> all-the-stuff file.
>>>
>>> David Lang
>>>
>>>>
>>>>
>>>> Dan.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://list/
>>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C02%7CDaniel.
>>>> D
>>>> r
>>>> u
>>>> mm%40dob.texas.gov%7Ca5b7d1daeca54918bec408dcbb106d71%7C32a53eeb275
>>>> 9
>>>> 4
>>>> e
>>>> f4af2858fd990c7d67%7C1%7C0%7C638590926965494771%7CUnknown%7CTWFpbGZ
>>>> s
>>>> b
>>>> 3
>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
>>>> D
>>>> %
>>>> 7
>>>> C60000%7C%7C%7C&sdata=jbqBlxisDwsVCMwjaT90N%2FKimmaD8t4q2sz482zP%2F
>>>> F
>>>> I
>>>> %
>>>> 3D&reserved=0
>>>> https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww%2F&data=05%7C02%7CDaniel.Drumm%40dob.texas.gov%7C64772cf33d5e413dcb3408dcbb24a633%7C32a53eeb27594ef4af2858fd990c7d67%7C1%7C0%7C638591013326152923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=ke4sm%2FZYh8E0%2FuVk7lFIp%2BarTAHXyK7LvKUOU%2ByNi%2Fc%3D&reserved=0.
>>>> r%2F&data=05%7C02%7CDaniel.Drumm%40dob.texas.gov%7C41a72e5934314348
>>>> 8
>>>> 7
>>>> 5e08dcbb1e8a28%7C32a53eeb27594ef4af2858fd990c7d67%7C1%7C0%7C6385909
>>>> 8
>>>> 7
>>>> 056813333%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM
>>>> z
>>>> I
>>>> iLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C40000%7C%7C%7C&sdata=0VDHHDdJX2r3
>>>> y
>>>> i
>>>> XUWqNO%2FD%2BEEbpPFWRRAaPIoTcHQ%2Bw%3D&reserved=0
>>>> syslog.com%2Fprofessional-services%2F&data=05%7C02%7CDaniel.Drumm%4
>>>> 0
>>>> d
>>>> o
>>>> b.texas.gov%7Ca5b7d1daeca54918bec408dcbb106d71%7C32a53eeb27594ef4af
>>>> 2
>>>> 8
>>>> 5
>>>> 8fd990c7d67%7C1%7C0%7C638590926965505555%7CUnknown%7CTWFpbGZsb3d8ey
>>>> J
>>>> W
>>>> I
>>>> joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C600
>>>> 0
>>>> 0
>>>> %
>>>> 7C%7C%7C&sdata=Y1xISoLoG4QqSzyfShnzV1At6WqTfmphvcwX27vYX%2Bo%3D&res
>>>> e
>>>> r
>>>> v
>>>> ed=0 What's up with rsyslog? Follow https://twit/
>>>> ter.com%2Frgerhards&data=05%7C02%7CDaniel.Drumm%40dob.texas.gov%7Ca
>>>> 5
>>>> b
>>>> 7
>>>> d1daeca54918bec408dcbb106d71%7C32a53eeb27594ef4af2858fd990c7d67%7C1
>>>> %
>>>> 7
>>>> C
>>>> 0%7C638590926965512721%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi
>>>> L
>>>> C
>>>> J
>>>> QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C60000%7C%7C%7C&sdata=
>>>> v
>>>> V
>>>> v
>>>> 5i4UP326ZNb2mtr1KEfnc4i9ZP58nSzWV5zmzPiw%3D&reserved=0
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>>
>>>
>>>
>>
>>
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
