On Thursday, February 9, 2017 at 10:00:49 AM UTC+1, Jochen Schalanda wrote:
>
> Hi,
>
> Graylog itself doesn't care where the packets are coming from.
>
> Is the routing to Graylog working for the "ignored" host?
> Is the networking set up correctly on all hosts?
> Are there any firewall rules in place?
> How did you configure the Syslog UDP and the Raw/Plaintext UDP inputs?
>
> Cheers,
> Jochen
>
> On Wednesday, 8 February 2017 19:43:38 UTC+1, [email protected] wrote:
>>
>> Hello,
>>
>> I've recently set up a working Graylog server. It's collecting logs from
>> many network switches and routers. One particular router (ironically, the
>> most important one) doesn't appear in the Sources list though. Graylog
>> keeps ignoring all packets coming from that host. Here's an example of a
>> packet which is *not* ignored by Graylog:
>>
>> 19:12:15.705167 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
>> UDP (17), length 115)
>> 10.50.255.44.40810 > Silenoz.syslog: [udp sum ok] [|syslog]
>> 0x0000: 4500 0073 0000 4000 4011 27e3 0a32 ff2c E..s..@.@.'..2.,
>> 0x0010: 0a32 ff06 9f6a 0202 005f 01d1 6468 6370 .2...j..._..dhcp
>> 0x0020: 2c77 6172 6e69 6e67 2067 706f 6e2d 6d6e ,warning.gpon-mn
>> 0x0030: 6720 6f66 6665 7269 6e67 206c 6561 7365 g.offering.lease
>> 0x0040: 2031 302e 3530 2e32 3338 2e33 3520 666f .10.50.238.35.fo
>> 0x0050: 7220 3030 3a30 323a 3731 3a35 413a 3036 r.00:02:71:5A:06
>> 0x0060: 3a42 3820 7769 7468 6f75 7420 7375 6363 :B8.without.succ
>> 0x0070: 6573 73
>>
>> And below you can see a packet which *is* ignored by Graylog:
>>
>> 10.50.255.111.56993 > Silenoz.syslog: [udp sum ok] SYSLOG, length:
>> 154
>> Facility local7 (23), Severity notice (5)
>> Msg: Feb 8 19:12:17: %SYSLOG-5-NOTICE: aaad: SubSessionAUTHFAIL user:
>> pppoe16344@mn (24) Authentication failure [Circuit handle: 1/4:511:63:31/
>> 6/2/47661]\0x0a
>> 0x0000: 3c31 3839 3e46 6562 2038 2031 393a 3132
>> 0x0010: 3a31 373a 2025 5359 534c 4f47 2d35 2d4e
>> 0x0020: 4f54 4943 453a 2061 6161 643a 2053 7562
>> 0x0030: 5365 7373 696f 6e41 5554 4846 4149 4c20
>> 0x0040: 7573 6572 3a20 7070 706f 6531 3633 3434
>> 0x0050: 406d 6e20 2832 3429 2041 7574 6865 6e74
>> 0x0060: 6963 6174 696f 6e20 6661 696c 7572 6520
>> 0x0070: 5b43 6972 6375 6974 2068 616e 646c 653a
>> 0x0080: 2031 2f34 3a35 3131 3a36 333a 3331 2f36
>> 0x0090: 2f32 2f34 3736 3631 5d0a
>> 0x0000: 4500 00b6 77da 0000 4011 ef82 0a32 ff6f [email protected]
>> 0x0010: 0a32 ff06 dea1 0202 00a2 28d8 3c31 3839 .2........(.<189
>> 0x0020: 3e46 6562 2038 2031 393a 3132 3a31 373a >Feb.8.19:12:17:
>> 0x0030: 2025 5359 534c 4f47 2d35 2d4e 4f54 4943 .%SYSLOG-5-NOTIC
>> 0x0040: 453a 2061 6161 643a 2053 7562 5365 7373 E:.aaad:.SubSess
>> 0x0050: 696f 6e41 5554 4846 4149 4c20 7573 6572 ionAUTHFAIL.user
>> 0x0060: 3a20 7070 706f 6531 3633 3434 406d 6e20 :.pppoe16344@mn.
>> 0x0070: 2832 3429 2041 7574 6865 6e74 6963 6174 (24).Authenticat
>> 0x0080: 696f 6e20 6661 696c 7572 6520 5b43 6972 ion.failure.[Cir
>> 0x0090: 6375 6974 2068 616e 646c 653a 2031 2f34 cuit.handle:.1/4
>> 0x00a0: 3a35 3131 3a36 333a 3331 2f36 2f32 2f34 :511:63:31/6/2/4
>> 0x00b0: 3736 3631 5d0a 7661].
>>
>> As you can see, the packet is much longer, but it doesn't exceed the
>> maximum UDP packet size that can be processed by Graylog (8192). My guess
>> is that logs coming from 10.50.255.111 are not RFC compatible and thus
>> they're discarded by Graylog. How can I debug it / fix it? I didn't find
>> any related messages in the Elasticsearch log (there were no errors related
>> to parsing a message).
>> I deleted the default Input object and added a new RAW UDP Input object.
>> It didn't fix the issue - logs from 10.50.255.111 are still not parsed.
>>
>
Thanks for your answer Jochen. It's not a problem with routing nor with
firewall. I can capture these packets with TCPDump (as shown in my post) on
the very same machine, on which the Graylog server is running. I don't have
a Syslog UDP input, I deleted it and created a Raw UDP Input instead.
Here's the configuration of this input:
- Type: Raw/Plaintext UDP
- Title: Main
- Global: unchecked
- Node: 01fd7feb/Silenoz (that's the only node I have)
- Bind address: 10.50.255.6
- Port: 10514
- Buffer size: 262144
As you can see I'm using the 10514 port, not 514. I can't use 514, because
the Graylog server would have to run with root privileges. As a workaround
I created an iptables rule which routes all packets from port 514 to 10514:
-A PREROUTING -i eth0 -p udp -m udp --dport 514 -j REDIRECT --to-ports 10514
I'm 100% sure that it works, because all other hosts *from the same subnet*
(10.50.255.0/24) are sending their logs to 10.50.255.6:10514 and they are
properly parsed by the Graylog server.
Here's a funny thing though: the following frame got parsed by Graylog (it
came from the problematic router):
uluru.miconet.pl.58254 > Silenoz.syslog: [udp sum ok] SYSLOG, length:
154
Facility local0 (16), Severity notice (5)
Msg: Feb 9 12:36:55: %SYSLOG-5-NOTICE: aaad: SubSessionAUTHFAIL user:
pppoe12431@mn (24) Authentication failure [Circuit handle: 1/4:511:63:31/6/2
/63670]\0x0a
0x0000: 3c31 3333 3e46 6562 2039 2031 323a 3336
0x0010: 3a35 353a 2025 5359 534c 4f47 2d35 2d4e
0x0020: 4f54 4943 453a 2061 6161 643a 2053 7562
0x0030: 5365 7373 696f 6e41 5554 4846 4149 4c20
0x0040: 7573 6572 3a20 7070 706f 6531 3234 3331
0x0050: 406d 6e20 2832 3429 2041 7574 6865 6e74
0x0060: 6963 6174 696f 6e20 6661 696c 7572 6520
0x0070: 5b43 6972 6375 6974 2068 616e 646c 653a
0x0080: 2031 2f34 3a35 3131 3a36 333a 3331 2f36
0x0090: 2f32 2f36 3336 3730 5d0a
0x0000: 4500 00b6 12d4 0000 3f11 7746 c3e1 2403 E.......?.wF..$.
0x0010: 0a32 ff06 e38e 0202 00a2 59a6 3c31 3333 .2........Y.<133
0x0020: 3e46 6562 2039 2031 323a 3336 3a35 353a >Feb.9.12:36:55:
0x0030: 2025 5359 534c 4f47 2d35 2d4e 4f54 4943 .%SYSLOG-5-NOTIC
0x0040: 453a 2061 6161 643a 2053 7562 5365 7373 E:.aaad:.SubSess
0x0050: 696f 6e41 5554 4846 4149 4c20 7573 6572 ionAUTHFAIL.user
0x0060: 3a20 7070 706f 6531 3234 3331 406d 6e20 :.pppoe12431@mn.
0x0070: 2832 3429 2041 7574 6865 6e74 6963 6174 (24).Authenticat
0x0080: 696f 6e20 6661 696c 7572 6520 5b43 6972 ion.failure.[Cir
0x0090: 6375 6974 2068 616e 646c 653a 2031 2f34 cuit.handle:.1/4
0x00a0: 3a35 3131 3a36 333a 3331 2f36 2f32 2f36 :511:63:31/6/2/6
0x00b0: 3336 3730 5d0a 3670].
As you can see, the content of this frame is very similar to the one that
didn't get parsed. So the RFC is not a problem. The *solution* was to
change a so called "context" on the problematic router. After doing that,
it sends all syslog packets to its default gateway, which then sends it to
10.50.255.6 (the Graylog server). You can see that the source host changed
from 10.50.255.111 (the problematic router) to uluru.miconet.pl (default
gateway).
It's very strange if you ask me, both ways of sending syslog packets (via
default gateway or directly) worked - they were reaching the Graylog server
machine, because TCPDump was able to capture them. I have no idea why
packets coming from 10.50.255.111 are ignored and the very same packets
coming from uluru.miconet.pl are properly parsed. I know that TCPDump is
placed before IPTables, but even after clearing all iptables rules besides
the port prerouting it still didn't work, so it's not a problem with a
firewall on this server.
I guess I'll just stick with this workaround.
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/72fb3e64-f60d-4b34-b792-bec8596d464b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
