On Tue, 22 Aug 2023, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
Hey guys.
So what you are mentioning here is that I could make something like:
Client --> TCP 10517 --> rsyslog(Frontend) --> process data, change message based on
Template --> LB Data to other rsyslog servers --> Store Data
What would be used to LB the data. Yet another "frontend" to do the connection
relay ?
once you change the logs, you can then use a proxy-type TCP load balancer since
you aren't depending on fromhost-ip, or you could use CLUSTERIP to balance on
the receiving machines
see my presentation/slides on the topic
https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david
David Lang
Best regards
Jan
-----Original Message-----
From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of Mariusz Kruk via
rsyslog
Sent: 22. august 2023 11:44
To: rsyslog@lists.adiscon.com
Cc: Mariusz Kruk <k...@epsilon.eu.org>
Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log messages
It's not about the number of nodes :-)
I have multi-layered rsyslog-based solution where the central node processes about 35k
EPS and load-balances them over several outgoing RELP connections and it doesn't break a
sweat (true, it sends the data out to "workers" for the heavier parts of event
processing).
@David Yes, I know you can embed your original message in json. You can do a similar
thing by cleverly formatting the forwarder event and appending the original IP to the
message in a form that can be "unpacked". But it of course does need a syslog
daemon as the load balancer, not a simple connection relay.
MK
On 22.08.2023 11:38, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
Huge amount.
2000+ servers
2200 network nodes where 200 is firewalls logging GB of date each every day.
This is where I wanted to be able to scale out when 1 node is not
enough, and 2 nodes is currently added for redundancy. (Yes could run
Active/Passive setup sharing a VIP with no LB and this would probably fix my
challenges for now, but will break the scalability option) I also do understand
that NIC's could be a bottleneck on the LB side in the huge amount of data
which is another headache I need to crack if possible.
But for making sure the the Log Servers can keep up and not hit IO Waite state
on disk side I'm scaling out.
Best regards
Jan
-----Original Message-----
From: David Lang <da...@lang.hm>
Sent: 22. august 2023 11:32
To: JPIM (Jan Primdahl Madsen) via rsyslog <rsyslog@lists.adiscon.com>
Cc: JPIM (Jan Primdahl Madsen) <j...@aevengroup.com>
Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log
messages
what are you considering 'a huge amount of syslogs'? Rsyslog is fast enough
that you seldom need multiple systems for capacity, just for redundency.
David Lang
On Tue, 22 Aug 2023, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
Having a syslog cluster with LB in front is for scalability MK.
I'm going to receive a huge amount of syslogs from servers, network equipment
as Firewalls connections.
So yes I want to be able to scale enough by LB between multiple log servers.
But got your point of not able to use header stuff.
I will try to look on some of the point David Lang mention by looking into the
use of corosync/pacemaker if this is something that can help solving our
challenge.
Thanks a lot for the answer.
Best regards
Jan
-----Original Message-----
From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of
Mariusz Kruk via rsyslog
Sent: 22. august 2023 11:09
To: rsyslog@lists.adiscon.com
Cc: Mariusz Kruk <k...@epsilon.eu.org>
Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log
messages
Caution: This email originates from outside the company. You should carefully examine the
sender and any links or actions before clicking. Use the "Report Phishing"
button on the ribbon to have Security analyze the email if in doubt.
The main question would be _why_ do you use LB in the first place. The
reasonable solution depends on the answer to this question.
If you had a network-level load-balancer, it would work without any further
configuration, as you have a proxy-type load-balancer, it's only natural to
have the LB IP as your $fromhost-ip and there is not much you can do about.it.
You either need to configure your sources to provide individually distinguishable source
identifiers in the events (so that if you have multiple logging servers they don't all
report hostname of "server1") or think about reorganizing your infrastructure.
In syslog there is no concept of "headers" so you can't add proxy headers on
your nginx. If you want to modify your event on the loadbalancer, you'd need to use a
syslog daemon (rsyslog?) for load-balancing. But that seems a bit overengineered since
probably just a single rsyslog receiving events would suffice.
MK
On 22.08.2023 10:58, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
Hello All rsyslog users and developers 😊
The main problem.
When receiving loadbalanced TCP syslog messages our Loadbalancer IP is the IP
added to the variable $fromhost-ip no matter what we do. This is due to the
fact that the LB is doing TCP routing and becomes the sending IP on the network
layer. The syslog message does not contain the IP of the sending node.
The Setup:
The setup contains only 2 servers which have 2 subnets (Frontend and
a Backend(For TCP messages)) The servers are using the following
software
* Nginx (For LB)
* Keepalived (For VIP handling)
* Rsyslog (yes for syslog messages)
Question 1:
I was thinking about adding some Proxy Headers to the log message on the LB
(nginx) side and then using that in rsyslog to overwrite the $fronhost-ip. Is
this possible in anyway to have rsyslog to use Proxy Headers like:
* X-Forwarded-For
* X-Real-IP
If possible howto do it ?
If not, any other great suggestions to preserve IP address of
sending source in a LB TCP syslog setup is appreciated
Please be aware, adding more HW to the setup is not an option.
Best regards
Jan P. Madsen
This e-mail (including any attachments) is intended for the addressee(s) stated
above only and may contain confidential information protected by law. You are
hereby notified that any unauthorized reading, disclosure, copying or
distribution of this e-mail or use of information contained herein is strictly
prohibited and may violate rights to proprietary information. If you are not an
intended recipient, please return this e-mail to the sender and delete it
immediately hereafter. Thank you.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
