Hey guys. So what you are mentioning here is that I could make something like:
Client --> TCP 10517 --> rsyslog(Frontend) --> process data, change message based on Template --> LB Data to other rsyslog servers --> Store Data What would be used to LB the data. Yet another "frontend" to do the connection relay ? Best regards Jan -----Original Message----- From: rsyslog <[email protected]> On Behalf Of Mariusz Kruk via rsyslog Sent: 22. august 2023 11:44 To: [email protected] Cc: Mariusz Kruk <[email protected]> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log messages It's not about the number of nodes :-) I have multi-layered rsyslog-based solution where the central node processes about 35k EPS and load-balances them over several outgoing RELP connections and it doesn't break a sweat (true, it sends the data out to "workers" for the heavier parts of event processing). @David Yes, I know you can embed your original message in json. You can do a similar thing by cleverly formatting the forwarder event and appending the original IP to the message in a form that can be "unpacked". But it of course does need a syslog daemon as the load balancer, not a simple connection relay. MK On 22.08.2023 11:38, JPIM (Jan Primdahl Madsen) via rsyslog wrote: > Huge amount. > 2000+ servers > 2200 network nodes where 200 is firewalls logging GB of date each every day. > > This is where I wanted to be able to scale out when 1 node is not > enough, and 2 nodes is currently added for redundancy. (Yes could run > Active/Passive setup sharing a VIP with no LB and this would probably fix my > challenges for now, but will break the scalability option) I also do > understand that NIC's could be a bottleneck on the LB side in the huge amount > of data which is another headache I need to crack if possible. > But for making sure the the Log Servers can keep up and not hit IO Waite > state on disk side I'm scaling out. > > Best regards > Jan > > -----Original Message----- > From: David Lang <[email protected]> > Sent: 22. august 2023 11:32 > To: JPIM (Jan Primdahl Madsen) via rsyslog <[email protected]> > Cc: JPIM (Jan Primdahl Madsen) <[email protected]> > Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log > messages > > what are you considering 'a huge amount of syslogs'? Rsyslog is fast enough > that you seldom need multiple systems for capacity, just for redundency. > > David Lang > > On Tue, 22 Aug 2023, JPIM (Jan Primdahl Madsen) via rsyslog wrote: > >> Having a syslog cluster with LB in front is for scalability MK. >> I'm going to receive a huge amount of syslogs from servers, network >> equipment as Firewalls connections. >> So yes I want to be able to scale enough by LB between multiple log servers. >> >> But got your point of not able to use header stuff. >> I will try to look on some of the point David Lang mention by looking into >> the use of corosync/pacemaker if this is something that can help solving our >> challenge. >> >> Thanks a lot for the answer. >> Best regards >> Jan >> >> >> -----Original Message----- >> From: rsyslog <[email protected]> On Behalf Of >> Mariusz Kruk via rsyslog >> Sent: 22. august 2023 11:09 >> To: [email protected] >> Cc: Mariusz Kruk <[email protected]> >> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log >> messages >> >> Caution: This email originates from outside the company. You should >> carefully examine the sender and any links or actions before clicking. Use >> the "Report Phishing" button on the ribbon to have Security analyze the >> email if in doubt. >> >> >> The main question would be _why_ do you use LB in the first place. The >> reasonable solution depends on the answer to this question. >> >> If you had a network-level load-balancer, it would work without any further >> configuration, as you have a proxy-type load-balancer, it's only natural to >> have the LB IP as your $fromhost-ip and there is not much you can do >> about.it. >> >> You either need to configure your sources to provide individually >> distinguishable source identifiers in the events (so that if you have >> multiple logging servers they don't all report hostname of "server1") or >> think about reorganizing your infrastructure. >> >> In syslog there is no concept of "headers" so you can't add proxy headers on >> your nginx. If you want to modify your event on the loadbalancer, you'd need >> to use a syslog daemon (rsyslog?) for load-balancing. But that seems a bit >> overengineered since probably just a single rsyslog receiving events would >> suffice. >> >> MK >> >> >> On 22.08.2023 10:58, JPIM (Jan Primdahl Madsen) via rsyslog wrote: >>> Hello All rsyslog users and developers 😊 >>> >>> The main problem. >>> When receiving loadbalanced TCP syslog messages our Loadbalancer IP is the >>> IP added to the variable $fromhost-ip no matter what we do. This is due to >>> the fact that the LB is doing TCP routing and becomes the sending IP on the >>> network layer. The syslog message does not contain the IP of the sending >>> node. >>> >>> The Setup: >>> The setup contains only 2 servers which have 2 subnets (Frontend and >>> a Backend(For TCP messages)) The servers are using the following >>> software >>> >>> * Nginx (For LB) >>> * Keepalived (For VIP handling) >>> * Rsyslog (yes for syslog messages) >>> >>> >>> Question 1: >>> I was thinking about adding some Proxy Headers to the log message on the LB >>> (nginx) side and then using that in rsyslog to overwrite the $fronhost-ip. >>> Is this possible in anyway to have rsyslog to use Proxy Headers like: >>> >>> * X-Forwarded-For >>> * X-Real-IP >>> >>> If possible howto do it ? >>> If not, any other great suggestions to preserve IP address of >>> sending source in a LB TCP syslog setup is appreciated >>> >>> Please be aware, adding more HW to the setup is not an option. >>> >>> Best regards >>> Jan P. Madsen >>> >>> This e-mail (including any attachments) is intended for the addressee(s) >>> stated above only and may contain confidential information protected by >>> law. You are hereby notified that any unauthorized reading, disclosure, >>> copying or distribution of this e-mail or use of information contained >>> herein is strictly prohibited and may violate rights to proprietary >>> information. If you are not an intended recipient, please return this >>> e-mail to the sender and delete it immediately hereafter. Thank you. >>> _______________________________________________ >>> rsyslog mailing list >>> https://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> https://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This >> is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our >> control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> https://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites >> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE >> THAT. > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
