On Tue, 14 May 2013, Chris Bartram wrote:
We are in the planning stages of setting up a rsyslog server pool to
accommodate syslog streams from a couple thousand *nix servers; including
auditd type data and potentially some application logs (so it's going to be a
VERY high volume of data) and we're looking to archive this data somewhere.We
have a 10Gb network infrastructure, and I can throw as many RHEL machines at
it as needed (as well as F5 load balancers in front).
Eventually the data may need to be searched, but highest priority is getting
it written somewhere quickly (and reliably - we need to minimize any possible
data loss so our archives can stand up to auditing requirements). In that
regard, any suggestions on file systems that can handle that kind of load?
Ideally we want all the log files written to the same storage somewhere - i.e.
we don't want to have to consolidate files from separate locations to search
all the log files for some specific host. On the other hand we can split up
load by subnet sources perhaps and route specific machines to specific rsyslog
clusters to ease the load on any one cluster (though our larger subnets still
may have around 1,000 systems reporting); as long as it's easy to identify
where to look for data from a given host.
I welcome any advice on setups that allow multiple concurrent (active) rsyslog
servers writing to a common-ish file system as well as any gotchas or
performance benchmarks we can use to help plan the system.
do you have any idea what sort of data volume you are talking about here?
you say "VERY high volume of data", but different people define that in
different ways :-)
I've built a system to handle 100K log messages/sec and I gave a presentation on
it at LISA in december, the video, paper and slides are available at
https://www.usenix.org/conference/lisa12/building-100k-logsec-logging-infrastructure
When I built it, I didn't have access to any 10G equipment, so I could only test
things up to ~380K log messages/sec.
At work I am currently part of a team defining how to take what I built for what
was an 800 person company (with an extremely large web presense) when it was
aquired and scale it up to the 8000 person company that aquired us. As part of
this, one of the other people tried to scare me about the total log volume by
saying that they handled 2B log messages in a month. I laughed and showed him
that my small subset of the business handled 18B log messages that same month,
without any of my systems breathing hard (other than the nightly log reporting
run, which will peg any server you use for it, the question is just how long it
will peg it :-)
Based on my experience, unless you have a lot more logs than I expect from only
a couple thousand servers, I don't think you need to do anything fancy. A
mid-range system with a modest RAID with XFS of ext4 should be able to handle
your log volume without a problem (well, you want it to be a HA pair of systems,
but only one needs to be active at a time)
If you are comfortable going into more details in public, we can continue the
discussion here on the list. If not, contact me directly.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
