On Fri, 12 Sep 2014, David Lang wrote:
On Thu, 11 Sep 2014, Yves Dorfsman wrote:
On 2014-09-10 22:26, David Lang wrote:
How many different logs are you talking about?
It depends on the server, apps server can have a dozen logs + the regular
system logs.
Do you have another method of getting the logs other than scraping the
files?
No, I thought graylog2 would come with its own mechanism.
Do these logs have any unusual features (extremely long lines, multi-line
log
messages, etc)
Do the files rotate by date, or with the 'standard' approach of mv the old
file and re-open it?
Mostly logrotate, some of them by date, some with a combination of
date/size.
I have seen logstash mentioned, but picture people sending files to the
logstash server which would then forward them to graylog2... if the
logstash
forwarder can send files directly to the graylog2 server, then that will
probably be the simplest solution.
logstash does have mechanisms to scrape files to send them to a log server.
This really is a simple task (the syslog UDP protocol is _really_ simple).
Unfortunantly a lot of existing tools have a hard-coded 1k size limitation,
but writing something to beat that is pretty simple.
I meant to add
In general, I dislike the methodology of writing data to disk for another file
to scrape and do something with. There are just so many things that can go wrong
disks fill up
files rotate too fast and are lost
extra wear on flash devices
The I/O load on the system causing other issues
filesystem layout issues from files being written to in small amounts over a
long time lead to fragmented files (because the filesystem can't see the future)
If you can arrange it, a direct handoff from the thing generating the message to
the thing receiving the message is far better.
For a lot of apps, even if the app doesn't support anything other than writing
to files, you can frequently fake it out by creating a named pipe and having the
app use it as it's "log file" and then have your log forwarder read from that
pipe. This is still not ideal, but it avoids the storage/filesystem/I/O problems
of using files.
David Lang
_______________________________________________
Tech mailing list
Tech@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
