Maybe I'm missing something, but isn't Splunk free for smaller needs? Also, despite what the hardware req says, it really doesn't take much to run Splunk unless you are really pumping out the logs. Maybe I missed it, but did you state how many logs you are producing as a log/sec or kb/sec estimate?
--- Puryear IT, LLC - Baton Rouge, LA - http://www.puryear-it.com/ Active Directory Integration : Web & Enterprise Single Sign-On Identity and Access Management : Linux/UNIX technologies Download our free ebook "Best Practices for Linux and UNIX Servers" http://www.puryear-it.com/pubs/linux-unix-best-practices/ -----Original Message----- From: discuss-boun...@lopsa.org [mailto:discuss-boun...@lopsa.org] On Behalf Of da...@lang.hm Sent: Monday, March 01, 2010 1:00 PM To: Rob Das Cc: discuss@lopsa.org Subject: Re: [lopsa-discuss] splunk alternatives Rob, one comment on the real-time search capability of splunk4, per recent conversations with splunk, the real-time search is not going to be able to be integrated with past data. in other words, you can say, 'do this search on data that arrives after now', but you cannot say 'do this search on data that arrived/arrives after 5 min ago' David Lang On Mon, 1 Mar 2010, Rob Das wrote: > Date: Mon, 1 Mar 2010 10:26:38 -0800 > From: Rob Das <rob...@gmail.com> > To: discuss@lopsa.org > Subject: [lopsa-discuss] splunk alternatives > > First, please forgive me if this email is overly long. > > Yes, SEC and Splunk are different in many ways - both useful in the > right context. I have a few questions. How much data per day are you > talking about? Are you interested in looking at historical data and > comparing it against current data? Do you need any sort of roll-ups > or more advanced aggregations/analytics on your data? You may not > now, but will you ever be interested in gathering events that cannot > be captured via syslog (extra large, application or multi-line events > for example)? Do you want different people to have access to > different types of data. Do you want different roles of users to see > different views? Do you foresee that the data volumes will grow over > time? Are your 20 users really concurrent or will they be searching randomly throughout the day? > > First of all the new version of Splunk (version 4.1), which will be > out very soon, includes real-time support. What this means is that > searches can optionally be executed at data input time as the data is > acquired. If events as they come in match a search, alerts can be triggered. > Furthermore, Splunk's dashboards, graphs and tables will update in > real time as the data comes in effectively providing a "heartbeat". > > If you need to "find the needle in the haystack", you can't find a > better tool. > > Simple stuff like "Tell me the top ten logins by IP address over the > last 24 hours or month" can't be done with SEC without writing code. > Splunk handles this via it's GUI and graphs like this can be placed on dashboards which > update in real-time. Splunk can easily filter out data that you are not > interested in or keep it for as long as you like - your choice. > > Splunk provides role-based access controls that can optionally filter > data at search time depending on who is allowed to see what. > > One of the most important concepts is that Splunk doesn't require or > impose any structure on the incoming data. You can apply structure at > search time, which means that as data changes in your data center > (because new versions of software/hardware are installed, etc), you > will not need to re-do any regular expressions. > > Depending on daily data volumes, Splunk will run very well on > commodity type hardware. As your business grows, it can scale to > handle it (to terrabytes/day). If your daily volume doesn't exceed > 500M/day, you can use the free version of the software. > > SEC is a low level tool written in Pearl that requires you to create > regular expressions that match patterns in your data. It also > requires quite a bit of scripting to make it work in many > environments. As things change, you will need to update your regular expressions or things will break. > > SEC implements a state machine that operates over incoming data. > There are many cool things you can do with it, but like David L says > keeps all of it's state in memory. Splunk does not currently > implement a state machine in the same way as SEC. However, Splunk's > search language, which is extremely robust, can handle many of the > same use cases - especially with the introduction of real-time searching version 4.1. > > I have not taken a look at logsurfer, so I can't comment on it. I'll > check it out. > > I am more than happy to field questions directly if you wish. > > Rob Das > r...@splunk.com > Co-founder / Chief Architect > Splunk, Inc. > >> Paul DiSciascio wrote: >>> I'm looking for a good way to share log files on a centralized >>> syslog server with about 10-20 people/developers who are familiar >>> with the log formats but not very much with unix tools. They want >>> an easy way to dig thru the logs and filter out junk they're not >>> interested in, but still have near realtime visibility. Obviously, >>> splunk can do this, but it's pricey and their documentation seems to >>> indicate that 20 concurrent users would be a lot to ask for without a lot of hardware. >>> I really only need an interface capable of some rudimentary >>> filtering, and if possible the ability to save those searches or >>> filters. Does anyone have any suggestions short of writing this myself? >>> >>> >> You might be interested in SEC (simple event correlator) for this >> purpose. But, if you just want a presentation interface, logsurfer >> might be more what you are looking for. SEC is much more like splunk >> while logsurfer is more of a realtime filtering monitor. > > I'm not sure what you have seen of splunk, but it and SEC have very > little in common. > > splunk allows for arbatrary search queries against your past log data > (and indexes it like crazy to make the search fairly efficiant) > > SEC watches for patterns (or combinations of patterns) to appear in > the logs and generates alerts. > > splunk can simulate SEC's functionality by doing repeated queries > against the logs, but that's fairly inefficant. > > > the answer to the original question, it depends a lot on the amount of > data that you are working with. > > If you can fit it all in ram on a machine, then there are a lot of > things that you can use to query it. The problem comes when you can no > longer fit it in ram and have to go to disk, at that point you need an > application that does a lot of indexing (and/or spreads the load > across multiple machines, depending on how much data you have and how > fast you want your > answers) > > you say that your users are not familiar with unix tools, are they > familiar with using SQL for queries? > > David Lang > _______________________________________________ Discuss mailing list Discuss@lopsa.org http://lopsa.org/cgi-bin/mailman/listinfo/discuss This list provided by the League of Professional System Administrators http://lopsa.org/
