Splunk free can handle (index) 500Meg/day. You can keep as much historical data around as you wish, and therefore your search may be across many many Gigs.
Rob Das On Mon, Mar 1, 2010 at 1:42 PM, Dustin Puryear <dpury...@puryear-it.com>wrote: > Maybe I'm missing something, but isn't Splunk free for smaller needs? > Also, despite what the hardware req says, it really doesn't take much to > run Splunk unless you are really pumping out the logs. Maybe I missed > it, but did you state how many logs you are producing as a log/sec or > kb/sec estimate? > > --- > Puryear IT, LLC - Baton Rouge, LA - http://www.puryear-it.com/ > Active Directory Integration : Web & Enterprise Single Sign-On > Identity and Access Management : Linux/UNIX technologies > > Download our free ebook "Best Practices for Linux and UNIX Servers" > http://www.puryear-it.com/pubs/linux-unix-best-practices/ > > > -----Original Message----- > From: discuss-boun...@lopsa.org [mailto:discuss-boun...@lopsa.org] On > Behalf Of da...@lang.hm > Sent: Monday, March 01, 2010 1:00 PM > To: Rob Das > Cc: discuss@lopsa.org > Subject: Re: [lopsa-discuss] splunk alternatives > > Rob, > one comment on the real-time search capability of splunk4, per recent > conversations with splunk, the real-time search is not going to be able > to be integrated with past data. > > in other words, you can say, 'do this search on data that arrives after > now', but you cannot say 'do this search on data that arrived/arrives > after 5 min ago' > > David Lang > > On Mon, 1 Mar 2010, Rob Das wrote: > > > Date: Mon, 1 Mar 2010 10:26:38 -0800 > > From: Rob Das <rob...@gmail.com> > > To: discuss@lopsa.org > > Subject: [lopsa-discuss] splunk alternatives > > > > First, please forgive me if this email is overly long. > > > > Yes, SEC and Splunk are different in many ways - both useful in the > > right context. I have a few questions. How much data per day are you > > > talking about? Are you interested in looking at historical data and > > comparing it against current data? Do you need any sort of roll-ups > > or more advanced aggregations/analytics on your data? You may not > > now, but will you ever be interested in gathering events that cannot > > be captured via syslog (extra large, application or multi-line events > > for example)? Do you want different people to have access to > > different types of data. Do you want different roles of users to see > > different views? Do you foresee that the data volumes will grow over > > time? Are your 20 users really concurrent or will they be searching > randomly throughout the day? > > > > First of all the new version of Splunk (version 4.1), which will be > > out very soon, includes real-time support. What this means is that > > searches can optionally be executed at data input time as the data is > > acquired. If events as they come in match a search, alerts can be > triggered. > > Furthermore, Splunk's dashboards, graphs and tables will update in > > real time as the data comes in effectively providing a "heartbeat". > > > > If you need to "find the needle in the haystack", you can't find a > > better tool. > > > > Simple stuff like "Tell me the top ten logins by IP address over the > > last 24 hours or month" can't be done with SEC without writing code. > > Splunk handles this via it's GUI and graphs like this can be placed on > dashboards which > > update in real-time. Splunk can easily filter out data that you are > not > > interested in or keep it for as long as you like - your choice. > > > > Splunk provides role-based access controls that can optionally filter > > data at search time depending on who is allowed to see what. > > > > One of the most important concepts is that Splunk doesn't require or > > impose any structure on the incoming data. You can apply structure at > > > search time, which means that as data changes in your data center > > (because new versions of software/hardware are installed, etc), you > > will not need to re-do any regular expressions. > > > > Depending on daily data volumes, Splunk will run very well on > > commodity type hardware. As your business grows, it can scale to > > handle it (to terrabytes/day). If your daily volume doesn't exceed > > 500M/day, you can use the free version of the software. > > > > SEC is a low level tool written in Pearl that requires you to create > > regular expressions that match patterns in your data. It also > > requires quite a bit of scripting to make it work in many > > environments. As things change, you will need to update your regular > expressions or things will break. > > > > SEC implements a state machine that operates over incoming data. > > There are many cool things you can do with it, but like David L says > > keeps all of it's state in memory. Splunk does not currently > > implement a state machine in the same way as SEC. However, Splunk's > > search language, which is extremely robust, can handle many of the > > same use cases - especially with the introduction of real-time > searching version 4.1. > > > > I have not taken a look at logsurfer, so I can't comment on it. I'll > > check it out. > > > > I am more than happy to field questions directly if you wish. > > > > Rob Das > > r...@splunk.com > > Co-founder / Chief Architect > > Splunk, Inc. > > > >> Paul DiSciascio wrote: > >>> I'm looking for a good way to share log files on a centralized > >>> syslog server with about 10-20 people/developers who are familiar > >>> with the log formats but not very much with unix tools. They want > >>> an easy way to dig thru the logs and filter out junk they're not > >>> interested in, but still have near realtime visibility. Obviously, > >>> splunk can do this, but it's pricey and their documentation seems to > > >>> indicate that 20 concurrent users would be a lot to ask for without > a lot of hardware. > >>> I really only need an interface capable of some rudimentary > >>> filtering, and if possible the ability to save those searches or > >>> filters. Does anyone have any suggestions short of writing this > myself? > >>> > >>> > >> You might be interested in SEC (simple event correlator) for this > >> purpose. But, if you just want a presentation interface, logsurfer > >> might be more what you are looking for. SEC is much more like splunk > >> while logsurfer is more of a realtime filtering monitor. > > > > I'm not sure what you have seen of splunk, but it and SEC have very > > little in common. > > > > splunk allows for arbatrary search queries against your past log data > > (and indexes it like crazy to make the search fairly efficiant) > > > > SEC watches for patterns (or combinations of patterns) to appear in > > the logs and generates alerts. > > > > splunk can simulate SEC's functionality by doing repeated queries > > against the logs, but that's fairly inefficant. > > > > > > the answer to the original question, it depends a lot on the amount of > > > data that you are working with. > > > > If you can fit it all in ram on a machine, then there are a lot of > > things that you can use to query it. The problem comes when you can no > > > longer fit it in ram and have to go to disk, at that point you need an > > > application that does a lot of indexing (and/or spreads the load > > across multiple machines, depending on how much data you have and how > > fast you want your > > answers) > > > > you say that your users are not familiar with unix tools, are they > > familiar with using SQL for queries? > > > > David Lang > > > -- Rob Das Splunk
_______________________________________________ Discuss mailing list Discuss@lopsa.org http://lopsa.org/cgi-bin/mailman/listinfo/discuss This list provided by the League of Professional System Administrators http://lopsa.org/
