On Thu, 14 Jan 2010, Ryan Pugatch wrote: > Does anyone have recommendations on running Splunk under Xen? > > http://www.splunk.com/base/Documentation/latest/installation/SystemRequirements > says "Note: Running Splunk in virtual machine (VM) mode on any platform > will degrade performance." and the recommended hardware config is at > least: 2x quad-core Xeon, 3GHz, 8GB RAM. > > I'm not really familiar with Splunk. Does it really require that much > power to run decently?
think of splunk as a data warehouse application. It does very interesting tricks for performance and if you need to do unpredictaable searches through log data (especially large volumes of it), it's hard to beat. I've seen it do a search for an IP that appeard 20 times out of 15 billion log entries (~500G compressed) in a couple of seconds. the current 4.0.8 release does excessive read-ahead so searching for something 'common' that appears 100K times in those same 15 billion logs currently takes ~20 min, when they fix the readahead I expect it to take ~2 min. It can also scale across multiple machines it does stress your system in interesting ways, and in my experiance the company doesn't know how it stresses the system, benchmarking with your data and your queries is _very_ important. if you are searching for something that's fairly common in your logs, it takes a little IO and a lot of processor time, if you are searching for something that's rare in your logs, it takes a lot of IO seeks and relativly little processor time. the config you list is overkill on the processor and doesn't mention the disk subsystem at all. In many ways the disk is the most important piece. In terms of the processor, splunk has trouble using more than 4 cores effectivly under any condition, so the dual quad core is definantly overkill for the processor. since virtualization suffers the most penalty when doing lots of IO (disk or network), running splunk in a VM is just about a worst-case scenerio. I'm in the process of doing a writeup of what we are doing for our high-volume splunk installation and will post it when I get it done. David Lang _______________________________________________ Discuss mailing list Discuss@lopsa.org http://lopsa.org/cgi-bin/mailman/listinfo/discuss This list provided by the League of Professional System Administrators http://lopsa.org/
