Re: buffer cache usage

2013-03-19 Thread Bruce Evans
On Tue, 19 Mar 2013, Richard Kearsley wrote: I'm using FreeBSD and nginx as a web server serving static files. I always like the "buffer cache" which lets frequently used files stay in memory rather than read from disk again. I have a new server with 32GB of ram and I notice a lot of memory is

Re: find vs ls performance for walking folders, are there any faster options?

2012-12-12 Thread Bruce Evans
On Wed, 12 Dec 2012, [ISO-8859-1] Olav Gr?n?s Gjerde wrote: I'm working on scanning filesystems to build a file search engine and came over something interesting. I can walk through 300 000 folders in ~19.5seconds with this command: ls -Ra | grep -e "./.*:" | sed "s/://" With find, it surprisi

Re: cmp(1) has a bottleneck, but where?

2012-01-15 Thread Bruce Evans
On Sun, 15 Jan 2012, Dieter BSD wrote: posix_fadvise() should probably be used for large files to tell the system not to cache the data. Its man page reminded me of the O_DIRECT flag. Certainly if the combined size exceeds the size of main memory, O_DIRECT would be good (even for benchmarks that

Re: cmp(1) has a bottleneck, but where?

2012-01-14 Thread Bruce Evans
On Thu, 12 Jan 2012, Dieter BSD wrote: A) Should the default vfs.read_max be increased? Maybe, but I don't buy most claims that larger block sizes are better. I didn't say anything about block sizes. There needs to be enough data in memory so that the CPU doesn't run out while the disk is se

Re: cmp(1) has a bottleneck, but where?

2012-01-09 Thread Bruce Evans
On Thu, 5 Jan 2012, Dieter BSD wrote: Something/somehow it's issuing smaller IOs when using mmap? On my box, 64K reads. ??Using the '-' to avoid mmap it uses 128K. The big difference I found was that the default mmap case isn't using read-ahead. So it has to wait on the disk every time. ??:-(

Re: cmp(1) has a bottleneck, but where?

2012-01-09 Thread Bruce Evans
On Wed, 4 Jan 2012, Bruce Evans wrote: On Tue, 3 Jan 2012, Marc Olzheim wrote: On Tue, Jan 03, 2012 at 12:21:10AM -0800, Garrett Cooper wrote: The file is 3.0GB in size. Look at all those page faults though! Thanks! -Garrett From usr.bin/cmp/c_regular.c: #define MMAP_CHUNK (8*1024

Re: cmp(1) has a bottleneck, but where?

2012-01-03 Thread Bruce Evans
On Tue, 3 Jan 2012, Marc Olzheim wrote: On Tue, Jan 03, 2012 at 12:21:10AM -0800, Garrett Cooper wrote: The file is 3.0GB in size. Look at all those page faults though! Thanks! -Garrett From usr.bin/cmp/c_regular.c: #define MMAP_CHUNK (8*1024*1024) ... for (..) { mmap() chunk of

Re: SCHED_ULE should not be the default

2011-12-13 Thread Bruce Evans
On Wed, 14 Dec 2011, Ivan Klymenko wrote: ?? Wed, 14 Dec 2011 00:04:42 +0100 Jilles Tjoelker ??: On Tue, Dec 13, 2011 at 10:40:48AM +0200, Ivan Klymenko wrote: If the algorithm ULE does not contain problems - it means the problem has Core2Duo, or in a piece of code that uses the ULE

Re: Interrupt performance

2011-02-01 Thread Bruce Evans
On Tue, 1 Feb 2011, Slawa Olhovchenkov wrote: I do some more test and build kernel with KTR. Now I don't think that inetrrupt overhead on FreeBSD weight: I try polling and don't see any difference. I see many reported by netperf send errors. I found this http://docs.freebsd.org/cgi/mid.cgi?E1Ai

Re: Interrupt performance

2011-01-29 Thread Bruce Evans
On Sat, 29 Jan 2011, Slawa Olhovchenkov wrote: On Sat, Jan 29, 2011 at 02:43:07PM +1100, Bruce Evans wrote: On Sat, 29 Jan 2011, Slawa Olhovchenkov wrote: On Sat, Jan 29, 2011 at 07:52:11AM +1100, Bruce Evans wrote: To see how much CPU is actually available, run something else and see how

Re: Interrupt performance

2011-01-28 Thread Bruce Evans
On Sat, 29 Jan 2011, Slawa Olhovchenkov wrote: On Sat, Jan 29, 2011 at 07:52:11AM +1100, Bruce Evans wrote: To see how much CPU is actually available, run something else and see how fast it runs. A simple counting loops works well on UP systems. === #include #include int Dummy; int

Re: Interrupt performance

2011-01-28 Thread Bruce Evans
On Fri, 28 Jan 2011, Slawa Olhovchenkov wrote: On Fri, Jan 28, 2011 at 09:10:20AM -0800, Julian Elischer wrote: On 1/28/11 8:15 AM, Stefan Lambrev wrote: The overhead comes from badly written software. This software is optimized for linux and you have to optimize it for freebsd, then you wil

Re: TTY task group scheduling

2010-11-20 Thread Bruce Evans
On Fri, 19 Nov 2010, Kostik Belousov wrote: On Fri, Nov 19, 2010 at 11:50:49AM +0200, Andriy Gapon wrote: on 19/11/2010 11:46 Bruce Cran said the following: [removed current@ and stable@ from the Cc list] On Fri, 19 Nov 2010 15:41:29 +1100 Andrew Reilly wrote: On Linux. Have you ever seen

Re: tuning FFS for large files Re: A specific example of a disk i/o problem

2009-10-06 Thread Bruce Evans
On Mon, 5 Oct 2009, Dieter wrote: I found a clue! The problem occurs with my big data partitions, which are newfs-ed with options intended to improve things. Reading a large file from the normal ad4s5b partition only delays other commands slightly, as expected. Reading a large file from the t

Re: ZFS, NFS and Network tuning

2009-01-29 Thread Bruce Evans
On Thu, 29 Jan 2009, Brent Jones wrote: On Wed, Jan 28, 2009 at 11:21 PM, Brent Jones wrote: ... The issue I am seeing, is that for certain file types, the FreeBSD NFS client will either issue an ASYNC write, or an FSYNC. However, NFSv3 and v4 both support "safe" ASYNC writes in the TCP vers

Re: Micro-benchmark for various time syscalls...

2008-06-03 Thread Bruce Evans
On Mon, 2 Jun 2008, Sean Chittenden wrote: I wouldn't expect SMP to make much difference between CLOCK_REALTIME and CLOCK_REALTIME_FAST. The only difference is that the former calls nanotime() where the latter calls getnanotime(). nanotime() always does more, but it doesn't have any extra SMP

Re: Micro-benchmark for various time syscalls...

2008-06-03 Thread Bruce Evans
On Mon, 2 Jun 2008, Sean Chittenden wrote: rozetta~/devel/c%>sysctl hw.model hw.model: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz rozetta~/devel/c%>./bench_time 9079882 | sort -rnk1 Timing micro-benchmark. 9079882 syscall iterations. Avg. us/callElapsed Name 1.40546912

Re: Micro-benchmark for various time syscalls...

2008-06-03 Thread Bruce Evans
On Mon, 2 Jun 2008, Gary Stanley wrote: At 06:19 AM 6/2/2008, Bruce Evans wrote: These are very slow. Are they on a 486? :-) I get about 262 ns for CLOCK_REALTIME using the TSC timecounter on all ~2GHz UP systems. The syscall overhead is about 200 nsec (170 nsec for a simpler syscall and

Re: Micro-benchmark for various time syscalls...

2008-06-03 Thread Bruce Evans
On Mon, 2 Jun 2008, Gary Stanley wrote: At 06:55 AM 6/2/2008, Bruce Evans wrote: On Mon, 2 Jun 2008, Gary Stanley wrote: At 12:54 AM 6/2/2008, Sean Chittenden wrote: PS Is there a reason that time(3) can't be implemented in terms of clock_gettime(CLOCK_SECOND)? 10ms seems precise e

Re: Micro-benchmark for various time syscalls...

2008-06-02 Thread Bruce Evans
On Sun, 1 Jun 2008, Sean Chittenden wrote: I wrote a small micro-benchmark utility[1] to test various time syscalls and the results were a bit surprising to me. The results were from a UP machine and I believe that the difference between gettimeofday(2) and clock_gettime(CLOCK_REALTIME_FAST)

Re: Micro-benchmark for various time syscalls...

2008-06-02 Thread Bruce Evans
On Mon, 2 Jun 2008, Claus Guttesen wrote: % ./bench_time 9079882 | sort -rnk1 Timing micro-benchmark. 9079882 syscall iterations. Avg. us/callElapsed Name 9.32248484.647053 gettimeofday(2) 8.95532481.313291 time(3) 8.64831578.525684 clock_gettime(2/CLOCK_RE

Re: Micro-benchmark for various time syscalls...

2008-06-02 Thread Bruce Evans
On Mon, 2 Jun 2008, Gary Stanley wrote: At 12:54 AM 6/2/2008, Sean Chittenden wrote: PS Is there a reason that time(3) can't be implemented in terms of clock_gettime(CLOCK_SECOND)? 10ms seems precise enough compared to time_t's whole second resolution. Another interesting idea is to map get

Re: maxphys and block sizes on slices

2008-02-24 Thread Bruce Evans
On Mon, 25 Feb 2008, Chris wrote: I got a server that is primarily handling large files not massive files but files that are 15meg+ in size and very few smaller files. So I decided to use the following options in newfs. -f 4096 -b 32768 Eventually I realised this was a bad decision especially

Re: Memory allocation performance

2008-02-03 Thread Bruce Evans
On Mon, 4 Feb 2008, Alexander Motin wrote: Kris Kennaway wrote: You can look at the raw output from pmcstat, which is a collection of instruction pointers that you can feed to e.g. addr2line to find out exactly where in those functions the events are occurring. This will often help to track

Re: Memory allocation performance

2008-02-01 Thread Bruce Evans
On Fri, 1 Feb 2008, Alexander Motin wrote: Robert Watson wrote: It would be very helpful if you could try doing some analysis with hwpmc -- "high resolution profiling" is of increasingly limited utility with modern You mean "of increasingly greater utility with modern CPUs". Low resolution k

Re: mysql scaling questions

2008-01-02 Thread Bruce Evans
On Wed, 2 Jan 2008, Gary Stanley wrote: At 06:00 AM 1/2/2008, Kris Kennaway wrote: Per later discussion you will also need to either comment out the syscalls that are (might be) being cached by glibc to artificially inflate its reported rate, or verify that it is not doing so. IIRC linux us

Re: mysql scaling questions

2008-01-01 Thread Bruce Evans
On Tue, 1 Jan 2008, Jeff Roberson wrote: On Tue, 1 Jan 2008, Gergely CZUCZY wrote: There's this SYSCALL CPU extension with the SYSENTER/SYSEXIT features. IIRC Linux takes advantage of this, while FreeBSD doesn't. I might be wrong here, of course. This is true on 32bit x86 and not true on

Re: mysql scaling questions

2007-12-31 Thread Bruce Evans
On Mon, 31 Dec 2007, Ivan Voras wrote: Kris Kennaway wrote: It's a general question. It looks like myisam either has a design deficiency in this regard or it has poor defaults. If it can be made to improve caching of the data in userland then performance should improve. Isn't this common fo

Re: ULE vs 4BSD in RELENG_7

2007-11-05 Thread Bruce Evans
On Sun, 4 Nov 2007, Jeff Roberson wrote: On Sun, 4 Nov 2007, Gelsema, P (Patrick) - FreeBSD wrote: w/o patch hulk# time make -j8 buildkernel 837.808u 138.167s 10:28.96 155.1% 6349+1349k 2873+7780io 303pf+0w w patch hulk# time make -j8 buildkernel 838.554u 168.316s 10:52.10 154.4%

Re: ULE vs. 4BSD in RELENG_7

2007-10-24 Thread Bruce Evans
On Tue, 23 Oct 2007, Kris Kennaway wrote: Josh Carroll wrote: Anyway, in summary, ULE is about 5-6 % slower than 4BSD for two workloads that I am sensitive to: building world with -j X, and ffmpeg -threads X. Other benchmarks seem to indicate relatively equal performance between the two. MySQL,

Re: MFC of UDP socket performance test

2007-03-16 Thread Bruce Evans
On Thu, 15 Mar 2007, Marcelo Gardini do Amaral wrote: I repeated that performance test done with bind [1] using now Robert's MFC [2]. Another tweak that I was supposed to do was to use libthr instead of libpthread (via libmap.conf) and build bind with threads option. In the new test I did this.

Re: (S)ATA performance in FBSD 6.2/7.0

2007-03-04 Thread Bruce Evans
On Mon, 5 Mar 2007, dima wrote: I think you're quite OK with dd. I do believe you'll get a comparable results for cp setting "noatime" option in mount(8)s. As you were told, the default mount mode is noasync which is "synchronous metadata + asynchronous data". But FreeBSD still updates metadat

Re: (S)ATA performance in FBSD 6.2/7.0

2007-03-04 Thread Bruce Evans
On Sat, 3 Mar 2007, Alexander Leidinger wrote: In FreeBSD we have 3 types, not 2. We have "sync", "noasync" (default) and "async". Are you sure? 2 independent flags give 4 states. I think the flags are not completely independent, but sync doesn't cancel async in all cases (partly because of

Re: (S)ATA performance in FBSD 6.2/7.0

2007-03-02 Thread Bruce Evans
On Fri, 2 Mar 2007, Brooks Davis wrote: Also, you should time the actual copy and do the math to verify that vmstat is actually producing valid results. It's possible there's a bug in vmstat or the underlying statistics it uses. There is certainly a bug in the underlying statistics. For ATA

Re: Cached file read performance

2006-12-23 Thread Bruce Evans
On Sat, 23 Dec 2006, I wrote: The problem becomes smaller as the read block size appoaches the file system block size and vanishes when the sizes are identical. Then there is apparently a different (smaller) problem: Read size 16K, random: %%% granularity: each sample hit covers 16 byte(s) for

Re: Cached file read performance

2006-12-23 Thread Bruce Evans
On Sat, 23 Dec 2006, Mark Kirkwood wrote: Bruce Evans wrote: None was attached. (meaning the c prog yes?) I notice that it is stripped out from the web archive... so here's a link: http://homepages.paradise.net.nz/markir/download/freebsd/readtest.c However, I couldn'

Re: Cached file read performance

2006-12-22 Thread Bruce Evans
On Fri, 22 Dec 2006, Adrian Chadd wrote: On 22/12/06, David Xu <[EMAIL PROTECTED]> wrote: I suspect in such a test, memory copying speed will be a key factor, I don't have number to back up my idea, but I think Linux has lots of tweaks, such as using MMX instruction to copy data. I had the o

Re: Cached file read performance

2006-12-22 Thread Bruce Evans
On Fri, 22 Dec 2006, Mark Kirkwood wrote: I recently did some testing on the performance of cached reads using two (almost identical) systems, one running FreeBSD 6.2PRE and the other running Gentoo Linux - the latter acting as a control. I initially started a thread of the same name on -stabl

Re: Polling tuning and performance

2006-12-15 Thread Bruce Evans
On Fri, 15 Dec 2006, Alan Amesbury wrote: Bruce Evans wrote: ... The (extremely busy) interface is exclusively incoming traffic, received promiscuously. Since that's provided enough clues as to what this box might actually be doing, I'll give away the secret: It's running

Re: Polling tuning and performance

2006-12-15 Thread Bruce Evans
On Thu, 14 Dec 2006, Alan Amesbury wrote: ... What I'm aiming for, of course, is zero packet loss. Realizing that's probably impossible for this system given its load, I'm trying to do what I can to minimize loss. ... * PREEMPTION disabled - /sys/conf/NOTES says this helps with

Re: DNS Performance Numbers

2006-11-18 Thread Bruce Evans
On Fri, 17 Nov 2006, Marcelo Gardini do Amaral wrote: I made some tests using 7.x with all the debugging disabled: queries / s Int bind (d_t) bind (e_t) nsd (1_s) nsd (2_s) --- -- -- - - bge

Re: DNS Performance Numbers

2006-11-17 Thread Bruce Evans
On Thu, 16 Nov 2006, Marcelo Gardini do Amaral wrote: I made some tests using 7.x with all the debugging disabled: queries / s Int bind (d_t) bind (e_t) nsd (1_s) nsd (2_s) --- -- -- - - bge

Re: Initial 6.1 questions

2006-06-13 Thread Bruce Evans
On Tue, 13 Jun 2006, Kip Macy wrote: ... Why do I say "non-interrupt blocking?". Currently we have roughly a half dozen locking primitives. The two that I am familiar with are blocking and spinning mutexes. The general policy is to use blocking locks except where a lock is used in interrupts or

Re: dd(1) performance when copiing a disk to another

2005-10-03 Thread Bruce Evans
On Mon, 3 Oct 2005, [ISO-8859-1] Tulio Guimar?es da Silva wrote: But just to clear out some questions... 1) Maxtor?s full specifications for Diamond Max+ 9 Series refers to maximum *sustained* transfer rates of 37MB/s and 67MB/s for "ID" and "OD", respectively (though I couldn?d find exactly w

Re: dd(1) performance when copiing a disk to another

2005-10-03 Thread Bruce Evans
On Mon, 3 Oct 2005, Patrick Proniewski wrote: # dd if=/dev/ad4 of=/dev/null bs=1m count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 17.647464 secs (59417943 bytes/sec) Many wrong answers to the original question have been given. dd with a blocks size of 1m betwee

Re: FreeBSD MySQL still WAY slower than Linux

2005-06-29 Thread Bruce Evans
On Wed, 29 Jun 2005, Jeremie Le Hen wrote: Could you try mounting the filesystem where the database lives with the noatime option, and re-run your tests ? IIRC from previous threads on this subject, Linux doesn't really honor this while FreeBSD does, which pulls down the performances. I think

Re: Slave IDE HDD not working in UDMA5

2005-06-20 Thread Bruce Evans
On Thu, 16 Jun 2005 [EMAIL PROTECTED] wrote: This was first posted on freebsd-question, but I could not find the solution yet. Maybe you could help me. Maybe a little. I don't use 5.4... I?ve installed FreeBSD 5.4 on a new machine with the following hardware:

Re: vn(4) performance on 4.11 versus md(4) on 5.4

2005-06-14 Thread Bruce Evans
On Sat, 4 Jun 2005, Glenn Dawson wrote: I have a number of systems running 4.11 that have file backed virtual disks, each of which contains a jail. I need to start using 5.4 for new servers. The catch is, file backed virtual disks using md(4) seem to be much slower than similar virtual disks

Re: rmt as a bottleneck - Was: Weird behaviour of AIT-3 and (g)tar

2005-05-30 Thread Bruce Evans
Observe that I bypassed rmt; that bumped the transfer rate to 10.976.153,96 Bytes/s, almost 30x faster. Should this really happen? (And yes, I read rmt(8), but found nothing about this. :( ). Thanks for your help; ISTR that remote tars have a delay of 10 msec or so for each block because the

Re: Very low disk performance on 5.x

2005-05-09 Thread Bruce Evans
On Sun, 8 May 2005, Scott Long wrote: Changing MAXPHYS is very dangerous, unfortunately. The root of the problem is that kernel virtual memory (KVA) gets assigned to each I/O buffer as it passes through the kernel. If we allow too much I/O through at once then we have the very real possibility of

Re: Performance Intel Pro 1000 MT (PWLA8490MT)

2005-04-19 Thread Bruce Evans
On Tue, 19 Apr 2005, Bosko Milekic wrote: My experience with 6.0-CURRENT has been that I am able to push at least about 400kpps INTO THE KERNEL from a gigE em card on its own 64-bit PCI-X 133MHz bus (i.e., the bus is uncontested) and that's A 64-bit bus doesn't seem to be essential for reasonabl