Re: rtorrent + OpenBSD = freeze
On Tue, 19 Feb 2008, Brian wrote:
I have seen this freeze with both xl(4) and nfe(4).
Maybe it's time folks start posting their dmesg.
Brian
I've seen this freeze, too. Seems to be related to rtorrent use.
More prevalent when rtorrent is handling multiple torrents.
The machine isn't setup as a router but after the freeze, the
computer responds to pings, but neither console nor sshd responds
leaving me no choice but hard reboot.
I originally had assumed that this was due to me using an old
-current, but since others seem to be experiencing similar freezes,
it may be worthwhile to post my dmesg, too.
I'd certainly be willing to help in any ongoing debugging effort.
dmesg below:
OpenBSD 4.2-current (GENERIC) #476: Fri Nov 2 14:41:26 MDT 2007
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: AMD Sempron(tm) 2200+ ("AuthenticAMD" 686-class, 256KB L2 cache) 1.50 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE
real mem = 234385408 (223MB)
avail mem = 218775552 (208MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 07/07/04, BIOS32 rev. 0 @ 0xfb590, SMBIOS
rev. 2.2 @ 0xf (34 entries)
bios0: vendor Phoenix Technologies, LTD version "6.00 PG" date 07/07/2004
apm0 at bios0: Power Management spec V1.2
apm0: AC on, battery charge unknown
pcibios0 at bios0: rev 2.1 @ 0xf/0xdf74
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfdec0/176 (9 entries)
pcibios0: PCI Exclusive IRQs: 3 5 10 11
pcibios0: PCI Interrupt Router at 000:17:0 ("VIA VT82C596A ISA" rev 0x00)
pcibios0: PCI bus #1 is the last bus
bios0: ROM list: 0xc/0xda00 0xd/0x8000!
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "VIA VT8378 PCI" rev 0x00
ppb0 at pci0 dev 1 function 0 "VIA VT8377 AGP" rev 0x00
pci1 at ppb0 bus 1
vga1 at pci1 dev 0 function 0 "VIA VT8378 VGA" rev 0x01: aperture at
0xe400, size 0x1000
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
uhci0 at pci0 dev 16 function 0 "VIA VT83C572 USB" rev 0x80: irq 11
uhci1 at pci0 dev 16 function 1 "VIA VT83C572 USB" rev 0x80: irq 3
uhci2 at pci0 dev 16 function 2 "VIA VT83C572 USB" rev 0x80: irq 10
ehci0 at pci0 dev 16 function 3 "VIA VT6202 USB" rev 0x82: irq 5
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "VIA EHCI root hub" rev 2.00/1.00 addr 1
viapm0 at pci0 dev 17 function 0 "VIA VT8235 ISA" rev 0x00
iic0 at viapm0
spdmem0 at iic0 addr 0x50: 256MB DDR SDRAM non-parity PC3200CL3.0
pciide0 at pci0 dev 17 function 1 "VIA VT82C571 IDE" rev 0x06: ATA133, channel
0 configured to compatibility, channel 1 configured to compatibility
wd0 at pciide0 channel 0 drive 0:
wd0: 16-sector PIO, LBA48, 238475MB, 488397168 sectors
wd1 at pciide0 channel 0 drive 1:
wd1: 16-sector PIO, LBA48, 305245MB, 625142448 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
wd1(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 5
atapiscsi0 at pciide0 channel 1 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: SCSI0 5/cdrom
removable
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2
auvia0 at pci0 dev 17 function 5 "VIA VT8233 AC97" rev 0x50: irq 10
ac97: codec id 0x414c4760 (Avance Logic ALC655 rev 0)
audio0 at auvia0
vr0 at pci0 dev 18 function 0 "VIA RhineII-2" rev 0x74: irq 11, address
00:11:5b:0a:44:14
ukphy0 at vr0 phy 1: Generic IEEE 802.3u media interface, rev. 8: OUI 0x004063,
model 0x0032
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "VIA UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "VIA UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "VIA UHCI root hub" rev 1.00/1.00 addr 1
isa0 at mainbus0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0:
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
it0 at isa0 port 0x290/8: IT87
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask ff6d netmask ff6d ttymask ffef
mtrr: Pentium Pro MTRR support
dkcsum: wd0 matches BIOS drive 0x80
dkcsum: wd1 matches BIOS drive 0x81
root on wd0a swap on wd0b dump on wd0b
WARNING: / was not properly unmounted
Re: massive memory leak in 3.8-stable samba
On Tue, 7 Mar 2006, Steve Fairhead wrote: One of my production machines (3.8-stable) has suddenly started panicing every couple of hours. I found out that the culprit is smbd, eating through memory like there's no tomorrow (approx. 10Mb / minute! ). Can't figure out what has triggered it, nothing changed on the machine lately and there is only one active w2k client, writing a 2.5kB file every 15 seconds or so. I'd be glad of any assistance, even pointing out any stupid mistakes I have made, because this is driving me nuts. I ran into something very similar recently. In my case I eventually discovered that one user was writing to a folder containing 22,000 files. Avoiding this folder has entirely solved the problem. (Or at least worked around it.) FWIW, the Samba logs were helpful only inasmuch as they pointed me to the user who was "causing" the problem. I had to sit down and watch her operate to find out what she was doing... Perhaps (indeed probably) not relevant to your problem, but might give you some ideas. If you're writing a file every 15s, perhaps your problem is related to mine. Steve http://www.fivetrees.com Have a similar problem, though mine appears when moving files between various directories when either has a large number of files in it. Finally decided to delve into the Samba code to see if I could track down the cause. I believe I tracked it to code that scans the directories prior to the file rename operation. The code that scans the directories does some odd things (like hammer on the system lib call: telldir() which appears to leak memory like a sieve if unaccompanied by a matching seekdir() -- approximately 16 bytes per file per directory scanned). This can add up to signifant loss if the directory has, say, 10,000 files or more in it, especially if the directory is scanned multiple times per operation and repeatedly over time. I'm at a bit of a loss how to proceed from here. Given the state and conventions of *BSD-ish directory library calls, Samba isn't scanning directories in a very memory efficient manner -- at least in the case of OpenBSD. But the directory traversals and possible dependencies on the scanning methods could spider badly to fix it properly and reliably within Samba. (ie, I believe that to fix the issue in Samba "properly" is a heck of a lot of work and effort, but I'm also not exactly a Samba expert/developer either) On the other hand, the fact that the system library call telldir() can leak as badly as it does probably isn't a good thing either as outlined here: http://mail-index.netbsd.org/netbsd-bugs/2004/02/05/0008.html It would appear that at least in OpenBSD 3.8-release, the library implementation suffers similar potental issues. I have no idea if the patch proposed in the URL above ever made it into NetBSD, since I don't run NetBSD anywhere; however, the patch looks promising. Changing the implementation of telldir() and related functions would likely fix this particular memory leak in Samba as well, though there may be underlying OS/userland issues about which I am unaware. I guess the bottom line here is that I can see that if you have a process writing into a directory that contains a LOT of files, Samba (or your client, or both) may be scanning the directory prior to any write, and possibly multiple times. If that's the case, the telldir() issue will likely affect you as well. While reducing the number of files in the directory in question won't stop the leak, it may significantly slow it ... I suppose this also assumes that your problem is related to the one I am seeing, and that my preliminary analysis is correct. Hope this helps, - Paul
Re: telldir(), etc: prevent memory leak
On Wed, 22 Mar 2006, Joshua Sandbrook wrote: Just today I found my openbsd server curiously stalled... not completely dead, could switch consoles, ping it.. but otherwise unresponsive. Found out that smbd was eating huge amounts of memory, and I put the crash down to smbd. I applied ( by hand, patch did not work ) the patches Paul wrote.. Yeah. I was having issues with my mail client (silly line wrapping and general corruption). There is a much cleaner patch offered (against -current) on the tech list: http://marc.theaimsgroup.com/?l=openbsd-tech&m=114275248318690&w=2 This one should work if run through patch. Don't know if it will apply against a 3.8 source tree, but I did test to ensure that it worked from the mailing list against -current. If it doesn't apply, please let me know, and I'll be happy to clean it up and try again. and now smbd is behaving in a more sane fashion, though still seems to be very slowly climbing in memory useage, though nothing like it was before. It was climbing at around 1Mb of ram per second before hand, now it is only using 6mb of ram ( started at 2.7 ish mb of ram ) according to top. That's about what I was experiencing, too. Leakage much worse if the directories were highly populated, but still slow leakage for smaller dirs, too. After the patch, some transitory memory usage, but typically it remains stable for me. So either there is still some memory leakage in telldir and friends, or smbd has other memory leaks in it, or this is just normal behaviour for smbd. I concentrated on the biggest culprit in my particular install. I'm quite sure that smbd probably leaks in other places, too, and I may not have caught all places in dirent that may leak if used ... "oddly". Perhaps I should go on another hunt if there is interest in the community. But the patch definately put a stop on the rapid memory consumtion. Glad it was useful to someone! Thanks Paul Most welcome. Hopefully the openbsd guys will provide feedback on it when they find some time, especially if it needs any further work to get it acceptable for inclusion into 3.9. Cheers, - Paul
Re: How to find memory leak in library/OS?
On Thu, 30 Mar 2006, Claus Assmann wrote: Is there some "simple" way to find a memory leak in some OS supplied library? I have a (constantly running) application that grows in a week from 5MB to 15MB in size (VSZ and RSS as reported by ps). The application can be compiled with an optional debugging memory allocator that tracks all (de)allocations to check whether any of its malloc()/free() calls leak memory; according to that tool the application behaves fine. Hence I'm wondering whether there is a memory leak in some library or the OS, which also could be triggered by the way my application uses it (see the recent thread about telldir()/seekdir()). The approach that I used with smbd to find the telldir()/seekdir() leaks wasn't "simple", and it did involve some patience and trial and error, but I'm not sure it was particularly complicated either. Basically, I recompiled smbd to link with the dmalloc library which is designed to augment the system malloc()-type calls. The samba devel team thoughtfully includes developer code to support this -- I simply needed to recompile the app and turn it on. Then I ran the program with the test case that caused it to leak memory and watched the dmalloc output as smbd exitted. This confirmed the leak; dmalloc catches leaks even if the underlying leaking malloc() is buried in a system call -- or at least it did in this case. Unfortunately, dmalloc only reported a stack return address to identify the culprit, so I ended up having to trace through the code and narrow the issue using dmalloc mark and reporting calls. If this is your own code, then this should be significantly easier than tracing samba code -- but I had to go through this step to understand what part of samba was leaking memory. At this point, my assumption was that it was something odd that samba was doing for my particular install. Eventually, once I'd narrowed the leaking code in samba, I ended up attaching to the process using gdb and determing where the return address on the stack was pointing. In my case, that was in the middle of telldir(). If you choose to try dmalloc (dmalloc.com), there are some very nice tutorials on their website for using a debugger to help track memory issues. I also used the internal libc malloc() debug options to help confirm the memory leak, though I wasn't as successful at identifying the leak with it. It did provide another avenue to confirm that the app was leaking. There is test code floating around on the telldir() threads in tech@ that might give you a template for using it, though this may require a recompile of libc to turn on the MALLOC_STATS option. It may be simpler to man malloc to see the easiest method for enabling the memory debugging code buried in libc. Maybe someone else on the list can give some insight into the "dump" results of the malloc() stats to see if there is a way to determine the caller, maybe in conjunction with gbd? So, that's the approach that worked for me. There may be much simpler approaches and/or tools depending on the code you are working on. I'm far from an expert at this ... Good hunting. - Paul My application uses pthreads and the DNS resolver, the latter by contacting it via UDP: sendto(), recvfrom(). Note: the memory leak seems to be unique to OpenBSD (3.8 and earlier), I can't reproduce it on SunOS 5.9 and others. That's why I'm asking for hints where to look for the leak: is there some "simple" way to show the allocated memory in the debugger or via system calls and to find out which functions made those allocations?
Encrypting content/filesystem on DVD?
Hi, This may not be OpenBSD specific, but I'm looking for a way to encrypt the contents of a DVD such that only a user with the correct passphrase would be able to mount the contents. Sort of an optical equivilent to: vnconfig -ck svnd0 my-encrypted-file mount /dev/svnd0c /mount-point My initial thoughts were to simply store an encrypted vnd file filesystem as the only contents of a normal ISO9660 DVD, mount the DVD as always and then attach a vnd device to the file stored on the DVD using vnconfig, as above. Unfortunately, neither mkisofs (and indeed the iso standard) nor growisofs appear to like 4G+ files ... The encrypted content may represent a reasonable large filesystem in one large file under this scheme. My attempts at burning an ffs filesystem to DVD/CDR to get around the filesize limitation of ISO9660 have been largely unsuccessful. See below for details on the (flawed) procedure I initially attempted. I'm sure I'm missing some crucial details -- blocksizes or similar. As an aside, I'm also curious how one might successfully burn an ffs filesystem to a DVD/CD such that OpenBSD can mount it, if such a thing is even possible. The contents only have to be mounted/read via an OpenBSD box. I'm not concerned with interoperability with other architectures or making the disk bootable. I'm not stuck on any particular method of producing the encrypted contents. Using vnd devices with a large file stored on a standard ISO filesystem only seemed like a logical and familiar approach for me and if the size of the file didn't trample ISO's limits, it would have worked fine, I suspect. I'm open to any suggestions on how else this might be most easily accomplished. Regards, - Paul *** cdrw-ffs filesystem procedure -- comments in () *** *** OpenBSD 3.8 GENERIC *** (create a virtual filesystem) # dd if=/dev/zero of=tst.fs bs=1024 count=10240 # vnconfig -c svnd2 tst.fs # newfs -f 2048 /dev/svnd2c newfs: /dev/svnd2c: not a character-special device Warning: cylinder groups must have a multiple of 8 cylinders Warning: 20 sector(s) in last cylinder unallocated /dev/svnd2c:20480 sectors in 205 cylinders of 1 tracks, 100 sectors 10.0MB in 1 cyl groups (208 c/g, 10.16MB/g, 1408 i/g) super-block backups (for fsck -b #) at: 32, (reference) # disklabel svnd2 # /dev/rsvnd2c: type: SCSI disk: vnd device label: fictitious flags: bytes/sector: 512 sectors/track: 100 tracks/cylinder: 1 sectors/cylinder: 100 cylinders: 204 total sectors: 20480 rpm: 3600 interleave: 1 trackskew: 0 cylinderskew: 0 headswitch: 0 # microseconds track-to-track seek: 0 # microseconds drivedata: 0 16 partitions: # sizeoffset fstype [fsize bsize cpg] c: 20480 0 4.2BSD 2048 16384 208 # Cyl 0 - 204* (put something into the ffs image file - tst.fs) # mkdir tstmnt # mount /dev/svnd2c tstmnt # touch tstmnt/hello_world # umount tstmnt # vnconfig -u svnd2 (burn it ...) (Note: cdrecord installed from binary package using pkg_add crdtools-2.01) # cdrecord -v dev=/dev/rcd0c tst.fs cdrecord: No write mode specified. cdrecord: Asuming -tao mode. cdrecord: Future versions of cdrecord may have different drive dependent defaults. cdrecord: Continuing in 5 seconds... Cdrecord-Clone 2.01 (i386-unknown-openbsd3.8) Copyright (C) 1995-2004 Jvrg Schilling TOC Type: 1 = CD-ROM scsidev: '/dev/rcd0c' devname: '/dev/rcd0c' scsibus: -2 target: -2 lun: -2 Using libscg version 'schily-0.8'. SCSI buffer size: 61440 atapi: 0 Device type: Removable CD-ROM Version: 0 Response Format: 2 Capabilities : Vendor_info: 'PIONEER ' Identifikation : 'DVD-RW DVR-106D' Revision : '1.06' Device seems to be: Generic mmc2 DVD-R/DVD-RW. Current: 0x000A Profile: 0x001B Profile: 0x001A Profile: 0x0014 Profile: 0x0013 Profile: 0x0011 Profile: 0x0010 Profile: 0x000A (current) Profile: 0x0009 (current) Profile: 0x0008 cdrecord: This version of cdrecord does not include DVD-R/DVD-RW support code. cdrecord: If you need DVD-R/DVD-RW support, ask the Author for cdrecord-ProDVD. cdrecord: Free test versions and free keys for personal use are at ftp://ftp.berlios.de/pub/cdrecord/ProDVD/ Using generic SCSI-3/mmc CD-R/CD-RW driver (mmc_cdr). Driver flags : MMC-3 SWABAUDIO BURNFREE Supported modes: TAO PACKET SAO SAO/R96P SAO/R96R RAW/R16 RAW/R96P RAW/R96R Drive buf size : 1267712 = 1238 KB FIFO size : 4194304 = 4096 KB Track 01: data10 MB Total size: 11 MB (01:08.29) = 5122 sectors Lout start: 11 MB (01:10/22) = 5122 sectors Current Secsize: 2048 ATIP info from disk: Indicated writing power: 2 Reference speed: 6 Is not unrestricted Is erasable Disk sub type: High speed Rewritable (CAV) media (1) ATIP start of lead in: -11077 (97:34/23) ATIP start of lead out: 336075 (74:43/00) 1T speed low: 4 1T speed high: 10 2T speed low: 2 2T speed high: 10 power mult factor: 2 6 recommended erase/write power: 5 A1 values: 24 2C DC A2 values: 14 A4 4A A3 values:
Re: Encrypting content/filesystem on DVD?
On Thu, 26 Jan 2006, Joachim Schipper wrote: On Wed, Jan 25, 2006 at 10:40:44AM -0500, Paul Thorn wrote: Hi, I'm open to any suggestions on how else this might be most easily accomplished. I don't know about the specific application, but since DVDs are read-only anyway, and encrypted data tends not be accessed that often, is there a good reason not to just pipe tar into gpg? That works very well, and very portably. Joachim I was trying to avoid encrypting individual files on the DVD. Otherwise, to retrieve and use the files involves a local copy. If possible, I'd prefer to access the DVD as a filesystem (even at the cost of speed -- or complexity to mount it) if this is at all possible. While the tar method would work if I split the data into smaller segments, retrieval would be cumbersome at best, I fear. The resulting encrypted tar files would need to be significantly < 4GB for the same reasons that the large vnd filesystem can't be written to the disk (ISO doesn't like these large files). These files may need to be accessed somewhat regularly, so I was looking for some method that is reasonably secure, but not overly cumbersome for someone authorised to access. Thanks for the suggestion, though. I'll probably be able to use something like this for encrypting pure backups that are less likely to be used regularly. - Paul
