Quoting "Jeremy Chadwick" <free...@jdc.parodius.com>:
On Tue, Jul 05, 2011 at 01:03:20PM -0400, Scott Sipe wrote:
I'm running virtualbox 3.2.12_1 if that has anything to do with it.
sysctl vfs.zfs.arc_max: 6200000000
While I'm trying to scp, kstat.zfs.misc.arcstats.size is hovering
right around that value, sometimes above, sometimes below (that's
as it should be, right?). I don't think that it dies when crossing
over arc_max. I can run the same scp 10 times and it might fail 1-3
times, with no correlation to the arcstats.size being above/below
arc_max that I can see.
Scott
On Jul 5, 2011, at 3:00 AM, Peter Ross wrote:
Hi all,
just as an addition: an upgrade to last Friday's FreeBSD-Stable
and to VirtualBox 4.0.8 does not fix the problem.
I will experiment a bit more tomorrow after hours and grab some statistics.
Regards
Peter
Quoting "Peter Ross" <peter.r...@bogen.in-berlin.de>:
Hi all,
I noticed a similar problem last week. It is also very similar to
one reported last year:
http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058708.html
My server is a Dell T410 server with the same bge card (the same
pciconf -lvc output as described by Mahlon:
http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058711.html
Yours, Scott, is a em(4)..
Another similarity: In all cases we are using VirtualBox. I just
want to mention it, in case it matters. I am still running
VirtualBox 3.2.
Most of the time kstat.zfs.misc.arcstats.size was reaching
vfs.zfs.arc_max then, but I could catch one or two cases then the
value was still below.
I added vfs.zfs.prefetch_disable=1 to sysctl.conf but it does not help.
BTW: It looks as ARC only gives back the memory when I destroy
the ZFS (a cloned snapshot containing virtual machines). Even if
nothing happens for hours the buffer isn't released..
My machine was still running 8.2-PRERELEASE so I am upgrading.
I am happy to give information gathered on old/new kernel if it helps.
Regards
Peter
Quoting "Scott Sipe" <csco...@gmail.com>:
On Jul 2, 2011, at 12:54 AM, jhell wrote:
On Fri, Jul 01, 2011 at 03:22:32PM -0700, Jeremy Chadwick wrote:
On Fri, Jul 01, 2011 at 03:13:17PM -0400, Scott Sipe wrote:
I'm running 8.2-RELEASE and am having new problems with scp.
When scping
files to a ZFS directory on the FreeBSD server -- most
notably large files
-- the transfer frequently dies after just a few seconds. In
my last test, I
tried to scp an 800mb file to the FreeBSD system and the
transfer died after
200mb. It completely copied the next 4 times I tried, and
then died again on
the next attempt.
On the client side:
"Connection to home closed by remote host.
lost connection"
In /var/log/auth.log:
Jul 1 14:54:42 freebsd sshd[18955]: fatal: Write failed:
Cannot allocate
memory
I've never seen this before and have used scp before to
transfer large files
without problems. This computer has been used in production
for months and
has a current uptime of 36 days. I have not been able to
notice any problems
copying files to the server via samba or netatalk, or any problems in
apache.
Uname:
FreeBSD xeon 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Sat Feb 19
01:02:54 EST
2011 root@xeon:/usr/obj/usr/src/sys/GENERIC amd64
I've attached my dmesg and output of vmstat -z.
I have not restarted the sshd daemon or rebooted the computer.
Am glad to provide any other information or test anything else.
{snip vmstat -z and dmesg}
You didn't provide details about your networking setup (rc.conf,
ifconfig -a, etc.). netstat -m would be useful too.
Next, please see this thread circa September 2010, titled "Network
memory allocation failures":
http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/thread.html#58708
The user in that thread is using rsync, which relies on scp by default.
I believe this problem is similar, if not identical, to yours.
Please also provide your output of ( /usr/bin/limits -a ) for the server
end and the client.
I am not quite sure I agree with the need for ifconfig -a but some
information about the networking driver your using for the interface
would be helpful, uptime of the boxes. And configuration of the pool.
e.g. ( zpool status -a ;zfs get all <poolname> ) You should probably
prop this information up somewhere so you can reference by URL whenever
needed.
rsync(1) does not rely on scp(1) whatsoever but rsync(1) can be made to
use ssh(1) instead of rsh(1) and I believe that is what Jeremy is
stating here but correct me if I am wrong. It does use ssh(1) by
default.
Its a possiblity as well that if using tmpfs(5) or mdmfs(8) for /tmp
type filesystems that rsync(1) may be just filling up your temp ram area
and causing the connection abort which would be expected. ( df
-h ) would
help here.
Hello,
I'm not using tmpfs/mdmfs at all. The clients yesterday were 3
different OSX computers (over gigabit). The FreeBSD server has
12gb of ram and no bce adapter. For what it's worth, the server
is backed up remotely every night with rsync (remote FreeBSD
uses rsync to pull) to an offsite (slow cable connection)
FreeBSD computer, and I have not seen any errors in the nightly
rsync.
Sorry for the omission of networking info, here's the output of
the requested commands and some that popped up in the other
thread:
http://www.cap-press.com/misc/
In rc.conf: ifconfig_em1="inet 10.1.1.1 netmask 255.255.0.0"
Scott
Just to make it crystal clear to everyone:
There is no correlation between this problem and use of ZFS. People are
attempting to correlate "cannot allocate memory" messages with "anything
on the system that uses memory". The VM is much more complex than that.
Given the nature of this problem, it's much more likely the issue is
"somewhere" within a networking layer within FreeBSD, whether it be
driver-level or some sort of intermediary layer.
Two people who have this issue in this thread are both using VirtualBox.
Can one, or both, of you remove VirtualBox from the configuration
entirely (kernel, etc. -- not sure what is required) and then see if the
issue goes away?
On the machine in question I only can do it after hours so I will do
it tonight.
I was _successfully_ sending the file over the loopback interface using
cat /zpool/temp/zimbra_oldroot.vdi | ssh localhost "cat > /dev/null"
I did it, btw, with the IPv6 localhost address first (accidently), and
then using IPv4. Both worked.
It always fails if I am sending it through the bce(4) interface, even
if my target is the VirtualBox bridged to the bce card (so it does not
"leave" the computer physically).
Below the uname -a, ifconfig -a, netstat -rn, pciconf -lv and kldstat output.
I have another box where I do not see that problem. It copies files
happily over the net using ssh.
It is an an older HP ML 150 with 3GB RAM only but with a bge(4) driver
instead. It runs the same last week's RELENG_8. I installed VirtualBox
and enabled vboxnet (so it loads the kernel modules). But I do not run
VirtualBox on it (because it hasn't enough RAM).
Regards
Peter
DellT410one# uname -a
FreeBSD DellT410one.vv.fda 8.2-STABLE FreeBSD 8.2-STABLE #1: Thu Jun
30 17:07:18 EST 2011
r...@dellt410one.vv.fda:/usr/obj/usr/src/sys/GENERIC amd64
DellT410one# ifconfig -a
bce0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 1500
options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 84:2b:2b:68:64:e4
inet 192.168.50.220 netmask 0xffffff00 broadcast 192.168.50.255
inet 192.168.50.221 netmask 0xffffff00 broadcast 192.168.50.255
inet 192.168.50.223 netmask 0xffffff00 broadcast 192.168.50.255
inet 192.168.50.224 netmask 0xffffff00 broadcast 192.168.50.255
inet 192.168.50.225 netmask 0xffffff00 broadcast 192.168.50.255
inet 192.168.50.226 netmask 0xffffff00 broadcast 192.168.50.255
inet 192.168.50.227 netmask 0xffffff00 broadcast 192.168.50.255
inet 192.168.50.219 netmask 0xffffff00 broadcast 192.168.50.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
bce1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 84:2b:2b:68:64:e5
media: Ethernet autoselect
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
vboxnet0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 0a:00:27:00:00:00
DellT410one# netstat -rn
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 192.168.50.201 UGS 0 52195 bce0
127.0.0.1 link#11 UH 0 6 lo0
192.168.50.0/24 link#1 U 0 1118212 bce0
192.168.50.219 link#1 UHS 0 9670 lo0
192.168.50.220 link#1 UHS 0 8347 lo0
192.168.50.221 link#1 UHS 0 103024 lo0
192.168.50.223 link#1 UHS 0 43614 lo0
192.168.50.224 link#1 UHS 0 8358 lo0
192.168.50.225 link#1 UHS 0 8438 lo0
192.168.50.226 link#1 UHS 0 8338 lo0
192.168.50.227 link#1 UHS 0 8333 lo0
192.168.165.0/24 192.168.50.200 UGS 0 3311 bce0
192.168.166.0/24 192.168.50.200 UGS 0 699 bce0
192.168.167.0/24 192.168.50.200 UGS 0 3012 bce0
192.168.168.0/24 192.168.50.200 UGS 0 552 bce0
Internet6:
Destination Gateway Flags
Netif Expire
::1 ::1 UH
lo0
fe80::%lo0/64 link#11 U
lo0
fe80::1%lo0 link#11 UHS
lo0
ff01::%lo0/32 fe80::1%lo0 U
lo0
ff02::%lo0/32 fe80::1%lo0 U
lo0
DellT410one# kldstat
Id Refs Address Size Name
1 19 0xffffffff80100000 dbf5d0 kernel
2 3 0xffffffff80ec0000 4c358 vboxdrv.ko
3 1 0xffffffff81012000 131998 zfs.ko
4 1 0xffffffff81144000 1ff1 opensolaris.ko
5 2 0xffffffff81146000 2940 vboxnetflt.ko
6 2 0xffffffff81149000 8e38 netgraph.ko
7 1 0xffffffff81152000 153c ng_ether.ko
8 1 0xffffffff81154000 e70 vboxnetadp.ko
DellT410one# pciconf -lv
..
bce0@pci0:1:0:0: class=0x020000 card=0x028d1028 chip=0x163b14e4
rev=0x20 hdr=0x00
vendor = 'Broadcom Corporation'
class = network
subclass = ethernet
bce1@pci0:1:0:1: class=0x020000 card=0x028d1028 chip=0x163b14e4
rev=0x20 hdr=0x00
vendor = 'Broadcom Corporation'
class = network
subclass = ethernet
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
