Date: Tue, 5 Jul 2011 13:03:20 -0400 From: Scott Sipe <cscotts@gmail.com> To: Peter Ross <Peter.Ross@bogen.in-berlin.de> Cc: freebsd-stable List <freebsd-stable@freebsd.org>, Jeremy Chadwick <freebsd@jdc.parodius.com> Subject: Re: scp: Write Failed: Cannot allocate memory Message-ID: <5BA0546B-DE70-4753-B445-551FD5336787@gmail.com> In-Reply-To: <20110705170023.87183xz6ehxbeft3@webmail.in-berlin.de> References: <BANLkTinGV6wBvVGyA2PjZ9fnvYt5hKsLOA@mail.gmail.com> <20110701222232.GA33935@icarus.home.lan> <20110702045435.GA81502@DataIX.net> <54D65EC5-9A9B-4F96-BB45-1904F2147CBA@gmail.com> <20110704100845.94513n3znbabpthp@webmail.in-berlin.de> <20110705170023.87183xz6ehxbeft3@webmail.in-berlin.de>
next in thread | previous in thread | raw e-mail | index | archive | help
I'm running virtualbox 3.2.12_1 if that has anything to do with it. sysctl vfs.zfs.arc_max: 6200000000 While I'm trying to scp, kstat.zfs.misc.arcstats.size is hovering right = around that value, sometimes above, sometimes below (that's as it should = be, right?). I don't think that it dies when crossing over arc_max. I = can run the same scp 10 times and it might fail 1-3 times, with no = correlation to the arcstats.size being above/below arc_max that I can = see. Scott On Jul 5, 2011, at 3:00 AM, Peter Ross wrote: > Hi all, >=20 > just as an addition: an upgrade to last Friday's FreeBSD-Stable and to = VirtualBox 4.0.8 does not fix the problem. >=20 > I will experiment a bit more tomorrow after hours and grab some = statistics. >=20 > Regards > Peter >=20 > Quoting "Peter Ross" <Peter.Ross@bogen.in-berlin.de>: >=20 >> Hi all, >>=20 >> I noticed a similar problem last week. It is also very similar to one = reported last year: >>=20 >> = http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058708.ht= ml >>=20 >> My server is a Dell T410 server with the same bge card (the same = pciconf -lvc output as described by Mahlon: >>=20 >> = http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058711.ht= ml >>=20 >> Yours, Scott, is a em(4).. >>=20 >> Another similarity: In all cases we are using VirtualBox. I just want = to mention it, in case it matters. I am still running VirtualBox 3.2. >>=20 >> Most of the time kstat.zfs.misc.arcstats.size was reaching = vfs.zfs.arc_max then, but I could catch one or two cases then the value = was still below. >>=20 >> I added vfs.zfs.prefetch_disable=3D1 to sysctl.conf but it does not = help. >>=20 >> BTW: It looks as ARC only gives back the memory when I destroy the = ZFS (a cloned snapshot containing virtual machines). Even if nothing = happens for hours the buffer isn't released.. >>=20 >> My machine was still running 8.2-PRERELEASE so I am upgrading. >>=20 >> I am happy to give information gathered on old/new kernel if it = helps. >>=20 >> Regards >> Peter >>=20 >> Quoting "Scott Sipe" <cscotts@gmail.com>: >>=20 >>>=20 >>> On Jul 2, 2011, at 12:54 AM, jhell wrote: >>>=20 >>>> On Fri, Jul 01, 2011 at 03:22:32PM -0700, Jeremy Chadwick wrote: >>>>> On Fri, Jul 01, 2011 at 03:13:17PM -0400, Scott Sipe wrote: >>>>>> I'm running 8.2-RELEASE and am having new problems with scp. When = scping >>>>>> files to a ZFS directory on the FreeBSD server -- most notably = large files >>>>>> -- the transfer frequently dies after just a few seconds. In my = last test, I >>>>>> tried to scp an 800mb file to the FreeBSD system and the transfer = died after >>>>>> 200mb. It completely copied the next 4 times I tried, and then = died again on >>>>>> the next attempt. >>>>>>=20 >>>>>> On the client side: >>>>>>=20 >>>>>> "Connection to home closed by remote host. >>>>>> lost connection" >>>>>>=20 >>>>>> In /var/log/auth.log: >>>>>>=20 >>>>>> Jul 1 14:54:42 freebsd sshd[18955]: fatal: Write failed: Cannot = allocate >>>>>> memory >>>>>>=20 >>>>>> I've never seen this before and have used scp before to transfer = large files >>>>>> without problems. This computer has been used in production for = months and >>>>>> has a current uptime of 36 days. I have not been able to notice = any problems >>>>>> copying files to the server via samba or netatalk, or any = problems in >>>>>> apache. >>>>>>=20 >>>>>> Uname: >>>>>>=20 >>>>>> FreeBSD xeon 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Sat Feb 19 = 01:02:54 EST >>>>>> 2011 root@xeon:/usr/obj/usr/src/sys/GENERIC amd64 >>>>>>=20 >>>>>> I've attached my dmesg and output of vmstat -z. >>>>>>=20 >>>>>> I have not restarted the sshd daemon or rebooted the computer. >>>>>>=20 >>>>>> Am glad to provide any other information or test anything else. >>>>>>=20 >>>>>> {snip vmstat -z and dmesg} >>>>>=20 >>>>> You didn't provide details about your networking setup (rc.conf, >>>>> ifconfig -a, etc.). netstat -m would be useful too. >>>>>=20 >>>>> Next, please see this thread circa September 2010, titled "Network >>>>> memory allocation failures": >>>>>=20 >>>>> = http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/thread.ht= ml#58708 >>>>>=20 >>>>> The user in that thread is using rsync, which relies on scp by = default. >>>>> I believe this problem is similar, if not identical, to yours. >>>>>=20 >>>>=20 >>>> Please also provide your output of ( /usr/bin/limits -a ) for the = server >>>> end and the client. >>>>=20 >>>> I am not quite sure I agree with the need for ifconfig -a but some >>>> information about the networking driver your using for the = interface >>>> would be helpful, uptime of the boxes. And configuration of the = pool. >>>> e.g. ( zpool status -a ;zfs get all <poolname> ) You should = probably >>>> prop this information up somewhere so you can reference by URL = whenever >>>> needed. >>>>=20 >>>> rsync(1) does not rely on scp(1) whatsoever but rsync(1) can be = made to >>>> use ssh(1) instead of rsh(1) and I believe that is what Jeremy is >>>> stating here but correct me if I am wrong. It does use ssh(1) by >>>> default. >>>>=20 >>>> Its a possiblity as well that if using tmpfs(5) or mdmfs(8) for = /tmp >>>> type filesystems that rsync(1) may be just filling up your temp ram = area >>>> and causing the connection abort which would be expected. ( df -h ) = would >>>> help here. >>>=20 >>> Hello, >>>=20 >>> I'm not using tmpfs/mdmfs at all. The clients yesterday were 3 = different OSX computers (over gigabit). The FreeBSD server has 12gb of = ram and no bce adapter. For what it's worth, the server is backed up = remotely every night with rsync (remote FreeBSD uses rsync to pull) to = an offsite (slow cable connection) FreeBSD computer, and I have not seen = any errors in the nightly rsync. >>>=20 >>> Sorry for the omission of networking info, here's the output of the = requested commands and some that popped up in the other thread: >>>=20 >>> http://www.cap-press.com/misc/ >>>=20 >>> In rc.conf: ifconfig_em1=3D"inet 10.1.1.1 netmask 255.255.0.0" >>>=20 >>> Scott >>>=20 >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >>>=20 >>=20 >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >>=20 >=20 >=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5BA0546B-DE70-4753-B445-551FD5336787>