From owner-freebsd-questions@FreeBSD.ORG Tue Dec 17 05:09:08 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C8F5E4E0 for ; Tue, 17 Dec 2013 05:09:08 +0000 (UTC) Received: from mail.aprec.ru (mail.aprec.ru [91.210.204.10]) by mx1.freebsd.org (Postfix) with ESMTP id 072251A12 for ; Tue, 17 Dec 2013 05:09:07 +0000 (UTC) Received: by mail.aprec.ru (Postfix, from userid 1002) id B47B81300878; Tue, 17 Dec 2013 09:03:08 +0400 (MSK) X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail.aprec.ru X-Spam-Level: X-Spam-Status: No, score=0.3 required=9.5 tests=ALL_TRUSTED,BAYES_50 autolearn=disabled version=3.2.5 Received: from [192.168.1.17] (office.relant.ru [91.210.204.169]) by mail.aprec.ru (Postfix) with ESMTPA id 666051300872 for ; Tue, 17 Dec 2013 09:03:08 +0400 (MSK) Message-ID: <52AFDB0B.3020603@aprec.ru> Date: Tue, 17 Dec 2013 09:03:07 +0400 From: =?UTF-8?B?0JHQtdC70L7Qs9GD0YDQvtCyINCh0LLRj9GC0L7RgdC70LDQsg==?= User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: freebsd-questions@freebsd.org Subject: Problem with amanda backup (cannot allocate memory) on FreeBSD-9.1-RELEASE amd64 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Dec 2013 05:09:08 -0000 Hello everyone. I've posted this problem on forums.zmanda.com, but without any luck. I think, maybe it's something FreeBSD-specific, or specific to it's version. So, here it is, any help will be greatly appreciated. I have a backup server with amanda-2.4.4p3, which is backup'ing numerous servers with different FreeBSD versions (from 6.x to 8.x, i386, amd64), some Linux machines. Everything works fine. Recently I've upgraded some servers to FreeBSD 9.1-amd64. I've installed amanda-2.4.4p3 on these servers, but when I tried to do backups, they complete with errors, on disk volumes that are larger than 500-1000mb approximately. The error message on client machines was: sendbackup: critical (fatal): index tee cannot write [Broken pipe] I decided, that I'm using too old version of amanda, and it's time to upgrade. I've installed amanda 3.3.4 to our backup server (as an another amanda version, alternative to working one), and to FreeBSD 9.1 clients. I've also installed GNU tar 1.27 on my clients. But, this does not help. I still receive errors on volumes that are larger than 500-1000mb, and backup does not complete. I only get error on /usr volume on problem host, / and /var are backuping normally. I suppose that's because of their size (/ - 740M used, /var - 42M used, /usr - 4.7G used). When I use disklist to exclude some directories on /usr volume (that are using most of it's space) - backup also completes normally. In amgtar.*.debug on client: Sat Dec 7 14:39:28 2013: thd-0x802c170c0: amgtar: Spawning "/usr/local/gnutar-1.27/bin/tar /usr/local/gnutar-1.27/bin/tar --create --verbose --file - --directory /usr --one-file-system --listed-incremental /usr/local/amanda-3.3.4/var/amanda/gnutar-lists/problem-host_usr_0.new --sparse --ignore-failed-read --totals ." in pipeline Sat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: 84: size(|): Total bytes written: 60026880 (58MiB, ?/s) Sat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: 0: strange(?): /usr/local/gnutar-1.27/bin/tar: -: Cannot write: Broken pipe Sat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: 0: strange(?): /usr/local/gnutar-1.27/bin/tar: Error is not recoverable: exiting now Sat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: after /usr/local/gnutar-1.27/bin/tar /usr wait Sat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: amgtar: /usr/local/gnutar-1.27/bin/tar: pid 19987 Sat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: /usr/local/gnutar-1.27/bin/tar exited with status 2: see /tmp/amanda/client/org/amgtar.20131207143928000.debugSat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: sendbackup: size 58620 Sat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: sendbackup: end In amandad.*.debug on clients I get strange error "Cannot allocate memory": Sat Dec 7 14:39:33 2013: thd-0x802c17100: amandad: opening security stream for fd 50 Sat Dec 7 14:39:33 2013: thd-0x802c17100: amandad: opening security stream for fd 51 Sat Dec 7 14:39:33 2013: thd-0x802c17100: amandad: opening security stream for fd 52 Sat Dec 7 14:39:33 2013: thd-0x802c17100: amandad: security_close(handle=0x802ca3280, driver=0x800c83680 (BSDTCP)) Sat Dec 7 14:39:33 2013: thd-0x802c17100: amandad: security_stream_close(0x802d23000) Sat Dec 7 14:39:44 2013: thd-0x802c17100: amandad: tcpm_send_token: data is still flowing Sat Dec 7 14:39:52 2013: thd-0x802c17100: amandad: security_stream_seterr(0x802d35000, write error to: Cannot allocate memory) Sat Dec 7 14:39:52 2013: thd-0x802c17100: amandad: sending NAK pkt: <<<<< ERROR write error on stream 499999: write error to: Cannot allocate memory >>>>> Sat Dec 7 14:39:52 2013: thd-0x802c17100: amandad: security_stream_close(0x802d35000) Sat Dec 7 14:39:52 2013: thd-0x802c17100: amandad: security_stream_close(0x802d3e000) Sat Dec 7 14:39:52 2013: thd-0x802c17100: amandad: security_stream_close(0x802d47000) Sat Dec 7 14:39:53 2013: thd-0x802c17100: amandad: pid 19988 finish time Sat Dec 7 14:39:53 2013 Strange, but that's not explicitly amanda problem. When I use SSH to login to problem machine's console, everything works OK. But when I start to compile some program package from source, or rebuild kernel (when there is much output from "make" to the console), sometimes SSH connection breaks, and when I connect again, I can see message from sshd in logs: "cannot allocate memory" So, maybe it's some bug with memory allocation on my FreeBSD box? How can I check this, or, maybe, someone had already encountered anything similar? -- С уважением, Белогуров Святослав http://relant.ru