From owner-freebsd-questions@FreeBSD.ORG Tue Mar 27 02:32:32 2012 Return-Path: Delivered-To: questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6013F106564A; Tue, 27 Mar 2012 02:32:32 +0000 (UTC) (envelope-from pgollucci@gmail.com) Received: from mail-qa0-f54.google.com (mail-qa0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id F1F3C8FC0A; Tue, 27 Mar 2012 02:32:31 +0000 (UTC) Received: by qao25 with SMTP id 25so2794276qao.13 for ; Mon, 26 Mar 2012 19:32:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:organization:user-agent:mime-version:to:cc :subject:references:in-reply-to:x-enigmail-version:content-type; bh=//mcH9VPSNAWrCDfJfAaw5HPTracc11XLO1MQBtwj8c=; b=EVTVXVgAtg5KO/MH3Ie+Tauf3fIXs0O9ffboiz7Fd+siSZ7dGR3CTqbUy1kt9gMc82 zw9005pnMfFJw7WJ8oj1iDhNguVWsQ731+eVGjQys/KT4cP2Os7G4nto0/1PMwAS1FU3 z8b1HW0TBUaiJwgljwYB5dTZHeYZsM9exV51WqT40aeTCdAOLkqfE4IdzvruNz6ykuPp AMHfPIf6g0j+ByIM+G75azRpa4LDxPRuPYhuwRu/Rd1PoCpYWhgxhJscfXog+Ma3NRRC sSNnas34pugIeby2XZCXfg5i67IQbUdkj4XtihCDFICGjVHs+HDazcWaHkXxQ1pyo1Fv j8Ew== Received: by 10.224.218.195 with SMTP id hr3mr30526561qab.26.1332815551484; Mon, 26 Mar 2012 19:32:31 -0700 (PDT) Received: from philip.hq.rws (wsip-174-79-184-239.dc.dc.cox.net. [174.79.184.239]) by mx.google.com with ESMTPS id 1sm33304848qac.3.2012.03.26.19.32.29 (version=SSLv3 cipher=OTHER); Mon, 26 Mar 2012 19:32:30 -0700 (PDT) Message-ID: <4F7126BA.1000202@p6m7g8.com> Date: Tue, 27 Mar 2012 02:32:26 +0000 From: "Philip M. Gollucci" Organization: P6M7G8 Inc. User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111029 Thunderbird/7.0.1 MIME-Version: 1.0 To: "Philip M. Gollucci" References: <4F708223.4060803@p6m7g8.com> In-Reply-To: <4F708223.4060803@p6m7g8.com> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigC256B7493D0030294755CCCD" Cc: questions@freebsd.org, current@freebsd.org Subject: Re: freebsd 9.0-release + zfs + mysqld(percona) = kernel: swap zone exhausted, increase kern.maxswzone X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Mar 2012 02:32:32 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigC256B7493D0030294755CCCD Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Some other tuning updates $ zfs set zfs:zfs_nocacheflush =3D 1 $ sysctl vfs.zfs.prefetch_disable=3D1 $ cat /etc/my.cnf skip-innodb-doublewrite innodb_flush_log_at_trx_commit=3D2 $ zfs set primarycache=3Dmetadata zmysqlD $ zfs set atime=3Doff zmysqlD $ zfs set recordsize=3D16k zmysqlD but not on zmysqlL my next plan is to turn off tmpfs and use ZVOL swaps then to simply use just zroot/tmp as a normal dir. after that I'll drastically increase maxswzone. still hoping someone has already done this. On 03/26/12 14:50, Philip M. Gollucci wrote: > /var/log/messages > Mar 23 22:21:50 sabertooth kernel: swap zone exhausted, increase > kern.maxswzone > Mar 23 22:21:50 sabertooth kernel: pid 86697 (mysqld), uid 88, was > killed: out of swap space >=20 > how to repeat: > $ mysql -ux < file.sql (~150GB) worth >=20 > basically, it slows down continually until it dies. IF you (suspend) > the process in time it recovers some, but eventually you have to suspen= d > it every 1s for ~3 minutes. The load is ~10 at this point. >=20 > I've looked at top, ps, iostat, zpool iostat, vmstat -z, vmstat -m > and I don't see anything wonky. I can provide more info on request. >=20 > system description: >=20 > $ df > zmysqlD 801G 658G 142G 82% /var/db/mysql/data > zmysqlL 133G 26G 107G 20% /var/db/mysql/log >=20 > its a 600GB innodb space, mysql has > innodb_buffer_pool_size =3D 80GB > about 1GB of data is MyISAM the rest is InnoDB >=20 > The machine has 96GB of RAM >=20 > $ cat /etc/fstab > /dev/gpt/swap0 none swap sw 0 0 > /dev/gpt/swap1 none swap sw 0 0 >=20 > tmpfs /tmp tmpfs rw 2 0 >=20 > swapinfo -h will show %6 and %6 usage on the swap devices > /tmp remains < 5% used >=20 > $ grep maxswzone /boot/loader.conf > kern.maxswzone=3D"67108864" ## double the default >=20 > $ gpart show > =3D> 34 286749421 da3 GPT (136G) > 34 128 1 freebsd-boot (64k) > 162 201326592 2 freebsd-swap (96G) > 201326754 85422701 3 freebsd-zfs (40G) >=20 > =3D> 34 286749421 da4 GPT (136G) > 34 128 1 freebsd-boot (64k) > 162 201326592 2 freebsd-swap (96G) > 201326754 85422701 3 freebsd-zfs (40G) >=20 > da[012] are SSDs, the rest are 15krpm >=20 > $ zpool status > pool: zmysqlD > state: ONLINE > scan: none requested > config: >=20 > NAME STATE READ WRITE CKSUM > zmysqlD ONLINE 0 0 0 > raidz2-0 ONLINE 0 0 0 > da7 ONLINE 0 0 0 > da8 ONLINE 0 0 0 > da9 ONLINE 0 0 0 > da10 ONLINE 0 0 0 > da11 ONLINE 0 0 0 > da12 ONLINE 0 0 0 > da13 ONLINE 0 0 0 > da14 ONLINE 0 0 0 > logs > da0 ONLINE 0 0 0 > cache > da2 ONLINE 0 0 0 >=20 > errors: No known data errors >=20 > pool: zmysqlL > state: ONLINE > scan: none requested > config: >=20 > NAME STATE READ WRITE CKSUM > zmysqlL ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > da5 ONLINE 0 0 0 > da6 ONLINE 0 0 0 > cache > da1 ONLINE 0 0 0 >=20 > errors: No known data errors >=20 > pool: zroot > state: ONLINE > scan: none requested > config: >=20 > NAME STATE READ WRITE CKSUM > zroot ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > da3p3 ONLINE 0 0 0 > da4p3 ONLINE 0 0 0 >=20 >=20 >=20 --=20 ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci@p6m7g8.com) c: 703.336.9354 Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Director Operations, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. --------------enigC256B7493D0030294755CCCD Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFPcSa8dbiP+9ubjBwRAgn+AJ467lTaFaaQs9OaV3fJnb1i25+1BACcCVh4 b8MyZzmS7/aGS8Zdh9+sHA8= =knaH -----END PGP SIGNATURE----- --------------enigC256B7493D0030294755CCCD--