From owner-freebsd-questions@FreeBSD.ORG Mon Mar 26 14:50:22 2012 Return-Path: Delivered-To: questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 119DA106566C; Mon, 26 Mar 2012 14:50:22 +0000 (UTC) (envelope-from pgollucci@gmail.com) Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id A56668FC2B; Mon, 26 Mar 2012 14:50:21 +0000 (UTC) Received: by qcsg15 with SMTP id g15so3928601qcs.13 for ; Mon, 26 Mar 2012 07:50:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:organization:user-agent:mime-version:to:cc :subject:x-enigmail-version:content-type; bh=nytDcmzMKZyfSI7vuge79swmh8U1yKrOkb4SJHFs9ew=; b=bnK+6dkKPPsFbmlSNzyv5a8Q5xi5F5gBr5DD+4KS1dmN+d2b3OpTaKtNQL6AxAQrro S4rnxzDekfuQHFlscm6/L/oRFk0eoj5Q/XtaDSTT+NgmVulzkNSAHDfdI9OnyXaBHGkX dtocT+vC1dsvw6k70ppt2JtQ24733d+TfnHa4PZOlgzPrZ5PtNM9bt1Hzf/QiD4GqkuP 0KOmny11yW+oBLzHLyo2Y3c7wLIMl0VAyYx5Zta4VjPFUXo0Z9idsblttq4bWxU4YfK1 oDFZGGfkHaXAqTnzT+GWMqBiasg3FtGfpNtq+o0wePpel/2H4D4W+w/sGcl1Gfcn3TeD d6Kg== Received: by 10.229.134.194 with SMTP id k2mr851949qct.125.1332773417247; Mon, 26 Mar 2012 07:50:17 -0700 (PDT) Received: from philip.hq.rws (wsip-174-79-184-239.dc.dc.cox.net. [174.79.184.239]) by mx.google.com with ESMTPS id j17sm30283659qaj.9.2012.03.26.07.50.15 (version=SSLv3 cipher=OTHER); Mon, 26 Mar 2012 07:50:15 -0700 (PDT) Message-ID: <4F708223.4060803@p6m7g8.com> Date: Mon, 26 Mar 2012 14:50:11 +0000 From: "Philip M. Gollucci" Organization: P6M7G8 Inc. User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111029 Thunderbird/7.0.1 MIME-Version: 1.0 To: questions@FreeBSD.org X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig8E80A1D254E6E3D78A7CAEFD" Cc: current@FreeBSD.org Subject: freebsd 9.0-release + zfs + mysqld(percona) = kernel: swap zone exhausted, increase kern.maxswzone X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2012 14:50:22 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig8E80A1D254E6E3D78A7CAEFD Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable /var/log/messages Mar 23 22:21:50 sabertooth kernel: swap zone exhausted, increase kern.maxswzone Mar 23 22:21:50 sabertooth kernel: pid 86697 (mysqld), uid 88, was killed: out of swap space how to repeat: $ mysql -ux < file.sql (~150GB) worth basically, it slows down continually until it dies. IF you (suspend) the process in time it recovers some, but eventually you have to suspend it every 1s for ~3 minutes. The load is ~10 at this point. I've looked at top, ps, iostat, zpool iostat, vmstat -z, vmstat -m and I don't see anything wonky. I can provide more info on request. system description: $ df zmysqlD 801G 658G 142G 82% /var/db/mysql/data zmysqlL 133G 26G 107G 20% /var/db/mysql/log its a 600GB innodb space, mysql has innodb_buffer_pool_size =3D 80GB about 1GB of data is MyISAM the rest is InnoDB The machine has 96GB of RAM $ cat /etc/fstab /dev/gpt/swap0 none swap sw 0 0 /dev/gpt/swap1 none swap sw 0 0 tmpfs /tmp tmpfs rw 2 0 swapinfo -h will show %6 and %6 usage on the swap devices /tmp remains < 5% used $ grep maxswzone /boot/loader.conf kern.maxswzone=3D"67108864" ## double the default $ gpart show =3D> 34 286749421 da3 GPT (136G) 34 128 1 freebsd-boot (64k) 162 201326592 2 freebsd-swap (96G) 201326754 85422701 3 freebsd-zfs (40G) =3D> 34 286749421 da4 GPT (136G) 34 128 1 freebsd-boot (64k) 162 201326592 2 freebsd-swap (96G) 201326754 85422701 3 freebsd-zfs (40G) da[012] are SSDs, the rest are 15krpm $ zpool status pool: zmysqlD state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zmysqlD ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 da7 ONLINE 0 0 0 da8 ONLINE 0 0 0 da9 ONLINE 0 0 0 da10 ONLINE 0 0 0 da11 ONLINE 0 0 0 da12 ONLINE 0 0 0 da13 ONLINE 0 0 0 da14 ONLINE 0 0 0 logs da0 ONLINE 0 0 0 cache da2 ONLINE 0 0 0 errors: No known data errors pool: zmysqlL state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zmysqlL ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 cache da1 ONLINE 0 0 0 errors: No known data errors pool: zroot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da3p3 ONLINE 0 0 0 da4p3 ONLINE 0 0 0 --=20 ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci@p6m7g8.com) c: 703.336.9354 Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Director Operations, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. --------------enig8E80A1D254E6E3D78A7CAEFD Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFPcIImdbiP+9ubjBwRArTUAJ9Ui2PEGq0BKIYIb5oZ03o4Gz1wxwCcChj2 x8hXjDSpox+q8isz+WJFCDs= =maYA -----END PGP SIGNATURE----- --------------enig8E80A1D254E6E3D78A7CAEFD--