From owner-freebsd-stable@freebsd.org Fri Dec 11 12:46:56 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15E3A9D7CD5 for ; Fri, 11 Dec 2015 12:46:56 +0000 (UTC) (envelope-from karl@denninger.net) Received: from mail.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D7815174D for ; Fri, 11 Dec 2015 12:46:55 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (Karl-Desktop.Denninger.net [192.168.1.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.denninger.net (Postfix) with ESMTPSA id 1524C2352A5 for ; Fri, 11 Dec 2015 06:46:48 -0600 (CST) Subject: Re: Something in r291926 (and earlier) causes reboots during periodic daily (14 jails on system) To: freebsd-stable@freebsd.org References: <20151209090049.000003db@Leidinger.net> <5667E86F.3050906@quip.cz> <20151209105144.000035be@Leidinger.net> <3923D6D6-E907-4B6C-931A-12CD7B863415@ebureau.com> <20151210120326.00005928@Leidinger.net> <56697E1F.6050604@denninger.net> <20151211085642.0000652e@Leidinger.net> From: Karl Denninger Message-ID: <566AC592.80106@denninger.net> Date: Fri, 11 Dec 2015 06:46:10 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <20151211085642.0000652e@Leidinger.net> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms040204030602080901050608" X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Dec 2015 12:46:56 -0000 This is a cryptographically signed message in MIME format. --------------ms040204030602080901050608 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable That's sorta what I expected..... First thing I'd try -- turn off UMA for ZFS. It will have a performance impact but it eliminates one of the issues that I had to do a lot of work to get around in the patch I developed. In /boot/loader.conf: vfs.zfs.zio.use_uma=3D0 And reboot. On 12/11/2015 01:56, Alexander Leidinger wrote: > On Thu, 10 Dec 2015 07:29:03 -0600 > Karl Denninger wrote: > >> On 12/10/2015 05:03, Alexander Leidinger wrote: >>> On Wed, 9 Dec 2015 15:15:10 -0600 >>> Dustin Wenz wrote: >>> =20 >>>> Are you by chance using the ARC patch from this PR? >>>> >>>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D187594 =20 >>> No (despite the fact that I would like to have it hit the tree). >>> =20 >>>> Do you have a vfs.zfs.dynamic_write_buffer tunable defined, if so, >>>> what is it set to? =20 >>> No. >>> >>> Bye, >>> Alexander. >>> =20 >> Wait a minute... you have a panic coming out of running the periodic >> jobs and you do _*not*_ have the patch loaded? Got a dump available >> from that? >> > To make it explicit about what is patched on this system or not: > # svnlite status > M etc/rc.d/localpkg > M lib/libc/Makefile > M lib/libutil/Makefile > M lib/msun/Makefile > M libexec/rtld-elf/Makefile > M sbin/dump/dumprmt.c > M sbin/dump/itime.c > M sbin/dump/optr.c > M sbin/dump/tape.c > M sbin/restore/dirs.c > M sbin/restore/symtab.c > M sbin/restore/tape.c > ? sys/amd64/conf/ANDROMEDA > M sys/dev/drm/drmP.h > M sys/dev/e1000/if_igb.c > M sys/dev/usb/serial/ulpt.c > M sys/fs/cd9660/cd9660_vfsops.c > M sys/fs/msdosfs/msdosfs_vfsops.c > M sys/fs/smbfs/smbfs_vfsops.c > M sys/fs/udf/udf_vfsops.c > M sys/fs/unionfs/union_vfsops.c > M sys/gnu/fs/reiserfs/reiserfs_vfsops.c > M sys/kern/kern_jail.c > M sys/kern/kern_sig.c > M sys/kern/subr_bus.c > M sys/sys/jail.h > M usr.sbin/adduser/adduser.sh > M usr.sbin/adduser/rmuser.sh > M usr.sbin/nfsd/nfsd.c > M usr.sbin/watchdogd/Makefile > > Unfortunately I don't have a dump. Every crash so far didn't lead to a > core in the dump device (the thread on stable has a stacktrace: > https://lists.freebsd.org/pipermail/freebsd-stable/2015-December/083807= =2Ehtml). > No idea why I don't get a dump. > > # grep dump /etc/rc.conf /boot/loader.conf > /etc/rc.conf:dumpdev=3D"/dev/ada3p2" > /boot/loader.conf:dumpdev=3D"/dev/ada3p2" > > # savecore -C /dev/ada3p2 > > # ll /var/crash > total 9 > -rw-r--r-- 1 root wheel 3B Nov 11 08:45 bounds > -rw-r--r-- 1 root wheel 5B Apr 28 2011 minfree > > Bye, > Alexander. > --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms040204030602080901050608 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNTEyMTExMjQ2MTBaME8GCSqGSIb3DQEJBDFCBEA7 /uXOy3c+mpCjpJ9JFV7iToPMDW6lY4fT9MVUdHk1kAL2xPi64QohV3wkxr9FthP3UwbmwZpw D0I72H3SyxurMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIATmF7ln+m orF9UzQhcfShxBISFxeqEAJY0magixzzdqeop6pWyLPdBeMLDnAQ8EQFv4JyWHQVdaRdGWo5 BBKgxPiEbMCAh36mYzRRlliw2QKA+0+C9j6RpDo0zXRrVPvmoMs0DOzYYhRWdD7kuz38+vZQ CC+WpCQFgqHfvSLU9ZHPGex53Mdug8EbUQsnEEH7iat7wV8Z1HXnCZy41vY/vpzWIszBSJY0 YbGu7RiXs0LxShaPHreOVUs3j0l4bKs1x0B/Y0UiYruXajVXh5pe1QorK6uUN1YDF3jKtvDC p94kruuVdqrN0GcA0F8NFdLHLmxV8c3LZpdVQO/1iz1uo6tgoviAkUiL677Q6Jp9JqLLLUZn 5/4O0C9sjrZgSx4v9l3AsWo87EfGQ/87Y/qVmEspC6z/lAqUATa0F7yTrfQEWplOTDoiMXcx E1lLNIN4P8zP9clQ0B9kMDxug2raQTmdWbEA3VEfq8wtRYt8P4xe7vAZKgFtZxReccTzmFI+ fVkVV0qDmS6xZhJbVLZH1pmVSHSXMga2Awpp20WCKSiWj2h5yCNoA6gFGlBVJ4iX4M3DZORz CAYkaMqgh/HPiVyLBesHriEdy8anVY36mPqpHCbJflsj7664j+G6yJ5f6A1upgIDGifTWEul uOZWQt0d+jkezhM3tK0PifVZ8o8AAAAAAAA= --------------ms040204030602080901050608--