From owner-freebsd-fs@FreeBSD.ORG Mon Mar 24 15:02:10 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A74FFEBC for ; Mon, 24 Mar 2014 15:02:10 +0000 (UTC) Received: from mx2.paymentallianceintl.com (mx2.paymentallianceintl.com [216.26.158.171]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 023E7F76 for ; Mon, 24 Mar 2014 15:02:08 +0000 (UTC) Received: from firewall.mikej.com (162-238-140-44.lightspeed.lsvlky.sbcglobal.net [162.238.140.44]) by mx2.paymentallianceintl.com (8.14.5/8.13.8) with ESMTP id s2OF1qIN089268 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 24 Mar 2014 11:01:54 -0400 (EDT) (envelope-from mikej@mikej.com) Received: from firewall.mikej.com (localhost.mikej.com [127.0.0.1]) by firewall.mikej.com (8.14.8/8.14.8) with ESMTP id s2OF1BpB090835 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 24 Mar 2014 11:01:51 -0400 (EDT) (envelope-from mikej@mikej.com) Received: (from www@localhost) by firewall.mikej.com (8.14.8/8.14.8/Submit) id s2OF19TN090834; Mon, 24 Mar 2014 11:01:09 -0400 (EDT) (envelope-from mikej@mikej.com) X-Authentication-Warning: firewall.mikej.com: www set sender to mikej@mikej.com using -f To: Karl Denninger , Subject: Re: kern/187594: [zfs] [patch] ZFS ARC behavior problem and fix [SB QUAR: Fri Mar 21 16:01:12 2014] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Mon, 24 Mar 2014 11:00:38 -0400 From: mikej In-Reply-To: <532CD3BC.5020305@denninger.net> References: <201403201710.s2KHA0e9043051@freefall.freebsd.org> <31e2f5092048c128add6f4cd3d136f4d@mail.mikej.com> <532CD3BC.5020305@denninger.net> Message-ID: X-Sender: mikej@mikej.com User-Agent: Roundcube Webmail/0.6-beta X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2014 15:02:10 -0000 Karl, Not being a C coder it appears a declaration is missing. --- arc.o --- /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2614:15: error: use of undeclared identifier 'cnt' freepages = cnt.v_free_target - (cnt.v_free_target / 33); Thanks again, Michael Jung On 2014-03-21 20:05, Karl Denninger wrote: > Here 'ya go... > > Please keep me posted (the list is best as the more public > commentary > the better, and if this needs more tuning that's the way to find > out!) > on how it works for you. > > I have it in production at this point and am happy with it -- the > current default is at the pager "wakeup" level less 3%, but it of > course can be tuned manually. > > On 3/21/2014 3:59 PM, mikej wrote: > >> Karl, >> >> I've looked at my raw mailbox and something is trashing tabs and >> line >> length for your more recent patches in email. >> >> I did not see any attachments, nor updates to the PR for download - >> would >> you mind sending me the latest patch as an attachment? >> >> Thanks for your work, I believe this is going to add real stability >> >> without having to set vfs.zfs.arc_max and other tunables. >> >> Kind regards, >> >> Michael Jung >> >> On 2014-03-20 13:10, Karl Denninger wrote: >> >>> The following reply was made to PR kern/187594; it has been noted >>> by GNATS. >>> >>> From: Karl Denninger [1] >>> To: bug-followup@FreeBSD.org [2], karl@fs.denninger.net [3] >>> Cc: >>> Subject: Re: kern/187594: [zfs] [patch] ZFS ARC behavior problem >>> and fix >>> Date: Thu, 20 Mar 2014 12:00:54 -0500 >>> >>>  This is a cryptographically signed message in MIME format. >>> >>>  --------------ms010508000607000909070805 >>>  Content-Type: text/plain; charset=ISO-8859-1; format=flowed >>>  Content-Transfer-Encoding: quoted-printable >>> >>>  Responsive to avg's comment and with another overnight and >>> daytime load=20 >>>  of testing on multiple machines with varying memory configs >>> from 4-24GB=20 >>>  of RAM here is another version of the patch. >>> >>>  The differences are: >>> >>>  1. No longer use kernel_sysctlbyname, include the VM header >>> file and get = >>> >>>  the values directly (less overhead.)  Remove the variables no >>> longer need= >>>  ed. >>> >>>  2. Set the default free RAM level for ARC shrinkage to >>> v_free_target=20 >>>  less 3% as I was able to provoke a stall once with it set to a >>> 5%=20 >>>  reservation, was able to provoke it with the parameter set to >>> 10% with a = >>> >>>  lot of work and was able to do so "on demand" with it set to >>> 20%.  With=20 >>>  a 5% invasion initiating a scrub with very heavy I/O and image >>> load=20 >>>  (hundreds of web and database processes) provoked a ~10 second >>> system=20 >>>  stall.  With it set to 3% I have not been able to reproduce >>> the stall=20 >>>  yet the inactive page count remains stable even under extremely >>> heavy=20 >>>  load, indicating that page-stealing remains effective when >>> required. =20 >>>  Note that for my workload even with this level set above >>> v_free_target,=20 >>>  which would imply no page stealing by the VM system before ARC >>> expansion = >>> >>>  is halted, I do not get unbridled inactive page growth. >>> >>>  As before vfs.zfs.zrc_freepages and >>> vfs.zfs.arc_freepage_percent remain=20 >>>  as accessible knobs if you wish to twist them for some reason >>> to=20 >>>  compensate for an unusual load profile or machine >>> configuration. >>> >>>  *** arc.c.original    Thu Mar 13 09:18:48 2014 >>>  --- arc.c    Thu Mar 20 11:51:48 2014 >>>  *************** >>>  *** 18,23 **** >>>  --- 18,94 ---- >>>      * >>>      * CDDL HEADER END >>>      */ >>>  + >>>  + /* Karl Denninger (karl@denninger.net [4]), 3/20/2014, >>> FreeBSD-specific >>>  +  * >>>  +  * If "NEWRECLAIM" is defined, change the "low memory" >>> warning that cau= >>>  ses >>>  +  * the ARC cache to be pared down.  The reason for the >>> change is that t= >>>  he >>>  +  * apparent attempted algorithm is to start evicting ARC >>> cache when fre= >>>  e >>>  +  * pages fall below 25% of installed RAM.  This maps >>> reasonably well to= >>>   how >>>  +  * Solaris is documented to behave; when "lotsfree" is >>> invaded ZFS is t= >>>  old >>>  +  * to pare down. >>>  +  * >>>  +  * The problem is that on FreeBSD machines the system >>> doesn't appear to= >>>   be >>>  +  * getting what the authors of the original code thought >>> they were look= >>>  ing at >>>  +  * with its test -- or at least not what Solaris did -- and >>> as a result= >>>   that >>>  +  * test never triggers.  That leaves the only reclaim >>> trigger as the "p= >>>  aging >>>  +  * needed" status flag, and by the time * that trips the >>> system is alre= >>>  ady >>>  +  * in low-memory trouble.  This can lead to severe >>> pathological behavio= >>>  r >>>  +  * under the following scenario: >>>  +  * - The system starts to page and ARC is evicted. >>>  +  * - The system stops paging as ARC's eviction drops wired >>> RAM a bit. >>>  +  * - ARC starts increasing its allocation again, and wired >>> memory grows= >>>  =2E >>>  +  * - A new image is activated, and the system once again >>> attempts to pa= >>>  ge. >>>  +  * - ARC starts to be evicted again. >>>  +  * - Back to #2 >>>  +  * >>>  +  * Note that ZFS's ARC default (unless you override it in >>> /boot/loader.= >>>  conf) >>>  +  * is to allow the ARC cache to grab nearly all of free RAM, >>> provided n= >>>  obody >>>  +  * else needs it.  That would be ok if we evicted cache >>> when required. >>>  +  * >>>  +  * Unfortunately the system can get into a state where it >>> never >>>  +  * manages to page anything of materiality back in, as if >>> there is acti= >>>  ve >>>  +  * I/O the ARC will start grabbing space once again as soon >>> as the memo= >>>  ry >>>  +  * contention state drops.  For this reason the "paging is >>> occurring" f= >>>  lag >>>  +  * should be the **last resort** condition for ARC eviction; >>> you want t= >>>  o >>>  +  * (as Solaris does) start when there is material free RAM >>> left BUT the= >>> >>>  +  * vm system thinks it needs to be active to steal pages >>> back in the at= >>>  tempt >>>  +  * to never get into the condition where you're potentially >>> paging off >>>  +  * executables in favor of leaving disk cache allocated. >>>  +  * >>>  +  * To fix this we change how we look at low memory, >>> declaring two new >>>  +  * runtime tunables. >>>  +  * >>>  +  * The new sysctls are: >>>  +  * vfs.zfs.arc_freepages (free pages required to call RAM >>> "sufficient")= >>> >>>  +  * vfs.zfs.arc_freepage_percent (additional reservation >>> percentage, def= >>>  ault 0) >>>  +  * >>>  +  * vfs.zfs.arc_freepages is initialized from >>> vm.v_free_target, less 3%.= >>> >>>  +  * This should insure that we allow the VM system to steal >>> pages first,= >>> >>>  +  * but pare the cache before we suspend processes attempting >>> to get mor= >>>  e >>>  +  * memory, thereby avoiding "stalls."  You can set this >>> higher if you w= >>>  ish, >>>  +  * or force a specific percentage reservation as well, but >>> doing so may= >>> >>>  +  * cause the cache to pare back while the VM system remains >>> willing to >>>  +  * allow "inactive" pages to accumulate.  The challenge is >>> that image >>>  +  * activation can force things into the page space on a >>> repeated basis >>>  +  * if you allow this level to be too small (the above >>> pathological >>>  +  * behavior); the defaults should avoid that behavior but >>> the sysctls >>>  +  * are exposed should your workload require adjustment. >>>  +  * >>>  +  * If we're using this check for low memory we are replacing >>> the previo= >>>  us >>>  +  * ones, including the oddball "random" reclaim that appears >>> to fire fa= >>>  r >>>  +  * more often than it should.  We still trigger if the >>> system pages. >>>  +  * >>>  +  * If you turn on NEWRECLAIM_DEBUG then the kernel will >>> print on the co= >>>  nsole >>>  +  * status messages when the reclaim status trips on and off, >>> along with= >>>   the >>>  +  * page count aggregate that triggered it (and the free >>> space) for each= >>> >>>  +  * event. >>>  +  */ >>>  + >>>  + #define    NEWRECLAIM >>>  + #undef    NEWRECLAIM_DEBUG >>>  + >>>  + >>>     /* >>>      * Copyright (c) 2005, 2010, Oracle and/or its >>> affiliates. All rights = >>>  reserved. >>>      * Copyright (c) 2013 by Delphix. All rights reserved. >>>  *************** >>>  *** 139,144 **** >>>  --- 210,222 ---- >>>    =20 >>>     #include >>>    =20 >>>  + #ifdef    NEWRECLAIM >>>  + #ifdef    __FreeBSD__ >>>  + #include >>>  + #include >>>  + #endif >>>  + #endif    /* NEWRECLAIM */ >>>  + >>>     #ifdef illumos >>>     #ifndef _KERNEL >>>     /* set with ZFS_DEBUG=3Dwatch, to enable watchpoints on >>> frozen buffers= >>>   */ >>>  *************** >>>  *** 203,218 **** >>>  --- 281,316 ---- >>>     int zfs_arc_shrink_shift =3D 0; >>>     int zfs_arc_p_min_shift =3D 0; >>>     int zfs_disable_dup_eviction =3D 0; >>>  + #ifdef    NEWRECLAIM >>>  + #ifdef  __FreeBSD__ >>>  + static    int freepages =3D 0;    /* This much memory >>> is considered critical = >>>  */ >>>  + static    int percent_target =3D 0;    /* Additionally >>> reserve "X" percent fr= >>>  ee RAM */ >>>  + #endif    /* __FreeBSD__ */ >>>  + #endif    /* NEWRECLAIM */ >>>    =20 >>>     TUNABLE_QUAD("vfs.zfs.arc_max", &zfs_arc_max); >>>     TUNABLE_QUAD("vfs.zfs.arc_min", &zfs_arc_min); >>>     TUNABLE_QUAD("vfs.zfs.arc_meta_limit", >>> &zfs_arc_meta_limit); >>>  + #ifdef    NEWRECLAIM >>>  + #ifdef  __FreeBSD__ >>>  + TUNABLE_INT("vfs.zfs.arc_freepages", &freepages); >>>  + TUNABLE_INT("vfs.zfs.arc_freepage_percent", &percent_target); >>> >>>  + #endif    /* __FreeBSD__ */ >>>  + #endif    /* NEWRECLAIM */ >>>  + >>>     SYSCTL_DECL(_vfs_zfs); >>>     SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, arc_max, CTLFLAG_RDTUN, >>> &zfs_arc_max,= >>>   0, >>>         "Maximum ARC size"); >>>     SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, arc_min, CTLFLAG_RDTUN, >>> &zfs_arc_min,= >>>   0, >>>         "Minimum ARC size"); >>>    =20 >>>  + #ifdef    NEWRECLAIM >>>  + #ifdef  __FreeBSD__ >>>  + SYSCTL_INT(_vfs_zfs, OID_AUTO, arc_freepages, CTLFLAG_RWTUN, >>> &freepages= >>>  , 0, "ARC Free RAM Pages Required"); >>>  + SYSCTL_INT(_vfs_zfs, OID_AUTO, arc_freepage_percent, >>> CTLFLAG_RWTUN, &pe= >>>  rcent_target, 0, "ARC Free RAM Target percentage"); >>>  + #endif    /* __FreeBSD__ */ >>>  + #endif    /* NEWRECLAIM */ >>>  + >>>     /* >>>      * Note that buffers can be in one of 6 states: >>>      *    ARC_anon    - anonymous (discussed below) >>>  *************** >>>  *** 2438,2443 **** >>>  --- 2536,2546 ---- >>>     { >>>    =20 >>>     #ifdef _KERNEL >>>  + #ifdef    NEWRECLAIM_DEBUG >>>  +     static    int    xval =3D -1; >>>  +     static    int    oldpercent =3D 0; >>>  +     static    int    oldfreepages =3D 0; >>>  + #endif    /* NEWRECLAIM_DEBUG */ >>>    =20 >>>         if (needfree) >>>             return (1); >>>  *************** >>>  *** 2476,2481 **** >>>  --- 2579,2585 ---- >>>             return (1); >>>    =20 >>>     #if defined(__i386) >>>  + >>>         /* >>>          * If we're on an i386 platform, it's possible >>> that we'll exhaust the= >>> >>>          * kernel heap space before we ever run out of >>> available physical >>>  *************** >>>  *** 2492,2502 **** >>>             return (1); >>>     #endif >>>     #else    /* !sun */ >>>         if (kmem_used() > (kmem_size() * 3) / 4) >>>             return (1); >>>     #endif    /* sun */ >>>    =20 >>>  - #else >>>         if (spa_get_random(100) =3D=3D 0) >>>             return (1); >>>     #endif >>>  --- 2596,2658 ---- >>>             return (1); >>>     #endif >>>     #else    /* !sun */ >>>  + >>>  + #ifdef    NEWRECLAIM >>>  + #ifdef  __FreeBSD__ >>>  + /* >>>  +  * Implement the new tunable free RAM algorithm.  We check >>> the free pag= >>>  es >>>  +  * against the minimum specified target and the percentage >>> that should = >>>  be >>>  +  * free.  If we're low we ask for ARC cache shrinkage.  If >>> this is defi= >>>  ned >>>  +  * on a FreeBSD system the older checks are not performed. >>>  +  * >>>  +  * Check first to see if we need to init freepages, then >>> test. >>>  +  */ >>>  +     if (!freepages) {        /* If zero then >>> (re)init */ >>>  +         freepages =3D cnt.v_free_target - >>> (cnt.v_free_target / 33); >>>  + #ifdef    NEWRECLAIM_DEBUG >>>  +         printf("ZFS ARC: Default >>> vfs.zfs.arc_freepages to [%u] [%u less 3%%]= >>>  n", freepages, cnt.v_free_target); >>>  + #endif    /* NEWRECLAIM_DEBUG */ >>>  +     } >>>  + #ifdef    NEWRECLAIM_DEBUG >>>  +     if (percent_target !=3D oldpercent) { >>>  +         printf("ZFS ARC: Reservation percent change >>> to [%d], [%d] pages, [%d]= >>>   freen", percent_target, cnt.v_page_count, cnt.v_free_count); >>>  +         oldpercent =3D percent_target; >>>  +     } >>>  +     if (freepages !=3D oldfreepages) { >>>  +         printf("ZFS ARC: Low RAM page change to [%d], >>> [%d] pages, [%d] freen= >>>  ", freepages, cnt.v_page_count, cnt.v_free_count); >>>  +         oldfreepages =3D freepages; >>>  +     } >>>  + #endif    /* NEWRECLAIM_DEBUG */ >>>  + /* >>>  +  * Now figure out how much free RAM we require to call the >>> ARC cache st= >>>  atus >>>  +  * "ok".  Add the percentage specified of the total to the >>> base require= >>>  ment. >>>  +  */ >>>  + >>>  +     if (cnt.v_free_count < freepages + ((cnt.v_page_count >>> / 100) * percent= >>>  _target)) { >>>  + #ifdef    NEWRECLAIM_DEBUG >>>  +         if (xval !=3D 1) { >>>  +             printf("ZFS ARC: RECLAIM total %u, >>> free %u, free pct (%u), reserved = >>>  (%u), target pct (%u)n", cnt.v_page_count, cnt.v_free_count, >>> ((cnt.v_fre= >>>  e_count * 100) / cnt.v_page_count), freepages, percent_target); >>> >>>  +             xval =3D 1; >>>  +         } >>>  + #endif    /* NEWRECLAIM_DEBUG */ >>>  +         return(1); >>>  +     } else { >>>  + #ifdef    NEWRECLAIM_DEBUG >>>  +         if (xval !=3D 0) { >>>  +             printf("ZFS ARC: NORMAL total %u, >>> free %u, free pct (%u), reserved (= >>>  %u), target pct (%u)n", cnt.v_page_count, cnt.v_free_count, >>> ((cnt.v_free= >>>  _count * 100) / cnt.v_page_count), freepages, percent_target); >>>  +             xval =3D 0; >>>  +         } >>>  + #endif    /* NEWRECLAIM_DEBUG */ >>>  +         return(0); >>>  +     } >>>  + >>>  + #endif    /* __FreeBSD__ */ >>>  + #endif    /* NEWRECLAIM */ >>>  + >>>         if (kmem_used() > (kmem_size() * 3) / 4) >>>             return (1); >>>     #endif    /* sun */ >>>    =20 >>>         if (spa_get_random(100) =3D=3D 0) >>>             return (1); >>>     #endif >>> >>>  --=20 >>>  -- Karl >>>  karl@denninger.net [5] >>> >>>  --------------ms010508000607000909070805 >>>  Content-Type: application/pkcs7-signature; name="smime.p7s" >>>  Content-Transfer-Encoding: base64 >>>  Content-Disposition: attachment; filename="smime.p7s" >>>  Content-Description: S/MIME Cryptographic Signature >>> >>>   >>> >> > > MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC >>> >>>   >>> >> > > BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI >>> >>>   >>> >> > > EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM >>> >>>   >>> >> > > TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv >>> >>>   >>> >> > > bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5 >>> >>>   >>> >> > > MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg >>> >>>   >>> >> > > RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG >>> >>>   >>> >> > > SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th >>> >>>   >>> >> > > d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W >>> >>>   >>> >> > > 6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV >>> >>>   >>> >> > > jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5 >>> >>>   >>> >> > > SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY >>> >>>   >>> >> > > 5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8 >>> >>>   >>> >> > > Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4 >>> >>>   >>> >> > > GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci >>> >>>   >>> >> > > WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN >>> >>>   >>> >> > > nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB >>> >>>   >>> >> > > o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg >>> >>>   >>> >> > > hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4 >>> >>>   >>> >> > > +LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG >>> >>>   >>> >> > > CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy >>> >>>   >>> >> > > bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO >>> >>>   >>> >> > > 31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c >>> >>>   >>> >> > > L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1 >>> >>>   >>> >> > > YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD >>> >>>   >>> >> > > pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE >>> >>>   >>> >> > > f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk >>> >>>   >>> >> > > YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD >>> >>>   >>> >> > > VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2 >>> >>>   >>> >> > > aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ >>> >>>   >>> >> > > KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDAzMjAxNzAwNTRaMCMGCSqGSIb3DQEJBDEW >>> >>>   >>> >> > > BBQrRNHcjN1dwAlZu7blrh+3Vu7++TBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL >>> >>>   >>> >> > > BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA >>> >>>   >>> >> > > MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV >>> >>>   >>> >> > > BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT >>> >>>   >>> >> > > EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq >>> >>>   >>> >> > > hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG >>> >>>   >>> >> > > 9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV >>> >>>   >>> >> > > BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk >>> >>>   >>> >> > > YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh >>> >>>   >>> >> > > c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAsIbg6OIk2XnrPw25FA9+s4FCnKlo >>> >>>   >>> >> > > Wz3KtfA59Gf2jX8yKEQW26k1QK6w62Zc8KaB5LypaOK1rZ5bipeu6rGHhgaG1oPXUMmcc2p4 >>> >>>   >>> >> > > E18URDskAfn5apoJ/n54nR94OqHfQ/EPBx711pxYtAGBLFOOzwrU2MEZCl2KBydI+Bw/E75R >>> >>>   >>> >> > > WRIk6y0NqSWjgVWU2tJwnOEZj/2UGQCSvJ7h5t1n7idbDIfT88/hvAW3b3knRwPxwpZretXq >>> >>>   >>> >> > > 2BGgmv8lojr7Zui5sR/YdDjSK2yGHqo0mWkSAHp0Wts8okcoJNZSEispFRh56MWCIoJ51cki >>> >>>   >>> >> > > pCZH/vX1EEsfka3CrlE7LWABAYf1biy+Xq/Bfxgq9oAaknGF2yM0jgR7xnjLYLvbv5pjt7ar >>> >>>   >>> >> > > TH2JslJMYkJPKiYFJNEgVJ9wTVQtrCPJQPTk3R1qD3YFraly5Mgjwy5Ax5n8SW858WWOxHeP >>> >>>   >>> >> > > vmL0j1boO0Re9qeAb9v/z8z3NPkFPZhBrEz3g6INCWil+2Vx1yruJvxm1oN9OMQSt2qY38rj >>> >>>   >>> >> > > XWhWVxoQtW39LZc/xSNR41DQXvPJ8VOvyrmvLm7uTm4+lQYVUwNuLNbDFlj8slkAeXF/eR1S >>> >>>   >>> >> > > 4VuWtwexxCco+xGjbPTcZgap976XsvlRWOmjmwqZyGNuW7ZmcODQPFjQvpnBkx9Rm5cLndK6 >>> >>>  OVorTQkAAAAAAAA= >>>  --------------ms010508000607000909070805-- >>> >>> _______________________________________________ >>> freebsd-fs@freebsd.org [6] mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs [7] >>> To unsubscribe, send any mail to >>> "freebsd-fs-unsubscribe@freebsd.org" [8] > > -- > Karl Denninger > karl@denninger.net [9] > _The Market Ticker_ > > > > Links: > ------ > [1] mailto:karl@denninger.net > [2] mailto:bug-followup@FreeBSD.org > [3] mailto:karl@fs.denninger.net > [4] mailto:karl@denninger.net > [5] mailto:karl@denninger.net > [6] mailto:freebsd-fs@freebsd.org > [7] http://lists.freebsd.org/mailman/listinfo/freebsd-fs > [8] mailto:freebsd-fs-unsubscribe@freebsd.org > [9] mailto:karl@denninger.net