to:stable+help@freebsd.org> List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 References: <5e1b5097-c1c0-4740-a491-63c709d01c25@sentex.net> <67721332-fa1d-4b3c-aa57-64594ad5d77a@shrew.net> <77e203b3-c555-408b-9634-c452cb3a57ac@sentex.net> In-Reply-To: <77e203b3-c555-408b-9634-c452cb3a57ac@sentex.net> From: Warner Losh Date: Thu, 2 May 2024 08:16:04 -0600 Message-ID: Subject: Re: how to tell if TRIM is working To: mike tancsa Cc: Matthew Grooms , stable@freebsd.org Content-Type: multipart/alternative; boundary="000000000000165b720617793f29" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4VVbZp5nf5z4HcV --000000000000165b720617793f29 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, May 2, 2024 at 6:30=E2=80=AFAM mike tancsa wrote: > On 5/1/2024 4:24 PM, Matthew Grooms wrote: > > On 5/1/24 14:38, mike tancsa wrote: > >> Kind of struggling to check if TRIM is actually working or not with > >> my SSDs on RELENG_14 in ZFS. > >> > >> On a pool that has almost no files on it (capacity at 0% out of 3TB), > >> should not > >> > >> zpool -w trim be almost instant after a couple of runs ? > >> Instead it seems to always take about 10min to complete. > >> > >> Looking at the stats, > >> > >> kstat.zfs.tortank1.misc.iostats.trim_bytes_failed: 0 > >> kstat.zfs.tortank1.misc.iostats.trim_extents_failed: 0 > >> kstat.zfs.tortank1.misc.iostats.trim_bytes_skipped: 2743435264 > >> kstat.zfs.tortank1.misc.iostats.trim_extents_skipped: 253898 > >> kstat.zfs.tortank1.misc.iostats.trim_bytes_written: 14835526799360 > >> kstat.zfs.tortank1.misc.iostats.trim_extents_written: 1169158 > >> > >> what and why are bytes being skipped ? > >> > >> One of the drives for example I had a hard time seeing evidence of > >> this at the disk level while fiddling with TRIM recently. It appeared > >> that at least some counters are driver and operation specific. For > >> example, the da driver appears to update counters in some paths but > >> not others. I assume that ada is different. There is a bug report for > >> da, but haven't seen any feedback ... > > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277673 > > > > You could try to run gstat with the -d flag during the time period > > when the delete operations are expected to occur. That should give you > > an idea of what's happening at the disk level in real time but may not > > offer more info than you're already seeing. > > > > It *seems* to be doing something. What I dont understand is why if I > run it once, do nothing (no writes / snapshots etc), and then run trim > again, it seems to be doing something with gstat even though there > should not be anything to mark as being trimmed ? > > dT: 1.002s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps > ms/d %busy Name > 0 1254 0 0 0.0 986 5202 2.0 244 > 8362733 4.5 55.6 ada0 > 12 1242 0 0 0.0 1012 5218 1.9 206 > 4972041 6.0 63.3 ada2 > 12 1242 0 0 0.0 1012 5218 1.9 206 > 4972041 6.0 63.3 ada2p1 > 0 4313 0 0 0.0 1024 5190 0.8 3266 > 6463815 0.4 62.8 ada3 > 0 1254 0 0 0.0 986 5202 2.0 244 > 8362733 4.5 55.6 ada0p1 > 0 4238 0 0 0.0 960 4874 0.7 3254 > 6280362 0.4 59.8 ada5 > 0 4313 0 0 0.0 1024 5190 0.8 3266 > 6463815 0.4 62.8 ada3p1 > 0 4238 0 0 0.0 960 4874 0.7 3254 > 6280362 0.4 59.8 ada5p1 > dT: 1.001s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps > ms/d %busy Name > 2 2381 0 0 0.0 1580 9946 0.9 767 > 5990286 1.8 70.0 ada0 > 2 2801 0 0 0.0 1540 9782 0.9 1227 > 11936510 1.0 65.2 ada2 > 2 2801 0 0 0.0 1540 9782 0.9 1227 > 11936510 1.0 65.2 ada2p1 > 0 2072 0 0 0.0 1529 9566 0.8 509 > 12549587 2.1 57.0 ada3 > 2 2381 0 0 0.0 1580 9946 0.9 767 > 5990286 1.8 70.0 ada0p1 > 0 2042 0 0 0.0 1517 9427 0.6 491 > 12549535 1.9 52.4 ada5 > 0 2072 0 0 0.0 1529 9566 0.8 509 > 12549587 2.1 57.0 ada3p1 > 0 2042 0 0 0.0 1517 9427 0.6 491 > 12549535 1.9 52.4 ada5p1 > dT: 1.002s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps > ms/d %busy Name > 2 1949 0 0 0.0 1094 5926 1.2 827 > 11267200 1.8 78.8 ada0 > 0 2083 0 0 0.0 1115 6034 0.7 939 > 16537981 1.4 67.2 ada2 > 0 2083 0 0 0.0 1115 6034 0.7 939 > 16537981 1.4 67.2 ada2p1 > 2 2525 0 0 0.0 1098 5914 0.8 1399 > 16021615 1.1 79.3 ada3 > 2 1949 0 0 0.0 1094 5926 1.2 827 > 11267200 1.8 78.8 ada0p1 > 12 2471 0 0 0.0 1018 5399 1.0 1425 > 15395566 1.1 80.5 ada5 > 2 2525 0 0 0.0 1098 5914 0.8 1399 > 16021615 1.1 79.3 ada3p1 > 12 2471 0 0 0.0 1018 5399 1.0 1425 > 15395566 1.1 80.5 ada5p1 > > The ultimate problem is that after a while with a lot of writes, the > disk performance will be toast until I do a manual trim -f of the disk > :( this is most notable on consumer WD SSDs. I havent done any > extensive tests with Samsung SSDs to see if there are performance > penalties or not. It might be that they are just better at masking the > problem. I dont see the same issue with ZFS on Linux with the same > disks / hardware > When trims are fast, you want to send them to the drive as soon as you know the blocks are freed. UFS always does this (if trim is enabled at all)= . ZFS has a lot of knobs to control when / how / if this is done. vfs.zfs.vdev.trim_min_active: 1 vfs.zfs.vdev.trim_max_active: 2 vfs.zfs.trim.queue_limit: 10 vfs.zfs.trim.txg_batch: 32 vfs.zfs.trim.metaslab_skip: 0 vfs.zfs.trim.extent_bytes_min: 32768 vfs.zfs.trim.extent_bytes_max: 134217728 vfs.zfs.l2arc.trim_ahead: 0 I've not tried to tune these in the past, but you can see how they affect things. Warner > I have an open PR in > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277992 that I think > might actually have 2 separate problems. > > ---Mike > > > >> e.g. here was one disk in the pool that was taking a long time for > >> each zpool trim > >> > >> # time trim -f /dev/ada1 > >> trim /dev/ada1 offset 0 length 1000204886016 > >> 0.000u 0.057s 1:29.33 0.0% 5+184k 0+0io 0pf+0w > >> and then if I re-run it > >> # time trim -f /dev/ada1 > >> trim /dev/ada1 offset 0 length 1000204886016 > >> 0.000u 0.052s 0:04.15 1.2% 1+52k 0+0io 0pf+0w > >> > >> 90 seconds and then 4 seconds after that. > >> > > > > -Matthew > > > > --000000000000165b720617793f29 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: base64 PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0ciI+PGJyPjwvZGl2Pjxicj48ZGl2IGNsYXNzPSJn bWFpbF9xdW90ZSI+PGRpdiBkaXI9Imx0ciIgY2xhc3M9ImdtYWlsX2F0dHIiPk9uIFRodSwgTWF5 IDIsIDIwMjQgYXQgNjozMOKAr0FNIG1pa2UgdGFuY3NhICZsdDs8YSBocmVmPSJtYWlsdG86bWlr ZUBzZW50ZXgubmV0Ij5taWtlQHNlbnRleC5uZXQ8L2E+Jmd0OyB3cm90ZTo8YnI+PC9kaXY+PGJs b2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAu OGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDox ZXgiPk9uIDUvMS8yMDI0IDQ6MjQgUE0sIE1hdHRoZXcgR3Jvb21zIHdyb3RlOjxicj4NCiZndDsg T24gNS8xLzI0IDE0OjM4LCBtaWtlIHRhbmNzYSB3cm90ZTo8YnI+DQomZ3Q7Jmd0OyBLaW5kIG9m IHN0cnVnZ2xpbmcgdG8gY2hlY2sgaWYgVFJJTSBpcyBhY3R1YWxseSB3b3JraW5nIG9yIG5vdCB3 aXRoIDxicj4NCiZndDsmZ3Q7IG15IFNTRHMgb24gUkVMRU5HXzE0IGluIFpGUy48YnI+DQomZ3Q7 Jmd0Ozxicj4NCiZndDsmZ3Q7IE9uIGEgcG9vbCB0aGF0IGhhcyBhbG1vc3Qgbm8gZmlsZXMgb24g aXQgKGNhcGFjaXR5IGF0IDAlIG91dCBvZiAzVEIpLCA8YnI+DQomZ3Q7Jmd0OyBzaG91bGQgbm90 PGJyPg0KJmd0OyZndDs8YnI+DQomZ3Q7Jmd0OyB6cG9vbCAtdyB0cmltICZsdDtwb29sJmd0OyBi ZSBhbG1vc3QgaW5zdGFudCBhZnRlciBhIGNvdXBsZSBvZiBydW5zID8gPGJyPg0KJmd0OyZndDsg SW5zdGVhZCBpdCBzZWVtcyB0byBhbHdheXMgdGFrZSBhYm91dCAxMG1pbiB0byBjb21wbGV0ZS48 YnI+DQomZ3Q7Jmd0Ozxicj4NCiZndDsmZ3Q7IExvb2tpbmcgYXQgdGhlIHN0YXRzLDxicj4NCiZn dDsmZ3Q7PGJyPg0KJmd0OyZndDsga3N0YXQuemZzLnRvcnRhbmsxLm1pc2MuaW9zdGF0cy50cmlt X2J5dGVzX2ZhaWxlZDogMDxicj4NCiZndDsmZ3Q7IGtzdGF0Lnpmcy50b3J0YW5rMS5taXNjLmlv c3RhdHMudHJpbV9leHRlbnRzX2ZhaWxlZDogMDxicj4NCiZndDsmZ3Q7IGtzdGF0Lnpmcy50b3J0 YW5rMS5taXNjLmlvc3RhdHMudHJpbV9ieXRlc19za2lwcGVkOiAyNzQzNDM1MjY0PGJyPg0KJmd0 OyZndDsga3N0YXQuemZzLnRvcnRhbmsxLm1pc2MuaW9zdGF0cy50cmltX2V4dGVudHNfc2tpcHBl ZDogMjUzODk4PGJyPg0KJmd0OyZndDsga3N0YXQuemZzLnRvcnRhbmsxLm1pc2MuaW9zdGF0cy50 cmltX2J5dGVzX3dyaXR0ZW46IDE0ODM1NTI2Nzk5MzYwPGJyPg0KJmd0OyZndDsga3N0YXQuemZz LnRvcnRhbmsxLm1pc2MuaW9zdGF0cy50cmltX2V4dGVudHNfd3JpdHRlbjogMTE2OTE1ODxicj4N CiZndDsmZ3Q7PGJyPg0KJmd0OyZndDsgd2hhdCBhbmQgd2h5IGFyZSBieXRlcyBiZWluZyBza2lw cGVkID88YnI+DQomZ3Q7Jmd0Ozxicj4NCiZndDsmZ3Q7IE9uZSBvZiB0aGUgZHJpdmVzIGZvciBl eGFtcGxlIEkgaGFkIGEgaGFyZCB0aW1lIHNlZWluZyBldmlkZW5jZSBvZiA8YnI+DQomZ3Q7Jmd0 OyB0aGlzIGF0IHRoZSBkaXNrIGxldmVsIHdoaWxlIGZpZGRsaW5nIHdpdGggVFJJTSByZWNlbnRs eS4gSXQgYXBwZWFyZWQgPGJyPg0KJmd0OyZndDsgdGhhdCBhdCBsZWFzdCBzb21lIGNvdW50ZXJz IGFyZSBkcml2ZXIgYW5kIG9wZXJhdGlvbiBzcGVjaWZpYy4gRm9yIDxicj4NCiZndDsmZ3Q7IGV4 YW1wbGUsIHRoZSBkYSBkcml2ZXIgYXBwZWFycyB0byB1cGRhdGUgY291bnRlcnMgaW4gc29tZSBw YXRocyBidXQgPGJyPg0KJmd0OyZndDsgbm90IG90aGVycy4gSSBhc3N1bWUgdGhhdCBhZGEgaXMg ZGlmZmVyZW50LiBUaGVyZSBpcyBhIGJ1ZyByZXBvcnQgZm9yIDxicj4NCiZndDsmZ3Q7IGRhLCBi dXQgaGF2ZW4mIzM5O3Qgc2VlbiBhbnkgZmVlZGJhY2sgLi4uIDxicj4NCiZndDs8YnI+DQomZ3Q7 IDxhIGhyZWY9Imh0dHBzOi8vYnVncy5mcmVlYnNkLm9yZy9idWd6aWxsYS9zaG93X2J1Zy5jZ2k/ aWQ9Mjc3NjczIiByZWw9Im5vcmVmZXJyZXIiIHRhcmdldD0iX2JsYW5rIj5odHRwczovL2J1Z3Mu ZnJlZWJzZC5vcmcvYnVnemlsbGEvc2hvd19idWcuY2dpP2lkPTI3NzY3MzwvYT48YnI+DQomZ3Q7 PGJyPg0KJmd0OyBZb3UgY291bGQgdHJ5IHRvIHJ1biBnc3RhdCB3aXRoIHRoZSAtZCBmbGFnIGR1 cmluZyB0aGUgdGltZSBwZXJpb2QgPGJyPg0KJmd0OyB3aGVuIHRoZSBkZWxldGUgb3BlcmF0aW9u cyBhcmUgZXhwZWN0ZWQgdG8gb2NjdXIuIFRoYXQgc2hvdWxkIGdpdmUgeW91IDxicj4NCiZndDsg YW4gaWRlYSBvZiB3aGF0JiMzOTtzIGhhcHBlbmluZyBhdCB0aGUgZGlzayBsZXZlbCBpbiByZWFs IHRpbWUgYnV0IG1heSBub3QgPGJyPg0KJmd0OyBvZmZlciBtb3JlIGluZm8gdGhhbiB5b3UmIzM5 O3JlIGFscmVhZHkgc2VlaW5nLjxicj4NCiZndDs8YnI+DQo8YnI+DQpJdCAqc2VlbXMqIHRvIGJl IGRvaW5nIHNvbWV0aGluZy7CoCBXaGF0IEkgZG9udCB1bmRlcnN0YW5kIGlzIHdoeSBpZiBJIDxi cj4NCnJ1biBpdCBvbmNlLCBkbyBub3RoaW5nIChubyB3cml0ZXMgLyBzbmFwc2hvdHMgZXRjKSwg YW5kIHRoZW4gcnVuIHRyaW0gPGJyPg0KYWdhaW4sIGl0IHNlZW1zIHRvIGJlIGRvaW5nIHNvbWV0 aGluZyB3aXRoIGdzdGF0IGV2ZW4gdGhvdWdoIHRoZXJlIDxicj4NCnNob3VsZCBub3QgYmUgYW55 dGhpbmcgdG8gbWFyayBhcyBiZWluZyB0cmltbWVkID88YnI+DQo8YnI+DQpkVDogMS4wMDJzwqAg dzogMS4wMDBzPGJyPg0KwqDCoEwocSnCoCBvcHMvc8KgwqDCoCByL3PCoMKgIGtCcHPCoMKgIG1z L3LCoMKgwqAgdy9zwqDCoCBrQnBzwqDCoCBtcy93wqDCoMKgIGQvcyBrQnBzwqDCoCA8YnI+DQpt cy9kwqDCoCAlYnVzeSBOYW1lPGJyPg0KwqDCoMKgwqAgMMKgwqAgMTI1NMKgwqDCoMKgwqAgMMKg wqDCoMKgwqAgMMKgwqDCoCAwLjDCoMKgwqAgOTg2wqDCoCA1MjAywqDCoMKgIDIuMMKgwqDCoCAy NDQgPGJyPg0KODM2MjczM8KgwqDCoCA0LjXCoMKgIDU1LjbCoCBhZGEwPGJyPg0KwqDCoMKgIDEy wqDCoCAxMjQywqDCoMKgwqDCoCAwwqDCoMKgwqDCoCAwwqDCoMKgIDAuMMKgwqAgMTAxMsKgwqAg NTIxOMKgwqDCoCAxLjnCoMKgwqAgMjA2IDxicj4NCjQ5NzIwNDHCoMKgwqAgNi4wwqDCoCA2My4z wqAgYWRhMjxicj4NCsKgwqDCoCAxMsKgwqAgMTI0MsKgwqDCoMKgwqAgMMKgwqDCoMKgwqAgMMKg wqDCoCAwLjDCoMKgIDEwMTLCoMKgIDUyMTjCoMKgwqAgMS45wqDCoMKgIDIwNiA8YnI+DQo0OTcy MDQxwqDCoMKgIDYuMMKgwqAgNjMuM8KgIGFkYTJwMTxicj4NCsKgwqDCoMKgIDDCoMKgIDQzMTPC oMKgwqDCoMKgIDDCoMKgwqDCoMKgIDDCoMKgwqAgMC4wwqDCoCAxMDI0wqDCoCA1MTkwwqDCoMKg IDAuOMKgwqAgMzI2NiA8YnI+DQo2NDYzODE1wqDCoMKgIDAuNMKgwqAgNjIuOMKgIGFkYTM8YnI+ DQrCoMKgwqDCoCAwwqDCoCAxMjU0wqDCoMKgwqDCoCAwwqDCoMKgwqDCoCAwwqDCoMKgIDAuMMKg wqDCoCA5ODbCoMKgIDUyMDLCoMKgwqAgMi4wwqDCoMKgIDI0NCA8YnI+DQo4MzYyNzMzwqDCoMKg IDQuNcKgwqAgNTUuNsKgIGFkYTBwMTxicj4NCsKgwqDCoMKgIDDCoMKgIDQyMzjCoMKgwqDCoMKg IDDCoMKgwqDCoMKgIDDCoMKgwqAgMC4wwqDCoMKgIDk2MMKgwqAgNDg3NMKgwqDCoCAwLjfCoMKg IDMyNTQgPGJyPg0KNjI4MDM2MsKgwqDCoCAwLjTCoMKgIDU5LjjCoCBhZGE1PGJyPg0KwqDCoMKg wqAgMMKgwqAgNDMxM8KgwqDCoMKgwqAgMMKgwqDCoMKgwqAgMMKgwqDCoCAwLjDCoMKgIDEwMjTC oMKgIDUxOTDCoMKgwqAgMC44wqDCoCAzMjY2IDxicj4NCjY0NjM4MTXCoMKgwqAgMC40wqDCoCA2 Mi44wqAgYWRhM3AxPGJyPg0KwqDCoMKgwqAgMMKgwqAgNDIzOMKgwqDCoMKgwqAgMMKgwqDCoMKg wqAgMMKgwqDCoCAwLjDCoMKgwqAgOTYwwqDCoCA0ODc0wqDCoMKgIDAuN8KgwqAgMzI1NCA8YnI+ DQo2MjgwMzYywqDCoMKgIDAuNMKgwqAgNTkuOMKgIGFkYTVwMTxicj4NCmRUOiAxLjAwMXPCoCB3 OiAxLjAwMHM8YnI+DQrCoMKgTChxKcKgIG9wcy9zwqDCoMKgIHIvc8KgwqAga0Jwc8KgwqAgbXMv csKgwqDCoCB3L3PCoMKgIGtCcHPCoMKgIG1zL3fCoMKgwqAgZC9zIGtCcHPCoMKgIDxicj4NCm1z L2TCoMKgICVidXN5IE5hbWU8YnI+DQrCoMKgwqDCoCAywqDCoCAyMzgxwqDCoMKgwqDCoCAwwqDC oMKgwqDCoCAwwqDCoMKgIDAuMMKgwqAgMTU4MMKgwqAgOTk0NsKgwqDCoCAwLjnCoMKgwqAgNzY3 IDxicj4NCjU5OTAyODbCoMKgwqAgMS44wqDCoCA3MC4wwqAgYWRhMDxicj4NCsKgwqDCoMKgIDLC oMKgIDI4MDHCoMKgwqDCoMKgIDDCoMKgwqDCoMKgIDDCoMKgwqAgMC4wwqDCoCAxNTQwwqDCoCA5 NzgywqDCoMKgIDAuOcKgwqAgMTIyNyA8YnI+DQoxMTkzNjUxMMKgwqDCoCAxLjDCoMKgIDY1LjLC oCBhZGEyPGJyPg0KwqDCoMKgwqAgMsKgwqAgMjgwMcKgwqDCoMKgwqAgMMKgwqDCoMKgwqAgMMKg wqDCoCAwLjDCoMKgIDE1NDDCoMKgIDk3ODLCoMKgwqAgMC45wqDCoCAxMjI3IDxicj4NCjExOTM2 NTEwwqDCoMKgIDEuMMKgwqAgNjUuMsKgIGFkYTJwMTxicj4NCsKgwqDCoMKgIDDCoMKgIDIwNzLC oMKgwqDCoMKgIDDCoMKgwqDCoMKgIDDCoMKgwqAgMC4wwqDCoCAxNTI5wqDCoCA5NTY2wqDCoMKg IDAuOMKgwqDCoCA1MDkgPGJyPg0KMTI1NDk1ODfCoMKgwqAgMi4xwqDCoCA1Ny4wwqAgYWRhMzxi cj4NCsKgwqDCoMKgIDLCoMKgIDIzODHCoMKgwqDCoMKgIDDCoMKgwqDCoMKgIDDCoMKgwqAgMC4w wqDCoCAxNTgwwqDCoCA5OTQ2wqDCoMKgIDAuOcKgwqDCoCA3NjcgPGJyPg0KNTk5MDI4NsKgwqDC oCAxLjjCoMKgIDcwLjDCoCBhZGEwcDE8YnI+DQrCoMKgwqDCoCAwwqDCoCAyMDQywqDCoMKgwqDC oCAwwqDCoMKgwqDCoCAwwqDCoMKgIDAuMMKgwqAgMTUxN8KgwqAgOTQyN8KgwqDCoCAwLjbCoMKg wqAgNDkxIDxicj4NCjEyNTQ5NTM1wqDCoMKgIDEuOcKgwqAgNTIuNMKgIGFkYTU8YnI+DQrCoMKg wqDCoCAwwqDCoCAyMDcywqDCoMKgwqDCoCAwwqDCoMKgwqDCoCAwwqDCoMKgIDAuMMKgwqAgMTUy OcKgwqAgOTU2NsKgwqDCoCAwLjjCoMKgwqAgNTA5IDxicj4NCjEyNTQ5NTg3wqDCoMKgIDIuMcKg wqAgNTcuMMKgIGFkYTNwMTxicj4NCsKgwqDCoMKgIDDCoMKgIDIwNDLCoMKgwqDCoMKgIDDCoMKg wqDCoMKgIDDCoMKgwqAgMC4wwqDCoCAxNTE3wqDCoCA5NDI3wqDCoMKgIDAuNsKgwqDCoCA0OTEg PGJyPg0KMTI1NDk1MzXCoMKgwqAgMS45wqDCoCA1Mi40wqAgYWRhNXAxPGJyPg0KZFQ6IDEuMDAy c8KgIHc6IDEuMDAwczxicj4NCsKgwqBMKHEpwqAgb3BzL3PCoMKgwqAgci9zwqDCoCBrQnBzwqDC oCBtcy9ywqDCoMKgIHcvc8KgwqAga0Jwc8KgwqAgbXMvd8KgwqDCoCBkL3Mga0Jwc8KgwqAgPGJy Pg0KbXMvZMKgwqAgJWJ1c3kgTmFtZTxicj4NCsKgwqDCoMKgIDLCoMKgIDE5NDnCoMKgwqDCoMKg IDDCoMKgwqDCoMKgIDDCoMKgwqAgMC4wwqDCoCAxMDk0wqDCoCA1OTI2wqDCoMKgIDEuMsKgwqDC oCA4MjcgPGJyPg0KMTEyNjcyMDDCoMKgwqAgMS44wqDCoCA3OC44wqAgYWRhMDxicj4NCsKgwqDC oMKgIDDCoMKgIDIwODPCoMKgwqDCoMKgIDDCoMKgwqDCoMKgIDDCoMKgwqAgMC4wwqDCoCAxMTE1 wqDCoCA2MDM0wqDCoMKgIDAuN8KgwqDCoCA5MzkgPGJyPg0KMTY1Mzc5ODHCoMKgwqAgMS40wqDC oCA2Ny4ywqAgYWRhMjxicj4NCsKgwqDCoMKgIDDCoMKgIDIwODPCoMKgwqDCoMKgIDDCoMKgwqDC oMKgIDDCoMKgwqAgMC4wwqDCoCAxMTE1wqDCoCA2MDM0wqDCoMKgIDAuN8KgwqDCoCA5MzkgPGJy Pg0KMTY1Mzc5ODHCoMKgwqAgMS40wqDCoCA2Ny4ywqAgYWRhMnAxPGJyPg0KwqDCoMKgwqAgMsKg wqAgMjUyNcKgwqDCoMKgwqAgMMKgwqDCoMKgwqAgMMKgwqDCoCAwLjDCoMKgIDEwOTjCoMKgIDU5 MTTCoMKgwqAgMC44wqDCoCAxMzk5IDxicj4NCjE2MDIxNjE1wqDCoMKgIDEuMcKgwqAgNzkuM8Kg IGFkYTM8YnI+DQrCoMKgwqDCoCAywqDCoCAxOTQ5wqDCoMKgwqDCoCAwwqDCoMKgwqDCoCAwwqDC oMKgIDAuMMKgwqAgMTA5NMKgwqAgNTkyNsKgwqDCoCAxLjLCoMKgwqAgODI3IDxicj4NCjExMjY3 MjAwwqDCoMKgIDEuOMKgwqAgNzguOMKgIGFkYTBwMTxicj4NCsKgwqDCoCAxMsKgwqAgMjQ3McKg wqDCoMKgwqAgMMKgwqDCoMKgwqAgMMKgwqDCoCAwLjDCoMKgIDEwMTjCoMKgIDUzOTnCoMKgwqAg MS4wwqDCoCAxNDI1IDxicj4NCjE1Mzk1NTY2wqDCoMKgIDEuMcKgwqAgODAuNcKgIGFkYTU8YnI+ DQrCoMKgwqDCoCAywqDCoCAyNTI1wqDCoMKgwqDCoCAwwqDCoMKgwqDCoCAwwqDCoMKgIDAuMMKg wqAgMTA5OMKgwqAgNTkxNMKgwqDCoCAwLjjCoMKgIDEzOTkgPGJyPg0KMTYwMjE2MTXCoMKgwqAg MS4xwqDCoCA3OS4zwqAgYWRhM3AxPGJyPg0KwqDCoMKgIDEywqDCoCAyNDcxwqDCoMKgwqDCoCAw wqDCoMKgwqDCoCAwwqDCoMKgIDAuMMKgwqAgMTAxOMKgwqAgNTM5OcKgwqDCoCAxLjDCoMKgIDE0 MjUgPGJyPg0KMTUzOTU1NjbCoMKgwqAgMS4xwqDCoCA4MC41wqAgYWRhNXAxPGJyPg0KPGJyPg0K VGhlIHVsdGltYXRlIHByb2JsZW0gaXMgdGhhdCBhZnRlciBhIHdoaWxlIHdpdGggYSBsb3Qgb2Yg d3JpdGVzLCB0aGUgPGJyPg0KZGlzayBwZXJmb3JtYW5jZSB3aWxsIGJlIHRvYXN0IHVudGlsIEkg ZG8gYSBtYW51YWwgdHJpbSAtZiBvZiB0aGUgZGlzayA8YnI+DQo6KMKgwqAgdGhpcyBpcyBtb3N0 IG5vdGFibGUgb24gY29uc3VtZXIgV0QgU1NEcy7CoCBJIGhhdmVudCBkb25lIGFueSA8YnI+DQpl eHRlbnNpdmUgdGVzdHMgd2l0aCBTYW1zdW5nIFNTRHMgdG8gc2VlIGlmIHRoZXJlIGFyZSBwZXJm b3JtYW5jZSA8YnI+DQpwZW5hbHRpZXMgb3Igbm90LiBJdCBtaWdodCBiZSB0aGF0IHRoZXkgYXJl IGp1c3QgYmV0dGVyIGF0IG1hc2tpbmcgdGhlIDxicj4NCnByb2JsZW0uwqAgSSBkb250IHNlZSB0 aGUgc2FtZSBpc3N1ZSB3aXRoIFpGUyBvbiBMaW51eCB3aXRoIHRoZSBzYW1lIDxicj4NCmRpc2tz IC8gaGFyZHdhcmU8YnI+PC9ibG9ja3F1b3RlPjxkaXY+PGJyPjwvZGl2PjxkaXY+V2hlbiB0cmlt cyBhcmUgZmFzdCwgeW91IHdhbnQgdG8gc2VuZCB0aGVtIHRvIHRoZSBkcml2ZSBhcyBzb29uIGFz IHlvdTwvZGl2PjxkaXY+a25vdyB0aGUgYmxvY2tzIGFyZSBmcmVlZC4gVUZTIGFsd2F5cyBkb2Vz IHRoaXMgKGlmIHRyaW0gaXMgZW5hYmxlZCBhdCBhbGwpLjwvZGl2PjxkaXY+WkZTIGhhcyBhIGxv dCBvZiBrbm9icyB0byBjb250cm9sIHdoZW4gLyBob3cgLyBpZiB0aGlzIGlzIGRvbmUuPC9kaXY+ PGRpdj48YnI+PC9kaXY+PGRpdj48cHJlIGlkPSJnbWFpbC1saW5lMSI+dmZzLnpmcy52ZGV2LnRy aW1fbWluX2FjdGl2ZTogMTxicj52ZnMuemZzLnZkZXYudHJpbV9tYXhfYWN0aXZlOiAyPGJyPnZm cy56ZnMudHJpbS5xdWV1ZV9saW1pdDogMTA8YnI+dmZzLnpmcy50cmltLnR4Z19iYXRjaDogMzI8 YnI+dmZzLnpmcy50cmltLm1ldGFzbGFiX3NraXA6IDA8YnI+dmZzLnpmcy50cmltLmV4dGVudF9i eXRlc19taW46IDMyNzY4PGJyPnZmcy56ZnMudHJpbS5leHRlbnRfYnl0ZXNfbWF4OiAxMzQyMTc3 Mjg8YnI+dmZzLnpmcy5sMmFyYy50cmltX2FoZWFkOiAwPGJyPjxicj48L3ByZT48cHJlIGlkPSJn bWFpbC1saW5lMSI+SSYjMzk7dmUgbm90IHRyaWVkIHRvIHR1bmUgdGhlc2UgaW4gdGhlIHBhc3Qs IGJ1dCB5b3UgY2FuIHNlZSBob3cgdGhleSBhZmZlY3QgdGhpbmdzLjxicj48L3ByZT48cHJlIGlk PSJnbWFpbC1saW5lMSI+PGJyPldhcm5lcjwvcHJlPjwvZGl2PjxkaXY+wqA8L2Rpdj48YmxvY2tx dW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7 Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+ DQpJIGhhdmUgYW4gb3BlbiBQUiBpbiA8YnI+DQo8YSBocmVmPSJodHRwczovL2J1Z3MuZnJlZWJz ZC5vcmcvYnVnemlsbGEvc2hvd19idWcuY2dpP2lkPTI3Nzk5MiIgcmVsPSJub3JlZmVycmVyIiB0 YXJnZXQ9Il9ibGFuayI+aHR0cHM6Ly9idWdzLmZyZWVic2Qub3JnL2J1Z3ppbGxhL3Nob3dfYnVn LmNnaT9pZD0yNzc5OTI8L2E+IHRoYXQgSSB0aGluayA8YnI+DQptaWdodCBhY3R1YWxseSBoYXZl IDIgc2VwYXJhdGUgcHJvYmxlbXMuPGJyPg0KPGJyPg0KwqDCoMKgwqAgLS0tTWlrZTxicj4NCjxi cj4NCjxicj4NCiZndDsmZ3Q7IGUuZy4gaGVyZSB3YXMgb25lIGRpc2sgaW4gdGhlIHBvb2wgdGhh dCB3YXMgdGFraW5nIGEgbG9uZyB0aW1lIGZvciA8YnI+DQomZ3Q7Jmd0OyBlYWNoIHpwb29sIHRy aW08YnI+DQomZ3Q7Jmd0Ozxicj4NCiZndDsmZ3Q7ICMgdGltZSB0cmltIC1mIC9kZXYvYWRhMTxi cj4NCiZndDsmZ3Q7IHRyaW0gL2Rldi9hZGExIG9mZnNldCAwIGxlbmd0aCAxMDAwMjA0ODg2MDE2 PGJyPg0KJmd0OyZndDsgMC4wMDB1IDAuMDU3cyAxOjI5LjMzIDAuMCXCoMKgwqDCoMKgIDUrMTg0 ayAwKzBpbyAwcGYrMHc8YnI+DQomZ3Q7Jmd0OyBhbmQgdGhlbiBpZiBJIHJlLXJ1biBpdDxicj4N CiZndDsmZ3Q7ICPCoCB0aW1lIHRyaW0gLWYgL2Rldi9hZGExPGJyPg0KJmd0OyZndDsgdHJpbSAv ZGV2L2FkYTEgb2Zmc2V0IDAgbGVuZ3RoIDEwMDAyMDQ4ODYwMTY8YnI+DQomZ3Q7Jmd0OyAwLjAw MHUgMC4wNTJzIDA6MDQuMTUgMS4yJcKgwqDCoMKgwqAgMSs1MmsgMCswaW8gMHBmKzB3PGJyPg0K Jmd0OyZndDs8YnI+DQomZ3Q7Jmd0OyA5MCBzZWNvbmRzIGFuZCB0aGVuIDQgc2Vjb25kcyBhZnRl ciB0aGF0Ljxicj4NCiZndDsmZ3Q7PGJyPg0KJmd0Ozxicj4NCiZndDsgLU1hdHRoZXc8YnI+DQom Z3Q7PGJyPg0KPGJyPg0KPC9ibG9ja3F1b3RlPjwvZGl2PjwvZGl2Pg0K --000000000000165b720617793f29--