From owner-freebsd-fs@freebsd.org Sun Jul 12 21:00:36 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C644D99B2C4 for ; Sun, 12 Jul 2015 21:00:36 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9F09F1A36 for ; Sun, 12 Jul 2015 21:00:36 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t6CL0axX057917 for ; Sun, 12 Jul 2015 21:00:36 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201507122100.t6CL0axX057917@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 12 Jul 2015 21:00:36 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jul 2015 21:00:36 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f 3 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Jul 13 11:48:02 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BC77999B7F4; Mon, 13 Jul 2015 11:48:02 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: from mail-qg0-x22f.google.com (mail-qg0-x22f.google.com [IPv6:2607:f8b0:400d:c04::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4E1D317B; Mon, 13 Jul 2015 11:48:02 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: by qgep37 with SMTP id p37so62637918qge.1; Mon, 13 Jul 2015 04:48:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=eiJ+314qHz0yeScIiVjlFL4qp2b73M0Nkg/CnY4Duok=; b=WVLaF+CTYXbfLSy7EiiMYL95Sw6nq/uAhZMrpG45Vr/KOCRpxGtvPu42lLg2xPenc/ uVbCFszb38VhIxOo7A3NxIDWNRklzDBpR651rm8ICXG3vrpq6fHL3C5rPfaEsvCWMC3l D/uoPTdLJrZ5BW7V7+GlMwYPQrzPoq7zI2ZNiDTuNeu/wRQ/ro9tlTwHzHCoJXpLCZSd ZkrOHVZoFG9A3rkPiMOYMB0l2YFK54kFHCSAp66hrII1ywPRO//RXXb8FOdpRFSHFNq8 cB4vNfBBSsfzmxBuiSUQbWOhXNoJ8CoFUkwYZIIUM7451m19AlDV1kLXeDdWr0LzkvsT 4xjg== MIME-Version: 1.0 X-Received: by 10.55.20.72 with SMTP id e69mr13924316qkh.63.1436788081309; Mon, 13 Jul 2015 04:48:01 -0700 (PDT) Received: by 10.96.174.39 with HTTP; Mon, 13 Jul 2015 04:48:01 -0700 (PDT) Date: Mon, 13 Jul 2015 08:48:01 -0300 Message-ID: Subject: FreeBSD 10.1 Memory Exhaustion From: Christopher Forgeron To: freebsd-stable@freebsd.org, FreeBSD Filesystems Cc: Adrian Chadd Content-Type: multipart/mixed; boundary=001a1145eae834d0fc051ac04b07 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 11:48:02 -0000 --001a1145eae834d0fc051ac04b07 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable TL;DR Summary: I can run FreeBSD out of memory quite consistently, and it= =E2=80=99s not a TOS/mbuf exhaustion issue. It=E2=80=99s quite possible that ZFS is th= e culprit, but shouldn=E2=80=99t the pager be able to handle aggressive memor= y requests in a low memory situation gracefully, without needing custom tuning of ZFS / VM? Hello, I=E2=80=99ve been dealing with some instability in my 10.1-RELEASE and STABLEr282701M machines for the last few months. These machines are NFS/iSCSI storage machines, running on Dell M610x or similar hardware, 96 Gig Memory, 10Gig Network Cards, dual Xeon Processors =E2=80=93 Fairly beefy stuff. Initially I thought it was more issues with TOS / jumbo mbufs, as I had this problem last year. I had thought that this was properly resolved, but setting my MTU to 1500, and turning off TOS did give me a bit more stability. Currently all my machines are set this way. Crashes were usually represented by loss of network connectivity, and the ctld daemon scrolling messages across the screen at full speed about lost connections. All of this did seem like more network stack problems, but with each crash I=E2=80=99d be able to learn a bit more. Usually there was nothing of any use in the logfile, but every now and then I=E2=80=99d get this: Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory Jun 3 13:02:04 san0 kernel: WARNING: icl_pdu_new: failed to allocate 80 bytes Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory Jun 3 13:02:04 san0 kernel: WARNING: icl_pdu_new: failed to allocate 80 bytes Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory --------- Jun 4 03:03:09 san0 kernel: WARNING: icl_pdu_new: failed to allocate 80 bytes Jun 4 03:03:09 san0 kernel: WARNING: icl_pdu_new: failed to allocate 80 bytes Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 (iqn.1998-01.com.vmware:esx5a-3387a188): connection error; dropping connection Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 (iqn.1998-01.com.vmware:esx5a-3387a188): connection error; dropping connection Jun 4 03:03:10 san0 kernel: WARNING: 172.16.0.97 (iqn.1998-01.com.vmware:esx5a-3387a188): waiting for CTL to terminate tasks, 1 remaining Jun 4 06:04:27 san0 syslogd: kernel boot file is /boot/kernel/kernel So knowing that it seemed to be running out of memory, I started leaving leaving =E2=80=98vmstat 5=E2=80=99 running on a console, to see what it was= displaying during the crash. It was always the same thing: 0 0 0 1520M 4408M 15 0 0 0 25 19 0 0 21962 1667 91390 0 33 67 0 0 0 1520M 4310M 9 0 0 0 2 15 3 0 21527 1385 95165 0 31 69 0 0 0 1520M 4254M 7 0 0 0 14 19 0 0 17664 1739 72873 0 18 82 0 0 0 1520M 4145M 2 0 0 0 0 19 0 0 23557 1447 96941 0 36 64 0 0 0 1520M 4013M 4 0 0 0 14 19 0 0 4288 490 34685 0 72 28 0 0 0 1520M 3885M 2 0 0 0 0 19 0 0 11141 1038 69242 0 52 48 0 0 0 1520M 3803M 10 0 0 0 14 19 0 0 24102 1834 91050 0 33 67 0 0 0 1520M 8192B 2 0 0 0 2 15 1 0 19037 1131 77470 0 45 55 0 0 0 1520M 8192B 0 22 0 0 2 0 6 0 146 82 578 0 0 100 0 0 0 1520M 8192B 1 0 0 0 0 0 0 0 130 40 510 0 0 100 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 143 40 501 0 0 100 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 201 62 660 0 0 100 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 101 28 404 0 0 100 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 97 27 398 0 0 100 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 93 28 377 0 0 100 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 92 27 373 0 0 100 I=E2=80=99d go from a decent amount of free memory to suddenly having none= . Vmstat would stop outputting, console commands would hang, etc. The whole system would be useless. Looking into this, I came across a similar issue; https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199189 I started increasing v.v_free_min, and it helped =E2=80=93 My crashes went = from being ~every 6 hours to every few days. Currently I=E2=80=99m running with vm.v_free_min=3D1254507 =E2=80=93 That= =E2=80=99s (1254507 * 4KiB) , or 4.78GiB of Reserve. The vmstat above is of a machine with that setting still running to 8B of memory. I have two issues here: 1) I don=E2=80=99t think I should ever be able to run the system into the g= round on memory. Deny me new memory until the pager can free more. 2) Setting =E2=80=98min=E2=80=99 doesn=E2=80=99t really mean =E2=80=98min= =E2=80=99 as it can obviously go below that threshold. I have plenty of local UFS swap (non-ZFS drives) Adrian requested that I output a few more diagnostic items, and this is what I=E2=80=99m running on a console now, in a loop: vmstat netstat -m vmstat -z sleep 1 The output of four crashes are attached here, as they can be a bit long. Let me know if that=E2=80=99s not a good way to report them. They will each= start mid-way through a vmstat =E2=80=93z output, as that=E2=80=99s as far back a= s my terminal buffer allows. Now, I have a good idea of the conditions that are causing this: ZFS Snapshots, run by cron, during times of high ZFS writes. The crashes are all nearly on the hour, as that=E2=80=99s when crontab trig= gers my python scripts to make new snapshots, and delete old ones. My average FreeBSD machine has ~ 30 zfs datasets, with each pool having ~20 TiB used. These all need to snapshot on the hour. By staggering the snapshots by a few minutes, I have been able to reduce crashing from every other day to perhaps once a week if I=E2=80=99m lucky = =E2=80=93 But if I start moving a lot of data around, I can cause daily crashes again. It=E2=80=99s looking to be the memory demand of snapshotting lots of ZFS da= tasets at the same time while accepting a lot of write traffic. Now perhaps the answer is =E2=80=98don=E2=80=99t do that=E2=80=99 but I fee= l that FreeBSD should be robust enough to handle this. I don=E2=80=99t mind tuning for now to reduce/eliminate this, but others shouldn=E2=80=99t run into this pain just= because they heavily load their machines =E2=80=93 There must be a way of avoiding = this condition. Here are the contents of my /boot/loader.conf and sysctl.conf, so show my minimal tuning to make this problem a little more bearable: /boot/loader.conf vfs.zfs.arc_meta_limit=3D49656727553 vfs.zfs.arc_max =3D 91489280512 /etc/sysctl.conf vm.v_free_min=3D1254507 Any suggestions/help is appreciated. Thank you. --001a1145eae834d0fc051ac04b07 Content-Type: text/plain; charset=US-ASCII; name="2015-07-03 - San2.txt" Content-Disposition: attachment; filename="2015-07-03 - San2.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_ic1ufftw0 U0FOMiBiZWZvcmUgQ3Jhc2gNCg0KemlvX2RhdGFfYnVmXzU3MzQ0OiAgIDU3MzQ0LCAgICAgIDAs ICAgICAgIDEsICAgICAgMjcsIDI4Nzk1MTcsICAgMCwgICAwDQp6aW9fYnVmXzYxNDQwOiAgICAg ICAgNjE0NDAsICAgICAgMCwgICAgICAgMCwgICAgICA1MywgODIxMTkzMiwgICAwLCAgIDANCnpp b19kYXRhX2J1Zl82MTQ0MDogICA2MTQ0MCwgICAgICAwLCAgICAgICAxLCAgICAgIDIxLCAgICAy MDg5LCAgIDAsICAgMA0KemlvX2J1Zl82NTUzNjogICAgICAgIDY1NTM2LCAgICAgIDAsICAgICA0 NTEsICAgICA2MTQsMzMyNjE2NjgyLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzY1NTM2OiAgIDY1 NTM2LCAgICAgIDAsICAyNzA2ODMsICAgIDEyMTksNDgzOTE1MzUyLCAgIDAsICAgMA0KemlvX2J1 Zl82OTYzMjogICAgICAgIDY5NjMyLCAgICAgIDAsICAgICAgIDAsICAgICAgMjIsIDYyNDQxODEs ICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNjk2MzI6ICAgNjk2MzIsICAgICAgMCwgICAgICAgMSwg ICAgICAyMiwgICAgMjQ0MiwgICAwLCAgIDANCnppb19idWZfNzM3Mjg6ICAgICAgICA3MzcyOCwg ICAgICAwLCAgICAgICAwLCAgICAgIDIyLCA3MTA1NDYzLCAgIDAsICAgMA0KemlvX2RhdGFfYnVm XzczNzI4OiAgIDczNzI4LCAgICAgIDAsICAgICAgIDAsICAgICAgMjUsICAgIDIwNzgsICAgMCwg ICAwDQp6aW9fYnVmXzc3ODI0OiAgICAgICAgNzc4MjQsICAgICAgMCwgICAgICAgMCwgICAgICAy MiwgNTM2MDM3NiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl83NzgyNDogICA3NzgyNCwgICAgICAw LCAgICAgICAwLCAgICAgIDIyLCAgICAyMDc2LCAgIDAsICAgMA0KemlvX2J1Zl84MTkyMDogICAg ICAgIDgxOTIwLCAgICAgIDAsICAgICAgIDAsICAgICAgMjUsIDY5NzYyMzMsICAgMCwgICAwDQp6 aW9fZGF0YV9idWZfODE5MjA6ICAgODE5MjAsICAgICAgMCwgICAgICAgMCwgICAgICAyMCwgICAg MjA3NywgICAwLCAgIDANCnppb19idWZfODYwMTY6ICAgICAgICA4NjAxNiwgICAgICAwLCAgICAg ICAwLCAgICAgIDI3LCA1Mzg3MjYyLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzg2MDE2OiAgIDg2 MDE2LCAgICAgIDAsICAgICAgIDAsICAgICAgMjEsICAgIDIwNzAsICAgMCwgICAwDQp6aW9fYnVm XzkwMTEyOiAgICAgICAgOTAxMTIsICAgICAgMCwgICAgICAgMiwgICAgICAyMSwgNjM2MjU3Miwg ICAwLCAgIDANCnppb19kYXRhX2J1Zl85MDExMjogICA5MDExMiwgICAgICAwLCAgICAgICAxLCAg ICAgIDIzLCAgICAyNTUxLCAgIDAsICAgMA0KemlvX2J1Zl85NDIwODogICAgICAgIDk0MjA4LCAg ICAgIDAsICAgICAgIDEsICAgICAgMjIsIDYwNjcxMDUsICAgMCwgICAwDQp6aW9fZGF0YV9idWZf OTQyMDg6ICAgOTQyMDgsICAgICAgMCwgICAgICAgMCwgICAgICAyNCwgICAgMjExNCwgICAwLCAg IDANCnppb19idWZfOTgzMDQ6ICAgICAgICA5ODMwNCwgICAgICAwLCAgICAgICAxLCAgICAgIDIx LCA4NTg3OTE1LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzk4MzA0OiAgIDk4MzA0LCAgICAgIDAs ICAgICAgIDAsICAgICAgMjEsICAgIDIxMDQsICAgMCwgICAwDQp6aW9fYnVmXzEwMjQwMDogICAg ICAxMDI0MDAsICAgICAgMCwgICAgICAgMCwgICAgICAyNiwgNjU4OTA2MiwgICAwLCAgIDANCnpp b19kYXRhX2J1Zl8xMDI0MDA6IDEwMjQwMCwgICAgICAwLCAgICAgICAwLCAgICAgIDIxLCAgICAy MDYxLCAgIDAsICAgMA0KemlvX2J1Zl8xMDY0OTY6ICAgICAgMTA2NDk2LCAgICAgIDAsICAgICAg IDEsICAgICAgMjMsIDcyODA1NzAsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTA2NDk2OiAxMDY0 OTYsICAgICAgMCwgICAgICAgMCwgICAgICAyNCwgICAgMjAwOSwgICAwLCAgIDANCnppb19idWZf MTEwNTkyOiAgICAgIDExMDU5MiwgICAgICAwLCAgICAgICA2LCAgICAgIDE5LCA3NzY3NjkyLCAg IDAsICAgMA0KemlvX2RhdGFfYnVmXzExMDU5MjogMTEwNTkyLCAgICAgIDAsICAgICAgIDAsICAg ICAgMjAsICAgIDE5MDMsICAgMCwgICAwDQp6aW9fYnVmXzExNDY4ODogICAgICAxMTQ2ODgsICAg ICAgMCwgICAgICAgMSwgICAgICAyNCwgOTExODYxNywgICAwLCAgIDANCnppb19kYXRhX2J1Zl8x MTQ2ODg6IDExNDY4OCwgICAgICAwLCAgICAgICAxLCAgICAgIDIxLCAgICAxOTAxLCAgIDAsICAg MA0KemlvX2J1Zl8xMTg3ODQ6ICAgICAgMTE4Nzg0LCAgICAgIDAsICAgICAgIDEsICAgICAgMjMs IDgyNjg4MTMsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTE4Nzg0OiAxMTg3ODQsICAgICAgMCwg ICAgICAgMCwgICAgICAyMywgICAgMTc0MCwgICAwLCAgIDANCnppb19idWZfMTIyODgwOiAgICAg IDEyMjg4MCwgICAgICAwLCAgICAgICAwLCAgICAgIDMxLDExMjI0OTE4LCAgIDAsICAgMA0Kemlv X2RhdGFfYnVmXzEyMjg4MDogMTIyODgwLCAgICAgIDAsICAgICAgIDAsICAgICAgMjAsICAgIDE2 NjgsICAgMCwgICAwDQp6aW9fYnVmXzEyNjk3NjogICAgICAxMjY5NzYsICAgICAgMCwgICAgICAg MCwgICAgICAyNywxMDcxMzQ5MSwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8xMjY5NzY6IDEyNjk3 NiwgICAgICAwLCAgICAgICAwLCAgICAgIDE5LCAgICAxNjQ4LCAgIDAsICAgMA0KemlvX2J1Zl8x MzEwNzI6ICAgICAgMTMxMDcyLCAgICAgIDAsICAgMTM1MTEsICAgIDEyMDgsMjU0MTkzMDEzLCAg IDAsICAgMA0KemlvX2RhdGFfYnVmXzEzMTA3MjogMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAg ICAgMTQsICAgNTY2OTYsICAgMCwgICAwDQpsejRfY3R4OiAgICAgICAgICAgICAgMTYzODQsICAg ICAgMCwgICAgICAgMCwgICAgICA1MiwxNjU4ODA1NjUxLCAgIDAsICAgMA0Kc2FfY2FjaGU6ICAg ICAgICAgICAgICAgIDgwLCAgICAgIDAsICAyMzQzMDEsICAgODM5NDksIDUwMzM4NzYsICAgMCwg ICAwDQpkbm9kZV90OiAgICAgICAgICAgICAgICA3NTIsICAgICAgMCwgIDMyNTE3MywgICAyMzgy MiwgNzg0NjEyOCwgICAwLCAgIDANCmRtdV9idWZfaW1wbF90OiAgICAgICAgIDIzMiwgICAgICAw LCAyNDIwNjEzLCAxMDkyNzk0LDEwMTg4MzU1MjIsICAgMCwgICAwDQphcmNfYnVmX2hkcl90OiAg ICAgICAgICAyMTYsICAgICAgMCwzMTQ3MDU2MCwgNjU4MDA3Miw3MDQ3NzUxMjYsICAgMCwgICAw DQphcmNfYnVmX3Q6ICAgICAgICAgICAgICAgNzIsICAgICAgMCwgMjMwMzE3NCwgMTQ0NDI1MSw4 NDI5Nzk4NjAsICAgMCwgICAwDQp6aWxfbHdiX2NhY2hlOiAgICAgICAgICAxOTIsICAgICAgMCwg ICAgICAzMiwgICAgMTkyOCwxMTg5Nzg1MDUsICAgMCwgICAwDQp6ZnNfem5vZGVfY2FjaGU6ICAg ICAgICAzNjgsICAgICAgMCwgIDIzNDMwMSwgICAzODEzOSwgNTAzMzg3NiwgICAwLCAgIDANCk1v dW50cG9pbnRzOiAgICAgICAgICAgIDgxNiwgICAgICAwLCAgICAgIDMzLCAgICAgMTM3LCAgICAg IDM3LCAgIDAsICAgMA0Ka3NpZ2luZm86ICAgICAgICAgICAgICAgMTEyLCAgICAgIDAsICAgICAy NjcsICAgIDMxMjgsICAgNTE0MjIsICAgMCwgICAwDQppdGltZXI6ICAgICAgICAgICAgICAgICAz NTIsICAgICAgMCwgICAgICAgMSwgICAgICAxMCwgICAgICAgMSwgICAwLCAgIDANCktOT1RFOiAg ICAgICAgICAgICAgICAgIDEyOCwgICAgICAwLCAgICAgICAwLCAgICAyMTM5LCAgMzg2MDQxLCAg IDAsICAgMA0Kc29ja2V0OiAgICAgICAgICAgICAgICAgNjk2LCAxNTcxNjIwLCAgICAgIDY3LCAg ICAgMjgzLCAgNTUxOTU3LCAgIDAsICAgMA0KaXBxOiAgICAgICAgICAgICAgICAgICAgIDU2LCAg OTU3MDgsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQp1ZHBfaW5wY2I6ICAg ICAgICAgICAgICAzOTIsIDE1NzE2MjAsICAgICAgMTgsICAgICA2MTIsICAyNDMyMTUsICAgMCwg ICAwDQp1ZHBjYjogICAgICAgICAgICAgICAgICAgMTYsIDE1NzE3NjIsICAgICAgMTgsICAgIDUw MDIsICAyNDMyMTUsICAgMCwgICAwDQp0Y3BfaW5wY2I6ICAgICAgICAgICAgICAzOTIsIDE1NzE2 MjAsICAgICAgMzMsICAgICA1OTcsICAyOTU2MTksICAgMCwgICAwDQp0Y3BjYjogICAgICAgICAg ICAgICAgIDEwMjQsIDE1NzE2MjAsICAgICAgMzMsICAgICAyODMsICAyOTU2MTksICAgMCwgICAw DQp0Y3B0dzogICAgICAgICAgICAgICAgICAgODgsICAyNzgxMCwgICAgICAgMCwgICAgMTY2NSwg ICAgNTA1OSwgICAwLCAgIDANCnN5bmNhY2hlOiAgICAgICAgICAgICAgIDE2MCwgIDE1Mzc1LCAg ICAgICAwLCAgICAgNTAwLCAgMjc5MzMxLCAgIDAsICAgMA0KaG9zdGNhY2hlOiAgICAgICAgICAg ICAgMTM2LCAgMTUzNzAsICAgICAgIDIsICAgICAzNzUsICAgICAgODUsICAgMCwgICAwDQp0Y3By ZWFzczogICAgICAgICAgICAgICAgNDAsIDE5MTQwMCwgICAgICAgMCwgICAgMjEwMCwgICAgOTUw NCwgICAwLCAgIDANCnNhY2tob2xlOiAgICAgICAgICAgICAgICAzMiwgICAgICAwLCAgICAgICAw LCAgICAyMTI1LCAgMTA3NTI4LCAgIDAsICAgMA0Kc2N0cF9lcDogICAgICAgICAgICAgICAxNDA4 LCAxNTcxNjIwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0Kc2N0cF9hc29j OiAgICAgICAgICAgICAyNDE2LCAgNDAwMDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAg MCwgICAwDQpzY3RwX2xhZGRyOiAgICAgICAgICAgICAgNDgsICA4MDAxMiwgICAgICAgMCwgICAg ICAgMCwgICAgICAgNCwgICAwLCAgIDANCnNjdHBfcmFkZHI6ICAgICAgICAgICAgIDcyOCwgIDgw MDAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0Kc2N0cF9jaHVuazogICAg ICAgICAgICAgMTM2LCA0MDAwMjYsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAw DQpzY3RwX3JlYWRxOiAgICAgICAgICAgICAxMDQsIDQwMDAyNiwgICAgICAgMCwgICAgICAgMCwg ICAgICAgMCwgICAwLCAgIDANCnNjdHBfc3RyZWFtX21zZ19vdXQ6ICAgIDEwNCwgNDAwMDI2LCAg ICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0Kc2N0cF9hc2NvbmY6ICAgICAgICAg ICAgIDQwLCA0MDAwMDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpzY3Rw X2FzY29uZl9hY2s6ICAgICAgICAgNDgsIDQwMDA2MCwgICAgICAgMCwgICAgICAgMCwgICAgICAg MCwgICAwLCAgIDANCnVkcGxpdGVfaW5wY2I6ICAgICAgICAgIDM5MiwgMTU3MTYyMCwgICAgICAg MCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnJpcGNiOiAgICAgICAgICAgICAgICAgIDM5 MiwgMTU3MTYyMCwgICAgICAgMCwgICAgIDE1MCwgICAgICAxNywgICAwLCAgIDANCnVucGNiOiAg ICAgICAgICAgICAgICAgIDI0MCwgMTU3MTYzMiwgICAgICAxNSwgICAgIDkyOSwgICAxMzEwMSwg ICAwLCAgIDANCnJ0ZW50cnk6ICAgICAgICAgICAgICAgIDIwMCwgICAgICAwLCAgICAgIDE0LCAg ICAgMTQ2LCAgICAgIDE0LCAgIDAsICAgMA0Kc2VsZmQ6ICAgICAgICAgICAgICAgICAgIDU2LCAg ICAgIDAsICAgICA1MTksICAgIDQzMDksMjU2NDQyMzIsICAgMCwgICAwDQpTV0FQTUVUQTogICAg ICAgICAgICAgICAyODgsIDYxMjM4MDYsICAgICA4ODQsICAgIDIyODgsICA0OTQzODcsICAgMCwg ICAwDQppY2xfY29ubjogICAgICAgICAgICAgICAxNTIsICAgICAgMCwgICAgICAgOCwgICAgIDMz MCwgICAgICA4MiwgICAwLCAgIDANCmljbF9wZHU6ICAgICAgICAgICAgICAgICA4MCwgICAgICAw LCAgICAgICAxLCAgICAzMDk5LDYxOTQzMjI0MiwgICAwLCAgIDANCmlzY3NpX291dHN0YW5kaW5n OiAgICAgICA0MCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0K Y2ZpX2x1bjogICAgICAgICAgICAgICAgMzQ0LCAgICAgIDAsICAgICAgIDksICAgICAgNDYsICAg ICAgMTEsICAgMCwgICAwDQpjZmlfbWV0YXRhc2s6ICAgICAgICAgICAzMjAsICAgICAgMCwgICAg ICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCmNmaXNjc2lfZGF0YV93YWl0OiAgICAg ICA1NiwgICAgICAwLCAgICAgICAwLCAgICA1MzI1LDIzOTY1NTI3MiwgICAwLCAgIDANCmJlaW86 ICAgICAgICAgICAgICAgICAgIDM2MCwgICAgICAwLCAgICAgICAxLCAgICAgNTM4LDI4NDQwOTAz NCwgICAwLCAgIDANCmNibGsyOiAgICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAw LCAgICAgIDI3LCAzMzQ0OTM1LCAgIDAsICAgMA0KY2JsazM6ICAgICAgICAgICAgICAgMTMxMDcy LCAgICAgIDAsICAgICAgIDAsICAgICAgMjMsICAgIDI3NTcsICAgMCwgICAwDQpjYmxrNDogICAg ICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICA1NSw0NjE1MTkxNywgICAw LCAgIDANCmNibGs1OiAgICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAg IDMzLDkwOTYxOTA3LCAgIDAsICAgMA0KY2JsazY6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAg IDAsICAgICAgIDAsICAgICAgMTcsMzUyNjQzODYsICAgMCwgICAwDQpjYmxrNzogICAgICAgICAg ICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICAyOCw1Njk4OTM1OSwgICAwLCAgIDAN CmNibGs4OiAgICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDMzLDM3 NzQ3MTIzLCAgIDAsICAgMA0KY2Jsazk6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAg ICAgIDEsICAgICAgMjYsIDcwOTgxNDYsICAgMCwgICAwDQpjYmxrMTA6ICAgICAgICAgICAgICAx MzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICAyNiwgOTU1NjkzNSwgICAwLCAgIDANCg0KIHBy b2NzICAgICAgbWVtb3J5ICAgICAgcGFnZSAgICAgICAgICAgICAgICAgICAgZGlza3MgICAgIGZh dWx0cyAgICAgICAgIGNwdQ0KIHIgYiB3ICAgICBhdm0gICAgZnJlICAgZmx0ICByZSAgcGkgIHBv ICAgIGZyICBzciBtZjAgbWYxICAgaW4gICBzeSAgIGNzIHVzIHN5IGlkDQogOCAwIDcgICAxMzI4 TSAgMjA1NE0gICA2MDYgICAwICAgMCAgIDAgIDIyODUgIDEwICAgMCAgIDAgIDgzNiAgNjE1IDI0 MzggIDAgIDQgOTYNCjE5MjM3LzE0MjEzLzMzNDUwIG1idWZzIGluIHVzZSAoY3VycmVudC9jYWNo ZS90b3RhbCkNCjE4NzE0LzgxNzAvMjY4ODQvMzA2MTkwMCBtYnVmIGNsdXN0ZXJzIGluIHVzZSAo Y3VycmVudC9jYWNoZS90b3RhbC9tYXgpDQoxODY0OC82Mzc0IG1idWYrY2x1c3RlcnMgb3V0IG9m IHBhY2tldCBzZWNvbmRhcnkgem9uZSBpbiB1c2UgKGN1cnJlbnQvY2FjaGUpDQowLzExOS8xMTkv MTUzMDk1MCA0ayAocGFnZSBzaXplKSBqdW1ibyBjbHVzdGVycyBpbiB1c2UgKGN1cnJlbnQvY2Fj aGUvdG90YWwvbWF4KQ0KMC8wLzAvNDUzNjE0IDlrIGp1bWJvIGNsdXN0ZXJzIGluIHVzZSAoY3Vy cmVudC9jYWNoZS90b3RhbC9tYXgpDQowLzAvMC8yNTUxNTggMTZrIGp1bWJvIGNsdXN0ZXJzIGlu IHVzZSAoY3VycmVudC9jYWNoZS90b3RhbC9tYXgpDQo0MjIzN0svMjAzNjlLLzYyNjA2SyBieXRl cyBhbGxvY2F0ZWQgdG8gbmV0d29yayAoY3VycmVudC9jYWNoZS90b3RhbCkNCjAvMC8wIHJlcXVl c3RzIGZvciBtYnVmcyBkZW5pZWQgKG1idWZzL2NsdXN0ZXJzL21idWYrY2x1c3RlcnMpDQowLzAv MCByZXF1ZXN0cyBmb3IgbWJ1ZnMgZGVsYXllZCAobWJ1ZnMvY2x1c3RlcnMvbWJ1ZitjbHVzdGVy cykNCjAvMC8wIHJlcXVlc3RzIGZvciBqdW1ibyBjbHVzdGVycyBkZWxheWVkICg0ay85ay8xNmsp DQowLzAvMCByZXF1ZXN0cyBmb3IganVtYm8gY2x1c3RlcnMgZGVuaWVkICg0ay85ay8xNmspDQow IHJlcXVlc3RzIGZvciBzZmJ1ZnMgZGVuaWVkDQowIHJlcXVlc3RzIGZvciBzZmJ1ZnMgZGVsYXll ZA0KMCByZXF1ZXN0cyBmb3IgSS9PIGluaXRpYXRlZCBieSBzZW5kZmlsZQ0KSVRFTSAgICAgICAg ICAgICAgICAgICBTSVpFICBMSU1JVCAgICAgVVNFRCAgICAgRlJFRSAgICAgIFJFUSBGQUlMIFNM RUVQDQoNClVNQSBLZWdzOiAgICAgICAgICAgICAgIDM4NCwgICAgICAwLCAgICAgMjE3LCAgICAg ICAzLCAgICAgMjE5LCAgIDAsICAgMA0KVU1BIFpvbmVzOiAgICAgICAgICAgICAyNjg4LCAgICAg IDAsICAgICAyMTcsICAgICAgIDAsICAgICAyMTksICAgMCwgICAwDQpVTUEgU2xhYnM6ICAgICAg ICAgICAgICAgODAsICAgICAgMCwgMjY0OTYwMiwgMTM5NTg5OCw0MzE4ODIzMzIsICAgMCwgICAw DQpVTUEgUkNudFNsYWJzOiAgICAgICAgICAgODgsICAgICAgMCwgICAxMzU2MSwgICAgNTE1OSwg MTg4MjQzNywgICAwLCAgIDANClVNQSBIYXNoOiAgICAgICAgICAgICAgIDI1NiwgICAgICAwLCAg ICAgIDIwLCAgICAgIDU1LCAgICAgIDkyLCAgIDAsICAgMA0KNCBCdWNrZXQ6ICAgICAgICAgICAg ICAgIDMyLCAgICAgIDAsICAgIDMxMTEsICAgMTMzODksMjU2MzYzNTM4LCAgIDAsICAgMA0KNiBC dWNrZXQ6ICAgICAgICAgICAgICAgIDQ4LCAgICAgIDAsICAgIDExMzIsICAgIDgyNDcsNjMxODAz NzMsICAgMCwgICAwDQo4IEJ1Y2tldDogICAgICAgICAgICAgICAgNjQsICAgICAgMCwgICAgIDM1 MCwgICAgODA4Miw2OTMyNzA2NiwgIDExLCAgIDANCjEyIEJ1Y2tldDogICAgICAgICAgICAgICA5 NiwgICAgICAwLCAgICAgNzQ0LCAgICAzNDc5LDgxMjA4NjQ3LDY0NTgsICAgMA0KMTYgQnVja2V0 OiAgICAgICAgICAgICAgMTI4LCAgICAgIDAsICAgICAzOTQsICAgIDUwNjIsNTMyODYyNzYsICAg MCwgICAwDQozMiBCdWNrZXQ6ICAgICAgICAgICAgICAyNTYsICAgICAgMCwgICAgIDM1NCwgICAg MjE2Niw2ODkyMzgwOSwgIDYzLCAgIDANCjY0IEJ1Y2tldDogICAgICAgICAgICAgIDUxMiwgICAg ICAwLCAgICAgNjM4LCAgICAgNzk0LDU3NTkxNjkzLDM1NjMwNTUsICAgMA0KMTI4IEJ1Y2tldDog ICAgICAgICAgICAxMDI0LCAgICAgIDAsICAgICA2NDIsICAgICAyNTQsNDc1NzQzNzIsNjUyNzMs ICAgMA0KMjU2IEJ1Y2tldDogICAgICAgICAgICAyMDQ4LCAgICAgIDAsICAgIDg0MzksICAgICAz NDUsNTAzNzU5MTY4LDcwMzk1LCAgIDANCnZtZW0gYnRhZzogICAgICAgICAgICAgICA1NiwgICAg ICAwLCAgNjI3MDkyLCAgMTU2ODkwLDE3NTU1MDcxLDU1MzUsICAgMA0KVk0gT0JKRUNUOiAgICAg ICAgICAgICAgMjU2LCAgICAgIDAsICAgNTQ2OTAsICAgMjIyMzAsMTk1ODY3NDksICAgMCwgICAw DQpSQURJWCBOT0RFOiAgICAgICAgICAgICAxNDQsICAgICAgMCwgIDQ1MjMwMSwgIDIxOTk5OSwx MTg0MzQ2NzgsICAgMCwgICAwDQpNQVA6ICAgICAgICAgICAgICAgICAgICAyNDAsICAgICAgMCwg ICAgICAgMywgICAgICA2MSwgICAgICAgMywgICAwLCAgIDANCktNQVAgRU5UUlk6ICAgICAgICAg ICAgIDEyOCwgICAgICAwLCAgICAgICA4LCAgICAgMTE2LCAgICAgICA4LCAgIDAsICAgMA0KTUFQ IEVOVFJZOiAgICAgICAgICAgICAgMTI4LCAgICAgIDAsICAgIDE0MzksICAgIDI3MTUsNTQ1MjYy MDQsICAgMCwgICAwDQpWTVNQQUNFOiAgICAgICAgICAgICAgICA0NDgsICAgICAgMCwgICAgICA0 MSwgICAgIDYxNiwgMTMzNzczNSwgICAwLCAgIDANCmZha2VwZzogICAgICAgICAgICAgICAgIDEw NCwgICAgICAwLCAgICAgICAwLCAgICAgMTUyLCAgICAgIDI1LCAgIDAsICAgMA0KbXRfem9uZTog ICAgICAgICAgICAgICA0MTEyLCAgICAgIDAsICAgICAzNjksICAgICAgIDAsICAgICAzNjksICAg MCwgICAwDQoxNjogICAgICAgICAgICAgICAgICAgICAgMTYsICAgICAgMCwgIDExMDUxMywgICAx NDIzNCwyMDA2Mjg4MDY1LCAgIDAsICAgMA0KMzI6ICAgICAgICAgICAgICAgICAgICAgIDMyLCAg ICAgIDAsNTgxMjI4NDMsMTE2NTI2NTcsNTAzNzU3MzQwOCwgICAwLCAgIDANCjY0OiAgICAgICAg ICAgICAgICAgICAgICA2NCwgICAgICAwLCAxMzU3MDM2LCAgNjIyNTAwLDEyODQ1NDg4NTQ3LCAg IDAsICAgMA0KMTI4OiAgICAgICAgICAgICAgICAgICAgMTI4LCAgICAgIDAsIDIwMTI0ODEsICA4 MzQ2MjEsMTEwMzIwMjgyMzIsICAgMCwgICAwDQoyNTY6ICAgICAgICAgICAgICAgICAgICAyNTYs ICAgICAgMCwgICA1NDk1MCwgIDExOTAyMCw4MTMwMTQyMTU1LCAgIDAsICAgMA0KNTEyOiAgICAg ICAgICAgICAgICAgICAgNTEyLCAgICAgIDAsICA0MTgzMTQsICAgICAzNTgsMjAxODUyMzMxOSwg ICAwLCAgIDANCjEwMjQ6ICAgICAgICAgICAgICAgICAgMTAyNCwgICAgICAwLCAgIDE1OTgzLCAg ICAgMjg5LDEzNDMwMTM5NSwgICAwLCAgIDANCjIwNDg6ICAgICAgICAgICAgICAgICAgMjA0OCwg ICAgICAwLCAgIDQyNDQ1LCAgICAgNjg1LDE5MzExMzc4MzcsICAgMCwgICAwDQo0MDk2OiAgICAg ICAgICAgICAgICAgIDQwOTYsICAgICAgMCwgICAxMzg3NCwgICAgICA2MiwxMDI5MDc5MzYsICAg MCwgICAwDQo4MTkyOiAgICAgICAgICAgICAgICAgIDgxOTIsICAgICAgMCwgICAgICAyNiwgICAg ICAyNywgMzA2Njg4MiwgICAwLCAgIDANCjE2Mzg0OiAgICAgICAgICAgICAgICAxNjM4NCwgICAg ICAwLCAgICAgIDQyLCAgICAgIDE2LCAgICA3NjExLCAgIDAsICAgMA0KMzI3Njg6ICAgICAgICAg ICAgICAgIDMyNzY4LCAgICAgIDAsICAgICAgMzksICAgICAgMTcsICAgNjc4MDUsICAgMCwgICAw DQo2NTUzNjogICAgICAgICAgICAgICAgNjU1MzYsICAgICAgMCwgICAgICA4MSwgICAgICAyMSwg MTA0MjYwNiwgICAwLCAgIDANCjY0IHBjcHU6ICAgICAgICAgICAgICAgICAgOCwgICAgICAwLCAg ICAxNjU0LCAgICAgNTIyLCAgICAxODE0LCAgIDAsICAgMA0KU0xFRVBRVUVVRTogICAgICAgICAg ICAgIDgwLCAgICAgIDAsICAgIDEyOTcsICAgICA4MTEsICAgIDE1NzksICAgMCwgICAwDQpGaWxl czogICAgICAgICAgICAgICAgICAgODAsICAgICAgMCwgICAgIDEzNSwgICAgMjY2NSwxMTczNjUw NSwgICAwLCAgIDANClRVUk5TVElMRTogICAgICAgICAgICAgIDEzNiwgICAgICAwLCAgICAxMjk3 LCAgICAgNTYzLCAgICAxNTc5LCAgIDAsICAgMA0KcmxfZW50cnk6ICAgICAgICAgICAgICAgIDQw LCAgICAgIDAsICAgICAzMzAsICAgIDMyNzAsICAgICAzNDEsICAgMCwgICAwDQp1bXR4IHBpOiAg ICAgICAgICAgICAgICAgOTYsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAw LCAgIDANCk1BQyBsYWJlbHM6ICAgICAgICAgICAgICA0MCwgICAgICAwLCAgICAgICAwLCAgICAg ICAwLCAgICAgICAwLCAgIDAsICAgMA0KUFJPQzogICAgICAgICAgICAgICAgICAxMjE2LCAgICAg IDAsICAgICAgNjEsICAgICAyMTUsIDEzMzc3NTIsICAgMCwgICAwDQpUSFJFQUQ6ICAgICAgICAg ICAgICAgIDExNjgsICAgICAgMCwgICAgMTE2OSwgICAgIDEyNywgICAgNTY4MSwgICAwLCAgIDAN CmNwdXNldDogICAgICAgICAgICAgICAgICA3MiwgICAgICAwLCAgICAgODcwLCAgICAxMTY1LCAg ICAxNjM3LCAgIDAsICAgMA0KYXVkaXRfcmVjb3JkOiAgICAgICAgICAxMjQ4LCAgICAgIDAsICAg ICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQptYnVmX3BhY2tldDogICAgICAgICAg ICAyNTYsIDE5NTk2MTY1LCAgIDE4NjU1LCAgICA2MzY3LDEwNzQ1OTMzODk0LCAgIDAsICAgMA0K bWJ1ZjogICAgICAgICAgICAgICAgICAgMjU2LCAxOTU5NjE2NSwgICAgIDUyNywgICAgNzkwMSwx Mjg3NDYwNDc0OCwgICAwLCAgIDANCm1idWZfY2x1c3RlcjogICAgICAgICAgMjA0OCwgMzA2MTkw MCwgICAyNTAyMiwgICAgMTg2MiwxNzM3NjM3OTgwLCAgIDAsICAgMA0KbWJ1Zl9qdW1ib19wYWdl OiAgICAgICA0MDk2LCAxNTMwOTUwLCAgICAgICAwLCAgICAgMTE5LDE0MjE3MTcxNDAsICAgMCwg ICAwDQptYnVmX2p1bWJvXzlrOiAgICAgICAgIDkyMTYsIDQ1MzYxNCwgICAgICAgMCwgICAgICAg MCwgICAgICAgMCwgICAwLCAgIDANCm1idWZfanVtYm9fMTZrOiAgICAgICAxNjM4NCwgMjU1MTU4 LCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KbWJ1Zl9leHRfcmVmY250OiAg ICAgICAgICA0LCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpn X2JpbzogICAgICAgICAgICAgICAgICAyNDgsICAgICAgMCwgICAgICA1MiwgICAgMjc0OCw3MzI1 MjE0NTA4LCAgIDAsICAgMA0KdHR5aW5xOiAgICAgICAgICAgICAgICAgMTYwLCAgICAgIDAsICAg ICAyNDAsICAgICAyMTAsICAgIDExMjUsICAgMCwgICAwDQp0dHlvdXRxOiAgICAgICAgICAgICAg ICAyNTYsICAgICAgMCwgICAgIDEyNiwgICAgIDE0NCwgICAgIDU5MywgICAwLCAgIDANCkRNQVJf TUFQX0VOVFJZOiAgICAgICAgIDEyMCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAw LCAgIDAsICAgMA0KYXRhX3JlcXVlc3Q6ICAgICAgICAgICAgMzM2LCAgICAgIDAsICAgICAgIDAs ICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQp2dG5ldF90eF9oZHI6ICAgICAgICAgICAgMjQs ICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCkZQVV9zYXZlX2Fy ZWE6ICAgICAgICAgIDUxMiwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAs ICAgMA0KdGFza3Ffem9uZTogICAgICAgICAgICAgIDQ4LCAgICAgIDAsICAgICAgIDAsICAgIDY3 MjMsMTcwMDgwNDc2LCAgIDAsICAgMA0KVk5PREU6ICAgICAgICAgICAgICAgICAgNDcyLCAgICAg IDAsICAyMzQzODYsICAgMjc0NzgsIDUwMzQ2NjQsICAgMCwgICAwDQpWTk9ERVBPTEw6ICAgICAg ICAgICAgICAxMTIsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMSwgICAwLCAgIDAN CkJVRiBUUklFOiAgICAgICAgICAgICAgIDE0NCwgICAgICAwLCAgICAgICAwLCAgMTA1OTQ4LCAg ICAgICAwLCAgIDAsICAgMA0KUyBWRlMgQ2FjaGU6ICAgICAgICAgICAgMTA4LCAgICAgIDAsICAx NDYyMzgsICAgODMxNTIsIDU3MzI2OTgsICAgMCwgICAwDQpTVFMgVkZTIENhY2hlOiAgICAgICAg ICAxNDgsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAxMiwgICAwLCAgIDANCkwgVkZT IENhY2hlOiAgICAgICAgICAgIDMyOCwgICAgICAwLCAgIDU2ODEwLCAgICAxODQ2LCAgODEwOTg4 LCAgIDAsICAgMA0KTFRTIFZGUyBDYWNoZTogICAgICAgICAgMzY4LCAgICAgIDAsICAgICAgIDAs ICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpOQU1FSTogICAgICAgICAgICAgICAgIDEwMjQs ICAgICAgMCwgICAgICAgMCwgICAgIDI2MCw0NDg4NTc2OCwgICAwLCAgIDANCk5DTE5PREU6ICAg ICAgICAgICAgICAgIDUyOCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgIDE2LCAgIDAs ICAgMA0KRElSSEFTSDogICAgICAgICAgICAgICAxMDI0LCAgICAgIDAsICAgICAgIDAsICAgICAg IDAsICAgICAgIDAsICAgMCwgICAwDQpwaXBlOiAgICAgICAgICAgICAgICAgICA3NDQsICAgICAg MCwgICAgICAgNiwgICAgIDIzOSwgIDEwODI2NSwgICAwLCAgIDANCnByb2NkZXNjOiAgICAgICAg ICAgICAgIDEyOCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0K cmFuZ2Vfc2VnX2NhY2hlOiAgICAgICAgIDY0LCAgICAgIDAsIDIyNDg5NTUsIDE2MzQ4NDksNjg2 MDI5MDkwNjcsICAgMCwgICAwDQp6aW9fY2FjaGU6ICAgICAgICAgICAgICA5MjAsICAgICAgMCwg ICAgNTE5OSwgIDEwMDI5MywzMTQxNzAzNzA0NSwgICAwLCAgIDANCnppb19saW5rX2NhY2hlOiAg ICAgICAgICA0OCwgICAgICAwLCAgICA1NDcwLCAgMTAwMzU1LDM0MDkxODkxMzI5LCAgIDAsICAg MA0KemlvX2J1Zl81MTI6ICAgICAgICAgICAgNTEyLCAgICAgIDAsICAyNzUxOTUsICAgNjgwMTMs NzEyNjI0NjYsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNTEyOiAgICAgICA1MTIsICAgICAgMCwg ICAgICAzMCwgICAgIDU3MCwgOTUxMjUyNywgICAwLCAgIDANCnppb19idWZfMTAyNDogICAgICAg ICAgMTAyNCwgICAgICAwLCAgICAgICA1LCAgICAgMjg3LCA4NDc1MDM5LCAgIDAsICAgMA0Kemlv X2RhdGFfYnVmXzEwMjQ6ICAgICAxMDI0LCAgICAgIDAsICAgICAgMjIsICAgICA3NzQsIDQyNjAz NzEsICAgMCwgICAwDQp6aW9fYnVmXzE1MzY6ICAgICAgICAgIDE1MzYsICAgICAgMCwgICAgICAg MywgICAgIDE4MywgNTI3NzA3NSwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8xNTM2OiAgICAgMTUz NiwgICAgICAwLCAgICAgICA3LCAgICAgMTc5LCA2NjA0ODA1LCAgIDAsICAgMA0KemlvX2J1Zl8y MDQ4OiAgICAgICAgICAyMDQ4LCAgICAgIDAsICAgICAzNjEsICAgICAxMjEsMTMzMjAyOTgsICAg MCwgICAwDQp6aW9fZGF0YV9idWZfMjA0ODogICAgIDIwNDgsICAgICAgMCwgICAgICAxNiwgICAg ICA5NiwgNjA2MDI0OSwgICAwLCAgIDANCnppb19idWZfMjU2MDogICAgICAgICAgMjU2MCwgICAg ICAwLCAgICAgICA3LCAgICAgMTQ1LCA0ODQxNzY4LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzI1 NjA6ICAgICAyNTYwLCAgICAgIDAsICAgICAgIDIsICAgICAgNjYsIDQzMjc4MjAsICAgMCwgICAw DQp6aW9fYnVmXzMwNzI6ICAgICAgICAgIDMwNzIsICAgICAgMCwgICAgIDEwOCwgICAgIDc2Niwg ODg0NDU2OSwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8zMDcyOiAgICAgMzA3MiwgICAgICAwLCAg ICAgICAyLCAgICAgIDYxLCAzNDM1NjE1LCAgIDAsICAgMA0KemlvX2J1Zl8zNTg0OiAgICAgICAg ICAzNTg0LCAgICAgIDAsICAgICAgIDMsICAgICAgNDQsIDQ3NTY2MTcsICAgMCwgICAwDQp6aW9f ZGF0YV9idWZfMzU4NDogICAgIDM1ODQsICAgICAgMCwgICAgICAgNiwgICAgICAyNiwgMzIzODk2 MiwgICAwLCAgIDANCnppb19idWZfNDA5NjogICAgICAgICAgNDA5NiwgICAgICAwLCAxNTk5NDE0 LCAgIDI4OTc2LDMzODU1MTc4MTQsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNDA5NjogICAgIDQw OTYsICAgICAgMCwgIDMwNTM0NiwgICAgMTQ1MiwyMzQ3NjA1NjEsICAgMCwgICAwDQp6aW9fYnVm XzUxMjA6ICAgICAgICAgIDUxMjAsICAgICAgMCwgICAgIDMwMCwgICAgICAyNCwxOTE5NDk2OCwg ICAwLCAgIDANCnppb19kYXRhX2J1Zl81MTIwOiAgICAgNTEyMCwgICAgICAwLCAgICAgICA0LCAg ICAgIDI2LCAxODE1ODA4LCAgIDAsICAgMA0KemlvX2J1Zl82MTQ0OiAgICAgICAgICA2MTQ0LCAg ICAgIDAsICAgICAgIDAsICAgICAgMjQsMTM0ODcwNDcsICAgMCwgICAwDQp6aW9fZGF0YV9idWZf NjE0NDogICAgIDYxNDQsICAgICAgMCwgICAgICAgNiwgICAgICAyNSwgMzM1MzU1NCwgICAwLCAg IDANCnppb19idWZfNzE2ODogICAgICAgICAgNzE2OCwgICAgICAwLCAgICAgICAwLCAgICAgIDIy LCA5NTMwNDc4LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzcxNjg6ICAgICA3MTY4LCAgICAgIDAs ICAgICAgIDAsICAgICAgMjcsIDIzMTU4NzUsICAgMCwgICAwDQp6aW9fYnVmXzgxOTI6ICAgICAg ICAgIDgxOTIsICAgICAgMCwgICAgIDE0MSwgICAgIDUyNywyNTEyNDIxMzMsICAgMCwgICAwDQp6 aW9fZGF0YV9idWZfODE5MjogICAgIDgxOTIsICAgICAgMCwgICAgICAgNSwgICAgICAyNiwgMTc2 NDA3MiwgICAwLCAgIDANCnppb19idWZfMTAyNDA6ICAgICAgICAxMDI0MCwgICAgICAwLCAgICAg ICAwLCAgICAgIDM5LDE1ODUwNDQwLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzEwMjQwOiAgIDEw MjQwLCAgICAgIDAsICAgICAgIDIsICAgICAgMjcsIDM0MDUwMTUsICAgMCwgICAwDQp6aW9fYnVm XzEyMjg4OiAgICAgICAgMTIyODgsICAgICAgMCwgICAgICAgMSwgICAgICA5MCw3NzgyOTAwNCwg ICAwLCAgIDANCnppb19kYXRhX2J1Zl8xMjI4ODogICAxMjI4OCwgICAgICAwLCAgICAgICAzLCAg ICAgIDMwLCAzMjIzMzgxLCAgIDAsICAgMA0KemlvX2J1Zl8xNDMzNjogICAgICAgIDE0MzM2LCAg ICAgIDAsICAgICAgIDEsICAgICAgNjksMTMxODAyNzEsICAgMCwgICAwDQp6aW9fZGF0YV9idWZf MTQzMzY6ICAgMTQzMzYsICAgICAgMCwgICAgICAgNCwgICAgICAyNSwgMjg2OTQ2MywgICAwLCAg IDANCnppb19idWZfMTYzODQ6ICAgICAgICAxNjM4NCwgICAgICAwLCAgMjczMTkzLCAgICAzNjY2 LDM1OTM0ODk5MSwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8xNjM4NDogICAxNjM4NCwgICAgICAw LCAgICA1ODI2LCAgICAgIDYyLCA0NDc4NjA3LCAgIDAsICAgMA0KemlvX2J1Zl8yMDQ4MDogICAg ICAgIDIwNDgwLCAgICAgIDAsICAgICAgIDAsICAgICAgOTIsNTg4NTgwMDAsICAgMCwgICAwDQp6 aW9fZGF0YV9idWZfMjA0ODA6ICAgMjA0ODAsICAgICAgMCwgICAgICAgOCwgICAgICAyNiwgMzY5 NDE3OSwgICAwLCAgIDANCnppb19idWZfMjQ1NzY6ICAgICAgICAyNDU3NiwgICAgICAwLCAgICAg ICAwLCAgICAgIDY1LDU3NjMyMzMxLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzI0NTc2OiAgIDI0 NTc2LCAgICAgIDAsICAgICAgIDIsICAgICAgMjYsIDMyNDE4MDIsICAgMCwgICAwDQp6aW9fYnVm XzI4NjcyOiAgICAgICAgMjg2NzIsICAgICAgMCwgICAgICAgMCwgICAgICA1NSw1NzE5NDI5OSwg ICAwLCAgIDANCnppb19kYXRhX2J1Zl8yODY3MjogICAyODY3MiwgICAgICAwLCAgICAgICAwLCAg ICAgIDI5LCAzMDU1MjcwLCAgIDAsICAgMA0KemlvX2J1Zl8zMjc2ODogICAgICAgIDMyNzY4LCAg ICAgIDAsICAgICAgIDAsICAgICAxMDQsNTU0MjI5MzQsICAgMCwgICAwDQp6aW9fZGF0YV9idWZf MzI3Njg6ICAgMzI3NjgsICAgICAgMCwgICAgICAzMywgICAgICA4MCw5NTk5NDgwOSwgICAwLCAg IDANCnppb19idWZfMzY4NjQ6ICAgICAgICAzNjg2NCwgICAgICAwLCAgICAgICAyLCAgICAgIDQ1 LDMwOTQ0OTU2LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzM2ODY0OiAgIDM2ODY0LCAgICAgIDAs ICAgICAgIDEsICAgICAgMjYsIDI5MDU3ODAsICAgMCwgICAwDQp6aW9fYnVmXzQwOTYwOiAgICAg ICAgNDA5NjAsICAgICAgMCwgICAgICAgMCwgICAgICA1NywzMDA4NDMyMCwgICAwLCAgIDANCnpp b19kYXRhX2J1Zl80MDk2MDogICA0MDk2MCwgICAgICAwLCAgICAgICA2LCAgICAgIDI2LCAzNDYz NzQ0LCAgIDAsICAgMA0KemlvX2J1Zl80NTA1NjogICAgICAgIDQ1MDU2LCAgICAgIDAsICAgICAg IDAsICAgICAgNjksMjgyNzUxOTksICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNDUwNTY6ICAgNDUw NTYsICAgICAgMCwgICAgICAgMSwgICAgICAyOSwgMzY0ODUyNiwgICAwLCAgIDANCnppb19idWZf NDkxNTI6ICAgICAgICA0OTE1MiwgICAgICAwLCAgICAgICAwLCAgICAgIDgxLDMxMjQ3NDE0LCAg IDAsICAgMA0KemlvX2RhdGFfYnVmXzQ5MTUyOiAgIDQ5MTUyLCAgICAgIDAsICAgICAgIDEsICAg ICAgMjksIDMxMDcyOTYsICAgMCwgICAwDQp6aW9fYnVmXzUzMjQ4OiAgICAgICAgNTMyNDgsICAg ICAgMCwgICAgICAgMCwgICAgIDEzNSwyNTE1NjcyNywgICAwLCAgIDANCnppb19kYXRhX2J1Zl81 MzI0ODogICA1MzI0OCwgICAgICAwLCAgICAgICAwLCAgICAgIDI2LCAyNTM0NzE3LCAgIDAsICAg MA0KemlvX2J1Zl81NzM0NDogICAgICAgIDU3MzQ0LCAgICAgIDAsICAgICAgIDAsICAgICAgOTQs MjY1NzkxNzQsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNTczNDQ6ICAgNTczNDQsICAgICAgMCwg ICAgICAgMSwgICAgICAyNywgMjg3OTUxNywgICAwLCAgIDANCnppb19idWZfNjE0NDA6ICAgICAg ICA2MTQ0MCwgICAgICAwLCAgICAgICAwLCAgICAgIDUzLCA4MjExOTUxLCAgIDAsICAgMA0Kemlv X2RhdGFfYnVmXzYxNDQwOiAgIDYxNDQwLCAgICAgIDAsICAgICAgIDEsICAgICAgMjEsICAgIDIw ODksICAgMCwgICAwDQp6aW9fYnVmXzY1NTM2OiAgICAgICAgNjU1MzYsICAgICAgMCwgICAgICAg MCwgICAgMTA2NSwzMzI2MTY3MTEsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNjU1MzY6ICAgNjU1 MzYsICAgICAgMCwgIDI3MDk4MSwgICAgIDkyMSw0ODM5MTYxMTMsICAgMCwgICAwDQp6aW9fYnVm XzY5NjMyOiAgICAgICAgNjk2MzIsICAgICAgMCwgICAgICAgMCwgICAgICAyMiwgNjI0NDE4Mywg ICAwLCAgIDANCnppb19kYXRhX2J1Zl82OTYzMjogICA2OTYzMiwgICAgICAwLCAgICAgICAxLCAg ICAgIDIyLCAgICAyNDQyLCAgIDAsICAgMA0KemlvX2J1Zl83MzcyODogICAgICAgIDczNzI4LCAg ICAgIDAsICAgICAgIDAsICAgICAgMjIsIDcxMDU0NjYsICAgMCwgICAwDQp6aW9fZGF0YV9idWZf NzM3Mjg6ICAgNzM3MjgsICAgICAgMCwgICAgICAgMCwgICAgICAyNSwgICAgMjA3OCwgICAwLCAg IDANCnppb19idWZfNzc4MjQ6ICAgICAgICA3NzgyNCwgICAgICAwLCAgICAgICAwLCAgICAgIDIy LCA1MzYwMzgwLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzc3ODI0OiAgIDc3ODI0LCAgICAgIDAs ICAgICAgIDAsICAgICAgMjIsICAgIDIwNzYsICAgMCwgICAwDQp6aW9fYnVmXzgxOTIwOiAgICAg ICAgODE5MjAsICAgICAgMCwgICAgICAgMCwgICAgICAyNSwgNjk3NjIzNywgICAwLCAgIDANCnpp b19kYXRhX2J1Zl84MTkyMDogICA4MTkyMCwgICAgICAwLCAgICAgICAwLCAgICAgIDIwLCAgICAy MDc3LCAgIDAsICAgMA0KemlvX2J1Zl84NjAxNjogICAgICAgIDg2MDE2LCAgICAgIDAsICAgICAg IDAsICAgICAgMjcsIDUzODcyNjksICAgMCwgICAwDQp6aW9fZGF0YV9idWZfODYwMTY6ICAgODYw MTYsICAgICAgMCwgICAgICAgMCwgICAgICAyMSwgICAgMjA3MCwgICAwLCAgIDANCnppb19idWZf OTAxMTI6ICAgICAgICA5MDExMiwgICAgICAwLCAgICAgICAwLCAgICAgIDIzLCA2MzYyNTc2LCAg IDAsICAgMA0KemlvX2RhdGFfYnVmXzkwMTEyOiAgIDkwMTEyLCAgICAgIDAsICAgICAgIDEsICAg ICAgMjMsICAgIDI1NTEsICAgMCwgICAwDQp6aW9fYnVmXzk0MjA4OiAgICAgICAgOTQyMDgsICAg ICAgMCwgICAgICAgMCwgICAgICAyMywgNjA2NzExMiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl85 NDIwODogICA5NDIwOCwgICAgICAwLCAgICAgICAwLCAgICAgIDI0LCAgICAyMTE0LCAgIDAsICAg MA0KemlvX2J1Zl85ODMwNDogICAgICAgIDk4MzA0LCAgICAgIDAsICAgICAgIDAsICAgICAgMjIs IDg1ODc5MjIsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfOTgzMDQ6ICAgOTgzMDQsICAgICAgMCwg ICAgICAgMCwgICAgICAyMSwgICAgMjEwNCwgICAwLCAgIDANCnppb19idWZfMTAyNDAwOiAgICAg IDEwMjQwMCwgICAgICAwLCAgICAgICAwLCAgICAgIDI2LCA2NTg5MDcxLCAgIDAsICAgMA0Kemlv X2RhdGFfYnVmXzEwMjQwMDogMTAyNDAwLCAgICAgIDAsICAgICAgIDAsICAgICAgMjEsICAgIDIw NjEsICAgMCwgICAwDQp6aW9fYnVmXzEwNjQ5NjogICAgICAxMDY0OTYsICAgICAgMCwgICAgICAg MCwgICAgICAyNCwgNzI4MDU4MywgICAwLCAgIDANCnppb19kYXRhX2J1Zl8xMDY0OTY6IDEwNjQ5 NiwgICAgICAwLCAgICAgICAwLCAgICAgIDI0LCAgICAyMDA5LCAgIDAsICAgMA0KemlvX2J1Zl8x MTA1OTI6ICAgICAgMTEwNTkyLCAgICAgIDAsICAgICAgIDAsICAgICAgMjUsIDc3Njc2OTgsICAg MCwgICAwDQp6aW9fZGF0YV9idWZfMTEwNTkyOiAxMTA1OTIsICAgICAgMCwgICAgICAgMCwgICAg ICAyMCwgICAgMTkwMywgICAwLCAgIDANCnppb19idWZfMTE0Njg4OiAgICAgIDExNDY4OCwgICAg ICAwLCAgICAgICAwLCAgICAgIDI1LCA5MTE4NjIyLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzEx NDY4ODogMTE0Njg4LCAgICAgIDAsICAgICAgIDEsICAgICAgMjEsICAgIDE5MDEsICAgMCwgICAw DQp6aW9fYnVmXzExODc4NDogICAgICAxMTg3ODQsICAgICAgMCwgICAgICAgMCwgICAgICAyNCwg ODI2ODgxNiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8xMTg3ODQ6IDExODc4NCwgICAgICAwLCAg ICAgICAwLCAgICAgIDIzLCAgICAxNzQwLCAgIDAsICAgMA0KemlvX2J1Zl8xMjI4ODA6ICAgICAg MTIyODgwLCAgICAgIDAsICAgICAgIDAsICAgICAgMzEsMTEyMjQ5MjUsICAgMCwgICAwDQp6aW9f ZGF0YV9idWZfMTIyODgwOiAxMjI4ODAsICAgICAgMCwgICAgICAgMCwgICAgICAyMCwgICAgMTY2 OCwgICAwLCAgIDANCnppb19idWZfMTI2OTc2OiAgICAgIDEyNjk3NiwgICAgICAwLCAgICAgICAx LCAgICAgIDI2LDEwNzEzNTE1LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzEyNjk3NjogMTI2OTc2 LCAgICAgIDAsICAgICAgIDAsICAgICAgMTksICAgIDE2NDgsICAgMCwgICAwDQp6aW9fYnVmXzEz MTA3MjogICAgICAxMzEwNzIsICAgICAgMCwgICAxNDAwMiwgICAgIDcxNywyNTQxOTM5MjIsICAg MCwgICAwDQp6aW9fZGF0YV9idWZfMTMxMDcyOiAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAg ICAxNCwgICA1NjY5NiwgICAwLCAgIDANCmx6NF9jdHg6ICAgICAgICAgICAgICAxNjM4NCwgICAg ICAwLCAgICAgICAwLCAgICAgIDUyLDE2NTg4MTQ3NDMsICAgMCwgICAwDQpzYV9jYWNoZTogICAg ICAgICAgICAgICAgODAsICAgICAgMCwgIDIzNDI5NCwgICA4Mzk1NiwgNTAzMzg3NiwgICAwLCAg IDANCmRub2RlX3Q6ICAgICAgICAgICAgICAgIDc1MiwgICAgICAwLCAgMzI1MTk0LCAgIDIzODAx LCA3ODQ2MTQ5LCAgIDAsICAgMA0KZG11X2J1Zl9pbXBsX3Q6ICAgICAgICAgMjMyLCAgICAgIDAs IDI0MjA3NDIsIDEwOTI2NjUsMTAxODgzNzQ3NiwgICAwLCAgIDANCmFyY19idWZfaGRyX3Q6ICAg ICAgICAgIDIxNiwgICAgICAwLDMxNDY5NzE4LCA2NTgwOTE0LDcwNDc3NjE5MiwgICAwLCAgIDAN CmFyY19idWZfdDogICAgICAgICAgICAgICA3MiwgICAgICAwLCAyMzAzNjA3LCAxNDQzODE4LDg0 Mjk4MTQ1NSwgICAwLCAgIDANCnppbF9sd2JfY2FjaGU6ICAgICAgICAgIDE5MiwgICAgICAwLCAg ICAgMTYyLCAgICAxNzk4LDExODk3ODY0OCwgICAwLCAgIDANCnpmc196bm9kZV9jYWNoZTogICAg ICAgIDM2OCwgICAgICAwLCAgMjM0Mjk0LCAgIDM4MTQ2LCA1MDMzODc2LCAgIDAsICAgMA0KTW91 bnRwb2ludHM6ICAgICAgICAgICAgODE2LCAgICAgIDAsICAgICAgMzMsICAgICAxMzcsICAgICAg MzcsICAgMCwgICAwDQprc2lnaW5mbzogICAgICAgICAgICAgICAxMTIsICAgICAgMCwgICAgIDI2 NywgICAgMzEyOCwgICA1MTQyMiwgICAwLCAgIDANCml0aW1lcjogICAgICAgICAgICAgICAgIDM1 MiwgICAgICAwLCAgICAgICAxLCAgICAgIDEwLCAgICAgICAxLCAgIDAsICAgMA0KS05PVEU6ICAg ICAgICAgICAgICAgICAgMTI4LCAgICAgIDAsICAgICAgIDAsICAgIDIxMzksICAzODYwNDEsICAg MCwgICAwDQpzb2NrZXQ6ICAgICAgICAgICAgICAgICA2OTYsIDE1NzE2MjAsICAgICAgNjcsICAg ICAyODMsICA1NTE5NTcsICAgMCwgICAwDQppcHE6ICAgICAgICAgICAgICAgICAgICAgNTYsICA5 NTcwOCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnVkcF9pbnBjYjogICAg ICAgICAgICAgIDM5MiwgMTU3MTYyMCwgICAgICAxOCwgICAgIDYxMiwgIDI0MzIxNSwgICAwLCAg IDANCnVkcGNiOiAgICAgICAgICAgICAgICAgICAxNiwgMTU3MTc2MiwgICAgICAxOCwgICAgNTAw MiwgIDI0MzIxNSwgICAwLCAgIDANCnRjcF9pbnBjYjogICAgICAgICAgICAgIDM5MiwgMTU3MTYy MCwgICAgICAzMywgICAgIDU5NywgIDI5NTYxOSwgICAwLCAgIDANCnRjcGNiOiAgICAgICAgICAg ICAgICAgMTAyNCwgMTU3MTYyMCwgICAgICAzMywgICAgIDI4MywgIDI5NTYxOSwgICAwLCAgIDAN CnRjcHR3OiAgICAgICAgICAgICAgICAgICA4OCwgIDI3ODEwLCAgICAgICAwLCAgICAxNjY1LCAg ICA1MDU5LCAgIDAsICAgMA0Kc3luY2FjaGU6ICAgICAgICAgICAgICAgMTYwLCAgMTUzNzUsICAg ICAgIDAsICAgICA1MDAsICAyNzkzMzEsICAgMCwgICAwDQpob3N0Y2FjaGU6ICAgICAgICAgICAg ICAxMzYsICAxNTM3MCwgICAgICAgMiwgICAgIDM3NSwgICAgICA4NSwgICAwLCAgIDANCnRjcHJl YXNzOiAgICAgICAgICAgICAgICA0MCwgMTkxNDAwLCAgICAgICAwLCAgICAyMTAwLCAgICA5NTA0 LCAgIDAsICAgMA0Kc2Fja2hvbGU6ICAgICAgICAgICAgICAgIDMyLCAgICAgIDAsICAgICAgIDAs ICAgIDIxMjUsICAxMDc1MjgsICAgMCwgICAwDQpzY3RwX2VwOiAgICAgICAgICAgICAgIDE0MDgs IDE1NzE2MjAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpzY3RwX2Fzb2M6 ICAgICAgICAgICAgIDI0MTYsICA0MDAwMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAw LCAgIDANCnNjdHBfbGFkZHI6ICAgICAgICAgICAgICA0OCwgIDgwMDEyLCAgICAgICAwLCAgICAg ICAwLCAgICAgICA0LCAgIDAsICAgMA0Kc2N0cF9yYWRkcjogICAgICAgICAgICAgNzI4LCAgODAw MDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpzY3RwX2NodW5rOiAgICAg ICAgICAgICAxMzYsIDQwMDAyNiwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDAN CnNjdHBfcmVhZHE6ICAgICAgICAgICAgIDEwNCwgNDAwMDI2LCAgICAgICAwLCAgICAgICAwLCAg ICAgICAwLCAgIDAsICAgMA0Kc2N0cF9zdHJlYW1fbXNnX291dDogICAgMTA0LCA0MDAwMjYsICAg ICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpzY3RwX2FzY29uZjogICAgICAgICAg ICAgNDAsIDQwMDAwMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnNjdHBf YXNjb25mX2FjazogICAgICAgICA0OCwgNDAwMDYwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAw LCAgIDAsICAgMA0KdWRwbGl0ZV9pbnBjYjogICAgICAgICAgMzkyLCAxNTcxNjIwLCAgICAgICAw LCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KcmlwY2I6ICAgICAgICAgICAgICAgICAgMzky LCAxNTcxNjIwLCAgICAgICAwLCAgICAgMTUwLCAgICAgIDE3LCAgIDAsICAgMA0KdW5wY2I6ICAg ICAgICAgICAgICAgICAgMjQwLCAxNTcxNjMyLCAgICAgIDE1LCAgICAgOTI5LCAgIDEzMTAxLCAg IDAsICAgMA0KcnRlbnRyeTogICAgICAgICAgICAgICAgMjAwLCAgICAgIDAsICAgICAgMTQsICAg ICAxNDYsICAgICAgMTQsICAgMCwgICAwDQpzZWxmZDogICAgICAgICAgICAgICAgICAgNTYsICAg ICAgMCwgICAgIDUxOSwgICAgNDMwOSwyNTY0NDM0NiwgICAwLCAgIDANClNXQVBNRVRBOiAgICAg ICAgICAgICAgIDI4OCwgNjEyMzgwNiwgICAgIDg4NCwgICAgMjI4OCwgIDQ5NDM4OSwgICAwLCAg IDANCmljbF9jb25uOiAgICAgICAgICAgICAgIDE1MiwgICAgICAwLCAgICAgICA4LCAgICAgMzMw LCAgICAgIDgyLCAgIDAsICAgMA0KaWNsX3BkdTogICAgICAgICAgICAgICAgIDgwLCAgICAgIDAs ICAgICAgIDYsICAgIDMwOTQsNjE5NDMzNjc0LCAgIDAsICAgMA0KaXNjc2lfb3V0c3RhbmRpbmc6 ICAgICAgIDQwLCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpj ZmlfbHVuOiAgICAgICAgICAgICAgICAzNDQsICAgICAgMCwgICAgICAgOSwgICAgICA0NiwgICAg ICAxMSwgICAwLCAgIDANCmNmaV9tZXRhdGFzazogICAgICAgICAgIDMyMCwgICAgICAwLCAgICAg ICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KY2Zpc2NzaV9kYXRhX3dhaXQ6ICAgICAg IDU2LCAgICAgIDAsICAgICAgIDAsICAgIDUzMjUsMjM5NjU1OTU1LCAgIDAsICAgMA0KYmVpbzog ICAgICAgICAgICAgICAgICAgMzYwLCAgICAgIDAsICAgICAgIDIsICAgICA1MzcsMjg0NDA5NzMz LCAgIDAsICAgMA0KY2JsazI6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAs ICAgICAgMjcsIDMzNDQ5NTcsICAgMCwgICAwDQpjYmxrMzogICAgICAgICAgICAgICAxMzEwNzIs ICAgICAgMCwgICAgICAgMCwgICAgICAyMywgICAgMjc1NywgICAwLCAgIDANCmNibGs0OiAgICAg ICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDU1LDQ2MTUxOTQyLCAgIDAs ICAgMA0KY2JsazU6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDEsICAgICAg MzIsOTA5NjE5MzUsICAgMCwgICAwDQpjYmxrNjogICAgICAgICAgICAgICAxMzEwNzIsICAgICAg MCwgICAgICAgMCwgICAgICAxNywzNTI2NDM4NiwgICAwLCAgIDANCmNibGs3OiAgICAgICAgICAg ICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDI4LDU2OTg5NTYyLCAgIDAsICAgMA0K Y2Jsazg6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDEsICAgICAgMzIsMzc3 NDczMjIsICAgMCwgICAwDQpjYmxrOTogICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAg ICAgMCwgICAgICAyNywgNzA5ODM0OSwgICAwLCAgIDANCmNibGsxMDogICAgICAgICAgICAgIDEz MTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDI2LCA5NTU2OTYzLCAgIDAsICAgMA0KDQogcHJv Y3MgICAgICBtZW1vcnkgICAgICBwYWdlICAgICAgICAgICAgICAgICAgICBkaXNrcyAgICAgZmF1 bHRzICAgICAgICAgY3B1DQogciBiIHcgICAgIGF2bSAgICBmcmUgICBmbHQgIHJlICBwaSAgcG8g ICAgZnIgIHNyIG1mMCBtZjEgICBpbiAgIHN5ICAgY3MgdXMgc3kgaWQNCiAwIDAgNyAgIDEzMjhN ICAxOTIwTSAgIDYwNiAgIDAgICAwICAgMCAgMjI4NSAgMTAgICAwICAgMCAgODM2ICA2MTUgMjQz OCAgMCAgNCA5Ng0KMTkwMTIvMTQ0MzgvMzM0NTAgbWJ1ZnMgaW4gdXNlIChjdXJyZW50L2NhY2hl L3RvdGFsKQ0KMTg0OTQvODM5MC8yNjg4NC8zMDYxOTAwIG1idWYgY2x1c3RlcnMgaW4gdXNlIChj dXJyZW50L2NhY2hlL3RvdGFsL21heCkNCjE4NDU3LzY1NjUgbWJ1ZitjbHVzdGVycyBvdXQgb2Yg cGFja2V0IHNlY29uZGFyeSB6b25lIGluIHVzZSAoY3VycmVudC9jYWNoZSkNCjAvMTM3LzEzNy8x NTMwOTUwIDRrIChwYWdlIHNpemUpIGp1bWJvIGNsdXN0ZXJzIGluIHVzZSAoY3VycmVudC9jYWNo ZS90b3RhbC9tYXgpDQowLzAvMC80NTM2MTQgOWsganVtYm8gY2x1c3RlcnMgaW4gdXNlIChjdXJy ZW50L2NhY2hlL3RvdGFsL21heCkNCjAvMC8wLzI1NTE1OCAxNmsganVtYm8gY2x1c3RlcnMgaW4g dXNlIChjdXJyZW50L2NhY2hlL3RvdGFsL21heCkNCjQxNzQxSy8yMDkzN0svNjI2NzhLIGJ5dGVz IGFsbG9jYXRlZCB0byBuZXR3b3JrIChjdXJyZW50L2NhY2hlL3RvdGFsKQ0KMC8wLzAgcmVxdWVz dHMgZm9yIG1idWZzIGRlbmllZCAobWJ1ZnMvY2x1c3RlcnMvbWJ1ZitjbHVzdGVycykNCjAvMC8w IHJlcXVlc3RzIGZvciBtYnVmcyBkZWxheWVkIChtYnVmcy9jbHVzdGVycy9tYnVmK2NsdXN0ZXJz KQ0KMC8wLzAgcmVxdWVzdHMgZm9yIGp1bWJvIGNsdXN0ZXJzIGRlbGF5ZWQgKDRrLzlrLzE2aykN CjAvMC8wIHJlcXVlc3RzIGZvciBqdW1ibyBjbHVzdGVycyBkZW5pZWQgKDRrLzlrLzE2aykNCjAg cmVxdWVzdHMgZm9yIHNmYnVmcyBkZW5pZWQNCjAgcmVxdWVzdHMgZm9yIHNmYnVmcyBkZWxheWVk DQowIHJlcXVlc3RzIGZvciBJL08gaW5pdGlhdGVkIGJ5IHNlbmRmaWxlDQpJVEVNICAgICAgICAg ICAgICAgICAgIFNJWkUgIExJTUlUICAgICBVU0VEICAgICBGUkVFICAgICAgUkVRIEZBSUwgU0xF RVANCg0KVU1BIEtlZ3M6ICAgICAgICAgICAgICAgMzg0LCAgICAgIDAsICAgICAyMTcsICAgICAg IDMsICAgICAyMTksICAgMCwgICAwDQpVTUEgWm9uZXM6ICAgICAgICAgICAgIDI2ODgsICAgICAg MCwgICAgIDIxNywgICAgICAgMCwgICAgIDIxOSwgICAwLCAgIDANClVNQSBTbGFiczogICAgICAg ICAgICAgICA4MCwgICAgICAwLCAyNjUyNzU4LCAxMzkyNzQyLDQzMTg4NTQ4OCwgICAwLCAgIDAN ClVNQSBSQ250U2xhYnM6ICAgICAgICAgICA4OCwgICAgICAwLCAgIDEzNTc5LCAgICA1MTQxLCAx ODgyNDU1LCAgIDAsICAgMA0KVU1BIEhhc2g6ICAgICAgICAgICAgICAgMjU2LCAgICAgIDAsICAg ICAgMjAsICAgICAgNTUsICAgICAgOTIsICAgMCwgICAwDQo0IEJ1Y2tldDogICAgICAgICAgICAg ICAgMzIsICAgICAgMCwgICAgMzI3NCwgICAxMzIyNiwyNTYzNjQ3OTEsICAgMCwgICAwDQo2IEJ1 Y2tldDogICAgICAgICAgICAgICAgNDgsICAgICAgMCwgICAgMTMwOSwgICAgODA3MCw2MzE4MDY4 MSwgICAwLCAgIDANCjggQnVja2V0OiAgICAgICAgICAgICAgICA2NCwgICAgICAwLCAgICAgMzY3 LCAgICA4MDY1LDY5MzI3MjEwLCAgMTEsICAgMA0KMTIgQnVja2V0OiAgICAgICAgICAgICAgIDk2 LCAgICAgIDAsICAgICA3NjIsICAgIDM0NjEsODEyMDg5MTEsNjQ1OCwgICAwDQoxNiBCdWNrZXQ6 ICAgICAgICAgICAgICAxMjgsICAgICAgMCwgICAgIDQwNSwgICAgNTA1MSw1MzI4NjI5MCwgICAw LCAgIDANCjMyIEJ1Y2tldDogICAgICAgICAgICAgIDI1NiwgICAgICAwLCAgICAgMzU0LCAgICAy MTY2LDY4OTI0MDUwLCAgNjMsICAgMA0KNjQgQnVja2V0OiAgICAgICAgICAgICAgNTEyLCAgICAg IDAsICAgIDcyNjYsICAgICAzNjYsNTc1OTg4MTAsMzU2MzA1NSwgICAwDQoxMjggQnVja2V0OiAg ICAgICAgICAgIDEwMjQsICAgICAgMCwgICAgIDQ0OSwgICAgIDQ0Nyw0NzU3ODE4Myw2NTI3Mywg ICAwDQoyNTYgQnVja2V0OiAgICAgICAgICAgIDIwNDgsICAgICAgMCwgICAgOTU2NCwgICAgIDEw MCw1MDM3NjA3MDcsNzAzOTUsICAgMA0Kdm1lbSBidGFnOiAgICAgICAgICAgICAgIDU2LCAgICAg IDAsICA2MjcwOTIsICAxNTY4OTAsMTc1NTUwNzEsNTUzNSwgICAwDQpWTSBPQkpFQ1Q6ICAgICAg ICAgICAgICAyNTYsICAgICAgMCwgICA1NDY5MCwgICAyMjIzMCwxOTU4NjgxNiwgICAwLCAgIDAN ClJBRElYIE5PREU6ICAgICAgICAgICAgIDE0NCwgICAgICAwLCAgNDUyMzA0LCAgMjE5OTk2LDEx ODQzNDk2NCwgICAwLCAgIDANCk1BUDogICAgICAgICAgICAgICAgICAgIDI0MCwgICAgICAwLCAg ICAgICAzLCAgICAgIDYxLCAgICAgICAzLCAgIDAsICAgMA0KS01BUCBFTlRSWTogICAgICAgICAg ICAgMTI4LCAgICAgIDAsICAgICAgIDgsICAgICAxMTYsICAgICAgIDgsICAgMCwgICAwDQpNQVAg RU5UUlk6ICAgICAgICAgICAgICAxMjgsICAgICAgMCwgICAgMTQzOSwgICAgMjcxNSw1NDUyNjQw MiwgICAwLCAgIDANClZNU1BBQ0U6ICAgICAgICAgICAgICAgIDQ0OCwgICAgICAwLCAgICAgIDQx LCAgICAgNjE2LCAxMzM3NzQwLCAgIDAsICAgMA0KZmFrZXBnOiAgICAgICAgICAgICAgICAgMTA0 LCAgICAgIDAsICAgICAgIDAsICAgICAxNTIsICAgICAgMjUsICAgMCwgICAwDQptdF96b25lOiAg ICAgICAgICAgICAgIDQxMTIsICAgICAgMCwgICAgIDM2OSwgICAgICAgMCwgICAgIDM2OSwgICAw LCAgIDANCjE2OiAgICAgICAgICAgICAgICAgICAgICAxNiwgICAgICAwLCAgMTEwNTE2LCAgIDE0 MjMxLDIwMDYyOTM5MDEsICAgMCwgICAwDQozMjogICAgICAgICAgICAgICAgICAgICAgMzIsICAg ICAgMCw1Nzg2OTI2MywxMTkwNjIzNyw1MDM3NTg2MDI5LCAgIDAsICAgMA0KNjQ6ICAgICAgICAg ICAgICAgICAgICAgIDY0LCAgICAgIDAsIDEzNTc3OTAsICA2MjE3NDYsMTI4NDU5NTMyMDcsICAg MCwgICAwDQoxMjg6ICAgICAgICAgICAgICAgICAgICAxMjgsICAgICAgMCwgMTk4MzE0MiwgIDg2 Mzk2MCwxMTAzMjA2NDQ3NywgICAwLCAgIDANCjI1NjogICAgICAgICAgICAgICAgICAgIDI1Niwg ICAgICAwLCAgNTUwMjc5LCAgICAxNDUxLDgxMzA2NDQ0MzYsICAgMCwgICAwDQo1MTI6ICAgICAg ICAgICAgICAgICAgICA1MTIsICAgICAgMCwgICAyODQ3MywgIDQwNTc2NywyMDE5MzAzNjgzLCAg IDAsICAgMA0KMTAyNDogICAgICAgICAgICAgICAgICAxMDI0LCAgICAgIDAsICAgMTU5OTEsICAg ICAyODEsMTM0MzAxNDEyLCAgIDAsICAgMA0KMjA0ODogICAgICAgICAgICAgICAgICAyMDQ4LCAg ICAgIDAsICAgNDI0NjQsICAgICA2NjYsMTkzMTEzOTczMSwgICAwLCAgIDANCjQwOTY6ICAgICAg ICAgICAgICAgICAgNDA5NiwgICAgICAwLCAgIDEzODcyLCAgICAgIDY0LDEwMjkwODYzMiwgICAw LCAgIDANCjgxOTI6ICAgICAgICAgICAgICAgICAgODE5MiwgICAgICAwLCAgICAgIDI2LCAgICAg IDI3LCAzMDY2ODgyLCAgIDAsICAgMA0KMTYzODQ6ICAgICAgICAgICAgICAgIDE2Mzg0LCAgICAg IDAsICAgICAgNDIsICAgICAgMTYsICAgIDc2MTEsICAgMCwgICAwDQozMjc2ODogICAgICAgICAg ICAgICAgMzI3NjgsICAgICAgMCwgICAgICAzOSwgICAgICAxNywgICA2NzgwNSwgICAwLCAgIDAN CjY1NTM2OiAgICAgICAgICAgICAgICA2NTUzNiwgICAgICAwLCAgICAgIDgxLCAgICAgIDIxLCAx MDQyNjA2LCAgIDAsICAgMA0KNjQgcGNwdTogICAgICAgICAgICAgICAgICA4LCAgICAgIDAsICAg IDE2NTQsICAgICA1MjIsICAgIDE4MTQsICAgMCwgICAwDQpTTEVFUFFVRVVFOiAgICAgICAgICAg ICAgODAsICAgICAgMCwgICAgMTI5NywgICAgIDgxMSwgICAgMTU3OSwgICAwLCAgIDANCkZpbGVz OiAgICAgICAgICAgICAgICAgICA4MCwgICAgICAwLCAgICAgMTM1LCAgICAyNjY1LDExNzM2NTMw LCAgIDAsICAgMA0KVFVSTlNUSUxFOiAgICAgICAgICAgICAgMTM2LCAgICAgIDAsICAgIDEyOTcs ICAgICA1NjMsICAgIDE1NzksICAgMCwgICAwDQpybF9lbnRyeTogICAgICAgICAgICAgICAgNDAs ICAgICAgMCwgICAgIDMzMCwgICAgMzI3MCwgICAgIDM0MSwgICAwLCAgIDANCnVtdHggcGk6ICAg ICAgICAgICAgICAgICA5NiwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAs ICAgMA0KTUFDIGxhYmVsczogICAgICAgICAgICAgIDQwLCAgICAgIDAsICAgICAgIDAsICAgICAg IDAsICAgICAgIDAsICAgMCwgICAwDQpQUk9DOiAgICAgICAgICAgICAgICAgIDEyMTYsICAgICAg MCwgICAgICA2MSwgICAgIDIxNSwgMTMzNzc1NywgICAwLCAgIDANClRIUkVBRDogICAgICAgICAg ICAgICAgMTE2OCwgICAgICAwLCAgICAxMTY5LCAgICAgMTI3LCAgICA1NjgxLCAgIDAsICAgMA0K Y3B1c2V0OiAgICAgICAgICAgICAgICAgIDcyLCAgICAgIDAsICAgICA4NzAsICAgIDExNjUsICAg IDE2MzcsICAgMCwgICAwDQphdWRpdF9yZWNvcmQ6ICAgICAgICAgIDEyNDgsICAgICAgMCwgICAg ICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCm1idWZfcGFja2V0OiAgICAgICAgICAg IDI1NiwgMTk1OTYxNjUsICAgMTg0MjUsICAgIDY1OTcsMTA3NDU5NzAyMTQsICAgMCwgICAwDQpt YnVmOiAgICAgICAgICAgICAgICAgICAyNTYsIDE5NTk2MTY1LCAgICAgNTQ1LCAgICA3ODgzLDEy ODc0NjQzNTQ4LCAgIDAsICAgMA0KbWJ1Zl9jbHVzdGVyOiAgICAgICAgICAyMDQ4LCAzMDYxOTAw LCAgIDI1MDU1LCAgICAxODI5LDE3Mzc2NDkwODcsICAgMCwgICAwDQptYnVmX2p1bWJvX3BhZ2U6 ICAgICAgIDQwOTYsIDE1MzA5NTAsICAgICAgIDAsICAgICAxMzcsMTQyMTcxNzIyMCwgICAwLCAg IDANCm1idWZfanVtYm9fOWs6ICAgICAgICAgOTIxNiwgNDUzNjE0LCAgICAgICAwLCAgICAgICAw LCAgICAgICAwLCAgIDAsICAgMA0KbWJ1Zl9qdW1ib18xNms6ICAgICAgIDE2Mzg0LCAyNTUxNTgs ICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQptYnVmX2V4dF9yZWZjbnQ6ICAg ICAgICAgIDQsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCmdf YmlvOiAgICAgICAgICAgICAgICAgIDI0OCwgICAgICAwLCAgICAgMTMyLCAgICAyNjY4LDczMjUy MjMwOTIsICAgMCwgICAwDQp0dHlpbnE6ICAgICAgICAgICAgICAgICAxNjAsICAgICAgMCwgICAg IDI0MCwgICAgIDIxMCwgICAgMTEyNSwgICAwLCAgIDANCnR0eW91dHE6ICAgICAgICAgICAgICAg IDI1NiwgICAgICAwLCAgICAgMTI2LCAgICAgMTQ0LCAgICAgNTkzLCAgIDAsICAgMA0KRE1BUl9N QVBfRU5UUlk6ICAgICAgICAgMTIwLCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAs ICAgMCwgICAwDQphdGFfcmVxdWVzdDogICAgICAgICAgICAzMzYsICAgICAgMCwgICAgICAgMCwg ICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnZ0bmV0X3R4X2hkcjogICAgICAgICAgICAyNCwg ICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KRlBVX3NhdmVfYXJl YTogICAgICAgICAgNTEyLCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwg ICAwDQp0YXNrcV96b25lOiAgICAgICAgICAgICAgNDgsICAgICAgMCwgICAgICAgMCwgICAgNjcy MywxNzAwODA1MzIsICAgMCwgICAwDQpWTk9ERTogICAgICAgICAgICAgICAgICA0NzIsICAgICAg MCwgIDIzNDM4NiwgICAyNzQ3OCwgNTAzNDY2NCwgICAwLCAgIDANClZOT0RFUE9MTDogICAgICAg ICAgICAgIDExMiwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAxLCAgIDAsICAgMA0K QlVGIFRSSUU6ICAgICAgICAgICAgICAgMTQ0LCAgICAgIDAsICAgICAgIDAsICAxMDU5NDgsICAg ICAgIDAsICAgMCwgICAwDQpTIFZGUyBDYWNoZTogICAgICAgICAgICAxMDgsICAgICAgMCwgIDE0 NjIzOCwgICA4MzE1MiwgNTczMjY5OCwgICAwLCAgIDANClNUUyBWRlMgQ2FjaGU6ICAgICAgICAg IDE0OCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgIDEyLCAgIDAsICAgMA0KTCBWRlMg Q2FjaGU6ICAgICAgICAgICAgMzI4LCAgICAgIDAsICAgNTY4MTAsICAgIDE4NDYsICA4MTA5ODgs ICAgMCwgICAwDQpMVFMgVkZTIENhY2hlOiAgICAgICAgICAzNjgsICAgICAgMCwgICAgICAgMCwg ICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCk5BTUVJOiAgICAgICAgICAgICAgICAgMTAyNCwg ICAgICAwLCAgICAgICAwLCAgICAgMjYwLDQ0ODg1ODUwLCAgIDAsICAgMA0KTkNMTk9ERTogICAg ICAgICAgICAgICAgNTI4LCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgMTYsICAgMCwg ICAwDQpESVJIQVNIOiAgICAgICAgICAgICAgIDEwMjQsICAgICAgMCwgICAgICAgMCwgICAgICAg MCwgICAgICAgMCwgICAwLCAgIDANCnBpcGU6ICAgICAgICAgICAgICAgICAgIDc0NCwgICAgICAw LCAgICAgICA2LCAgICAgMjM5LCAgMTA4MjY1LCAgIDAsICAgMA0KcHJvY2Rlc2M6ICAgICAgICAg ICAgICAgMTI4LCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpy YW5nZV9zZWdfY2FjaGU6ICAgICAgICAgNjQsICAgICAgMCwgMjI1ODY0MCwgMTYyNTE2NCw2ODYw MjkxOTExMSwgICAwLCAgIDANCnppb19jYWNoZTogICAgICAgICAgICAgIDkyMCwgICAgICAwLCAg ICAgMjM0LCAgMTA1MjU4LDMxNDE3MTEzMzgzLCAgIDAsICAgMA0KemlvX2xpbmtfY2FjaGU6ICAg ICAgICAgIDQ4LCAgICAgIDAsICAgICAyMzAsICAxMDU1OTUsMzQwOTE5NTkzMzgsICAgMCwgICAw DQp6aW9fYnVmXzUxMjogICAgICAgICAgICA1MTIsICAgICAgMCwgIDI3NTE5OCwgICA2ODAxMCw3 MTI2MjQ4MywgICAwLCAgIDANCnppb19kYXRhX2J1Zl81MTI6ICAgICAgIDUxMiwgICAgICAwLCAg ICAgIDMwLCAgICAgNTcwLCA5NTEyNTYxLCAgIDAsICAgMA0KemlvX2J1Zl8xMDI0OiAgICAgICAg ICAxMDI0LCAgICAgIDAsICAgICAgIDQsICAgICAyODgsIDg0NzUwNTAsICAgMCwgICAwDQp6aW9f ZGF0YV9idWZfMTAyNDogICAgIDEwMjQsICAgICAgMCwgICAgICAyMiwgICAgIDc3NCwgNDI2MDM3 NywgICAwLCAgIDANCnppb19idWZfMTUzNjogICAgICAgICAgMTUzNiwgICAgICAwLCAgICAgICAz LCAgICAgMTgzLCA1Mjc3MDk4LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzE1MzY6ICAgICAxNTM2 LCAgICAgIDAsICAgICAgIDcsICAgICAxNzksIDY2MDQ4MjcsICAgMCwgICAwDQp6aW9fYnVmXzIw NDg6ICAgICAgICAgIDIwNDgsICAgICAgMCwgICAgIDM2MSwgICAgIDEyMSwxMzMyMDMyNSwgICAw LCAgIDANCnppb19kYXRhX2J1Zl8yMDQ4OiAgICAgMjA0OCwgICAgICAwLCAgICAgIDE2LCAgICAg IDk2LCA2MDYwMzE0LCAgIDAsICAgMA0KemlvX2J1Zl8yNTYwOiAgICAgICAgICAyNTYwLCAgICAg IDAsICAgICAgIDcsICAgICAxNDUsIDQ4NDE3ODEsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMjU2 MDogICAgIDI1NjAsICAgICAgMCwgICAgICAgMiwgICAgICA2NiwgNDMyNzg0MCwgICAwLCAgIDAN Cnppb19idWZfMzA3MjogICAgICAgICAgMzA3MiwgICAgICAwLCAgICAgMTA4LCAgICAgNzY2LCA4 ODQ0NTczLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzMwNzI6ICAgICAzMDcyLCAgICAgIDAsICAg ICAgIDIsICAgICAgNjEsIDM0MzU2MTcsICAgMCwgICAwDQp6aW9fYnVmXzM1ODQ6ICAgICAgICAg IDM1ODQsICAgICAgMCwgICAgICAgMywgICAgICA0NCwgNDc1NjYyMywgICAwLCAgIDANCnppb19k YXRhX2J1Zl8zNTg0OiAgICAgMzU4NCwgICAgICAwLCAgICAgICA2LCAgICAgIDI2LCAzMjM4OTYy LCAgIDAsICAgMA0KemlvX2J1Zl80MDk2OiAgICAgICAgICA0MDk2LCAgICAgIDAsIDE2MDAyMTEs ICAgMjgxNzksMzM4NTUxODczMiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl80MDk2OiAgICAgNDA5 NiwgICAgICAwLCAgMzA0NzY5LCAgICAyMDI5LDIzNDc2MDc2NywgICAwLCAgIDANCnppb19idWZf NTEyMDogICAgICAgICAgNTEyMCwgICAgICAwLCAgICAgMzAwLCAgICAgIDI0LDE5MTk0OTgwLCAg IDAsICAgMA0KemlvX2RhdGFfYnVmXzUxMjA6ICAgICA1MTIwLCAgICAgIDAsICAgICAgIDQsICAg ICAgMjYsIDE4MTU4MTMsICAgMCwgICAwDQp6aW9fYnVmXzYxNDQ6ICAgICAgICAgIDYxNDQsICAg ICAgMCwgICAgICAgMCwgICAgICAyNCwxMzQ4NzA3OSwgICAwLCAgIDANCnppb19kYXRhX2J1Zl82 MTQ0OiAgICAgNjE0NCwgICAgICAwLCAgICAgICA2LCAgICAgIDI1LCAzMzUzNTYxLCAgIDAsICAg MA0KemlvX2J1Zl83MTY4OiAgICAgICAgICA3MTY4LCAgICAgIDAsICAgICAgIDAsICAgICAgMjIs IDk1MzA0OTEsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNzE2ODogICAgIDcxNjgsICAgICAgMCwg ICAgICAgMCwgICAgICAyNywgMjMxNTg4MCwgICAwLCAgIDANCnppb19idWZfODE5MjogICAgICAg ICAgODE5MiwgICAgICAwLCAgICAgICAxLCAgICAgNjY3LDI1MTI0MjE5NSwgICAwLCAgIDANCnpp b19kYXRhX2J1Zl84MTkyOiAgICAgODE5MiwgICAgICAwLCAgICAgICA1LCAgICAgIDI2LCAxNzY0 MDc0LCAgIDAsICAgMA0KemlvX2J1Zl8xMDI0MDogICAgICAgIDEwMjQwLCAgICAgIDAsICAgICAg IDAsICAgICAgMzksMTU4NTA0NjQsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTAyNDA6ICAgMTAy NDAsICAgICAgMCwgICAgICAgMiwgICAgICAyNywgMzQwNTAzMSwgICAwLCAgIDANCnppb19idWZf MTIyODg6ICAgICAgICAxMjI4OCwgICAgICAwLCAgICAgICAxLCAgICAgIDkwLDc3ODI5MTg1LCAg IDAsICAgMA0KemlvX2RhdGFfYnVmXzEyMjg4OiAgIDEyMjg4LCAgICAgIDAsICAgICAgIDMsICAg ICAgMzAsIDMyMjM0MDAsICAgMCwgICAwDQp6aW9fYnVmXzE0MzM2OiAgICAgICAgMTQzMzYsICAg ICAgMCwgICAgICAgMSwgICAgICA2OSwxMzE4MDMwNiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8x NDMzNjogICAxNDMzNiwgICAgICAwLCAgICAgICA0LCAgICAgIDI1LCAyODY5NDY3LCAgIDAsICAg MA0KemlvX2J1Zl8xNjM4NDogICAgICAgIDE2Mzg0LCAgICAgIDAsICAyNzI2MzgsICAgIDQyMjEs MzU5MzQ5MDcxLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzE2Mzg0OiAgIDE2Mzg0LCAgICAgIDAs ICAgIDU2MzYsICAgICAyNTIsIDQ0Nzg2MjIsICAgMCwgICAwDQp6aW9fYnVmXzIwNDgwOiAgICAg ICAgMjA0ODAsICAgICAgMCwgICAgICAgMCwgICAgICA5Miw1ODg1ODA5MywgICAwLCAgIDANCnpp b19kYXRhX2J1Zl8yMDQ4MDogICAyMDQ4MCwgICAgICAwLCAgICAgICA4LCAgICAgIDI2LCAzNjk0 MTgyLCAgIDAsICAgMA0KemlvX2J1Zl8yNDU3NjogICAgICAgIDI0NTc2LCAgICAgIDAsICAgICAg IDAsICAgICAgNjUsNTc2MzI0NjQsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMjQ1NzY6ICAgMjQ1 NzYsICAgICAgMCwgICAgICAgMiwgICAgICAyNiwgMzI0MTgwOCwgICAwLCAgIDANCnppb19idWZf Mjg2NzI6ICAgICAgICAyODY3MiwgICAgICAwLCAgICAgICAxLCAgICAgIDU0LDU3MTk0NTc2LCAg IDAsICAgMA0KemlvX2RhdGFfYnVmXzI4NjcyOiAgIDI4NjcyLCAgICAgIDAsICAgICAgIDAsICAg ICAgMjksIDMwNTUyNzMsICAgMCwgICAwDQp6aW9fYnVmXzMyNzY4OiAgICAgICAgMzI3NjgsICAg ICAgMCwgICAgICAgMCwgICAgIDEwNCw1NTQyMjk2NCwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8z Mjc2ODogICAzMjc2OCwgICAgICAwLCAgICAgIDQyLCAgICAgIDcxLDk1OTk1NTEwLCAgIDAsICAg MA0KemlvX2J1Zl8zNjg2NDogICAgICAgIDM2ODY0LCAgICAgIDAsICAgICAgIDAsICAgICAgNDcs MzA5NDQ5NzAsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMzY4NjQ6ICAgMzY4NjQsICAgICAgMCwg ICAgICAgMSwgICAgICAyNiwgMjkwNTc4MiwgICAwLCAgIDANCnppb19idWZfNDA5NjA6ICAgICAg ICA0MDk2MCwgICAgICAwLCAgICAgICAwLCAgICAgIDU3LDMwMDg0MzM5LCAgIDAsICAgMA0Kemlv X2RhdGFfYnVmXzQwOTYwOiAgIDQwOTYwLCAgICAgIDAsICAgICAgIDYsICAgICAgMjYsIDM0NjM3 NDcsICAgMCwgICAwDQp6aW9fYnVmXzQ1MDU2OiAgICAgICAgNDUwNTYsICAgICAgMCwgICAgICAg MCwgICAgICA2OSwyODI3NTIxOCwgICAwLCAgIDANCnppb19kYXRhX2J1Zl80NTA1NjogICA0NTA1 NiwgICAgICAwLCAgICAgICAxLCAgICAgIDI5LCAzNjQ4NTMzLCAgIDAsICAgMA0KemlvX2J1Zl80 OTE1MjogICAgICAgIDQ5MTUyLCAgICAgIDAsICAgICAgIDAsICAgICAgODEsMzEyNDc0MzcsICAg MCwgICAwDQp6aW9fZGF0YV9idWZfNDkxNTI6ICAgNDkxNTIsICAgICAgMCwgICAgICAgMSwgICAg ICAyOSwgMzEwNzI5OSwgICAwLCAgIDANCnppb19idWZfNTMyNDg6ICAgICAgICA1MzI0OCwgICAg ICAwLCAgICAgICAwLCAgICAgMTM1LDI1MTU2NzY5LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzUz MjQ4OiAgIDUzMjQ4LCAgICAgIDAsICAgICAgIDAsICAgICAgMjYsIDI1MzQ3MTksICAgMCwgICAw DQp6aW9fYnVmXzU3MzQ0OiAgICAgICAgNTczNDQsICAgICAgMCwgICAgICAgMCwgICAgICA5NCwy NjU3OTIyNiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl81NzM0NDogICA1NzM0NCwgICAgICAwLCAg ICAgICAxLCAgICAgIDI3LCAyODc5NTE5LCAgIDAsICAgMA0KemlvX2J1Zl82MTQ0MDogICAgICAg IDYxNDQwLCAgICAgIDAsICAgICAgIDAsICAgICAgNTMsIDgyMTE5NTIsICAgMCwgICAwDQp6aW9f ZGF0YV9idWZfNjE0NDA6ICAgNjE0NDAsICAgICAgMCwgICAgICAgMSwgICAgICAyMSwgICAgMjA4 OSwgICAwLCAgIDANCnppb19idWZfNjU1MzY6ICAgICAgICA2NTUzNiwgICAgICAwLCAgICAgICAw LCAgICAxMDY1LDMzMjYxNjcxMiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl82NTUzNjogICA2NTUz NiwgICAgICAwLCAgMjcxMDE3LCAgICAgODg1LDQ4MzkxNzQ2MywgICAwLCAgIDANCnppb19idWZf Njk2MzI6ICAgICAgICA2OTYzMiwgICAgICAwLCAgICAgICAwLCAgICAgIDIyLCA2MjQ0MTg2LCAg IDAsICAgMA0KemlvX2RhdGFfYnVmXzY5NjMyOiAgIDY5NjMyLCAgICAgIDAsICAgICAgIDEsICAg ICAgMjIsICAgIDI0NDIsICAgMCwgICAwDQp6aW9fYnVmXzczNzI4OiAgICAgICAgNzM3MjgsICAg ICAgMCwgICAgICAgMCwgICAgICAyMiwgNzEwNTQ2NywgICAwLCAgIDANCnppb19kYXRhX2J1Zl83 MzcyODogICA3MzcyOCwgICAgICAwLCAgICAgICAwLCAgICAgIDI1LCAgICAyMDc4LCAgIDAsICAg MA0KemlvX2J1Zl83NzgyNDogICAgICAgIDc3ODI0LCAgICAgIDAsICAgICAgIDAsICAgICAgMjIs IDUzNjAzODQsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNzc4MjQ6ICAgNzc4MjQsICAgICAgMCwg ICAgICAgMCwgICAgICAyMiwgICAgMjA3NiwgICAwLCAgIDANCnppb19idWZfODE5MjA6ICAgICAg ICA4MTkyMCwgICAgICAwLCAgICAgICAwLCAgICAgIDI1LCA2OTc2MjM5LCAgIDAsICAgMA0Kemlv X2RhdGFfYnVmXzgxOTIwOiAgIDgxOTIwLCAgICAgIDAsICAgICAgIDAsICAgICAgMjAsICAgIDIw NzcsICAgMCwgICAwDQp6aW9fYnVmXzg2MDE2OiAgICAgICAgODYwMTYsICAgICAgMCwgICAgICAg MCwgICAgICAyNywgNTM4NzI3MywgICAwLCAgIDANCnppb19kYXRhX2J1Zl84NjAxNjogICA4NjAx NiwgICAgICAwLCAgICAgICAwLCAgICAgIDIxLCAgICAyMDcwLCAgIDAsICAgMA0KemlvX2J1Zl85 MDExMjogICAgICAgIDkwMTEyLCAgICAgIDAsICAgICAgIDAsICAgICAgMjMsIDYzNjI1ODAsICAg MCwgICAwDQp6aW9fZGF0YV9idWZfOTAxMTI6ICAgOTAxMTIsICAgICAgMCwgICAgICAgMSwgICAg ICAyMywgICAgMjU1MSwgICAwLCAgIDANCnppb19idWZfOTQyMDg6ICAgICAgICA5NDIwOCwgICAg ICAwLCAgICAgICAwLCAgICAgIDIzLCA2MDY3MTE1LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzk0 MjA4OiAgIDk0MjA4LCAgICAgIDAsICAgICAgIDAsICAgICAgMjQsICAgIDIxMTQsICAgMCwgICAw DQp6aW9fYnVmXzk4MzA0OiAgICAgICAgOTgzMDQsICAgICAgMCwgICAgICAgMCwgICAgICAyMiwg ODU4NzkyOSwgICAwLCAgIDANCnppb19kYXRhX2J1Zl85ODMwNDogICA5ODMwNCwgICAgICAwLCAg ICAgICAwLCAgICAgIDIxLCAgICAyMTA0LCAgIDAsICAgMA0KemlvX2J1Zl8xMDI0MDA6ICAgICAg MTAyNDAwLCAgICAgIDAsICAgICAgIDAsICAgICAgMjYsIDY1ODkwNzcsICAgMCwgICAwDQp6aW9f ZGF0YV9idWZfMTAyNDAwOiAxMDI0MDAsICAgICAgMCwgICAgICAgMCwgICAgICAyMSwgICAgMjA2 MSwgICAwLCAgIDANCnppb19idWZfMTA2NDk2OiAgICAgIDEwNjQ5NiwgICAgICAwLCAgICAgICAw LCAgICAgIDI0LCA3MjgwNTg4LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzEwNjQ5NjogMTA2NDk2 LCAgICAgIDAsICAgICAgIDAsICAgICAgMjQsICAgIDIwMDksICAgMCwgICAwDQp6aW9fYnVmXzEx MDU5MjogICAgICAxMTA1OTIsICAgICAgMCwgICAgICAgMCwgICAgICAyNSwgNzc2NzcwNiwgICAw LCAgIDANCnppb19kYXRhX2J1Zl8xMTA1OTI6IDExMDU5MiwgICAgICAwLCAgICAgICAwLCAgICAg IDIwLCAgICAxOTAzLCAgIDAsICAgMA0KemlvX2J1Zl8xMTQ2ODg6ICAgICAgMTE0Njg4LCAgICAg IDAsICAgICAgIDAsICAgICAgMjUsIDkxMTg2MzMsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTE0 Njg4OiAxMTQ2ODgsICAgICAgMCwgICAgICAgMSwgICAgICAyMSwgICAgMTkwMSwgICAwLCAgIDAN Cnppb19idWZfMTE4Nzg0OiAgICAgIDExODc4NCwgICAgICAwLCAgICAgICAwLCAgICAgIDI0LCA4 MjY4ODMwLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzExODc4NDogMTE4Nzg0LCAgICAgIDAsICAg ICAgIDAsICAgICAgMjMsICAgIDE3NDAsICAgMCwgICAwDQp6aW9fYnVmXzEyMjg4MDogICAgICAx MjI4ODAsICAgICAgMCwgICAgICAgMCwgICAgICAzMSwxMTIyNDk0MCwgICAwLCAgIDANCnppb19k YXRhX2J1Zl8xMjI4ODA6IDEyMjg4MCwgICAgICAwLCAgICAgICAwLCAgICAgIDIwLCAgICAxNjY4 LCAgIDAsICAgMA0KemlvX2J1Zl8xMjY5NzY6ICAgICAgMTI2OTc2LCAgICAgIDAsICAgICAgIDAs ICAgICAgMjcsMTA3MTM1MjUsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTI2OTc2OiAxMjY5NzYs ICAgICAgMCwgICAgICAgMCwgICAgICAxOSwgICAgMTY0OCwgICAwLCAgIDANCnppb19idWZfMTMx MDcyOiAgICAgIDEzMTA3MiwgICAgICAwLCAgIDE0MDU2LCAgICAgNjYzLDI1NDE5NDcyMCwgICAw LCAgIDANCnppb19kYXRhX2J1Zl8xMzEwNzI6IDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAg IDE0LCAgIDU2Njk2LCAgIDAsICAgMA0KbHo0X2N0eDogICAgICAgICAgICAgIDE2Mzg0LCAgICAg IDAsICAgICAgIDAsICAgICAgNTIsMTY1ODgxNTQwNiwgICAwLCAgIDANCnNhX2NhY2hlOiAgICAg ICAgICAgICAgICA4MCwgICAgICAwLCAgMjM0Mjk0LCAgIDgzOTU2LCA1MDMzODc2LCAgIDAsICAg MA0KZG5vZGVfdDogICAgICAgICAgICAgICAgNzUyLCAgICAgIDAsICAzMjUxOTcsICAgMjM3OTgs IDc4NDYxNTIsICAgMCwgICAwDQpkbXVfYnVmX2ltcGxfdDogICAgICAgICAyMzIsICAgICAgMCwg MjQyMTEzOSwgMTA5MjI2OCwxMDE4ODQwODU0LCAgIDAsICAgMA0KYXJjX2J1Zl9oZHJfdDogICAg ICAgICAgMjE2LCAgICAgIDAsMzE0NzAxNDEsIDY1ODA0OTEsNzA0Nzc4ODgwLCAgIDAsICAgMA0K YXJjX2J1Zl90OiAgICAgICAgICAgICAgIDcyLCAgICAgIDAsIDIzMDM1MjksIDE0NDM4OTYsODQy OTg0NTQwLCAgIDAsICAgMA0KemlsX2x3Yl9jYWNoZTogICAgICAgICAgMTkyLCAgICAgIDAsICAg ICA4NDUsICAgIDExMTUsMTE4OTc5MzMxLCAgIDAsICAgMA0KemZzX3pub2RlX2NhY2hlOiAgICAg ICAgMzY4LCAgICAgIDAsICAyMzQyOTQsICAgMzgxNDYsIDUwMzM4NzYsICAgMCwgICAwDQpNb3Vu dHBvaW50czogICAgICAgICAgICA4MTYsICAgICAgMCwgICAgICAzMywgICAgIDEzNywgICAgICAz NywgICAwLCAgIDANCmtzaWdpbmZvOiAgICAgICAgICAgICAgIDExMiwgICAgICAwLCAgICAgMjY3 LCAgICAzMTI4LCAgIDUxNDIyLCAgIDAsICAgMA0KaXRpbWVyOiAgICAgICAgICAgICAgICAgMzUy LCAgICAgIDAsICAgICAgIDEsICAgICAgMTAsICAgICAgIDEsICAgMCwgICAwDQpLTk9URTogICAg ICAgICAgICAgICAgICAxMjgsICAgICAgMCwgICAgICAgMCwgICAgMjEzOSwgIDM4NjA0MSwgICAw LCAgIDANCnNvY2tldDogICAgICAgICAgICAgICAgIDY5NiwgMTU3MTYyMCwgICAgICA2NywgICAg IDI4MywgIDU1MTk1NywgICAwLCAgIDANCmlwcTogICAgICAgICAgICAgICAgICAgICA1NiwgIDk1 NzA4LCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KdWRwX2lucGNiOiAgICAg ICAgICAgICAgMzkyLCAxNTcxNjIwLCAgICAgIDE4LCAgICAgNjEyLCAgMjQzMjE1LCAgIDAsICAg MA0KdWRwY2I6ICAgICAgICAgICAgICAgICAgIDE2LCAxNTcxNzYyLCAgICAgIDE4LCAgICA1MDAy LCAgMjQzMjE1LCAgIDAsICAgMA0KdGNwX2lucGNiOiAgICAgICAgICAgICAgMzkyLCAxNTcxNjIw LCAgICAgIDMzLCAgICAgNTk3LCAgMjk1NjE5LCAgIDAsICAgMA0KdGNwY2I6ICAgICAgICAgICAg ICAgICAxMDI0LCAxNTcxNjIwLCAgICAgIDMzLCAgICAgMjgzLCAgMjk1NjE5LCAgIDAsICAgMA0K dGNwdHc6ICAgICAgICAgICAgICAgICAgIDg4LCAgMjc4MTAsICAgICAgIDAsICAgIDE2NjUsICAg IDUwNTksICAgMCwgICAwDQpzeW5jYWNoZTogICAgICAgICAgICAgICAxNjAsICAxNTM3NSwgICAg ICAgMCwgICAgIDUwMCwgIDI3OTMzMSwgICAwLCAgIDANCmhvc3RjYWNoZTogICAgICAgICAgICAg IDEzNiwgIDE1MzcwLCAgICAgICAyLCAgICAgMzc1LCAgICAgIDg1LCAgIDAsICAgMA0KdGNwcmVh c3M6ICAgICAgICAgICAgICAgIDQwLCAxOTE0MDAsICAgICAgIDAsICAgIDIxMDAsICAgIDk1MDQs ICAgMCwgICAwDQpzYWNraG9sZTogICAgICAgICAgICAgICAgMzIsICAgICAgMCwgICAgICAgMCwg ICAgMjEyNSwgIDEwNzUyOCwgICAwLCAgIDANCnNjdHBfZXA6ICAgICAgICAgICAgICAgMTQwOCwg MTU3MTYyMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnNjdHBfYXNvYzog ICAgICAgICAgICAgMjQxNiwgIDQwMDAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAs ICAgMA0Kc2N0cF9sYWRkcjogICAgICAgICAgICAgIDQ4LCAgODAwMTIsICAgICAgIDAsICAgICAg IDAsICAgICAgIDQsICAgMCwgICAwDQpzY3RwX3JhZGRyOiAgICAgICAgICAgICA3MjgsICA4MDAw MCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnNjdHBfY2h1bms6ICAgICAg ICAgICAgIDEzNiwgNDAwMDI2LCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0K c2N0cF9yZWFkcTogICAgICAgICAgICAgMTA0LCA0MDAwMjYsICAgICAgIDAsICAgICAgIDAsICAg ICAgIDAsICAgMCwgICAwDQpzY3RwX3N0cmVhbV9tc2dfb3V0OiAgICAxMDQsIDQwMDAyNiwgICAg ICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnNjdHBfYXNjb25mOiAgICAgICAgICAg ICA0MCwgNDAwMDAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0Kc2N0cF9h c2NvbmZfYWNrOiAgICAgICAgIDQ4LCA0MDAwNjAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAs ICAgMCwgICAwDQp1ZHBsaXRlX2lucGNiOiAgICAgICAgICAzOTIsIDE1NzE2MjAsICAgICAgIDAs ICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpyaXBjYjogICAgICAgICAgICAgICAgICAzOTIs IDE1NzE2MjAsICAgICAgIDAsICAgICAxNTAsICAgICAgMTcsICAgMCwgICAwDQp1bnBjYjogICAg ICAgICAgICAgICAgICAyNDAsIDE1NzE2MzIsICAgICAgMTUsICAgICA5MjksICAgMTMxMDEsICAg MCwgICAwDQpydGVudHJ5OiAgICAgICAgICAgICAgICAyMDAsICAgICAgMCwgICAgICAxNCwgICAg IDE0NiwgICAgICAxNCwgICAwLCAgIDANCnNlbGZkOiAgICAgICAgICAgICAgICAgICA1NiwgICAg ICAwLCAgICAgNTE5LCAgICA0MzA5LDI1NjQ0NDUwLCAgIDAsICAgMA0KU1dBUE1FVEE6ICAgICAg ICAgICAgICAgMjg4LCA2MTIzODA2LCAgICAgODg0LCAgICAyMjg4LCAgNDk0MzkxLCAgIDAsICAg MA0KaWNsX2Nvbm46ICAgICAgICAgICAgICAgMTUyLCAgICAgIDAsICAgICAgIDgsICAgICAzMzAs ICAgICAgODIsICAgMCwgICAwDQppY2xfcGR1OiAgICAgICAgICAgICAgICAgODAsICAgICAgMCwg ICAgICAgMCwgICAgMzEwMCw2MTk0MzY0ODksICAgMCwgICAwDQppc2NzaV9vdXRzdGFuZGluZzog ICAgICAgNDAsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCmNm aV9sdW46ICAgICAgICAgICAgICAgIDM0NCwgICAgICAwLCAgICAgICA5LCAgICAgIDQ2LCAgICAg IDExLCAgIDAsICAgMA0KY2ZpX21ldGF0YXNrOiAgICAgICAgICAgMzIwLCAgICAgIDAsICAgICAg IDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpjZmlzY3NpX2RhdGFfd2FpdDogICAgICAg NTYsICAgICAgMCwgICAgICAgMCwgICAgNTMyNSwyMzk2NTczMDUsICAgMCwgICAwDQpiZWlvOiAg ICAgICAgICAgICAgICAgICAzNjAsICAgICAgMCwgICAgICAgMCwgICAgIDUzOSwyODQ0MTExMjQs ICAgMCwgICAwDQpjYmxrMjogICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwg ICAgICAyNywgMzM0NDk3MiwgICAwLCAgIDANCmNibGszOiAgICAgICAgICAgICAgIDEzMTA3Miwg ICAgICAwLCAgICAgICAwLCAgICAgIDIzLCAgICAyNzU3LCAgIDAsICAgMA0KY2JsazQ6ICAgICAg ICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgNTUsNDYxNTIxMDYsICAgMCwg ICAwDQpjYmxrNTogICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICAz Myw5MDk2MjMxNywgICAwLCAgIDANCmNibGs2OiAgICAgICAgICAgICAgIDEzMTA3MiwgICAgICAw LCAgICAgICAwLCAgICAgIDE3LDM1MjY0Mzg2LCAgIDAsICAgMA0KY2Jsazc6ICAgICAgICAgICAg ICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgMjgsNTY5ODk2NzUsICAgMCwgICAwDQpj YmxrODogICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICAzNSwzNzc0 NzkxNiwgICAwLCAgIDANCmNibGs5OiAgICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAg ICAwLCAgICAgIDI3LCA3MDk4NDQ5LCAgIDAsICAgMA0KY2JsazEwOiAgICAgICAgICAgICAgMTMx MDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgMjYsIDk1NTY5NzYsICAgMCwgICAw --001a1145eae834d0fc051ac04b07 Content-Type: text/plain; charset=US-ASCII; name="2015-07-06 - San1.txt" Content-Disposition: attachment; filename="2015-07-06 - San1.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_ic1uffuk1 DQoNCiBwcm9jcyAgICAgIG1lbW9yeSAgICAgIHBhZ2UgICAgICAgICAgICAgICAgICAgIGRpc2tz ICAgICBmYXVsdHMgICAgICAgICBjcHUNCiByIGIgdyAgICAgYXZtICAgIGZyZSAgIGZsdCAgcmUg IHBpICBwbyAgICBmciAgc3IgZGEwIGRhMSAgIGluICAgc3kgICBjcyB1cyBzeSBpZA0KIDAgMCA0 ICAgIDc1OU0gIDk5NjlNICAgICAwICAgMCAgIDAgICAwICAgICAwICAgMCAgMTQgIDEwIDUzMjYg ICAzMyAyNDc4NSAgMCAgMiA5OA0KIDAgMCA0ICAgIDc1OU0gIDk5NjBNICAgICAwICAgMCAgIDAg ICAwICAgICAwICAgMCAgIDYgIDEzIDM4NjUgICAzNiAxODk0NSAgMCAgMSA5OQ0KIDEgMCA0ICAg IDc1OU0gIDk4ODVNICAgICAwICAgMCAgIDAgICAwICAgICAwICAgMCAgMzcgIDM1IDQ2MjEgICAz MiAyMjQ3MyAgMCAyMiA3OA0KIDAgMCA0ICAgIDc2OE0gIDk2OTVNICAgIDExICAgMCAgIDAgICAw ICAgICAwICAgMCAgIDggIDEwIDI4NTggICA0MiAxMjQ5MiAgMCA1MiA0OA0KIDAgMCA0ICAgIDc2 OE0gIDk0OTJNICAgICAwICAgMCAgIDAgICAwICAgICAwICAgMCAgMTggIDM0IDI0OTUgICAzMSAx NjAwNiAgMCA4NiAxNA0KIDAgMCA0ICAgIDc2OE0gIDk0MTRNICAgICAwICAgMCAgIDAgICAwICAg ICAwICAgMCAgIDYgICA3IDIyNzEgICAzMiAxMTM4NCAgMCAzMiA2OA0KIDEgMCA0ICAgIDc2OE0g IDQ2NjdNICAgICAwICAgMCAgIDAgICAwICAgICAwICAgMCAgIDYgICA0IDE3NzIgICAzMCA3Mjk0 ICAwIDYzIDM3DQogMCAwIDQgICAgNzY4TSAgODE5MkIgICAgIDAgICAwICAgMCAgIDAgICAgIDAg ICAwICAgMSAgIDYgIDU3NyAgIDMyIDIwNTcgIDAgNDAgNjANCiAwIDAgNCAgICA3NjhNICA4MTky QiAgICAgMCAgIDAgICAwICAgMCAgICAgMCAgIDAgICAwICAgMCAgIDgyICAgMzMgIDM3MiAgMCAg MCAxMDANCiAwIDAgNCAgICA3NjhNICA4MTkyQiAgICAgMCAgIDAgICAwICAgMCAgICAgMCAgIDAg ICAwICAgMCAgMTE0ICAgMzcgIDUxNiAgMCAgMCAxMDANCiAwIDAgNCAgICA3NjhNICA4MTkyQiAg ICAgMCAgIDAgICAwICAgMCAgICAgMCAgIDAgICAwICAgMCAgIDYxICAgMzQgIDMyNyAgMCAgMCAx MDANCiAwIDAgNCAgICA3NjhNICA4MTkyQiAgICAgMCAgIDAgICAwICAgMCAgICAgMCAgIDAgICAw ICAgMCAgIDgyICAgMzQgIDM5NCAgMCAgMCAxMDANCiAwIDAgNCAgICA3NjhNICA4MTkyQiAgICAg MCAgIDAgICAwICAgMCAgICAgMCAgIDAgICAwICAgMCAgIDU5ICAgMzYgIDM1OCAgMCAgMCAxMDAN CiAwIDAgNCAgICA3NjhNICA4MTkyQiAgICAgMCAgIDAgICAwICAgMCAgICAgMCAgIDAgICAwICAg MCAgIDc2ICAgMzQgIDM2OSAgMCAgMCAxMDANCiAwIDAgNCAgICA3NjhNICA4MTkyQiAgICAgMCAg IDAgICAwICAgMCAgICAgMCAgIDAgICAwICAgMCAgIDYzICAgMzUgIDMxOCAgMCAgMCAxMDANCiAw IDAgNCAgICA3NjhNICA4MTkyQiAgICAgMCAgIDAgICAwICAgMCAgICAgMCAgIDAgICAwICAgMCAg IDc2ICAgMzUgIDM2NyAgMCAgMCAxMDANCiAwIDAgNCAgICA3NjhNICA4MTkyQiAgICAgMCAgIDAg ICAwICAgMCAgICAgMCAgIDAgICAwICAgMCAgIDY0ICAgMzQgIDMzNyAgMCAgMCAxMDANCiAwIDAg NCAgICA3NjhNICA4MTkyQiAgICAgMCAgIDAgICAwICAgMCAgICAgMCAgIDAgICAwICAgMCAgIDc2 ICAgMzUgIDM4MCAgMCAgMCAxMDANCiAwIDAgNCAgICA3NjhNICA4MTkyQiAgICAgMCAgIDAgICAw ICAgMCAgICAgMCAgIDAgICAwICAgMCAgIDc3ICAgMzQgIDM3OSAgMCAgMCAxMDANCiAwIDAgNCAg ICA3NjhNICA4MTkyQiAgICAgMCAgIDAgICAwICAgMCAgICAgMCAgIDAgICAwICAgMCAgIDc0ICAg MzUgIDYyMSAgMCAgMSA5OQ0KIDAgMCA0ICAgIDc2OE0gIDgxOTJCICAgICAwICAgMCAgIDAgICAw ICAgIDEwICAgMCAgIDAgICAwICAgNTcgICAzNSAxMjMyICAwICAyIDk4DQogMCAwIDQgICAgNzY4 TSAgODE5MkIgICAgIDAgICAwICAgMCAgIDAgICAgIDAgICAwICAgMCAgIDAgICA2NiAgIDMzIDE0 MDIgIDAgIDIgOTgNCiAwIDAgNCAgICA3NjhNICA4MTkyQiAgICAgMCAgIDAgICAwICAgMCAgICAg MCAgIDAgICAwICAgMCAgIDUwICAgMzMgMTM3NSAgMCAgMiA5OA0KIDAgMCA0ICAgIDc2OE0gIDgx OTJCICAgICAwICAgMCAgIDAgICAwICAgICAwICAgMCAgIDAgICAwICAgNDkgICAzMyAxNDg3ICAw ICAyIDk4DQogMCAwIDQgICAgNzY4TSAgODE5MkIgICAgIDAgICAwICAgMCAgIDAgICAgIDAgICAw ICAgMCAgIDAgICA0NyAgIDMzIDE5NzcgIDAgIDMgOTcNCiAwIDAgNCAgICA3NjhNICA4MTkyQiAg ICAgMCAgIDAgICAwICAgMCAgICAgMCAgIDAgICAwICAgMCAgIDQzICAgMzMgMjI1MSAgMCAgNCA5 Ng0KIDAgMCA0ICAgIDc2OE0gIDgxOTJCICAgICAwICAgMCAgIDAgICAwICAgICAwICAgMCAgIDAg ICAwICAgNDYgICAzMyAyMjk3ICAwICA0IDk2DQogMCAwIDQgICAgNzY4TSAgODE5MkIgICAgIDAg ICAwICAgMCAgIDAgICAgIDAgICAwICAgMCAgIDAgICA0MSAgIDMzIDIzODcgIDAgIDQgOTYNCiAw IDAgNCAgICA3NjhNICA4MTkyQiAgICAgMCAgIDAgICAwICAgMCAgICAgMCAgIDAgICAwICAgMCAg IDM4ICAgMzMgMjM5MSAgMCAgNSA5NQ0KIDAgMCA0ICAgIDc2OE0gIDgxOTJCICAgICAwICAgMCAg IDAgICAwICAgICAwICAgMCAgIDAgICAwICAgMzYgICAzMyAyNDI3ICAwICA2IDk0DQogMCAwIDQg ICAgNzY4TSAgODE5MkIgICAgIDAgICAwICAgMCAgIDAgICAgIDAgICAwICAgMCAgIDAgICAyOSAg IDMzIDI0MTYgIDAgIDUgOTUNCiAwIDAgNCAgICA3NjhNICA4MTkyQiAgICAgMCAgIDAgICAwICAg MCAgICAgMCAgIDAgICAwICAgMCAgIDM1ICAgMzMgMjQzOSAgMCAgNiA5NA0KIDAgMCA0ICAgIDc2 OE0gIDgxOTJCICAgICAwICAgMCAgIDAgICAwICAgICAwICAgMCAgIDAgICAwICAgMzMgICAzMyAy NDgyICAwICA2IDk0DQogMCAwIDQgICAgNzY4TSAgODE5MkIgICAgIDAgICAwICAgMCAgIDAgICAg IDAgICAwICAgMCAgIDAgICAyNyAgIDMzIDI0MzIgIDAgIDYgOTQNCiAwIDAgNCAgICA3NjhNICA4 MTkyQiAgICAgMCAgIDAgICAwICAgMCAgICAgMCAgIDAgICAwICAgMCAgIDMyICAgMzMgMjQ3NyAg MCAgNSA5NQ0KIDAgMCA0ICAgIDc2OE0gIDgxOTJCICAgICAwICAgMCAgIDAgICAwICAgICAwICAg MCAgIDAgICAwICAgMjUgICAzMyAyNDEyICAwICA1IDk1DQogMCAwIDQgICAgNzY4TSAgODE5MkIg ICAgIDAgICAwICAgMCAgIDAgICAgIDAgICAwICAgMCAgIDAgICAzMCAgIDMzIDI0MTMgIDAgIDYg OTQNCiAwIDAgNCAgICA3NjhNICA4MTkyQiAgICAgMCAgIDAgICAwICAgMCAgICAgMCAgIDAgICAw ICAgMCAgIDMwICAgMzMgMjQ4MSAgMCAgNyA5Mw0KIDAgMCA0ICAgIDc2OE0gIDgxOTJCICAgICAw ICAgMCAgIDAgICAwICAgICAwICAgMCAgIDAgICAwICAgNDQgICAzMyAyNTA3ICAwICA4IDkyDQog MCAwIDQgICAgNzY4TSAgODE5MkIgICAgIDAgICAwICAgMCAgIDAgICAgIDAgICAwICAgMCAgIDAg ICAyMiAgIDMzIDI0NTcgIDAgIDggOTINCiAwIDAgNCAgICA3NjhNICA4MTkyQiAgICAgMCAgIDAg ICAwICAgMCAgICAgMCAgIDAgICAwICAgMCAgIDI2ICAgMzIgMjQwOCAgMCAxMiA4OA0KIDAgMCA0 ICAgIDc2OE0gIDgxOTJCICAgICAwICAgMCAgIDAgICAwICAgICAwICAgMCAgIDAgICAwICAgMjQg ICAzMyAyNDA0ICAwIDEyIDg4DQogMCAwIDQgICAgNzY4TSAgODE5MkIgICAgIDAgICAwICAgMCAg IDAgICAgIDAgICAwICAgMCAgIDAgICAyMiAgIDMzIDIzOTYgIDAgMTIgODgNCiAwIDAgNCAgICA3 NjhNICA4MTkyQiAgICAgMCAgIDAgICAwICAgMCAgICAgMCAgIDAgICAwICAgMCAgIDI1ICAgMzMg MjQwMiAgMCAxMiA4OA0KIDAgMCA0ICAgIDc2OE0gIDgxOTJCICAgICAwICAgMCAgIDAgICAwICAg ICAwICAgMCAgIDAgICAwICAgMjUgICAzMyAyNDAyICAwIDEyIDg4DQogMCAwIDQgICAgNzY4TSAg ODE5MkIgICAgIDAgICAwICAgMCAgIDAgICAgIDAgICAwICAgMCAgIDAgICAyNiAgIDMzIDI0MTAg IDAgMTIgODgNCiAwIDAgNCAgICA3NjhNICA4MTkyQiAgICAgMCAgIDAgICAwICAgMCAgICAgMCAg IDAgICAwICAgMCAgIDE5ICAgMzMgMjM5NiAgMCAxMiA4OA0KIDAgMCA0ICAgIDc2OE0gIDgxOTJC ICAgICAwICAgMCAgIDAgICAwICAgICAwICAgMCAgIDAgICAwICAgMjAgICAzMyAyMzkyICAwIDEy IDg4DQogMCAwIDQgICAgNzY4TSAgODE5MkIgICAgIDAgICAwICAgMCAgIDAgICAgIDAgICAwICAg MCAgIDAgICAyNCAgIDMzIDI0MDEgIDAgMTIgODgNCiAwIDAgNCAgICA3NjhNICA4MTkyQiAgICAg MCAgIDAgICAwICAgMCAgICAgMCAgIDAgICAwICAgMCAgIDIwICAgMzMgMjQ1NiAgMCAxMiA4OA0K IDAgMCA0ICAgIDc2OE0gIDgxOTJCICAgICAwICAgMCAgIDAgICAwICAgICAwICAgMCAgIDAgICAw ICAgMjIgICAzMyAyNDExICAwIDEyIDg4DQogMCAwIDQgICAgNzY4TSAgODE5MkIgICAgIDAgICAw ICAgMCAgIDAgICAgIDAgICAwICAgMCAgIDAgICAxOSAgIDMzIDIzODcgIDAgMTIgODg= --001a1145eae834d0fc051ac04b07 Content-Type: text/plain; charset=US-ASCII; name="2015-07-07 - San1.txt" Content-Disposition: attachment; filename="2015-07-07 - San1.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_ic1uffuq2 DQpsejRfY3R4OiAgICAgICAgICAgICAgMTYzODQsICAgICAgMCwgICAgICAgMCwgICAgICA5NCw1 MDIzNzEyMCwgICAwLCAgIDANCnNhX2NhY2hlOiAgICAgICAgICAgICAgICA4MCwgICAgICAwLCAg ICAzODQzLCAgICAxMzA3LCAgICA1NDcxLCAgIDAsICAgMA0KZG5vZGVfdDogICAgICAgICAgICAg ICAgNzUyLCAgICAgIDAsICAgMzg2ODMsICAgICA1MjIsIDU5Mjc2NDgsICAgMCwgICAwDQpkbXVf YnVmX2ltcGxfdDogICAgICAgICAyMzIsICAgICAgMCwgNDY4OTA4OSwgIDk1MzMzMCwyNjQzODE4 NiwgICAwLCAgIDANCmFyY19idWZfaGRyX3Q6ICAgICAgICAgIDIxNiwgICAgICAwLCA5NjkyMDc1 LCAgICAgODM1LDE2NTUyNzkyLCAgIDAsICAgMA0KYXJjX2J1Zl90OiAgICAgICAgICAgICAgIDcy LCAgICAgIDAsIDQ4NDY5ODIsIDIwMjQyMjMsMjAwNjQ5NzIsICAgMCwgICAwDQp6aWxfbHdiX2Nh Y2hlOiAgICAgICAgICAxOTIsICAgICAgMCwgICAyMDUyOCwgICAgIDY1MiwgIDg1MzMzMywgICAw LCAgIDANCnpmc196bm9kZV9jYWNoZTogICAgICAgIDM2OCwgICAgICAwLCAgICAzODQzLCAgICAg MzM3LCAgICA1NDcxLCAgIDAsICAgMA0KTW91bnRwb2ludHM6ICAgICAgICAgICAgODE2LCAgICAg IDAsICAgICAgMjMsICAgICAxNDcsICAgICAgMjMsICAgMCwgICAwDQprc2lnaW5mbzogICAgICAg ICAgICAgICAxMTIsICAgICAgMCwgICAgIDM3NCwgICAgMzQwNiwgICAgMTg4NiwgICAwLCAgIDAN Cml0aW1lcjogICAgICAgICAgICAgICAgIDM1MiwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAg ICAgICAwLCAgIDAsICAgMA0KS05PVEU6ICAgICAgICAgICAgICAgICAgMTI4LCAgICAgIDAsICAg ICAgIDAsICAgIDMwMDcsICAgIDg3NDYsICAgMCwgICAwDQpzb2NrZXQ6ICAgICAgICAgICAgICAg ICA2OTYsIDMxNDQ0ODUsICAgICAgNjksICAgICAzMzYsICAgMjM3OTEsICAgMCwgICAwDQppcHE6 ICAgICAgICAgICAgICAgICAgICAgNTYsIDE5MTQ4NywgICAgICAgMCwgICAgICAgMCwgICAgICAg MCwgICAwLCAgIDANCnVkcF9pbnBjYjogICAgICAgICAgICAgIDM5MiwgMzE0NDQ5MCwgICAgICAx MSwgICAgIDc0OSwgICAgOTA1MCwgICAwLCAgIDANCnVkcGNiOiAgICAgICAgICAgICAgICAgICAx NiwgMzE0NDUyOCwgICAgICAxMSwgICAgNjI2NCwgICAgOTA1MCwgICAwLCAgIDANCnRjcF9pbnBj YjogICAgICAgICAgICAgIDM5MiwgMzE0NDQ5MCwgICAgICA0NSwgICAgIDY1NSwgICAgMzAxNiwg ICAwLCAgIDANCnRjcGNiOiAgICAgICAgICAgICAgICAgMTAyNCwgMzE0NDQ4OCwgICAgICA0NSwg ICAgIDI2NywgICAgMzAxNiwgICAwLCAgIDANCnRjcHR3OiAgICAgICAgICAgICAgICAgICA4OCwg IDI3ODEwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0Kc3luY2FjaGU6ICAg ICAgICAgICAgICAgMTYwLCAgMTUzNzUsICAgICAgIDAsICAgICA1MDAsICAgICAgODcsICAgMCwg ICAwDQpob3N0Y2FjaGU6ICAgICAgICAgICAgICAxMzYsICAxNTM3MCwgICAgICAgMCwgICAgICAg MCwgICAgICAgMCwgICAwLCAgIDANCnRjcHJlYXNzOiAgICAgICAgICAgICAgICA0MCwgMzgzMDAw LCAgICAgICAwLCAgICAgODAwLCAgICAgIDU3LCAgIDAsICAgMA0Kc2Fja2hvbGU6ICAgICAgICAg ICAgICAgIDMyLCAgICAgIDAsICAgICAgIDAsICAgIDE2MjUsICAgICAyMDgsICAgMCwgICAwDQpz Y3RwX2VwOiAgICAgICAgICAgICAgIDE0MDgsIDMxNDQ0ODYsICAgICAgIDAsICAgICAgIDAsICAg ICAgIDAsICAgMCwgICAwDQpzY3RwX2Fzb2M6ICAgICAgICAgICAgIDI0MTYsICA0MDAwMCwgICAg ICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnNjdHBfbGFkZHI6ICAgICAgICAgICAg ICA0OCwgIDgwMDEyLCAgICAgICAwLCAgICAxMDc5LCAgICAgICA0LCAgIDAsICAgMA0Kc2N0cF9y YWRkcjogICAgICAgICAgICAgNzI4LCAgODAwMDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAs ICAgMCwgICAwDQpzY3RwX2NodW5rOiAgICAgICAgICAgICAxMzYsIDQwMDAyNiwgICAgICAgMCwg ICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnNjdHBfcmVhZHE6ICAgICAgICAgICAgIDEwNCwg NDAwMDI2LCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0Kc2N0cF9zdHJlYW1f bXNnX291dDogICAgMTA0LCA0MDAwMjYsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwg ICAwDQpzY3RwX2FzY29uZjogICAgICAgICAgICAgNDAsIDQwMDAwMCwgICAgICAgMCwgICAgICAg MCwgICAgICAgMCwgICAwLCAgIDANCnNjdHBfYXNjb25mX2FjazogICAgICAgICA0OCwgNDAwMDYw LCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KdWRwbGl0ZV9pbnBjYjogICAg ICAgICAgMzkyLCAzMTQ0NDkwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0K cmlwY2I6ICAgICAgICAgICAgICAgICAgMzkyLCAzMTQ0NDkwLCAgICAgICAwLCAgICAgICAwLCAg ICAgICAwLCAgIDAsICAgMA0KdW5wY2I6ICAgICAgICAgICAgICAgICAgMjQwLCAzMTQ0NDk2LCAg ICAgIDEzLCAgICAxNDU5LCAgIDExNzIwLCAgIDAsICAgMA0KcnRlbnRyeTogICAgICAgICAgICAg ICAgMjAwLCAgICAgIDAsICAgICAgMTUsICAgICA0ODUsICAgICAgMTUsICAgMCwgICAwDQpzZWxm ZDogICAgICAgICAgICAgICAgICAgNTYsICAgICAgMCwgICAgIDQ2OCwgICAgNTYzOCwgMjU4Njg0 NCwgICAwLCAgIDANClNXQVBNRVRBOiAgICAgICAgICAgICAgIDI4OCwgMTIyNTQzMDcsICAgICAg IDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQppY2xfY29ubjogICAgICAgICAgICAgICAx NTIsICAgICAgMCwgICAgICAyMCwgICAgIDY1NiwgICAgICAyMCwgICAwLCAgIDANCmljbF9wZHU6 ICAgICAgICAgICAgICAgICA4MCwgICAgICAwLCAgICAgIDc5LCAgICA3MzIxLDE4OTA3NTIzLCAg IDAsICAgMA0KaXNjc2lfb3V0c3RhbmRpbmc6ICAgICAgIDQwLCAgICAgIDAsICAgICAgIDAsICAg ICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpjZmlfbHVuOiAgICAgICAgICAgICAgICAzNDQsICAg ICAgMCwgICAgICAyNCwgICAgIDI5NSwgICAgICAyNCwgICAwLCAgIDANCmNmaV9tZXRhdGFzazog ICAgICAgICAgIDMyMCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAg MA0KY2Zpc2NzaV9kYXRhX3dhaXQ6ICAgICAgIDU2LCAgICAgIDAsICAgICAgIDAsICAgIDcxMDAs IDY2NTY0MDQsICAgMCwgICAwDQpiZWlvOiAgICAgICAgICAgICAgICAgICAzNjAsICAgICAgMCwg ICAgICA0MCwgICAgIDg0MCwgODI0NTE1NCwgICAwLCAgIDANCmNibGswOiAgICAgICAgICAgICAg IDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDEwLCAgICAgIDI5LCAgIDAsICAgMA0KY2Js azE6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgMzQsICAgIDEw MjgsICAgMCwgICAwDQpjYmxrMjogICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAg MCwgICAgICAxNCwgICAgICAyOSwgICAwLCAgIDANCmNibGszOiAgICAgICAgICAgICAgIDEzMTA3 MiwgICAgICAwLCAgICAgICAwLCAgICAgIDEwLCAgICAgIDI5LCAgIDAsICAgMA0KY2JsazQ6ICAg ICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgIDcsICAgICAgMjksICAg MCwgICAwDQpjYmxrNTogICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAg ICAyMCwgICAgICA5NCwgICAwLCAgIDANCmNibGs2OiAgICAgICAgICAgICAgIDEzMTA3MiwgICAg ICAwLCAgICAgICAxLCAgICAgIDc5LCAyMjE4MTY3LCAgIDAsICAgMA0KY2Jsazc6ICAgICAgICAg ICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgMTAsICAgICAgMjMsICAgMCwgICAw DQpjYmxrODogICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICA1Miwg ICAyOTU0NywgICAwLCAgIDANCmNibGs5OiAgICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAg ICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KY2JsazEwOiAgICAgICAgICAgICAg MTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpjYmxr MTE6ICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICA3OSwgICAyMDMy NCwgICAwLCAgIDANCmNibGsxMjogICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAw LCAgICAgICA5LCAgICAgIDk0LCAgIDAsICAgMA0KY2JsazEzOiAgICAgICAgICAgICAgMTMxMDcy LCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpjYmxrMTQ6ICAg ICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAw LCAgIDANCmNibGsxNTogICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgIDExLCAgICAg IDI0LCAgMzA2MTY3LCAgIDAsICAgMA0KY2JsazE2OiAgICAgICAgICAgICAgMTMxMDcyLCAgICAg IDAsICAgICAgMTAsICAgICAgMjUsICA2MzU1MDUsICAgMCwgICAwDQpjYmxrMTc6ICAgICAgICAg ICAgICAxMzEwNzIsICAgICAgMCwgICAgICAxMSwgICAgICAyNSwgMjE0MDc4NSwgICAwLCAgIDAN CmNibGsxODogICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDEzLCAg ICAgIDk0LCAgIDAsICAgMA0KY2JsazE5OiAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAg ICAgIDAsICAgICAgMTMsICAgICAgOTQsICAgMCwgICAwDQpjYmxrMjA6ICAgICAgICAgICAgICAx MzEwNzIsICAgICAgMCwgICAgICAgMiwgICAgICA3NiwgMTE3MzcwNiwgICAwLCAgIDANCmNibGsy MTogICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAxLCAgICAgIDU4LCAxMjc1Mjg1 LCAgIDAsICAgMA0KY2JsazIyOiAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDIs ICAgICAgMzUsICAgMzUxODIsICAgMCwgICAwDQpjYmxrMjM6ICAgICAgICAgICAgICAxMzEwNzIs ICAgICAgMCwgICAgICAgMiwgICAgICAxOSwgIDQxODkwNSwgICAwLCAgIDANCg0KIHByb2NzICAg ICAgbWVtb3J5ICAgICAgcGFnZSAgICAgICAgICAgICAgICAgICAgZGlza3MgICAgIGZhdWx0cyAg ICAgICAgIGNwdQ0KIHIgYiB3ICAgICBhdm0gICAgZnJlICAgZmx0ICByZSAgcGkgIHBvICAgIGZy ICBzciBkYTAgZGExICAgaW4gICBzeSAgIGNzIHVzIHN5IGlkDQogMiAwIDAgICAyMDEwTSAgMjI0 OU0gIDIyMDUgICAwICAgMCAgIDAgIDI5MDEgICAyICAgMCAgIDAgMTk4MiA0NjMyNiAxMDQ2OCAg MCAgMyA5Nw0KNDA2MDIvMjMxNDgvNjM3NTAgbWJ1ZnMgaW4gdXNlIChjdXJyZW50L2NhY2hlL3Rv dGFsKQ0KMzkwNDMvMTEzNTMvNTAzOTYvNjEyNzE0OCBtYnVmIGNsdXN0ZXJzIGluIHVzZSAoY3Vy cmVudC9jYWNoZS90b3RhbC9tYXgpDQozOTAzMC83NjYzIG1idWYrY2x1c3RlcnMgb3V0IG9mIHBh Y2tldCBzZWNvbmRhcnkgem9uZSBpbiB1c2UgKGN1cnJlbnQvY2FjaGUpDQowLzQ4NS80ODUvMzA2 MzU3NCA0ayAocGFnZSBzaXplKSBqdW1ibyBjbHVzdGVycyBpbiB1c2UgKGN1cnJlbnQvY2FjaGUv dG90YWwvbWF4KQ0KMC8wLzAvOTA3NzI1IDlrIGp1bWJvIGNsdXN0ZXJzIGluIHVzZSAoY3VycmVu dC9jYWNoZS90b3RhbC9tYXgpDQowLzAvMC81MTA1OTUgMTZrIGp1bWJvIGNsdXN0ZXJzIGluIHVz ZSAoY3VycmVudC9jYWNoZS90b3RhbC9tYXgpDQo4ODIzNksvMzA0MzNLLzExODY2OUsgYnl0ZXMg YWxsb2NhdGVkIHRvIG5ldHdvcmsgKGN1cnJlbnQvY2FjaGUvdG90YWwpDQowLzAvMCByZXF1ZXN0 cyBmb3IgbWJ1ZnMgZGVuaWVkIChtYnVmcy9jbHVzdGVycy9tYnVmK2NsdXN0ZXJzKQ0KMC8wLzAg cmVxdWVzdHMgZm9yIG1idWZzIGRlbGF5ZWQgKG1idWZzL2NsdXN0ZXJzL21idWYrY2x1c3RlcnMp DQowLzAvMCByZXF1ZXN0cyBmb3IganVtYm8gY2x1c3RlcnMgZGVsYXllZCAoNGsvOWsvMTZrKQ0K MC8wLzAgcmVxdWVzdHMgZm9yIGp1bWJvIGNsdXN0ZXJzIGRlbmllZCAoNGsvOWsvMTZrKQ0KMCBy ZXF1ZXN0cyBmb3Igc2ZidWZzIGRlbmllZA0KMCByZXF1ZXN0cyBmb3Igc2ZidWZzIGRlbGF5ZWQN CjAgcmVxdWVzdHMgZm9yIEkvTyBpbml0aWF0ZWQgYnkgc2VuZGZpbGUNCklURU0gICAgICAgICAg ICAgICAgICAgU0laRSAgTElNSVQgICAgIFVTRUQgICAgIEZSRUUgICAgICBSRVEgRkFJTCBTTEVF UA0KDQpVTUEgS2VnczogICAgICAgICAgICAgICAzODQsICAgICAgMCwgICAgIDIzMiwgICAgICAg OCwgICAgIDIzMiwgICAwLCAgIDANClVNQSBab25lczogICAgICAgICAgICAgMzcxMiwgICAgICAw LCAgICAgMjMyLCAgICAgICAwLCAgICAgMjMyLCAgIDAsICAgMA0KVU1BIFNsYWJzOiAgICAgICAg ICAgICAgIDgwLCAgICAgIDAsIDU0OTk0NzQsIDI1NDk2MjYsMTc0MDcyOTgsICAgMCwgICAwDQpV TUEgUkNudFNsYWJzOiAgICAgICAgICAgODgsICAgICAgMCwgICAyNTY4MywgICAgNDg3MiwgICAz NTc5MywgICAwLCAgIDANClVNQSBIYXNoOiAgICAgICAgICAgICAgIDI1NiwgICAgICAwLCAgICAg IDMxLCAgICAgIDQ0LCAgICAgMTA1LCAgIDAsICAgMA0KNCBCdWNrZXQ6ICAgICAgICAgICAgICAg IDMyLCAgICAgIDAsICAgIDM0ODMsICAgMTQxNDIsIDMyNDcxMTQsICAgMCwgICAwDQo2IEJ1Y2tl dDogICAgICAgICAgICAgICAgNDgsICAgICAgMCwgICAgIDQ3MSwgICAxMjQ3NywgMTE4MTU1MSwg ICAwLCAgIDANCjggQnVja2V0OiAgICAgICAgICAgICAgICA2NCwgICAgICAwLCAgICAgMjI3LCAg IDEwNjg1LCAgNTUwMTU3LCAgMTEsICAgMA0KMTIgQnVja2V0OiAgICAgICAgICAgICAgIDk2LCAg ICAgIDAsICAgICAyNjQsICAgIDY3NDcsICA3NzkwMDAsICAgMCwgICAwDQoxNiBCdWNrZXQ6ICAg ICAgICAgICAgICAxMjgsICAgICAgMCwgICAgIDY0NywgICAgNzI1OCwgIDczNzU3MywgICAwLCAg IDANCjMyIEJ1Y2tldDogICAgICAgICAgICAgIDI1NiwgICAgICAwLCAgICAgNjA1LCAgICA1NjUw LCAxNDM4Nzc5LCAgNTEsICAgMA0KNjQgQnVja2V0OiAgICAgICAgICAgICAgNTEyLCAgICAgIDAs ICAgICA0NTQsICAgIDUxNTQsIDEzMzg0NTcsIDEwMywgICAwDQoxMjggQnVja2V0OiAgICAgICAg ICAgIDEwMjQsICAgICAgMCwgICAgNDA5OSwgICAgIDE3NywgMTE5OTQzMCwgICAwLCAgIDANCjI1 NiBCdWNrZXQ6ICAgICAgICAgICAgMjA0OCwgICAgICAwLCAgICAyNzc4LCAgICAzMDg4LDQxMDc3 NDEyLCAgNjQsICAgMA0Kdm1lbSBidGFnOiAgICAgICAgICAgICAgIDU2LCAgICAgIDAsIDEyODM5 ODYsICAgMjE4NDYsIDIwMzM2NjMsOTIxNywgICAwDQpWTSBPQkpFQ1Q6ICAgICAgICAgICAgICAy NTYsICAgICAgMCwgICAgNzExMywgICAgMTk2MiwgMjk2MjkxMywgICAwLCAgIDANClJBRElYIE5P REU6ICAgICAgICAgICAgIDE0NCwgICAgICAwLCAxMjA1MTk0LCAgIDMxMjcxLDE0MzkyNDUyLCAg IDAsICAgMA0KTUFQOiAgICAgICAgICAgICAgICAgICAgMjQwLCAgICAgIDAsICAgICAgIDMsICAg ICAgNjEsICAgICAgIDMsICAgMCwgICAwDQpLTUFQIEVOVFJZOiAgICAgICAgICAgICAxMjgsICAg ICAgMCwgICAgICAgOSwgICAgIDM5NCwgICAgICAgOSwgICAwLCAgIDANCk1BUCBFTlRSWTogICAg ICAgICAgICAgIDEyOCwgICAgICAwLCAgICAyNzg0LCAgICA2MTc1LCA4NzIzNzIzLCAgIDAsICAg MA0KVk1TUEFDRTogICAgICAgICAgICAgICAgNDQ4LCAgICAgIDAsICAgICAgNzAsICAgICA4MzAs ICAxOTU2MDksICAgMCwgICAwDQpmYWtlcGc6ICAgICAgICAgICAgICAgICAxMDQsICAgICAgMCwg ICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCm10X3pvbmU6ICAgICAgICAgICAg ICAgNDExMiwgICAgICAwLCAgICAgMzY5LCAgICAgICAwLCAgICAgMzY5LCAgIDAsICAgMA0KMTY6 ICAgICAgICAgICAgICAgICAgICAgIDE2LCAgICAgIDAsICAgMzAyMzYsICAgIDU5MDgsNTQyNDU0 OTIsICAgMCwgICAwDQozMjogICAgICAgICAgICAgICAgICAgICAgMzIsICAgICAgMCwgOTg2Nzk5 OCwgMTA1Mjc1MiwxMjg1NTg4MDEsICAgMCwgICAwDQo2NDogICAgICAgICAgICAgICAgICAgICAg NjQsICAgICAgMCwgMzI3MjU0OSwgMjI2NTQxNSw2MjYyMDY3OTIsICAgMCwgICAwDQoxMjg6ICAg ICAgICAgICAgICAgICAgICAxMjgsICAgICAgMCwgIDk1MzU1NywgMzQ1MjY5MCwzNDc0MzY0NDAs ICAgMCwgICAwDQoyNTY6ICAgICAgICAgICAgICAgICAgICAyNTYsICAgICAgMCwgMTMzNTI1MCwg ICAgMzQ3MCwyMjM3NzEyMDQsICAgMCwgICAwDQo1MTI6ICAgICAgICAgICAgICAgICAgICA1MTIs ICAgICAgMCwgICAgMTQ3NiwgIDY2NTcwMCw3Mjk3OTQ5OCwgICAwLCAgIDANCjEwMjQ6ICAgICAg ICAgICAgICAgICAgMTAyNCwgICAgICAwLCAgIDI2ODY1LCAgICAgNjYzLDYwODk3NDEzLCAgIDAs ICAgMA0KMjA0ODogICAgICAgICAgICAgICAgICAyMDQ4LCAgICAgIDAsICAxMDU0MzcsICAgIDE4 NTEsNTQ5MDgyMzksICAgMCwgICAwDQo0MDk2OiAgICAgICAgICAgICAgICAgIDQwOTYsICAgICAg MCwgICAgNDI0NCwgICAgIDExOSwgMjYyMzY3NCwgICAwLCAgIDANCjgxOTI6ICAgICAgICAgICAg ICAgICAgODE5MiwgICAgICAwLCAgICAgIDM4LCAgICAgIDc5LCAxOTA5NTY1LCAgIDAsICAgMA0K MTYzODQ6ICAgICAgICAgICAgICAgIDE2Mzg0LCAgICAgIDAsICAgICAgNDYsICAgICAgNDIsICAg IDUxMTgsICAgMCwgICAwDQozMjc2ODogICAgICAgICAgICAgICAgMzI3NjgsICAgICAgMCwgICAg ICA0NywgICAgICAyNywgICAgNjQxMSwgICAwLCAgIDANCjY1NTM2OiAgICAgICAgICAgICAgICA2 NTUzNiwgICAgICAwLCAgICAgMTA2LCAgICAgIDMyLCAgMTc1ODMyLCAgIDAsICAgMA0KNjQgcGNw dTogICAgICAgICAgICAgICAgICA4LCAgICAgIDAsICAgIDIxMTAsICAgIDE5ODYsICAgIDIxMTAs ICAgMCwgICAwDQpTTEVFUFFVRVVFOiAgICAgICAgICAgICAgODAsICAgICAgMCwgICAgMjAwOCwg ICAgMTg2NywgICAgMjAwOCwgICAwLCAgIDANCkZpbGVzOiAgICAgICAgICAgICAgICAgICA4MCwg ICAgICAwLCAgICAgMjQ0LCAgICAzMjU2LCAxMzM1NzU4LCAgIDAsICAgMA0KVFVSTlNUSUxFOiAg ICAgICAgICAgICAgMTM2LCAgICAgIDAsICAgIDIwMDgsICAgICA4MTIsICAgIDIwMDgsICAgMCwg ICAwDQpybF9lbnRyeTogICAgICAgICAgICAgICAgNDAsICAgICAgMCwgICAgIDE2MywgICAgNTkz NywgICAgIDE2MywgICAwLCAgIDANCnVtdHggcGk6ICAgICAgICAgICAgICAgICA5NiwgICAgICAw LCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KTUFDIGxhYmVsczogICAgICAg ICAgICAgIDQwLCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpQ Uk9DOiAgICAgICAgICAgICAgICAgIDEyMTYsICAgICAgMCwgICAgICA5MSwgICAgIDMxNCwgIDE5 NTYzMCwgICAwLCAgIDANClRIUkVBRDogICAgICAgICAgICAgICAgMTE2OCwgICAgICAwLCAgICAx ODIzLCAgICAgMTg0LCAgICA0MDQwLCAgIDAsICAgMA0KY3B1c2V0OiAgICAgICAgICAgICAgICAg IDcyLCAgICAgIDAsICAgIDE0NjEsICAgIDIxNjksICAgIDM2OTQsICAgMCwgICAwDQphdWRpdF9y ZWNvcmQ6ICAgICAgICAgIDEyNDgsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwg ICAwLCAgIDANCm1idWZfcGFja2V0OiAgICAgICAgICAgIDI1NiwgMzkyMTM3NTAsICAgMzkwMzAs ICAgIDc2NjMsMTQ3MTcxODYwLCAgIDAsICAgMA0KbWJ1ZjogICAgICAgICAgICAgICAgICAgMjU2 LCAzOTIxMzc1MCwgICAgMTU3MiwgICAxNTQ4NSwxOTM1NzYxMjgsICAgMCwgICAwDQptYnVmX2Ns dXN0ZXI6ICAgICAgICAgIDIwNDgsIDYxMjcxNDgsICAgNDY3MDYsICAgIDM2OTAsIDI4NTYwNjIs ICAgMCwgICAwDQptYnVmX2p1bWJvX3BhZ2U6ICAgICAgIDQwOTYsIDMwNjM1NzQsICAgICAgIDAs ICAgICA0ODUsMTIyMDEzMTcsICAgMCwgICAwDQptYnVmX2p1bWJvXzlrOiAgICAgICAgIDkyMTYs IDkwNzcyNSwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCm1idWZfanVtYm9f MTZrOiAgICAgICAxNjM4NCwgNTEwNTk1LCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAs ICAgMA0KbWJ1Zl9leHRfcmVmY250OiAgICAgICAgICA0LCAgICAgIDAsICAgICAgIDAsICAgICAg IDAsICAgICAgIDAsICAgMCwgICAwDQpnX2JpbzogICAgICAgICAgICAgICAgICAyNDgsICAgICAg MCwgICAgICAxMywgICAgNzg0MywyMDEzNzY4MTMsICAgMCwgICAwDQp0dHlpbnE6ICAgICAgICAg ICAgICAgICAxNjAsICAgICAgMCwgICAgIDI0MCwgICAgMTA2MCwgICAgIDgyNSwgICAwLCAgIDAN CnR0eW91dHE6ICAgICAgICAgICAgICAgIDI1NiwgICAgICAwLCAgICAgMTI2LCAgICAxMTY0LCAg ICAgNDM4LCAgIDAsICAgMA0KRE1BUl9NQVBfRU5UUlk6ICAgICAgICAgMTIwLCAgICAgIDAsICAg ICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQphdGFfcmVxdWVzdDogICAgICAgICAg ICAzMzYsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnZ0bmV0 X3R4X2hkcjogICAgICAgICAgICAyNCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAw LCAgIDAsICAgMA0KRlBVX3NhdmVfYXJlYTogICAgICAgICAgNTEyLCAgICAgIDAsICAgICAgIDAs ICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQp0YXNrcV96b25lOiAgICAgICAgICAgICAgNDgs ICAgICAgMCwgICAgICAgMCwgICAgNzYzNiwgIDUyNzIwMywgICAwLCAgIDANClZOT0RFOiAgICAg ICAgICAgICAgICAgIDQ3MiwgICAgICAwLCAgICAzOTQ1LCAgICAgMzUxLCAgICA1NjE4LCAgIDAs ICAgMA0KVk5PREVQT0xMOiAgICAgICAgICAgICAgMTEyLCAgICAgIDAsICAgICAgIDAsICAgICAg IDAsICAgICAgIDAsICAgMCwgICAwDQpCVUYgVFJJRTogICAgICAgICAgICAgICAxNDQsICAgICAg MCwgICAgICAgMCwgIDEwNTk0OCwgICAgICAgMCwgICAwLCAgIDANClMgVkZTIENhY2hlOiAgICAg ICAgICAgIDEwOCwgICAgICAwLCAgICAgODYxLCAgICAyOTE5LCAgIDQzODYwLCAgIDAsICAgMA0K U1RTIFZGUyBDYWNoZTogICAgICAgICAgMTQ4LCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAg ICAgIDAsICAgMCwgICAwDQpMIFZGUyBDYWNoZTogICAgICAgICAgICAzMjgsICAgICAgMCwgICAg ICAgOCwgICAgIDE3MiwgICAgICAgOSwgICAwLCAgIDANCkxUUyBWRlMgQ2FjaGU6ICAgICAgICAg IDM2OCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KTkFNRUk6 ICAgICAgICAgICAgICAgICAxMDI0LCAgICAgIDAsICAgICAgMTIsICAgICAzOTIsIDQ0NzM0MDcs ICAgMCwgICAwDQpOQ0xOT0RFOiAgICAgICAgICAgICAgICA1MjgsICAgICAgMCwgICAgICAgMCwg ICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCkRJUkhBU0g6ICAgICAgICAgICAgICAgMTAyNCwg ICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KcGlwZTogICAgICAg ICAgICAgICAgICAgNzQ0LCAgICAgIDAsICAgICAgNTAsICAgICAzOTUsICAgMTc2NjcsICAgMCwg ICAwDQpwcm9jZGVzYzogICAgICAgICAgICAgICAxMjgsICAgICAgMCwgICAgICAgMCwgICAgICAg MCwgICAgICAgMCwgICAwLCAgIDANCnJhbmdlX3NlZ19jYWNoZTogICAgICAgICA2NCwgICAgICAw LDEyMzc4MjkwLCA0NTM5NDY0LDkyODk4MTkzNTMsICAgMCwgICAwDQp6aW9fY2FjaGU6ICAgICAg ICAgICAgICA5MjAsICAgICAgMCwgMTUxNDk3NiwgICAgNTEwNSwxMDQwNDI1Nzk1LCAgIDAsICAg MA0KemlvX2xpbmtfY2FjaGU6ICAgICAgICAgIDQ4LCAgICAgIDAsIDE1MTg3NTMsICAgIDQ5NjEs MTIwODg2MjIzOCwgICAwLCAgIDANCnppb19idWZfNTEyOiAgICAgICAgICAgIDUxMiwgICAgICAw LCAgIDEyODM5LCAgICAgOTY5LCAxMDY4MzkyLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzUxMjog ICAgICAgNTEyLCAgICAgIDAsICAgICAgMjMsICAgICA3MjksICAzMzQ0ODksICAgMCwgICAwDQp6 aW9fYnVmXzEwMjQ6ICAgICAgICAgIDEwMjQsICAgICAgMCwgICAgICAyOCwgICAgIDQ2NCwgIDM2 MTgxMiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8xMDI0OiAgICAgMTAyNCwgICAgICAwLCAgICAg NDk5LCAgICAgNDEzLCAgICA2MjM5LCAgIDAsICAgMA0KemlvX2J1Zl8xNTM2OiAgICAgICAgICAx NTM2LCAgICAgIDAsICAgICAgMjYsICAgICA0NDAsICAzMDAxMzksICAgMCwgICAwDQp6aW9fZGF0 YV9idWZfMTUzNjogICAgIDE1MzYsICAgICAgMCwgICAgIDIwMiwgICAgIDM3MCwgIDExOTY5Mywg ICAwLCAgIDANCnppb19idWZfMjA0ODogICAgICAgICAgMjA0OCwgICAgICAwLCAgICAgNzE0LCAg ICAgMzQyLCAyNDczNzk1LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzIwNDg6ICAgICAyMDQ4LCAg ICAgIDAsICAgICAgMzEsICAgICA1ODEsICA0NjIzNDMsICAgMCwgICAwDQp6aW9fYnVmXzI1NjA6 ICAgICAgICAgIDI1NjAsICAgICAgMCwgICAgICAgNSwgICAgIDEyOCwgMjQ4ODcwNCwgICAwLCAg IDANCnppb19kYXRhX2J1Zl8yNTYwOiAgICAgMjU2MCwgICAgICAwLCAgICAgIDUwLCAgICAgMzMx LCAgNDY4NDk0LCAgIDAsICAgMA0KemlvX2J1Zl8zMDcyOiAgICAgICAgICAzMDcyLCAgICAgIDAs ICAgICAgNTgsICAgICAyMTcsICA1NzQ1OTYsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMzA3Mjog ICAgIDMwNzIsICAgICAgMCwgICAgICAgNCwgICAgIDEzOCwgIDE0MTc2MSwgICAwLCAgIDANCnpp b19idWZfMzU4NDogICAgICAgICAgMzU4NCwgICAgICAwLCAgICAgNDA1LCAgICAgIDg1LCAgMTgz ODc0LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzM1ODQ6ICAgICAzNTg0LCAgICAgIDAsICAgICAg IDcsICAgICAgMzUsICAgIDYxNjEsICAgMCwgICAwDQp6aW9fYnVmXzQwOTY6ICAgICAgICAgIDQw OTYsICAgICAgMCwgMzExNDM5NSwgICAgNDU0OSwxNTc1OTgyMjUsICAgMCwgICAwDQp6aW9fZGF0 YV9idWZfNDA5NjogICAgIDQwOTYsICAgICAgMCwgIDk4ODcwMiwgICAgMTI4NSwxMDU4OTgyOSwg ICAwLCAgIDANCnppb19idWZfNTEyMDogICAgICAgICAgNTEyMCwgICAgICAwLCAgICAgMTk0LCAg ICAgIDI0LCAgMTQ3MTU2LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzUxMjA6ICAgICA1MTIwLCAg ICAgIDAsICAgICAgIDgsICAgICAgNjcsICAgMTA5OTIsICAgMCwgICAwDQp6aW9fYnVmXzYxNDQ6 ICAgICAgICAgIDYxNDQsICAgICAgMCwgICAgICAgMCwgICAgICAyOCwgIDIwMTAxOCwgICAwLCAg IDANCnppb19kYXRhX2J1Zl82MTQ0OiAgICAgNjE0NCwgICAgICAwLCAgICAgICA1LCAgICAgIDM2 LCAgICA2OTY5LCAgIDAsICAgMA0KemlvX2J1Zl83MTY4OiAgICAgICAgICA3MTY4LCAgICAgIDAs ICAgICAgIDEsICAgICAgMjIsICAxMjIxMjcsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNzE2ODog ICAgIDcxNjgsICAgICAgMCwgICAgICAgMCwgICAgICAyOSwgICAgNTk0MywgICAwLCAgIDANCnpp b19idWZfODE5MjogICAgICAgICAgODE5MiwgICAgICAwLCAgICAgIDMzLCAgICAgNDI1LDE0NzI5 ODA0LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzgxOTI6ICAgICA4MTkyLCAgICAgIDAsICAgIDEy MDQsICAgICAgMjEsICAgMTYxMzAsICAgMCwgICAwDQp6aW9fYnVmXzEwMjQwOiAgICAgICAgMTAy NDAsICAgICAgMCwgICAgICAgMiwgICAgICAyNSwgIDE0ODI4MywgICAwLCAgIDANCnppb19kYXRh X2J1Zl8xMDI0MDogICAxMDI0MCwgICAgICAwLCAgICAgICAzLCAgICAgIDM4LCAgICA4MDgyLCAg IDAsICAgMA0KemlvX2J1Zl8xMjI4ODogICAgICAgIDEyMjg4LCAgICAgIDAsICAgICAgIDUsICAg ICAxMzksIDQ0NTMyMTcsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTIyODg6ICAgMTIyODgsICAg ICAgMCwgICAgICAgMywgICAgICAzOSwgICAgNzA2OSwgICAwLCAgIDANCnppb19idWZfMTQzMzY6 ICAgICAgICAxNDMzNiwgICAgICAwLCAgICAgICAxLCAgICAgIDIzLCAgIDk0Mzk0LCAgIDAsICAg MA0KemlvX2RhdGFfYnVmXzE0MzM2OiAgIDE0MzM2LCAgICAgIDAsICAgICAgIDUsICAgICAgMzcs ICAgIDY5ODcsICAgMCwgICAwDQp6aW9fYnVmXzE2Mzg0OiAgICAgICAgMTYzODQsICAgICAgMCwg IDIxMjE2NywgICAgIDg5OSwxOTI0OTE5MCwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8xNjM4NDog ICAxNjM4NCwgICAgICAwLCAgICAgNzQ0LCAgICAgIDI3LCAgIDkzNjAxLCAgIDAsICAgMA0Kemlv X2J1Zl8yMDQ4MDogICAgICAgIDIwNDgwLCAgICAgIDAsICAgICAgIDUsICAgICAgMzMsIDExNTM5 ODMsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMjA0ODA6ICAgMjA0ODAsICAgICAgMCwgICAgICAg NywgICAgICAzNywgICAxODE5MywgICAwLCAgIDANCnppb19idWZfMjQ1NzY6ICAgICAgICAyNDU3 NiwgICAgICAwLCAgICAgICA3LCAgICAgIDM5LCAxOTM4NTA3LCAgIDAsICAgMA0KemlvX2RhdGFf YnVmXzI0NTc2OiAgIDI0NTc2LCAgICAgIDAsICAgICAgIDIsICAgICAgNDIsICAgMjU0MjgsICAg MCwgICAwDQp6aW9fYnVmXzI4NjcyOiAgICAgICAgMjg2NzIsICAgICAgMCwgICAgICAgNiwgICAg ICAzNiwgMTA4ODU5NiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8yODY3MjogICAyODY3MiwgICAg ICAwLCAgICAgICAwLCAgICAgIDQxLCAgIDMwMzI1LCAgIDAsICAgMA0KemlvX2J1Zl8zMjc2ODog ICAgICAgIDMyNzY4LCAgICAgIDAsICAgICAgIDYsICAgICAgMzQsIDE2ODI5NDQsICAgMCwgICAw DQp6aW9fZGF0YV9idWZfMzI3Njg6ICAgMzI3NjgsICAgICAgMCwgICAgMTU1NSwgICAgICAzMSwg ICA0MTQyMSwgICAwLCAgIDANCnppb19idWZfMzY4NjQ6ICAgICAgICAzNjg2NCwgICAgICAwLCAg ICAgICAzLCAgICAgIDY5LCAgNzA5OTk5LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzM2ODY0OiAg IDM2ODY0LCAgICAgIDAsICAgICAgIDIsICAgICAgNDMsICAgMjE0NTksICAgMCwgICAwDQp6aW9f YnVmXzQwOTYwOiAgICAgICAgNDA5NjAsICAgICAgMCwgICAgICAgNywgICAgICAyNywgIDgzODE5 OCwgICAwLCAgIDANCnppb19kYXRhX2J1Zl80MDk2MDogICA0MDk2MCwgICAgICAwLCAgICAgICA1 LCAgICAgIDQyLCAgIDEzMjIwLCAgIDAsICAgMA0KemlvX2J1Zl80NTA1NjogICAgICAgIDQ1MDU2 LCAgICAgIDAsICAgICAgIDEsICAgICAgMzMsICA0OTc2NjksICAgMCwgICAwDQp6aW9fZGF0YV9i dWZfNDUwNTY6ICAgNDUwNTYsICAgICAgMCwgICAgICAgMiwgICAgICAzNiwgICAgOTI2MiwgICAw LCAgIDANCnppb19idWZfNDkxNTI6ICAgICAgICA0OTE1MiwgICAgICAwLCAgICAgICAyLCAgICAg IDM2LCAgOTAwNjQ2LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzQ5MTUyOiAgIDQ5MTUyLCAgICAg IDAsICAgICAgIDAsICAgICAgNDEsICAgIDczNTcsICAgMCwgICAwDQp6aW9fYnVmXzUzMjQ4OiAg ICAgICAgNTMyNDgsICAgICAgMCwgICAgICAgMCwgICAgICAzMywgIDQwNDgwMywgICAwLCAgIDAN Cnppb19kYXRhX2J1Zl81MzI0ODogICA1MzI0OCwgICAgICAwLCAgICAgICAwLCAgICAgIDM3LCAg ICA1NTkwLCAgIDAsICAgMA0KemlvX2J1Zl81NzM0NDogICAgICAgIDU3MzQ0LCAgICAgIDAsICAg ICAgIDIsICAgICAgMjksICA2MjE3NzgsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNTczNDQ6ICAg NTczNDQsICAgICAgMCwgICAgICAgMiwgICAgICAzNywgICAgNDk0MywgICAwLCAgIDANCnppb19i dWZfNjE0NDA6ICAgICAgICA2MTQ0MCwgICAgICAwLCAgICAgICAxLCAgICAgIDMwLCAgMTU0ODUx LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzYxNDQwOiAgIDYxNDQwLCAgICAgIDAsICAgICAgIDAs ICAgICAgMjMsICAgICAgNjIsICAgMCwgICAwDQp6aW9fYnVmXzY1NTM2OiAgICAgICAgNjU1MzYs ICAgICAgMCwgICAgICAgMywgICAgIDIyMywgNjA3ODU1NCwgICAwLCAgIDANCnppb19kYXRhX2J1 Zl82NTUzNjogICA2NTUzNiwgICAgICAwLCAxMDA1MDQzLCAgICAgNjM5LCA3MTIxNzI0LCAgIDAs ICAgMA0KemlvX2J1Zl82OTYzMjogICAgICAgIDY5NjMyLCAgICAgIDAsICAgICAgIDEsICAgICAg MzAsICAxNDE1MzEsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNjk2MzI6ICAgNjk2MzIsICAgICAg MCwgICAgICAgMSwgICAgICAyNSwgICAgICA2OCwgICAwLCAgIDANCnppb19idWZfNzM3Mjg6ICAg ICAgICA3MzcyOCwgICAgICAwLCAgICAgICAyLCAgICAgIDMwLCAgMjczMzc0LCAgIDAsICAgMA0K emlvX2RhdGFfYnVmXzczNzI4OiAgIDczNzI4LCAgICAgIDAsICAgICAgIDAsICAgICAgMjYsICAg ICAgNjIsICAgMCwgICAwDQp6aW9fYnVmXzc3ODI0OiAgICAgICAgNzc4MjQsICAgICAgMCwgICAg ICAgMSwgICAgICAzMSwgIDEyNTI2NSwgICAwLCAgIDANCnppb19kYXRhX2J1Zl83NzgyNDogICA3 NzgyNCwgICAgICAwLCAgICAgICAwLCAgICAgIDI3LCAgICAgIDYyLCAgIDAsICAgMA0KemlvX2J1 Zl84MTkyMDogICAgICAgIDgxOTIwLCAgICAgIDAsICAgICAgIDEsICAgICAgMjUsICAzMDQ0OTEs ICAgMCwgICAwDQp6aW9fZGF0YV9idWZfODE5MjA6ICAgODE5MjAsICAgICAgMCwgICAgICAgMCwg ICAgICAyMiwgICAgICA2NCwgICAwLCAgIDANCnppb19idWZfODYwMTY6ICAgICAgICA4NjAxNiwg ICAgICAwLCAgICAgICAyLCAgICAgIDMzLCAgMTE5MTcyLCAgIDAsICAgMA0KemlvX2RhdGFfYnVm Xzg2MDE2OiAgIDg2MDE2LCAgICAgIDAsICAgICAgIDAsICAgICAgMjAsICAgICAgNjIsICAgMCwg ICAwDQp6aW9fYnVmXzkwMTEyOiAgICAgICAgOTAxMTIsICAgICAgMCwgICAgICAgMywgICAgICAz MywgIDIxODU3NiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl85MDExMjogICA5MDExMiwgICAgICAw LCAgICAgICAxLCAgICAgIDI2LCAgICAgIDcwLCAgIDAsICAgMA0KemlvX2J1Zl85NDIwODogICAg ICAgIDk0MjA4LCAgICAgIDAsICAgICAgIDAsICAgICAgMzEsICAxMTcwNTIsICAgMCwgICAwDQp6 aW9fZGF0YV9idWZfOTQyMDg6ICAgOTQyMDgsICAgICAgMCwgICAgICAgMCwgICAgICAyMywgICAg ICA1NCwgICAwLCAgIDANCnppb19idWZfOTgzMDQ6ICAgICAgICA5ODMwNCwgICAgICAwLCAgICAg ICAyLCAgICAgIDMwLCAgMjc2MDc2LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzk4MzA0OiAgIDk4 MzA0LCAgICAgIDAsICAgICAgIDEsICAgICAgMTYsICAgICAgNDIsICAgMCwgICAwDQp6aW9fYnVm XzEwMjQwMDogICAgICAxMDI0MDAsICAgICAgMCwgICAgICAgMiwgICAgICAzMSwgIDExMDA2MCwg ICAwLCAgIDANCnppb19kYXRhX2J1Zl8xMDI0MDA6IDEwMjQwMCwgICAgICAwLCAgICAgICAwLCAg ICAgIDIzLCAgICAgIDQzLCAgIDAsICAgMA0KemlvX2J1Zl8xMDY0OTY6ICAgICAgMTA2NDk2LCAg ICAgIDAsICAgICAgIDEsICAgICAgMzcsICAxOTAzNjksICAgMCwgICAwDQp6aW9fZGF0YV9idWZf MTA2NDk2OiAxMDY0OTYsICAgICAgMCwgICAgICAgMCwgICAgICAxOCwgICAgICA0NSwgICAwLCAg IDANCnppb19idWZfMTEwNTkyOiAgICAgIDExMDU5MiwgICAgICAwLCAgICAgICAwLCAgICAgIDM5 LCAgMTEyNDM5LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzExMDU5MjogMTEwNTkyLCAgICAgIDAs ICAgICAgIDAsICAgICAgMTgsICAgICAgMzMsICAgMCwgICAwDQp6aW9fYnVmXzExNDY4ODogICAg ICAxMTQ2ODgsICAgICAgMCwgICAgICAgMCwgICAgICA0MywgIDIzMjMzMSwgICAwLCAgIDANCnpp b19kYXRhX2J1Zl8xMTQ2ODg6IDExNDY4OCwgICAgICAwLCAgICAgICAwLCAgICAgIDE4LCAgICAg IDMzLCAgIDAsICAgMA0KemlvX2J1Zl8xMTg3ODQ6ICAgICAgMTE4Nzg0LCAgICAgIDAsICAgICAg IDAsICAgICAgMzYsICAxMDY1MzgsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTE4Nzg0OiAxMTg3 ODQsICAgICAgMCwgICAgICAgMCwgICAgICAxNSwgICAgICAxOSwgICAwLCAgIDANCnppb19idWZf MTIyODgwOiAgICAgIDEyMjg4MCwgICAgICAwLCAgICAgICAyLCAgICAgIDM3LCAgMjc1NTEyLCAg IDAsICAgMA0KemlvX2RhdGFfYnVmXzEyMjg4MDogMTIyODgwLCAgICAgIDAsICAgICAgIDAsICAg ICAgIDUsICAgICAgIDgsICAgMCwgICAwDQp6aW9fYnVmXzEyNjk3NjogICAgICAxMjY5NzYsICAg ICAgMCwgICAgICAgMCwgICAgICAzMiwgIDMxMTE1MSwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8x MjY5NzY6IDEyNjk3NiwgICAgICAwLCAgICAgICAwLCAgICAgICA4LCAgICAgIDEwLCAgIDAsICAg MA0KemlvX2J1Zl8xMzEwNzI6ICAgICAgMTMxMDcyLCAgICAgIDAsICAgIDQxOTIsICAgICAxNDYs IDY1NDA5NjQsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTMxMDcyOiAxMzEwNzIsICAgICAgMCwg ICAgMTgzOSwgICAgICA0NywgIDIzMDUwNywgICAwLCAgIDANCmx6NF9jdHg6ICAgICAgICAgICAg ICAxNjM4NCwgICAgICAwLCAgICAgICAxLCAgICAgIDkzLDUwMzYzOTExLCAgIDAsICAgMA0Kc2Ff Y2FjaGU6ICAgICAgICAgICAgICAgIDgwLCAgICAgIDAsICAgIDM4NDMsICAgIDEzMDcsICAgIDU0 NzEsICAgMCwgICAwDQpkbm9kZV90OiAgICAgICAgICAgICAgICA3NTIsICAgICAgMCwgICAzODY4 MywgICAgIDUyMiwgNTkyNzY0OCwgICAwLCAgIDANCmRtdV9idWZfaW1wbF90OiAgICAgICAgIDIz MiwgICAgICAwLCA0Njg4ODg2LCAgOTUzNTMzLDI2NDM4NDU5LCAgIDAsICAgMA0KYXJjX2J1Zl9o ZHJfdDogICAgICAgICAgMjE2LCAgICAgIDAsIDk2OTIzMjgsICAgICA4MzQsMTY1NTMwNjUsICAg MCwgICAwDQphcmNfYnVmX3Q6ICAgICAgICAgICAgICAgNzIsICAgICAgMCwgNDg0NzIwMywgMjAy NDAwMiwyMDA2NTI0NCwgICAwLCAgIDANCnppbF9sd2JfY2FjaGU6ICAgICAgICAgIDE5MiwgICAg ICAwLCAgIDIwNTI4LCAgICAgNjUyLCAgODUzMzMzLCAgIDAsICAgMA0KemZzX3pub2RlX2NhY2hl OiAgICAgICAgMzY4LCAgICAgIDAsICAgIDM4NDMsICAgICAzMzcsICAgIDU0NzEsICAgMCwgICAw DQpNb3VudHBvaW50czogICAgICAgICAgICA4MTYsICAgICAgMCwgICAgICAyMywgICAgIDE0Nywg ICAgICAyMywgICAwLCAgIDANCmtzaWdpbmZvOiAgICAgICAgICAgICAgIDExMiwgICAgICAwLCAg ICAgMzc0LCAgICAzNDA2LCAgICAxODg2LCAgIDAsICAgMA0KaXRpbWVyOiAgICAgICAgICAgICAg ICAgMzUyLCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpLTk9U RTogICAgICAgICAgICAgICAgICAxMjgsICAgICAgMCwgICAgICAgMCwgICAgMzAwNywgICAgODc0 NiwgICAwLCAgIDANCnNvY2tldDogICAgICAgICAgICAgICAgIDY5NiwgMzE0NDQ4NSwgICAgICA2 OSwgICAgIDMzNiwgICAyMzc5MSwgICAwLCAgIDANCmlwcTogICAgICAgICAgICAgICAgICAgICA1 NiwgMTkxNDg3LCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KdWRwX2lucGNi OiAgICAgICAgICAgICAgMzkyLCAzMTQ0NDkwLCAgICAgIDExLCAgICAgNzQ5LCAgICA5MDUwLCAg IDAsICAgMA0KdWRwY2I6ICAgICAgICAgICAgICAgICAgIDE2LCAzMTQ0NTI4LCAgICAgIDExLCAg ICA2MjY0LCAgICA5MDUwLCAgIDAsICAgMA0KdGNwX2lucGNiOiAgICAgICAgICAgICAgMzkyLCAz MTQ0NDkwLCAgICAgIDQ1LCAgICAgNjU1LCAgICAzMDE2LCAgIDAsICAgMA0KdGNwY2I6ICAgICAg ICAgICAgICAgICAxMDI0LCAzMTQ0NDg4LCAgICAgIDQ1LCAgICAgMjY3LCAgICAzMDE2LCAgIDAs ICAgMA0KdGNwdHc6ICAgICAgICAgICAgICAgICAgIDg4LCAgMjc4MTAsICAgICAgIDAsICAgICAg IDAsICAgICAgIDAsICAgMCwgICAwDQpzeW5jYWNoZTogICAgICAgICAgICAgICAxNjAsICAxNTM3 NSwgICAgICAgMCwgICAgIDUwMCwgICAgICA4NywgICAwLCAgIDANCmhvc3RjYWNoZTogICAgICAg ICAgICAgIDEzNiwgIDE1MzcwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0K dGNwcmVhc3M6ICAgICAgICAgICAgICAgIDQwLCAzODMwMDAsICAgICAgIDAsICAgICA4MDAsICAg ICAgNTcsICAgMCwgICAwDQpzYWNraG9sZTogICAgICAgICAgICAgICAgMzIsICAgICAgMCwgICAg ICAgMCwgICAgMTYyNSwgICAgIDIwOCwgICAwLCAgIDANCnNjdHBfZXA6ICAgICAgICAgICAgICAg MTQwOCwgMzE0NDQ4NiwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnNjdHBf YXNvYzogICAgICAgICAgICAgMjQxNiwgIDQwMDAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAw LCAgIDAsICAgMA0Kc2N0cF9sYWRkcjogICAgICAgICAgICAgIDQ4LCAgODAwMTIsICAgICAgIDAs ICAgIDEwNzksICAgICAgIDQsICAgMCwgICAwDQpzY3RwX3JhZGRyOiAgICAgICAgICAgICA3Mjgs ICA4MDAwMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnNjdHBfY2h1bms6 ICAgICAgICAgICAgIDEzNiwgNDAwMDI2LCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAs ICAgMA0Kc2N0cF9yZWFkcTogICAgICAgICAgICAgMTA0LCA0MDAwMjYsICAgICAgIDAsICAgICAg IDAsICAgICAgIDAsICAgMCwgICAwDQpzY3RwX3N0cmVhbV9tc2dfb3V0OiAgICAxMDQsIDQwMDAy NiwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnNjdHBfYXNjb25mOiAgICAg ICAgICAgICA0MCwgNDAwMDAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0K c2N0cF9hc2NvbmZfYWNrOiAgICAgICAgIDQ4LCA0MDAwNjAsICAgICAgIDAsICAgICAgIDAsICAg ICAgIDAsICAgMCwgICAwDQp1ZHBsaXRlX2lucGNiOiAgICAgICAgICAzOTIsIDMxNDQ0OTAsICAg ICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpyaXBjYjogICAgICAgICAgICAgICAg ICAzOTIsIDMxNDQ0OTAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQp1bnBj YjogICAgICAgICAgICAgICAgICAyNDAsIDMxNDQ0OTYsICAgICAgMTMsICAgIDE0NTksICAgMTE3 MjAsICAgMCwgICAwDQpydGVudHJ5OiAgICAgICAgICAgICAgICAyMDAsICAgICAgMCwgICAgICAx NSwgICAgIDQ4NSwgICAgICAxNSwgICAwLCAgIDANCnNlbGZkOiAgICAgICAgICAgICAgICAgICA1 NiwgICAgICAwLCAgICAgNDY4LCAgICA1NjM4LCAyNTg2OTg0LCAgIDAsICAgMA0KU1dBUE1FVEE6 ICAgICAgICAgICAgICAgMjg4LCAxMjI1NDMwNywgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwg ICAwLCAgIDANCmljbF9jb25uOiAgICAgICAgICAgICAgIDE1MiwgICAgICAwLCAgICAgIDIwLCAg ICAgNjU2LCAgICAgIDIwLCAgIDAsICAgMA0KaWNsX3BkdTogICAgICAgICAgICAgICAgIDgwLCAg ICAgIDAsICAgICAgNzgsICAgIDczMjIsMTg5MDc1NjIsICAgMCwgICAwDQppc2NzaV9vdXRzdGFu ZGluZzogICAgICAgNDAsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAg IDANCmNmaV9sdW46ICAgICAgICAgICAgICAgIDM0NCwgICAgICAwLCAgICAgIDI0LCAgICAgMjk1 LCAgICAgIDI0LCAgIDAsICAgMA0KY2ZpX21ldGF0YXNrOiAgICAgICAgICAgMzIwLCAgICAgIDAs ICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpjZmlzY3NpX2RhdGFfd2FpdDog ICAgICAgNTYsICAgICAgMCwgICAgICAgMCwgICAgNzEwMCwgNjY1NjQwNCwgICAwLCAgIDANCmJl aW86ICAgICAgICAgICAgICAgICAgIDM2MCwgICAgICAwLCAgICAgIDQwLCAgICAgODQwLCA4MjQ1 MTU0LCAgIDAsICAgMA0KY2JsazA6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAg IDAsICAgICAgMTAsICAgICAgMjksICAgMCwgICAwDQpjYmxrMTogICAgICAgICAgICAgICAxMzEw NzIsICAgICAgMCwgICAgICAgMCwgICAgICAzNCwgICAgMTAyOCwgICAwLCAgIDANCmNibGsyOiAg ICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDE0LCAgICAgIDI5LCAg IDAsICAgMA0KY2JsazM6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAg ICAgMTAsICAgICAgMjksICAgMCwgICAwDQpjYmxrNDogICAgICAgICAgICAgICAxMzEwNzIsICAg ICAgMCwgICAgICAgMCwgICAgICAgNywgICAgICAyOSwgICAwLCAgIDANCmNibGs1OiAgICAgICAg ICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDIwLCAgICAgIDk0LCAgIDAsICAg MA0KY2JsazY6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDEsICAgICAgNzks IDIyMTgxNjcsICAgMCwgICAwDQpjYmxrNzogICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwg ICAgICAgMCwgICAgICAxMCwgICAgICAyMywgICAwLCAgIDANCmNibGs4OiAgICAgICAgICAgICAg IDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDUyLCAgIDI5NTQ3LCAgIDAsICAgMA0KY2Js azk6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAg IDAsICAgMCwgICAwDQpjYmxrMTA6ICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAg MCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCmNibGsxMTogICAgICAgICAgICAgIDEzMTA3 MiwgICAgICAwLCAgICAgICAwLCAgICAgIDc5LCAgIDIwMzI0LCAgIDAsICAgMA0KY2JsazEyOiAg ICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgIDksICAgICAgOTQsICAg MCwgICAwDQpjYmxrMTM6ICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAg ICAgMCwgICAgICAgMCwgICAwLCAgIDANCmNibGsxNDogICAgICAgICAgICAgIDEzMTA3MiwgICAg ICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KY2JsazE1OiAgICAgICAg ICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgMTEsICAgICAgMjQsICAzMDYxNjcsICAgMCwgICAw DQpjYmxrMTY6ICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAxMCwgICAgICAyNSwg IDYzNTUwNSwgICAwLCAgIDANCmNibGsxNzogICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAg ICAgIDExLCAgICAgIDI1LCAyMTQwNzg1LCAgIDAsICAgMA0KY2JsazE4OiAgICAgICAgICAgICAg MTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgMTMsICAgICAgOTQsICAgMCwgICAwDQpjYmxr MTk6ICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICAxMywgICAgICA5 NCwgICAwLCAgIDANCmNibGsyMDogICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAy LCAgICAgIDc2LCAxMTczNzA2LCAgIDAsICAgMA0KY2JsazIxOiAgICAgICAgICAgICAgMTMxMDcy LCAgICAgIDAsICAgICAgIDEsICAgICAgNTgsIDEyNzUyODUsICAgMCwgICAwDQpjYmxrMjI6ICAg ICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMiwgICAgICAzNSwgICAzNTE4MiwgICAw LCAgIDANCmNibGsyMzogICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAyLCAgICAg IDE5LCAgNDE4OTA1LCAgIDAsICAgMA0KDQogcHJvY3MgICAgICBtZW1vcnkgICAgICBwYWdlICAg ICAgICAgICAgICAgICAgICBkaXNrcyAgICAgZmF1bHRzICAgICAgICAgY3B1DQogciBiIHcgICAg IGF2bSAgICBmcmUgICBmbHQgIHJlICBwaSAgcG8gICAgZnIgIHNyIGRhMCBkYTEgICBpbiAgIHN5 ICAgY3MgdXMgc3kgaWQNCiAwIDAgMCAgIDIwMTBNICAgNjk1TSAgMjIwNSAgIDAgICAwICAgMCAg MjkxMSAgIDIgICAwICAgMCAxOTgyIDQ2MzI1IDEwNDY4ICAwICAzIDk3DQo0MDYwNi8yMzAwOS82 MzYxNSBtYnVmcyBpbiB1c2UgKGN1cnJlbnQvY2FjaGUvdG90YWwpDQozOTA0NC8xMTIzMi81MDI3 Ni82MTI3MTQ4IG1idWYgY2x1c3RlcnMgaW4gdXNlIChjdXJyZW50L2NhY2hlL3RvdGFsL21heCkN CjM5MDMxLzUzODUgbWJ1ZitjbHVzdGVycyBvdXQgb2YgcGFja2V0IHNlY29uZGFyeSB6b25lIGlu IHVzZSAoY3VycmVudC9jYWNoZSkNCjAvNDg1LzQ4NS8zMDYzNTc0IDRrIChwYWdlIHNpemUpIGp1 bWJvIGNsdXN0ZXJzIGluIHVzZSAoY3VycmVudC9jYWNoZS90b3RhbC9tYXgpDQowLzAvMC85MDc3 MjUgOWsganVtYm8gY2x1c3RlcnMgaW4gdXNlIChjdXJyZW50L2NhY2hlL3RvdGFsL21heCkNCjAv MC8wLzUxMDU5NSAxNmsganVtYm8gY2x1c3RlcnMgaW4gdXNlIChjdXJyZW50L2NhY2hlL3RvdGFs L21heCkNCjg4MjM5Sy8zMDE1NksvMTE4Mzk1SyBieXRlcyBhbGxvY2F0ZWQgdG8gbmV0d29yayAo Y3VycmVudC9jYWNoZS90b3RhbCkNCjAvMC8wIHJlcXVlc3RzIGZvciBtYnVmcyBkZW5pZWQgKG1i dWZzL2NsdXN0ZXJzL21idWYrY2x1c3RlcnMpDQowLzAvMCByZXF1ZXN0cyBmb3IgbWJ1ZnMgZGVs YXllZCAobWJ1ZnMvY2x1c3RlcnMvbWJ1ZitjbHVzdGVycykNCjAvMC8wIHJlcXVlc3RzIGZvciBq dW1ibyBjbHVzdGVycyBkZWxheWVkICg0ay85ay8xNmspDQowLzAvMCByZXF1ZXN0cyBmb3IganVt Ym8gY2x1c3RlcnMgZGVuaWVkICg0ay85ay8xNmspDQowIHJlcXVlc3RzIGZvciBzZmJ1ZnMgZGVu aWVkDQowIHJlcXVlc3RzIGZvciBzZmJ1ZnMgZGVsYXllZA0KMCByZXF1ZXN0cyBmb3IgSS9PIGlu aXRpYXRlZCBieSBzZW5kZmlsZQ0KSVRFTSAgICAgICAgICAgICAgICAgICBTSVpFICBMSU1JVCAg ICAgVVNFRCAgICAgRlJFRSAgICAgIFJFUSBGQUlMIFNMRUVQDQoNClVNQSBLZWdzOiAgICAgICAg ICAgICAgIDM4NCwgICAgICAwLCAgICAgMjMyLCAgICAgICA4LCAgICAgMjMyLCAgIDAsICAgMA0K VU1BIFpvbmVzOiAgICAgICAgICAgICAzNzEyLCAgICAgIDAsICAgICAyMzIsICAgICAgIDAsICAg ICAyMzIsICAgMCwgICAwDQpVTUEgU2xhYnM6ICAgICAgICAgICAgICAgODAsICAgICAgMCwgNTk1 OTc1MywgMjA4ODk0NywxODA4NzMyOSwgICAwLCAgIDANClVNQSBSQ250U2xhYnM6ICAgICAgICAg ICA4OCwgICAgICAwLCAgIDI0NzY5LCAgICA1Nzg2LCAgIDM1NzkzLCAgIDAsICAgMA0KVU1BIEhh c2g6ICAgICAgICAgICAgICAgMjU2LCAgICAgIDAsICAgICAgMzEsICAgICAgNDQsICAgICAxMDUs ICAgMCwgICAwDQo0IEJ1Y2tldDogICAgICAgICAgICAgICAgMzIsICAgICAgMCwgICAgICAgMCwg ICAgIDM3NSwgMzI0NzUzOSwgICAwLCAgIDANCjYgQnVja2V0OiAgICAgICAgICAgICAgICA0OCwg ICAgICAwLCAgICAgICAwLCAgICAgICAwLCAxMTgxNTY0LCAgIDAsICAgMA0KOCBCdWNrZXQ6ICAg ICAgICAgICAgICAgIDY0LCAgICAgIDAsICAgICAgIDAsICAgICAzMTAsICA1NTAxNzUsICAxMSwg ICAwDQoxMiBCdWNrZXQ6ICAgICAgICAgICAgICAgOTYsICAgICAgMCwgICAgICAgMCwgICAgICAg MCwgIDc3OTAwMiwgICAwLCAgIDANCjE2IEJ1Y2tldDogICAgICAgICAgICAgIDEyOCwgICAgICAw LCAgICAgICAwLCAgICAgIDMxLCAgNzQwODI5LCAgIDAsICAgMA0KMzIgQnVja2V0OiAgICAgICAg ICAgICAgMjU2LCAgICAgIDAsICAgICAgIDAsICAgICAgMTUsIDE0Mzg3ODUsICA1MSwgICAwDQo2 NCBCdWNrZXQ6ICAgICAgICAgICAgICA1MTIsICAgICAgMCwgICAgICAgMCwgICAgICAxNiwgMTMz OTA0MSwgMTAzLCAgIDANCjEyOCBCdWNrZXQ6ICAgICAgICAgICAgMTAyNCwgICAgICAwLCAgICAg ICAwLCAgICAgIDE2LCAxMjAwMjc1LCAgIDAsICAgMA0KMjU2IEJ1Y2tldDogICAgICAgICAgICAy MDQ4LCAgICAgIDAsICAgICAgIDAsICAgICAgIDgsNDEwOTY3ODIsICA2NCwgICAwDQp2bWVtIGJ0 YWc6ICAgICAgICAgICAgICAgNTYsICAgICAgMCwgMTI0OTE1NSwgICA1NjY3NywgMjAzMzY2Myw5 MjE3LCAgIDANClZNIE9CSkVDVDogICAgICAgICAgICAgIDI1NiwgICAgICAwLCAgICA3MTEzLCAg ICAxOTYyLCAyOTYyOTgwLCAgIDAsICAgMA0KUkFESVggTk9ERTogICAgICAgICAgICAgMTQ0LCAg ICAgIDAsIDExNjEzNzQsICAgNzUwOTEsMTQzOTMyNDksICAgMCwgICAwDQpNQVA6ICAgICAgICAg ICAgICAgICAgICAyNDAsICAgICAgMCwgICAgICAgMywgICAgICA2MSwgICAgICAgMywgICAwLCAg IDANCktNQVAgRU5UUlk6ICAgICAgICAgICAgIDEyOCwgICAgICAwLCAgICAgICA5LCAgICAgMTE1 LCAgICAgICA5LCAgIDAsICAgMA0KTUFQIEVOVFJZOiAgICAgICAgICAgICAgMTI4LCAgICAgIDAs ICAgIDI3ODQsICAgIDU5NTgsIDg3MjM5MjEsICAgMCwgICAwDQpWTVNQQUNFOiAgICAgICAgICAg ICAgICA0NDgsICAgICAgMCwgICAgICA3MCwgICAgIDgzMCwgIDE5NTYxNCwgICAwLCAgIDANCmZh a2VwZzogICAgICAgICAgICAgICAgIDEwNCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAg ICAwLCAgIDAsICAgMA0KbXRfem9uZTogICAgICAgICAgICAgICA0MTEyLCAgICAgIDAsICAgICAz NjksICAgICAgIDAsICAgICAzNjksICAgMCwgICAwDQoxNjogICAgICAgICAgICAgICAgICAgICAg MTYsICAgICAgMCwgICAzMDIzNiwgICAgNTY1Nyw1NDI0NTUyNiwgICAwLCAgIDANCjMyOiAgICAg ICAgICAgICAgICAgICAgICAzMiwgICAgICAwLCA5ODY4NjQ4LCAxMDUyMTAyLDEyODU1OTUyOCwg ICAwLCAgIDANCjY0OiAgICAgICAgICAgICAgICAgICAgICA2NCwgICAgICAwLCA0Mzg5MjUyLCAx MTQ4NzExLDYyNzQxNTQyNSwgICAwLCAgIDANCjEyODogICAgICAgICAgICAgICAgICAgIDEyOCwg ICAgICAwLCAxNDUzODg5LCAyOTQ2OTMzLDM0ODAxMDU0MywgICAwLCAgIDANCjI1NjogICAgICAg ICAgICAgICAgICAgIDI1NiwgICAgICAwLCAxNTE1NTU2LCAgICAgIDE0LDIyMzk3MTkyNCwgICAw LCAgIDANCjUxMjogICAgICAgICAgICAgICAgICAgIDUxMiwgICAgICAwLCAgICAxNDc2LCAgICA0 OTU2LDcyOTc5NDk4LCAgIDAsICAgMA0KMTAyNDogICAgICAgICAgICAgICAgICAxMDI0LCAgICAg IDAsICAgMjY4NjIsICAgICAzODIsNjA4OTc0MTcsICAgMCwgICAwDQoyMDQ4OiAgICAgICAgICAg ICAgICAgIDIwNDgsICAgICAgMCwgIDEwNTUwNSwgICAgIDY5MSw1NDkxOTY5MywgICAwLCAgIDAN CjQwOTY6ICAgICAgICAgICAgICAgICAgNDA5NiwgICAgICAwLCAgICA0MjQ0LCAgICAgICAwLCAy NjIzNjgwLCAgIDAsICAgMA0KODE5MjogICAgICAgICAgICAgICAgICA4MTkyLCAgICAgIDAsICAg ICAgMzgsICAgICAgIDAsIDE5MDk1NjUsICAgMCwgICAwDQoxNjM4NDogICAgICAgICAgICAgICAg MTYzODQsICAgICAgMCwgICAgICA0NiwgICAgICAgMCwgICAgNTExOCwgICAwLCAgIDANCjMyNzY4 OiAgICAgICAgICAgICAgICAzMjc2OCwgICAgICAwLCAgICAgIDQ3LCAgICAgICAwLCAgICA2NDEx LCAgIDAsICAgMA0KNjU1MzY6ICAgICAgICAgICAgICAgIDY1NTM2LCAgICAgIDAsICAgICAxMDYs ICAgICAgIDAsICAxNzU4MzIsICAgMCwgICAwDQo2NCBwY3B1OiAgICAgICAgICAgICAgICAgIDgs ICAgICAgMCwgICAgMTc5MCwgICAgMTAyNiwgICAgMjExMCwgICAwLCAgIDANClNMRUVQUVVFVUU6 ICAgICAgICAgICAgICA4MCwgICAgICAwLCAgICAxOTMwLCAgICAgNjQzLCAgICAyMDA4LCAgIDAs ICAgMA0KRmlsZXM6ICAgICAgICAgICAgICAgICAgIDgwLCAgICAgIDAsICAgICAyNDQsICAgIDMy NTYsIDEzMzU3ODMsICAgMCwgICAwDQpUVVJOU1RJTEU6ICAgICAgICAgICAgICAxMzYsICAgICAg MCwgICAgMTkzMCwgICAgIDg5MCwgICAgMjAwOCwgICAwLCAgIDANCnJsX2VudHJ5OiAgICAgICAg ICAgICAgICA0MCwgICAgICAwLCAgICAgMTYzLCAgICAyMzM3LCAgICAgMTYzLCAgIDAsICAgMA0K dW10eCBwaTogICAgICAgICAgICAgICAgIDk2LCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAg ICAgIDAsICAgMCwgICAwDQpNQUMgbGFiZWxzOiAgICAgICAgICAgICAgNDAsICAgICAgMCwgICAg ICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANClBST0M6ICAgICAgICAgICAgICAgICAg MTIxNiwgICAgICAwLCAgICAgIDkxLCAgICAgMzE0LCAgMTk1NjM1LCAgIDAsICAgMA0KVEhSRUFE OiAgICAgICAgICAgICAgICAxMTY4LCAgICAgIDAsICAgIDE4MjMsICAgICAxMDYsICAgIDQwNDAs ICAgMCwgICAwDQpjcHVzZXQ6ICAgICAgICAgICAgICAgICAgNzIsICAgICAgMCwgICAgMTQ2MSwg ICAgMTI4OSwgICAgMzY5NCwgICAwLCAgIDANCmF1ZGl0X3JlY29yZDogICAgICAgICAgMTI0OCwg ICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KbWJ1Zl9wYWNrZXQ6 ICAgICAgICAgICAgMjU2LCAzOTIxMzc1MCwgICAzOTAzMywgICAgICAgMCwxNDcxNzE4ODksICAg MCwgICAwDQptYnVmOiAgICAgICAgICAgICAgICAgICAyNTYsIDM5MjEzNzUwLCAgICAxNTcyLCAg IDIyMDk1LDE5MzU3Njg3MCwgICAwLCAgIDANCm1idWZfY2x1c3RlcjogICAgICAgICAgMjA0OCwg NjEyNzE0OCwgICAzOTA0NiwgICAxMDQ5MiwgMjg1NjA2NCwgICAwLCAgIDANCm1idWZfanVtYm9f cGFnZTogICAgICAgNDA5NiwgMzA2MzU3NCwgICAgICAgMCwgICAgICAgMCwxMjIwMTMxNywgICAw LCAgIDANCm1idWZfanVtYm9fOWs6ICAgICAgICAgOTIxNiwgOTA3NzI1LCAgICAgICAwLCAgICAg ICAwLCAgICAgICAwLCAgIDAsICAgMA0KbWJ1Zl9qdW1ib18xNms6ICAgICAgIDE2Mzg0LCA1MTA1 OTUsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQptYnVmX2V4dF9yZWZjbnQ6 ICAgICAgICAgIDQsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDAN CmdfYmlvOiAgICAgICAgICAgICAgICAgIDI0OCwgICAgICAwLCAgICAgMjg1LCAgICAgMjU5LDIw MTQyMjYwNSwgICAwLCAgIDANCnR0eWlucTogICAgICAgICAgICAgICAgIDE2MCwgICAgICAwLCAg ICAgMjQwLCAgICAgMjYwLCAgICAgODI1LCAgIDAsICAgMA0KdHR5b3V0cTogICAgICAgICAgICAg ICAgMjU2LCAgICAgIDAsICAgICAxMjYsICAgICAxODksICAgICA0MzgsICAgMCwgICAwDQpETUFS X01BUF9FTlRSWTogICAgICAgICAxMjAsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAg MCwgICAwLCAgIDANCmF0YV9yZXF1ZXN0OiAgICAgICAgICAgIDMzNiwgICAgICAwLCAgICAgICAw LCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KdnRuZXRfdHhfaGRyOiAgICAgICAgICAgIDI0 LCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpGUFVfc2F2ZV9h cmVhOiAgICAgICAgICA1MTIsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAw LCAgIDANCnRhc2txX3pvbmU6ICAgICAgICAgICAgICA0OCwgICAgICAwLCAgICAgICAwLCAgICAg ICAwLCAgNTI3MjA2LCAgIDAsICAgMA0KVk5PREU6ICAgICAgICAgICAgICAgICAgNDcyLCAgICAg IDAsICAgIDM5NDUsICAgICAxNTEsICAgIDU2MTgsICAgMCwgICAwDQpWTk9ERVBPTEw6ICAgICAg ICAgICAgICAxMTIsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDAN CkJVRiBUUklFOiAgICAgICAgICAgICAgIDE0NCwgICAgICAwLCAgICAgICAwLCAgMTA1OTQ4LCAg ICAgICAwLCAgIDAsICAgMA0KUyBWRlMgQ2FjaGU6ICAgICAgICAgICAgMTA4LCAgICAgIDAsICAg ICA4NjEsICAgIDE4MzQsICAgNDM4NjAsICAgMCwgICAwDQpTVFMgVkZTIENhY2hlOiAgICAgICAg ICAxNDgsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCkwgVkZT IENhY2hlOiAgICAgICAgICAgIDMyOCwgICAgICAwLCAgICAgICA4LCAgICAgIDc2LCAgICAgICA5 LCAgIDAsICAgMA0KTFRTIFZGUyBDYWNoZTogICAgICAgICAgMzY4LCAgICAgIDAsICAgICAgIDAs ICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpOQU1FSTogICAgICAgICAgICAgICAgIDEwMjQs ICAgICAgMCwgICAgICAxMiwgICAgICAzNiwgNDQ3MzQ4OSwgICAwLCAgIDANCk5DTE5PREU6ICAg ICAgICAgICAgICAgIDUyOCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAs ICAgMA0KRElSSEFTSDogICAgICAgICAgICAgICAxMDI0LCAgICAgIDAsICAgICAgIDAsICAgICAg IDAsICAgICAgIDAsICAgMCwgICAwDQpwaXBlOiAgICAgICAgICAgICAgICAgICA3NDQsICAgICAg MCwgICAgICA1MCwgICAgIDE2MCwgICAxNzY2NywgICAwLCAgIDANCnByb2NkZXNjOiAgICAgICAg ICAgICAgIDEyOCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0K cmFuZ2Vfc2VnX2NhY2hlOiAgICAgICAgIDY0LCAgICAgIDAsMTIzNzgyNDcsIDQ1Mzk1MDcsOTI4 OTg4Njk3MCwgICAwLCAgIDANCnppb19jYWNoZTogICAgICAgICAgICAgIDkyMCwgICAgICAwLCAz NDQ3OTA3LCAgICAgICAxLDEwNDI2MDIwMjYsICAgMCwgICAwDQp6aW9fbGlua19jYWNoZTogICAg ICAgICAgNDgsICAgICAgMCwgMzQ1MDQxNiwgICAgICA2MCwxMjExMTc2MjUwLCAgIDAsICAgMA0K emlvX2J1Zl81MTI6ICAgICAgICAgICAgNTEyLCAgICAgIDAsICAgMTI4MzksICAgICA2ODEsIDEw Njg0MDEsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNTEyOiAgICAgICA1MTIsICAgICAgMCwgICAg ICAyMywgICAgIDExMywgIDMzNDQ5NywgICAwLCAgIDANCnppb19idWZfMTAyNDogICAgICAgICAg MTAyNCwgICAgICAwLCAgICAgIDI2LCAgICAgIDYyLCAgMzYxODM5LCAgIDAsICAgMA0KemlvX2Rh dGFfYnVmXzEwMjQ6ICAgICAxMDI0LCAgICAgIDAsICAgICA0NTAsICAgICAzNTAsICAgIDYyNDAs ICAgMCwgICAwDQp6aW9fYnVmXzE1MzY6ICAgICAgICAgIDE1MzYsICAgICAgMCwgICAgICAyNiwg ICAgICAyNiwgIDMwMDE0MywgICAwLCAgIDANCnppb19kYXRhX2J1Zl8xNTM2OiAgICAgMTUzNiwg ICAgICAwLCAgICAgMTgzLCAgICAgMTI1LCAgMTE5NjkzLCAgIDAsICAgMA0KemlvX2J1Zl8yMDQ4 OiAgICAgICAgICAyMDQ4LCAgICAgIDAsICAgICA3MTMsICAgICAxNDMsIDI0NzM3OTksICAgMCwg ICAwDQp6aW9fZGF0YV9idWZfMjA0ODogICAgIDIwNDgsICAgICAgMCwgICAgICAyOSwgICAgICAy OSwgIDQ2MjM0MywgICAwLCAgIDANCnppb19idWZfMjU2MDogICAgICAgICAgMjU2MCwgICAgICAw LCAgICAgICA1LCAgICAgICAwLCAyNDg4NzA0LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzI1NjA6 ICAgICAyNTYwLCAgICAgIDAsICAgICAgNDAsICAgICAgIDAsICA0Njg0OTQsICAgMCwgICAwDQp6 aW9fYnVmXzMwNzI6ICAgICAgICAgIDMwNzIsICAgICAgMCwgICAgICA1OCwgICAgICAgMCwgIDU3 NDU5NiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8zMDcyOiAgICAgMzA3MiwgICAgICAwLCAgICAg ICA0LCAgICAgICAwLCAgMTQxNzYxLCAgIDAsICAgMA0KemlvX2J1Zl8zNTg0OiAgICAgICAgICAz NTg0LCAgICAgIDAsICAgICA0MDUsICAgICAgIDAsICAxODM4NzQsICAgMCwgICAwDQp6aW9fZGF0 YV9idWZfMzU4NDogICAgIDM1ODQsICAgICAgMCwgICAgICAgNywgICAgICAgMCwgICAgNjE2MSwg ICAwLCAgIDANCnppb19idWZfNDA5NjogICAgICAgICAgNDA5NiwgICAgICAwLCAzNzk3MzkxLCAg ICAgICAwLDE1ODM2Mzc2MywgICAwLCAgIDANCnppb19kYXRhX2J1Zl80MDk2OiAgICAgNDA5Niwg ICAgICAwLCAgOTQ3NTA1LCAgICAgICAwLDEwNTg5ODQ4LCAgIDAsICAgMA0KemlvX2J1Zl81MTIw OiAgICAgICAgICA1MTIwLCAgICAgIDAsICAgICAxOTQsICAgICAgIDAsICAxNDcxNTYsICAgMCwg ICAwDQp6aW9fZGF0YV9idWZfNTEyMDogICAgIDUxMjAsICAgICAgMCwgICAgICAgNywgICAgICAg MCwgICAxMDk5MywgICAwLCAgIDANCnppb19idWZfNjE0NDogICAgICAgICAgNjE0NCwgICAgICAw LCAgICAgICAwLCAgICAgICAwLCAgMjAxMDMyLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzYxNDQ6 ICAgICA2MTQ0LCAgICAgIDAsICAgICAgIDUsICAgICAgIDAsICAgIDY5NjksICAgMCwgICAwDQp6 aW9fYnVmXzcxNjg6ICAgICAgICAgIDcxNjgsICAgICAgMCwgICAgICAgMSwgICAgICAgMCwgIDEy MjEyNywgICAwLCAgIDANCnppb19kYXRhX2J1Zl83MTY4OiAgICAgNzE2OCwgICAgICAwLCAgICAg ICAwLCAgICAgICAwLCAgICA1OTQzLCAgIDAsICAgMA0KemlvX2J1Zl84MTkyOiAgICAgICAgICA4 MTkyLCAgICAgIDAsICAgICAgMzMsICAgICAgMjIsMTQ3MzE3NzksICAgMCwgICAwDQp6aW9fZGF0 YV9idWZfODE5MjogICAgIDgxOTIsICAgICAgMCwgICAgMTIwNCwgICAgICAgMCwgICAxNjEzMCwg ICAwLCAgIDANCnppb19idWZfMTAyNDA6ICAgICAgICAxMDI0MCwgICAgICAwLCAgICAgICAyLCAg ICAgICAwLCAgMTQ4MjkxLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzEwMjQwOiAgIDEwMjQwLCAg ICAgIDAsICAgICAgIDMsICAgICAgIDAsICAgIDgwODIsICAgMCwgICAwDQp6aW9fYnVmXzEyMjg4 OiAgICAgICAgMTIyODgsICAgICAgMCwgICAgICAgOSwgICAgICAgOCwgNDQ1MzY0OSwgICAwLCAg IDANCnppb19kYXRhX2J1Zl8xMjI4ODogICAxMjI4OCwgICAgICAwLCAgICAgICAzLCAgICAgICAw LCAgICA3MDY5LCAgIDAsICAgMA0KemlvX2J1Zl8xNDMzNjogICAgICAgIDE0MzM2LCAgICAgIDAs ICAgICAgIDEsICAgICAgIDAsICAgOTQzOTQsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTQzMzY6 ICAgMTQzMzYsICAgICAgMCwgICAgICAgNSwgICAgICAgMCwgICAgNjk4NywgICAwLCAgIDANCnpp b19idWZfMTYzODQ6ICAgICAgICAxNjM4NCwgICAgICAwLCAgMjEyMTExLCAgICAgICA0LDE5MjUw MDI1LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzE2Mzg0OiAgIDE2Mzg0LCAgICAgIDAsICAgICA3 NDQsICAgICAgIDAsICAgOTM2MDEsICAgMCwgICAwDQp6aW9fYnVmXzIwNDgwOiAgICAgICAgMjA0 ODAsICAgICAgMCwgICAgICAgOCwgICAgICAgNCwgMTE1NDIzNywgICAwLCAgIDANCnppb19kYXRh X2J1Zl8yMDQ4MDogICAyMDQ4MCwgICAgICAwLCAgICAgICA3LCAgICAgICAwLCAgIDE4MTkzLCAg IDAsICAgMA0KemlvX2J1Zl8yNDU3NjogICAgICAgIDI0NTc2LCAgICAgIDAsICAgICAgIDksICAg ICAgMTUsIDE5MzkwMDEsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMjQ1NzY6ICAgMjQ1NzYsICAg ICAgMCwgICAgICAgMiwgICAgICAgMCwgICAyNTQyOCwgICAwLCAgIDANCnppb19idWZfMjg2NzI6 ICAgICAgICAyODY3MiwgICAgICAwLCAgICAgICA1LCAgICAgICAzLCAxMDg4NzM4LCAgIDAsICAg MA0KemlvX2RhdGFfYnVmXzI4NjcyOiAgIDI4NjcyLCAgICAgIDAsICAgICAgIDAsICAgICAgIDAs ICAgMzAzMjUsICAgMCwgICAwDQp6aW9fYnVmXzMyNzY4OiAgICAgICAgMzI3NjgsICAgICAgMCwg ICAgICAxMywgICAgICAgMywgMTY4MzMxNywgICAwLCAgIDANCnppb19kYXRhX2J1Zl8zMjc2ODog ICAzMjc2OCwgICAgICAwLCAgICAxNDc4LCAgICAgICAwLCAgIDQxNDIxLCAgIDAsICAgMA0Kemlv X2J1Zl8zNjg2NDogICAgICAgIDM2ODY0LCAgICAgIDAsICAgICAgIDYsICAgICAgIDIsICA3MTAx MDUsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMzY4NjQ6ICAgMzY4NjQsICAgICAgMCwgICAgICAg MiwgICAgICAgMCwgICAyMTQ1OSwgICAwLCAgIDANCnppb19idWZfNDA5NjA6ICAgICAgICA0MDk2 MCwgICAgICAwLCAgICAgICA0LCAgICAgICA4LCAgODM4Mzk4LCAgIDAsICAgMA0KemlvX2RhdGFf YnVmXzQwOTYwOiAgIDQwOTYwLCAgICAgIDAsICAgICAgIDUsICAgICAgIDAsICAgMTMyMjAsICAg MCwgICAwDQp6aW9fYnVmXzQ1MDU2OiAgICAgICAgNDUwNTYsICAgICAgMCwgICAgICAgMiwgICAg ICAgMywgIDQ5Nzc0MCwgICAwLCAgIDANCnppb19kYXRhX2J1Zl80NTA1NjogICA0NTA1NiwgICAg ICAwLCAgICAgICAyLCAgICAgICAwLCAgICA5MjYyLCAgIDAsICAgMA0KemlvX2J1Zl80OTE1Mjog ICAgICAgIDQ5MTUyLCAgICAgIDAsICAgICAgIDksICAgICAgIDAsICA5MDA4NjQsICAgMCwgICAw DQp6aW9fZGF0YV9idWZfNDkxNTI6ICAgNDkxNTIsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwg ICAgNzM1NywgICAwLCAgIDANCnppb19idWZfNTMyNDg6ICAgICAgICA1MzI0OCwgICAgICAwLCAg ICAgICAxLCAgICAgICA0LCAgNDA0ODY5LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzUzMjQ4OiAg IDUzMjQ4LCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgIDU1OTAsICAgMCwgICAwDQp6aW9f YnVmXzU3MzQ0OiAgICAgICAgNTczNDQsICAgICAgMCwgICAgICAgNSwgICAgICAgMiwgIDYyMTky MywgICAwLCAgIDANCnppb19kYXRhX2J1Zl81NzM0NDogICA1NzM0NCwgICAgICAwLCAgICAgICAy LCAgICAgICAwLCAgICA0OTQzLCAgIDAsICAgMA0KemlvX2J1Zl82MTQ0MDogICAgICAgIDYxNDQw LCAgICAgIDAsICAgICAgIDEsICAgICAgIDIsICAxNTQ5MDEsICAgMCwgICAwDQp6aW9fZGF0YV9i dWZfNjE0NDA6ICAgNjE0NDAsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICA2MiwgICAw LCAgIDANCnppb19idWZfNjU1MzY6ICAgICAgICA2NTUzNiwgICAgICAwLCAgICAgIDE1LCAgICAg ICAzLCA2MDc4ODk1LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzY1NTM2OiAgIDY1NTM2LCAgICAg IDAsICA5MjQ4MjgsICAgICAgODYsIDcxMjE5NzQsICAgMCwgICAwDQp6aW9fYnVmXzY5NjMyOiAg ICAgICAgNjk2MzIsICAgICAgMCwgICAgICAgMCwgICAgICAgMiwgIDE0MTU3MCwgICAwLCAgIDAN Cnppb19kYXRhX2J1Zl82OTYzMjogICA2OTYzMiwgICAgICAwLCAgICAgICAxLCAgICAgICAwLCAg ICAgIDY4LCAgIDAsICAgMA0KemlvX2J1Zl83MzcyODogICAgICAgIDczNzI4LCAgICAgIDAsICAg ICAgIDAsICAgICAgIDIsICAyNzM0NjksICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNzM3Mjg6ICAg NzM3MjgsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICA2MiwgICAwLCAgIDANCnppb19i dWZfNzc4MjQ6ICAgICAgICA3NzgyNCwgICAgICAwLCAgICAgICAxLCAgICAgICAwLCAgMTI1Mjk2 LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzc3ODI0OiAgIDc3ODI0LCAgICAgIDAsICAgICAgIDAs ICAgICAgIDAsICAgICAgNjIsICAgMCwgICAwDQp6aW9fYnVmXzgxOTIwOiAgICAgICAgODE5MjAs ICAgICAgMCwgICAgICAgNCwgICAgICAgMywgIDMwNDU5MiwgICAwLCAgIDANCnppb19kYXRhX2J1 Zl84MTkyMDogICA4MTkyMCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgIDY0LCAgIDAs ICAgMA0KemlvX2J1Zl84NjAxNjogICAgICAgIDg2MDE2LCAgICAgIDAsICAgICAgIDEsICAgICAg IDAsICAxMTkyMDYsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfODYwMTY6ICAgODYwMTYsICAgICAg MCwgICAgICAgMCwgICAgICAgMCwgICAgICA2MiwgICAwLCAgIDANCnppb19idWZfOTAxMTI6ICAg ICAgICA5MDExMiwgICAgICAwLCAgICAgICAwLCAgICAgICAxLCAgMjE4NjQxLCAgIDAsICAgMA0K emlvX2RhdGFfYnVmXzkwMTEyOiAgIDkwMTEyLCAgICAgIDAsICAgICAgIDEsICAgICAgIDAsICAg ICAgNzAsICAgMCwgICAwDQp6aW9fYnVmXzk0MjA4OiAgICAgICAgOTQyMDgsICAgICAgMCwgICAg ICAgMSwgICAgICAgMiwgIDExNzA4NSwgICAwLCAgIDANCnppb19kYXRhX2J1Zl85NDIwODogICA5 NDIwOCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgIDU0LCAgIDAsICAgMA0KemlvX2J1 Zl85ODMwNDogICAgICAgIDk4MzA0LCAgICAgIDAsICAgICAgIDAsICAgICAgIDUsICAyNzYxNzgs ICAgMCwgICAwDQp6aW9fZGF0YV9idWZfOTgzMDQ6ICAgOTgzMDQsICAgICAgMCwgICAgICAgMSwg ICAgICAgMCwgICAgICA0MiwgICAwLCAgIDANCnppb19idWZfMTAyNDAwOiAgICAgIDEwMjQwMCwg ICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgMTEwMDg1LCAgIDAsICAgMA0KemlvX2RhdGFfYnVm XzEwMjQwMDogMTAyNDAwLCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgNDMsICAgMCwg ICAwDQp6aW9fYnVmXzEwNjQ5NjogICAgICAxMDY0OTYsICAgICAgMCwgICAgICAgMCwgICAgICAg MCwgIDE5MDQyNCwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8xMDY0OTY6IDEwNjQ5NiwgICAgICAw LCAgICAgICAwLCAgICAgICAwLCAgICAgIDQ1LCAgIDAsICAgMA0KemlvX2J1Zl8xMTA1OTI6ICAg ICAgMTEwNTkyLCAgICAgIDAsICAgICAgIDAsICAgICAgIDQsICAxMTI0NjIsICAgMCwgICAwDQp6 aW9fZGF0YV9idWZfMTEwNTkyOiAxMTA1OTIsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAg ICAzMywgICAwLCAgIDANCnppb19idWZfMTE0Njg4OiAgICAgIDExNDY4OCwgICAgICAwLCAgICAg ICAxLCAgICAgICAyLCAgMjMyMzg4LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzExNDY4ODogMTE0 Njg4LCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgMzMsICAgMCwgICAwDQp6aW9fYnVm XzExODc4NDogICAgICAxMTg3ODQsICAgICAgMCwgICAgICAgMSwgICAgICAgMSwgIDEwNjU3MSwg ICAwLCAgIDANCnppb19kYXRhX2J1Zl8xMTg3ODQ6IDExODc4NCwgICAgICAwLCAgICAgICAwLCAg ICAgICAwLCAgICAgIDE5LCAgIDAsICAgMA0KemlvX2J1Zl8xMjI4ODA6ICAgICAgMTIyODgwLCAg ICAgIDAsICAgICAgIDAsICAgICAgIDIsICAyNzU1NjEsICAgMCwgICAwDQp6aW9fZGF0YV9idWZf MTIyODgwOiAxMjI4ODAsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgOCwgICAwLCAg IDANCnppb19idWZfMTI2OTc2OiAgICAgIDEyNjk3NiwgICAgICAwLCAgICAgICAwLCAgICAgICAw LCAgMzExMTY4LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzEyNjk3NjogMTI2OTc2LCAgICAgIDAs ICAgICAgIDAsICAgICAgIDAsICAgICAgMTAsICAgMCwgICAwDQp6aW9fYnVmXzEzMTA3MjogICAg ICAxMzEwNzIsICAgICAgMCwgICAgNDE0MywgICAgICAxMCwgNjU0NDA2NCwgICAwLCAgIDANCnpp b19kYXRhX2J1Zl8xMzEwNzI6IDEzMTA3MiwgICAgICAwLCAgICAxODMyLCAgICAgICAwLCAgMjMw NTA3LCAgIDAsICAgMA0KbHo0X2N0eDogICAgICAgICAgICAgIDE2Mzg0LCAgICAgIDAsICAgICAg IDQsICAgICAgIDgsNTA1NTU2MTAsICAgMCwgICAwDQpzYV9jYWNoZTogICAgICAgICAgICAgICAg ODAsICAgICAgMCwgICAgMzg0MywgICAgIDgwNywgICAgNTQ3MSwgICAwLCAgIDANCmRub2RlX3Q6 ICAgICAgICAgICAgICAgIDc1MiwgICAgICAwLCAgIDM4NjgzLCAgICAgNDQ3LCA1OTI3NjQ4LCAg IDAsICAgMA0KZG11X2J1Zl9pbXBsX3Q6ICAgICAgICAgMjMyLCAgICAgIDAsIDQ2MTYxNDYsIDEw MjE4MTksMjY0Mzg0NzIsICAgMCwgICAwDQphcmNfYnVmX2hkcl90OiAgICAgICAgICAyMTYsICAg ICAgMCwgOTY0NTU3NiwgICA0NzI2MiwxNjU1MzI3MCwgICAwLCAgIDANCmFyY19idWZfdDogICAg ICAgICAgICAgICA3MiwgICAgICAwLCA0NzY1MTI5LCAyMTAwNDY2LDIwMDY1NDQ4LCAgIDAsICAg MA0KemlsX2x3Yl9jYWNoZTogICAgICAgICAgMTkyLCAgICAgIDAsICAgMjA1MjgsICAgICA0MzIs ICA4NTMzMzMsICAgMCwgICAwDQp6ZnNfem5vZGVfY2FjaGU6ICAgICAgICAzNjgsICAgICAgMCwg ICAgMzg0MywgICAgIDE5NywgICAgNTQ3MSwgICAwLCAgIDANCk1vdW50cG9pbnRzOiAgICAgICAg ICAgIDgxNiwgICAgICAwLCAgICAgIDIzLCAgICAgMTQ3LCAgICAgIDIzLCAgIDAsICAgMA0Ka3Np Z2luZm86ICAgICAgICAgICAgICAgMTEyLCAgICAgIDAsICAgICAzNzQsICAgIDEyMzYsICAgIDE4 ODcsICAgMCwgICAwDQppdGltZXI6ICAgICAgICAgICAgICAgICAzNTIsICAgICAgMCwgICAgICAg MCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCktOT1RFOiAgICAgICAgICAgICAgICAgIDEy OCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICA4NzQ2LCAgIDAsICAgMA0Kc29ja2V0OiAg ICAgICAgICAgICAgICAgNjk2LCAzMTQ0NDg1LCAgICAgIDY5LCAgICAgMzM2LCAgIDIzNzkxLCAg IDAsICAgMA0KaXBxOiAgICAgICAgICAgICAgICAgICAgIDU2LCAxOTE0ODcsICAgICAgIDAsICAg ICAgIDAsICAgICAgIDAsICAgMCwgICAwDQp1ZHBfaW5wY2I6ICAgICAgICAgICAgICAzOTIsIDMx NDQ0OTAsICAgICAgMTEsICAgICA3NDksICAgIDkwNTAsICAgMCwgICAwDQp1ZHBjYjogICAgICAg ICAgICAgICAgICAgMTYsIDMxNDQ1MjgsICAgICAgMTEsICAgIDYyNjQsICAgIDkwNTAsICAgMCwg ICAwDQp0Y3BfaW5wY2I6ICAgICAgICAgICAgICAzOTIsIDMxNDQ0OTAsICAgICAgNDUsICAgICA2 NTUsICAgIDMwMTYsICAgMCwgICAwDQp0Y3BjYjogICAgICAgICAgICAgICAgIDEwMjQsIDMxNDQ0 ODgsICAgICAgNDUsICAgICAyNjcsICAgIDMwMTYsICAgMCwgICAwDQp0Y3B0dzogICAgICAgICAg ICAgICAgICAgODgsICAyNzgxMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDAN CnN5bmNhY2hlOiAgICAgICAgICAgICAgIDE2MCwgIDE1Mzc1LCAgICAgICAwLCAgICAgICAwLCAg ICAgIDg3LCAgIDAsICAgMA0KaG9zdGNhY2hlOiAgICAgICAgICAgICAgMTM2LCAgMTUzNzAsICAg ICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQp0Y3ByZWFzczogICAgICAgICAgICAg ICAgNDAsIDM4MzAwMCwgICAgICAgMCwgICAgIDgwMCwgICAgICA1NywgICAwLCAgIDANCnNhY2to b2xlOiAgICAgICAgICAgICAgICAzMiwgICAgICAwLCAgICAgICAwLCAgICAxNjI1LCAgICAgMjA4 LCAgIDAsICAgMA0Kc2N0cF9lcDogICAgICAgICAgICAgICAxNDA4LCAzMTQ0NDg2LCAgICAgICAw LCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0Kc2N0cF9hc29jOiAgICAgICAgICAgICAyNDE2 LCAgNDAwMDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpzY3RwX2xhZGRy OiAgICAgICAgICAgICAgNDgsICA4MDAxMiwgICAgICAgMCwgICAgICAgMCwgICAgICAgNCwgICAw LCAgIDANCnNjdHBfcmFkZHI6ICAgICAgICAgICAgIDcyOCwgIDgwMDAwLCAgICAgICAwLCAgICAg ICAwLCAgICAgICAwLCAgIDAsICAgMA0Kc2N0cF9jaHVuazogICAgICAgICAgICAgMTM2LCA0MDAw MjYsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpzY3RwX3JlYWRxOiAgICAg ICAgICAgICAxMDQsIDQwMDAyNiwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDAN CnNjdHBfc3RyZWFtX21zZ19vdXQ6ICAgIDEwNCwgNDAwMDI2LCAgICAgICAwLCAgICAgICAwLCAg ICAgICAwLCAgIDAsICAgMA0Kc2N0cF9hc2NvbmY6ICAgICAgICAgICAgIDQwLCA0MDAwMDAsICAg ICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpzY3RwX2FzY29uZl9hY2s6ICAgICAg ICAgNDgsIDQwMDA2MCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnVkcGxp dGVfaW5wY2I6ICAgICAgICAgIDM5MiwgMzE0NDQ5MCwgICAgICAgMCwgICAgICAgMCwgICAgICAg MCwgICAwLCAgIDANCnJpcGNiOiAgICAgICAgICAgICAgICAgIDM5MiwgMzE0NDQ5MCwgICAgICAg MCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnVucGNiOiAgICAgICAgICAgICAgICAgIDI0 MCwgMzE0NDQ5NiwgICAgICAxMywgICAgIDE0NywgICAxMTcyMCwgICAwLCAgIDANCnJ0ZW50cnk6 ICAgICAgICAgICAgICAgIDIwMCwgICAgICAwLCAgICAgIDE1LCAgICAgMTY1LCAgICAgIDE1LCAg IDAsICAgMA0Kc2VsZmQ6ICAgICAgICAgICAgICAgICAgIDU2LCAgICAgIDAsICAgICA0NjgsICAg IDI3OTgsIDI1ODc3MTMsICAgMCwgICAwDQpTV0FQTUVUQTogICAgICAgICAgICAgICAyODgsIDEy MjU0MzA3LCAgICAgICA0LCAgICAgICA5LCAgICAgICA0LCAgIDAsICAgMA0KaWNsX2Nvbm46ICAg ICAgICAgICAgICAgMTUyLCAgICAgIDAsICAgICAgMjAsICAgICAyNjYsICAgICAgMjAsICAgMCwg ICAwDQppY2xfcGR1OiAgICAgICAgICAgICAgICAgODAsICAgICAgMCwgICAgICA3OCwgICAgMzAy MiwxODkwNzY0MiwgICAwLCAgIDANCmlzY3NpX291dHN0YW5kaW5nOiAgICAgICA0MCwgICAgICAw LCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KY2ZpX2x1bjogICAgICAgICAg ICAgICAgMzQ0LCAgICAgIDAsICAgICAgMjQsICAgICAxMDgsICAgICAgMjQsICAgMCwgICAwDQpj ZmlfbWV0YXRhc2s6ICAgICAgICAgICAzMjAsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAg ICAgMCwgICAwLCAgIDANCmNmaXNjc2lfZGF0YV93YWl0OiAgICAgICA1NiwgICAgICAwLCAgICAg ICAwLCAgICAgICAwLCA2NjU2NDA0LCAgIDAsICAgMA0KYmVpbzogICAgICAgICAgICAgICAgICAg MzYwLCAgICAgIDAsICAgICAgNDAsICAgICAzMjMsIDgyNDUxNTQsICAgMCwgICAwDQpjYmxrMDog ICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAyOSwg ICAwLCAgIDANCmNibGsxOiAgICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAg ICAgICAwLCAgICAxMDI4LCAgIDAsICAgMA0KY2JsazI6ICAgICAgICAgICAgICAgMTMxMDcyLCAg ICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgMjksICAgMCwgICAwDQpjYmxrMzogICAgICAg ICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAyOSwgICAwLCAg IDANCmNibGs0OiAgICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgICAw LCAgICAgIDI5LCAgIDAsICAgMA0KY2JsazU6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAs ICAgICAgIDAsICAgICAgIDAsICAgICAgOTQsICAgMCwgICAwDQpjYmxrNjogICAgICAgICAgICAg ICAxMzEwNzIsICAgICAgMCwgICAgICAgMSwgICAgICAgMCwgMjIxODE2NywgICAwLCAgIDANCmNi bGs3OiAgICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAg IDIzLCAgIDAsICAgMA0KY2Jsazg6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAg IDAsICAgICAgIDAsICAgMjk1NDcsICAgMCwgICAwDQpjYmxrOTogICAgICAgICAgICAgICAxMzEw NzIsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCmNibGsxMDog ICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAg IDAsICAgMA0KY2JsazExOiAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAg ICAgIDAsICAgMjAzMjQsICAgMCwgICAwDQpjYmxrMTI6ICAgICAgICAgICAgICAxMzEwNzIsICAg ICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICA5NCwgICAwLCAgIDANCmNibGsxMzogICAgICAg ICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAg MA0KY2JsazE0OiAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgIDAs ICAgICAgIDAsICAgMCwgICAwDQpjYmxrMTU6ICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwg ICAgICAxMSwgICAgICAgMCwgIDMwNjE2NywgICAwLCAgIDANCmNibGsxNjogICAgICAgICAgICAg IDEzMTA3MiwgICAgICAwLCAgICAgIDEwLCAgICAgICAwLCAgNjM1NTA1LCAgIDAsICAgMA0KY2Js azE3OiAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgMTEsICAgICAgIDAsIDIxNDA3 ODUsICAgMCwgICAwDQpjYmxrMTg6ICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAg MCwgICAgICAgMCwgICAgICA5NCwgICAwLCAgIDANCmNibGsxOTogICAgICAgICAgICAgIDEzMTA3 MiwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgIDk0LCAgIDAsICAgMA0KY2JsazIwOiAg ICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDIsICAgICAgIDAsIDExNzM3MDYsICAg MCwgICAwDQpjYmxrMjE6ICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMSwgICAg ICAgMCwgMTI3NTI4NSwgICAwLCAgIDANCmNibGsyMjogICAgICAgICAgICAgIDEzMTA3MiwgICAg ICAwLCAgICAgICAyLCAgICAgICAwLCAgIDM1MTgyLCAgIDAsICAgMA0KY2JsazIzOiAgICAgICAg ICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDIsICAgICAgIDAsICA0MTg5MDUsICAgMCwgICAw DQo= --001a1145eae834d0fc051ac04b07 Content-Type: text/plain; charset=US-ASCII; name="2015-07-09 - San2.txt" Content-Disposition: attachment; filename="2015-07-09 - San2.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_ic1uffuv3 DQp6aW9fZGF0YV9idWZfOTQyMDg6ICAgOTQyMDgsICAgICAgMCwgICAgICAgMCwgICAgICAyNCwg ICAgIDYyMywgICAwLCAgIDANCnppb19idWZfOTgzMDQ6ICAgICAgICA5ODMwNCwgICAgICAwLCAg ICAgICAwLCAgICAgIDIxLCAyNDU1Mzg0LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzk4MzA0OiAg IDk4MzA0LCAgICAgIDAsICAgICAgIDAsICAgICAgMjEsICAgICA2MTksICAgMCwgICAwDQp6aW9f YnVmXzEwMjQwMDogICAgICAxMDI0MDAsICAgICAgMCwgICAgICAgMCwgICAgICAyNCwgMTgyODUz MiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8xMDI0MDA6IDEwMjQwMCwgICAgICAwLCAgICAgICAw LCAgICAgIDIxLCAgICAgNjA5LCAgIDAsICAgMA0KemlvX2J1Zl8xMDY0OTY6ICAgICAgMTA2NDk2 LCAgICAgIDAsICAgICAgIDAsICAgICAgMjIsIDE5NTQ2MDEsICAgMCwgICAwDQp6aW9fZGF0YV9i dWZfMTA2NDk2OiAxMDY0OTYsICAgICAgMCwgICAgICAgMCwgICAgICAyMiwgICAgIDYwNiwgICAw LCAgIDANCnppb19idWZfMTEwNTkyOiAgICAgIDExMDU5MiwgICAgICAwLCAgICAgICAwLCAgICAg IDIzLCAxOTg1MDk3LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzExMDU5MjogMTEwNTkyLCAgICAg IDAsICAgICAgIDAsICAgICAgMTksICAgICA1NzUsICAgMCwgICAwDQp6aW9fYnVmXzExNDY4ODog ICAgICAxMTQ2ODgsICAgICAgMCwgICAgICAgMCwgICAgICA0MywgMjM3MjkwMiwgICAwLCAgIDAN Cnppb19kYXRhX2J1Zl8xMTQ2ODg6IDExNDY4OCwgICAgICAwLCAgICAgICAwLCAgICAgIDI1LCAg ICAgNTgxLCAgIDAsICAgMA0KemlvX2J1Zl8xMTg3ODQ6ICAgICAgMTE4Nzg0LCAgICAgIDAsICAg ICAgIDAsICAgICAgMTgsIDIwNTE3NDUsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTE4Nzg0OiAx MTg3ODQsICAgICAgMCwgICAgICAgMCwgICAgICAyNCwgICAgIDU4NCwgICAwLCAgIDANCnppb19i dWZfMTIyODgwOiAgICAgIDEyMjg4MCwgICAgICAwLCAgICAgICAwLCAgICAgIDE3LCAyNDkyNTYw LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzEyMjg4MDogMTIyODgwLCAgICAgIDAsICAgICAgIDAs ICAgICAgMjAsICAgICA1ODIsICAgMCwgICAwDQp6aW9fYnVmXzEyNjk3NjogICAgICAxMjY5NzYs ICAgICAgMCwgICAgICAgMCwgICAgICAyMiwgMzU4Mjg5MiwgICAwLCAgIDANCnppb19kYXRhX2J1 Zl8xMjY5NzY6IDEyNjk3NiwgICAgICAwLCAgICAgICAwLCAgICAgIDIxLCAgICAgNTgxLCAgIDAs ICAgMA0KemlvX2J1Zl8xMzEwNzI6ICAgICAgMTMxMDcyLCAgICAgIDAsICAgMTAxNjUsICAgICAy MjAsNjYwODk2OTIsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTMxMDcyOiAxMzEwNzIsICAgICAg MCwgICAgICAgMywgICAgICAyMiwgICAxNjY3NCwgICAwLCAgIDANCmx6NF9jdHg6ICAgICAgICAg ICAgICAxNjM4NCwgICAgICAwLCAgICAgICAwLCAgICAgIDM5LDQ2Njk0NjQwNCwgICAwLCAgIDAN CnNhX2NhY2hlOiAgICAgICAgICAgICAgICA4MCwgICAgICAwLCAgMjM0MTMxLCAgIDI5NjE5LCAx NTQ1NTcyLCAgIDAsICAgMA0KZG5vZGVfdDogICAgICAgICAgICAgICAgNzUyLCAgICAgIDAsICAz MDE3ODQsICAgMTAzNTEsIDE1NTc5NjUsICAgMCwgICAwDQpkbXVfYnVmX2ltcGxfdDogICAgICAg ICAyMzIsICAgICAgMCwgMjU1NTE5MCwgIDU1NTEzMCwyNDk0OTY4NjAsICAgMCwgICAwDQphcmNf YnVmX2hkcl90OiAgICAgICAgICAyMTYsICAgICAgMCwxNzU5MTA5MCwgICAyODc2OCwxNzIyMjg1 MDMsICAgMCwgICAwDQphcmNfYnVmX3Q6ICAgICAgICAgICAgICAgNzIsICAgICAgMCwgMjQ1MzY3 NiwgIDY4MDYwOSwxODY0NTgwNjIsICAgMCwgICAwDQp6aWxfbHdiX2NhY2hlOiAgICAgICAgICAx OTIsICAgICAgMCwgICAgIDQ4MCwgICAgMjM0MCwzMzYyNjIyNywgICAwLCAgIDANCnpmc196bm9k ZV9jYWNoZTogICAgICAgIDM2OCwgICAgICAwLCAgMjM0MTMxLCAgIDE4ODk5LCAxNTQ1NTcyLCAg IDAsICAgMA0KTW91bnRwb2ludHM6ICAgICAgICAgICAgODE2LCAgICAgIDAsICAgICAgMzcsICAg ICAxMjMsICAgICAgMzcsICAgMCwgICAwDQprc2lnaW5mbzogICAgICAgICAgICAgICAxMTIsICAg ICAgMCwgICAgIDIyOSwgICAgMjY0MSwgICAxNTA1NCwgICAwLCAgIDANCml0aW1lcjogICAgICAg ICAgICAgICAgIDM1MiwgICAgICAwLCAgICAgICAxLCAgICAgIDEwLCAgICAgICAxLCAgIDAsICAg MA0KS05PVEU6ICAgICAgICAgICAgICAgICAgMTI4LCAgICAgIDAsICAgICAgIDAsICAgIDIxMzks ICAxNTExNzYsICAgMCwgICAwDQpzb2NrZXQ6ICAgICAgICAgICAgICAgICA2OTYsIDE1NzE2MjAs ICAgICAgNzAsICAgICAyNzAsICAxODU2MDAsICAgMCwgICAwDQppcHE6ICAgICAgICAgICAgICAg ICAgICAgNTYsICA5NTcwOCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnVk cF9pbnBjYjogICAgICAgICAgICAgIDM5MiwgMTU3MTYyMCwgICAgICAxOCwgICAgIDUyMiwgICA4 NzY5MiwgICAwLCAgIDANCnVkcGNiOiAgICAgICAgICAgICAgICAgICAxNiwgMTU3MTc2MiwgICAg ICAxOCwgICAgNDc1MSwgICA4NzY5MiwgICAwLCAgIDANCnRjcF9pbnBjYjogICAgICAgICAgICAg IDM5MiwgMTU3MTYyMCwgICAgICAzNiwgICAgIDYxNCwgICA5MTUzOCwgICAwLCAgIDANCnRjcGNi OiAgICAgICAgICAgICAgICAgMTAyNCwgMTU3MTYyMCwgICAgICAzNiwgICAgIDI3MiwgICA5MTUz OCwgICAwLCAgIDANCnRjcHR3OiAgICAgICAgICAgICAgICAgICA4OCwgIDI3ODEwLCAgICAgICAw LCAgICAgNTQwLCAgICAgICA4LCAgIDAsICAgMA0Kc3luY2FjaGU6ICAgICAgICAgICAgICAgMTYw LCAgMTUzNzUsICAgICAgIDAsICAgICA1MDAsICAgODg1NzcsICAgMCwgICAwDQpob3N0Y2FjaGU6 ICAgICAgICAgICAgICAxMzYsICAxNTM3MCwgICAgICAgMSwgICAgIDIwMiwgICAgICAxMCwgICAw LCAgIDANCnRjcHJlYXNzOiAgICAgICAgICAgICAgICA0MCwgMTkxNDAwLCAgICAgICAwLCAgICAy MTAwLCAgICAzMzcwLCAgIDAsICAgMA0Kc2Fja2hvbGU6ICAgICAgICAgICAgICAgIDMyLCAgICAg IDAsICAgICAgIDAsICAgIDE4NzUsICAgIDUwNDYsICAgMCwgICAwDQpzY3RwX2VwOiAgICAgICAg ICAgICAgIDE0MDgsIDE1NzE2MjAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAw DQpzY3RwX2Fzb2M6ICAgICAgICAgICAgIDI0MTYsICA0MDAwMCwgICAgICAgMCwgICAgICAgMCwg ICAgICAgMCwgICAwLCAgIDANCnNjdHBfbGFkZHI6ICAgICAgICAgICAgICA0OCwgIDgwMDEyLCAg ICAgICAwLCAgICAgICAwLCAgICAgICA0LCAgIDAsICAgMA0Kc2N0cF9yYWRkcjogICAgICAgICAg ICAgNzI4LCAgODAwMDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpzY3Rw X2NodW5rOiAgICAgICAgICAgICAxMzYsIDQwMDAyNiwgICAgICAgMCwgICAgICAgMCwgICAgICAg MCwgICAwLCAgIDANCnNjdHBfcmVhZHE6ICAgICAgICAgICAgIDEwNCwgNDAwMDI2LCAgICAgICAw LCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0Kc2N0cF9zdHJlYW1fbXNnX291dDogICAgMTA0 LCA0MDAwMjYsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpzY3RwX2FzY29u ZjogICAgICAgICAgICAgNDAsIDQwMDAwMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAw LCAgIDANCnNjdHBfYXNjb25mX2FjazogICAgICAgICA0OCwgNDAwMDYwLCAgICAgICAwLCAgICAg ICAwLCAgICAgICAwLCAgIDAsICAgMA0KdWRwbGl0ZV9pbnBjYjogICAgICAgICAgMzkyLCAxNTcx NjIwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KcmlwY2I6ICAgICAgICAg ICAgICAgICAgMzkyLCAxNTcxNjIwLCAgICAgICAwLCAgICAgIDkwLCAgICAgICA1LCAgIDAsICAg MA0KdW5wY2I6ICAgICAgICAgICAgICAgICAgMjQwLCAxNTcxNjMyLCAgICAgIDE1LCAgICAgODY1 LCAgICA2MzYwLCAgIDAsICAgMA0KcnRlbnRyeTogICAgICAgICAgICAgICAgMjAwLCAgICAgIDAs ICAgICAgMTQsICAgICAxNDYsICAgICAgMTQsICAgMCwgICAwDQpzZWxmZDogICAgICAgICAgICAg ICAgICAgNTYsICAgICAgMCwgICAgIDM1NiwgICAgNTA0MCwxMTc5ODQzNCwgICAwLCAgIDANClNX QVBNRVRBOiAgICAgICAgICAgICAgIDI4OCwgNjEyMzgwNiwgICAgIDgxNiwgICAgIDc1NywgICA1 ODMyNCwgICAwLCAgIDANCmljbF9jb25uOiAgICAgICAgICAgICAgIDE1MiwgICAgICAwLCAgICAg IDEzLCAgICAgMjQ3LCAgICAgIDI4LCAgIDAsICAgMA0KaWNsX3BkdTogICAgICAgICAgICAgICAg IDgwLCAgICAgIDAsICAgICAgIDMsICAgIDI2NDcsMjQzMDQ0NTg2LCAgIDAsICAgMA0KaXNjc2lf b3V0c3RhbmRpbmc6ICAgICAgIDQwLCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAs ICAgMCwgICAwDQpjZmlfbHVuOiAgICAgICAgICAgICAgICAzNDQsICAgICAgMCwgICAgICAxNCwg ICAgICA4NSwgICAgICAxNSwgICAwLCAgIDANCmNmaV9tZXRhdGFzazogICAgICAgICAgIDMyMCwg ICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KY2Zpc2NzaV9kYXRh X3dhaXQ6ICAgICAgIDU2LCAgICAgIDAsICAgICAgIDAsICAgIDYzMTksMTAyNzY1MDYxLCAgIDAs ICAgMA0KYmVpbzogICAgICAgICAgICAgICAgICAgMzYwLCAgICAgIDAsICAgICAgIDIsICAgICA3 MzUsMTE0NzQzODA1LCAgIDAsICAgMA0KY2JsazA6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAg IDAsICAgICAgIDAsICAgICAgMTEsICAgNTg2NDgsICAgMCwgICAwDQpjYmxrMTogICAgICAgICAg ICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICAyMCwgICAgMTg0OSwgICAwLCAgIDAN CmNibGsyOiAgICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAxLCAgICAgIDIzLCA5 Mzk4MjA3LCAgIDAsICAgMA0KY2JsazM6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAg ICAgIDAsICAgICAgMjUsMTM0NDcxOTYsICAgMCwgICAwDQpjYmxrNDogICAgICAgICAgICAgICAx MzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCmNibGs1 OiAgICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDI5LDIxNjUwMjAz LCAgIDAsICAgMA0KY2JsazY6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAs ICAgICAgNTIsNDM1NjgwOTEsICAgMCwgICAwDQpjYmxrNzogICAgICAgICAgICAgICAxMzEwNzIs ICAgICAgMCwgICAgICAgMSwgICAgICAyMSwxMDc2OTc0OSwgICAwLCAgIDANCmNibGs4OiAgICAg ICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDIzLCAyMTM4NzE1LCAgIDAs ICAgMA0KY2Jsazk6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAg MjQsMTE0MTE2NDAsICAgMCwgICAwDQpjYmxrMTA6ICAgICAgICAgICAgICAxMzEwNzIsICAgICAg MCwgICAgICAgMCwgICAgICAzNCwgMjY4MzAyMCwgICAwLCAgIDANCmNibGsxMTogICAgICAgICAg ICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDI3LCAgIDM5MjEwLCAgIDAsICAgMA0K Y2JsazEyOiAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgMjIsICA0 MDQ3NDMsICAgMCwgICAwDQpjYmxrMTM6ICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAg ICAgMCwgICAgICAyMiwgICAgMjU5MiwgICAwLCAgIDANCg0KIHByb2NzICAgICAgbWVtb3J5ICAg ICAgcGFnZSAgICAgICAgICAgICAgICAgICAgZGlza3MgICAgIGZhdWx0cyAgICAgICAgIGNwdQ0K IHIgYiB3ICAgICBhdm0gICAgZnJlICAgZmx0ICByZSAgcGkgIHBvICAgIGZyICBzciBtZjAgbWYx ICAgaW4gICBzeSAgIGNzIHVzIHN5IGlkDQogMCAwIDYgICAxMzA3TSAgMjUxMU0gICA2OTYgICAw ICAgMCAgIDAgIDE3MTIgICAzICAgMCAgIDAgMzQxMiAxNzQxIDc3MDkgIDAgIDMgOTcNCjE4OTk1 LzEyNTA1LzMxNTAwIG1idWZzIGluIHVzZSAoY3VycmVudC9jYWNoZS90b3RhbCkNCjE4NDI0Lzk2 MzYvMjgwNjAvMzA2MTkwMCBtYnVmIGNsdXN0ZXJzIGluIHVzZSAoY3VycmVudC9jYWNoZS90b3Rh bC9tYXgpDQoxODQyNC82MjMxIG1idWYrY2x1c3RlcnMgb3V0IG9mIHBhY2tldCBzZWNvbmRhcnkg em9uZSBpbiB1c2UgKGN1cnJlbnQvY2FjaGUpDQozNC8xOTMvMjI3LzE1MzA5NTAgNGsgKHBhZ2Ug c2l6ZSkganVtYm8gY2x1c3RlcnMgaW4gdXNlIChjdXJyZW50L2NhY2hlL3RvdGFsL21heCkNCjAv MC8wLzQ1MzYxNCA5ayBqdW1ibyBjbHVzdGVycyBpbiB1c2UgKGN1cnJlbnQvY2FjaGUvdG90YWwv bWF4KQ0KMC8wLzAvMjU1MTU4IDE2ayBqdW1ibyBjbHVzdGVycyBpbiB1c2UgKGN1cnJlbnQvY2Fj aGUvdG90YWwvbWF4KQ0KNDE3MzJLLzIzMTcwSy82NDkwM0sgYnl0ZXMgYWxsb2NhdGVkIHRvIG5l dHdvcmsgKGN1cnJlbnQvY2FjaGUvdG90YWwpDQowLzAvMCByZXF1ZXN0cyBmb3IgbWJ1ZnMgZGVu aWVkIChtYnVmcy9jbHVzdGVycy9tYnVmK2NsdXN0ZXJzKQ0KMC8wLzAgcmVxdWVzdHMgZm9yIG1i dWZzIGRlbGF5ZWQgKG1idWZzL2NsdXN0ZXJzL21idWYrY2x1c3RlcnMpDQowLzAvMCByZXF1ZXN0 cyBmb3IganVtYm8gY2x1c3RlcnMgZGVsYXllZCAoNGsvOWsvMTZrKQ0KMC8wLzAgcmVxdWVzdHMg Zm9yIGp1bWJvIGNsdXN0ZXJzIGRlbmllZCAoNGsvOWsvMTZrKQ0KMCByZXF1ZXN0cyBmb3Igc2Zi dWZzIGRlbmllZA0KMCByZXF1ZXN0cyBmb3Igc2ZidWZzIGRlbGF5ZWQNCjAgcmVxdWVzdHMgZm9y IEkvTyBpbml0aWF0ZWQgYnkgc2VuZGZpbGUNCklURU0gICAgICAgICAgICAgICAgICAgU0laRSAg TElNSVQgICAgIFVTRUQgICAgIEZSRUUgICAgICBSRVEgRkFJTCBTTEVFUA0KDQpVTUEgS2Vnczog ICAgICAgICAgICAgICAzODQsICAgICAgMCwgICAgIDIyMiwgICAgICAgOCwgICAgIDIyMywgICAw LCAgIDANClVNQSBab25lczogICAgICAgICAgICAgMjY4OCwgICAgICAwLCAgICAgMjIyLCAgICAg ICAwLCAgICAgMjIzLCAgIDAsICAgMA0KVU1BIFNsYWJzOiAgICAgICAgICAgICAgIDgwLCAgICAg IDAsIDI2MzgzMTcsICA1NDA3ODMsNjcyNDg0NDIsICAgMCwgICAwDQpVTUEgUkNudFNsYWJzOiAg ICAgICAgICAgODgsICAgICAgMCwgICAxNDI1NywgICAgMjkzMywgIDMyMzQ4OSwgICAwLCAgIDAN ClVNQSBIYXNoOiAgICAgICAgICAgICAgIDI1NiwgICAgICAwLCAgICAgIDIzLCAgICAgIDUyLCAg ICAgIDk2LCAgIDAsICAgMA0KNCBCdWNrZXQ6ICAgICAgICAgICAgICAgIDMyLCAgICAgIDAsICAg IDI4ODUsICAgIDg2MTUsNjY0ODE1OTUsMzM3NiwgICAwDQo2IEJ1Y2tldDogICAgICAgICAgICAg ICAgNDgsICAgICAgMCwgICAgIDI3MSwgICAgNjk1MCwxODgxMDA1MSwgICAwLCAgIDANCjggQnVj a2V0OiAgICAgICAgICAgICAgICA2NCwgICAgICAwLCAgICAgNTE0LCAgICA2MDU4LCA5MTQzODIw LCAgMTEsICAgMA0KMTIgQnVja2V0OiAgICAgICAgICAgICAgIDk2LCAgICAgIDAsICAgICA3MDQs ICAgIDM4ODgsMjA1MDg1MjksMTIxMDYyLCAgIDANCjE2IEJ1Y2tldDogICAgICAgICAgICAgIDEy OCwgICAgICAwLCAgICAgMzQ2LCAgICA1NzYxLDE0OTg5MjQ3LDY2MjI5LCAgIDANCjMyIEJ1Y2tl dDogICAgICAgICAgICAgIDI1NiwgICAgICAwLCAgICAgMzQzLCAgICAyMjUyLDE2NDA5OTMyLCAg NTEsICAgMA0KNjQgQnVja2V0OiAgICAgICAgICAgICAgNTEyLCAgICAgIDAsICAgIDQxMjIsICAg ICAyODYsMTAxMjkxMzgsODUxODAzLCAgIDANCjEyOCBCdWNrZXQ6ICAgICAgICAgICAgMTAyNCwg ICAgICAwLCAgICAgNDMzLCAgICAgODAzLDEzODE0MjgxLCA5MDUsICAgMA0KMjU2IEJ1Y2tldDog ICAgICAgICAgICAyMDQ4LCAgICAgIDAsICAgIDMyNzcsICAgIDE1ODEsODg1NjE5MjEsMTA3NzUs ICAgMA0Kdm1lbSBidGFnOiAgICAgICAgICAgICAgIDU2LCAgICAgIDAsICA3NjM2NzMsICAgODU0 MTYsIDQxNjI0MDAsNjA1MywgICAwDQpWTSBPQkpFQ1Q6ICAgICAgICAgICAgICAyNTYsICAgICAg MCwgICA2MTI4OCwgICAgMTI5MiwgNjcwMjY3MSwgICAwLCAgIDANClJBRElYIE5PREU6ICAgICAg ICAgICAgIDE0NCwgICAgICAwLCAgNTY0NTE0LCAgIDU2NzI5LDM0MTI4ODI4LCAgIDAsICAgMA0K TUFQOiAgICAgICAgICAgICAgICAgICAgMjQwLCAgICAgIDAsICAgICAgIDMsICAgICAgNjEsICAg ICAgIDMsICAgMCwgICAwDQpLTUFQIEVOVFJZOiAgICAgICAgICAgICAxMjgsICAgICAgMCwgICAg ICAgOCwgICAgIDExNiwgICAgICAgOCwgICAwLCAgIDANCk1BUCBFTlRSWTogICAgICAgICAgICAg IDEyOCwgICAgICAwLCAgICAxMjgxLCAgICAyOTk3LDE4Nzk2NzI2LCAgIDAsICAgMA0KVk1TUEFD RTogICAgICAgICAgICAgICAgNDQ4LCAgICAgIDAsICAgICAgMzcsICAgICA1ODQsICA0NTg4Mjgs ICAgMCwgICAwDQpmYWtlcGc6ICAgICAgICAgICAgICAgICAxMDQsICAgICAgMCwgICAgICAgMCwg ICAgIDI2NiwgICAgICA1MCwgICAwLCAgIDANCm10X3pvbmU6ICAgICAgICAgICAgICAgNDExMiwg ICAgICAwLCAgICAgMzY5LCAgICAgICAwLCAgICAgMzY5LCAgIDAsICAgMA0KMTY6ICAgICAgICAg ICAgICAgICAgICAgIDE2LCAgICAgIDAsICAxMDkzNjgsICAgIDYwOTIsNDA4MzcyMTE2LCAgIDAs ICAgMA0KMzI6ICAgICAgICAgICAgICAgICAgICAgIDMyLCAgICAgIDAsMzAyNDg5MzcsICAgNjI0 MzgsODg1MzM3MTg2LCAgIDAsICAgMA0KNjQ6ICAgICAgICAgICAgICAgICAgICAgIDY0LCAgICAg IDAsIDE2NDA3ODMsICAzNDc3NDMsMzQyOTM3NTY3NSwgICAwLCAgIDANCjEyODogICAgICAgICAg ICAgICAgICAgIDEyOCwgICAgICAwLCAxMDU2NTAxLCAxMzA2OTA4LDMxODcyMDU5ODksICAgMCwg ICAwDQoyNTY6ICAgICAgICAgICAgICAgICAgICAyNTYsICAgICAgMCwgIDUxMzY4OSwgICAgMjUz NiwyMjk0MzE3NTA2LCAgIDAsICAgMA0KNTEyOiAgICAgICAgICAgICAgICAgICAgNTEyLCAgICAg IDAsICAxNjU0MTEsICAyMzIwNTMsNjM5Nzk3NTk3LCAgIDAsICAgMA0KMTAyNDogICAgICAgICAg ICAgICAgICAxMDI0LCAgICAgIDAsICAgMTU5ODksICAgICAyODcsMjAyMzU4NTAsICAgMCwgICAw DQoyMDQ4OiAgICAgICAgICAgICAgICAgIDIwNDgsICAgICAgMCwgICA2MzAwNSwgICAgMTQzMyw0 ODM3MzAzODksICAgMCwgICAwDQo0MDk2OiAgICAgICAgICAgICAgICAgIDQwOTYsICAgICAgMCwg ICAxMzAyOSwgICAgICA1NCw0MTg2NDk2OSwgICAwLCAgIDANCjgxOTI6ICAgICAgICAgICAgICAg ICAgODE5MiwgICAgICAwLCAgICAgIDI2LCAgICAgIDIxLCAgNDUxNzY0LCAgIDAsICAgMA0KMTYz ODQ6ICAgICAgICAgICAgICAgIDE2Mzg0LCAgICAgIDAsICAgICAgMzUsICAgICAgMTcsICAgIDI3 NjYsICAgMCwgICAwDQozMjc2ODogICAgICAgICAgICAgICAgMzI3NjgsICAgICAgMCwgICAgICAz OSwgICAgICAxNiwgICAxNzgxMCwgICAwLCAgIDANCjY1NTM2OiAgICAgICAgICAgICAgICA2NTUz NiwgICAgICAwLCAgICAgIDg2LCAgICAgIDIwLCAgMzA0NzM0LCAgIDAsICAgMA0KNjQgcGNwdTog ICAgICAgICAgICAgICAgICA4LCAgICAgIDAsICAgIDE2NTQsICAgICA2NTAsICAgIDE4NzQsICAg MCwgICAwDQpTTEVFUFFVRVVFOiAgICAgICAgICAgICAgODAsICAgICAgMCwgICAgMTI2MSwgICAg MTAzMywgICAgMTM0MiwgICAwLCAgIDANCkZpbGVzOiAgICAgICAgICAgICAgICAgICA4MCwgICAg ICAwLCAgICAgMTIxLCAgICAyMjc5LCA0MDE3MDc0LCAgIDAsICAgMA0KVFVSTlNUSUxFOiAgICAg ICAgICAgICAgMTM2LCAgICAgIDAsICAgIDEyNjEsICAgICA1MTksICAgIDEzNDIsICAgMCwgICAw DQpybF9lbnRyeTogICAgICAgICAgICAgICAgNDAsICAgICAgMCwgICAgIDE5OSwgICAgMjUwMSwg ICAgIDE5OSwgICAwLCAgIDANCnVtdHggcGk6ICAgICAgICAgICAgICAgICA5NiwgICAgICAwLCAg ICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KTUFDIGxhYmVsczogICAgICAgICAg ICAgIDQwLCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpQUk9D OiAgICAgICAgICAgICAgICAgIDEyMTYsICAgICAgMCwgICAgICA1NSwgICAgIDE4NSwgIDQ1ODg1 OCwgICAwLCAgIDANClRIUkVBRDogICAgICAgICAgICAgICAgMTE2OCwgICAgICAwLCAgICAxMTk5 LCAgICAgIDYxLCAgICAxNzY3LCAgIDAsICAgMA0KY3B1c2V0OiAgICAgICAgICAgICAgICAgIDcy LCAgICAgIDAsICAgICA5NTksICAgIDEwNzYsICAgIDE1MjIsICAgMCwgICAwDQphdWRpdF9yZWNv cmQ6ICAgICAgICAgIDEyNDgsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAw LCAgIDANCm1idWZfcGFja2V0OiAgICAgICAgICAgIDI1NiwgMTk1OTYxNjUsICAgMTg0MjMsICAg IDYyMzIsMjY3NzEzNzU5OCwgICAwLCAgIDANCm1idWY6ICAgICAgICAgICAgICAgICAgIDI1Niwg MTk1OTYxNjUsICAgICA1MzUsICAgIDYzMTAsMjc4MDYxMjcyOSwgICAwLCAgIDANCm1idWZfY2x1 c3RlcjogICAgICAgICAgMjA0OCwgMzA2MTkwMCwgICAyNDY1NSwgICAgMzQwNSw1NDg2Mjc2ODgs ICAgMCwgICAwDQptYnVmX2p1bWJvX3BhZ2U6ICAgICAgIDQwOTYsIDE1MzA5NTAsICAgICAgIDgs ICAgICAyMTksMTQ0MjI3MjI1LCAgIDAsICAgMA0KbWJ1Zl9qdW1ib185azogICAgICAgICA5MjE2 LCA0NTM2MTQsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQptYnVmX2p1bWJv XzE2azogICAgICAgMTYzODQsIDI1NTE1OCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAw LCAgIDANCm1idWZfZXh0X3JlZmNudDogICAgICAgICAgNCwgICAgICAwLCAgICAgICAwLCAgICAg ICAwLCAgICAgICAwLCAgIDAsICAgMA0KZ19iaW86ICAgICAgICAgICAgICAgICAgMjQ4LCAgICAg IDAsICAgICAgNDQsICAgIDM0NDQsMTkzMTg5MDQzOSwgICAwLCAgIDANCnR0eWlucTogICAgICAg ICAgICAgICAgIDE2MCwgICAgICAwLCAgICAgMjQwLCAgICAgMjM1LCAgICAgODI1LCAgIDAsICAg MA0KdHR5b3V0cTogICAgICAgICAgICAgICAgMjU2LCAgICAgIDAsICAgICAxMjYsICAgICAxMTQs ICAgICA0MzgsICAgMCwgICAwDQpETUFSX01BUF9FTlRSWTogICAgICAgICAxMjAsICAgICAgMCwg ICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCmF0YV9yZXF1ZXN0OiAgICAgICAg ICAgIDMzNiwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KdnRu ZXRfdHhfaGRyOiAgICAgICAgICAgIDI0LCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAg IDAsICAgMCwgICAwDQpGUFVfc2F2ZV9hcmVhOiAgICAgICAgICA1MTIsICAgICAgMCwgICAgICAg MCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnRhc2txX3pvbmU6ICAgICAgICAgICAgICA0 OCwgICAgICAwLCAgICAgICAwLCAgICA2MjI1LCA1OTgxODMxLCAgIDAsICAgMA0KVk5PREU6ICAg ICAgICAgICAgICAgICAgNDcyLCAgICAgIDAsICAyMzQyMzEsICAgMTc5MjEsIDE1NDYwMzEsICAg MCwgICAwDQpWTk9ERVBPTEw6ICAgICAgICAgICAgICAxMTIsICAgICAgMCwgICAgICAgMCwgICAg ICAgMCwgICAgICAgMCwgICAwLCAgIDANCkJVRiBUUklFOiAgICAgICAgICAgICAgIDE0NCwgICAg ICAwLCAgICAgICAwLCAgMTA1OTQ4LCAgICAgICAwLCAgIDAsICAgMA0KUyBWRlMgQ2FjaGU6ICAg ICAgICAgICAgMTA4LCAgICAgIDAsICAxMzg0NjQsICAgNjM2NjEsIDE4MjUwMDYsICAgMCwgICAw DQpTVFMgVkZTIENhY2hlOiAgICAgICAgICAxNDgsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwg ICAgICAgMCwgICAwLCAgIDANCkwgVkZTIENhY2hlOiAgICAgICAgICAgIDMyOCwgICAgICAwLCAg IDU2Nzk3LCAgICAgNTI3LCAgMzEyNDcwLCAgIDAsICAgMA0KTFRTIFZGUyBDYWNoZTogICAgICAg ICAgMzY4LCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpOQU1F STogICAgICAgICAgICAgICAgIDEwMjQsICAgICAgMCwgICAgICAgMCwgICAgIDI3NiwxNDI3MDk4 NiwgICAwLCAgIDANCk5DTE5PREU6ICAgICAgICAgICAgICAgIDUyOCwgICAgICAwLCAgICAgICAw LCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KRElSSEFTSDogICAgICAgICAgICAgICAxMDI0 LCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpwaXBlOiAgICAg ICAgICAgICAgICAgICA3NDQsICAgICAgMCwgICAgICAgNCwgICAgIDI1NiwgICAyMTE2NSwgICAw LCAgIDANCnByb2NkZXNjOiAgICAgICAgICAgICAgIDEyOCwgICAgICAwLCAgICAgICAwLCAgICAg ICAwLCAgICAgICAwLCAgIDAsICAgMA0KcmFuZ2Vfc2VnX2NhY2hlOiAgICAgICAgIDY0LCAgICAg IDAsIDE4OTE5MTIsIDExNTU0NTAsNTI0MTMwNzA1NCwgICAwLCAgIDANCnppb19jYWNoZTogICAg ICAgICAgICAgIDkyMCwgICAgICAwLCAgICA0NjIzLCAgMzA3MzQ1LDk0NzgzNTMyODUsICAgMCwg ICAwDQp6aW9fbGlua19jYWNoZTogICAgICAgICAgNDgsICAgICAgMCwgICAgNDY0NywgIDMwNzY4 MiwxMDg5MzE5MzY0NywgICAwLCAgIDANCnppb19idWZfNTEyOiAgICAgICAgICAgIDUxMiwgICAg ICAwLCAgMjUzNDQxLCAgIDMxNzY3LDE3MzM3MjU0LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzUx MjogICAgICAgNTEyLCAgICAgIDAsICAgICAgMTAsICAgICA2NTQsIDEyNzE2MzUsICAgMCwgICAw DQp6aW9fYnVmXzEwMjQ6ICAgICAgICAgIDEwMjQsICAgICAgMCwgICAgICAzNSwgICAgIDM1Nywg MjI4MTYyMywgICAwLCAgIDANCnppb19kYXRhX2J1Zl8xMDI0OiAgICAgMTAyNCwgICAgICAwLCAg ICAgICAxLCAgICAgODUxLCAgNDA1ODcyLCAgIDAsICAgMA0KemlvX2J1Zl8xNTM2OiAgICAgICAg ICAxNTM2LCAgICAgIDAsICAgICAgMzEsICAgICAxOTksIDEzMTczOTcsICAgMCwgICAwDQp6aW9f ZGF0YV9idWZfMTUzNjogICAgIDE1MzYsICAgICAgMCwgICAgICAgMCwgICAgIDE5NCwgIDUzOTgz MSwgICAwLCAgIDANCnppb19idWZfMjA0ODogICAgICAgICAgMjA0OCwgICAgICAwLCAgICAgIDcy LCAgICAgMTY4LCA1OTQyNzc3LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzIwNDg6ICAgICAyMDQ4 LCAgICAgIDAsICAgICAgIDAsICAgICAxMjAsICA3MTgyNDMsICAgMCwgICAwDQp6aW9fYnVmXzI1 NjA6ICAgICAgICAgIDI1NjAsICAgICAgMCwgICAgICAgMCwgICAgICA1OCwgMjY4MDg4MywgICAw LCAgIDANCnppb19kYXRhX2J1Zl8yNTYwOiAgICAgMjU2MCwgICAgICAwLCAgICAgICAxLCAgICAg IDYzLCAgNTYyMTY5LCAgIDAsICAgMA0KemlvX2J1Zl8zMDcyOiAgICAgICAgICAzMDcyLCAgICAg IDAsICAgICAgODcsICAgICAgNjAsIDIzMjIxNzcsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMzA3 MjogICAgIDMwNzIsICAgICAgMCwgICAgICAgMCwgICAgICA2MywgIDIwNzcwNiwgICAwLCAgIDAN Cnppb19idWZfMzU4NDogICAgICAgICAgMzU4NCwgICAgICAwLCAgICAgICAxLCAgICAgIDE5LCAg ODUxOTc5LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzM1ODQ6ICAgICAzNTg0LCAgICAgIDAsICAg ICAgIDEsICAgICAgMjAsICAxODcwNzgsICAgMCwgICAwDQp6aW9fYnVmXzQwOTY6ICAgICAgICAg IDQwOTYsICAgICAgMCwgMTgxNDg1OCwgICAgNDQ0OSwxMDM2NjA5NjI2LCAgIDAsICAgMA0Kemlv X2RhdGFfYnVmXzQwOTY6ICAgICA0MDk2LCAgICAgIDAsICAgNDA2OTgsICAgIDIxNTUsMjM3MjQw ODksICAgMCwgICAwDQp6aW9fYnVmXzUxMjA6ICAgICAgICAgIDUxMjAsICAgICAgMCwgICAgIDIw OSwgICAgICAyNCwgNTA1MTEzMSwgICAwLCAgIDANCnppb19kYXRhX2J1Zl81MTIwOiAgICAgNTEy MCwgICAgICAwLCAgICAgICAwLCAgICAgIDI0LCAgMjgxOTc4LCAgIDAsICAgMA0KemlvX2J1Zl82 MTQ0OiAgICAgICAgICA2MTQ0LCAgICAgIDAsICAgICAgIDAsICAgICAgMjksIDI5NTk0MjIsICAg MCwgICAwDQp6aW9fZGF0YV9idWZfNjE0NDogICAgIDYxNDQsICAgICAgMCwgICAgICAgMSwgICAg ICAyOCwgIDUyMjI4MywgICAwLCAgIDANCnppb19idWZfNzE2ODogICAgICAgICAgNzE2OCwgICAg ICAwLCAgICAgICAwLCAgICAgIDI3LCAyNzgzNzg4LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzcx Njg6ICAgICA3MTY4LCAgICAgIDAsICAgICAgIDAsICAgICAgMjYsICA0NDE1NDksICAgMCwgICAw DQp6aW9fYnVmXzgxOTI6ICAgICAgICAgIDgxOTIsICAgICAgMCwgICAgICAgMCwgICAgIDE1Niw5 NTg2NzYwNiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl84MTkyOiAgICAgODE5MiwgICAgICAwLCAg ICAgICAwLCAgICAgIDI0LCAgMzE2MjIwLCAgIDAsICAgMA0KemlvX2J1Zl8xMDI0MDogICAgICAg IDEwMjQwLCAgICAgIDAsICAgICAgIDAsICAgICAgMzYsIDQ5ODQ5MjMsICAgMCwgICAwDQp6aW9f ZGF0YV9idWZfMTAyNDA6ICAgMTAyNDAsICAgICAgMCwgICAgICAgMSwgICAgICAyNiwgIDU0OTky NiwgICAwLCAgIDANCnppb19idWZfMTIyODg6ICAgICAgICAxMjI4OCwgICAgICAwLCAgICAgICAw LCAgICAgIDY2LDI3OTc4ODY5LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzEyMjg4OiAgIDEyMjg4 LCAgICAgIDAsICAgICAgIDEsICAgICAgMjYsICA1MDY4MjMsICAgMCwgICAwDQp6aW9fYnVmXzE0 MzM2OiAgICAgICAgMTQzMzYsICAgICAgMCwgICAgICAgMCwgICAgICAyOSwgNDI3NDQyNywgICAw LCAgIDANCnppb19kYXRhX2J1Zl8xNDMzNjogICAxNDMzNiwgICAgICAwLCAgICAgICAwLCAgICAg IDI3LCAgNDE3Njc5LCAgIDAsICAgMA0KemlvX2J1Zl8xNjM4NDogICAgICAgIDE2Mzg0LCAgICAg IDAsICAyNDM5NzQsICAgICA2ODMsMTQyNjgyNjUyLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzE2 Mzg0OiAgIDE2Mzg0LCAgICAgIDAsICAgICA2NjgsICAgICAgMzIsICA3NzI0NjIsICAgMCwgICAw DQp6aW9fYnVmXzIwNDgwOiAgICAgICAgMjA0ODAsICAgICAgMCwgICAgICAgMCwgICAgICA1Nywy MTAwMTczMCwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8yMDQ4MDogICAyMDQ4MCwgICAgICAwLCAg ICAgICAwLCAgICAgIDI4LCAgNTIyNzc0LCAgIDAsICAgMA0KemlvX2J1Zl8yNDU3NjogICAgICAg IDI0NTc2LCAgICAgIDAsICAgICAgIDAsICAgICAgNTMsMjExMzc5OTAsICAgMCwgICAwDQp6aW9f ZGF0YV9idWZfMjQ1NzY6ICAgMjQ1NzYsICAgICAgMCwgICAgICAgMCwgICAgICAyNywgIDQzMDA5 MCwgICAwLCAgIDANCnppb19idWZfMjg2NzI6ICAgICAgICAyODY3MiwgICAgICAwLCAgICAgICAw LCAgICAgIDU1LDE5NTIyODQ5LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzI4NjcyOiAgIDI4Njcy LCAgICAgIDAsICAgICAgIDAsICAgICAgMjYsICA0MjM0NzksICAgMCwgICAwDQp6aW9fYnVmXzMy NzY4OiAgICAgICAgMzI3NjgsICAgICAgMCwgICAgICAgMCwgICAgIDI2NywyMDUxMTEyMywgICAw LCAgIDANCnppb19kYXRhX2J1Zl8zMjc2ODogICAzMjc2OCwgICAgICAwLCAgICAgMTQ0LCAgICAg IDgyLDI5MjkzNDM0LCAgIDAsICAgMA0KemlvX2J1Zl8zNjg2NDogICAgICAgIDM2ODY0LCAgICAg IDAsICAgICAgIDAsICAgICAgNDQsMTEwMzM4MDAsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMzY4 NjQ6ICAgMzY4NjQsICAgICAgMCwgICAgICAgMCwgICAgICAyOCwgIDM5OTA5MywgICAwLCAgIDAN Cnppb19idWZfNDA5NjA6ICAgICAgICA0MDk2MCwgICAgICAwLCAgICAgICAwLCAgICAgIDQyLDEw NTk2MDY1LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzQwOTYwOiAgIDQwOTYwLCAgICAgIDAsICAg ICAgIDAsICAgICAgMjgsICA0NjEwMTksICAgMCwgICAwDQp6aW9fYnVmXzQ1MDU2OiAgICAgICAg NDUwNTYsICAgICAgMCwgICAgICAgMCwgICAgICAzMiwgODk3ODEwMCwgICAwLCAgIDANCnppb19k YXRhX2J1Zl80NTA1NjogICA0NTA1NiwgICAgICAwLCAgICAgICAwLCAgICAgIDI5LCAgNDcyMjAy LCAgIDAsICAgMA0KemlvX2J1Zl80OTE1MjogICAgICAgIDQ5MTUyLCAgICAgIDAsICAgICAgIDAs ICAgICAgMjMsMTA0NjY0NjMsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNDkxNTI6ICAgNDkxNTIs ICAgICAgMCwgICAgICAgMCwgICAgICAyNCwgIDM5NDIzMSwgICAwLCAgIDANCnppb19idWZfNTMy NDg6ICAgICAgICA1MzI0OCwgICAgICAwLCAgICAgICAwLCAgICAgIDI0LCA3NjAzOTMwLCAgIDAs ICAgMA0KemlvX2RhdGFfYnVmXzUzMjQ4OiAgIDUzMjQ4LCAgICAgIDAsICAgICAgIDAsICAgICAg MjcsICAyOTg3MjIsICAgMCwgICAwDQp6aW9fYnVmXzU3MzQ0OiAgICAgICAgNTczNDQsICAgICAg MCwgICAgICAgMCwgICAgICA0NiwgNzUzOTI5OCwgICAwLCAgIDANCnppb19kYXRhX2J1Zl81NzM0 NDogICA1NzM0NCwgICAgICAwLCAgICAgICAwLCAgICAgIDI0LCAgMzcyNzE5LCAgIDAsICAgMA0K emlvX2J1Zl82MTQ0MDogICAgICAgIDYxNDQwLCAgICAgIDAsICAgICAgIDAsICAgICAgMTYsIDMz MTMyNDEsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNjE0NDA6ICAgNjE0NDAsICAgICAgMCwgICAg ICAgMCwgICAgICAyMywgICAgIDYyMywgICAwLCAgIDANCnppb19idWZfNjU1MzY6ICAgICAgICA2 NTUzNiwgICAgICAwLCAgICAgICAwLCAgICAxODkzLDk5MTM0MTA3LCAgIDAsICAgMA0KemlvX2Rh dGFfYnVmXzY1NTM2OiAgIDY1NTM2LCAgICAgIDAsICAzNzQ5NzAsICAgICAxODIsMTE2Mzk0NTQ1 LCAgIDAsICAgMA0KemlvX2J1Zl82OTYzMjogICAgICAgIDY5NjMyLCAgICAgIDAsICAgICAgIDAs ICAgICAgMTksIDIzNjg4MDMsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNjk2MzI6ICAgNjk2MzIs ICAgICAgMCwgICAgICAgMCwgICAgICAyMSwgICAgIDY3MiwgICAwLCAgIDANCnppb19idWZfNzM3 Mjg6ICAgICAgICA3MzcyOCwgICAgICAwLCAgICAgICAwLCAgICAgIDE4LCAyNDM1MzU0LCAgIDAs ICAgMA0KemlvX2RhdGFfYnVmXzczNzI4OiAgIDczNzI4LCAgICAgIDAsICAgICAgIDAsICAgICAg MjEsICAgICA2MjIsICAgMCwgICAwDQp6aW9fYnVmXzc3ODI0OiAgICAgICAgNzc4MjQsICAgICAg MCwgICAgICAgMCwgICAgICAxOCwgMTk3NDU3OSwgICAwLCAgIDANCnppb19kYXRhX2J1Zl83Nzgy NDogICA3NzgyNCwgICAgICAwLCAgICAgICAwLCAgICAgIDIyLCAgICAgNjI5LCAgIDAsICAgMA0K emlvX2J1Zl84MTkyMDogICAgICAgIDgxOTIwLCAgICAgIDAsICAgICAgIDAsICAgICAgMTksIDI0 MjE2MjIsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfODE5MjA6ICAgODE5MjAsICAgICAgMCwgICAg ICAgMCwgICAgICAyNCwgICAgIDYyMSwgICAwLCAgIDANCnppb19idWZfODYwMTY6ICAgICAgICA4 NjAxNiwgICAgICAwLCAgICAgICAwLCAgICAgIDE4LCAxODExNTIxLCAgIDAsICAgMA0KemlvX2Rh dGFfYnVmXzg2MDE2OiAgIDg2MDE2LCAgICAgIDAsICAgICAgIDAsICAgICAgMjEsICAgICA2MTUs ICAgMCwgICAwDQp6aW9fYnVmXzkwMTEyOiAgICAgICAgOTAxMTIsICAgICAgMCwgICAgICAgMCwg ICAgICAxOCwgMTk2NTY1NiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl85MDExMjogICA5MDExMiwg ICAgICAwLCAgICAgICAwLCAgICAgIDIxLCAgICAgNjk0LCAgIDAsICAgMA0KemlvX2J1Zl85NDIw ODogICAgICAgIDk0MjA4LCAgICAgIDAsICAgICAgIDAsICAgICAgMTksIDE3ODY5OTEsICAgMCwg ICAwDQp6aW9fZGF0YV9idWZfOTQyMDg6ICAgOTQyMDgsICAgICAgMCwgICAgICAgMCwgICAgICAy NCwgICAgIDYyMywgICAwLCAgIDANCnppb19idWZfOTgzMDQ6ICAgICAgICA5ODMwNCwgICAgICAw LCAgICAgICAwLCAgICAgIDIxLCAyNDU1Mzg2LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzk4MzA0 OiAgIDk4MzA0LCAgICAgIDAsICAgICAgIDAsICAgICAgMjEsICAgICA2MTksICAgMCwgICAwDQp6 aW9fYnVmXzEwMjQwMDogICAgICAxMDI0MDAsICAgICAgMCwgICAgICAgMCwgICAgICAyNCwgMTgy ODUzMiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8xMDI0MDA6IDEwMjQwMCwgICAgICAwLCAgICAg ICAwLCAgICAgIDIxLCAgICAgNjA5LCAgIDAsICAgMA0KemlvX2J1Zl8xMDY0OTY6ICAgICAgMTA2 NDk2LCAgICAgIDAsICAgICAgIDAsICAgICAgMjIsIDE5NTQ2MDYsICAgMCwgICAwDQp6aW9fZGF0 YV9idWZfMTA2NDk2OiAxMDY0OTYsICAgICAgMCwgICAgICAgMCwgICAgICAyMiwgICAgIDYwNiwg ICAwLCAgIDANCnppb19idWZfMTEwNTkyOiAgICAgIDExMDU5MiwgICAgICAwLCAgICAgICAwLCAg ICAgIDIzLCAxOTg1MTA0LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzExMDU5MjogMTEwNTkyLCAg ICAgIDAsICAgICAgIDAsICAgICAgMTksICAgICA1NzUsICAgMCwgICAwDQp6aW9fYnVmXzExNDY4 ODogICAgICAxMTQ2ODgsICAgICAgMCwgICAgICAgMCwgICAgICA0MywgMjM3MjkwNCwgICAwLCAg IDANCnppb19kYXRhX2J1Zl8xMTQ2ODg6IDExNDY4OCwgICAgICAwLCAgICAgICAwLCAgICAgIDI1 LCAgICAgNTgxLCAgIDAsICAgMA0KemlvX2J1Zl8xMTg3ODQ6ICAgICAgMTE4Nzg0LCAgICAgIDAs ICAgICAgIDAsICAgICAgMTgsIDIwNTE3NDYsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTE4Nzg0 OiAxMTg3ODQsICAgICAgMCwgICAgICAgMCwgICAgICAyNCwgICAgIDU4NCwgICAwLCAgIDANCnpp b19idWZfMTIyODgwOiAgICAgIDEyMjg4MCwgICAgICAwLCAgICAgICAwLCAgICAgIDE4LCAyNDky NTY4LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzEyMjg4MDogMTIyODgwLCAgICAgIDAsICAgICAg IDAsICAgICAgMjAsICAgICA1ODIsICAgMCwgICAwDQp6aW9fYnVmXzEyNjk3NjogICAgICAxMjY5 NzYsICAgICAgMCwgICAgICAgMCwgICAgICAyMiwgMzU4Mjg5NiwgICAwLCAgIDANCnppb19kYXRh X2J1Zl8xMjY5NzY6IDEyNjk3NiwgICAgICAwLCAgICAgICAwLCAgICAgIDIxLCAgICAgNTgxLCAg IDAsICAgMA0KemlvX2J1Zl8xMzEwNzI6ICAgICAgMTMxMDcyLCAgICAgIDAsICAgMTE3NTQsICAg ICAzNDMsNjYwOTQwNjEsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTMxMDcyOiAxMzEwNzIsICAg ICAgMCwgICAgICAgMywgICAgICAyMiwgICAxNjY3NCwgICAwLCAgIDANCmx6NF9jdHg6ICAgICAg ICAgICAgICAxNjM4NCwgICAgICAwLCAgICAgICAwLCAgICAgIDM5LDQ2Njk3MTAzMSwgICAwLCAg IDANCnNhX2NhY2hlOiAgICAgICAgICAgICAgICA4MCwgICAgICAwLCAgMjM0MTMxLCAgIDI5NjE5 LCAxNTQ1NTcyLCAgIDAsICAgMA0KZG5vZGVfdDogICAgICAgICAgICAgICAgNzUyLCAgICAgIDAs ICAzMDE3ODQsICAgMTAzNTEsIDE1NTc5NjUsICAgMCwgICAwDQpkbXVfYnVmX2ltcGxfdDogICAg ICAgICAyMzIsICAgICAgMCwgMjYwMDM3NSwgIDUwOTk0NSwyNDk1NTQ5MDUsICAgMCwgICAwDQph cmNfYnVmX2hkcl90OiAgICAgICAgICAyMTYsICAgICAgMCwxNzU5NzE4NywgICAyMjY3MSwxNzIy MzkzMzEsICAgMCwgICAwDQphcmNfYnVmX3Q6ICAgICAgICAgICAgICAgNzIsICAgICAgMCwgMjQ1 ODIyMSwgIDY3NjA2NCwxODY0NjkyODEsICAgMCwgICAwDQp6aWxfbHdiX2NhY2hlOiAgICAgICAg ICAxOTIsICAgICAgMCwgICAgMzk4OSwgICAgIDYxMSwzMzYyOTczNiwgICAwLCAgIDANCnpmc196 bm9kZV9jYWNoZTogICAgICAgIDM2OCwgICAgICAwLCAgMjM0MTMxLCAgIDE4ODk5LCAxNTQ1NTcy LCAgIDAsICAgMA0KTW91bnRwb2ludHM6ICAgICAgICAgICAgODE2LCAgICAgIDAsICAgICAgMzcs ICAgICAxMjMsICAgICAgMzcsICAgMCwgICAwDQprc2lnaW5mbzogICAgICAgICAgICAgICAxMTIs ICAgICAgMCwgICAgIDIyOSwgICAgMjY0MSwgICAxNTA1NCwgICAwLCAgIDANCml0aW1lcjogICAg ICAgICAgICAgICAgIDM1MiwgICAgICAwLCAgICAgICAxLCAgICAgIDEwLCAgICAgICAxLCAgIDAs ICAgMA0KS05PVEU6ICAgICAgICAgICAgICAgICAgMTI4LCAgICAgIDAsICAgICAgIDAsICAgIDIx MzksICAxNTExNzYsICAgMCwgICAwDQpzb2NrZXQ6ICAgICAgICAgICAgICAgICA2OTYsIDE1NzE2 MjAsICAgICAgNzAsICAgICAyNzAsICAxODU2MDAsICAgMCwgICAwDQppcHE6ICAgICAgICAgICAg ICAgICAgICAgNTYsICA5NTcwOCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDAN CnVkcF9pbnBjYjogICAgICAgICAgICAgIDM5MiwgMTU3MTYyMCwgICAgICAxOCwgICAgIDUyMiwg ICA4NzY5MiwgICAwLCAgIDANCnVkcGNiOiAgICAgICAgICAgICAgICAgICAxNiwgMTU3MTc2Miwg ICAgICAxOCwgICAgNDc1MSwgICA4NzY5MiwgICAwLCAgIDANCnRjcF9pbnBjYjogICAgICAgICAg ICAgIDM5MiwgMTU3MTYyMCwgICAgICAzNiwgICAgIDYxNCwgICA5MTUzOCwgICAwLCAgIDANCnRj cGNiOiAgICAgICAgICAgICAgICAgMTAyNCwgMTU3MTYyMCwgICAgICAzNiwgICAgIDI3MiwgICA5 MTUzOCwgICAwLCAgIDANCnRjcHR3OiAgICAgICAgICAgICAgICAgICA4OCwgIDI3ODEwLCAgICAg ICAwLCAgICAgNTQwLCAgICAgICA4LCAgIDAsICAgMA0Kc3luY2FjaGU6ICAgICAgICAgICAgICAg MTYwLCAgMTUzNzUsICAgICAgIDAsICAgICA1MDAsICAgODg1NzcsICAgMCwgICAwDQpob3N0Y2Fj aGU6ICAgICAgICAgICAgICAxMzYsICAxNTM3MCwgICAgICAgMSwgICAgIDIwMiwgICAgICAxMCwg ICAwLCAgIDANCnRjcHJlYXNzOiAgICAgICAgICAgICAgICA0MCwgMTkxNDAwLCAgICAgICAwLCAg ICAyMTAwLCAgICAzMzcwLCAgIDAsICAgMA0Kc2Fja2hvbGU6ICAgICAgICAgICAgICAgIDMyLCAg ICAgIDAsICAgICAgIDAsICAgIDE4NzUsICAgIDUwNDcsICAgMCwgICAwDQpzY3RwX2VwOiAgICAg ICAgICAgICAgIDE0MDgsIDE1NzE2MjAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwg ICAwDQpzY3RwX2Fzb2M6ICAgICAgICAgICAgIDI0MTYsICA0MDAwMCwgICAgICAgMCwgICAgICAg MCwgICAgICAgMCwgICAwLCAgIDANCnNjdHBfbGFkZHI6ICAgICAgICAgICAgICA0OCwgIDgwMDEy LCAgICAgICAwLCAgICAgICAwLCAgICAgICA0LCAgIDAsICAgMA0Kc2N0cF9yYWRkcjogICAgICAg ICAgICAgNzI4LCAgODAwMDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpz Y3RwX2NodW5rOiAgICAgICAgICAgICAxMzYsIDQwMDAyNiwgICAgICAgMCwgICAgICAgMCwgICAg ICAgMCwgICAwLCAgIDANCnNjdHBfcmVhZHE6ICAgICAgICAgICAgIDEwNCwgNDAwMDI2LCAgICAg ICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0Kc2N0cF9zdHJlYW1fbXNnX291dDogICAg MTA0LCA0MDAwMjYsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpzY3RwX2Fz Y29uZjogICAgICAgICAgICAgNDAsIDQwMDAwMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwg ICAwLCAgIDANCnNjdHBfYXNjb25mX2FjazogICAgICAgICA0OCwgNDAwMDYwLCAgICAgICAwLCAg ICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KdWRwbGl0ZV9pbnBjYjogICAgICAgICAgMzkyLCAx NTcxNjIwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KcmlwY2I6ICAgICAg ICAgICAgICAgICAgMzkyLCAxNTcxNjIwLCAgICAgICAwLCAgICAgIDkwLCAgICAgICA1LCAgIDAs ICAgMA0KdW5wY2I6ICAgICAgICAgICAgICAgICAgMjQwLCAxNTcxNjMyLCAgICAgIDE1LCAgICAg ODY1LCAgICA2MzYwLCAgIDAsICAgMA0KcnRlbnRyeTogICAgICAgICAgICAgICAgMjAwLCAgICAg IDAsICAgICAgMTQsICAgICAxNDYsICAgICAgMTQsICAgMCwgICAwDQpzZWxmZDogICAgICAgICAg ICAgICAgICAgNTYsICAgICAgMCwgICAgIDM1NiwgICAgNTA0MCwxMTc5ODUzNSwgICAwLCAgIDAN ClNXQVBNRVRBOiAgICAgICAgICAgICAgIDI4OCwgNjEyMzgwNiwgICAgIDgxNiwgICAgIDc1Nywg ICA1ODMyNiwgICAwLCAgIDANCmljbF9jb25uOiAgICAgICAgICAgICAgIDE1MiwgICAgICAwLCAg ICAgIDEzLCAgICAgMjQ3LCAgICAgIDI4LCAgIDAsICAgMA0KaWNsX3BkdTogICAgICAgICAgICAg ICAgIDgwLCAgICAgIDAsICAgICAgIDgsICAgIDI2NDIsMjQzMDU1NDk5LCAgIDAsICAgMA0KaXNj c2lfb3V0c3RhbmRpbmc6ICAgICAgIDQwLCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAg IDAsICAgMCwgICAwDQpjZmlfbHVuOiAgICAgICAgICAgICAgICAzNDQsICAgICAgMCwgICAgICAx NCwgICAgICA4NSwgICAgICAxNSwgICAwLCAgIDANCmNmaV9tZXRhdGFzazogICAgICAgICAgIDMy MCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KY2Zpc2NzaV9k YXRhX3dhaXQ6ICAgICAgIDU2LCAgICAgIDAsICAgICAgIDAsICAgIDYzMTksMTAyNzY4ODY5LCAg IDAsICAgMA0KYmVpbzogICAgICAgICAgICAgICAgICAgMzYwLCAgICAgIDAsICAgICAgIDMsICAg ICA3MzQsMTE0NzQ4NzA4LCAgIDAsICAgMA0KY2JsazA6ICAgICAgICAgICAgICAgMTMxMDcyLCAg ICAgIDAsICAgICAgIDAsICAgICAgMTEsICAgNTg2NDgsICAgMCwgICAwDQpjYmxrMTogICAgICAg ICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICAyMCwgICAgMTg0OSwgICAwLCAg IDANCmNibGsyOiAgICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDI2 LCA5Mzk4NTQ3LCAgIDAsICAgMA0KY2JsazM6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAs ICAgICAgIDAsICAgICAgMjUsMTM0NDc0NDIsICAgMCwgICAwDQpjYmxrNDogICAgICAgICAgICAg ICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCmNi bGs1OiAgICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDI5LDIxNjUw NTYzLCAgIDAsICAgMA0KY2JsazY6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAg IDAsICAgICAgNTIsNDM1Njg5NTEsICAgMCwgICAwDQpjYmxrNzogICAgICAgICAgICAgICAxMzEw NzIsICAgICAgMCwgICAgICAgMCwgICAgICAyMywxMDc2OTkyMiwgICAwLCAgIDANCmNibGs4OiAg ICAgICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDIzLCAyMTM4NzQ2LCAg IDAsICAgMA0KY2Jsazk6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAg ICAgMjYsMTE0MTI2ODIsICAgMCwgICAwDQpjYmxrMTA6ICAgICAgICAgICAgICAxMzEwNzIsICAg ICAgMCwgICAgICAgMCwgICAgICAzNSwgMjY4NDgzNiwgICAwLCAgIDANCmNibGsxMTogICAgICAg ICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDI3LCAgIDM5MjEwLCAgIDAsICAg MA0KY2JsazEyOiAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgMjIs ICA0MDQ3ODIsICAgMCwgICAwDQpjYmxrMTM6ICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwg ICAgICAgMCwgICAgICAyMiwgICAgMjU5MiwgICAwLCAgIDANCg0KIHByb2NzICAgICAgbWVtb3J5 ICAgICAgcGFnZSAgICAgICAgICAgICAgICAgICAgZGlza3MgICAgIGZhdWx0cyAgICAgICAgIGNw dQ0KIHIgYiB3ICAgICBhdm0gICAgZnJlICAgZmx0ICByZSAgcGkgIHBvICAgIGZyICBzciBtZjAg bWYxICAgaW4gICBzeSAgIGNzIHVzIHN5IGlkDQogMCAwIDYgICAxMzA3TSAgMjI0MU0gICA2OTYg ICAwICAgMCAgIDAgIDE3MTIgICAzICAgMCAgIDAgMzQxMiAxNzQxIDc3MTAgIDAgIDMgOTcNCjE5 MDI1LzEyNDc1LzMxNTAwIG1idWZzIGluIHVzZSAoY3VycmVudC9jYWNoZS90b3RhbCkNCjE4NDk1 Lzk1NjUvMjgwNjAvMzA2MTkwMCBtYnVmIGNsdXN0ZXJzIGluIHVzZSAoY3VycmVudC9jYWNoZS90 b3RhbC9tYXgpDQoxODQ2Mi82NDQ2IG1idWYrY2x1c3RlcnMgb3V0IG9mIHBhY2tldCBzZWNvbmRh cnkgem9uZSBpbiB1c2UgKGN1cnJlbnQvY2FjaGUpDQowLzI5OS8yOTkvMTUzMDk1MCA0ayAocGFn ZSBzaXplKSBqdW1ibyBjbHVzdGVycyBpbiB1c2UgKGN1cnJlbnQvY2FjaGUvdG90YWwvbWF4KQ0K MC8wLzAvNDUzNjE0IDlrIGp1bWJvIGNsdXN0ZXJzIGluIHVzZSAoY3VycmVudC9jYWNoZS90b3Rh bC9tYXgpDQowLzAvMC8yNTUxNTggMTZrIGp1bWJvIGNsdXN0ZXJzIGluIHVzZSAoY3VycmVudC9j YWNoZS90b3RhbC9tYXgpDQo0MTc0NksvMjM0NDRLLzY1MTkxSyBieXRlcyBhbGxvY2F0ZWQgdG8g bmV0d29yayAoY3VycmVudC9jYWNoZS90b3RhbCkNCjAvMC8wIHJlcXVlc3RzIGZvciBtYnVmcyBk ZW5pZWQgKG1idWZzL2NsdXN0ZXJzL21idWYrY2x1c3RlcnMpDQowLzAvMCByZXF1ZXN0cyBmb3Ig bWJ1ZnMgZGVsYXllZCAobWJ1ZnMvY2x1c3RlcnMvbWJ1ZitjbHVzdGVycykNCjAvMC8wIHJlcXVl c3RzIGZvciBqdW1ibyBjbHVzdGVycyBkZWxheWVkICg0ay85ay8xNmspDQowLzAvMCByZXF1ZXN0 cyBmb3IganVtYm8gY2x1c3RlcnMgZGVuaWVkICg0ay85ay8xNmspDQowIHJlcXVlc3RzIGZvciBz ZmJ1ZnMgZGVuaWVkDQowIHJlcXVlc3RzIGZvciBzZmJ1ZnMgZGVsYXllZA0KMCByZXF1ZXN0cyBm b3IgSS9PIGluaXRpYXRlZCBieSBzZW5kZmlsZQ0KSVRFTSAgICAgICAgICAgICAgICAgICBTSVpF ICBMSU1JVCAgICAgVVNFRCAgICAgRlJFRSAgICAgIFJFUSBGQUlMIFNMRUVQDQoNClVNQSBLZWdz OiAgICAgICAgICAgICAgIDM4NCwgICAgICAwLCAgICAgMjIyLCAgICAgICA4LCAgICAgMjIzLCAg IDAsICAgMA0KVU1BIFpvbmVzOiAgICAgICAgICAgICAyNjg4LCAgICAgIDAsICAgICAyMjIsICAg ICAgIDAsICAgICAyMjMsICAgMCwgICAwDQpVTUEgU2xhYnM6ICAgICAgICAgICAgICAgODAsICAg ICAgMCwgMjcyMjEwNywgIDQ1Njk5Myw2NzMzMjIzMiwgICAwLCAgIDANClVNQSBSQ250U2xhYnM6 ICAgICAgICAgICA4OCwgICAgICAwLCAgIDE0MzI5LCAgICAyODYxLCAgMzIzNTYxLCAgIDAsICAg MA0KVU1BIEhhc2g6ICAgICAgICAgICAgICAgMjU2LCAgICAgIDAsICAgICAgMjMsICAgICAgNTIs ICAgICAgOTYsICAgMCwgICAwDQo0IEJ1Y2tldDogICAgICAgICAgICAgICAgMzIsICAgICAgMCwg ICAgMzIxOCwgICAgODI4Miw2NjQ4NTc5NCwzMzc2LCAgIDANCjYgQnVja2V0OiAgICAgICAgICAg ICAgICA0OCwgICAgICAwLCAgICAgMzA2LCAgICA2OTE1LDE4ODEwMzMzLCAgIDAsICAgMA0KOCBC dWNrZXQ6ICAgICAgICAgICAgICAgIDY0LCAgICAgIDAsICAgICA3NDcsICAgIDU4MjUsIDkxNDQ0 MDQsICAxMSwgICAwDQoxMiBCdWNrZXQ6ICAgICAgICAgICAgICAgOTYsICAgICAgMCwgICAgIDkx NiwgICAgMzY3NiwyMDUxNTM0MywxMjEwNjIsICAgMA0KMTYgQnVja2V0OiAgICAgICAgICAgICAg MTI4LCAgICAgIDAsICAgICAzMzksICAgIDU3NjgsMTQ5ODkyNjMsNjYyMjksICAgMA0KMzIgQnVj a2V0OiAgICAgICAgICAgICAgMjU2LCAgICAgIDAsICAgICAzODEsICAgIDIyMTQsMTY0MTAxOTMs ICA1MSwgICAwDQo2NCBCdWNrZXQ6ICAgICAgICAgICAgICA1MTIsICAgICAgMCwgICAgNjgxNSwg ICAgIDE3NywxMDEzMTk5NSw4NTE4MDMsICAgMA0KMTI4IEJ1Y2tldDogICAgICAgICAgICAxMDI0 LCAgICAgIDAsICAgICA0NzEsICAgICA3NjUsMTM4MTQ1OTUsIDkwNSwgICAwDQoyNTYgQnVja2V0 OiAgICAgICAgICAgIDIwNDgsICAgICAgMCwgICAgMTM1NywgICAgMzUwMSw4ODU2Njg5OSwxMDc3 NSwgICAwDQp2bWVtIGJ0YWc6ICAgICAgICAgICAgICAgNTYsICAgICAgMCwgIDc2MzY4MywgICA4 NTQwNiwgNDE2MjQxMCw2MDUzLCAgIDANClZNIE9CSkVDVDogICAgICAgICAgICAgIDI1NiwgICAg ICAwLCAgIDYxMjg4LCAgICAxMjkyLCA2NzAyNzM4LCAgIDAsICAgMA0KUkFESVggTk9ERTogICAg ICAgICAgICAgMTQ0LCAgICAgIDAsICA1NjQ3NzAsICAgNTY0NzMsMzQxMjkzNjksICAgMCwgICAw DQpNQVA6ICAgICAgICAgICAgICAgICAgICAyNDAsICAgICAgMCwgICAgICAgMywgICAgICA2MSwg ICAgICAgMywgICAwLCAgIDANCktNQVAgRU5UUlk6ICAgICAgICAgICAgIDEyOCwgICAgICAwLCAg ICAgICA4LCAgICAgMTE2LCAgICAgICA4LCAgIDAsICAgMA0KTUFQIEVOVFJZOiAgICAgICAgICAg ICAgMTI4LCAgICAgIDAsICAgIDEyODEsICAgIDI5OTcsMTg3OTY5MjQsICAgMCwgICAwDQpWTVNQ QUNFOiAgICAgICAgICAgICAgICA0NDgsICAgICAgMCwgICAgICAzNywgICAgIDU4NCwgIDQ1ODgz MywgICAwLCAgIDANCmZha2VwZzogICAgICAgICAgICAgICAgIDEwNCwgICAgICAwLCAgICAgICAw LCAgICAgMjY2LCAgICAgIDUwLCAgIDAsICAgMA0KbXRfem9uZTogICAgICAgICAgICAgICA0MTEy LCAgICAgIDAsICAgICAzNjksICAgICAgIDAsICAgICAzNjksICAgMCwgICAwDQoxNjogICAgICAg ICAgICAgICAgICAgICAgMTYsICAgICAgMCwgIDEwOTM3MiwgICAgNjA4OCw0MDgzODUwMzMsICAg MCwgICAwDQozMjogICAgICAgICAgICAgICAgICAgICAgMzIsICAgICAgMCwzMDE4OTAwNSwgIDEy MjM3MCw4ODU0NTQyMTYsICAgMCwgICAwDQo2NDogICAgICAgICAgICAgICAgICAgICAgNjQsICAg ICAgMCwgMTc1NTIwNiwgIDIzMzMyMCwzNDI5NjgwNDQ5LCAgIDAsICAgMA0KMTI4OiAgICAgICAg ICAgICAgICAgICAgMTI4LCAgICAgIDAsIDExMzcwNTUsIDEyMjYzNTQsMzE4NzMzNDg5NywgICAw LCAgIDANCjI1NjogICAgICAgICAgICAgICAgICAgIDI1NiwgICAgICAwLCAgODI0NjgwLCAgICAx MDQwLDIyOTQ2NDUyMDYsICAgMCwgICAwDQo1MTI6ICAgICAgICAgICAgICAgICAgICA1MTIsICAg ICAgMCwgICAgIDg2MiwgIDM5NjYwMiw2Mzk5ODU4MzksICAgMCwgICAwDQoxMDI0OiAgICAgICAg ICAgICAgICAgIDEwMjQsICAgICAgMCwgICAxNTk5MywgICAgIDI4MywyMDIzNTg3MywgICAwLCAg IDANCjIwNDg6ICAgICAgICAgICAgICAgICAgMjA0OCwgICAgICAwLCAgIDYzMDI2LCAgICAxNDEy LDQ4MzczNzE5NSwgICAwLCAgIDANCjQwOTY6ICAgICAgICAgICAgICAgICAgNDA5NiwgICAgICAw LCAgIDEzMDI5LCAgICAgIDU0LDQxODY1NTUyLCAgIDAsICAgMA0KODE5MjogICAgICAgICAgICAg ICAgICA4MTkyLCAgICAgIDAsICAgICAgMjYsICAgICAgMjEsICA0NTE3NjQsICAgMCwgICAwDQox NjM4NDogICAgICAgICAgICAgICAgMTYzODQsICAgICAgMCwgICAgICAzNSwgICAgICAxNywgICAg Mjc2NiwgICAwLCAgIDANCjMyNzY4OiAgICAgICAgICAgICAgICAzMjc2OCwgICAgICAwLCAgICAg IDM5LCAgICAgIDE2LCAgIDE3ODEwLCAgIDAsICAgMA0KNjU1MzY6ICAgICAgICAgICAgICAgIDY1 NTM2LCAgICAgIDAsICAgICAgODYsICAgICAgMjAsICAzMDQ3MzQsICAgMCwgICAwDQo2NCBwY3B1 OiAgICAgICAgICAgICAgICAgIDgsICAgICAgMCwgICAgMTY1NCwgICAgIDY1MCwgICAgMTg3NCwg ICAwLCAgIDANClNMRUVQUVVFVUU6ICAgICAgICAgICAgICA4MCwgICAgICAwLCAgICAxMjYxLCAg ICAxMDMzLCAgICAxMzQyLCAgIDAsICAgMA0KRmlsZXM6ICAgICAgICAgICAgICAgICAgIDgwLCAg ICAgIDAsICAgICAxMjQsICAgIDIyNzYsIDQwMTcxMDksICAgMCwgICAwDQpUVVJOU1RJTEU6ICAg ICAgICAgICAgICAxMzYsICAgICAgMCwgICAgMTI2MSwgICAgIDUxOSwgICAgMTM0MiwgICAwLCAg IDANCnJsX2VudHJ5OiAgICAgICAgICAgICAgICA0MCwgICAgICAwLCAgICAgMTk5LCAgICAyNTAx LCAgICAgMTk5LCAgIDAsICAgMA0KdW10eCBwaTogICAgICAgICAgICAgICAgIDk2LCAgICAgIDAs ICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpNQUMgbGFiZWxzOiAgICAgICAg ICAgICAgNDAsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANClBS T0M6ICAgICAgICAgICAgICAgICAgMTIxNiwgICAgICAwLCAgICAgIDU1LCAgICAgMTg1LCAgNDU4 ODYzLCAgIDAsICAgMA0KVEhSRUFEOiAgICAgICAgICAgICAgICAxMTY4LCAgICAgIDAsICAgIDEx OTksICAgICAgNjEsICAgIDE3NjcsICAgMCwgICAwDQpjcHVzZXQ6ICAgICAgICAgICAgICAgICAg NzIsICAgICAgMCwgICAgIDk1OSwgICAgMTA3NiwgICAgMTUyMiwgICAwLCAgIDANCmF1ZGl0X3Jl Y29yZDogICAgICAgICAgMTI0OCwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAg IDAsICAgMA0KbWJ1Zl9wYWNrZXQ6ICAgICAgICAgICAgMjU2LCAxOTU5NjE2NSwgICAxODQ2NCwg ICAgNjQ0NCwyNjc3MjQ0MDgzLCAgIDAsICAgMA0KbWJ1ZjogICAgICAgICAgICAgICAgICAgMjU2 LCAxOTU5NjE2NSwgICAgIDU2MSwgICAgNjAzMSwyNzgwNzQ0MDgxLCAgIDAsICAgMA0KbWJ1Zl9j bHVzdGVyOiAgICAgICAgICAyMDQ4LCAzMDYxOTAwLCAgIDI0OTQxLCAgICAzMTE5LDU0ODYzMjkz MSwgICAwLCAgIDANCm1idWZfanVtYm9fcGFnZTogICAgICAgNDA5NiwgMTUzMDk1MCwgICAgICAg MCwgICAgIDI5OSwxNDQyNTA3MTksICAgMCwgICAwDQptYnVmX2p1bWJvXzlrOiAgICAgICAgIDky MTYsIDQ1MzYxNCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCm1idWZfanVt Ym9fMTZrOiAgICAgICAxNjM4NCwgMjU1MTU4LCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAg IDAsICAgMA0KbWJ1Zl9leHRfcmVmY250OiAgICAgICAgICA0LCAgICAgIDAsICAgICAgIDAsICAg ICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpnX2JpbzogICAgICAgICAgICAgICAgICAyNDgsICAg ICAgMCwgICAgIDEwNiwgICAgMzM4MiwxOTMxOTI5NjE5LCAgIDAsICAgMA0KdHR5aW5xOiAgICAg ICAgICAgICAgICAgMTYwLCAgICAgIDAsICAgICAyNDAsICAgICAyMzUsICAgICA4MjUsICAgMCwg ICAwDQp0dHlvdXRxOiAgICAgICAgICAgICAgICAyNTYsICAgICAgMCwgICAgIDEyNiwgICAgIDEx NCwgICAgIDQzOCwgICAwLCAgIDANCkRNQVJfTUFQX0VOVFJZOiAgICAgICAgIDEyMCwgICAgICAw LCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KYXRhX3JlcXVlc3Q6ICAgICAg ICAgICAgMzM2LCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQp2 dG5ldF90eF9oZHI6ICAgICAgICAgICAgMjQsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAg ICAgMCwgICAwLCAgIDANCkZQVV9zYXZlX2FyZWE6ICAgICAgICAgIDUxMiwgICAgICAwLCAgICAg ICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KdGFza3Ffem9uZTogICAgICAgICAgICAg IDQ4LCAgICAgIDAsICAgICAgIDAsICAgIDYyMjUsIDU5ODI4ODgsICAgMCwgICAwDQpWTk9ERTog ICAgICAgICAgICAgICAgICA0NzIsICAgICAgMCwgIDIzNDIzMSwgICAxNzkyMSwgMTU0NjAzMSwg ICAwLCAgIDANClZOT0RFUE9MTDogICAgICAgICAgICAgIDExMiwgICAgICAwLCAgICAgICAwLCAg ICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0KQlVGIFRSSUU6ICAgICAgICAgICAgICAgMTQ0LCAg ICAgIDAsICAgICAgIDAsICAxMDU5NDgsICAgICAgIDAsICAgMCwgICAwDQpTIFZGUyBDYWNoZTog ICAgICAgICAgICAxMDgsICAgICAgMCwgIDEzODQ2NCwgICA2MzY2MSwgMTgyNTAwNiwgICAwLCAg IDANClNUUyBWRlMgQ2FjaGU6ICAgICAgICAgIDE0OCwgICAgICAwLCAgICAgICAwLCAgICAgICAw LCAgICAgICAwLCAgIDAsICAgMA0KTCBWRlMgQ2FjaGU6ICAgICAgICAgICAgMzI4LCAgICAgIDAs ICAgNTY3OTcsICAgICA1MjcsICAzMTI0NzAsICAgMCwgICAwDQpMVFMgVkZTIENhY2hlOiAgICAg ICAgICAzNjgsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCk5B TUVJOiAgICAgICAgICAgICAgICAgMTAyNCwgICAgICAwLCAgICAgICAwLCAgICAgMjc2LDE0Mjcx MDc0LCAgIDAsICAgMA0KTkNMTk9ERTogICAgICAgICAgICAgICAgNTI4LCAgICAgIDAsICAgICAg IDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpESVJIQVNIOiAgICAgICAgICAgICAgIDEw MjQsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnBpcGU6ICAg ICAgICAgICAgICAgICAgIDc0NCwgICAgICAwLCAgICAgICA0LCAgICAgMjU2LCAgIDIxMTY1LCAg IDAsICAgMA0KcHJvY2Rlc2M6ICAgICAgICAgICAgICAgMTI4LCAgICAgIDAsICAgICAgIDAsICAg ICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpyYW5nZV9zZWdfY2FjaGU6ICAgICAgICAgNjQsICAg ICAgMCwgMTg4ODc5MywgMTE1ODU2OSw1MjQxMzEyNjc5LCAgIDAsICAgMA0KemlvX2NhY2hlOiAg ICAgICAgICAgICAgOTIwLCAgICAgIDAsICAyNzg3NzgsICAgMzMxOTAsOTQ3ODcyNjI5NywgICAw LCAgIDANCnppb19saW5rX2NhY2hlOiAgICAgICAgICA0OCwgICAgICAwLCAgMjc5MDA0LCAgIDMz MzI1LDEwODkzNTk2MzY3LCAgIDAsICAgMA0KemlvX2J1Zl81MTI6ICAgICAgICAgICAgNTEyLCAg ICAgIDAsICAyNTM0NDEsICAgMzE3NjcsMTczMzczMTIsICAgMCwgICAwDQp6aW9fZGF0YV9idWZf NTEyOiAgICAgICA1MTIsICAgICAgMCwgICAgICAxMCwgICAgIDY1NCwgMTI3MTY1OCwgICAwLCAg IDANCnppb19idWZfMTAyNDogICAgICAgICAgMTAyNCwgICAgICAwLCAgICAgIDM1LCAgICAgMzU3 LCAyMjgxNjYxLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzEwMjQ6ICAgICAxMDI0LCAgICAgIDAs ICAgICAgIDEsICAgICA4NTEsICA0MDU5MjcsICAgMCwgICAwDQp6aW9fYnVmXzE1MzY6ICAgICAg ICAgIDE1MzYsICAgICAgMCwgICAgICAzMSwgICAgIDE5OSwgMTMxNzQyNCwgICAwLCAgIDANCnpp b19kYXRhX2J1Zl8xNTM2OiAgICAgMTUzNiwgICAgICAwLCAgICAgICAwLCAgICAgMTk0LCAgNTM5 ODk3LCAgIDAsICAgMA0KemlvX2J1Zl8yMDQ4OiAgICAgICAgICAyMDQ4LCAgICAgIDAsICAgICAg NzIsICAgICAxNjgsIDU5NDI4NjksICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMjA0ODogICAgIDIw NDgsICAgICAgMCwgICAgICAgMCwgICAgIDEyMCwgIDcxODMyOCwgICAwLCAgIDANCnppb19idWZf MjU2MDogICAgICAgICAgMjU2MCwgICAgICAwLCAgICAgICAwLCAgICAgIDU4LCAyNjgxMDQzLCAg IDAsICAgMA0KemlvX2RhdGFfYnVmXzI1NjA6ICAgICAyNTYwLCAgICAgIDAsICAgICAgIDEsICAg ICAgNjMsICA1NjIyMjEsICAgMCwgICAwDQp6aW9fYnVmXzMwNzI6ICAgICAgICAgIDMwNzIsICAg ICAgMCwgICAgICA4NywgICAgICA2OSwgMjMyMjYzMSwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8z MDcyOiAgICAgMzA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDYzLCAgMjA3NzE4LCAgIDAsICAg MA0KemlvX2J1Zl8zNTg0OiAgICAgICAgICAzNTg0LCAgICAgIDAsICAgICAgIDEsICAgICAgMjMs ICA4NTIwMTksICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMzU4NDogICAgIDM1ODQsICAgICAgMCwg ICAgICAgMSwgICAgICAyMCwgIDE4NzA4NSwgICAwLCAgIDANCnppb19idWZfNDA5NjogICAgICAg ICAgNDA5NiwgICAgICAwLCAxODk3NjUxLCAgICAxODE1LDEwMzY2OTUwNzgsICAgMCwgICAwDQp6 aW9fZGF0YV9idWZfNDA5NjogICAgIDQwOTYsICAgICAgMCwgICA0MjM4NSwgICAgMjAyNSwyMzc2 MDg5NywgICAwLCAgIDANCnppb19idWZfNTEyMDogICAgICAgICAgNTEyMCwgICAgICAwLCAgICAg MjA5LCAgICAgIDI0LCA1MDUxMTcyLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzUxMjA6ICAgICA1 MTIwLCAgICAgIDAsICAgICAgIDAsICAgICAgMjQsICAyODE5ODIsICAgMCwgICAwDQp6aW9fYnVm XzYxNDQ6ICAgICAgICAgIDYxNDQsICAgICAgMCwgICAgICAgMCwgICAgICAyOSwgMjk1OTUwNCwg ICAwLCAgIDANCnppb19kYXRhX2J1Zl82MTQ0OiAgICAgNjE0NCwgICAgICAwLCAgICAgICAxLCAg ICAgIDI4LCAgNTIyMzA2LCAgIDAsICAgMA0KemlvX2J1Zl83MTY4OiAgICAgICAgICA3MTY4LCAg ICAgIDAsICAgICAgIDAsICAgICAgMjgsIDI3ODM5NjUsICAgMCwgICAwDQp6aW9fZGF0YV9idWZf NzE2ODogICAgIDcxNjgsICAgICAgMCwgICAgICAgMCwgICAgICAyNiwgIDQ0MTU2NiwgICAwLCAg IDANCnppb19idWZfODE5MjogICAgICAgICAgODE5MiwgICAgICAwLCAgICAgICAyLCAgICAgMjI2 LDk1ODY4MzA4LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzgxOTI6ICAgICA4MTkyLCAgICAgIDAs ICAgICAgIDAsICAgICAgMjQsICAzMTYyMjYsICAgMCwgICAwDQp6aW9fYnVmXzEwMjQwOiAgICAg ICAgMTAyNDAsICAgICAgMCwgICAgICAgMCwgICAgICAzNiwgNDk4NTAyMCwgICAwLCAgIDANCnpp b19kYXRhX2J1Zl8xMDI0MDogICAxMDI0MCwgICAgICAwLCAgICAgICAxLCAgICAgIDI2LCAgNTQ5 OTM4LCAgIDAsICAgMA0KemlvX2J1Zl8xMjI4ODogICAgICAgIDEyMjg4LCAgICAgIDAsICAgICAg IDEsICAgICAgNjUsMjc5NzkwOTUsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTIyODg6ICAgMTIy ODgsICAgICAgMCwgICAgICAgMSwgICAgICAyNiwgIDUwNjgzMSwgICAwLCAgIDANCnppb19idWZf MTQzMzY6ICAgICAgICAxNDMzNiwgICAgICAwLCAgICAgICAyLCAgICAgIDI3LCA0Mjc0NTA5LCAg IDAsICAgMA0KemlvX2RhdGFfYnVmXzE0MzM2OiAgIDE0MzM2LCAgICAgIDAsICAgICAgIDAsICAg ICAgMjcsICA0MTc2OTUsICAgMCwgICAwDQp6aW9fYnVmXzE2Mzg0OiAgICAgICAgMTYzODQsICAg ICAgMCwgIDI0Mzg2NCwgICAgIDc5MywxNDI2ODI5MDYsICAgMCwgICAwDQp6aW9fZGF0YV9idWZf MTYzODQ6ICAgMTYzODQsICAgICAgMCwgICAgMTA1MCwgICAgICAyMCwgIDc3MzI1NiwgICAwLCAg IDANCnppb19idWZfMjA0ODA6ICAgICAgICAyMDQ4MCwgICAgICAwLCAgICAgICAyLCAgICAgIDc0 LDIxMDAyMDgwLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzIwNDgwOiAgIDIwNDgwLCAgICAgIDAs ICAgICAgIDAsICAgICAgMjgsICA1MjI4NDksICAgMCwgICAwDQp6aW9fYnVmXzI0NTc2OiAgICAg ICAgMjQ1NzYsICAgICAgMCwgICAgICAgMCwgICAgICA1NiwyMTEzODQyMiwgICAwLCAgIDANCnpp b19kYXRhX2J1Zl8yNDU3NjogICAyNDU3NiwgICAgICAwLCAgICAgICAwLCAgICAgIDI4LCAgNDMw MTc4LCAgIDAsICAgMA0KemlvX2J1Zl8yODY3MjogICAgICAgIDI4NjcyLCAgICAgIDAsICAgICAg IDAsICAgICAgODQsMTk1MjMyMzcsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMjg2NzI6ICAgMjg2 NzIsICAgICAgMCwgICAgICAgMCwgICAgICAyNiwgIDQyMzU5MywgICAwLCAgIDANCnppb19idWZf MzI3Njg6ICAgICAgICAzMjc2OCwgICAgICAwLCAgICAgICAwLCAgICAgMjY3LDIwNTExMzgxLCAg IDAsICAgMA0KemlvX2RhdGFfYnVmXzMyNzY4OiAgIDMyNzY4LCAgICAgIDAsICAgICAxNzAsICAg ICAgNTYsMjkyOTUwNjcsICAgMCwgICAwDQp6aW9fYnVmXzM2ODY0OiAgICAgICAgMzY4NjQsICAg ICAgMCwgICAgICAgMCwgICAgICA1NywxMTAzNDExOCwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8z Njg2NDogICAzNjg2NCwgICAgICAwLCAgICAgICAwLCAgICAgIDI5LCAgMzk5MTQ5LCAgIDAsICAg MA0KemlvX2J1Zl80MDk2MDogICAgICAgIDQwOTYwLCAgICAgIDAsICAgICAgIDAsICAgICAgNTcs MTA1OTYzNDgsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfNDA5NjA6ICAgNDA5NjAsICAgICAgMCwg ICAgICAgMCwgICAgICAyOCwgIDQ2MTA0NCwgICAwLCAgIDANCnppb19idWZfNDUwNTY6ICAgICAg ICA0NTA1NiwgICAgICAwLCAgICAgICAwLCAgICAgIDkxLCA4OTc4NTM5LCAgIDAsICAgMA0Kemlv X2RhdGFfYnVmXzQ1MDU2OiAgIDQ1MDU2LCAgICAgIDAsICAgICAgIDAsICAgICAgMjksICA0NzIy MzQsICAgMCwgICAwDQp6aW9fYnVmXzQ5MTUyOiAgICAgICAgNDkxNTIsICAgICAgMCwgICAgICAg MCwgICAgICA5MCwxMDQ2NjkwMCwgICAwLCAgIDANCnppb19kYXRhX2J1Zl80OTE1MjogICA0OTE1 MiwgICAgICAwLCAgICAgICAwLCAgICAgIDI0LCAgMzk0MjU2LCAgIDAsICAgMA0KemlvX2J1Zl81 MzI0ODogICAgICAgIDUzMjQ4LCAgICAgIDAsICAgICAgIDIsICAgICAgODAsIDc2MDQyMTksICAg MCwgICAwDQp6aW9fZGF0YV9idWZfNTMyNDg6ICAgNTMyNDgsICAgICAgMCwgICAgICAgMCwgICAg ICAyNywgIDI5ODczNywgICAwLCAgIDANCnppb19idWZfNTczNDQ6ICAgICAgICA1NzM0NCwgICAg ICAwLCAgICAgICAwLCAgICAgIDUwLCA3NTM5NTAyLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzU3 MzQ0OiAgIDU3MzQ0LCAgICAgIDAsICAgICAgIDAsICAgICAgMjUsICAzNzI3MzAsICAgMCwgICAw DQp6aW9fYnVmXzYxNDQwOiAgICAgICAgNjE0NDAsICAgICAgMCwgICAgICAgMCwgICAgICAxNywg MzMxMzI1MiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl82MTQ0MDogICA2MTQ0MCwgICAgICAwLCAg ICAgICAwLCAgICAgIDIzLCAgICAgNjIzLCAgIDAsICAgMA0KemlvX2J1Zl82NTUzNjogICAgICAg IDY1NTM2LCAgICAgIDAsICAgICAgIDAsICAgIDE4OTMsOTkxMzQxMjMsICAgMCwgICAwDQp6aW9f ZGF0YV9idWZfNjU1MzY6ICAgNjU1MzYsICAgICAgMCwgIDM3NDg0NSwgICAgMTMxNSwxMTY0MDA0 NjksICAgMCwgICAwDQp6aW9fYnVmXzY5NjMyOiAgICAgICAgNjk2MzIsICAgICAgMCwgICAgICAg MCwgICAgICAxOSwgMjM2ODgxNCwgICAwLCAgIDANCnppb19kYXRhX2J1Zl82OTYzMjogICA2OTYz MiwgICAgICAwLCAgICAgICAwLCAgICAgIDIxLCAgICAgNjcyLCAgIDAsICAgMA0KemlvX2J1Zl83 MzcyODogICAgICAgIDczNzI4LCAgICAgIDAsICAgICAgIDAsICAgICAgMTksIDI0MzUzNjQsICAg MCwgICAwDQp6aW9fZGF0YV9idWZfNzM3Mjg6ICAgNzM3MjgsICAgICAgMCwgICAgICAgMCwgICAg ICAyMSwgICAgIDYyMiwgICAwLCAgIDANCnppb19idWZfNzc4MjQ6ICAgICAgICA3NzgyNCwgICAg ICAwLCAgICAgICAwLCAgICAgIDE4LCAxOTc0NTkxLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzc3 ODI0OiAgIDc3ODI0LCAgICAgIDAsICAgICAgIDAsICAgICAgMjIsICAgICA2MjksICAgMCwgICAw DQp6aW9fYnVmXzgxOTIwOiAgICAgICAgODE5MjAsICAgICAgMCwgICAgICAgMCwgICAgICAyMSwg MjQyMTYzMiwgICAwLCAgIDANCnppb19kYXRhX2J1Zl84MTkyMDogICA4MTkyMCwgICAgICAwLCAg ICAgICAwLCAgICAgIDI0LCAgICAgNjIxLCAgIDAsICAgMA0KemlvX2J1Zl84NjAxNjogICAgICAg IDg2MDE2LCAgICAgIDAsICAgICAgIDAsICAgICAgMTgsIDE4MTE1MzEsICAgMCwgICAwDQp6aW9f ZGF0YV9idWZfODYwMTY6ICAgODYwMTYsICAgICAgMCwgICAgICAgMCwgICAgICAyMSwgICAgIDYx NSwgICAwLCAgIDANCnppb19idWZfOTAxMTI6ICAgICAgICA5MDExMiwgICAgICAwLCAgICAgICAw LCAgICAgIDE4LCAxOTY1Njc4LCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzkwMTEyOiAgIDkwMTEy LCAgICAgIDAsICAgICAgIDAsICAgICAgMjEsICAgICA2OTQsICAgMCwgICAwDQp6aW9fYnVmXzk0 MjA4OiAgICAgICAgOTQyMDgsICAgICAgMCwgICAgICAgMCwgICAgICAyMCwgMTc4NzAyMCwgICAw LCAgIDANCnppb19kYXRhX2J1Zl85NDIwODogICA5NDIwOCwgICAgICAwLCAgICAgICAwLCAgICAg IDI0LCAgICAgNjIzLCAgIDAsICAgMA0KemlvX2J1Zl85ODMwNDogICAgICAgIDk4MzA0LCAgICAg IDAsICAgICAgIDAsICAgICAgMjEsIDI0NTU0MDcsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfOTgz MDQ6ICAgOTgzMDQsICAgICAgMCwgICAgICAgMCwgICAgICAyMSwgICAgIDYxOSwgICAwLCAgIDAN Cnppb19idWZfMTAyNDAwOiAgICAgIDEwMjQwMCwgICAgICAwLCAgICAgICAwLCAgICAgIDI4LCAx ODI4NTUwLCAgIDAsICAgMA0KemlvX2RhdGFfYnVmXzEwMjQwMDogMTAyNDAwLCAgICAgIDAsICAg ICAgIDAsICAgICAgMjEsICAgICA2MDksICAgMCwgICAwDQp6aW9fYnVmXzEwNjQ5NjogICAgICAx MDY0OTYsICAgICAgMCwgICAgICAgMCwgICAgICAyMiwgMTk1NDYyMiwgICAwLCAgIDANCnppb19k YXRhX2J1Zl8xMDY0OTY6IDEwNjQ5NiwgICAgICAwLCAgICAgICAwLCAgICAgIDIyLCAgICAgNjA2 LCAgIDAsICAgMA0KemlvX2J1Zl8xMTA1OTI6ICAgICAgMTEwNTkyLCAgICAgIDAsICAgICAgIDAs ICAgICAgMjMsIDE5ODUxMTYsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTEwNTkyOiAxMTA1OTIs ICAgICAgMCwgICAgICAgMCwgICAgICAxOSwgICAgIDU3NSwgICAwLCAgIDANCnppb19idWZfMTE0 Njg4OiAgICAgIDExNDY4OCwgICAgICAwLCAgICAgICAwLCAgICAgIDQzLCAyMzcyOTE4LCAgIDAs ICAgMA0KemlvX2RhdGFfYnVmXzExNDY4ODogMTE0Njg4LCAgICAgIDAsICAgICAgIDAsICAgICAg MjUsICAgICA1ODEsICAgMCwgICAwDQp6aW9fYnVmXzExODc4NDogICAgICAxMTg3ODQsICAgICAg MCwgICAgICAgMSwgICAgICAxNywgMjA1MTc2OCwgICAwLCAgIDANCnppb19kYXRhX2J1Zl8xMTg3 ODQ6IDExODc4NCwgICAgICAwLCAgICAgICAwLCAgICAgIDI0LCAgICAgNTg0LCAgIDAsICAgMA0K emlvX2J1Zl8xMjI4ODA6ICAgICAgMTIyODgwLCAgICAgIDAsICAgICAgIDAsICAgICAgMTgsIDI0 OTI1ODUsICAgMCwgICAwDQp6aW9fZGF0YV9idWZfMTIyODgwOiAxMjI4ODAsICAgICAgMCwgICAg ICAgMCwgICAgICAyMCwgICAgIDU4MiwgICAwLCAgIDANCnppb19idWZfMTI2OTc2OiAgICAgIDEy Njk3NiwgICAgICAwLCAgICAgICAwLCAgICAgIDIyLCAzNTgyOTI0LCAgIDAsICAgMA0KemlvX2Rh dGFfYnVmXzEyNjk3NjogMTI2OTc2LCAgICAgIDAsICAgICAgIDAsICAgICAgMjEsICAgICA1ODEs ICAgMCwgICAwDQp6aW9fYnVmXzEzMTA3MjogICAgICAxMzEwNzIsICAgICAgMCwgICAxMDQ3NCwg ICAgMTYyMyw2NjA5NzI3MywgICAwLCAgIDANCnppb19kYXRhX2J1Zl8xMzEwNzI6IDEzMTA3Miwg ICAgICAwLCAgICAgIDE1LCAgICAgIDIyLCAgIDE2Njk3LCAgIDAsICAgMA0KbHo0X2N0eDogICAg ICAgICAgICAgIDE2Mzg0LCAgICAgIDAsICAgICAgIDEsICAgICAgMzgsNDY3MDM1OTI1LCAgIDAs ICAgMA0Kc2FfY2FjaGU6ICAgICAgICAgICAgICAgIDgwLCAgICAgIDAsICAyMzQxMzEsICAgMjk2 MTksIDE1NDU1NzIsICAgMCwgICAwDQpkbm9kZV90OiAgICAgICAgICAgICAgICA3NTIsICAgICAg MCwgIDMwMTc4NCwgICAxMDM1MSwgMTU1Nzk2NSwgICAwLCAgIDANCmRtdV9idWZfaW1wbF90OiAg ICAgICAgIDIzMiwgICAgICAwLCAyNTk4NjMzLCAgNTExNjg3LDI0OTU2NzEzNCwgICAwLCAgIDAN CmFyY19idWZfaGRyX3Q6ICAgICAgICAgIDIxNiwgICAgICAwLDE3NjAyMzc4LCAgIDE3NDgwLDE3 MjI0ODM5NCwgICAwLCAgIDANCmFyY19idWZfdDogICAgICAgICAgICAgICA3MiwgICAgICAwLCAy NDU2OTkzLCAgNjc3MjkyLDE4NjQ4MDAwMCwgICAwLCAgIDANCnppbF9sd2JfY2FjaGU6ICAgICAg ICAgIDE5MiwgICAgICAwLCAgICA2MDI0LCAgICAgNTk2LDMzNjMxNzcxLCAgIDAsICAgMA0KemZz X3pub2RlX2NhY2hlOiAgICAgICAgMzY4LCAgICAgIDAsICAyMzQxMzEsICAgMTg4OTksIDE1NDU1 NzIsICAgMCwgICAwDQpNb3VudHBvaW50czogICAgICAgICAgICA4MTYsICAgICAgMCwgICAgICAz NywgICAgIDEyMywgICAgICAzNywgICAwLCAgIDANCmtzaWdpbmZvOiAgICAgICAgICAgICAgIDEx MiwgICAgICAwLCAgICAgMjI5LCAgICAyNjQxLCAgIDE1MDU0LCAgIDAsICAgMA0KaXRpbWVyOiAg ICAgICAgICAgICAgICAgMzUyLCAgICAgIDAsICAgICAgIDEsICAgICAgMTAsICAgICAgIDEsICAg MCwgICAwDQpLTk9URTogICAgICAgICAgICAgICAgICAxMjgsICAgICAgMCwgICAgICAgMiwgICAg MjEzNywgIDE1MTE3OCwgICAwLCAgIDANCnNvY2tldDogICAgICAgICAgICAgICAgIDY5NiwgMTU3 MTYyMCwgICAgICA3MywgICAgIDI2NywgIDE4NTYwNSwgICAwLCAgIDANCmlwcTogICAgICAgICAg ICAgICAgICAgICA1NiwgIDk1NzA4LCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAg MA0KdWRwX2lucGNiOiAgICAgICAgICAgICAgMzkyLCAxNTcxNjIwLCAgICAgIDE5LCAgICAgNTIx LCAgIDg3NjkzLCAgIDAsICAgMA0KdWRwY2I6ICAgICAgICAgICAgICAgICAgIDE2LCAxNTcxNzYy LCAgICAgIDE5LCAgICA0NzUwLCAgIDg3NjkzLCAgIDAsICAgMA0KdGNwX2lucGNiOiAgICAgICAg ICAgICAgMzkyLCAxNTcxNjIwLCAgICAgIDM4LCAgICAgNjEyLCAgIDkxNTQyLCAgIDAsICAgMA0K dGNwY2I6ICAgICAgICAgICAgICAgICAxMDI0LCAxNTcxNjIwLCAgICAgIDM4LCAgICAgMjcwLCAg IDkxNTQyLCAgIDAsICAgMA0KdGNwdHc6ICAgICAgICAgICAgICAgICAgIDg4LCAgMjc4MTAsICAg ICAgIDAsICAgICA1NDAsICAgICAgIDgsICAgMCwgICAwDQpzeW5jYWNoZTogICAgICAgICAgICAg ICAxNjAsICAxNTM3NSwgICAgICAgMCwgICAgIDUwMCwgICA4ODU4MSwgICAwLCAgIDANCmhvc3Rj YWNoZTogICAgICAgICAgICAgIDEzNiwgIDE1MzcwLCAgICAgICAxLCAgICAgMjAyLCAgICAgIDEw LCAgIDAsICAgMA0KdGNwcmVhc3M6ICAgICAgICAgICAgICAgIDQwLCAxOTE0MDAsICAgICAgIDAs ICAgIDIxMDAsICAgIDMzNzAsICAgMCwgICAwDQpzYWNraG9sZTogICAgICAgICAgICAgICAgMzIs ICAgICAgMCwgICAgICAgMCwgICAgMTg3NSwgICAgNTA0OSwgICAwLCAgIDANCnNjdHBfZXA6ICAg ICAgICAgICAgICAgMTQwOCwgMTU3MTYyMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAw LCAgIDANCnNjdHBfYXNvYzogICAgICAgICAgICAgMjQxNiwgIDQwMDAwLCAgICAgICAwLCAgICAg ICAwLCAgICAgICAwLCAgIDAsICAgMA0Kc2N0cF9sYWRkcjogICAgICAgICAgICAgIDQ4LCAgODAw MTIsICAgICAgIDAsICAgICAgIDAsICAgICAgIDQsICAgMCwgICAwDQpzY3RwX3JhZGRyOiAgICAg ICAgICAgICA3MjgsICA4MDAwMCwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDAN CnNjdHBfY2h1bms6ICAgICAgICAgICAgIDEzNiwgNDAwMDI2LCAgICAgICAwLCAgICAgICAwLCAg ICAgICAwLCAgIDAsICAgMA0Kc2N0cF9yZWFkcTogICAgICAgICAgICAgMTA0LCA0MDAwMjYsICAg ICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpzY3RwX3N0cmVhbV9tc2dfb3V0OiAg ICAxMDQsIDQwMDAyNiwgICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAwLCAgIDANCnNjdHBf YXNjb25mOiAgICAgICAgICAgICA0MCwgNDAwMDAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAw LCAgIDAsICAgMA0Kc2N0cF9hc2NvbmZfYWNrOiAgICAgICAgIDQ4LCA0MDAwNjAsICAgICAgIDAs ICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQp1ZHBsaXRlX2lucGNiOiAgICAgICAgICAzOTIs IDE1NzE2MjAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpyaXBjYjogICAg ICAgICAgICAgICAgICAzOTIsIDE1NzE2MjAsICAgICAgIDAsICAgICAgOTAsICAgICAgIDUsICAg MCwgICAwDQp1bnBjYjogICAgICAgICAgICAgICAgICAyNDAsIDE1NzE2MzIsICAgICAgMTUsICAg ICA4NjUsICAgIDYzNjAsICAgMCwgICAwDQpydGVudHJ5OiAgICAgICAgICAgICAgICAyMDAsICAg ICAgMCwgICAgICAxNCwgICAgIDE0NiwgICAgICAxNCwgICAwLCAgIDANCnNlbGZkOiAgICAgICAg ICAgICAgICAgICA1NiwgICAgICAwLCAgICAgMzUzLCAgICA1MDQzLDExNzk4NzAwLCAgIDAsICAg MA0KU1dBUE1FVEE6ICAgICAgICAgICAgICAgMjg4LCA2MTIzODA2LCAgICAgODE2LCAgICAgNzU3 LCAgIDU4MzI4LCAgIDAsICAgMA0KaWNsX2Nvbm46ICAgICAgICAgICAgICAgMTUyLCAgICAgIDAs ICAgICAgMTMsICAgICAyNDcsICAgICAgMjgsICAgMCwgICAwDQppY2xfcGR1OiAgICAgICAgICAg ICAgICAgODAsICAgICAgMCwgICAgICAxNSwgICAgMjYzNSwyNDMwNjQwNTYsICAgMCwgICAwDQpp c2NzaV9vdXRzdGFuZGluZzogICAgICAgNDAsICAgICAgMCwgICAgICAgMCwgICAgICAgMCwgICAg ICAgMCwgICAwLCAgIDANCmNmaV9sdW46ICAgICAgICAgICAgICAgIDM0NCwgICAgICAwLCAgICAg IDE0LCAgICAgIDg1LCAgICAgIDE1LCAgIDAsICAgMA0KY2ZpX21ldGF0YXNrOiAgICAgICAgICAg MzIwLCAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgICAgIDAsICAgMCwgICAwDQpjZmlzY3Np X2RhdGFfd2FpdDogICAgICAgNTYsICAgICAgMCwgICAgICAgMCwgICAgNjMxOSwxMDI3NzExNjcs ICAgMCwgICAwDQpiZWlvOiAgICAgICAgICAgICAgICAgICAzNjAsICAgICAgMCwgICAgICAgNCwg ICAgIDczMywxMTQ3NTIzMTUsICAgMCwgICAwDQpjYmxrMDogICAgICAgICAgICAgICAxMzEwNzIs ICAgICAgMCwgICAgICAgMCwgICAgICAxMSwgICA1ODY0OCwgICAwLCAgIDANCmNibGsxOiAgICAg ICAgICAgICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDIwLCAgICAxODQ5LCAgIDAs ICAgMA0KY2JsazI6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAg MjYsIDkzOTg2NzksICAgMCwgICAwDQpjYmxrMzogICAgICAgICAgICAgICAxMzEwNzIsICAgICAg MCwgICAgICAgMSwgICAgICAyNSwxMzQ0Nzg0MywgICAwLCAgIDANCmNibGs0OiAgICAgICAgICAg ICAgIDEzMTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgICAgICAwLCAgIDAsICAgMA0K Y2JsazU6ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgMjksMjE2 NTA3OTgsICAgMCwgICAwDQpjYmxrNjogICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAg ICAgMCwgICAgICA1Miw0MzU2OTczOCwgICAwLCAgIDANCmNibGs3OiAgICAgICAgICAgICAgIDEz MTA3MiwgICAgICAwLCAgICAgICAwLCAgICAgIDI1LDEwNzcwMDA5LCAgIDAsICAgMA0KY2Jsazg6 ICAgICAgICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgMjQsIDIxMzg4MTAs ICAgMCwgICAwDQpjYmxrOTogICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwg ICAgICAzMiwxMTQxMzU1OSwgICAwLCAgIDANCmNibGsxMDogICAgICAgICAgICAgIDEzMTA3Miwg ICAgICAwLCAgICAgICAwLCAgICAgIDM1LCAyNjg1ODQxLCAgIDAsICAgMA0KY2JsazExOiAgICAg ICAgICAgICAgMTMxMDcyLCAgICAgIDAsICAgICAgIDAsICAgICAgMjcsICAgMzkyMTAsICAgMCwg ICAwDQpjYmxrMTI6ICAgICAgICAgICAgICAxMzEwNzIsICAgICAgMCwgICAgICAgMCwgICAgICAy MiwgIDQwNDgxMSwgICAwLCAgIDANCmNibGsxMzogICAgICAgICAgICAgIDEzMTA3MiwgICAgICAw LCAgICAgICAwLCAgICAgIDIyLCAgICAyNTkyLCAgIDAsICAgMA0K --001a1145eae834d0fc051ac04b07-- From owner-freebsd-fs@freebsd.org Mon Jul 13 11:59:19 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8712F99BB5E for ; Mon, 13 Jul 2015 11:59:19 +0000 (UTC) (envelope-from karl@denninger.net) Received: from fs.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "NewFS.denninger.net", Issuer "NewFS.denninger.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1BCC1DE8 for ; Mon, 13 Jul 2015 11:59:17 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (localhost [127.0.0.1]) by fs.denninger.net (8.14.9/8.14.8) with ESMTP id t6DBx8rh005817 for ; Mon, 13 Jul 2015 06:59:08 -0500 (CDT) (envelope-from karl@denninger.net) Received: from [192.168.1.40] [192.168.1.40] (Via SSLv3 AES128-SHA) ; by Spamblock-sys (LOCAL/AUTH) Mon Jul 13 06:59:08 2015 Message-ID: <55A3A800.5060904@denninger.net> Date: Mon, 13 Jul 2015 06:58:56 -0500 From: Karl Denninger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: FreeBSD 10.1 Memory Exhaustion References: In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms030401070204080609030509" X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 11:59:19 -0000 This is a cryptographically signed message in MIME format. --------------ms030401070204080609030509 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Put this on your box and see if the problem goes away.... :-) https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D187594 The 2015-02-10 refactor will apply against 10.1-STABLE and 10.2-PRE (the latter will give you a 10-line fuzz in one block but applies and works.) I've been unable to provoke misbehavior with this patch in and I run a cron job that does auto-snapshotting. There are others that have run this patch with similarly positive results. On 7/13/2015 06:48, Christopher Forgeron wrote: > TL;DR Summary: I can run FreeBSD out of memory quite consistently, and = it=92s > not a TOS/mbuf exhaustion issue. It=92s quite possible that ZFS is the > culprit, but shouldn=92t the pager be able to handle aggressive memory > requests in a low memory situation gracefully, without needing custom > tuning of ZFS / VM? > > > Hello, > > I=92ve been dealing with some instability in my 10.1-RELEASE and > STABLEr282701M machines for the last few months. > > These machines are NFS/iSCSI storage machines, running on Dell M610x or= > similar hardware, 96 Gig Memory, 10Gig Network Cards, dual Xeon Process= ors > =96 Fairly beefy stuff. > > Initially I thought it was more issues with TOS / jumbo mbufs, as I had= > this problem last year. I had thought that this was properly resolved, = but > setting my MTU to 1500, and turning off TOS did give me a bit more > stability. Currently all my machines are set this way. > > Crashes were usually represented by loss of network connectivity, and t= he > ctld daemon scrolling messages across the screen at full speed about lo= st > connections. > > All of this did seem like more network stack problems, but with each cr= ash > I=92d be able to learn a bit more. > > Usually there was nothing of any use in the logfile, but every now and = then > I=92d get this: > > Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory > Jun 3 13:02:04 san0 kernel: WARNING: icl_pdu_new: failed to allocate 8= 0 > bytes > Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory > Jun 3 13:02:04 san0 kernel: WARNING: icl_pdu_new: failed to allocate 8= 0 > bytes > Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory > --------- > Jun 4 03:03:09 san0 kernel: WARNING: icl_pdu_new: failed to allocate 8= 0 > bytes > Jun 4 03:03:09 san0 kernel: WARNING: icl_pdu_new: failed to allocate 8= 0 > bytes > Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory > Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 > (iqn.1998-01.com.vmware:esx5a-3387a188): connection error; dropping > connection > Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 > (iqn.1998-01.com.vmware:esx5a-3387a188): connection error; dropping > connection > Jun 4 03:03:10 san0 kernel: WARNING: 172.16.0.97 > (iqn.1998-01.com.vmware:esx5a-3387a188): waiting for CTL to terminate > tasks, 1 remaining > Jun 4 06:04:27 san0 syslogd: kernel boot file is /boot/kernel/kernel > > So knowing that it seemed to be running out of memory, I started leavin= g > leaving =91vmstat 5=92 running on a console, to see what it was display= ing > during the crash. > > It was always the same thing: > > 0 0 0 1520M 4408M 15 0 0 0 25 19 0 0 21962 1667 91= 390 > 0 33 67 > 0 0 0 1520M 4310M 9 0 0 0 2 15 3 0 21527 1385 95= 165 > 0 31 69 > 0 0 0 1520M 4254M 7 0 0 0 14 19 0 0 17664 1739 72= 873 > 0 18 82 > 0 0 0 1520M 4145M 2 0 0 0 0 19 0 0 23557 1447 96= 941 > 0 36 64 > 0 0 0 1520M 4013M 4 0 0 0 14 19 0 0 4288 490 346= 85 > 0 72 28 > 0 0 0 1520M 3885M 2 0 0 0 0 19 0 0 11141 1038 69= 242 > 0 52 48 > 0 0 0 1520M 3803M 10 0 0 0 14 19 0 0 24102 1834 91= 050 > 0 33 67 > 0 0 0 1520M 8192B 2 0 0 0 2 15 1 0 19037 1131 77= 470 > 0 45 55 > 0 0 0 1520M 8192B 0 22 0 0 2 0 6 0 146 82 57= 8 0 > 0 100 > 0 0 0 1520M 8192B 1 0 0 0 0 0 0 0 130 40 51= 0 0 > 0 100 > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 143 40 50= 1 0 > 0 100 > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 201 62 66= 0 0 > 0 100 > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 101 28 40= 4 0 > 0 100 > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 97 27 39= 8 0 > 0 100 > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 93 28 37= 7 0 > 0 100 > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 92 27 37= 3 0 > 0 100 > > > I=92d go from a decent amount of free memory to suddenly having none. = Vmstat > would stop outputting, console commands would hang, etc. The whole syst= em > would be useless. > > Looking into this, I came across a similar issue; > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199189 > > I started increasing v.v_free_min, and it helped =96 My crashes went fr= om > being ~every 6 hours to every few days. > > Currently I=92m running with vm.v_free_min=3D1254507 =96 That=92s (1254= 507 * 4KiB) > , or 4.78GiB of Reserve. The vmstat above is of a machine with that > setting still running to 8B of memory. > > I have two issues here: > > 1) I don=92t think I should ever be able to run the system into the gro= und on > memory. Deny me new memory until the pager can free more. > 2) Setting =91min=92 doesn=92t really mean =91min=92 as it can obviousl= y go below > that threshold. > > > I have plenty of local UFS swap (non-ZFS drives) > > Adrian requested that I output a few more diagnostic items, and this i= s > what I=92m running on a console now, in a loop: > > vmstat > netstat -m > vmstat -z > sleep 1 > > The output of four crashes are attached here, as they can be a bit long= =2E > Let me know if that=92s not a good way to report them. They will each s= tart > mid-way through a vmstat =96z output, as that=92s as far back as my ter= minal > buffer allows. > > > > Now, I have a good idea of the conditions that are causing this: ZFS > Snapshots, run by cron, during times of high ZFS writes. > > The crashes are all nearly on the hour, as that=92s when crontab trigge= rs my > python scripts to make new snapshots, and delete old ones. > > My average FreeBSD machine has ~ 30 zfs datasets, with each pool having= ~20 > TiB used. These all need to snapshot on the hour. > > By staggering the snapshots by a few minutes, I have been able to reduc= e > crashing from every other day to perhaps once a week if I=92m lucky =96= But if > I start moving a lot of data around, I can cause daily crashes again. > > It=92s looking to be the memory demand of snapshotting lots of ZFS data= sets > at the same time while accepting a lot of write traffic. > > Now perhaps the answer is =91don=92t do that=92 but I feel that FreeBSD= should be > robust enough to handle this. I don=92t mind tuning for now to > reduce/eliminate this, but others shouldn=92t run into this pain just b= ecause > they heavily load their machines =96 There must be a way of avoiding th= is > condition. > > Here are the contents of my /boot/loader.conf and sysctl.conf, so show = my > minimal tuning to make this problem a little more bearable: > > /boot/loader.conf > vfs.zfs.arc_meta_limit=3D49656727553 > vfs.zfs.arc_max =3D 91489280512 > > /etc/sysctl.conf > vm.v_free_min=3D1254507 > > > Any suggestions/help is appreciated. > > Thank you. > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms030401070204080609030509 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGXzCC BlswggRDoAMCAQICASkwDQYJKoZIhvcNAQELBQAwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEg U3lzdGVtcyBMTEMgQ0EwHhcNMTUwNDIxMDIyMTU5WhcNMjAwNDE5MDIyMTU5WjBaMQswCQYD VQQGEwJVUzEQMA4GA1UECBMHRmxvcmlkYTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEe MBwGA1UEAxMVS2FybCBEZW5uaW5nZXIgKE9DU1ApMIICIjANBgkqhkiG9w0BAQEFAAOCAg8A MIICCgKCAgEAuYRY+EB2mGtZ3grlVO8TmnEvduVFA/IYXcCmNSOC1q+pTVjylsjcHKBcOPb9 TP1KLxdWP+Q1soSORGHlKw2/HcVzShDW5WPIKrvML+Ry0XvIvNBu9adTiCsA9nci4Cnf98XE hVpenER0qbJkBUOGT1rP4iAcfjet0lEgzPEnm+pAxv6fYSNp1WqIY9u0b1pkQiaWrt8hgNOc rJOiLbc8CeQ/DBP6rUiQjYNO9/aPNauEtHkNNfR9RgLSfGUdZuOCmJqnIla1HsrZhA5p69Bv /e832BKiNPaH5wF6btAiPpTr2sRhwQO8/IIxcRX1Vxd1yZbjYtJGw+9lwEcWRYAmoxkzKLPi S6Zo/6z5wgNpeK1H+zOioMoZIczgI8BlX1iHxqy/FAvm4PHPnC8s+BLnJLwr+jvMNHm82QwL J9hC5Ho8AnFU6TkCuq+P2V8/clJVqnBuvTUKhYMGSm4mUp+lAgR4L+lwIEqSeWVsxirIcE7Z OKkvI7k5x3WeE3+c6w74L6PfWVAd84xFlo9DKRdU9YbkFuFZPu21fi/LmE5brImB5P+jdqnK eWnVwRq+RBFLy4kehCzMXooitAwgP8l/JJa9VDiSyd/PAHaVGiat2vCdDh4b8cFL7SV6jPA4 k0MgGUA/6Et7wDmhZmCigggr9K6VQCx8jpKB3x1NlNNiaWECAwEAAaOB9DCB8TA3BggrBgEF BQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNV HRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIFoDALBgNVHQ8EBAMCBeAwLAYJYIZIAYb4QgENBB8W HU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1UdDgQWBBTFHJQt6cloXBdG1Pv1 o2YgH+7lWTAfBgNVHSMEGDAWgBQkcZudhX383d29sMqSlAOh+tNtNTAdBgNVHREEFjAUgRJr YXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcNAQELBQADggIBAE9/dxi2YqjCYYhiybp4GKcm 7tBVa/GLW+qcHPcoT4dqmqghlLz8+iUH+HCJjRQATVGyMEnvISOKFVHC6aZIG+Sg7J8bfS4+ fjKDi9smRH2VPPx3bV8+yFYRNroMGHaPHZB/Xctmmvc+PZ9O2W7rExgrODtxIOB3Zs6wkYf+ ty+9r1KmTHlV+rRHI6timH1uiyFE3cPi1taAEBxf0851cJV8k40PGF8G48ewnq8SY9sCf5cv liXbpdgU+I4ND5BuTjg63WS32zuhLd1VSuH3ZC/QbcncMX5W3oLXmcQP5/5uTiBJy74kdPtG MSZ9rXwZPwNxP/8PXMSR7ViaFvjUkf4bJlyENFa2PGxLk4EUzOuO7t3brjMlQW1fuInfG+ko 3tVxko20Hp0tKGPe/9cOxBVBZeZH/VgpZn3cLculGzZjmdh2fqAQ6kv9Z9AVOG1+dq0c1zt8 2zm+Oi1pikGXkfz5UJq60psY6zbX25BuEZkthO/qiS4pxjxb7gQkS0rTEHTy+qv0l3QVL0wa NAT74Zaj7l5DEW3qdQQ0dtVieyvptg9CxkfQJE3JyBMb0zBj9Qhc5/hbTfhSlHzZMEbUuIyx h9vxqFAmGzfB1/WfOKkiNHChkpPW8ZeH9yPeDBKvrgZ96dREHFoVkDk7Vpw5lSM+tFOfdyLg xxhb/RZVUDeUMYIE4zCCBN8CAQEwgZYwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQIEwdGbG9y aWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBMTEMxHDAa BgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEgU3lzdGVt cyBMTEMgQ0ECASkwCQYFKw4DAhoFAKCCAiEwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAc BgkqhkiG9w0BCQUxDxcNMTUwNzEzMTE1ODU2WjAjBgkqhkiG9w0BCQQxFgQUn7/qalbe53fp ZqLqyBH8XFk3o7owbAYJKoZIhvcNAQkPMV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAEC MAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzAN BggqhkiG9w0DAgIBKDCBpwYJKwYBBAGCNxAEMYGZMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBAgEpMIGpBgsqhkiG9w0BCRACCzGBmaCBljCBkDELMAkGA1UE BhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQ Q3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqG SIb3DQEJARYTQ3VkYSBTeXN0ZW1zIExMQyBDQQIBKTANBgkqhkiG9w0BAQEFAASCAgBZkAw+ z2e0+ZzQkpD4BDW9dWBwV0ZZVpMq9/80TKuILBKjDSDM3Bto3mMtBZ+lO9y163Syx1q0oex1 znKkAvL5P9LExozyp2F2Y+cqDfZWZEBmo8gkyvyoYRiYYj87tzjjOCUw7MoTaj3YTqyQyY/8 TnRYrljNjqUIIcI++fC8jtce6kGUpHVb/G3/ZrPaSzR8KxkRfHxgC1L1g4VBfOFKguCOPEQt RSTqkK/qnN0KRKEFdZsIu1lTSlD8j01D7gfoIFSR2z8aPf36zkx9/7jtWnptWhc9dy/+dgeG d/ewZDn0QisNj8jYdIFCAfbe8Vq68cI1bgkm3X2QP2yHFjv4jpYbH6BAQYK84j5YiqFEanoK RGpXOBP4PfepHENY9cLQS6Fz31Zolz01R+/SxA6yLWqwl5AD03fcCQs0ixJj7SjzTbKv91WY eRE8Wbq3TZz7AqWiOyCXljFJHxYW/M9avNqMxaOoYYhiVElzVN0t3dVDdOnBEIIT8WzJiv3D b75WMt7P+0w8J7hMxeUR1+kurN4KaW5gFyz8eUOm1IYusjuas108lcwMKOTwEnNVvRDx4AnF uwPWbNPQVWJVdANAGBJhwugISZ+bYrv9oWBvnPKHiaESuwnf6P1HEmz7xdW6P4iOgc1HVzfj clfmMlFBWCM56bUsKxPkOeFs/UuxvAAAAAAAAA== --------------ms030401070204080609030509-- From owner-freebsd-fs@freebsd.org Mon Jul 13 12:48:40 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD92A999D84 for ; Mon, 13 Jul 2015 12:48:39 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: from mail-qk0-x234.google.com (mail-qk0-x234.google.com [IPv6:2607:f8b0:400d:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9048715DD for ; Mon, 13 Jul 2015 12:48:39 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: by qkcl188 with SMTP id l188so69617418qkc.1 for ; Mon, 13 Jul 2015 05:48:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=up/wkHdkiGODLQUC5Qo+uI9pr5vjhVhdM32CN4I3B0w=; b=q2mSA0s3npuaWYM56YAQ0VowSft3fuTEqWvZy0fQoIyGvmS/3nBSunIUG8CxTr+DXs 8wpPoF3iNLEsGvp2m4bO33KoFI0EEzb4M0UEojsBpt8h6aMY9sOQtN7NSStz+gyZwTjQ aONtvB424BbSj2sdH5+jmtUMiFyCc9yPQC4XK8CctHyMDKyfg49wHR88IUfFxpZVS5Ct 69KBHaGKLdK4d49zDyVi3M3ACso5Jfo2FwFZqdNM6HoXalsjHhOfezfL7/wxXZNR1dWH BsfiK7x5G+Isysp7/WNheSq/FINtGua+HzTx/omJ/mXYwcuwy5amZO5yrm2idgLmlfTx f9uw== MIME-Version: 1.0 X-Received: by 10.140.239.136 with SMTP id k130mr55655180qhc.90.1436791718476; Mon, 13 Jul 2015 05:48:38 -0700 (PDT) Received: by 10.96.174.39 with HTTP; Mon, 13 Jul 2015 05:48:38 -0700 (PDT) In-Reply-To: <55A3A800.5060904@denninger.net> References: <55A3A800.5060904@denninger.net> Date: Mon, 13 Jul 2015 09:48:38 -0300 Message-ID: Subject: Re: FreeBSD 10.1 Memory Exhaustion From: Christopher Forgeron To: Karl Denninger Cc: FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 12:48:40 -0000 Hey, thanks, that's sounding exactly like my issue. I'll patch and build that kernel this week and have it up for testing on at least one of my boxes. While I agree that ZFS is the instigator of this behaviour, shouldn't the kernel protect against this type of behaviour? What would happen if a web server started chewing memory aggressively due to a DoS attack? On Mon, Jul 13, 2015 at 8:58 AM, Karl Denninger wrote: > Put this on your box and see if the problem goes away.... :-) > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D187594 > > The 2015-02-10 refactor will apply against 10.1-STABLE and 10.2-PRE (the > latter will give you a 10-line fuzz in one block but applies and works.) > > I've been unable to provoke misbehavior with this patch in and I run a > cron job that does auto-snapshotting. There are others that have run > this patch with similarly positive results. > > On 7/13/2015 06:48, Christopher Forgeron wrote: > > TL;DR Summary: I can run FreeBSD out of memory quite consistently, and > it=E2=80=99s > > not a TOS/mbuf exhaustion issue. It=E2=80=99s quite possible that ZFS i= s the > > culprit, but shouldn=E2=80=99t the pager be able to handle aggressive m= emory > > requests in a low memory situation gracefully, without needing custom > > tuning of ZFS / VM? > > > > > > Hello, > > > > I=E2=80=99ve been dealing with some instability in my 10.1-RELEASE and > > STABLEr282701M machines for the last few months. > > > > These machines are NFS/iSCSI storage machines, running on Dell M610x or > > similar hardware, 96 Gig Memory, 10Gig Network Cards, dual Xeon > Processors > > =E2=80=93 Fairly beefy stuff. > > > > Initially I thought it was more issues with TOS / jumbo mbufs, as I had > > this problem last year. I had thought that this was properly resolved, > but > > setting my MTU to 1500, and turning off TOS did give me a bit more > > stability. Currently all my machines are set this way. > > > > Crashes were usually represented by loss of network connectivity, and t= he > > ctld daemon scrolling messages across the screen at full speed about lo= st > > connections. > > > > All of this did seem like more network stack problems, but with each > crash > > I=E2=80=99d be able to learn a bit more. > > > > Usually there was nothing of any use in the logfile, but every now and > then > > I=E2=80=99d get this: > > > > Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 > > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory > > Jun 3 13:02:04 san0 kernel: WARNING: icl_pdu_new: failed to allocate 8= 0 > > bytes > > Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 > > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory > > Jun 3 13:02:04 san0 kernel: WARNING: icl_pdu_new: failed to allocate 8= 0 > > bytes > > Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 > > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory > > --------- > > Jun 4 03:03:09 san0 kernel: WARNING: icl_pdu_new: failed to allocate 8= 0 > > bytes > > Jun 4 03:03:09 san0 kernel: WARNING: icl_pdu_new: failed to allocate 8= 0 > > bytes > > Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 > > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory > > Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 > > (iqn.1998-01.com.vmware:esx5a-3387a188): connection error; dropping > > connection > > Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 > > (iqn.1998-01.com.vmware:esx5a-3387a188): connection error; dropping > > connection > > Jun 4 03:03:10 san0 kernel: WARNING: 172.16.0.97 > > (iqn.1998-01.com.vmware:esx5a-3387a188): waiting for CTL to terminate > > tasks, 1 remaining > > Jun 4 06:04:27 san0 syslogd: kernel boot file is /boot/kernel/kernel > > > > So knowing that it seemed to be running out of memory, I started leavin= g > > leaving =E2=80=98vmstat 5=E2=80=99 running on a console, to see what it= was displaying > > during the crash. > > > > It was always the same thing: > > > > 0 0 0 1520M 4408M 15 0 0 0 25 19 0 0 21962 1667 > 91390 > > 0 33 67 > > 0 0 0 1520M 4310M 9 0 0 0 2 15 3 0 21527 1385 > 95165 > > 0 31 69 > > 0 0 0 1520M 4254M 7 0 0 0 14 19 0 0 17664 1739 > 72873 > > 0 18 82 > > 0 0 0 1520M 4145M 2 0 0 0 0 19 0 0 23557 1447 > 96941 > > 0 36 64 > > 0 0 0 1520M 4013M 4 0 0 0 14 19 0 0 4288 490 346= 85 > > 0 72 28 > > 0 0 0 1520M 3885M 2 0 0 0 0 19 0 0 11141 1038 > 69242 > > 0 52 48 > > 0 0 0 1520M 3803M 10 0 0 0 14 19 0 0 24102 1834 > 91050 > > 0 33 67 > > 0 0 0 1520M 8192B 2 0 0 0 2 15 1 0 19037 1131 > 77470 > > 0 45 55 > > 0 0 0 1520M 8192B 0 22 0 0 2 0 6 0 146 82 > 578 0 > > 0 100 > > 0 0 0 1520M 8192B 1 0 0 0 0 0 0 0 130 40 > 510 0 > > 0 100 > > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 143 40 > 501 0 > > 0 100 > > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 201 62 > 660 0 > > 0 100 > > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 101 28 > 404 0 > > 0 100 > > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 97 27 > 398 0 > > 0 100 > > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 93 28 > 377 0 > > 0 100 > > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 92 27 > 373 0 > > 0 100 > > > > > > I=E2=80=99d go from a decent amount of free memory to suddenly having = none. > Vmstat > > would stop outputting, console commands would hang, etc. The whole syst= em > > would be useless. > > > > Looking into this, I came across a similar issue; > > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199189 > > > > I started increasing v.v_free_min, and it helped =E2=80=93 My crashes w= ent from > > being ~every 6 hours to every few days. > > > > Currently I=E2=80=99m running with vm.v_free_min=3D1254507 =E2=80=93 Th= at=E2=80=99s (1254507 * > 4KiB) > > , or 4.78GiB of Reserve. The vmstat above is of a machine with that > > setting still running to 8B of memory. > > > > I have two issues here: > > > > 1) I don=E2=80=99t think I should ever be able to run the system into t= he ground > on > > memory. Deny me new memory until the pager can free more. > > 2) Setting =E2=80=98min=E2=80=99 doesn=E2=80=99t really mean =E2=80=98m= in=E2=80=99 as it can obviously go below > > that threshold. > > > > > > I have plenty of local UFS swap (non-ZFS drives) > > > > Adrian requested that I output a few more diagnostic items, and this i= s > > what I=E2=80=99m running on a console now, in a loop: > > > > vmstat > > netstat -m > > vmstat -z > > sleep 1 > > > > The output of four crashes are attached here, as they can be a bit long= . > > Let me know if that=E2=80=99s not a good way to report them. They will = each start > > mid-way through a vmstat =E2=80=93z output, as that=E2=80=99s as far ba= ck as my terminal > > buffer allows. > > > > > > > > Now, I have a good idea of the conditions that are causing this: ZFS > > Snapshots, run by cron, during times of high ZFS writes. > > > > The crashes are all nearly on the hour, as that=E2=80=99s when crontab = triggers > my > > python scripts to make new snapshots, and delete old ones. > > > > My average FreeBSD machine has ~ 30 zfs datasets, with each pool having > ~20 > > TiB used. These all need to snapshot on the hour. > > > > By staggering the snapshots by a few minutes, I have been able to reduc= e > > crashing from every other day to perhaps once a week if I=E2=80=99m luc= ky =E2=80=93 But > if > > I start moving a lot of data around, I can cause daily crashes again. > > > > It=E2=80=99s looking to be the memory demand of snapshotting lots of ZF= S datasets > > at the same time while accepting a lot of write traffic. > > > > Now perhaps the answer is =E2=80=98don=E2=80=99t do that=E2=80=99 but I= feel that FreeBSD should > be > > robust enough to handle this. I don=E2=80=99t mind tuning for now to > > reduce/eliminate this, but others shouldn=E2=80=99t run into this pain = just > because > > they heavily load their machines =E2=80=93 There must be a way of avoid= ing this > > condition. > > > > Here are the contents of my /boot/loader.conf and sysctl.conf, so show = my > > minimal tuning to make this problem a little more bearable: > > > > /boot/loader.conf > > vfs.zfs.arc_meta_limit=3D49656727553 > > vfs.zfs.arc_max =3D 91489280512 > > > > /etc/sysctl.conf > > vm.v_free_min=3D1254507 > > > > > > Any suggestions/help is appreciated. > > > > Thank you. > > > > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > -- > Karl Denninger > karl@denninger.net > /The Market Ticker/ > /[S/MIME encrypted email preferred]/ > From owner-freebsd-fs@freebsd.org Mon Jul 13 13:44:15 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A51BE99B9D3 for ; Mon, 13 Jul 2015 13:44:15 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (smtp.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.infracaninophile.co.uk", Issuer "infracaninophile.co.uk" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 39D341EE6 for ; Mon, 13 Jul 2015 13:44:14 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from host-4-75.office.adestra.com (vpn-1.adestra.com [46.236.37.122]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.15.1/8.15.1) with ESMTPSA id t6DDi8tm009707 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Mon, 13 Jul 2015 14:44:09 +0100 (BST) (envelope-from m.seaman@infracaninophile.co.uk) Authentication-Results: smtp.infracaninophile.co.uk; dmarc=none header.from=infracaninophile.co.uk DKIM-Filter: OpenDKIM Filter v2.9.2 smtp.infracaninophile.co.uk t6DDi8tm009707 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=infracaninophile.co.uk; s=201001-infracaninophile; t=1436795049; bh=wp02A6PhrZfwj2cqjQdhS7v3WuG1pqF+m/l1qG5bubQ=; h=Date:From:To:Subject:References:In-Reply-To; z=Date:=20Mon,=2013=20Jul=202015=2014:44:01=20+0100|From:=20Matthew =20Seaman=20|To:=20freebsd-fs@fre ebsd.org|Subject:=20Re:=20FreeBSD=2010.1=20Memory=20Exhaustion|Ref erences:=20=20<55A3A800.5060904@denninger.net>|In-Reply-To:=20<5 5A3A800.5060904@denninger.net>; b=cR57LFY0RYsiNUgt/AvYruo9I1cJtkZzzQaYVx5/CacnbptniL5Am1zWCXVz0RVcX Ai7AqcTEd+u/k63MrNQYfD5yHiljcfN+st8pvrH9AIIuYfdBYrR7GcakvYZndNv1hI 6nPWom1OM565iJUOc188iace2fsrPEeAqwAyhw8w= X-Authentication-Warning: lucid-nonsense.infracaninophile.co.uk: Host vpn-1.adestra.com [46.236.37.122] claimed to be host-4-75.office.adestra.com Message-ID: <55A3C0A1.7040505@infracaninophile.co.uk> Date: Mon, 13 Jul 2015 14:44:01 +0100 From: Matthew Seaman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: FreeBSD 10.1 Memory Exhaustion References: <55A3A800.5060904@denninger.net> In-Reply-To: <55A3A800.5060904@denninger.net> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Bi5uDFNfv6uhNo3bRv1e3b7FJh05FUkAC" X-Virus-Scanned: clamav-milter 0.98.7 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-1.5 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on lucid-nonsense.infracaninophile.co.uk X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 13:44:15 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Bi5uDFNfv6uhNo3bRv1e3b7FJh05FUkAC Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2015/07/13 12:58, Karl Denninger wrote: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D187594 >=20 > The 2015-02-10 refactor will apply against 10.1-STABLE and 10.2-PRE (th= e > latter will give you a 10-line fuzz in one block but applies and works.= ) Is there a version that will apply to releng/10.1 ? I'm getting: backup-4:/usr/src/sys:# svn patch ~matthew/patch.10-1.STABLE C cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > applied hunk @@ -190,6 +190,15 @@ with offset -1 > applied hunk @@ -2697,11 +2718,29 @@ with offset -97 > applied hunk @@ -2889,6 +2928,19 @@ with offset -54 > rejected hunk @@ -212,7 +221,7 @@ > rejected hunk @@ -233,7 +242,9 @@ > rejected hunk @@ -2635,6 +2646,28 @@ > rejected hunk @@ -2666,19 +2699,7 @@ U cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c > applied hunk @@ -1058,7 +1060,7 @@ with offset -10 > applied hunk @@ -1070,11 +1072,11 @@ with offset -10 > applied hunk @@ -1279,6 +1281,7 @@ with offset -10 > applied hunk @@ -1291,6 +1294,42 @@ with offset -10 > applied hunk @@ -1321,7 +1360,7 @@ with offset -10 U cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c > applied hunk @@ -547,7 +550,7 @@ with offset -20 > applied hunk @@ -726,7 +729,7 @@ with offset -20 U cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_pool.h Summary of conflicts: Text conflicts: 1 If not, I'll just put this box onto STABLE/10. Cheers, Matthew --Bi5uDFNfv6uhNo3bRv1e3b7FJh05FUkAC Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQJ8BAEBCgBmBQJVo8CnXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQxOUYxNTRFQ0JGMTEyRTUwNTQ0RTNGMzAw MDUxM0YxMEUwQTlFNEU3AAoJEABRPxDgqeTnvxUP/iMtRyZVjHrBNul7cNq5km3P rY3H16MdNabQ1DYLuBDdycOwizQ02ki2vFkAijsWDebAN8W9L3hP8fcisPBdxlUd AUjh020Jltjg/pF5340FO9mmZeMDaVKylyA8v+UwC2qdgD6IJjflHTAGKyLseETx Hwx5g2CNqMGAF+47mJ+tcJgQEyP+Y97lRVui3tUIVek9YjXp0332BG86mTQhWR0R XdcWiRdVqo2UszJWxhgP/zxpckaxTSyG+Tb1vpYVFZKVk2D4AVIHepqy91CUBOZ4 AgARMVtWFKN+Ul1TZCKppnL43HtUGgN1pTTDRcNnoqkBamAMSzwa1zzeEjYqLKRw v+4ZSThWjzlTS0XVPDZo7ZgADJ3Zy9DotrE29kGAbuiJCXMb4Xe27p+PJu3vaYCf 1Bs2mroUnBqDSxq0QJE+Xr4sc/fE0YOUT3cdDvIXdqlTGryXRXJYRROyo4qcfn8b mCElPbx2jelvI00rxJtdsFWfvDxe12iGYUd80Trod93epqLoUj6CJfXWYDEVrjCc wHiCXdRsisa7vNTQgVclhST5U8Lnskt0t/TgZpBZjVkn4gLrgETEGHlD7X0sVSW+ MXx5z8Z0KJTy2uu8DDufrRJ43Dbg7FMJun/vvKGiH48njPBOtmotkQUgwitaHWfj lRyyWfePFNUNWxKNPbFR =BLs+ -----END PGP SIGNATURE----- --Bi5uDFNfv6uhNo3bRv1e3b7FJh05FUkAC-- From owner-freebsd-fs@freebsd.org Mon Jul 13 13:55:50 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4827599BD97 for ; Mon, 13 Jul 2015 13:55:50 +0000 (UTC) (envelope-from karl@denninger.net) Received: from fs.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "NewFS.denninger.net", Issuer "NewFS.denninger.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id F0665AB3 for ; Mon, 13 Jul 2015 13:55:49 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (localhost [127.0.0.1]) by fs.denninger.net (8.14.9/8.14.8) with ESMTP id t6DDtlf2051218 for ; Mon, 13 Jul 2015 08:55:48 -0500 (CDT) (envelope-from karl@denninger.net) Received: from [192.168.1.40] [192.168.1.40] (Via SSLv3 AES128-SHA) ; by Spamblock-sys (LOCAL/AUTH) Mon Jul 13 08:55:48 2015 Message-ID: <55A3C357.5070209@denninger.net> Date: Mon, 13 Jul 2015 08:55:35 -0500 From: Karl Denninger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: FreeBSD 10.1 Memory Exhaustion References: <55A3A800.5060904@denninger.net> <55A3C0A1.7040505@infracaninophile.co.uk> In-Reply-To: <55A3C0A1.7040505@infracaninophile.co.uk> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms000009050200010404080408" X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 13:55:50 -0000 This is a cryptographically signed message in MIME format. --------------ms000009050200010404080408 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable The previous version of the patch MAY apply against RELENG/10.1. On 7/13/2015 08:44, Matthew Seaman wrote: > On 2015/07/13 12:58, Karl Denninger wrote: >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D187594 >> >> The 2015-02-10 refactor will apply against 10.1-STABLE and 10.2-PRE (t= he >> latter will give you a 10-line fuzz in one block but applies and works= =2E) > Is there a version that will apply to releng/10.1 ? I'm getting: > > backup-4:/usr/src/sys:# svn patch ~matthew/patch.10-1.STABLE > C cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c >> applied hunk @@ -190,6 +190,15 @@ with offset -1 >> applied hunk @@ -2697,11 +2718,29 @@ with offset -97 >> applied hunk @@ -2889,6 +2928,19 @@ with offset -54 >> rejected hunk @@ -212,7 +221,7 @@ >> rejected hunk @@ -233,7 +242,9 @@ >> rejected hunk @@ -2635,6 +2646,28 @@ >> rejected hunk @@ -2666,19 +2699,7 @@ > U cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c >> applied hunk @@ -1058,7 +1060,7 @@ with offset -10 >> applied hunk @@ -1070,11 +1072,11 @@ with offset -10 >> applied hunk @@ -1279,6 +1281,7 @@ with offset -10 >> applied hunk @@ -1291,6 +1294,42 @@ with offset -10 >> applied hunk @@ -1321,7 +1360,7 @@ with offset -10 > U cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c >> applied hunk @@ -547,7 +550,7 @@ with offset -20 >> applied hunk @@ -726,7 +729,7 @@ with offset -20 > U cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_pool.h > Summary of conflicts: > Text conflicts: 1 > > If not, I'll just put this box onto STABLE/10. > > Cheers, > > Matthew > > > --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms000009050200010404080408 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGXzCC BlswggRDoAMCAQICASkwDQYJKoZIhvcNAQELBQAwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEg U3lzdGVtcyBMTEMgQ0EwHhcNMTUwNDIxMDIyMTU5WhcNMjAwNDE5MDIyMTU5WjBaMQswCQYD VQQGEwJVUzEQMA4GA1UECBMHRmxvcmlkYTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEe MBwGA1UEAxMVS2FybCBEZW5uaW5nZXIgKE9DU1ApMIICIjANBgkqhkiG9w0BAQEFAAOCAg8A MIICCgKCAgEAuYRY+EB2mGtZ3grlVO8TmnEvduVFA/IYXcCmNSOC1q+pTVjylsjcHKBcOPb9 TP1KLxdWP+Q1soSORGHlKw2/HcVzShDW5WPIKrvML+Ry0XvIvNBu9adTiCsA9nci4Cnf98XE hVpenER0qbJkBUOGT1rP4iAcfjet0lEgzPEnm+pAxv6fYSNp1WqIY9u0b1pkQiaWrt8hgNOc rJOiLbc8CeQ/DBP6rUiQjYNO9/aPNauEtHkNNfR9RgLSfGUdZuOCmJqnIla1HsrZhA5p69Bv /e832BKiNPaH5wF6btAiPpTr2sRhwQO8/IIxcRX1Vxd1yZbjYtJGw+9lwEcWRYAmoxkzKLPi S6Zo/6z5wgNpeK1H+zOioMoZIczgI8BlX1iHxqy/FAvm4PHPnC8s+BLnJLwr+jvMNHm82QwL J9hC5Ho8AnFU6TkCuq+P2V8/clJVqnBuvTUKhYMGSm4mUp+lAgR4L+lwIEqSeWVsxirIcE7Z OKkvI7k5x3WeE3+c6w74L6PfWVAd84xFlo9DKRdU9YbkFuFZPu21fi/LmE5brImB5P+jdqnK eWnVwRq+RBFLy4kehCzMXooitAwgP8l/JJa9VDiSyd/PAHaVGiat2vCdDh4b8cFL7SV6jPA4 k0MgGUA/6Et7wDmhZmCigggr9K6VQCx8jpKB3x1NlNNiaWECAwEAAaOB9DCB8TA3BggrBgEF BQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNV HRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIFoDALBgNVHQ8EBAMCBeAwLAYJYIZIAYb4QgENBB8W HU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1UdDgQWBBTFHJQt6cloXBdG1Pv1 o2YgH+7lWTAfBgNVHSMEGDAWgBQkcZudhX383d29sMqSlAOh+tNtNTAdBgNVHREEFjAUgRJr YXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcNAQELBQADggIBAE9/dxi2YqjCYYhiybp4GKcm 7tBVa/GLW+qcHPcoT4dqmqghlLz8+iUH+HCJjRQATVGyMEnvISOKFVHC6aZIG+Sg7J8bfS4+ fjKDi9smRH2VPPx3bV8+yFYRNroMGHaPHZB/Xctmmvc+PZ9O2W7rExgrODtxIOB3Zs6wkYf+ ty+9r1KmTHlV+rRHI6timH1uiyFE3cPi1taAEBxf0851cJV8k40PGF8G48ewnq8SY9sCf5cv liXbpdgU+I4ND5BuTjg63WS32zuhLd1VSuH3ZC/QbcncMX5W3oLXmcQP5/5uTiBJy74kdPtG MSZ9rXwZPwNxP/8PXMSR7ViaFvjUkf4bJlyENFa2PGxLk4EUzOuO7t3brjMlQW1fuInfG+ko 3tVxko20Hp0tKGPe/9cOxBVBZeZH/VgpZn3cLculGzZjmdh2fqAQ6kv9Z9AVOG1+dq0c1zt8 2zm+Oi1pikGXkfz5UJq60psY6zbX25BuEZkthO/qiS4pxjxb7gQkS0rTEHTy+qv0l3QVL0wa NAT74Zaj7l5DEW3qdQQ0dtVieyvptg9CxkfQJE3JyBMb0zBj9Qhc5/hbTfhSlHzZMEbUuIyx h9vxqFAmGzfB1/WfOKkiNHChkpPW8ZeH9yPeDBKvrgZ96dREHFoVkDk7Vpw5lSM+tFOfdyLg xxhb/RZVUDeUMYIE4zCCBN8CAQEwgZYwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQIEwdGbG9y aWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBMTEMxHDAa BgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEgU3lzdGVt cyBMTEMgQ0ECASkwCQYFKw4DAhoFAKCCAiEwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAc BgkqhkiG9w0BCQUxDxcNMTUwNzEzMTM1NTM1WjAjBgkqhkiG9w0BCQQxFgQUsurODoiInUAt PXJEGp/UCa5LnWEwbAYJKoZIhvcNAQkPMV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAEC MAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzAN BggqhkiG9w0DAgIBKDCBpwYJKwYBBAGCNxAEMYGZMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBAgEpMIGpBgsqhkiG9w0BCRACCzGBmaCBljCBkDELMAkGA1UE BhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQ Q3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqG SIb3DQEJARYTQ3VkYSBTeXN0ZW1zIExMQyBDQQIBKTANBgkqhkiG9w0BAQEFAASCAgCPSQS7 sMfSP9XA9+yQIqqbReHZtSCV5h5hKXR5YKovuvjQqF85LqeOM6Qz/sFxRakDmdsIacKfXkum ca3nG9XG3FQ4vfeJMDThuEPyLiycLWrCHBNtc3xLiDawfUQtM122SE62hZuwW6+8U6ZRx1uR D5ubLKQzJkf+vwmlzAUUafeDncAYZLynF0tqWb2HNwGZU8yNbWAiYyGwyAqBz8QML4m73Djt 3bO10KCuDSjs6JV2dzIzgYTyLmA+0Guc3NgIw2mnVR/mu1zJSfRLkXb58rSPG7gpQYZK/bhj YhxpmwEvfOGBck8pOGjQ78g0bOfGNX5fpIG7CKAAidbxxKj6/CI6tl2lpyQGrZ5rIJwgl1/w z6YG0a/yQCj3kOz13bPJ6Q0H+GXVZUr8n4B6DrrOCvVRLzbEyV5lrueohgvHf6K9ck8554Ub fjX5PRhCWjker2lmjmLZ79ti+h/f4wwtHlDBXR4jR4QMnhmQfDO3ee/g5X/2t8ROVJ5u6aiN Wg6rShR1oe+EaSL152JUVSZPcK7Wf9SEYURRpfZ82eFiLHUcfRLWRDeA/bDZgL0OqNxAHUjh N5w5ZK5nKiiaNoqVYgDMBABcrKOJNPfD9mJSK/WtLFTaL4RF4PQboQ5LzHcwZ4+9UV447DcO Uj8o1P2nqvdHI3m/RHevjdtAr/gcBAAAAAAAAA== --------------ms000009050200010404080408-- From owner-freebsd-fs@freebsd.org Mon Jul 13 14:16:28 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2244A9993DD for ; Mon, 13 Jul 2015 14:16:28 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (smtp.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.infracaninophile.co.uk", Issuer "infracaninophile.co.uk" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 87E4419BE for ; Mon, 13 Jul 2015 14:16:27 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from host-4-75.office.adestra.com (vpn-1.adestra.com [46.236.37.122]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.15.1/8.15.1) with ESMTPSA id t6DEGK4M010398 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Mon, 13 Jul 2015 15:16:21 +0100 (BST) (envelope-from m.seaman@infracaninophile.co.uk) Authentication-Results: smtp.infracaninophile.co.uk; dmarc=none header.from=infracaninophile.co.uk DKIM-Filter: OpenDKIM Filter v2.9.2 smtp.infracaninophile.co.uk t6DEGK4M010398 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=infracaninophile.co.uk; s=201001-infracaninophile; t=1436796981; bh=BMOro6Dw3XZjWkkYWnm+G43FZtnMiLq4BN8X9pvxAq8=; h=Date:From:To:Subject:References:In-Reply-To; z=Date:=20Mon,=2013=20Jul=202015=2015:15:43=20+0100|From:=20Matthew =20Seaman=20|To:=20freebsd-fs@fre ebsd.org|Subject:=20Re:=20FreeBSD=2010.1=20Memory=20Exhaustion|Ref erences:=20=20<55A3A800.5060904@denninger.net>=20<55A3C0A1.70405 05@infracaninophile.co.uk>=20<55A3C357.5070209@denninger.net>|In-R eply-To:=20<55A3C357.5070209@denninger.net>; b=IK/dYcFHF9qRG4H1QbiZ+LxQtyE0U7k4knMhlddOAW2lQpeoCSwleHzFvB/wTU0rE v/wpNxnlTlQy0ALI1klm+zxxJN2Ufc7oDnV1WE4SAtDkFIIFZifo991QbDKNZW2mkr ry/U2L4Qm/8Dsuc7AEIIGuN6dJpsCYBJZ5V8QjY0= X-Authentication-Warning: lucid-nonsense.infracaninophile.co.uk: Host vpn-1.adestra.com [46.236.37.122] claimed to be host-4-75.office.adestra.com Message-ID: <55A3C80F.4060407@infracaninophile.co.uk> Date: Mon, 13 Jul 2015 15:15:43 +0100 From: Matthew Seaman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: FreeBSD 10.1 Memory Exhaustion References: <55A3A800.5060904@denninger.net> <55A3C0A1.7040505@infracaninophile.co.uk> <55A3C357.5070209@denninger.net> In-Reply-To: <55A3C357.5070209@denninger.net> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="SfocwpETkb4LOJEoTsmnFsWQUDHu8oFhk" X-Virus-Scanned: clamav-milter 0.98.7 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-1.5 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on lucid-nonsense.infracaninophile.co.uk X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 14:16:28 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --SfocwpETkb4LOJEoTsmnFsWQUDHu8oFhk Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2015/07/13 14:55, Karl Denninger wrote: > The previous version of the patch MAY apply against RELENG/10.1. Close.... backup-4:/usr/src:# svn patch ~matthew/arc_cache_reclaim_uma_stable10.pat= ch U sys/cddl/compat/opensolaris/kern/opensolaris_kmem.c U sys/cddl/compat/opensolaris/sys/kmem.h U sys/cddl/compat/opensolaris/sys/param.h U sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > applied hunk @@ -592,6 +687,13 @@ with offset 2 > applied hunk @@ -2421,6 +2523,7 @@ with offset 28 > applied hunk @@ -2429,6 +2532,9 @@ with offset 28 > applied hunk @@ -2439,12 +2545,19 @@ with offset 28 > applied hunk @@ -2454,16 +2567,26 @@ with offset 28 > applied hunk @@ -2491,8 +2614,19 @@ with offset 28 > applied hunk @@ -2503,26 +2637,43 @@ with offset 28 > applied hunk @@ -2529,6 +2680,7 @@ with offset 28 > applied hunk @@ -2537,7 +2689,7 @@ with offset 28 > applied hunk @@ -2552,20 +2704,131 @@ with offset 28 > applied hunk @@ -2577,10 +2840,36 @@ with offset 28 > applied hunk @@ -2588,6 +2877,7 @@ with offset 28 > applied hunk @@ -2602,6 +2892,7 @@ with offset 28 > applied hunk @@ -2618,6 +2909,7 @@ with offset 28 > applied hunk @@ -2692,6 +2984,7 @@ with offset 28 > applied hunk @@ -2713,20 +3006,6 @@ with offset 28 > applied hunk @@ -2763,6 +3042,9 @@ with offset 28 > applied hunk @@ -3885,20 +4167,16 @@ with offset 29 > applied hunk @@ -3911,7 +4189,7 @@ with offset 29 > applied hunk @@ -3939,8 +4217,10 @@ with offset 29 > applied hunk @@ -3994,6 +4274,7 @@ with offset 29 U sys/cddl/contrib/opensolaris/uts/common/fs/zfs/range_tree.c U sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c U sys/vm/uma.h U sys/vm/uma_core.c C sys/vm/vm_pageout.c > rejected hunk @@ -921,10 +932,11 @@ Summary of conflicts: Text conflicts: 1 =2E.. so close in fact that it's a fairly obvious merge even for someone like me. Cheers, Matthew --SfocwpETkb4LOJEoTsmnFsWQUDHu8oFhk Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQJ8BAEBCgBmBQJVo8g0XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQxOUYxNTRFQ0JGMTEyRTUwNTQ0RTNGMzAw MDUxM0YxMEUwQTlFNEU3AAoJEABRPxDgqeTnMUQP/jXYegwLvVLBlfau3ZqEYPJH 29N9KhqZZb/dgvWjnjuvfSq+G1TzjYeqlIyZfdJCGHUqwMl+2MCVV4erIy/802k9 VAfAnhFMvGLodj1HvxJFRPsGIkpLJcVhobKVLPQdGlsxVj/SnlkUxU98NwJg3ZGA WXCvfC7TLzoidx8CvpEiVIyBUG7hJBdcqUGX2GkYbD0wu3/1tbOaOGBKCmqyDFBM 0ilkW3gGmxBGMBpGgzR6XzAa9ewTQ8sJoME1Fp0+X8/rmZqu+P1bqLmlltHO2nUT z8LYBtCjAWrg4T+fgEGcWsIv4hwiNlL3b998+uVJmD83ThAkTkS8e0HmWJczJ/f6 iEHQ534j2/yoUkfPJy4WnuVjj7DbJXAEb07bWboSUV/SAfLf5tXZo1nCkgrcfWsw +NaDFSFf84p82A2lJdAgBVQL+HRrG6FPtbCxNJIQXr08+9ktobuN0d7pJB1Di1V7 G3fRRRLg7EjRDlGTUH8C4j9VHawjfIo2gdtw+hCusCH3eVfiTO3lt4iYJKTp9M/h Gdlwtbj2GaE7NovYrMhrGVLDwXQdwdcUy+5R5b+VSYgU0Vc4sNf08ChUl9inpri3 /+Kfg4JwP7rnHGC/wXfVhK0KisWO1PbHKdcHfl/yjS4Xcj1ekC0L97xghknuxjNE SW/1smgM1dBVxeXdjJqm =MkFo -----END PGP SIGNATURE----- --SfocwpETkb4LOJEoTsmnFsWQUDHu8oFhk-- From owner-freebsd-fs@freebsd.org Mon Jul 13 17:29:41 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2580C99CFFE; Mon, 13 Jul 2015 17:29:41 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x22d.google.com (mail-ig0-x22d.google.com [IPv6:2607:f8b0:4001:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E464318BD; Mon, 13 Jul 2015 17:29:40 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by igcqs7 with SMTP id qs7so62753165igc.0; Mon, 13 Jul 2015 10:29:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=t07gxu5c4lEz16l2iBPS8buAH8xas5O3GSbgjLbof0c=; b=UuFmMVTMOL7Xwlu9B4Wu8I/VpPDj/v/JzyKW8XQdMeBHaX6Rf7pTzpZrFiY0eKvWeO G94nkxS636GvrN5QzsjYWF2SktfMkGPzpdMph+Qw1wicB1E6WgTffZJkPLeufKc5h0th kzKts8n8sw7k4yLRp7dC5N7VxvbgAWoQBHlF8ZPE/rg2/2fhAVQO7ozh0X2MTLuvlTo+ miu8en+k0N6nOTPDNLoE543tIZWyTv9xJvZ9XFrs9kOY29EeKA925vgbJrAWUrWDQp5w xT9CLJAnl2NkAXQFGvn1jQiEKzcjhA5ko2zxMtaSnDlQFKzM7jDWYwFpq4F95zBuQJD9 /5Uw== MIME-Version: 1.0 X-Received: by 10.107.35.144 with SMTP id j138mr23062716ioj.105.1436808580237; Mon, 13 Jul 2015 10:29:40 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.38.133 with HTTP; Mon, 13 Jul 2015 10:29:40 -0700 (PDT) In-Reply-To: References: Date: Mon, 13 Jul 2015 10:29:40 -0700 X-Google-Sender-Auth: 8CZ9Nmttbd4B-ppWCvEV4KLnEBE Message-ID: Subject: Re: FreeBSD 10.1 Memory Exhaustion From: Adrian Chadd To: Christopher Forgeron Cc: FreeBSD Stable Mailing List , FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 17:29:41 -0000 hi, With that much storage and that many snapshots, I do think you need more than 96GB of RAM in the box. I'm hoping someone doing active ZFS work can comment.. I don't think the ZFS code is completely "memory usage" safe. The "old" Sun suggestions when I started using ZFS was "if your server panics due to out of memory with ZFS, buy more memory." That said, there doesn't look like there's a leak anywhere - those dumps show you're using at least 32gig on each just in zfs data buffers. Try tuning the ARC down a little? -adrian On 13 July 2015 at 04:48, Christopher Forgeron wrote= : > > > TL;DR Summary: I can run FreeBSD out of memory quite consistently, and it= =E2=80=99s > not a TOS/mbuf exhaustion issue. It=E2=80=99s quite possible that ZFS is = the > culprit, but shouldn=E2=80=99t the pager be able to handle aggressive mem= ory > requests in a low memory situation gracefully, without needing custom tun= ing > of ZFS / VM? > > > Hello, > > I=E2=80=99ve been dealing with some instability in my 10.1-RELEASE and > STABLEr282701M machines for the last few months. > > These machines are NFS/iSCSI storage machines, running on Dell M610x or > similar hardware, 96 Gig Memory, 10Gig Network Cards, dual Xeon Processor= s =E2=80=93 > Fairly beefy stuff. > > Initially I thought it was more issues with TOS / jumbo mbufs, as I had t= his > problem last year. I had thought that this was properly resolved, but > setting my MTU to 1500, and turning off TOS did give me a bit more > stability. Currently all my machines are set this way. > > Crashes were usually represented by loss of network connectivity, and the > ctld daemon scrolling messages across the screen at full speed about lost > connections. > > All of this did seem like more network stack problems, but with each cras= h > I=E2=80=99d be able to learn a bit more. > > Usually there was nothing of any use in the logfile, but every now and th= en > I=E2=80=99d get this: > > Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory > Jun 3 13:02:04 san0 kernel: WARNING: icl_pdu_new: failed to allocate 80 > bytes > Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory > Jun 3 13:02:04 san0 kernel: WARNING: icl_pdu_new: failed to allocate 80 > bytes > Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory > --------- > Jun 4 03:03:09 san0 kernel: WARNING: icl_pdu_new: failed to allocate 80 > bytes > Jun 4 03:03:09 san0 kernel: WARNING: icl_pdu_new: failed to allocate 80 > bytes > Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory > Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 > (iqn.1998-01.com.vmware:esx5a-3387a188): connection error; dropping > connection > Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 > (iqn.1998-01.com.vmware:esx5a-3387a188): connection error; dropping > connection > Jun 4 03:03:10 san0 kernel: WARNING: 172.16.0.97 > (iqn.1998-01.com.vmware:esx5a-3387a188): waiting for CTL to terminate tas= ks, > 1 remaining > Jun 4 06:04:27 san0 syslogd: kernel boot file is /boot/kernel/kernel > > So knowing that it seemed to be running out of memory, I started leaving > leaving =E2=80=98vmstat 5=E2=80=99 running on a console, to see what it w= as displaying > during the crash. > > It was always the same thing: > > 0 0 0 1520M 4408M 15 0 0 0 25 19 0 0 21962 1667 9139= 0 > 0 33 67 > 0 0 0 1520M 4310M 9 0 0 0 2 15 3 0 21527 1385 9516= 5 > 0 31 69 > 0 0 0 1520M 4254M 7 0 0 0 14 19 0 0 17664 1739 7287= 3 > 0 18 82 > 0 0 0 1520M 4145M 2 0 0 0 0 19 0 0 23557 1447 9694= 1 > 0 36 64 > 0 0 0 1520M 4013M 4 0 0 0 14 19 0 0 4288 490 34685= 0 > 72 28 > 0 0 0 1520M 3885M 2 0 0 0 0 19 0 0 11141 1038 6924= 2 > 0 52 48 > 0 0 0 1520M 3803M 10 0 0 0 14 19 0 0 24102 1834 9105= 0 > 0 33 67 > 0 0 0 1520M 8192B 2 0 0 0 2 15 1 0 19037 1131 7747= 0 > 0 45 55 > 0 0 0 1520M 8192B 0 22 0 0 2 0 6 0 146 82 578 = 0 > 0 100 > 0 0 0 1520M 8192B 1 0 0 0 0 0 0 0 130 40 510 = 0 > 0 100 > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 143 40 501 = 0 > 0 100 > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 201 62 660 = 0 > 0 100 > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 101 28 404 = 0 > 0 100 > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 97 27 398 = 0 > 0 100 > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 93 28 377 = 0 > 0 100 > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 92 27 373 = 0 > 0 100 > > > I=E2=80=99d go from a decent amount of free memory to suddenly having no= ne. Vmstat > would stop outputting, console commands would hang, etc. The whole system > would be useless. > > Looking into this, I came across a similar issue; > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199189 > > I started increasing v.v_free_min, and it helped =E2=80=93 My crashes wen= t from > being ~every 6 hours to every few days. > > Currently I=E2=80=99m running with vm.v_free_min=3D1254507 =E2=80=93 That= =E2=80=99s (1254507 * 4KiB) , > or 4.78GiB of Reserve. The vmstat above is of a machine with that settin= g > still running to 8B of memory. > > I have two issues here: > > 1) I don=E2=80=99t think I should ever be able to run the system into the= ground on > memory. Deny me new memory until the pager can free more. > 2) Setting =E2=80=98min=E2=80=99 doesn=E2=80=99t really mean =E2=80=98min= =E2=80=99 as it can obviously go below that > threshold. > > > I have plenty of local UFS swap (non-ZFS drives) > > Adrian requested that I output a few more diagnostic items, and this is > what I=E2=80=99m running on a console now, in a loop: > > vmstat > netstat -m > vmstat -z > sleep 1 > > The output of four crashes are attached here, as they can be a bit long. = Let > me know if that=E2=80=99s not a good way to report them. They will each s= tart > mid-way through a vmstat =E2=80=93z output, as that=E2=80=99s as far back= as my terminal > buffer allows. > > > > Now, I have a good idea of the conditions that are causing this: ZFS > Snapshots, run by cron, during times of high ZFS writes. > > The crashes are all nearly on the hour, as that=E2=80=99s when crontab tr= iggers my > python scripts to make new snapshots, and delete old ones. > > My average FreeBSD machine has ~ 30 zfs datasets, with each pool having ~= 20 > TiB used. These all need to snapshot on the hour. > > By staggering the snapshots by a few minutes, I have been able to reduce > crashing from every other day to perhaps once a week if I=E2=80=99m lucky= =E2=80=93 But if I > start moving a lot of data around, I can cause daily crashes again. > > It=E2=80=99s looking to be the memory demand of snapshotting lots of ZFS = datasets at > the same time while accepting a lot of write traffic. > > Now perhaps the answer is =E2=80=98don=E2=80=99t do that=E2=80=99 but I f= eel that FreeBSD should be > robust enough to handle this. I don=E2=80=99t mind tuning for now to > reduce/eliminate this, but others shouldn=E2=80=99t run into this pain ju= st because > they heavily load their machines =E2=80=93 There must be a way of avoidin= g this > condition. > > Here are the contents of my /boot/loader.conf and sysctl.conf, so show my > minimal tuning to make this problem a little more bearable: > > /boot/loader.conf > vfs.zfs.arc_meta_limit=3D49656727553 > vfs.zfs.arc_max =3D 91489280512 > > /etc/sysctl.conf > vm.v_free_min=3D1254507 > > > Any suggestions/help is appreciated. > > Thank you. > From owner-freebsd-fs@freebsd.org Mon Jul 13 17:30:13 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B422599B0B6; Mon, 13 Jul 2015 17:30:13 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x22e.google.com (mail-ig0-x22e.google.com [IPv6:2607:f8b0:4001:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 815CD1C24; Mon, 13 Jul 2015 17:30:13 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by igcqs7 with SMTP id qs7so62764868igc.0; Mon, 13 Jul 2015 10:30:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=RYPgV682VdUCmWvxSYdVC8gmZ/ZzN8XKGt7/4q80ETI=; b=balkdc8zedON1falyGHf3VeSgG2zhvXHSyFx9t2dR6K6jfDkw/5MwtH66wHLRqdGSb O1juo9eTEK3fWzxPJYhAHkwoo4i2Pr5GlHP/9Q9qu2hoEQvTw4fngXL8vvlhXtArf3uH oOEcIBShCwTCJ3iLoOPCad/5I73lf6whEcKFOCGP/UxQA8Y29MXmfeE1oq2eLLNHJ6ip HAWH5z4LYKRNHsqV8p0FUbjFsdilTSTKRlSQiUodJF20hRmhiXY/VcRMr15f+S6hqy// lXl+BjzDMdba4xOJeXjY24Ci/5i5GFr0PTZ6MP/NcX4QbyBTHG9epVvMGrErnHO1pyyY KjnA== MIME-Version: 1.0 X-Received: by 10.50.111.167 with SMTP id ij7mr14180226igb.49.1436808612831; Mon, 13 Jul 2015 10:30:12 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.38.133 with HTTP; Mon, 13 Jul 2015 10:30:12 -0700 (PDT) In-Reply-To: References: Date: Mon, 13 Jul 2015 10:30:12 -0700 X-Google-Sender-Auth: PrY1_B36uxAbNHWnO1CNtQxqluU Message-ID: Subject: Re: FreeBSD 10.1 Memory Exhaustion From: Adrian Chadd To: Christopher Forgeron Cc: FreeBSD Stable Mailing List , FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 17:30:13 -0000 .. and in one instance, you had 57gig of RAM allocated just to ZFS 65536 size buffers. -adrian From owner-freebsd-fs@freebsd.org Mon Jul 13 18:26:32 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1772499BCA7 for ; Mon, 13 Jul 2015 18:26:32 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: from mail-qk0-x231.google.com (mail-qk0-x231.google.com [IPv6:2607:f8b0:400d:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C77841D6 for ; Mon, 13 Jul 2015 18:26:31 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: by qkbp125 with SMTP id p125so260181636qkb.2 for ; Mon, 13 Jul 2015 11:26:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=md2ja3HttpbOpd6uKQ2emM7FliY5z2+EHyxeF5gAcyY=; b=Jj0s4yCeZgS/XCBq1KFJj1bR4gPtWFGvew3Z2T6L0R+6ew4QN+Lt94Bnxjxq1UKgge 5EbU/Qz65Fqe1dd+g9z/FBwzH3Ezy39ri/g3U3sXpNSvJYWUkcBx1kd0Ep0M44UFD9+D qYFP52lquXkZUQLOrLlOg0bP5d+EUQAuKXFPMe0Vgymqs5IAwjape2gcvPEj8unAdR5J JrqZxPh98Yv8boGP0ZewdfjFQYLmcX8X9RtpydRDQwmrtWg8zj7iHCI0jX8L0nsWU17t eSfTjonaHY7QdVQH4D3Ps4vNtph4QKMFLdzjfOAUmNxGHhsyvskq6WmeSuDJ+RZpVsW4 467g== MIME-Version: 1.0 X-Received: by 10.140.239.136 with SMTP id k130mr58366694qhc.90.1436811990801; Mon, 13 Jul 2015 11:26:30 -0700 (PDT) Received: by 10.96.174.39 with HTTP; Mon, 13 Jul 2015 11:26:30 -0700 (PDT) In-Reply-To: <55A3C80F.4060407@infracaninophile.co.uk> References: <55A3A800.5060904@denninger.net> <55A3C0A1.7040505@infracaninophile.co.uk> <55A3C357.5070209@denninger.net> <55A3C80F.4060407@infracaninophile.co.uk> Date: Mon, 13 Jul 2015 15:26:30 -0300 Message-ID: Subject: Re: FreeBSD 10.1 Memory Exhaustion From: Christopher Forgeron To: Matthew Seaman Cc: FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 18:26:32 -0000 I found the same thing - The older patch does apply on 10.1 with one line needing to be added manually. Did that, but I'm running into this when I try a make buildworld: /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2726:30: error: use of undeclared identifier 'freemem' if (zfs_arc_cache_target && freemem < zfs_arc_cache_target + btop(size)) { ^ /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2726:63: warning: implicit declaration of function 'btop' is invalid in C99 [-Wimplicit-function-declaration] if (zfs_arc_cache_target && freemem < zfs_arc_cache_target + btop(size)) { ^ /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2750:3: warning: implicit declaration of function 'kmem_cache_reap' is invalid in C99 [-Wimplicit-function-declaration] kmem_cache_reap(cache, zfs_arc_cache_free_max); ^ /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2812:14: warning: implicit declaration of function 'kmem_cache_free_size' is invalid in C99 [-Wimplicit-function-declaration] cachefree = kmem_cache_free_size(buf_cache) + I see freemem defined in sys/cddl/compat/opensolaris/sys/kmem.h, and I've done a make clean; make clean; make cleanworld; make depend before running make buildworld. What am I forgetting to make here? On Mon, Jul 13, 2015 at 11:15 AM, Matthew Seaman < m.seaman@infracaninophile.co.uk> wrote: > On 2015/07/13 14:55, Karl Denninger wrote: > > The previous version of the patch MAY apply against RELENG/10.1. > > Close.... > > backup-4:/usr/src:# svn patch ~matthew/arc_cache_reclaim_uma_stable10.patch > U sys/cddl/compat/opensolaris/kern/opensolaris_kmem.c > U sys/cddl/compat/opensolaris/sys/kmem.h > U sys/cddl/compat/opensolaris/sys/param.h > U sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > > applied hunk @@ -592,6 +687,13 @@ with offset 2 > > applied hunk @@ -2421,6 +2523,7 @@ with offset 28 > > applied hunk @@ -2429,6 +2532,9 @@ with offset 28 > > applied hunk @@ -2439,12 +2545,19 @@ with offset 28 > > applied hunk @@ -2454,16 +2567,26 @@ with offset 28 > > applied hunk @@ -2491,8 +2614,19 @@ with offset 28 > > applied hunk @@ -2503,26 +2637,43 @@ with offset 28 > > applied hunk @@ -2529,6 +2680,7 @@ with offset 28 > > applied hunk @@ -2537,7 +2689,7 @@ with offset 28 > > applied hunk @@ -2552,20 +2704,131 @@ with offset 28 > > applied hunk @@ -2577,10 +2840,36 @@ with offset 28 > > applied hunk @@ -2588,6 +2877,7 @@ with offset 28 > > applied hunk @@ -2602,6 +2892,7 @@ with offset 28 > > applied hunk @@ -2618,6 +2909,7 @@ with offset 28 > > applied hunk @@ -2692,6 +2984,7 @@ with offset 28 > > applied hunk @@ -2713,20 +3006,6 @@ with offset 28 > > applied hunk @@ -2763,6 +3042,9 @@ with offset 28 > > applied hunk @@ -3885,20 +4167,16 @@ with offset 29 > > applied hunk @@ -3911,7 +4189,7 @@ with offset 29 > > applied hunk @@ -3939,8 +4217,10 @@ with offset 29 > > applied hunk @@ -3994,6 +4274,7 @@ with offset 29 > U sys/cddl/contrib/opensolaris/uts/common/fs/zfs/range_tree.c > U sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c > U sys/vm/uma.h > U sys/vm/uma_core.c > C sys/vm/vm_pageout.c > > rejected hunk @@ -921,10 +932,11 @@ > Summary of conflicts: > Text conflicts: 1 > > ... so close in fact that it's a fairly obvious merge even for someone > like me. > > Cheers, > > Matthew > > > From owner-freebsd-fs@freebsd.org Mon Jul 13 18:34:19 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3A81A99BFA6 for ; Mon, 13 Jul 2015 18:34:19 +0000 (UTC) (envelope-from matthew@FreeBSD.org) Received: from smtp.infracaninophile.co.uk (smtp.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.infracaninophile.co.uk", Issuer "infracaninophile.co.uk" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B29EC9CF for ; Mon, 13 Jul 2015 18:34:18 +0000 (UTC) (envelope-from matthew@FreeBSD.org) Received: from liminal.local ([192.168.100.2]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.15.1/8.15.1) with ESMTPSA id t6DIYBHw015466 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 13 Jul 2015 19:34:11 +0100 (BST) (envelope-from matthew@FreeBSD.org) Authentication-Results: smtp.infracaninophile.co.uk; dmarc=none header.from=FreeBSD.org DKIM-Filter: OpenDKIM Filter v2.9.2 smtp.infracaninophile.co.uk t6DIYBHw015466 Authentication-Results: smtp.infracaninophile.co.uk/t6DIYBHw015466; dkim=none reason="no signature"; dkim-adsp=none; dkim-atps=neutral X-Authentication-Warning: lucid-nonsense.infracaninophile.co.uk: Host [192.168.100.2] claimed to be liminal.local Message-ID: <55A40498.80608@FreeBSD.org> Date: Mon, 13 Jul 2015 19:34:00 +0100 From: Matthew Seaman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Christopher Forgeron CC: FreeBSD Filesystems Subject: Re: FreeBSD 10.1 Memory Exhaustion References: <55A3A800.5060904@denninger.net> <55A3C0A1.7040505@infracaninophile.co.uk> <55A3C357.5070209@denninger.net> <55A3C80F.4060407@infracaninophile.co.uk> In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="XC87Jui1ipdOgDA08d1W1jUWMG3XngQ8m" X-Virus-Scanned: clamav-milter 0.98.7 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on lucid-nonsense.infracaninophile.co.uk X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 18:34:19 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --XC87Jui1ipdOgDA08d1W1jUWMG3XngQ8m Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 13/07/2015 19:26, Christopher Forgeron wrote: > I found the same thing - The older patch does apply on 10.1 with one li= ne > needing to be added manually. Did that, but I'm running into this when = I > try a make buildworld: I only did a buildkernel as the patch doesn't affect any of the userland code. Cheers, Matthew --XC87Jui1ipdOgDA08d1W1jUWMG3XngQ8m Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.20 (Darwin) iQJ8BAEBCgBmBQJVpASfXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ2NTNBNjhCOTEzQTRFNkNGM0UxRTEzMjZC QjIzQUY1MThFMUE0MDEzAAoJELsjr1GOGkATBLIP/0OO45yhA8RQi6c3MhlQ56yf fmg8Q9w4FgCgGbk8J7p+6NjIJ/tKgMW/hurhlDwNaOe539T0Be7f/jApvCA255f/ HEADZIRgzBn0FH9t8GHWrwOMlowYZTjATnAaSQagWubjT97100qT2x0qa/uEOHrC +Qv2yBNCdXrCBEq0SQ22WM9Yt145asA9Gv6GhD8YT4Xex0BI6FZsBaSPhxe4FDp+ sy/22XpQvl1cPW04k5C71xWjf87ADP4vqogWnLacoD7PVp/7JgSfrANZ69ij1XbE 7bswblXTU9u1t4dFjU5Mdud9/G8Dkkq9ucj35KSBzKEZFhyusGB0DtKtMRd12tMm Z/8nugDA3/N5fMNb+4YBhA9OxSMl6VeAW/9H2A96WLUfbLIXtKYPLQ8zT2ePTDrg iVV/SBqBpGtO6nuXtI4UMDAUSUDJ6OQgi0o5ct6PRtLuIZQn3o0MxA9t7Ntu2sr+ 5fDMUhqYcCldQ1bcTuj5maNemoFIyAXsMH8LK40OV8Hgzbod7xbLQ5fKfboBi22z QhoKw4Ca8BnCSn+sRNFk5j5fH7c0m2EDPPvw97umr/48Dcz66KZgwIJeE0CSGRlZ p5+WYMuS3/96KVGAVI6Npzi+1YQT/QkWBrvWHhXoAicS9RLZS7LtRGvDS3dQbK/6 GIaLOm3iibEtiSh+O0JH =aycZ -----END PGP SIGNATURE----- --XC87Jui1ipdOgDA08d1W1jUWMG3XngQ8m-- From owner-freebsd-fs@freebsd.org Mon Jul 13 23:22:38 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A58B7995B0B for ; Mon, 13 Jul 2015 23:22:38 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from mango.stankevitz.com (mango.stankevitz.com [208.79.93.194]) by mx1.freebsd.org (Postfix) with ESMTP id 95B33D4 for ; Mon, 13 Jul 2015 23:22:38 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from Chriss-MacBook-Pro.local (209-203-101-124.static.twtelecom.net [209.203.101.124]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mango.stankevitz.com (Postfix) with ESMTPSA id D90685EF for ; Mon, 13 Jul 2015 16:13:35 -0700 (PDT) Message-ID: <55A4461F.6020308@stankevitz.com> Date: Mon, 13 Jul 2015 16:13:35 -0700 From: Chris Stankevitz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Offline/online a drive from a mounted RW RAIDZx pool: resilver needed? Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 23:22:38 -0000 When I offline a drive from a RW RAIDZx pool and later bring it online, do I have to endure a full resilver? If not, ZFS must keep track of history (sort of like subversion) so that it can replay missing data. I believe the answer is "no" given my experience today: 1. 11 drive pool healthy (~10 TB of data) 2. I (accidentally) pull a drive from hot-swap bay 3. pool degraded 4. 30 seconds later I reinsert the drive 5. pool still degraded (disk missing) 6. 300 seconds later I issue "zpool online /dev/daX" 7. A few seconds later, pool healthy again 8. I did not notice any resilvering Chris From owner-freebsd-fs@freebsd.org Mon Jul 13 23:26:09 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E53EC995B62 for ; Mon, 13 Jul 2015 23:26:08 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-ob0-x231.google.com (mail-ob0-x231.google.com [IPv6:2607:f8b0:4003:c01::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AD352231 for ; Mon, 13 Jul 2015 23:26:08 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by obbgp5 with SMTP id gp5so127074058obb.0 for ; Mon, 13 Jul 2015 16:26:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=P6kZj7XF++uBhjW7JeV/dzpZrWOSNx2pZb0l1KNVfFE=; b=iICUFeXQ7tWaaA1rnvvUpeOLrZJ9QyC0ulJUWpdwqj9VmctmZwGAWIGP15Ifq3wzQJ yFcxa2b/gEm7ndUBN1nggwPgT/pHfI9Xuiucih5yUwgH0ziIK1xae7MnIHqqQtYcrfRu 1OXfbQWU4dWKnWquwNwLEpxihPEcFntgkpDtVQ5ox4QwySYaIy6TJRtwrwwJy7oaaJC4 sznQcsPkZa5IBwsWBk8SdgjbhKcSdQ2ha7ibfDpxAII7wB9wAgQJrQ+9gu5IiEOg1Tn7 hxd4wgyYCDVasggTzCxqQHBXNJumlSbSU72ZTU666mwFnLgp4VsnLHgjQpg0IxTsUmAQ cpJg== MIME-Version: 1.0 X-Received: by 10.182.215.226 with SMTP id ol2mr32050695obc.56.1436829968007; Mon, 13 Jul 2015 16:26:08 -0700 (PDT) Received: by 10.76.81.100 with HTTP; Mon, 13 Jul 2015 16:26:07 -0700 (PDT) Received: by 10.76.81.100 with HTTP; Mon, 13 Jul 2015 16:26:07 -0700 (PDT) In-Reply-To: <55A4461F.6020308@stankevitz.com> References: <55A4461F.6020308@stankevitz.com> Date: Mon, 13 Jul 2015 16:26:07 -0700 Message-ID: Subject: Re: Offline/online a drive from a mounted RW RAIDZx pool: resilver needed? From: Freddie Cash To: Chris Stankevitz Cc: FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 23:26:09 -0000 If you don't write anything to the pool, or at least, nothing gets written to that vdev, then there's On Jul 13, 2015 4:22 PM, "Chris Stankevitz" wrote: > > When I offline a drive from a RW RAIDZx pool and later bring it online, do > I have to endure a full resilver? > > If not, ZFS must keep track of history (sort of like subversion) so that > it can replay missing data. I believe the answer is "no" given my > experience today: > > 1. 11 drive pool healthy (~10 TB of data) > > 2. I (accidentally) pull a drive from hot-swap bay > > 3. pool degraded > > 4. 30 seconds later I reinsert the drive > > 5. pool still degraded (disk missing) > > 6. 300 seconds later I issue "zpool online /dev/daX" > > 7. A few seconds later, pool healthy again > > 8. I did not notice any resilvering > > Chris > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Tue Jul 14 00:01:59 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C007C9993A3 for ; Tue, 14 Jul 2015 00:01:59 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: from mail-qg0-x232.google.com (mail-qg0-x232.google.com [IPv6:2607:f8b0:400d:c04::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 57E29107B; Tue, 14 Jul 2015 00:01:59 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: by qgy5 with SMTP id 5so28531847qgy.3; Mon, 13 Jul 2015 17:01:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=vXA9uH3yRbhjWp3/IvxxKAS5/2aEZkYuz53MhFXP8UA=; b=pRJuZluj6bmomAdr3aOpYD+fH7m52e+JCHN+wXd9dWWuc14twVAw9ffZon7uZBaKLN ny41QGTUWWRlNQ7t3nw5a6R/dhsF3t2nBot2RboomzuXt6y/BCsAExmyTQdWMAKthRol jiJtL1WApysgWILPFa44X0ki96YG8BM2r70mK9c0GVPMaw1VxAvShFB0migc4yWmQR3Y jnqtPPYhyJqYfurjgXXrJtHJl25glgihcyik7OaGtEd2HUdCFnolDKd6Z9RsBkVfrwFn WULZfs38p9/b+TQqgp/G1Hg10d3hxlOr9il1m8PvSnXTTtE0+RPH0OMhDGcUgzRhViVE C2nA== MIME-Version: 1.0 X-Received: by 10.140.232.206 with SMTP id d197mr28018294qhc.18.1436832116840; Mon, 13 Jul 2015 17:01:56 -0700 (PDT) Received: by 10.96.174.39 with HTTP; Mon, 13 Jul 2015 17:01:56 -0700 (PDT) In-Reply-To: <55A40498.80608@FreeBSD.org> References: <55A3A800.5060904@denninger.net> <55A3C0A1.7040505@infracaninophile.co.uk> <55A3C357.5070209@denninger.net> <55A3C80F.4060407@infracaninophile.co.uk> <55A40498.80608@FreeBSD.org> Date: Mon, 13 Jul 2015 21:01:56 -0300 Message-ID: Subject: Re: FreeBSD 10.1 Memory Exhaustion From: Christopher Forgeron To: Matthew Seaman Cc: FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 00:01:59 -0000 Ah, confirmed buildkernel only works. Testing now on a small box (24 gig, 2 Xeon, 5TB storage), I may get the chance to put this on a less important production box before the weekend. On Mon, Jul 13, 2015 at 3:34 PM, Matthew Seaman wrote: > On 13/07/2015 19:26, Christopher Forgeron wrote: > > I found the same thing - The older patch does apply on 10.1 with one line > > needing to be added manually. Did that, but I'm running into this when I > > try a make buildworld: > > I only did a buildkernel as the patch doesn't affect any of the userland > code. > > Cheers, > > Matthew > > > From owner-freebsd-fs@freebsd.org Tue Jul 14 09:26:33 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A04899CDEC for ; Tue, 14 Jul 2015 09:26:33 +0000 (UTC) (envelope-from matthew@freebsd.org) Received: from smtp.infracaninophile.co.uk (smtp.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.infracaninophile.co.uk", Issuer "infracaninophile.co.uk" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BFF30A95 for ; Tue, 14 Jul 2015 09:26:32 +0000 (UTC) (envelope-from matthew@freebsd.org) Received: from ox-dell39.ox.adestra.com (no-reverse-dns.metronet-uk.com [85.199.232.226] (may be forged)) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.15.1/8.15.1) with ESMTPSA id t6E9QMMl034358 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 14 Jul 2015 10:26:23 +0100 (BST) (envelope-from matthew@freebsd.org) Authentication-Results: smtp.infracaninophile.co.uk; dmarc=none header.from=freebsd.org DKIM-Filter: OpenDKIM Filter v2.9.2 smtp.infracaninophile.co.uk t6E9QMMl034358 Authentication-Results: smtp.infracaninophile.co.uk/t6E9QMMl034358; dkim=none reason="no signature"; dkim-adsp=none; dkim-atps=neutral X-Authentication-Warning: lucid-nonsense.infracaninophile.co.uk: Host no-reverse-dns.metronet-uk.com [85.199.232.226] (may be forged) claimed to be ox-dell39.ox.adestra.com Message-ID: <55A4D5B7.2030603@freebsd.org> Date: Tue, 14 Jul 2015 10:26:15 +0100 From: Matthew Seaman User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: FreeBSD 10.1 Memory Exhaustion References: <55A3A800.5060904@denninger.net> In-Reply-To: <55A3A800.5060904@denninger.net> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="CnuH8DIVhwl4sPakXbhMRW56gHDDcql24" X-Virus-Scanned: clamav-milter 0.98.7 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-2.8 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on lucid-nonsense.infracaninophile.co.uk X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 09:26:33 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --CnuH8DIVhwl4sPakXbhMRW56gHDDcql24 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 07/13/15 12:58, Karl Denninger wrote: > Put this on your box and see if the problem goes away.... :-) >=20 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D187594 >=20 > The 2015-02-10 refactor will apply against 10.1-STABLE and 10.2-PRE (th= e > latter will give you a 10-line fuzz in one block but applies and works.= ) >=20 > I've been unable to provoke misbehavior with this patch in and I run a > cron job that does auto-snapshotting. There are others that have run > this patch with similarly positive results. I know that you, Karl, and a number of others have been advocating to get this patch set committed. Having now personally run into the sort of problems that this addresses I can say that I would very much like to see this go in. Conditional of course on this actually solving the problems I and others have been experiencing without introducing significant regressions elsewhere. It's only had a day's testing from me so far, but it's looking good. If it survives a week without the system locking up, I'll be convinced. I also know that some of the most experienced ZFS developers are cautious about applying this patchset, and I do not doubt that they have good reason to think so. What I wonder is could it be possible to generate some sort of regression tests where * The memory exhaustion effect or equivalent memory pressures can be triggered at will * The test doesn't require unfeasibly large resources to run * The behaviour provides a good model for real-world deployments Maybe these tests would be too large-scale to run every day in Jenkins, but having them available as part of, say, the release process, seems like a no-brainer to me. ZFS is one of FreeBSD's big selling points, and not having it work reliably at scale is an embarrassment, to say the least. Cheers, Matthew --CnuH8DIVhwl4sPakXbhMRW56gHDDcql24 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVpNW3AAoJEABRPxDgqeTnJ4oP/2J7iAvWZUXaQzESb0IBc91N ZKG1P9XtsEkN5FJOn963s2Vhc8HHdB3PGgKYbRVMu+WVUPXPrKSihxHVSD8QkWAt D1TGuDcPZ7oDiS/wfzROZi0zOxghot0NPY+LcLKaDVdNe9bv2HQYuTTCOKLKG/nm krOSK7jIRcng7Ll89TJxz2gEEDr+U+gcFZq5R9M/ljGin0mxK1HjrPiGP2tFCJ/F F+AGkxDPdSLRK/xgpRwcq7chhmftYlSxd95kH/eSckI9U4bufq2s2I9dNfs0qkT5 Uh1ODFa0/Ze+RTFH9Y0UHOOU2xxlqA1/ju1zI0RLMAfk4R+8LyHqJeQa2VhIlTvE MZPPTJaCzdokK73DYdRVMTIRrw6WIYWAx04pBZtzit/tak1B1JxkwCLItFM3E0H3 YgjdOKB7vb5LIZoTMTURS355mpa+OfhMcobmpZt4fTg1MprgF+9DHlacSE5azJmh 6yTA7qQfgihw74KPKrPsxCYcLz0Vsf28/UcXghH+zaO645Vplt5f8ZjwfEjE2xAO Q2xi+dhU/Oi40FjJNgNFTMEaWj3j+zGNNHgwfnv6tpvUei67UpqCXzbE+liypYoq OBfPwHdqv6a6NHhIvaqCZFkBmNwmFPoH+w9Kz3X5aJzmstOCVagsIxRw+tqHIpha uib2r6lYOjUZLFnogFp8 =3wyn -----END PGP SIGNATURE----- --CnuH8DIVhwl4sPakXbhMRW56gHDDcql24-- From owner-freebsd-fs@freebsd.org Tue Jul 14 09:56:03 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06AD799B30D for ; Tue, 14 Jul 2015 09:56:03 +0000 (UTC) (envelope-from accounts-ag@netlabs.org) Received: from r2-d2.netlabs.org (r2-d2.netlabs.org [213.238.45.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 599C8AE7 for ; Tue, 14 Jul 2015 09:56:01 +0000 (UTC) (envelope-from accounts-ag@netlabs.org) Received: (qmail 24499 invoked by uid 89); 14 Jul 2015 09:49:17 -0000 Received: from unknown (HELO eternal-3.metropolis.netlabs.org) (accounts-ag@netlabs.org@213.144.156.18) by 0 with ESMTPA; 14 Jul 2015 09:49:17 -0000 To: freebsd-fs@freebsd.org From: accounts-ag Subject: ZFS volume does not seem to free space for deleted files Message-ID: <55A4DB1D.7000404@netlabs.org> Date: Tue, 14 Jul 2015 11:49:17 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.0.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 09:56:03 -0000 Hi, I'm trying to clean up a zfs volume used by a FreeBSD jail: NAME USED AVAIL REFER MOUNTPOINT tank/jail/www 116G 68.3G 4.57G /tank/jail/www The USED space was probably accurate before I freed a lot of space on this device by cleaning up /var/tmp and some (a lot...) system mails which accumulated over years. According to "du" this at least freed +50GB. However, somehow ZFS does not free that yet. In the stopped jail starting from /tank/jail/www, "du" (or ncdu to be precise) reports: Total disk usage: 4.6 GiB Apparent size: 4.6 GiB Items: 76516 This seems to correspond with what is listed as REFER. So I checked space usage over everything on tank/jail/www: # zfs list -t all -o space -r tank/jail/www NAME AVAIL USED USEDSNAP USEDDS tank/jail/www 68.3G 116G 112G 4.57G tank/jail/www@monthly-2015-07-01_00.00.00--3m - 29.0M - - tank/jail/www@daily-2015-07-01_00.00.00--2w - 27.8M - - tank/jail/www@daily-2015-07-02_00.00.00--2w - 1.22G - - tank/jail/www@daily-2015-07-03_00.00.00--2w - 1.27G - - tank/jail/www@daily-2015-07-04_00.00.00--2w - 1.52G - - tank/jail/www@daily-2015-07-05_00.00.00--2w - 62.6M - - tank/jail/www@weekly-2015-07-05_00.00.00--1m - 62.5M - - tank/jail/www@daily-2015-07-06_00.00.00--2w - 1.51G - - tank/jail/www@daily-2015-07-07_00.00.00--2w - 1.38G - - tank/jail/www@daily-2015-07-08_00.00.00--2w - 1.39G - - tank/jail/www@daily-2015-07-09_00.00.00--2w - 1.49G - - tank/jail/www@daily-2015-07-10_00.00.00--2w - 1.17G - - tank/jail/www@daily-2015-07-11_00.00.00--2w - 1.33G - - tank/jail/www@weekly-2015-07-12_00.00.00--1m - 119M - - tank/jail/www@daily-2015-07-12_00.00.00--2w - 117M - - tank/jail/www@hourly-2015-07-12_04.04.00--2d - 255M - - tank/jail/www@hourly-2015-07-12_10.04.00--2d - 265M - - tank/jail/www@hourly-2015-07-12_16.04.00--2d - 267M - - tank/jail/www@hourly-2015-07-12_22.04.00--2d - 204M - - tank/jail/www@daily-2015-07-13_00.00.00--2w - 257M - - tank/jail/www@hourly-2015-07-13_04.04.00--2d - 257M - - tank/jail/www@hourly-2015-07-13_10.04.00--2d - 260M - - tank/jail/www@hourly-2015-07-13_16.04.00--2d - 1.70G - - tank/jail/www@hourly-2015-07-13_22.04.00--2d - 48.4M - - tank/jail/www@daily-2015-07-14_00.00.00--2w - 32.4M - - tank/jail/www@hourly-2015-07-14_04.04.00--2d - 20.0M - - tank/jail/www@hourly-2015-07-14_10.04.00--2d - 18.9M - - So according to my calculation it is using less than 20GB of space for snapshots. Which does not make much sense to me as I clearly do expect the deleted files to show up somewhere (I did it on 2015-07-13). So how can the big difference to USED be explained which reports 116G in total? Would I have to delete all snapshots to properly free the disk space? I was unmounting the volume meanwhile and remounting it but that didn't change anything. zpool info: # zpool list -o name,size,allocated,free,freeing NAME SIZE ALLOC FREE FREEING tank 295G 222G 72.9G 0 thanks regards Adrian From owner-freebsd-fs@freebsd.org Tue Jul 14 10:00:25 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D17E599B49D for ; Tue, 14 Jul 2015 10:00:25 +0000 (UTC) (envelope-from fullermd@over-yonder.net) Received: from thyme.infocus-llc.com (thyme.infocus-llc.com [199.15.120.10]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ADF6ABF8 for ; Tue, 14 Jul 2015 10:00:25 +0000 (UTC) (envelope-from fullermd@over-yonder.net) Received: from draco.over-yonder.net (c-75-65-60-66.hsd1.ms.comcast.net [75.65.60.66]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by thyme.infocus-llc.com (Postfix) with ESMTPSA id B96B437B5CF; Tue, 14 Jul 2015 05:00:23 -0500 (CDT) Received: by draco.over-yonder.net (Postfix, from userid 100) id 3mVy5M0lbFz1jw; Tue, 14 Jul 2015 05:00:23 -0500 (CDT) Date: Tue, 14 Jul 2015 05:00:23 -0500 From: "Matthew D. Fuller" To: accounts-ag Cc: freebsd-fs@freebsd.org Subject: Re: ZFS volume does not seem to free space for deleted files Message-ID: <20150714100023.GC96394@over-yonder.net> References: <55A4DB1D.7000404@netlabs.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <55A4DB1D.7000404@netlabs.org> X-Editor: vi X-OS: FreeBSD User-Agent: Mutt/1.5.23-fullermd.4 (2014-03-12) X-Virus-Scanned: clamav-milter 0.98.7 at thyme.infocus-llc.com X-Virus-Status: Clean X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 10:00:25 -0000 On Tue, Jul 14, 2015 at 11:49:17AM +0200 I heard the voice of accounts-ag, and lo! it spake thus: > > So according to my calculation it is using less than 20GB of space > for snapshots. Which does not make much sense to me as I clearly do > expect the deleted files to show up somewhere (I did it on > 2015-07-13). So how can the big difference to USED be explained > which reports 116G in total? I'm pretty sure you can't just add up those USED's and get a number suggesting how much space deleting them all will free up. AIUI, the USED number for a snapshot means how much that one alone is using; e.g., how much space would be freed by deleting just that one. So any blocks referenced by 2 or more wouldn't show up in those numbers. A bunch of files that have been sitting around a long time would be a good candidate for existing in most (/all) of them, so you won't see them in per-snapshot USED until you delete all but 1. -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. From owner-freebsd-fs@freebsd.org Tue Jul 14 10:34:22 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6648199C342 for ; Tue, 14 Jul 2015 10:34:22 +0000 (UTC) (envelope-from ml-ktk@netlabs.org) Received: from r2-d2.netlabs.org (r2-d2.netlabs.org [213.238.45.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D337DA for ; Tue, 14 Jul 2015 10:34:21 +0000 (UTC) (envelope-from ml-ktk@netlabs.org) Received: (qmail 29436 invoked by uid 89); 14 Jul 2015 10:34:19 -0000 Received: from unknown (HELO eternal-3.metropolis.netlabs.org) (ml-ktk@netlabs.org@213.144.156.18) by 0 with ESMTPA; 14 Jul 2015 10:34:19 -0000 Subject: Re: FreeBSD 10.1 Memory Exhaustion To: freebsd-fs@freebsd.org References: <55A3A800.5060904@denninger.net> <55A4D5B7.2030603@freebsd.org> From: Adrian Gschwend X-Enigmail-Draft-Status: N1110 Message-ID: <55A4E5AB.8060909@netlabs.org> Date: Tue, 14 Jul 2015 12:34:19 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.0.1 MIME-Version: 1.0 In-Reply-To: <55A4D5B7.2030603@freebsd.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 10:34:22 -0000 On 14.07.15 11:26, Matthew Seaman wrote: > On 07/13/15 12:58, Karl Denninger wrote: >> Put this on your box and see if the problem goes away.... :-) [...] > I know that you, Karl, and a number of others have been advocating to > get this patch set committed. Having now personally run into the sort > of problems that this addresses I can say that I would very much like to > see this go in. Conditional of course on this actually solving the > problems I and others have been experiencing without introducing > significant regressions elsewhere. It's only had a day's testing from me > so far, but it's looking good. If it survives a week without the system > locking up, I'll be convinced. I was the one which posted the message last year which triggered Karl to analyze it as he saw similar issues: https://lists.freebsd.org/pipermail/freebsd-fs/2014-March/019043.html https://lists.freebsd.org/pipermail/freebsd-fs/2014-March/019057.html Since then I run on Karls patch and never had any issue anymore. Not that my boxes were basically unusable without the patch. So I'm basically hoping since then that the patch will be committed soon. > * The memory exhaustion effect or equivalent memory pressures can be > triggered at will > * The test doesn't require unfeasibly large resources to run > * The behaviour provides a good model for real-world deployments > > Maybe these tests would be too large-scale to run every day in Jenkins, > but having them available as part of, say, the release process, seems > like a no-brainer to me. I wouldn't consider my setup as "unfeasibly large resources", in fact I triggered it with a bunch of jails running on a machine and providing various Internet-services for a small Open Source community. I was always surprised that not more people ran into this issue as I had it since 8.x. regards Adrian From owner-freebsd-fs@freebsd.org Tue Jul 14 12:56:45 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2BADC99994D for ; Tue, 14 Jul 2015 12:56:45 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-wg0-x229.google.com (mail-wg0-x229.google.com [IPv6:2a00:1450:400c:c00::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BA4F5229 for ; Tue, 14 Jul 2015 12:56:44 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: by wgxm20 with SMTP id m20so8090169wgx.3 for ; Tue, 14 Jul 2015 05:56:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=bk+0OwK+FAoD/Q81SBbHct3+5EFRTUIr022LW9cefik=; b=L1D7VdqycqmNgYz+PIY1dxwvrHXymfGNVjof23nVujTNKz4S/ZVsWXzf0ocXfWPrmK xYmZL0urDFOEnJdMC3FmeZF54frn4tDg/95A7Sg6BckvLA8CmVc0KkScROupKy7KFVKn KbvAiBC/bXx9uet5zvnlCChB8tzRcAF7GvYTnuHWs/E5i6OVGXBilUP2lc9eYvcX52mu hCJEDLAAL0djRVCQJCG7G0booqIpkDobYl6a9ALnnZz10nQO/mS1ZjbqTdq4vVd5JtYY rImAI2kCku9YlyFX/90xYHwYKKXT9doO5mBTVYHdffI93h3MN71tkkKdvOl5PN1xXDtQ 5HUg== MIME-Version: 1.0 X-Received: by 10.180.97.7 with SMTP id dw7mr33630385wib.74.1436878602942; Tue, 14 Jul 2015 05:56:42 -0700 (PDT) Received: by 10.180.73.5 with HTTP; Tue, 14 Jul 2015 05:56:42 -0700 (PDT) In-Reply-To: <20150714100023.GC96394@over-yonder.net> References: <55A4DB1D.7000404@netlabs.org> <20150714100023.GC96394@over-yonder.net> Date: Tue, 14 Jul 2015 13:56:42 +0100 Message-ID: Subject: Re: ZFS volume does not seem to free space for deleted files From: krad To: "Matthew D. Fuller" Cc: accounts-ag , FreeBSD FS Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 12:56:45 -0000 also if compression was enabled and it was mainly mails you deleted dont expect ot get the space savings you think you might get as the compression ratios would be good. Lets not even start on dedup... On 14 July 2015 at 11:00, Matthew D. Fuller wrote: > On Tue, Jul 14, 2015 at 11:49:17AM +0200 I heard the voice of > accounts-ag, and lo! it spake thus: > > > > So according to my calculation it is using less than 20GB of space > > for snapshots. Which does not make much sense to me as I clearly do > > expect the deleted files to show up somewhere (I did it on > > 2015-07-13). So how can the big difference to USED be explained > > which reports 116G in total? > > I'm pretty sure you can't just add up those USED's and get a number > suggesting how much space deleting them all will free up. AIUI, the > USED number for a snapshot means how much that one alone is using; > e.g., how much space would be freed by deleting just that one. So any > blocks referenced by 2 or more wouldn't show up in those numbers. A > bunch of files that have been sitting around a long time would be a > good candidate for existing in most (/all) of them, so you won't see > them in per-snapshot USED until you delete all but 1. > > > -- > Matthew Fuller (MF4839) | fullermd@over-yonder.net > Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ > On the Internet, nobody can hear you scream. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Tue Jul 14 15:10:31 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 440F49A1761 for ; Tue, 14 Jul 2015 15:10:31 +0000 (UTC) (envelope-from schittenden@groupon.com) Received: from mail-yk0-x22f.google.com (mail-yk0-x22f.google.com [IPv6:2607:f8b0:4002:c07::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 10AC8D68 for ; Tue, 14 Jul 2015 15:10:30 +0000 (UTC) (envelope-from schittenden@groupon.com) Received: by ykay190 with SMTP id y190so10404128yka.3 for ; Tue, 14 Jul 2015 08:10:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=groupon.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=+9J7FmOApWdCr4Nlj+EgHIEhABHWskultPV++8sZPfg=; b=fBBj4EGea15pHe/GyHStZ+u2rdoGQKP+ZPei7Cpc0+g1f0gKxVejXhm4WYPVAePONQ JNwNjaHzssjBkqadIdjQBH/XmS7fXNVfjac794xRYeKLpaTexE3efHbSzYK0Rd7tTUg/ xVskk3XJzLQnDI+X0rSD8UmZlTI55tzEEgc6g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=+9J7FmOApWdCr4Nlj+EgHIEhABHWskultPV++8sZPfg=; b=C9gl/JhQTrJE3Wn4+doUIqjv+IjO3S0BhTy/XVY3aZPm/Zg+/fd/lNscrOGn3syBKF Tjv3yjgtN6y/2khf+mqAn2nTTiSXBdAC/sgn6USo6k7EOU9rFfOTlt2zc0PieE+/Hkm4 64yLqXLoo3ZrHhJxO4p65BV9BTsbK2/FoIgDMGgy5HmOf8zNRKQnZufEI8yfadUeMPb6 lIEzr7VDA506Fm38GR+mZCXQbgts+flg92uPyIxSlQ/7mF2qjc9lmFkJ6qIOJvD5E9vL l0Ky5Zo+7Kq0jf0RFacryCkfcrTXmv0fqTgaCdagPaZIT8omVm+YyREoS8QNlp6P5NRO MtpQ== X-Gm-Message-State: ALoCoQlyYrmmwAXF/foN61Rzqj2+egEcLw/cz4BTNjnw0OMEn68r55H4Hj6XPSjVSJmlLAetYyTE MIME-Version: 1.0 X-Received: by 10.170.110.82 with SMTP id c79mr44902121ykb.113.1436886629924; Tue, 14 Jul 2015 08:10:29 -0700 (PDT) Received: by 10.13.242.7 with HTTP; Tue, 14 Jul 2015 08:10:29 -0700 (PDT) In-Reply-To: <55A4E5AB.8060909@netlabs.org> References: <55A3A800.5060904@denninger.net> <55A4D5B7.2030603@freebsd.org> <55A4E5AB.8060909@netlabs.org> Date: Tue, 14 Jul 2015 08:10:29 -0700 Message-ID: Subject: Re: FreeBSD 10.1 Memory Exhaustion From: Sean Chittenden To: Adrian Gschwend Cc: FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 15:10:31 -0000 I think the reason this is not seen more often is because people frequently throw limits on the arc in /boot/loader.conf: vfs.zfs.arc_min="18G" vfs.zfs.arc_max="149G" ZFS ARC *should* not require those settings, but does currently for mixed workloads (i.e. databases) in order to be "stable". By setting fixed sizes on the ARC, UMA and ARC are much more cooperative in that they have their own memory regions to manage so this behavior is not seen as often. To be clear, however, it should not be necessary to set parameters like these in /boot/loader.conf in order to obtain consistent operational behavior. I'd be curious to know if someone running 10.2 BETA without patches is able to trigger this behavior or not. There was work done that reported helped with this between 10.1 and now. To what extent it helped, however, I don't have any advice yet. -sc On Tue, Jul 14, 2015 at 3:34 AM, Adrian Gschwend wrote: > On 14.07.15 11:26, Matthew Seaman wrote: > > > > On 07/13/15 12:58, Karl Denninger wrote: > >> Put this on your box and see if the problem goes away.... :-) > > [...] > > > I know that you, Karl, and a number of others have been advocating to > > get this patch set committed. Having now personally run into the sort > > of problems that this addresses I can say that I would very much like to > > see this go in. Conditional of course on this actually solving the > > problems I and others have been experiencing without introducing > > significant regressions elsewhere. It's only had a day's testing from me > > so far, but it's looking good. If it survives a week without the system > > locking up, I'll be convinced. > > I was the one which posted the message last year which triggered Karl to > analyze it as he saw similar issues: > > https://lists.freebsd.org/pipermail/freebsd-fs/2014-March/019043.html > > https://lists.freebsd.org/pipermail/freebsd-fs/2014-March/019057.html > > Since then I run on Karls patch and never had any issue anymore. Not > that my boxes were basically unusable without the patch. > > So I'm basically hoping since then that the patch will be committed soon. > > > * The memory exhaustion effect or equivalent memory pressures can be > > triggered at will > > * The test doesn't require unfeasibly large resources to run > > * The behaviour provides a good model for real-world deployments > > > > Maybe these tests would be too large-scale to run every day in Jenkins, > > but having them available as part of, say, the release process, seems > > like a no-brainer to me. > > I wouldn't consider my setup as "unfeasibly large resources", in fact I > triggered it with a bunch of jails running on a machine and providing > various Internet-services for a small Open Source community. I was > always surprised that not more people ran into this issue as I had it > since 8.x. > > regards > > Adrian > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > -- Sean Chittenden From owner-freebsd-fs@freebsd.org Tue Jul 14 16:11:11 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 94A119A17C5 for ; Tue, 14 Jul 2015 16:11:11 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-outbound.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.userve.net", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E4B53EF5 for ; Tue, 14 Jul 2015 16:11:10 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) by smtp-outbound.userve.net (8.14.7/8.14.7) with ESMTP id t6EFn5qb076141 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=FAIL); Tue, 14 Jul 2015 16:49:05 +0100 (BST) (envelope-from matt.churchyard@userve.net) Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Tue, 14 Jul 2015 16:49:05 +0100 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Tue, 14 Jul 2015 16:49:05 +0100 From: Matt Churchyard To: Sean Chittenden , Adrian Gschwend CC: FreeBSD Filesystems Subject: RE: FreeBSD 10.1 Memory Exhaustion Thread-Topic: FreeBSD 10.1 Memory Exhaustion Thread-Index: AQHQvWHT+mqLQQIA90upWPLaPGA5Dp3ZOw8AgAFnrICAABMFgIAATSmAgAARnMA= Date: Tue, 14 Jul 2015 15:49:04 +0000 Message-ID: References: <55A3A800.5060904@denninger.net> <55A4D5B7.2030603@freebsd.org> <55A4E5AB.8060909@netlabs.org> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 16:11:11 -0000 Yes, I'm one of those and I suspect it's very common. I generally just use official FreeBSD releases and let the devs decide what= patches are/aren't applied. However, I limit max ARC on all my ZFS systems to leave a few GB or so (my = latest system has limit of 28G with 32GB total) Last time I didn't limit ARC was a new system about a year ago and it panic= ed due to memory after a few days I don't bother letting it near all my RAM anymore. For me (and probably many other ZFS users that want their machines to stay = up more than a few days) it's much easier to do this, than to run manually = patched kernels that *might* fix it, but *might* also cause other problems.= Also allows me control over how much ZFS has, and how much I leave for my = other applications. Would be nice if it 'just worked', but I'll be very reluctant to take the l= imits off. -Matt -----Original Message----- From: owner-freebsd-fs@freebsd.org [mailto:owner-freebsd-fs@freebsd.org] On= Behalf Of Sean Chittenden Sent: 14 July 2015 16:10 To: Adrian Gschwend Cc: FreeBSD Filesystems Subject: Re: FreeBSD 10.1 Memory Exhaustion I think the reason this is not seen more often is because people frequently= throw limits on the arc in /boot/loader.conf: vfs.zfs.arc_min=3D"18G" vfs.zfs.arc_max=3D"149G" ZFS ARC *should* not require those settings, but does currently for mixed w= orkloads (i.e. databases) in order to be "stable". By setting fixed sizes = on the ARC, UMA and ARC are much more cooperative in that they have their o= wn memory regions to manage so this behavior is not seen as often. To be clear, however, it should not be necessary to set parameters like the= se in /boot/loader.conf in order to obtain consistent operational behavior.= I'd be curious to know if someone running 10.2 BETA without patches is ab= le to trigger this behavior or not. There was work done that reported help= ed with this between 10.1 and now. To what extent it helped, however, I do= n't have any advice yet. -sc On Tue, Jul 14, 2015 at 3:34 AM, Adrian Gschwend wrote= : > On 14.07.15 11:26, Matthew Seaman wrote: > > > > On 07/13/15 12:58, Karl Denninger wrote: > >> Put this on your box and see if the problem goes away.... :-) > > [...] > > > I know that you, Karl, and a number of others have been advocating=20 > > to get this patch set committed. Having now personally run into the=20 > > sort of problems that this addresses I can say that I would very=20 > > much like to see this go in. Conditional of course on this actually=20 > > solving the problems I and others have been experiencing without=20 > > introducing significant regressions elsewhere. It's only had a day's=20 > > testing from me so far, but it's looking good. If it survives a=20 > > week without the system locking up, I'll be convinced. > > I was the one which posted the message last year which triggered Karl=20 > to analyze it as he saw similar issues: > > https://lists.freebsd.org/pipermail/freebsd-fs/2014-March/019043.html > > https://lists.freebsd.org/pipermail/freebsd-fs/2014-March/019057.html > > Since then I run on Karls patch and never had any issue anymore. Not=20 > that my boxes were basically unusable without the patch. > > So I'm basically hoping since then that the patch will be committed soon. > > > * The memory exhaustion effect or equivalent memory pressures can be > > triggered at will > > * The test doesn't require unfeasibly large resources to run > > * The behaviour provides a good model for real-world deployments > > > > Maybe these tests would be too large-scale to run every day in=20 > > Jenkins, but having them available as part of, say, the release=20 > > process, seems like a no-brainer to me. > > I wouldn't consider my setup as "unfeasibly large resources", in fact=20 > I triggered it with a bunch of jails running on a machine and=20 > providing various Internet-services for a small Open Source community.=20 > I was always surprised that not more people ran into this issue as I=20 > had it since 8.x. > > regards > > Adrian > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > -- Sean Chittenden _______________________________________________ freebsd-fs@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Jul 14 16:13:06 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BA8389A18C7 for ; Tue, 14 Jul 2015 16:13:06 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: from mail-qk0-x234.google.com (mail-qk0-x234.google.com [IPv6:2607:f8b0:400d:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 761A112 for ; Tue, 14 Jul 2015 16:13:06 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: by qkcl188 with SMTP id l188so9666168qkc.1 for ; Tue, 14 Jul 2015 09:13:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=QUf3A/nUs2e39j8G0az62/db2/ajrG9Kb0TwU5mdn8o=; b=qwdek+6MVEgMs4qpH5IVX0le0oNkv3YDJ27f3cmMNOR50n+vycP4Ix893vcT+3QESI xrK4MUymg/Gj5Yddsv+tKajYRyoKRiilMHfLUqJ3GrU1t5NE3yoslv5hJtPihPgGc6U0 +e0OoNencFTkjrb3/p6i6Qui9z+mr4qu1in3dPPV1Lfa8yZs1nhVyQGWgC0jonGjvcs8 lFCyUbrtUBSQijm1uQDJoaprV5vjDxfAwSTQyxh6WdK/ubmuBRgBvY6R/lqrWwZMfYxq QFtHTxYOKeifTxiwGGRW1aGiGd91GQQvTiqvlXr4klVLdi+ZDDd3lmyysOyrzdApmP2M GOow== MIME-Version: 1.0 X-Received: by 10.140.91.66 with SMTP id y60mr62961884qgd.90.1436890385528; Tue, 14 Jul 2015 09:13:05 -0700 (PDT) Received: by 10.96.174.39 with HTTP; Tue, 14 Jul 2015 09:13:05 -0700 (PDT) In-Reply-To: References: <55A3A800.5060904@denninger.net> <55A4D5B7.2030603@freebsd.org> <55A4E5AB.8060909@netlabs.org> Date: Tue, 14 Jul 2015 13:13:05 -0300 Message-ID: Subject: Re: FreeBSD 10.1 Memory Exhaustion From: Christopher Forgeron To: Sean Chittenden Cc: Adrian Gschwend , FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 16:13:06 -0000 I now have the patch on two of my production machines after early morning crashes that opened up a 'maintenance window'. With crashing happening quite regularly for months now, I think we have progress if I can make it to the end of the week crash-free. On Tue, Jul 14, 2015 at 12:10 PM, Sean Chittenden wrote: > > > ZFS ARC *should* not require those settings, but does currently for mixed > workloads (i.e. databases) in order to be "stable". This illustrates one of my main concerns: - I shouldn't have to tweak/patch FreeBSD to keep it from being unstable. Tweaking to improve performance is perfectly fine with me, that's part of what we do as sysadmins to make the software fit the layout of the required needs. Right now stock FreeBSD is unstable for me, and I don't think my workload is unusual in any way. My smallest production machine is 48 GiB RAM on a 8 TiB Pool, and it still faults like my larger 96GiB RAM machines with larger pools. You can buy a 5 TiB drive for ~$200 - $300 now. It's well within the budget of a home user to have a 10TiB ZFS pool, and if that home user is torrenting video, they are setting up the same situations for crashing. ZFS needs to know when it's consuming swap instead of raw memory, and from what I understand, this patch allows that knowledge, and thus this patch is very important. I'll take a small, possibly performance regression to achieve stability. If we can't come together to help people who's jobs and income literally depend on keeping these boxes running, then can we do it for the common punter trying to torrent movies? :-) From owner-freebsd-fs@freebsd.org Tue Jul 14 16:42:08 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3AC1C9A1E29 for ; Tue, 14 Jul 2015 16:42:08 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from mango.stankevitz.com (mango.stankevitz.com [208.79.93.194]) by mx1.freebsd.org (Postfix) with ESMTP id 28E4CE98 for ; Tue, 14 Jul 2015 16:42:07 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from Chriss-MacBook-Pro.local (209-203-101-124.static.twtelecom.net [209.203.101.124]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mango.stankevitz.com (Postfix) with ESMTPSA id D6B6F6E7; Tue, 14 Jul 2015 09:42:06 -0700 (PDT) Message-ID: <55A53BE2.7060609@stankevitz.com> Date: Tue, 14 Jul 2015 09:42:10 -0700 From: Chris Stankevitz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Freddie Cash CC: FreeBSD Filesystems Subject: Re: Offline/online a drive from a mounted RW RAIDZx pool: resilver needed? References: <55A4461F.6020308@stankevitz.com> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 16:42:08 -0000 On 7/13/15 4:26 PM, Freddie Cash wrote: > If you don't write anything to the pool, or at least, nothing gets > written to that vdev, then there's Freddie, I missed part of your message. Are you saying that there are two possibilities in the offline/online scenario: the drive comes online immediately (no missed writes) or the drive undergoes a complete resilver (1 or more missing bytes)? Thank you, Chris From owner-freebsd-fs@freebsd.org Tue Jul 14 16:46:17 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EDAD09A1F15 for ; Tue, 14 Jul 2015 16:46:16 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-ob0-x235.google.com (mail-ob0-x235.google.com [IPv6:2607:f8b0:4003:c01::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B4B56FD4 for ; Tue, 14 Jul 2015 16:46:16 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by obre1 with SMTP id e1so9758751obr.1 for ; Tue, 14 Jul 2015 09:46:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=xI/+W4UUv1roi3NbpfcRm/ZEzDRKQ0qwOs3xLX+PAis=; b=vJNuEZ03ZqAkpEuV5d5uVhFmo9LUGCujeis+aGXef6SaNv1M3TQadh85pvH/Btdc7V Yy6lJda/tJfdzi9o9tjGzp9TRYFoNvoZSWwAeVvAZquoH4Tb0prUf5V4E/JxW6gQJqxs I6GdEqGpCqcr2n7UN2ubJDcMf2Dn9LRYaZcyt9GkU2R3w0XbK1rjBV+7MoLcWFa27oW5 d0iD5BfjA8NhSJIxu+WgV0abGZxk4gxf+5gtXtfheONGzB4DbgSPy1NWS7oNc/eIurYB xf+0qaOZlEaFMwE+jFd1p/a8Z8PijLbJObs2OO34dxOHTv5P1ju8rzZvjZKEUFKe0QaA 20yw== MIME-Version: 1.0 X-Received: by 10.202.232.67 with SMTP id f64mr34700071oih.63.1436892376010; Tue, 14 Jul 2015 09:46:16 -0700 (PDT) Received: by 10.76.81.100 with HTTP; Tue, 14 Jul 2015 09:46:15 -0700 (PDT) In-Reply-To: <55A53BE2.7060609@stankevitz.com> References: <55A4461F.6020308@stankevitz.com> <55A53BE2.7060609@stankevitz.com> Date: Tue, 14 Jul 2015 09:46:15 -0700 Message-ID: Subject: Re: Offline/online a drive from a mounted RW RAIDZx pool: resilver needed? From: Freddie Cash To: Chris Stankevitz Cc: FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 16:46:17 -0000 On Tue, Jul 14, 2015 at 9:42 AM, Chris Stankevitz wrote: > On 7/13/15 4:26 PM, Freddie Cash wrote: > >> If you don't write anything to the pool, or at least, nothing gets >> written to that vdev, then there's >> > > Freddie, > > I missed part of your message. Are you saying that there are two > possibilities in the offline/online scenario: the drive comes online > immediately (no missed writes) or the drive undergoes a complete resilver > (1 or more missing bytes)? > =E2=80=8BSorry, was typing that on my phone, and meant to hit back not send= . If nothing is written to the pool (or, at least that specific vdev) while the drive is removed, then the metadata on the drive will match the pool, and it can be onlined without a resilver. If only a little bit of data has been written, and a short bit of time has passed, =E2=80=8B =E2=80=8Bthen the drive will only need to sync (resilver) that changed data= to the drive. If a lot of data has been written, or a long time has passed (not sure what qualifies as "long", maybe until the last txg synced to the disk is no longer in the txg history of the pool?), then the entire drive will be resilvered. Or, something along those lines. That's what I've observed by doing various tests (offline drive, pull drive, write to pool, online drive; or offline drive, replace with new one, start resilver, new drive dies, stick in old drive, online it, wait for resilver; and other scenarios). =E2=80=8B --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@freebsd.org Tue Jul 14 16:59:22 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6B3E99C288 for ; Tue, 14 Jul 2015 16:59:22 +0000 (UTC) (envelope-from karl@denninger.net) Received: from fs.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "NewFS.denninger.net", Issuer "NewFS.denninger.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 99717B5A for ; Tue, 14 Jul 2015 16:59:21 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (localhost [127.0.0.1]) by fs.denninger.net (8.14.9/8.14.8) with ESMTP id t6EGxJGo089603 for ; Tue, 14 Jul 2015 11:59:19 -0500 (CDT) (envelope-from karl@denninger.net) Received: from [192.168.1.40] [192.168.1.40] (Via SSLv3 AES128-SHA) ; by Spamblock-sys (LOCAL/AUTH) Tue Jul 14 11:59:19 2015 Message-ID: <55A53FD9.90400@denninger.net> Date: Tue, 14 Jul 2015 11:59:05 -0500 From: Karl Denninger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: FreeBSD 10.1 Memory Exhaustion References: <55A3A800.5060904@denninger.net> <55A4D5B7.2030603@freebsd.org> <55A4E5AB.8060909@netlabs.org> In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms030302000908000202050400" X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 16:59:23 -0000 This is a cryptographically signed message in MIME format. --------------ms030302000908000202050400 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 7/14/2015 10:10, Sean Chittenden wrote: > I think the reason this is not seen more often is because people freque= ntly > throw limits on the arc in /boot/loader.conf: > > vfs.zfs.arc_min=3D"18G" > vfs.zfs.arc_max=3D"149G" > > ZFS ARC *should* not require those settings, but does currently for mix= ed > workloads (i.e. databases) in order to be "stable". By setting fixed s= izes > on the ARC, UMA and ARC are much more cooperative in that they have the= ir > own memory regions to manage so this behavior is not seen as often. However, this is a false God unless you have very tight control over the RSS requirements on your machine. For a NFS fileserver or similar you might be able to get away with that, because the number of nfsds (for example) is a nominally-known quantity and you can probably quantify RSS requirements. For a server that accepts connections from outside sources and is subject to burst loads this strategy moves the wall but probably doesn't prevent the problem entirely (what happens when a bunch of web clients hit your machine at once, for example, spiking memory demand?) The fundamental issue is that the base code will under certain load patterns (and surprisingly often) prefer to keep pages allocated to ARC (disk cache) in memory over RSS, causing RSS to be paged. This is exacerbated by UMA's "lazy" return allocated kernel memory (which is a good thing most of the time for performance reasons.) That decision is almost always wrong because paging RSS requires one guaranteed I/O (to place the paged RSS on the swap) and may require two (to later recover it if it is referenced); discarding cached disk data carries no I/O guarantee, with one future I/O only being required if the cached page is again referenced. The patch in question does not change the base code behavior until and unless memory is constrained. It then pares back ARC instead of allowing the system to page RSS, and in addition when under memory pressure UMA is patrolled to keep the lazily-held kernel memory in check along with cutting back dmu_tx write buffer size so as to prevent heavily burst-loading memory during write-intensive operations. The latter, IMHO, is a poorly chosen value in the first instance. The ideal situation would be one where the dmu_tx write buffer size is selected based on the performance of each vdev so as to always have at least one full buffer available when the previous DMA'd transfer completes, but not materially more than one (e.g. perhaps two maximum.) As it stands there is only one set for the entire system (rather than one per vdev) and it's sized based not on I/O channel performance but system memory with a cap of 4Gb, which takes a hell of a long time to drain to small-parallel (or non-parallel) spinning media vdevs. Such a flush can implicate a sequencing lock (e.g. you wish to modify something that is pending write in that buffer) which has the potential to lead to further misbehavior in the form of long delays before the system responds. If you have all spinning media and few parallel channels on the vdevs (or none) for writes lowering the max_max will be of material benefit in leveling out I/O performance with no penalty on peak write rates. However, if your machine is mixed SSD and spinning rust there's no "one size fits all", nor is there if you have pools with varying degrees of parallelism. > > To be clear, however, it should not be necessary to set parameters like= > these in /boot/loader.conf in order to obtain consistent operational > behavior. I'd be curious to know if someone running 10.2 BETA without > patches is able to trigger this behavior or not. There was work done t= hat > reported helped with this between 10.1 and now. To what extent it help= ed, > however, I don't have any advice yet. > > -sc I looked at the changes in 10.2-PRE and didn't see anything that led me to believe would materially change behavior. I have not, however, had the time to run an exhaustive test suite against unpatched 10.2-PRE. --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms030302000908000202050400 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGXzCC BlswggRDoAMCAQICASkwDQYJKoZIhvcNAQELBQAwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEg U3lzdGVtcyBMTEMgQ0EwHhcNMTUwNDIxMDIyMTU5WhcNMjAwNDE5MDIyMTU5WjBaMQswCQYD VQQGEwJVUzEQMA4GA1UECBMHRmxvcmlkYTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEe MBwGA1UEAxMVS2FybCBEZW5uaW5nZXIgKE9DU1ApMIICIjANBgkqhkiG9w0BAQEFAAOCAg8A MIICCgKCAgEAuYRY+EB2mGtZ3grlVO8TmnEvduVFA/IYXcCmNSOC1q+pTVjylsjcHKBcOPb9 TP1KLxdWP+Q1soSORGHlKw2/HcVzShDW5WPIKrvML+Ry0XvIvNBu9adTiCsA9nci4Cnf98XE hVpenER0qbJkBUOGT1rP4iAcfjet0lEgzPEnm+pAxv6fYSNp1WqIY9u0b1pkQiaWrt8hgNOc rJOiLbc8CeQ/DBP6rUiQjYNO9/aPNauEtHkNNfR9RgLSfGUdZuOCmJqnIla1HsrZhA5p69Bv /e832BKiNPaH5wF6btAiPpTr2sRhwQO8/IIxcRX1Vxd1yZbjYtJGw+9lwEcWRYAmoxkzKLPi S6Zo/6z5wgNpeK1H+zOioMoZIczgI8BlX1iHxqy/FAvm4PHPnC8s+BLnJLwr+jvMNHm82QwL J9hC5Ho8AnFU6TkCuq+P2V8/clJVqnBuvTUKhYMGSm4mUp+lAgR4L+lwIEqSeWVsxirIcE7Z OKkvI7k5x3WeE3+c6w74L6PfWVAd84xFlo9DKRdU9YbkFuFZPu21fi/LmE5brImB5P+jdqnK eWnVwRq+RBFLy4kehCzMXooitAwgP8l/JJa9VDiSyd/PAHaVGiat2vCdDh4b8cFL7SV6jPA4 k0MgGUA/6Et7wDmhZmCigggr9K6VQCx8jpKB3x1NlNNiaWECAwEAAaOB9DCB8TA3BggrBgEF BQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNV HRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIFoDALBgNVHQ8EBAMCBeAwLAYJYIZIAYb4QgENBB8W HU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1UdDgQWBBTFHJQt6cloXBdG1Pv1 o2YgH+7lWTAfBgNVHSMEGDAWgBQkcZudhX383d29sMqSlAOh+tNtNTAdBgNVHREEFjAUgRJr YXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcNAQELBQADggIBAE9/dxi2YqjCYYhiybp4GKcm 7tBVa/GLW+qcHPcoT4dqmqghlLz8+iUH+HCJjRQATVGyMEnvISOKFVHC6aZIG+Sg7J8bfS4+ fjKDi9smRH2VPPx3bV8+yFYRNroMGHaPHZB/Xctmmvc+PZ9O2W7rExgrODtxIOB3Zs6wkYf+ ty+9r1KmTHlV+rRHI6timH1uiyFE3cPi1taAEBxf0851cJV8k40PGF8G48ewnq8SY9sCf5cv liXbpdgU+I4ND5BuTjg63WS32zuhLd1VSuH3ZC/QbcncMX5W3oLXmcQP5/5uTiBJy74kdPtG MSZ9rXwZPwNxP/8PXMSR7ViaFvjUkf4bJlyENFa2PGxLk4EUzOuO7t3brjMlQW1fuInfG+ko 3tVxko20Hp0tKGPe/9cOxBVBZeZH/VgpZn3cLculGzZjmdh2fqAQ6kv9Z9AVOG1+dq0c1zt8 2zm+Oi1pikGXkfz5UJq60psY6zbX25BuEZkthO/qiS4pxjxb7gQkS0rTEHTy+qv0l3QVL0wa NAT74Zaj7l5DEW3qdQQ0dtVieyvptg9CxkfQJE3JyBMb0zBj9Qhc5/hbTfhSlHzZMEbUuIyx h9vxqFAmGzfB1/WfOKkiNHChkpPW8ZeH9yPeDBKvrgZ96dREHFoVkDk7Vpw5lSM+tFOfdyLg xxhb/RZVUDeUMYIE4zCCBN8CAQEwgZYwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQIEwdGbG9y aWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBMTEMxHDAa BgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEgU3lzdGVt cyBMTEMgQ0ECASkwCQYFKw4DAhoFAKCCAiEwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAc BgkqhkiG9w0BCQUxDxcNMTUwNzE0MTY1OTA1WjAjBgkqhkiG9w0BCQQxFgQUpBLxuXKzV6Xf jjmA1nBPkGX01SQwbAYJKoZIhvcNAQkPMV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAEC MAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzAN BggqhkiG9w0DAgIBKDCBpwYJKwYBBAGCNxAEMYGZMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBAgEpMIGpBgsqhkiG9w0BCRACCzGBmaCBljCBkDELMAkGA1UE BhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQ Q3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqG SIb3DQEJARYTQ3VkYSBTeXN0ZW1zIExMQyBDQQIBKTANBgkqhkiG9w0BAQEFAASCAgCtr/mp 2xtCp9lqANDC3n4kvVD7qrikc7O0mC9ynESVFyU7kgbYW0fiZctA3R0kNUlpWcLrPKBi+7Ob hS8G1ryf/fJYYn5PhLQ6NIekk/EI7P1wG0T+HqjdhtF1feY3IZ0Ox10DuDXoxOegp+y0yR4x pGEmDRDChew6fReVazpUeViQBKZahmfwjETIhQp33MgN+ZPXKfCh3A0HlGENB9qBymEcy8x/ zp/JcstHkuBLgV1IAUGW0bt4QopUkw3fqjGNK1IJ1L9NQyAmbYpGsp5TaRbay2WYkxF7nB8z 7AmjRU3zsvYEfORrjpsR69DY0wK3+IIKwdSD1HxDRzF1fEWexJxxX4L1gmeuRIm6/LKTid7x I5xFsNWdhmTXZ3Xmzoci9jRBebFSPX0MuD7t5KvV5LLlA79hyaph821TUCKylBty9jeH3VeB qW4y6I+IwQt++Cs43ORPFclQ/7hi0fU/SSWg9khXzTmxMr4/1+T5ksD7/XBLu7Y8N44XEiVY 1cvk4oeUQpSGIZ4HvBqP/3Dl4eOydlVvoFTwPFhEiR9r5dZkzZ+ujdPtZY/cdykOl+3MJhgU wwl68QvLMlVGBvFLfGMHbSmXlYAZgclbXcNXOOzyKHadlJF9WYpKYIXFocSIHM8/5Oh1w8Fn Zga0kX7AlMJMWWivxrv6YZzG5PQIBAAAAAAAAA== --------------ms030302000908000202050400-- From owner-freebsd-fs@freebsd.org Tue Jul 14 17:27:37 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C32899CA01 for ; Tue, 14 Jul 2015 17:27:37 +0000 (UTC) (envelope-from accounts-ag@netlabs.org) Received: from r2-d2.netlabs.org (r2-d2.netlabs.org [213.238.45.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D12961CFF for ; Tue, 14 Jul 2015 17:27:36 +0000 (UTC) (envelope-from accounts-ag@netlabs.org) Received: (qmail 74368 invoked by uid 89); 14 Jul 2015 17:27:33 -0000 Received: from unknown (HELO eternal-3.metropolis.netlabs.org) (accounts-ag@netlabs.org@213.144.156.18) by 0 with ESMTPA; 14 Jul 2015 17:27:33 -0000 Subject: Re: ZFS volume does not seem to free space for deleted files To: "Matthew D. Fuller" References: <55A4DB1D.7000404@netlabs.org> <20150714100023.GC96394@over-yonder.net> Cc: freebsd-fs@freebsd.org From: accounts-ag Message-ID: <55A54684.7060408@netlabs.org> Date: Tue, 14 Jul 2015 19:27:32 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.0.1 MIME-Version: 1.0 In-Reply-To: <20150714100023.GC96394@over-yonder.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 17:27:37 -0000 On 14.07.15 12:00, Matthew D. Fuller wrote: > I'm pretty sure you can't just add up those USED's and get a number > suggesting how much space deleting them all will free up. AIUI, the > USED number for a snapshot means how much that one alone is using; > e.g., how much space would be freed by deleting just that one. So any > blocks referenced by 2 or more wouldn't show up in those numbers. A > bunch of files that have been sitting around a long time would be a > good candidate for existing in most (/all) of them, so you won't see > them in per-snapshot USED until you delete all but 1. looks right, I found this later: # zfs list -t all -r tank/jail/www NAME USED AVAIL REFER MOUNTPOINT [...] tank/jail/www@hourly-2015-07-13_04.04.00--2d 257M - 46.1G - tank/jail/www@hourly-2015-07-13_10.04.00--2d 260M - 46.5G - tank/jail/www@hourly-2015-07-13_16.04.00--2d 1.70G - 46.7G - tank/jail/www@hourly-2015-07-13_22.04.00--2d 48.4M - 22.3G - tank/jail/www@daily-2015-07-14_00.00.00--2w 32.4M - 4.61G - tank/jail/www@hourly-2015-07-14_04.04.00--2d 20.0M - 4.60G - tank/jail/www@hourly-2015-07-14_10.04.00--2d 20.4M - 4.61G - tank/jail/www@hourly-2015-07-14_16.04.00--2d 19.8M - 4.57G - so one can clearly see where I deleted the files. Will now delete the snapshots before that point and see if this solves it. regards Adrian From owner-freebsd-fs@freebsd.org Tue Jul 14 18:09:29 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D53849A12EC for ; Tue, 14 Jul 2015 18:09:29 +0000 (UTC) (envelope-from ml-ktk@netlabs.org) Received: from r2-d2.netlabs.org (r2-d2.netlabs.org [213.238.45.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 35AA118F for ; Tue, 14 Jul 2015 18:09:28 +0000 (UTC) (envelope-from ml-ktk@netlabs.org) Received: (qmail 78892 invoked by uid 89); 14 Jul 2015 18:09:27 -0000 Received: from unknown (HELO eternal-3.metropolis.netlabs.org) (ml-ktk@netlabs.org@213.144.156.18) by 0 with ESMTPA; 14 Jul 2015 18:09:27 -0000 Subject: Re: ZFS volume does not seem to free space for deleted files To: freebsd-fs@freebsd.org References: <55A4DB1D.7000404@netlabs.org> <20150714100023.GC96394@over-yonder.net> <55A54684.7060408@netlabs.org> From: Adrian Gschwend Message-ID: <55A55056.5010208@netlabs.org> Date: Tue, 14 Jul 2015 20:09:26 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.0.1 MIME-Version: 1.0 In-Reply-To: <55A54684.7060408@netlabs.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 18:09:29 -0000 On 14.07.15 19:27, accounts-ag wrote: > so one can clearly see where I deleted the files. Will now delete the > snapshots before that point and see if this solves it. ok that solved it: # zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT tank 295G 111G 184G 37% 1.00x ONLINE - regards Adrian From owner-freebsd-fs@freebsd.org Tue Jul 14 18:19:10 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9B449A1744 for ; Tue, 14 Jul 2015 18:19:10 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from mango.stankevitz.com (mango.stankevitz.com [208.79.93.194]) by mx1.freebsd.org (Postfix) with ESMTP id B6BFABCB for ; Tue, 14 Jul 2015 18:19:10 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from Chriss-MacBook-Pro.local (209-203-101-124.static.twtelecom.net [209.203.101.124]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mango.stankevitz.com (Postfix) with ESMTPSA id A9CAC6F8; Tue, 14 Jul 2015 11:19:09 -0700 (PDT) Message-ID: <55A552A0.3010408@stankevitz.com> Date: Tue, 14 Jul 2015 11:19:12 -0700 From: Chris Stankevitz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Freddie Cash CC: FreeBSD Filesystems Subject: Re: Offline/online a drive from a mounted RW RAIDZx pool: resilver needed? References: <55A4461F.6020308@stankevitz.com> <55A53BE2.7060609@stankevitz.com> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 18:19:10 -0000 On 7/14/15 9:46 AM, Freddie Cash wrote: > If nothing is written to the pool (or, at least that specific vdev) > while the drive is removed, then the metadata on the drive will match > the pool, and it can be onlined without a resilver. > > If only a little bit of data has been written, and a short bit of time > has passed, ​ > ​then the drive will only need to sync (resilver) that changed data to > the drive. > > If a lot of data has been written, or a long time has passed (not sure > what qualifies as "long", maybe until the last txg synced to the disk is > no longer in the txg history of the pool?), then the entire drive will > be resilvered. > > Or, something along those lines. That's what I've observed by doing > various tests (offline drive, pull drive, write to pool, online drive; > or offline drive, replace with new one, start resilver, new drive > dies, stick in old drive, online it, wait for resilver; and other > scenarios). Freddie, Thank you. That certainly agrees with what I experienced. You can imagine my terror when I yanked the wrong drive... and my fear/confusion when the drive did not resilver as I expected. Chris From owner-freebsd-fs@freebsd.org Wed Jul 15 02:36:19 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DDDB9A2847 for ; Wed, 15 Jul 2015 02:36:19 +0000 (UTC) (envelope-from janm@transactionware.com) Received: from mail3.transactionware.com (mail.transactionware.com [203.14.245.7]) by mx1.freebsd.org (Postfix) with SMTP id 71CED11B0 for ; Wed, 15 Jul 2015 02:36:17 +0000 (UTC) (envelope-from janm@transactionware.com) Received: (qmail 42186 invoked by uid 907); 15 Jul 2015 02:36:14 -0000 Received: from Unknown (HELO jmmacpro.tmst.com.au) (203.14.245.130) (smtp-auth username janm, mechanism plain) by mail3.transactionware.com (qpsmtpd/0.84) with (ECDHE-RSA-AES256-SHA encrypted) ESMTPSA; Wed, 15 Jul 2015 12:36:14 +1000 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: FreeBSD 10.1 Memory Exhaustion From: Jan Mikkelsen In-Reply-To: <55A3A800.5060904@denninger.net> Date: Wed, 15 Jul 2015 12:36:13 +1000 Cc: freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <55A3A800.5060904@denninger.net> To: Karl Denninger X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2015 02:36:19 -0000 Hi, To make the 2015-02-05 patch work on 10.2-BETA1 I had to add this patch = as well: --- = //depot/vendor/freebsd/10-stable-local/src/sys/cddl/contrib/opensolaris/ut= s/common/fs/zfs/zio.c 2015-07-03 02:03:52.000000000 1000 +++ = /home/janm/p4/freebsd-std/FreeBSD/src/sys/cddl/contrib/opensolaris/uts/com= mon/fs/zfs/zio.c 2015-07-03 02:03:52.000000000 1000 @@ -44,9 +44,9 @@ SYSCTL_DECL(_vfs_zfs); SYSCTL_NODE(_vfs_zfs, OID_AUTO, zio, CTLFLAG_RW, 0, "ZFS ZIO"); #if defined(__amd64__) -static int zio_use_uma =3D 1; +int zio_use_uma =3D 1; #else -static int zio_use_uma =3D 0; +int zio_use_uma =3D 0; #endif TUNABLE_INT("vfs.zfs.zio.use_uma", &zio_use_uma); SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, use_uma, CTLFLAG_RDTUN, = &zio_use_uma, 0, Without changing the storage class of zio_use_uma this error appeared at = runtime "link_elf: symbol zio_use_uma undefined=94. This is unsurprising = because the linker can=92t resolve an extern symbol to a static in = another file. How are other people using this patch, or am I missing = something fundamental? Regards, Jan. > On 13 Jul 2015, at 21:58, Karl Denninger wrote: >=20 > Put this on your box and see if the problem goes away.... :-) >=20 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D187594 >=20 > The 2015-02-10 refactor will apply against 10.1-STABLE and 10.2-PRE = (the > latter will give you a 10-line fuzz in one block but applies and = works.) >=20 > I've been unable to provoke misbehavior with this patch in and I run a > cron job that does auto-snapshotting. There are others that have run > this patch with similarly positive results. >=20 > On 7/13/2015 06:48, Christopher Forgeron wrote: >> TL;DR Summary: I can run FreeBSD out of memory quite consistently, = and it=92s >> not a TOS/mbuf exhaustion issue. It=92s quite possible that ZFS is = the >> culprit, but shouldn=92t the pager be able to handle aggressive = memory >> requests in a low memory situation gracefully, without needing custom >> tuning of ZFS / VM? >>=20 >>=20 >> Hello, >>=20 >> I=92ve been dealing with some instability in my 10.1-RELEASE and >> STABLEr282701M machines for the last few months. >>=20 >> These machines are NFS/iSCSI storage machines, running on Dell M610x = or >> similar hardware, 96 Gig Memory, 10Gig Network Cards, dual Xeon = Processors >> =96 Fairly beefy stuff. >>=20 >> Initially I thought it was more issues with TOS / jumbo mbufs, as I = had >> this problem last year. I had thought that this was properly = resolved, but >> setting my MTU to 1500, and turning off TOS did give me a bit more >> stability. Currently all my machines are set this way. >>=20 >> Crashes were usually represented by loss of network connectivity, and = the >> ctld daemon scrolling messages across the screen at full speed about = lost >> connections. >>=20 >> All of this did seem like more network stack problems, but with each = crash >> I=92d be able to learn a bit more. >>=20 >> Usually there was nothing of any use in the logfile, but every now = and then >> I=92d get this: >>=20 >> Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 >> (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory >> Jun 3 13:02:04 san0 kernel: WARNING: icl_pdu_new: failed to allocate = 80 >> bytes >> Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 >> (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory >> Jun 3 13:02:04 san0 kernel: WARNING: icl_pdu_new: failed to allocate = 80 >> bytes >> Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 >> (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory >> --------- >> Jun 4 03:03:09 san0 kernel: WARNING: icl_pdu_new: failed to allocate = 80 >> bytes >> Jun 4 03:03:09 san0 kernel: WARNING: icl_pdu_new: failed to allocate = 80 >> bytes >> Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 >> (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory >> Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 >> (iqn.1998-01.com.vmware:esx5a-3387a188): connection error; dropping >> connection >> Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 >> (iqn.1998-01.com.vmware:esx5a-3387a188): connection error; dropping >> connection >> Jun 4 03:03:10 san0 kernel: WARNING: 172.16.0.97 >> (iqn.1998-01.com.vmware:esx5a-3387a188): waiting for CTL to terminate >> tasks, 1 remaining >> Jun 4 06:04:27 san0 syslogd: kernel boot file is /boot/kernel/kernel >>=20 >> So knowing that it seemed to be running out of memory, I started = leaving >> leaving =91vmstat 5=92 running on a console, to see what it was = displaying >> during the crash. >>=20 >> It was always the same thing: >>=20 >> 0 0 0 1520M 4408M 15 0 0 0 25 19 0 0 21962 1667 = 91390 >> 0 33 67 >> 0 0 0 1520M 4310M 9 0 0 0 2 15 3 0 21527 1385 = 95165 >> 0 31 69 >> 0 0 0 1520M 4254M 7 0 0 0 14 19 0 0 17664 1739 = 72873 >> 0 18 82 >> 0 0 0 1520M 4145M 2 0 0 0 0 19 0 0 23557 1447 = 96941 >> 0 36 64 >> 0 0 0 1520M 4013M 4 0 0 0 14 19 0 0 4288 490 = 34685 >> 0 72 28 >> 0 0 0 1520M 3885M 2 0 0 0 0 19 0 0 11141 1038 = 69242 >> 0 52 48 >> 0 0 0 1520M 3803M 10 0 0 0 14 19 0 0 24102 1834 = 91050 >> 0 33 67 >> 0 0 0 1520M 8192B 2 0 0 0 2 15 1 0 19037 1131 = 77470 >> 0 45 55 >> 0 0 0 1520M 8192B 0 22 0 0 2 0 6 0 146 82 = 578 0 >> 0 100 >> 0 0 0 1520M 8192B 1 0 0 0 0 0 0 0 130 40 = 510 0 >> 0 100 >> 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 143 40 = 501 0 >> 0 100 >> 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 201 62 = 660 0 >> 0 100 >> 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 101 28 = 404 0 >> 0 100 >> 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 97 27 = 398 0 >> 0 100 >> 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 93 28 = 377 0 >> 0 100 >> 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 92 27 = 373 0 >> 0 100 >>=20 >>=20 >> I=92d go from a decent amount of free memory to suddenly having none. = Vmstat >> would stop outputting, console commands would hang, etc. The whole = system >> would be useless. >>=20 >> Looking into this, I came across a similar issue; >>=20 >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199189 >>=20 >> I started increasing v.v_free_min, and it helped =96 My crashes went = from >> being ~every 6 hours to every few days. >>=20 >> Currently I=92m running with vm.v_free_min=3D1254507 =96 That=92s = (1254507 * 4KiB) >> , or 4.78GiB of Reserve. The vmstat above is of a machine with that >> setting still running to 8B of memory. >>=20 >> I have two issues here: >>=20 >> 1) I don=92t think I should ever be able to run the system into the = ground on >> memory. Deny me new memory until the pager can free more. >> 2) Setting =91min=92 doesn=92t really mean =91min=92 as it can = obviously go below >> that threshold. >>=20 >>=20 >> I have plenty of local UFS swap (non-ZFS drives) >>=20 >> Adrian requested that I output a few more diagnostic items, and this = is >> what I=92m running on a console now, in a loop: >>=20 >> vmstat >> netstat -m >> vmstat -z >> sleep 1 >>=20 >> The output of four crashes are attached here, as they can be a bit = long. >> Let me know if that=92s not a good way to report them. They will each = start >> mid-way through a vmstat =96z output, as that=92s as far back as my = terminal >> buffer allows. >>=20 >>=20 >>=20 >> Now, I have a good idea of the conditions that are causing this: ZFS >> Snapshots, run by cron, during times of high ZFS writes. >>=20 >> The crashes are all nearly on the hour, as that=92s when crontab = triggers my >> python scripts to make new snapshots, and delete old ones. >>=20 >> My average FreeBSD machine has ~ 30 zfs datasets, with each pool = having ~20 >> TiB used. These all need to snapshot on the hour. >>=20 >> By staggering the snapshots by a few minutes, I have been able to = reduce >> crashing from every other day to perhaps once a week if I=92m lucky =96= But if >> I start moving a lot of data around, I can cause daily crashes again. >>=20 >> It=92s looking to be the memory demand of snapshotting lots of ZFS = datasets >> at the same time while accepting a lot of write traffic. >>=20 >> Now perhaps the answer is =91don=92t do that=92 but I feel that = FreeBSD should be >> robust enough to handle this. I don=92t mind tuning for now to >> reduce/eliminate this, but others shouldn=92t run into this pain just = because >> they heavily load their machines =96 There must be a way of avoiding = this >> condition. >>=20 >> Here are the contents of my /boot/loader.conf and sysctl.conf, so = show my >> minimal tuning to make this problem a little more bearable: >>=20 >> /boot/loader.conf >> vfs.zfs.arc_meta_limit=3D49656727553 >> vfs.zfs.arc_max =3D 91489280512 >>=20 >> /etc/sysctl.conf >> vm.v_free_min=3D1254507 >>=20 >>=20 >> Any suggestions/help is appreciated. >>=20 >> Thank you. >>=20 >>=20 >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >=20 > --=20 > Karl Denninger > karl@denninger.net > /The Market Ticker/ > /[S/MIME encrypted email preferred]/ From owner-freebsd-fs@freebsd.org Wed Jul 15 17:37:20 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E1359A29F9; Wed, 15 Jul 2015 17:37:20 +0000 (UTC) (envelope-from dieterbsd@gmail.com) Received: from mail-ig0-x236.google.com (mail-ig0-x236.google.com [IPv6:2607:f8b0:4001:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4E3F61C92; Wed, 15 Jul 2015 17:37:20 +0000 (UTC) (envelope-from dieterbsd@gmail.com) Received: by igvi1 with SMTP id i1so82616523igv.1; Wed, 15 Jul 2015 10:37:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=Gy3H2s4CSpQWi0u5RhQBajRjbP49PON3N+867t4x0Ak=; b=LbUPURhYDF/GMwYWtX8neQ+4FwW4ecO+95c0v7fQdXeg1MPwlWuiohYNjXBLgFHdI2 XthF1RFHvkHPHYhDhXYpC4eCR8/Wew1GB8G4vTQjRrDTDfsT/cX7SU0GjNaWK4/VYBRJ q4sMqQq+tgvvKPKPlT9P4hS9LXHmcV0Eaeewkm/g1C8mpolDrfeU4OcT9kNSJ9EvX1Hk P/3zYd+t6dtfiK3bijFlxdh3Kr2noxJdxS6wjI3o76t/r8yRHnJWI4CM6jqGqzAxDD1e 4h3uJ3tP1UXChTlubOqsPXKsbUR844g1KRoh9Qh3YyrnKFKHLvia5HOZqdAZ8XuSKm0f fmvg== MIME-Version: 1.0 X-Received: by 10.107.6.194 with SMTP id f63mr5901717ioi.61.1436981839598; Wed, 15 Jul 2015 10:37:19 -0700 (PDT) Received: by 10.64.2.132 with HTTP; Wed, 15 Jul 2015 10:37:19 -0700 (PDT) Date: Wed, 15 Jul 2015 10:37:19 -0700 Message-ID: Subject: Re: format/newfs larger external consumer drives From: Dieter BSD To: freebsd-hackers@freebsd.org, freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2015 17:37:20 -0000 [ freebsd-fs@ added ] >> If the average filesize will be large, use large block/frag sizes. >> I use 64 KiB / 8 KiB. And reduce the number of inodes. I reduce >> inodes as much as newfs allows and there are still way too many. > > Can you think of an algorithmic way to express this? I.e., you don't > want blocks to get *too* large as you risk greater losses in "partial > fragments", etc. Likewise, you don't want to run out of inodes. I look at df -i for existing filesystems with similar filesizes. My data filesystems usually get an entire disk (..., 2TB, 3TB, recently 5TB) and with 64/8 block/frag and as few inodes as newfs will allow df still reports numbers like 97% full but only using 0% or 1% of inodes. density reduced from 67108864 to 14860288 /dev/ada1: 4769307.0MB (9767541168 sectors) block size 65536, fragment size 8192 using 1315 cylinder groups of 3628.00MB, 58048 blks, 256 inodes. with soft updates I should take another look at increasing the size of cylinder groups. Newfs likes very small cylinder groups, which made sense 30 years when disks were like 40 MB and file sizes were a lot smaller. IIRC, each cylinder group gets at least one block of inodes, and with file sizes of 1-20 GB I get way too many inodes. Yes, a larger frag size will waste some space in the last frag of a file, but having smaller block&frag sizes uses a lot of space to keep track of all those blocks and frags. And makes more work for fsck. > "risk" of loss/cost of recovery (when the medium > *is* unceremoniously dismounted Some panics don't sync the disks. Sometimes disks just go into a coma. Soft updates is supposed to limit problems to those that fsck -p will automagicly fix. (assuming the disk's write cache is turned off) There is at least one case where it does not. See PR 166499 (from 2012, still not fixed). As long as I'm whining about unfixed filesystem PRs, see also bin/170676: Newfs creates a filesystem that does not pass fsck. (also from 2012) > I am concerned with the fact that users can so easily/carelessly "unplug" > a USB device without the proper incantations beforehand. of course, *their* > mistake is seen as a "product design flaw"! :-/ Superglue the cable in place? :-) Perhaps print up something like "Unmount filesystem(s) before unplugging or powering off external disk, or you might lose your data.", laminate it and attach it to the cables? > The "demo app" that I'm working on is a sort of (low performance) NAS > built on SBC's and external drives. I assume that the drives *have* to be external? Do they have to be usb? Could they be e-sata? E-sata is faster and avoids the various usb problems. They used to sell external drives where the sata-to-usb bridge was in a separate little module box. They had alternate modules with e-sata, firewire, etc. The disk box had a standard internal ('L') sata connector, except a standard sata connector was too large to fit. So I took out my Swiss Army Knife and carved off some plastic from the connector on a standard sata cable so that it would fit. You could also put a standard sata drive into an enclosure (with a small fan) and use your choice of connection to the computer. >> USB specific stuff: There is an off by 1 sector problem, which will >> bite you if you switch a drive between using the sata-usb bridge >> and connecting the drive directly to a sata controller. I had to > > Ouch! I've not seen that with PATA-USB bridges. OTOH, I tend not > to pull a drive *from* an external enclosure but, rather, rely on > the external enclosures to provide portability. E.g., easier to > move 500G of files from machineA to machineB by physically moving > the volume containing them! Apparently they vary, see the message from Warren. Mine was missing the first sector, so I had to have the kernel hunt for the partitioning info. The external drives I've seen do not have fans, and have little or no ventilation. If the drive will be spinning for awhile I worry about it overheating. > The "demo app" will try to use the large multi-TB drives of which I > have little long-term experience. OTOH, the usage model is "fire it > up, pull off whichever files you need, then spin everything down"... > until the next time you might need to retrieve an ISO (a week later?) With this usage model it sounds like you could use a read-only mount. Would an optical drive work for this application? >> If the drive disappears with filesystem(s) mounted. the kernel might >> very well panic. There was a discussion of this problem recently. >> I thought that FUSE was suggested as a possible solution, but I >> can't find the discussion. This problem is not limited to users >> disconnecting usb drives without unmounting them. The problem >> happens all by itself with internal drives, as the drive, port >> multiplier, controller, or device driver decides to go out to lunch, >> and the kernel panics. This happens *far* too often, and *kills* >> reliability. We really need a solution for this. > > I think its hard to back-port these sorts of things. Much easier > to consider the possibility of failure when initially designing the > system, interfaces, etc. I wonder how hard it would be to create a FUSE version of FFS? Any thoughts from the filesystem wizards? Alternately, instead of panicing, could the filesystem just umount -f the offending filesystem? (And whine to log(9).) I am very tired of having an entire machine panic just because one disk decided to take a nap. This is not how you get 5 9s. :-( From owner-freebsd-fs@freebsd.org Wed Jul 15 19:43:32 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3BCD49A2915 for ; Wed, 15 Jul 2015 19:43:32 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1032A1A2A for ; Wed, 15 Jul 2015 19:43:31 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 0DC12208E7 for ; Wed, 15 Jul 2015 15:43:31 -0400 (EDT) Received: from web3 ([10.202.2.213]) by compute5.internal (MEProxy); Wed, 15 Jul 2015 15:43:31 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=qFLf8LH1a9u4yj0 qCJWvQbW4sBM=; b=rquQnQ4BGKeYlALYp8WIkyl9Su5n+U8tUW+KaUud6xiUsOe wI1tiUeAEuH3Br+C33ivBWZIIOjkbQzskA7zPRaePfaWIj8VEQsRJJu/LamDPwPw jtYy5NroBRbolAueXMnjuP1Hlqvkb0dwPZjDjJv4LoMtqjnFnqvkD9G32PkI= Received: by web3.nyi.internal (Postfix, from userid 99) id DF7F510DF90; Wed, 15 Jul 2015 15:43:30 -0400 (EDT) Message-Id: <1436989410.1427298.324703241.421E814B@webmail.messagingengine.com> X-Sasl-Enc: fccPh+gYR3qspyxbwjAFaBU4kW6O3je3VtA4a1iB1j+l 1436989410 From: Mark Felder To: Sean Chittenden , Adrian Gschwend Cc: FreeBSD Filesystems MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Mailer: MessagingEngine.com Webmail Interface - ajax-63a5d8c6 Subject: Re: FreeBSD 10.1 Memory Exhaustion Date: Wed, 15 Jul 2015 14:43:30 -0500 In-Reply-To: References: <55A3A800.5060904@denninger.net> <55A4D5B7.2030603@freebsd.org> <55A4E5AB.8060909@netlabs.org> X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2015 19:43:32 -0000 On Tue, Jul 14, 2015, at 10:10, Sean Chittenden wrote: > I think the reason this is not seen more often is because people > frequently > throw limits on the arc in /boot/loader.conf: > > vfs.zfs.arc_min="18G" > vfs.zfs.arc_max="149G" > > ZFS ARC *should* not require those settings, but does currently for mixed > workloads (i.e. databases) in order to be "stable". By setting fixed > sizes > on the ARC, UMA and ARC are much more cooperative in that they have their > own memory regions to manage so this behavior is not seen as often. > > To be clear, however, it should not be necessary to set parameters like > these in /boot/loader.conf in order to obtain consistent operational > behavior. I'd be curious to know if someone running 10.2 BETA without > patches is able to trigger this behavior or not. There was work done > that > reported helped with this between 10.1 and now. To what extent it > helped, > however, I don't have any advice yet. > I was about to email "I have 12TB at home and 4GB of RAM with a very erratic workload and never run into any issues" and then I looked at /boot/loader.conf and saw vfs.zfs.arc_max="2G" Now I'm too scared to turn it off... :-) From owner-freebsd-fs@freebsd.org Wed Jul 15 20:21:33 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9C109A2082 for ; Wed, 15 Jul 2015 20:21:33 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [31.223.170.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 97A931ED0; Wed, 15 Jul 2015 20:21:32 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 86342153431; Wed, 15 Jul 2015 22:21:21 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XUz8g5rF6mx0; Wed, 15 Jul 2015 22:20:51 +0200 (CEST) Received: from [IPv6:2001:4cb8:3:1:6994:6919:ad85:1cbb] (unknown [IPv6:2001:4cb8:3:1:6994:6919:ad85:1cbb]) by smtp.digiware.nl (Postfix) with ESMTPA id A21E715340A; Wed, 15 Jul 2015 22:20:51 +0200 (CEST) Subject: Re: FreeBSD 10.1 Memory Exhaustion To: Mark Felder , Sean Chittenden , Adrian Gschwend References: <55A3A800.5060904@denninger.net> <55A4D5B7.2030603@freebsd.org> <55A4E5AB.8060909@netlabs.org> <1436989410.1427298.324703241.421E814B@webmail.messagingengine.com> Cc: FreeBSD Filesystems From: Willem Jan Withagen Message-ID: <55A6C0A4.1030300@digiware.nl> Date: Wed, 15 Jul 2015 22:20:52 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.0.1 MIME-Version: 1.0 In-Reply-To: <1436989410.1427298.324703241.421E814B@webmail.messagingengine.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2015 20:21:33 -0000 On 15/07/2015 21:43, Mark Felder wrote: > > > On Tue, Jul 14, 2015, at 10:10, Sean Chittenden wrote: >> I think the reason this is not seen more often is because people >> frequently >> throw limits on the arc in /boot/loader.conf: >> >> vfs.zfs.arc_min="18G" >> vfs.zfs.arc_max="149G" >> >> ZFS ARC *should* not require those settings, but does currently for mixed >> workloads (i.e. databases) in order to be "stable". By setting fixed >> sizes >> on the ARC, UMA and ARC are much more cooperative in that they have their >> own memory regions to manage so this behavior is not seen as often. >> >> To be clear, however, it should not be necessary to set parameters like >> these in /boot/loader.conf in order to obtain consistent operational >> behavior. I'd be curious to know if someone running 10.2 BETA without >> patches is able to trigger this behavior or not. There was work done >> that >> reported helped with this between 10.1 and now. To what extent it >> helped, >> however, I don't have any advice yet. >> > > I was about to email "I have 12TB at home and 4GB of RAM with a very > erratic workload and never run into any issues" and then I looked at > /boot/loader.conf and saw vfs.zfs.arc_max="2G" > > Now I'm too scared to turn it off... :-) Same here. Just a leftover of all the advise to limit arc in the past. Just bit the bullit: installed BETA1, killed the settings and rebooted. We'll see what comes of it. --WjW From owner-freebsd-fs@freebsd.org Wed Jul 15 20:38:44 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E29479A23EA for ; Wed, 15 Jul 2015 20:38:44 +0000 (UTC) (envelope-from karl@denninger.net) Received: from fs.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "NewFS.denninger.net", Issuer "NewFS.denninger.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A6B4B1B2B for ; Wed, 15 Jul 2015 20:38:43 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (localhost [127.0.0.1]) by fs.denninger.net (8.14.9/8.14.8) with ESMTP id t6FKcZfd016568 for ; Wed, 15 Jul 2015 15:38:35 -0500 (CDT) (envelope-from karl@denninger.net) Received: from [192.168.1.40] [192.168.1.40] (Via SSLv3 AES128-SHA) ; by Spamblock-sys (LOCAL/AUTH) Wed Jul 15 15:38:35 2015 Message-ID: <55A6C4BD.5090500@denninger.net> Date: Wed, 15 Jul 2015 15:38:21 -0500 From: Karl Denninger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: FreeBSD 10.1 Memory Exhaustion References: <55A3A800.5060904@denninger.net> <55A4D5B7.2030603@freebsd.org> <55A4E5AB.8060909@netlabs.org> <1436989410.1427298.324703241.421E814B@webmail.messagingengine.com> <55A6C0A4.1030300@digiware.nl> In-Reply-To: <55A6C0A4.1030300@digiware.nl> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms090206030707000801090107" X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2015 20:38:45 -0000 This is a cryptographically signed message in MIME format. --------------ms090206030707000801090107 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 7/15/2015 15:20, Willem Jan Withagen wrote: > On 15/07/2015 21:43, Mark Felder wrote: >> >> On Tue, Jul 14, 2015, at 10:10, Sean Chittenden wrote: >>> I think the reason this is not seen more often is because people >>> frequently >>> throw limits on the arc in /boot/loader.conf: >>> >>> vfs.zfs.arc_min=3D"18G" >>> vfs.zfs.arc_max=3D"149G" >>> >>> ZFS ARC *should* not require those settings, but does currently for m= ixed >>> workloads (i.e. databases) in order to be "stable". By setting fixed= >>> sizes >>> on the ARC, UMA and ARC are much more cooperative in that they have t= heir >>> own memory regions to manage so this behavior is not seen as often. >>> >>> To be clear, however, it should not be necessary to set parameters li= ke >>> these in /boot/loader.conf in order to obtain consistent operational >>> behavior. I'd be curious to know if someone running 10.2 BETA withou= t >>> patches is able to trigger this behavior or not. There was work done= >>> that >>> reported helped with this between 10.1 and now. To what extent it >>> helped, >>> however, I don't have any advice yet. >>> >> I was about to email "I have 12TB at home and 4GB of RAM with a very >> erratic workload and never run into any issues" and then I looked at >> /boot/loader.conf and saw vfs.zfs.arc_max=3D"2G" >> >> Now I'm too scared to turn it off... :-) > Same here. > Just a leftover of all the advise to limit arc in the past. > Just bit the bullit: installed BETA1, killed the settings and rebooted.= > We'll see what comes of it. > > --WjW If you get bit I have refactored the patch for 10.2-BETA1 to get rid of the fudges and i386 linker problem and uploaded it, so if you run into trouble try applying that and see if that fixes it. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D187594 --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms090206030707000801090107 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGXzCC BlswggRDoAMCAQICASkwDQYJKoZIhvcNAQELBQAwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEg U3lzdGVtcyBMTEMgQ0EwHhcNMTUwNDIxMDIyMTU5WhcNMjAwNDE5MDIyMTU5WjBaMQswCQYD VQQGEwJVUzEQMA4GA1UECBMHRmxvcmlkYTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEe MBwGA1UEAxMVS2FybCBEZW5uaW5nZXIgKE9DU1ApMIICIjANBgkqhkiG9w0BAQEFAAOCAg8A MIICCgKCAgEAuYRY+EB2mGtZ3grlVO8TmnEvduVFA/IYXcCmNSOC1q+pTVjylsjcHKBcOPb9 TP1KLxdWP+Q1soSORGHlKw2/HcVzShDW5WPIKrvML+Ry0XvIvNBu9adTiCsA9nci4Cnf98XE hVpenER0qbJkBUOGT1rP4iAcfjet0lEgzPEnm+pAxv6fYSNp1WqIY9u0b1pkQiaWrt8hgNOc rJOiLbc8CeQ/DBP6rUiQjYNO9/aPNauEtHkNNfR9RgLSfGUdZuOCmJqnIla1HsrZhA5p69Bv /e832BKiNPaH5wF6btAiPpTr2sRhwQO8/IIxcRX1Vxd1yZbjYtJGw+9lwEcWRYAmoxkzKLPi S6Zo/6z5wgNpeK1H+zOioMoZIczgI8BlX1iHxqy/FAvm4PHPnC8s+BLnJLwr+jvMNHm82QwL J9hC5Ho8AnFU6TkCuq+P2V8/clJVqnBuvTUKhYMGSm4mUp+lAgR4L+lwIEqSeWVsxirIcE7Z OKkvI7k5x3WeE3+c6w74L6PfWVAd84xFlo9DKRdU9YbkFuFZPu21fi/LmE5brImB5P+jdqnK eWnVwRq+RBFLy4kehCzMXooitAwgP8l/JJa9VDiSyd/PAHaVGiat2vCdDh4b8cFL7SV6jPA4 k0MgGUA/6Et7wDmhZmCigggr9K6VQCx8jpKB3x1NlNNiaWECAwEAAaOB9DCB8TA3BggrBgEF BQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNV HRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIFoDALBgNVHQ8EBAMCBeAwLAYJYIZIAYb4QgENBB8W HU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1UdDgQWBBTFHJQt6cloXBdG1Pv1 o2YgH+7lWTAfBgNVHSMEGDAWgBQkcZudhX383d29sMqSlAOh+tNtNTAdBgNVHREEFjAUgRJr YXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcNAQELBQADggIBAE9/dxi2YqjCYYhiybp4GKcm 7tBVa/GLW+qcHPcoT4dqmqghlLz8+iUH+HCJjRQATVGyMEnvISOKFVHC6aZIG+Sg7J8bfS4+ fjKDi9smRH2VPPx3bV8+yFYRNroMGHaPHZB/Xctmmvc+PZ9O2W7rExgrODtxIOB3Zs6wkYf+ ty+9r1KmTHlV+rRHI6timH1uiyFE3cPi1taAEBxf0851cJV8k40PGF8G48ewnq8SY9sCf5cv liXbpdgU+I4ND5BuTjg63WS32zuhLd1VSuH3ZC/QbcncMX5W3oLXmcQP5/5uTiBJy74kdPtG MSZ9rXwZPwNxP/8PXMSR7ViaFvjUkf4bJlyENFa2PGxLk4EUzOuO7t3brjMlQW1fuInfG+ko 3tVxko20Hp0tKGPe/9cOxBVBZeZH/VgpZn3cLculGzZjmdh2fqAQ6kv9Z9AVOG1+dq0c1zt8 2zm+Oi1pikGXkfz5UJq60psY6zbX25BuEZkthO/qiS4pxjxb7gQkS0rTEHTy+qv0l3QVL0wa NAT74Zaj7l5DEW3qdQQ0dtVieyvptg9CxkfQJE3JyBMb0zBj9Qhc5/hbTfhSlHzZMEbUuIyx h9vxqFAmGzfB1/WfOKkiNHChkpPW8ZeH9yPeDBKvrgZ96dREHFoVkDk7Vpw5lSM+tFOfdyLg xxhb/RZVUDeUMYIE4zCCBN8CAQEwgZYwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQIEwdGbG9y aWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBMTEMxHDAa BgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEgU3lzdGVt cyBMTEMgQ0ECASkwCQYFKw4DAhoFAKCCAiEwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAc BgkqhkiG9w0BCQUxDxcNMTUwNzE1MjAzODIxWjAjBgkqhkiG9w0BCQQxFgQUTWU8hGAivyHM Vnw22QEhQJwA5hMwbAYJKoZIhvcNAQkPMV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAEC MAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzAN BggqhkiG9w0DAgIBKDCBpwYJKwYBBAGCNxAEMYGZMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBAgEpMIGpBgsqhkiG9w0BCRACCzGBmaCBljCBkDELMAkGA1UE BhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQ Q3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqG SIb3DQEJARYTQ3VkYSBTeXN0ZW1zIExMQyBDQQIBKTANBgkqhkiG9w0BAQEFAASCAgAl+pFu /06vAB31ivchq8G4UzUkbFUWqtcKuYdm4CZeV5n5nFgjB3roI0WYqAKxq5HuvmqN5UpOmOJt Ve//qUt83aEDIMf5Y1RqKtJd+aty3omx7czllLn/eMU2J2QXjj0QptefcudTVb4Uop/BKqQq nY8Xd8bK91fn+58FKlHx/oekojosUkAmWPt8bZIo47aktuGTx0EvKup9g0nzZTZPl2PrE5AH dx8whKC3Ttir/VjTI9A65yBqa3jDyQwZPGEe6Mo2gyeMxX3I3ZgcG+RTHRYA0IgsZ7VASNV/ 1qBgPcVbvfybERiWWmNurYN9P7+W+pJeIV3MVr9jZm/2QiHAfWIX17Nd2fx4a3zEmMPO4TMp Uyb5f0h0C7dkyD4NWCWM0lVxrQAWuGxoVnmYEKpBI95W89GydgFQ5kA28Zz13Qi+22D0qg3e 08a6fll1xvJ8vFm6k7jYu4qbAxYG9sZEU5LWb8Eo9Mpd/xZwc6jzXJvikd+TcpElJMqMqq5x HgW23vDD1lMEC/NjzALmKyvM9OE6kTF0lXl3fySfCZwS71Ed5HdCTUyRD8GchzhidhN0ir0t KJoEdWbz2N91cCXzjQ0vf7A9DZI52RurxPBmOaJhfG6TEBx3ntd3SzaGjVUleK3KQhC9rCbB i26J48FS9W1vizsLmqVXOwc/XBEQbwAAAAAAAA== --------------ms090206030707000801090107-- From owner-freebsd-fs@freebsd.org Wed Jul 15 20:42:26 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37F179A2585; Wed, 15 Jul 2015 20:42:26 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (wonkity.com [67.158.26.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "wonkity.com", Issuer "wonkity.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BF71C1FDD; Wed, 15 Jul 2015 20:42:25 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.14.9/8.14.9) with ESMTP id t6FKgNp9051714 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 15 Jul 2015 14:42:23 -0600 (MDT) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.14.9/8.14.9/Submit) with ESMTP id t6FKgNv8051711; Wed, 15 Jul 2015 14:42:23 -0600 (MDT) (envelope-from wblock@wonkity.com) Date: Wed, 15 Jul 2015 14:42:23 -0600 (MDT) From: Warren Block To: Dieter BSD cc: freebsd-hackers@freebsd.org, freebsd-fs@freebsd.org Subject: Re: format/newfs larger external consumer drives In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (wonkity.com [127.0.0.1]); Wed, 15 Jul 2015 14:42:23 -0600 (MDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2015 20:42:26 -0000 On Wed, 15 Jul 2015, Dieter BSD wrote: > I wonder how hard it would be to create a FUSE version of FFS? > Any thoughts from the filesystem wizards? http://sourceforge.net/projects/fuse-ufs2/ Untested by me, though. From owner-freebsd-fs@freebsd.org Wed Jul 15 22:06:22 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BC4A79A3613; Wed, 15 Jul 2015 22:06:22 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 847721B28; Wed, 15 Jul 2015 22:06:22 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t6FM6LpY072052 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 15 Jul 2015 15:06:21 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t6FM6LBS072051; Wed, 15 Jul 2015 15:06:21 -0700 (PDT) (envelope-from jmg) Date: Wed, 15 Jul 2015 15:06:21 -0700 From: John-Mark Gurney To: Dieter BSD Cc: freebsd-hackers@freebsd.org, freebsd-fs@freebsd.org Subject: Re: format/newfs larger external consumer drives Message-ID: <20150715220621.GP8523@funkthat.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Wed, 15 Jul 2015 15:06:21 -0700 (PDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2015 22:06:22 -0000 Dieter BSD wrote this message on Wed, Jul 15, 2015 at 10:37 -0700: > [ freebsd-fs@ added ] > > >> If the average filesize will be large, use large block/frag sizes. > >> I use 64 KiB / 8 KiB. And reduce the number of inodes. I reduce > >> inodes as much as newfs allows and there are still way too many. > > > > Can you think of an algorithmic way to express this? I.e., you don't > > want blocks to get *too* large as you risk greater losses in "partial > > fragments", etc. Likewise, you don't want to run out of inodes. > > I look at df -i for existing filesystems with similar filesizes. > My data filesystems usually get an entire disk (..., 2TB, 3TB, recently 5TB) > and with 64/8 block/frag and as few inodes as newfs will allow > df still reports numbers like 97% full but only using 0% or 1% > of inodes. > > density reduced from 67108864 to 14860288 > /dev/ada1: 4769307.0MB (9767541168 sectors) block size 65536, fragment size 8192 > using 1315 cylinder groups of 3628.00MB, 58048 blks, 256 inodes. > with soft updates > > I should take another look at increasing the size of cylinder groups. Right now the cg by default is made to fill a block... I don't believe it can be made larger without a major overhaul of the code... The default used to be even smaller than a full block causing even more cg's to be created and you had to do trial and error to figure out how to make a cg a full block... > Newfs likes very small cylinder groups, which made sense 30 years when > disks were like 40 MB and file sizes were a lot smaller. IIRC, each > cylinder group gets at least one block of inodes, and with file sizes > of 1-20 GB I get way too many inodes. This is partly the default number of inodes are too large... The current documented default is an inode for every 4 * frag_size bytes of data space, which isn't correct!!! This was changed to 2 in r228794 to keep the number of inodes the same when the transition from 16k/2k to 32k/4k happened, but the documentation was not updated... It has now been updated in r285615 and will be MFC'd... On my dev server where I have a few source trees checked out: Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/ada0s2d 185G 122G 48G 72% 2.8M 9.5M 23% /a This fs has non-standard config in that my frag size is 8k... If it was standard, I'd have twice as many inodes... Increaseing the frag size both cuts the # of inodes in half, but also increases the cg size... Standard: /dev/ada0s2d: 192068.0MB (393355264 sectors) block size 32768, fragment size 4096 using 307 cylinder groups of 626.09MB, 20035 blks, 80256 inodes. Non-standard: /dev/ada0s2d: 192068.0MB (393355264 sectors) block size 32768, fragment size 8192 using 166 cylinder groups of 1162.97MB, 37215 blks, 74496 inodes. The other thing I didn't realize (and would be useful for someone to benchmark) is that many SSD's now use 8k page size instead of the previous 4k.. Maybe this needs to be more of a sliding scale based upon disk size? Maybe go from 2 * frag to 4 * frag at fs's larger than 1TB? Though this is still something that a system admin needs to address, it's impossible to make the defaults sane for all use cases... There are some people that will only keep multi GB files on their 5 TB fs, and so only need a few thousand inodes, but others may keep more smaller files... It'd be nice to put together a fs survey to see what sizes of filesystems people have, and the distribution of files sizes... I'll try to do that... > Yes, a larger frag size will waste some space in the last frag of a file, > but having smaller block&frag sizes uses a lot of space to keep track of > all those blocks and frags. And makes more work for fsck. Yep... > > "risk" of loss/cost of recovery (when the medium > > *is* unceremoniously dismounted > > Some panics don't sync the disks. Sometimes disks just go into a coma. > Soft updates is supposed to limit problems to those that fsck -p will > automagicly fix. (assuming the disk's write cache is turned off) There > is at least one case where it does not. See PR 166499 (from 2012, > still not fixed). > > As long as I'm whining about unfixed filesystem PRs, see also > bin/170676: Newfs creates a filesystem that does not pass fsck. > (also from 2012) > > > I am concerned with the fact that users can so easily/carelessly "unplug" > > a USB device without the proper incantations beforehand. of course, *their* > > mistake is seen as a "product design flaw"! :-/ > > Superglue the cable in place? :-) > > Perhaps print up something like "Unmount filesystem(s) before unplugging > or powering off external disk, or you might lose your data.", > laminate it and attach it to the cables? Same problem goes for Windows.. They have a policy of turning of write buffering on pluggable thumb drives to help eliminate this.. For UFS, the sync flag should be provided to mount... [...] > Alternately, instead of panicing, could the filesystem just > umount -f the offending filesystem? (And whine to log(9).) > > I am very tired of having an entire machine panic just because > one disk decided to take a nap. This is not how you get 5 9s. :-( There has been lots of work to try to make file systems not panic when the underlying drives disappear, though clearly more work is needed... Patches welcome! :) -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-fs@freebsd.org Fri Jul 17 00:15:57 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C0FF9A397C for ; Fri, 17 Jul 2015 00:15:57 +0000 (UTC) (envelope-from allan@physics.umn.edu) Received: from mail.physics.umn.edu (smtp.spa.umn.edu [128.101.220.4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47CB61E0F for ; Fri, 17 Jul 2015 00:15:56 +0000 (UTC) (envelope-from allan@physics.umn.edu) Received: from peevish.spa.umn.edu ([128.101.220.230]) by mail.physics.umn.edu with esmtp (Exim 4.77 (FreeBSD)) (envelope-from ) id 1ZFsv0-000KAE-D3 for freebsd-fs@freebsd.org; Thu, 16 Jul 2015 18:50:22 -0500 Received: by peevish.spa.umn.edu (Postfix, from userid 5000) id 521EED34; Thu, 16 Jul 2015 18:50:22 -0500 (CDT) Date: Thu, 16 Jul 2015 18:50:22 -0500 From: Graham Allan To: Ahmed Kamal via freebsd-fs Subject: Re: Linux NFSv4 clients are getting (bad sequence-id error!) Message-ID: <20150716235022.GF32479@physics.umn.edu> References: <684628776.2772174.1435793776748.JavaMail.zimbra@uoguelph.ca> <55947C6E.5060409@delphij.net> <1491630362.2785531.1435799383802.JavaMail.zimbra@uoguelph.ca> <5594B008.10202@freebsd.org> <1022558302.2863702.1435838360534.JavaMail.zimbra@uoguelph.ca> <791936587.3443190.1435873993955.JavaMail.zimbra@uoguelph.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-12-10) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jul 2015 00:15:57 -0000 I'm curious how things are going for you with this? Reading your thread did pique my interest since we have a lot of Scientific Linux (RHEL clone) boxes with FreeBSD NFSv4 servers. I meant to glance through our logs for signs of the same issue, but today I started investigating a machine which appeared to have hung processes, high rpciod load, and high traffic to the NFS server. Of course it is exactly this issue. The affected machine is running SL5 though most of our server nodes are now SL6. I can see errors from most of them but the SL6 systems appear less affected - I see a stream of the sequence-id errors in their logs but things in general keep working. The one SL5 machine I'm looking at has a single sequence-id error in today's logs, but then goes into a stream of "state recovery failed" then "Lock reclaim failed". It's probably partly related to the particular workload on this machine. I would try switching our SL6 machines to NFS 4.1 to see if the behaviour changes, but 4.1 isn't supported by our 9.3 servers (is it in 10.1?). At the NFS servers, most of the sysctl settings are already tuned from defaults. eg tcp.highwater=100000, vfs.nfsd.tcpcachetimeo=300, 128-256 nfs kernel threads. Graham On Fri, Jul 03, 2015 at 01:21:00AM +0200, Ahmed Kamal via freebsd-fs wrote: > PS: Today (after adjusting tcp.highwater) I didn't get any screaming > reports from users about hung vnc sessions. So maybe just maybe, linux > clients are able to somehow recover from this bad sequence messages. I > could still see the bad sequence error message in logs though > > Why isn't the highwater tunable set to something better by default ? I mean > this server is certainly not under a high or unusual load (it's only 40 PCs > mounting from it) > > On Fri, Jul 3, 2015 at 1:15 AM, Ahmed Kamal > wrote: > > > Thanks all .. I understand now we're doing the "right thing" .. Although > > if mounting keeps wedging, I will have to solve it somehow! Either using > > Xin's patch .. or Upgrading RHEL to 6.x and using NFS4.1. > > > > Regarding Xin's patch, is it possible to build the patched nfsd code, as a > > kernel module ? I'm looking to minimize my delta to upstream. > > > > Also would adopting Xin's patch and hiding it behind a > > kern.nfs.allow_linux_broken_client be an option (I'm probably not the last > > person on earth to hit this) ? > > > > Thanks a lot for all the help! > > > > On Thu, Jul 2, 2015 at 11:53 PM, Rick Macklem > > wrote: > > > >> Ahmed Kamal wrote: > >> > Appreciating the fruitful discussion! Can someone please explain to me, > >> > what would happen in the current situation (linux client doing this > >> > skip-by-1 thing, and freebsd not doing it) ? What is the effect of that? > >> Well, as you've seen, the Linux client doesn't function correctly against > >> the FreeBSD server (and probably others that don't support this > >> "skip-by-1" > >> case). > >> > >> > What do users see? Any chances of data loss? > >> Hmm. Mostly it will cause Opens to fail, but I can't guess what the Linux > >> client behaviour is after receiving NFS4ERR_BAD_SEQID. You're the guy > >> observing > >> it. > >> > >> > > >> > Also, I find it strange that netapp have acknowledged this is a bug on > >> > their side, which has been fixed since then! > >> Yea, I think Netapp screwed up. For some reason their server allowed this, > >> then was fixed to not allow it and then someone decided that was broken > >> and > >> reversed it. > >> > >> > I also find it strange that I'm the first to hit this :) Is no one > >> running > >> > nfs4 yet! > >> > > >> Well, it seems to be slowly catching on. I suspect that the Linux client > >> mounting a Netapp is the most common use of it. Since it appears that they > >> flip flopped w.r.t. who's bug this is, it has probably persisted. > >> > >> It may turn out that the Linux client has been fixed or it may turn out > >> that most servers allowed this "skip-by-1" even though David Noveck (one > >> of the main authors of the protocol) seems to agree with me that it should > >> not be allowed. > >> > >> It is possible that others have bumped into this, but it wasn't isolated > >> (I wouldn't have guessed it, so it was good you pointed to the RedHat > >> discussion) > >> and they worked around it by reverting to NFSv3 or similar. > >> The protocol is rather complex in this area and changed completely for > >> NFSv4.1, > >> so many have also probably moved onto NFSv4.1 where this won't be an > >> issue. > >> (NFSv4.1 uses sessions to provide exactly once RPC semantics and doesn't > >> use > >> these seqid fields.) > >> > >> This is all just mho, rick > >> > >> > On Thu, Jul 2, 2015 at 1:59 PM, Rick Macklem > >> wrote: > >> > > >> > > Julian Elischer wrote: > >> > > > On 7/2/15 9:09 AM, Rick Macklem wrote: > >> > > > > I am going to post to nfsv4@ietf.org to see what they say. Please > >> > > > > let me know if Xin Li's patch resolves your problem, even though I > >> > > > > don't believe it is correct except for the UINT32_MAX case. Good > >> > > > > luck with it, rick > >> > > > and please keep us all in the loop as to what they say! > >> > > > > >> > > > the general N+2 bit sounds like bullshit to me.. its always N+1 in a > >> > > > number field that has a > >> > > > bit of slack at wrap time (probably due to some ambiguity in the > >> > > > original spec). > >> > > > > >> > > Actually, since N is the lock op already done, N + 1 is the next lock > >> > > operation in order. Since lock ops need to be strictly ordered, > >> allowing > >> > > N + 2 (which means N + 2 would be done before N + 1) makes no sense. > >> > > > >> > > I think the author of the RFC meant that N + 2 or greater fails, but > >> it > >> > > was poorly worded. > >> > > > >> > > I will pass along whatever I get from nfsv4@ietf.org. (There is an > >> archive > >> > > of it somewhere, but I can't remember where.;-) > >> > > > >> > > rick > >> > > _______________________________________________ > >> > > freebsd-fs@freebsd.org mailing list > >> > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > >> > > > >> > > >> > > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" -- ------------------------------------------------------------------------- Graham Allan - allan@physics.umn.edu - gta@umn.edu - (612) 624-5040 School of Physics and Astronomy - University of Minnesota ------------------------------------------------------------------------- From owner-freebsd-fs@freebsd.org Fri Jul 17 11:22:18 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 29EA29A31FE for ; Fri, 17 Jul 2015 11:22:18 +0000 (UTC) (envelope-from email.ahmedkamal@googlemail.com) Received: from mail-wi0-x232.google.com (mail-wi0-x232.google.com [IPv6:2a00:1450:400c:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A24A91A68 for ; Fri, 17 Jul 2015 11:22:17 +0000 (UTC) (envelope-from email.ahmedkamal@googlemail.com) Received: by wicmv11 with SMTP id mv11so40005964wic.1 for ; Fri, 17 Jul 2015 04:22:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=Mtklms+aZyIlGk05TbTVT/Q3JiVP3EzBDtIREmEsIx0=; b=FHG0fmSewelbdbUwN5r9HH7UQ/G4SxohlyCkQFzRxDIXynv7ig6eAMgbb++aLNlAse eAXS9ff3xLfjVEndE7HnbOLYEhBKipD6XMK4HeVVm7PHi3H7W/HoJFCuGtkrDUniDf4I We7jTZC9PdbZDwtyAUTLrsBHBAbvRPqF42+tWhssbWfsid8I4hn4hcf+i6QtUFsIOU2h Qgvfp9bhD/onhnpaQMT5hEMPuxysARgT4xy8TIfpwQLJ525Mt60SC6Wvl7L6cRroGzG9 h8wk6G8I9FnvNRDP5kKFcUOFsE9sMu3pH/dr2p8+BtjhpsZVjXKl7jJyq8e+gqbIMwAC ZiOQ== X-Received: by 10.194.59.98 with SMTP id y2mr29193036wjq.42.1437132135994; Fri, 17 Jul 2015 04:22:15 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.6.143 with HTTP; Fri, 17 Jul 2015 04:21:56 -0700 (PDT) In-Reply-To: <20150716235022.GF32479@physics.umn.edu> References: <684628776.2772174.1435793776748.JavaMail.zimbra@uoguelph.ca> <55947C6E.5060409@delphij.net> <1491630362.2785531.1435799383802.JavaMail.zimbra@uoguelph.ca> <5594B008.10202@freebsd.org> <1022558302.2863702.1435838360534.JavaMail.zimbra@uoguelph.ca> <791936587.3443190.1435873993955.JavaMail.zimbra@uoguelph.ca> <20150716235022.GF32479@physics.umn.edu> From: Ahmed Kamal Date: Fri, 17 Jul 2015 13:21:56 +0200 Message-ID: Subject: Re: Linux NFSv4 clients are getting (bad sequence-id error!) To: Graham Allan Cc: Ahmed Kamal via freebsd-fs Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jul 2015 11:22:18 -0000 Hi Graham, So my RHEL5 boxes certainly have trouble with nfs4 .. I'm running about 20 boxes and almost all of them develop a choking process every day or two. I'm now in the process of upgrading our RHEL boxes to v6.x .. This is motivated to migrate to NFS4.1, although now that you say NFS4 is more tolerant on EL6, I might just remain on that. So far I did one week of basic testing of a VM on el6 with nfs4.1 vs my FreeBSD 10.1, so far I didn't hit problems (although the testing was light). Next week, I'll probably upgrade one of our production machines to el6 and see how it fares. PS: I had upgraded our el5 box with elrepo kernel (v3.2) .. which I thought would be way much newer (even newer than el6) .. But I still had trouble with it .. so I reverted to stock el5 kernel! Not sure if this means Linux is not the only component at fault ?! On Fri, Jul 17, 2015 at 1:50 AM, Graham Allan wrote: > I'm curious how things are going for you with this? > > Reading your thread did pique my interest since we have a lot of > Scientific Linux (RHEL clone) boxes with FreeBSD NFSv4 servers. I meant > to glance through our logs for signs of the same issue, but today I > started investigating a machine which appeared to have hung processes, > high rpciod load, and high traffic to the NFS server. Of course it is > exactly this issue. > > The affected machine is running SL5 though most of our server nodes are > now SL6. I can see errors from most of them but the SL6 systems appear > less affected - I see a stream of the sequence-id errors in their logs but > things in general keep working. The one SL5 machine I'm looking at > has a single sequence-id error in today's logs, but then goes into a > stream of "state recovery failed" then "Lock reclaim failed". It's > probably partly related to the particular workload on this machine. > > I would try switching our SL6 machines to NFS 4.1 to see if the > behaviour changes, but 4.1 isn't supported by our 9.3 servers (is it in > 10.1?). > > At the NFS servers, most of the sysctl settings are already tuned > from defaults. eg tcp.highwater=100000, vfs.nfsd.tcpcachetimeo=300, > 128-256 nfs kernel threads. > > Graham > > On Fri, Jul 03, 2015 at 01:21:00AM +0200, Ahmed Kamal via freebsd-fs wrote: > > PS: Today (after adjusting tcp.highwater) I didn't get any screaming > > reports from users about hung vnc sessions. So maybe just maybe, linux > > clients are able to somehow recover from this bad sequence messages. I > > could still see the bad sequence error message in logs though > > > > Why isn't the highwater tunable set to something better by default ? I > mean > > this server is certainly not under a high or unusual load (it's only 40 > PCs > > mounting from it) > > > > On Fri, Jul 3, 2015 at 1:15 AM, Ahmed Kamal < > email.ahmedkamal@googlemail.com > > > wrote: > > > > > Thanks all .. I understand now we're doing the "right thing" .. > Although > > > if mounting keeps wedging, I will have to solve it somehow! Either > using > > > Xin's patch .. or Upgrading RHEL to 6.x and using NFS4.1. > > > > > > Regarding Xin's patch, is it possible to build the patched nfsd code, > as a > > > kernel module ? I'm looking to minimize my delta to upstream. > > > > > > Also would adopting Xin's patch and hiding it behind a > > > kern.nfs.allow_linux_broken_client be an option (I'm probably not the > last > > > person on earth to hit this) ? > > > > > > Thanks a lot for all the help! > > > > > > On Thu, Jul 2, 2015 at 11:53 PM, Rick Macklem > > > wrote: > > > > > >> Ahmed Kamal wrote: > > >> > Appreciating the fruitful discussion! Can someone please explain to > me, > > >> > what would happen in the current situation (linux client doing this > > >> > skip-by-1 thing, and freebsd not doing it) ? What is the effect of > that? > > >> Well, as you've seen, the Linux client doesn't function correctly > against > > >> the FreeBSD server (and probably others that don't support this > > >> "skip-by-1" > > >> case). > > >> > > >> > What do users see? Any chances of data loss? > > >> Hmm. Mostly it will cause Opens to fail, but I can't guess what the > Linux > > >> client behaviour is after receiving NFS4ERR_BAD_SEQID. You're the guy > > >> observing > > >> it. > > >> > > >> > > > >> > Also, I find it strange that netapp have acknowledged this is a bug > on > > >> > their side, which has been fixed since then! > > >> Yea, I think Netapp screwed up. For some reason their server allowed > this, > > >> then was fixed to not allow it and then someone decided that was > broken > > >> and > > >> reversed it. > > >> > > >> > I also find it strange that I'm the first to hit this :) Is no one > > >> running > > >> > nfs4 yet! > > >> > > > >> Well, it seems to be slowly catching on. I suspect that the Linux > client > > >> mounting a Netapp is the most common use of it. Since it appears that > they > > >> flip flopped w.r.t. who's bug this is, it has probably persisted. > > >> > > >> It may turn out that the Linux client has been fixed or it may turn > out > > >> that most servers allowed this "skip-by-1" even though David Noveck > (one > > >> of the main authors of the protocol) seems to agree with me that it > should > > >> not be allowed. > > >> > > >> It is possible that others have bumped into this, but it wasn't > isolated > > >> (I wouldn't have guessed it, so it was good you pointed to the RedHat > > >> discussion) > > >> and they worked around it by reverting to NFSv3 or similar. > > >> The protocol is rather complex in this area and changed completely for > > >> NFSv4.1, > > >> so many have also probably moved onto NFSv4.1 where this won't be an > > >> issue. > > >> (NFSv4.1 uses sessions to provide exactly once RPC semantics and > doesn't > > >> use > > >> these seqid fields.) > > >> > > >> This is all just mho, rick > > >> > > >> > On Thu, Jul 2, 2015 at 1:59 PM, Rick Macklem > > >> wrote: > > >> > > > >> > > Julian Elischer wrote: > > >> > > > On 7/2/15 9:09 AM, Rick Macklem wrote: > > >> > > > > I am going to post to nfsv4@ietf.org to see what they say. > Please > > >> > > > > let me know if Xin Li's patch resolves your problem, even > though I > > >> > > > > don't believe it is correct except for the UINT32_MAX case. > Good > > >> > > > > luck with it, rick > > >> > > > and please keep us all in the loop as to what they say! > > >> > > > > > >> > > > the general N+2 bit sounds like bullshit to me.. its always N+1 > in a > > >> > > > number field that has a > > >> > > > bit of slack at wrap time (probably due to some ambiguity in the > > >> > > > original spec). > > >> > > > > > >> > > Actually, since N is the lock op already done, N + 1 is the next > lock > > >> > > operation in order. Since lock ops need to be strictly ordered, > > >> allowing > > >> > > N + 2 (which means N + 2 would be done before N + 1) makes no > sense. > > >> > > > > >> > > I think the author of the RFC meant that N + 2 or greater fails, > but > > >> it > > >> > > was poorly worded. > > >> > > > > >> > > I will pass along whatever I get from nfsv4@ietf.org. (There is > an > > >> archive > > >> > > of it somewhere, but I can't remember where.;-) > > >> > > > > >> > > rick > > >> > > _______________________________________________ > > >> > > freebsd-fs@freebsd.org mailing list > > >> > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > >> > > To unsubscribe, send any mail to " > freebsd-fs-unsubscribe@freebsd.org" > > >> > > > > >> > > > >> > > > > > > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > -- > ------------------------------------------------------------------------- > Graham Allan - allan@physics.umn.edu - gta@umn.edu - (612) 624-5040 > School of Physics and Astronomy - University of Minnesota > ------------------------------------------------------------------------- > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Fri Jul 17 16:05:50 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5FA309A4E35 for ; Fri, 17 Jul 2015 16:05:50 +0000 (UTC) (envelope-from allan@physics.umn.edu) Received: from mail.physics.umn.edu (smtp.spa.umn.edu [128.101.220.4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 385A61F2E for ; Fri, 17 Jul 2015 16:05:49 +0000 (UTC) (envelope-from allan@physics.umn.edu) Received: from c-66-41-25-68.hsd1.mn.comcast.net ([66.41.25.68] helo=[192.168.0.2]) by mail.physics.umn.edu with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.77 (FreeBSD)) (envelope-from ) id 1ZG88x-000Dmj-Di; Fri, 17 Jul 2015 11:05:47 -0500 Message-ID: <55A927CE.5010505@physics.umn.edu> Date: Fri, 17 Jul 2015 11:05:34 -0500 From: Graham Allan User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Ahmed Kamal CC: Ahmed Kamal via freebsd-fs Subject: Re: Linux NFSv4 clients are getting (bad sequence-id error!) References: <684628776.2772174.1435793776748.JavaMail.zimbra@uoguelph.ca> <55947C6E.5060409@delphij.net> <1491630362.2785531.1435799383802.JavaMail.zimbra@uoguelph.ca> <5594B008.10202@freebsd.org> <1022558302.2863702.1435838360534.JavaMail.zimbra@uoguelph.ca> <791936587.3443190.1435873993955.JavaMail.zimbra@uoguelph.ca> <20150716235022.GF32479@physics.umn.edu> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jul 2015 16:05:50 -0000 I have maintenance scheduled this weekend so maybe I will try to add Xin Li's patch on one of our 9.3 servers and can see if the sequence-id messages diminish (even though it didn't help for you - possibly SL6 will behave differently). As for SL6/NFS4 being more tolerant, I suspect the problem is dependent on the specific job. This is the first time I have seen it at all (that is, with the stuck processes and high rpciod load), and I only see one person running this code. Although looking back ~60 days in logs I can see the sequence-id messages occurring all over the place from other machines, apparently without incident. For the more intense users who are running on 200 servers at once, I wonder if they are not hitting the NFS server in the same way - possibly they are mostly writing somewhere else like hadoop and only reading from NFS. However our compute farm conversions to SL6 and NFSv4 are fairly recent, so something may yet show up. I wonder if we have any avenue to file a bug with Redhat. I have a very basic subscription which only lets me look at their KB, but I could upgrade it - but then, as I'm running a clone product I probably don't have a viable report. Graham On 7/17/2015 6:21 AM, Ahmed Kamal wrote: > Hi Graham, > > So my RHEL5 boxes certainly have trouble with nfs4 .. I'm running about > 20 boxes and almost all of them develop a choking process every day or > two. I'm now in the process of upgrading our RHEL boxes to v6.x .. This > is motivated to migrate to NFS4.1, although now that you say NFS4 is > more tolerant on EL6, I might just remain on that. So far I did one week > of basic testing of a VM on el6 with nfs4.1 vs my FreeBSD 10.1, so far I > didn't hit problems (although the testing was light). Next week, I'll > probably upgrade one of our production machines to el6 and see how it fares. > > PS: I had upgraded our el5 box with elrepo kernel (v3.2) .. which I > thought would be way much newer (even newer than el6) .. But I still had > trouble with it .. so I reverted to stock el5 kernel! Not sure if this > means Linux is not the only component at fault ?! > -- ------------------------------------------------------------------------- Graham Allan - gta@umn.edu - allan@physics.umn.edu School of Physics and Astronomy - University of Minnesota ------------------------------------------------------------------------- From owner-freebsd-fs@freebsd.org Fri Jul 17 19:30:53 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89D7A9A2194 for ; Fri, 17 Jul 2015 19:30:53 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 293071F28 for ; Fri, 17 Jul 2015 19:30:52 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2ADAwAoWKdV/61jaINUAwMOg1lpBoMer1mIRAmBbAqFLUoCggUUAQEBAQEBAYEKhCMBAQEDAQEBASAEJyALBQsCAQgYAgINGQICJwEJJgIECAcEARwEh3gDCggNuQiQWQ2FQAEBAQEBBQEBAQEBAQEbgSKKKoJNgVYQAgEFCAEOJBAHEYIcOxKBMQWHC40zhG2CX4F2hBFGg1ODD4wvg18CJoINHIEVWiIxB39BgQQBAQE X-IronPort-AV: E=Sophos;i="5.15,497,1432612800"; d="scan'208";a="226411215" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 17 Jul 2015 15:29:43 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id C309D15F542; Fri, 17 Jul 2015 15:29:43 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id RpdVbG8E4fTR; Fri, 17 Jul 2015 15:29:42 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id E57BD15F55D; Fri, 17 Jul 2015 15:29:42 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id qqgenWBziywL; Fri, 17 Jul 2015 15:29:42 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id CCF2415F542; Fri, 17 Jul 2015 15:29:42 -0400 (EDT) Date: Fri, 17 Jul 2015 15:29:42 -0400 (EDT) From: Rick Macklem To: Graham Allan Cc: Ahmed Kamal via freebsd-fs Message-ID: <746249653.10948386.1437161382672.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <20150716235022.GF32479@physics.umn.edu> References: <684628776.2772174.1435793776748.JavaMail.zimbra@uoguelph.ca> <5594B008.10202@freebsd.org> <1022558302.2863702.1435838360534.JavaMail.zimbra@uoguelph.ca> <791936587.3443190.1435873993955.JavaMail.zimbra@uoguelph.ca> <20150716235022.GF32479@physics.umn.edu> Subject: Re: Linux NFSv4 clients are getting (bad sequence-id error!) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: Linux NFSv4 clients are getting (bad sequence-id error!) Thread-Index: keN759NpxQHHEv3H5bqHsFUzs/VLeQ== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jul 2015 19:30:53 -0000 Graham Allan wrote: > I'm curious how things are going for you with this? > > Reading your thread did pique my interest since we have a lot of > Scientific Linux (RHEL clone) boxes with FreeBSD NFSv4 servers. I meant > to glance through our logs for signs of the same issue, but today I > started investigating a machine which appeared to have hung processes, > high rpciod load, and high traffic to the NFS server. Of course it is > exactly this issue. > > The affected machine is running SL5 though most of our server nodes are > now SL6. I can see errors from most of them but the SL6 systems appear > less affected - I see a stream of the sequence-id errors in their logs but > things in general keep working. The one SL5 machine I'm looking at > has a single sequence-id error in today's logs, but then goes into a > stream of "state recovery failed" then "Lock reclaim failed". It's > probably partly related to the particular workload on this machine. > Normally, reclaims for Opens and byte range Locks only work after a server reboot (during the grace time period just after reboot). However, Opens can be re-acquired via a normal (non-reclaim) open. I don't know what the Linux client does in this case, but I suspect that might be doing a normal Open when the reclaim one fails. --> As such, things would start working normally again, but byte range Locks would get lost. - I suspect the high load on rpciod occurs when a read-ahead or write-behind keeps retrying with the "stale" stateid. (Maybe that behaviour was changed for SL6?) > I would try switching our SL6 machines to NFS 4.1 to see if the > behaviour changes, but 4.1 isn't supported by our 9.3 servers (is it in > 10.1?). > Yes, 10.1 has an NFSv4.1 server in it. (Since NFSv4.1 doesn't use the seqid#s, it shouldn't have the "bad seqid" errors.) Good luck with it, rick > At the NFS servers, most of the sysctl settings are already tuned > from defaults. eg tcp.highwater=100000, vfs.nfsd.tcpcachetimeo=300, > 128-256 nfs kernel threads. > > Graham > > On Fri, Jul 03, 2015 at 01:21:00AM +0200, Ahmed Kamal via freebsd-fs wrote: > > PS: Today (after adjusting tcp.highwater) I didn't get any screaming > > reports from users about hung vnc sessions. So maybe just maybe, linux > > clients are able to somehow recover from this bad sequence messages. I > > could still see the bad sequence error message in logs though > > > > Why isn't the highwater tunable set to something better by default ? I mean > > this server is certainly not under a high or unusual load (it's only 40 PCs > > mounting from it) > > > > On Fri, Jul 3, 2015 at 1:15 AM, Ahmed Kamal > > > > wrote: > > > > > Thanks all .. I understand now we're doing the "right thing" .. Although > > > if mounting keeps wedging, I will have to solve it somehow! Either using > > > Xin's patch .. or Upgrading RHEL to 6.x and using NFS4.1. > > > > > > Regarding Xin's patch, is it possible to build the patched nfsd code, as > > > a > > > kernel module ? I'm looking to minimize my delta to upstream. > > > > > > Also would adopting Xin's patch and hiding it behind a > > > kern.nfs.allow_linux_broken_client be an option (I'm probably not the > > > last > > > person on earth to hit this) ? > > > > > > Thanks a lot for all the help! > > > > > > On Thu, Jul 2, 2015 at 11:53 PM, Rick Macklem > > > wrote: > > > > > >> Ahmed Kamal wrote: > > >> > Appreciating the fruitful discussion! Can someone please explain to > > >> > me, > > >> > what would happen in the current situation (linux client doing this > > >> > skip-by-1 thing, and freebsd not doing it) ? What is the effect of > > >> > that? > > >> Well, as you've seen, the Linux client doesn't function correctly > > >> against > > >> the FreeBSD server (and probably others that don't support this > > >> "skip-by-1" > > >> case). > > >> > > >> > What do users see? Any chances of data loss? > > >> Hmm. Mostly it will cause Opens to fail, but I can't guess what the > > >> Linux > > >> client behaviour is after receiving NFS4ERR_BAD_SEQID. You're the guy > > >> observing > > >> it. > > >> > > >> > > > >> > Also, I find it strange that netapp have acknowledged this is a bug on > > >> > their side, which has been fixed since then! > > >> Yea, I think Netapp screwed up. For some reason their server allowed > > >> this, > > >> then was fixed to not allow it and then someone decided that was broken > > >> and > > >> reversed it. > > >> > > >> > I also find it strange that I'm the first to hit this :) Is no one > > >> running > > >> > nfs4 yet! > > >> > > > >> Well, it seems to be slowly catching on. I suspect that the Linux client > > >> mounting a Netapp is the most common use of it. Since it appears that > > >> they > > >> flip flopped w.r.t. who's bug this is, it has probably persisted. > > >> > > >> It may turn out that the Linux client has been fixed or it may turn out > > >> that most servers allowed this "skip-by-1" even though David Noveck (one > > >> of the main authors of the protocol) seems to agree with me that it > > >> should > > >> not be allowed. > > >> > > >> It is possible that others have bumped into this, but it wasn't isolated > > >> (I wouldn't have guessed it, so it was good you pointed to the RedHat > > >> discussion) > > >> and they worked around it by reverting to NFSv3 or similar. > > >> The protocol is rather complex in this area and changed completely for > > >> NFSv4.1, > > >> so many have also probably moved onto NFSv4.1 where this won't be an > > >> issue. > > >> (NFSv4.1 uses sessions to provide exactly once RPC semantics and doesn't > > >> use > > >> these seqid fields.) > > >> > > >> This is all just mho, rick > > >> > > >> > On Thu, Jul 2, 2015 at 1:59 PM, Rick Macklem > > >> wrote: > > >> > > > >> > > Julian Elischer wrote: > > >> > > > On 7/2/15 9:09 AM, Rick Macklem wrote: > > >> > > > > I am going to post to nfsv4@ietf.org to see what they say. > > >> > > > > Please > > >> > > > > let me know if Xin Li's patch resolves your problem, even though > > >> > > > > I > > >> > > > > don't believe it is correct except for the UINT32_MAX case. Good > > >> > > > > luck with it, rick > > >> > > > and please keep us all in the loop as to what they say! > > >> > > > > > >> > > > the general N+2 bit sounds like bullshit to me.. its always N+1 in > > >> > > > a > > >> > > > number field that has a > > >> > > > bit of slack at wrap time (probably due to some ambiguity in the > > >> > > > original spec). > > >> > > > > > >> > > Actually, since N is the lock op already done, N + 1 is the next > > >> > > lock > > >> > > operation in order. Since lock ops need to be strictly ordered, > > >> allowing > > >> > > N + 2 (which means N + 2 would be done before N + 1) makes no sense. > > >> > > > > >> > > I think the author of the RFC meant that N + 2 or greater fails, but > > >> it > > >> > > was poorly worded. > > >> > > > > >> > > I will pass along whatever I get from nfsv4@ietf.org. (There is an > > >> archive > > >> > > of it somewhere, but I can't remember where.;-) > > >> > > > > >> > > rick > > >> > > _______________________________________________ > > >> > > freebsd-fs@freebsd.org mailing list > > >> > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > >> > > To unsubscribe, send any mail to > > >> > > "freebsd-fs-unsubscribe@freebsd.org" > > >> > > > > >> > > > >> > > > > > > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > -- > ------------------------------------------------------------------------- > Graham Allan - allan@physics.umn.edu - gta@umn.edu - (612) 624-5040 > School of Physics and Astronomy - University of Minnesota > ------------------------------------------------------------------------- > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Fri Jul 17 19:32:09 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0FEB59A22FC for ; Fri, 17 Jul 2015 19:32:09 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 9BEEC1074 for ; Fri, 17 Jul 2015 19:32:07 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BKBQASV6lV/61jaINUAwMOg1lpBoMdr3SKQAqFLUoCggQRAQEBAQEBAYEKhCMBAQEDAQEBASArIAsFCwIBCBgCAg0ZAgInAQkmAgQIBwQBHASHeAMKCA26KJBRDYUuAQEBAQEBBAEBAQEBAQEbgSKKKoJNgVYQAgEFCAEOJBAHEYIcOxKBMQWHDo0+hG+CYIF8hBdGg1OPRoNfAiaCDRyBFVoiMQd/QYEEAQEB X-IronPort-AV: E=Sophos;i="5.15,497,1432612800"; d="scan'208";a="224652241" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 17 Jul 2015 15:32:00 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 4B44C15F542; Fri, 17 Jul 2015 15:32:00 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id UvmjOSL53b_8; Fri, 17 Jul 2015 15:31:59 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 78EDB15F55D; Fri, 17 Jul 2015 15:31:59 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id QDL8xeOeRvYB; Fri, 17 Jul 2015 15:31:59 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 61A4C15F542; Fri, 17 Jul 2015 15:31:59 -0400 (EDT) Date: Fri, 17 Jul 2015 15:31:59 -0400 (EDT) From: Rick Macklem To: Graham Allan Cc: Ahmed Kamal via freebsd-fs Message-ID: <184170291.10949389.1437161519387.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <20150716235022.GF32479@physics.umn.edu> References: <684628776.2772174.1435793776748.JavaMail.zimbra@uoguelph.ca> <5594B008.10202@freebsd.org> <1022558302.2863702.1435838360534.JavaMail.zimbra@uoguelph.ca> <791936587.3443190.1435873993955.JavaMail.zimbra@uoguelph.ca> <20150716235022.GF32479@physics.umn.edu> Subject: Re: Linux NFSv4 clients are getting (bad sequence-id error!) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.10] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: Linux NFSv4 clients are getting (bad sequence-id error!) Thread-Index: eZKd+hQdka2jJ4f723b3ygvnl3fiEw== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jul 2015 19:32:09 -0000 Graham Allan wrote: > I'm curious how things are going for you with this? > > Reading your thread did pique my interest since we have a lot of > Scientific Linux (RHEL clone) boxes with FreeBSD NFSv4 servers. I meant > to glance through our logs for signs of the same issue, but today I > started investigating a machine which appeared to have hung processes, > high rpciod load, and high traffic to the NFS server. Of course it is > exactly this issue. > > The affected machine is running SL5 though most of our server nodes are > now SL6. I can see errors from most of them but the SL6 systems appear > less affected - I see a stream of the sequence-id errors in their logs but > things in general keep working. The one SL5 machine I'm looking at > has a single sequence-id error in today's logs, but then goes into a > stream of "state recovery failed" then "Lock reclaim failed". It's > probably partly related to the particular workload on this machine. > > I would try switching our SL6 machines to NFS 4.1 to see if the > behaviour changes, but 4.1 isn't supported by our 9.3 servers (is it in > 10.1?). > Btw, I've done some testing against a fairly recent Fedora and haven't seen the problem. If either of you guys could load a recent Fedora on a test client box, it would be interesting to see if it suffers from this. (My experience is that the Fedora distros have more up to date Linux NFS clients.) rick > At the NFS servers, most of the sysctl settings are already tuned > from defaults. eg tcp.highwater=100000, vfs.nfsd.tcpcachetimeo=300, > 128-256 nfs kernel threads. > > Graham > > On Fri, Jul 03, 2015 at 01:21:00AM +0200, Ahmed Kamal via freebsd-fs wrote: > > PS: Today (after adjusting tcp.highwater) I didn't get any screaming > > reports from users about hung vnc sessions. So maybe just maybe, linux > > clients are able to somehow recover from this bad sequence messages. I > > could still see the bad sequence error message in logs though > > > > Why isn't the highwater tunable set to something better by default ? I mean > > this server is certainly not under a high or unusual load (it's only 40 PCs > > mounting from it) > > > > On Fri, Jul 3, 2015 at 1:15 AM, Ahmed Kamal > > > > wrote: > > > > > Thanks all .. I understand now we're doing the "right thing" .. Although > > > if mounting keeps wedging, I will have to solve it somehow! Either using > > > Xin's patch .. or Upgrading RHEL to 6.x and using NFS4.1. > > > > > > Regarding Xin's patch, is it possible to build the patched nfsd code, as > > > a > > > kernel module ? I'm looking to minimize my delta to upstream. > > > > > > Also would adopting Xin's patch and hiding it behind a > > > kern.nfs.allow_linux_broken_client be an option (I'm probably not the > > > last > > > person on earth to hit this) ? > > > > > > Thanks a lot for all the help! > > > > > > On Thu, Jul 2, 2015 at 11:53 PM, Rick Macklem > > > wrote: > > > > > >> Ahmed Kamal wrote: > > >> > Appreciating the fruitful discussion! Can someone please explain to > > >> > me, > > >> > what would happen in the current situation (linux client doing this > > >> > skip-by-1 thing, and freebsd not doing it) ? What is the effect of > > >> > that? > > >> Well, as you've seen, the Linux client doesn't function correctly > > >> against > > >> the FreeBSD server (and probably others that don't support this > > >> "skip-by-1" > > >> case). > > >> > > >> > What do users see? Any chances of data loss? > > >> Hmm. Mostly it will cause Opens to fail, but I can't guess what the > > >> Linux > > >> client behaviour is after receiving NFS4ERR_BAD_SEQID. You're the guy > > >> observing > > >> it. > > >> > > >> > > > >> > Also, I find it strange that netapp have acknowledged this is a bug on > > >> > their side, which has been fixed since then! > > >> Yea, I think Netapp screwed up. For some reason their server allowed > > >> this, > > >> then was fixed to not allow it and then someone decided that was broken > > >> and > > >> reversed it. > > >> > > >> > I also find it strange that I'm the first to hit this :) Is no one > > >> running > > >> > nfs4 yet! > > >> > > > >> Well, it seems to be slowly catching on. I suspect that the Linux client > > >> mounting a Netapp is the most common use of it. Since it appears that > > >> they > > >> flip flopped w.r.t. who's bug this is, it has probably persisted. > > >> > > >> It may turn out that the Linux client has been fixed or it may turn out > > >> that most servers allowed this "skip-by-1" even though David Noveck (one > > >> of the main authors of the protocol) seems to agree with me that it > > >> should > > >> not be allowed. > > >> > > >> It is possible that others have bumped into this, but it wasn't isolated > > >> (I wouldn't have guessed it, so it was good you pointed to the RedHat > > >> discussion) > > >> and they worked around it by reverting to NFSv3 or similar. > > >> The protocol is rather complex in this area and changed completely for > > >> NFSv4.1, > > >> so many have also probably moved onto NFSv4.1 where this won't be an > > >> issue. > > >> (NFSv4.1 uses sessions to provide exactly once RPC semantics and doesn't > > >> use > > >> these seqid fields.) > > >> > > >> This is all just mho, rick > > >> > > >> > On Thu, Jul 2, 2015 at 1:59 PM, Rick Macklem > > >> wrote: > > >> > > > >> > > Julian Elischer wrote: > > >> > > > On 7/2/15 9:09 AM, Rick Macklem wrote: > > >> > > > > I am going to post to nfsv4@ietf.org to see what they say. > > >> > > > > Please > > >> > > > > let me know if Xin Li's patch resolves your problem, even though > > >> > > > > I > > >> > > > > don't believe it is correct except for the UINT32_MAX case. Good > > >> > > > > luck with it, rick > > >> > > > and please keep us all in the loop as to what they say! > > >> > > > > > >> > > > the general N+2 bit sounds like bullshit to me.. its always N+1 in > > >> > > > a > > >> > > > number field that has a > > >> > > > bit of slack at wrap time (probably due to some ambiguity in the > > >> > > > original spec). > > >> > > > > > >> > > Actually, since N is the lock op already done, N + 1 is the next > > >> > > lock > > >> > > operation in order. Since lock ops need to be strictly ordered, > > >> allowing > > >> > > N + 2 (which means N + 2 would be done before N + 1) makes no sense. > > >> > > > > >> > > I think the author of the RFC meant that N + 2 or greater fails, but > > >> it > > >> > > was poorly worded. > > >> > > > > >> > > I will pass along whatever I get from nfsv4@ietf.org. (There is an > > >> archive > > >> > > of it somewhere, but I can't remember where.;-) > > >> > > > > >> > > rick > > >> > > _______________________________________________ > > >> > > freebsd-fs@freebsd.org mailing list > > >> > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > >> > > To unsubscribe, send any mail to > > >> > > "freebsd-fs-unsubscribe@freebsd.org" > > >> > > > > >> > > > >> > > > > > > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > -- > ------------------------------------------------------------------------- > Graham Allan - allan@physics.umn.edu - gta@umn.edu - (612) 624-5040 > School of Physics and Astronomy - University of Minnesota > ------------------------------------------------------------------------- > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >