From owner-freebsd-stable@FreeBSD.ORG Mon Feb 15 21:20:37 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA2891065695 for ; Mon, 15 Feb 2010 21:20:37 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) by mx1.freebsd.org (Postfix) with ESMTP id DCE278FC1B for ; Mon, 15 Feb 2010 21:20:36 +0000 (UTC) Received: from elsa.codelab.cz (localhost.codelab.cz [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 61A5019E019; Mon, 15 Feb 2010 22:20:35 +0100 (CET) Received: from [192.168.1.2] (r5bb235.net.upc.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 03D0A19E027; Mon, 15 Feb 2010 22:20:28 +0100 (CET) Message-ID: <4B79BA9C.3020402@quip.cz> Date: Mon, 15 Feb 2010 22:20:28 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1.7) Gecko/20100104 SeaMonkey/2.0.2 MIME-Version: 1.0 To: Alexander Leidinger References: <20100215090756.GA54764@icarus.home.lan> <20100215105000.101326yj01j0f64g@webmail.leidinger.net> <20100215122744.GA57382@icarus.home.lan> <20100215161105.14071eiflhc9le68@webmail.leidinger.net> In-Reply-To: <20100215161105.14071eiflhc9le68@webmail.leidinger.net> Content-Type: multipart/mixed; boundary="------------050007080204090204030503" Cc: freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: ZFS tuning [was: hardware for home use large storage] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Feb 2010 21:20:37 -0000 This is a multi-part message in MIME format. --------------050007080204090204030503 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Alexander Leidinger wrote: [...] >> kstat.zfs.misc.arcstats.c >> kstat.zfs.misc.arcstats.c_min >> kstat.zfs.misc.arcstats.c_max > > c_max is vfs.zfs.arc_max, c_min is vfs.zfs.arc_min. > >> kstat.zfs.misc.arcstats.evict_skip >> kstat.zfs.misc.arcstats.memory_throttle_count >> kstat.zfs.misc.arcstats.size > > I'm not very sure about size and c... both represent some kind of > current size, but they are not the same. > > > About the tuning I would recommend to depend upon a more human readable > representation. I've seen someone posting something like this, but I do > not know how it was generated (some kind of script, but I do not know > where to get it). I think you are referring to this script: http://cuddletech.com/arc_summary/ and its FreeBSD version http://bitbucket.org/koie/arc_summary/changeset/dbe14d2cf52b/ It gives output like this: # arc_summary.sh System Memory: Physical RAM: 4978 MB Free Memory : 755 MB ARC Size: Current Size: 1028 MB (arcsize) Target Size (Adaptive): 1028 MB (c) Min Size (Hard Limit): 50 MB (zfs_arc_min) Max Size (Hard Limit): 1205 MB (zfs_arc_max) ARC Size Breakdown: Most Recently Used Cache Size: 93% 963 MB (p) Most Frequently Used Cache Size: 6% 65 MB (c-p) ARC Efficency: Cache Access Total: 358720716 Cache Hit Ratio: 97% 350351031 [Defined State for buffer] Cache Miss Ratio: 2% 8369685 [Undefined State for Buffer] REAL Hit Ratio: 76% 272917080 [MRU/MFU Hits Only] Data Demand Efficiency: 96% Data Prefetch Efficiency: 27% CACHE HITS BY CACHE LIST: Anon: 22% 77179355 [ New Customer, First Cache Hit ] Most Recently Used: 45% 158252587 (mru) [ Return Customer ] Most Frequently Used: 32% 114664493 (mfu) [ Frequent Customer ] Most Recently Used Ghost: 0% 9777 (mru_ghost) [ Return Customer Evicted, Now Back ] Most Frequently Used Ghost: 0% 244819 (mfu_ghost) [ Frequent Customer Evicted, Now Back ] CACHE HITS BY DATA TYPE: Demand Data: 1% 4375918 Prefetch Data: 0% 150148 Demand Metadata: 76% 267830502 Prefetch Metadata: 22% 77994463 CACHE MISSES BY DATA TYPE: Demand Data: 1% 135956 Prefetch Data: 4% 400434 Demand Metadata: 73% 6177748 Prefetch Metadata: 19% 1655547 --------------------------------------------- Another useful script is arcstat.pl from http://blogs.sun.com/realneel/entry/zfs_arc_statistics There are FreeBSD version floating around but I can't find a link to it, so I am sending it as attachment. It would be nice to have some "standardized" scripts for monitoring & debugging ZFS issues attached to FreeBSD Wiki page about ZFS tuning. Then everebody will use the same scripts, same output format. It will be easier to compare results in discussions etc. So if anybody of you have write permissions to Wiki, can you add those two scripts? (or make some better ;]) Understanding to tuning of ZFS is really hard with lack of documentation ;( Miroslav Lachman --------------050007080204090204030503 Content-Type: text/plain; name="arcstat.freebsd.pl.txt" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="arcstat.freebsd.pl.txt" IyEvdXNyL2Jpbi9wZXJsIC13CiMKIyBQcmludCBvdXQgWkZTIEFSQyBTdGF0aXN0aWNzIGV4 cG9ydGVkIHZpYSBrc3RhdCgxKQojIEZvciBhIGRlZmluaXRpb24gb2YgZmllbGRzLCBvciB1 c2FnZSwgdXNlIGFyY3RzdGF0LnBsIC12CiMKIyBBdXRob3I6IE5lZWxha2FudGggTmFkZ2ly IGh0dHA6Ly9ibG9ncy5zdW4uY29tL3JlYWxuZWVsCiMgQ29tbWVudHMvUXVlc3Rpb25zL0Zl ZWRiYWNrIHRvIG5lZWxfc3VuLmNvbSBvciBuZWVsX2dudS5vcmcKIwojIENEREwgSEVBREVS IFNUQVJUCiMgCiMgVGhlIGNvbnRlbnRzIG9mIHRoaXMgZmlsZSBhcmUgc3ViamVjdCB0byB0 aGUgdGVybXMgb2YgdGhlCiMgQ29tbW9uIERldmVsb3BtZW50IGFuZCBEaXN0cmlidXRpb24g TGljZW5zZSwgVmVyc2lvbiAxLjAgb25seQojICh0aGUgIkxpY2Vuc2UiKS4gIFlvdSBtYXkg bm90IHVzZSB0aGlzIGZpbGUgZXhjZXB0IGluIGNvbXBsaWFuY2UKIyB3aXRoIHRoZSBMaWNl bnNlLgojIAojIFlvdSBjYW4gb2J0YWluIGEgY29weSBvZiB0aGUgbGljZW5zZSBhdCB1c3Iv c3JjL09QRU5TT0xBUklTLkxJQ0VOU0UKIyBvciBodHRwOi8vd3d3Lm9wZW5zb2xhcmlzLm9y Zy9vcy9saWNlbnNpbmcuCiMgU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFu Z3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zCiMgYW5kIGxpbWl0YXRpb25zIHVuZGVyIHRo ZSBMaWNlbnNlLgojIAojIFdoZW4gZGlzdHJpYnV0aW5nIENvdmVyZWQgQ29kZSwgaW5jbHVk ZSB0aGlzIENEREwgSEVBREVSIGluIGVhY2gKIyBmaWxlIGFuZCBpbmNsdWRlIHRoZSBMaWNl bnNlIGZpbGUgYXQgdXNyL3NyYy9PUEVOU09MQVJJUy5MSUNFTlNFLgojIElmIGFwcGxpY2Fi bGUsIGFkZCB0aGUgZm9sbG93aW5nIGJlbG93IHRoaXMgQ0RETCBIRUFERVIsIHdpdGggdGhl CiMgZmllbGRzIGVuY2xvc2VkIGJ5IGJyYWNrZXRzICJbXSIgcmVwbGFjZWQgd2l0aCB5b3Vy IG93biBpZGVudGlmeWluZwojIGluZm9ybWF0aW9uOiBQb3J0aW9ucyBDb3B5cmlnaHQgW3l5 eXldIFtuYW1lIG9mIGNvcHlyaWdodCBvd25lcl0KIyAKIyBDRERMIEhFQURFUiBFTkQKIwoj CiMgRmllbGRzIGhhdmUgYSBmaXhlZCB3aWR0aC4gRXZlcnkgaW50ZXJ2YWwsIHdlIGZpbGwg dGhlICJ2IgojIGhhc2ggd2l0aCBpdHMgY29ycmVzcG9uZGluZyB2YWx1ZSAodltmaWVsZF09 dmFsdWUpIHVzaW5nIGNhbGN1bGF0ZSgpLiAKIyBAaGRyIGlzIHRoZSBhcnJheSBvZiBmaWVs ZHMgdGhhdCBuZWVkcyB0byBiZSBwcmludGVkLCBzbyB3ZQojIGp1c3QgaXRlcmF0ZSBvdmVy IHRoaXMgYXJyYXkgYW5kIHByaW50IHRoZSB2YWx1ZXMgdXNpbmcgb3VyIHByZXR0eSBwcmlu dGVyLgoKdXNlIHN0cmljdDsKdXNlIFBPU0lYIHF3KHN0cmZ0aW1lKTsKI3VzZSBTdW46OlNv bGFyaXM6OktzdGF0Owp1c2UgR2V0b3B0OjpMb25nOwp1c2UgSU86OkhhbmRsZTsKCm15ICVj b2xzID0gKCMgSERSID0+IFtTaXplLCBEZXNjcmlwdGlvbl0KCSJUaW1lIgk9Pls4LCAiVGlt ZSJdLAoJImhpdHMiCT0+WzQsICJBcmMgcmVhZHMgcGVyIHNlY29uZCJdLAoJIm1pc3MiCT0+ WzQsICJBcmMgbWlzc2VzIHBlciBzZWNvbmQiXSwKCSJyZWFkIgk9Pls0LCAiVG90YWwgQXJj IGFjY2Vzc2VzIHBlciBzZWNvbmQiXSwKCSJIaXQlIgk9Pls0LCAiQXJjIEhpdCBwZXJjZW50 YWdlIl0sCgkibWlzcyUiCT0+WzUsICJBcmMgbWlzcyBwZXJjZW50YWdlIl0sCgkiZGhpdCIJ PT5bNCwgIkRlbWFuZCBEYXRhIGhpdHMgcGVyIHNlY29uZCJdLAoJImRtaXMiCT0+WzQsICJE ZW1hbmQgRGF0YSBtaXNzZXMgcGVyIHNlY29uZCJdLAoJImRoJSIJPT5bMywgIkRlbWFuZCBE YXRhIGhpdCBwZXJjZW50YWdlIl0sCgkiZG0lIgk9PlszLCAiRGVtYW5kIERhdGEgbWlzcyBw ZXJjZW50YWdlIl0sCgkicGhpdCIJPT5bNCwgIlByZWZldGNoIGhpdHMgcGVyIHNlY29uZCJd LAoJInBtaXMiCT0+WzQsICJQcmVmZXRjaCBtaXNzZXMgcGVyIHNlY29uZCJdLAoJInBoJSIJ PT5bMywgIlByZWZldGNoIGhpdHMgcGVyY2VudGFnZSJdLAoJInBtJSIJPT5bMywgIlByZWZl dGNoIG1pc3MgcGVyY2VudGFnZSJdLAoJIm1oaXQiCT0+WzQsICJNZXRhZGF0YSBoaXRzIHBl ciBzZWNvbmQiXSwKCSJtbWlzIgk9Pls0LCAiTWV0YWRhdGEgbWlzc2VzIHBlciBzZWNvbmQi XSwKCSJtcmVhZCIJPT5bNSwgIk1ldGFkYXRhIGFjY2Vzc2VzIHBlciBzZWNvbmQiXSwKCSJt aCUiCT0+WzMsICJNZXRhZGF0YSBoaXQgcGVyY2VudGFnZSJdLAoJIm1tJSIJPT5bMywgIk1l dGFkYXRhIG1pc3MgcGVyY2VudGFnZSJdLAoJImFyY3N6Igk9Pls1LCAiQXJjIFNpemUiXSwK CSJjIiAJPT5bNCwgIkFyYyBUYXJnZXQgU2l6ZSJdLAoJIm1mdSIgCT0+WzQsICJNRlUgTGlz dCBoaXRzIHBlciBzZWNvbmQiXSwKCSJtcnUiIAk9Pls0LCAiTVJVIExpc3QgaGl0cyBwZXIg c2Vjb25kIl0sCgkibWZ1ZyIgCT0+WzQsICJNRlUgR2hvc3QgTGlzdCBoaXRzIHBlciBzZWNv bmQiXSwKCSJtcnVnIiAJPT5bNCwgIk1SVSBHaG9zdCBMaXN0IGhpdHMgcGVyIHNlY29uZCJd LAoJImVza2lwIgk9Pls1LCAiZXZpY3Rfc2tpcCBwZXIgc2Vjb25kIl0sCgkibXR4bWlzIj0+ WzYsICJtdXRleF9taXNzIHBlciBzZWNvbmQiXSwKCSJybWlzIgk9Pls0LCAicmVjeWNsZV9t aXNzIHBlciBzZWNvbmQiXSwKCSJkcmVhZCIJPT5bNSwgIkRlbWFuZCBkYXRhIGFjY2Vzc2Vz IHBlciBzZWNvbmQiXSwKCSJwcmVhZCIJPT5bNSwgIlByZWZldGNoIGFjY2Vzc2VzIHBlciBz ZWNvbmQiXSwKKTsKbXkgJXY9KCk7Cm15IEBoZHIgPSBxdyhUaW1lIHJlYWQgbWlzcyBtaXNz JSBkbWlzIGRtJSBwbWlzIHBtJSBtbWlzIG1tJSBhcmNzeiBjKTsKbXkgQHhoZHIgPSBxdyhU aW1lIG1mdSBtcnUgbWZ1ZyBtcnVnIGVza2lwIG10eG1pcyBybWlzIGRyZWFkIHByZWFkIHJl YWQpOwpteSAkaW50ID0gMTsJCSMgUHJpbnQgc3RhdHMgZXZlcnkgMSBzZWNvbmQgYnkgZGVm YXVsdApteSAkY291bnQgPSAwOwkJIyBQcmludCBzdGF0cyBmb3JldmVyCm15ICRoZHJfaW50 ciA9IDIwOwkjIFByaW50IGhlYWRlciBldmVyeSAyMCBsaW5lcyBvZiBvdXRwdXQKbXkgJG9w ZmlsZSA9ICIiOwpteSAkc2VwID0gIiAgIjsJCSMgRGVmYXVsdCBzZXBlcmF0b3IgaXMgMiBz cGFjZXMKbXkgJHZlcnNpb24gPSAiMC4xIjsKbXkgJGNtZCA9ICJVc2FnZTogYXJjc3RhdC5w bCBbLWh2eF0gWy1mIGZpZWxkc10gWy1vIGZpbGVdIFtpbnRlcnZhbCBbY291bnRdXVxuIjsK bXkgJWN1cjsKbXkgJWQ7Cm15ICRvdXQ7Cm15ICRrc3RhdDsgIyA9IFN1bjo6U29sYXJpczo6 S3N0YXQtPm5ldygpOwpTVERPVVQtPmF1dG9mbHVzaDsKCnN1YiBrc3RhdF91cGRhdGUgewoJ bXkgQGsgPSBgc3lzY3RsICdrc3RhdC56ZnMubWlzYy5hcmNzdGF0cydgOwoKCXVuZGVmICRr c3RhdDsKCglmb3JlYWNoIG15ICRrIChAaykgewoJICBjaG9tcCAkazsKCSAgbXkgKCRuYW1l LCR2YWx1ZSkgPSBzcGxpdCAvOi8sICRrOwoJICBteSBAeiA9IHNwbGl0IC9cLi8sICRuYW1l OwoJICBteSAkbiA9IHBvcCBAejsKCSAgJHtrc3RhdH0tPnt6ZnN9LT57MH0tPnthcmNzdGF0 c30tPnskbn0gPSAkdmFsdWU7Cgl9Cn0KCnN1YiBkZXRhaWxlZF91c2FnZSB7CglwcmludCBT VERFUlIgIkFyY3N0YXQgdmVyc2lvbiAkdmVyc2lvblxuJGNtZCI7CglwcmludCBTVERFUlIg IkZpZWxkIGRlZmluaXRpb25zIGFyZSBhcyBmb2xsb3dzXG4iOwoJZm9yZWFjaCBteSAkaGRy IChrZXlzICVjb2xzKSB7CgkJcHJpbnQgU1RERVJSIHNwcmludGYoIiU2cyA6ICVzXG4iLCAk aGRyLCAkY29sc3skaGRyfVsxXSk7Cgl9CglwcmludCBTVERFUlIgIlxuTm90ZTogSz0xMF4z IE09MTBeNiBHPTEwXjkgYW5kIHNvIG9uXG4iOwoJZXhpdCgxKTsKCn0KCnN1YiB1c2FnZSB7 CglwcmludCBTVERFUlIgIkFyY3N0YXQgdmVyc2lvbiAkdmVyc2lvblxuJGNtZCI7Cglwcmlu dCBTVERFUlIgIlx0IC14IDogUHJpbnQgZXh0ZW5kZWQgc3RhdHNcbiI7CglwcmludCBTVERF UlIgIlx0IC1mIDogU3BlY2lmeSBzcGVjaWZpYyBmaWVsZHMgdG8gcHJpbnQgKHNlZSAtdilc biI7CglwcmludCBTVERFUlIgIlx0IC1vIDogUHJpbnQgc3RhdHMgdG8gZmlsZVxuIjsKCXBy aW50IFNUREVSUiAiXHQgLXMgOiBTcGVjaWZ5IGEgc2VwZXJhdG9yXG5cbkV4YW1wbGVzOlxu IjsKCXByaW50IFNUREVSUiAiXHRhcmNzdGF0IC1vIC90bXAvYS5sb2cgMiAxMFxuIjsKCXBy aW50IFNUREVSUiAiXHRhcmNzdGF0IC1zICwgLW8gL3RtcC9hLmxvZyAyIDEwXG4iOwoJcHJp bnQgU1RERVJSICJcdGFyY3N0YXQgLXZcbiI7CglwcmludCBTVERFUlIgIlx0YXJjc3RhdCAt ZiBUaW1lLEhpdCUsZGglLHBoJSxtaCVcbiI7CglleGl0KDEpOwp9CgpzdWIgaW5pdCB7Cglt eSAkZGVzaXJlZF9jb2xzOwoJbXkgJHhmbGFnID0gJyc7CglteSAkaGZsYWcgPSAnJzsKCW15 ICR2ZmxhZzsKCW15ICRyZXMgPSBHZXRPcHRpb25zKCd4JyA9PiBcJHhmbGFnLAoJCSdvPXMn ID0+IFwkb3BmaWxlLAoJCSdoZWxwfGh8PycgPT4gXCRoZmxhZywKCQkndicgPT4gXCR2Zmxh ZywKCQkncz1zJyA9PiBcJHNlcCwKCQknZj1zJyA9PiBcJGRlc2lyZWRfY29scyk7CgkkaW50 ID0gJEFSR1ZbMF0gfHwgJGludDsKCSRjb3VudCA9ICRBUkdWWzFdIHx8ICRjb3VudDsKCXVz YWdlKCkgaWYgISRyZXMgb3IgJGhmbGFnIG9yICgkeGZsYWcgYW5kICRkZXNpcmVkX2NvbHMp OwoJZGV0YWlsZWRfdXNhZ2UoKSBpZiAkdmZsYWc7CglAaGRyID0gQHhoZHIgaWYgJHhmbGFn OwkJI3Jlc2V0IGhlYWRlcnMgdG8geGhkcgoJaWYgKCRkZXNpcmVkX2NvbHMpIHsKCQlAaGRy ID0gc3BsaXQoL1sgLF0rLywgJGRlc2lyZWRfY29scyk7CgkJIyBOb3cgY2hlY2sgaWYgdGhl eSBhcmUgdmFsaWQgZmllbGRzCgkJbXkgQGludmFsaWQgPSAoKTsKCQlmb3JlYWNoIG15ICRl bGUgKEBoZHIpIHsKCQkJcHVzaChAaW52YWxpZCwgJGVsZSkgaWYgbm90IGV4aXN0cygkY29s c3skZWxlfSk7CgkJfQoJCWlmIChzY2FsYXIgQGludmFsaWQgPiAwKSB7CgkJCXByaW50IFNU REVSUiAiSW52YWxpZCBjb2x1bW4gZGVmaW5pdGlvbiEgLS0gIgoJCQkJLiAiQGludmFsaWRc blxuIjsKCQkJdXNhZ2UoKTsKCQl9Cgl9CglpZiAoJG9wZmlsZSkgewoJCW9wZW4oJG91dCwg Ij4kb3BmaWxlIikgfHxkaWUgIkNhbm5vdCBvcGVuICRvcGZpbGUgZm9yIHdyaXRpbmciOwoJ CSRvdXQtPmF1dG9mbHVzaDsKCQlzZWxlY3QgJG91dDsKCX0KfQoKIyBDYXB0dXJlIGtzdGF0 IHN0YXRpc3RpY3MuIFdlIG1haW50YWluIDMgaGFzaGVzLCBwcmV2LCBjdXIsIGFuZAojIGQg KGRlbHRhKS4gQXMgdGhlaXIgbmFtZXMgaW1wbHkgdGhleSBtYWludGFpbiB0aGUgcHJldmlv dXMsIGN1cnJlbnQsCiMgYW5kIGRlbHRhIChjdXIgLSBwcmV2KSBzdGF0aXN0aWNzLgpzdWIg c25hcF9zdGF0cyB7CglteSAlcHJldiA9ICVjdXI7Cglrc3RhdF91cGRhdGUoKTsKCglteSAk aGFzaHJlZl9jdXIgPSAka3N0YXQtPnsiemZzIn17MH17ImFyY3N0YXRzIn07CgklY3VyID0g JSRoYXNocmVmX2N1cjsKCWZvcmVhY2ggbXkgJGtleSAoa2V5cyAlY3VyKSB7CgkJbmV4dCBp ZiAka2V5ID1+IC9jbGFzcy87CgkJaWYgKGRlZmluZWQgJHByZXZ7JGtleX0pIHsKCQkJJGR7 JGtleX0gPSAkY3VyeyRrZXl9IC0gJHByZXZ7JGtleX07CgkJfSBlbHNlIHsKCQkJJGR7JGtl eX0gPSAkY3VyeyRrZXl9OwoJCX0KCX0KfQoKIyBQcmV0dHkgcHJpbnQgbnVtLiBBcmd1bWVu dHMgYXJlIHdpZHRoIGFuZCBudW0Kc3ViIHByZXR0eW51bSB7CglteSBAc3VmZml4PSgnICcs J0snLCAnTScsICdHJywgJ1QnLCAnUCcsICdFJywgJ1onKTsKCW15ICRudW0gPSAkX1sxXSB8 fCAwOwoJbXkgJHN6ID0gJF9bMF07CglteSAkaW5kZXggPSAwOwoJcmV0dXJuIHNwcmludGYo IiVzIiwgJG51bSkgaWYgbm90ICRudW0gPX4gL15bMC05XC5dKyQvOwoJd2hpbGUgKCRudW0g PiAxMDAwIGFuZCAkaW5kZXggPCA4KSB7CgkJJG51bSA9ICRudW0vMTAwMDsKCQkkaW5kZXgr KzsKCX0KCXJldHVybiBzcHJpbnRmKCIlKmQiLCAkc3osICRudW0pIGlmICgkaW5kZXggPT0g MCk7CglyZXR1cm4gc3ByaW50ZigiJSpkJXMiLCAkc3ogLSAxLCAkbnVtLCRzdWZmaXhbJGlu ZGV4XSk7Cn0KCnN1YiBwcmludF92YWx1ZXMgewoJZm9yZWFjaCBteSAkY29sIChAaGRyKSB7 CgkJcHJpbnRmKCIlcyVzIiwgcHJldHR5bnVtKCRjb2xzeyRjb2x9WzBdLCAkdnskY29sfSks ICRzZXApOwoJfQoJcHJpbnRmKCJcbiIpOwp9CgpzdWIgcHJpbnRfaGVhZGVyIHsKCWZvcmVh Y2ggbXkgJGNvbCAoQGhkcikgewoJCXByaW50ZigiJSpzJXMiLCAkY29sc3skY29sfVswXSwg JGNvbCwgJHNlcCk7Cgl9CglwcmludGYoIlxuIik7Cn0KCnN1YiBjYWxjdWxhdGUgewoJJXY9 KCk7CgkkdnsiVGltZSJ9ID0gc3RyZnRpbWUoIiVIOiVNOiVTIiwgbG9jYWx0aW1lKTsKCSR2 eyJoaXRzIn0gPSAkZHsiaGl0cyJ9LyRpbnQ7CgkkdnsibWlzcyJ9ID0gJGR7Im1pc3NlcyJ9 LyRpbnQ7CgkkdnsicmVhZCJ9ID0gJHZ7ImhpdHMifSArICR2eyJtaXNzIn07CgkkdnsiSGl0 JSJ9ID0gMTAwKiR2eyJoaXRzIn0vJHZ7InJlYWQifSBpZiAkdnsicmVhZCJ9ID4gMDsKCSR2 eyJtaXNzJSJ9ID0gMTAwIC0gJHZ7IkhpdCUifSBpZiAkdnsicmVhZCJ9ID4gMDsKCgkkdnsi ZGhpdCJ9ID0gKCRkeyJkZW1hbmRfZGF0YV9oaXRzIn0gKyAkZHsiZGVtYW5kX21ldGFkYXRh X2hpdHMifSkvJGludDsKCSR2eyJkbWlzIn0gPSAoJGR7ImRlbWFuZF9kYXRhX21pc3NlcyJ9 KyRkeyJkZW1hbmRfbWV0YWRhdGFfbWlzc2VzIn0pLyRpbnQ7CgkkdnsiZHJlYWQifSA9ICR2 eyJkaGl0In0gKyAkdnsiZG1pcyJ9OwoJJHZ7ImRoJSJ9ID0gMTAwKiR2eyJkaGl0In0vJHZ7 ImRyZWFkIn0gaWYgJHZ7ImRyZWFkIn0gPiAwOwoJJHZ7ImRtJSJ9ID0gMTAwIC0gJHZ7ImRo JSJ9IGlmICR2eyJkcmVhZCJ9ID4gMDsKCgkkdnsicGhpdCJ9PSgkZHsicHJlZmV0Y2hfZGF0 YV9oaXRzIn0gKyAkZHsicHJlZmV0Y2hfbWV0YWRhdGFfaGl0cyJ9KS8kaW50OwoJJHZ7InBt aXMifT0oJGR7InByZWZldGNoX2RhdGFfbWlzc2VzIn0KCQkrJGR7InByZWZldGNoX21ldGFk YXRhX21pc3NlcyJ9KS8kaW50OwoJJHZ7InByZWFkIn0gPSAkdnsicGhpdCJ9ICsgJHZ7InBt aXMifTsKCSR2eyJwaCUifSA9IDEwMCokdnsicGhpdCJ9LyR2eyJwcmVhZCJ9IGlmICR2eyJw cmVhZCJ9ID4gMDsKCSR2eyJwbSUifSA9IDEwMCAtICR2eyJwaCUifSBpZiAkdnsicHJlYWQi fSA+IDA7CgoJJHZ7Im1oaXQifT0oJGR7InByZWZldGNoX21ldGFkYXRhX2hpdHMifSskZHsi ZGVtYW5kX21ldGFkYXRhX2hpdHMifSkvJGludDsKCSR2eyJtbWlzIn09KCRkeyJwcmVmZXRj aF9tZXRhZGF0YV9taXNzZXMifQoJCSskZHsiZGVtYW5kX21ldGFkYXRhX21pc3NlcyJ9KS8k aW50OwoJJHZ7Im1yZWFkIn0gPSAkdnsibWhpdCJ9ICsgJHZ7Im1taXMifTsKCSR2eyJtaCUi fSA9IDEwMCokdnsibWhpdCJ9LyR2eyJtcmVhZCJ9IGlmICR2eyJtcmVhZCJ9ID4gMDsKCSR2 eyJtbSUifSA9IDEwMCAtICR2eyJtaCUifSBpZiAkdnsibXJlYWQifSA+IDA7CgoJJHZ7ImFy Y3N6In0gPSAkY3VyeyJzaXplIn07CgkkdnsiYyJ9ID0gJGN1cnsiYyJ9OwoJJHZ7Im1mdSJ9 ID0gJGR7ImhpdHMifS8kaW50OwoJJHZ7Im1ydSJ9ID0gJGR7Im1ydV9oaXRzIn0vJGludDsK CSR2eyJtcnVnIn0gPSAkZHsibXJ1X2dob3N0X2hpdHMifS8kaW50OwoJJHZ7Im1mdWcifSA9 ICRkeyJtcnVfZ2hvc3RfaGl0cyJ9LyRpbnQ7CgkkdnsiZXNraXAifSA9ICRkeyJldmljdF9z a2lwIn0vJGludDsKCSR2eyJybWlzcyJ9ID0gJGR7InJlY3ljbGVfbWlzcyJ9LyRpbnQ7Cgkk dnsibXR4bWlzIn0gPSAkZHsibXV0ZXhfbWlzcyJ9LyRpbnQ7Cn0KCnN1YiBtYWluIHsKCW15 ICRpID0gMDsKCW15ICRjb3VudF9mbGFnID0gMDsKCglpbml0KCk7CglpZiAoJGNvdW50ID4g MCkgeyAkY291bnRfZmxhZyA9IDE7IH0KCXdoaWxlICgxKSB7CgkJcHJpbnRfaGVhZGVyKCkg aWYgKCRpID09IDApOwoJCXNuYXBfc3RhdHMoKTsKCQljYWxjdWxhdGUoKTsKCQlwcmludF92 YWx1ZXMoKTsKCQlsYXN0IGlmICgkY291bnRfZmxhZyA9PSAxICYmICRjb3VudC0tIDw9IDEp OwoJCSRpID0gKCRpID09ICRoZHJfaW50cikgPyAwIDogJGkrMTsKCQlzbGVlcCgkaW50KTsK CX0KCWNsb3NlKCRvdXQpIGlmIGRlZmluZWQgJG91dDsKfQoKJm1haW47Cg== --------------050007080204090204030503--