From owner-freebsd-current@FreeBSD.ORG Wed Oct 16 14:55:11 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id AD4E0F81; Wed, 16 Oct 2013 14:55:11 +0000 (UTC) (envelope-from prvs=100105a393=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 224F724B5; Wed, 16 Oct 2013 14:55:10 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50006421385.msg; Wed, 16 Oct 2013 15:55:08 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 16 Oct 2013 15:55:08 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=100105a393=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <6DE320B20F7844B9ADC34A214AED8055@multiplay.co.uk> From: "Steven Hartland" To: "Vitalij Satanivskij" References: <74C1D072-77BB-4D6C-B78F-C8D2731FA0CF@sarenet.es> <1379333192.127359970.ma5jnbc5@fmst-6.ukr.net> <1379334340.567465877.0b1lli6r@fmst-6.ukr.net> <8365CE736DC749DF95D0030A725211F6@multiplay.co.uk> <02549AD9-C456-4E17-927C-B4BCC97F8CC8@freebsd.org> <4AA28730F331444AB13108ABF0CD68B7@multiplay.co.uk> <1379496242.750778745.m0ksff1m@fmst-6.ukr.net> <20131016080100.GA27758@hell.ukr.net> <0023A5B16E614B67A2D8A4A63641D1D8@multiplay.co.uk> <20131016141053.GA43384@hell.ukr.net> Subject: Re: ZFS secondarycache on SSD problem on r255173 Date: Wed, 16 Oct 2013 15:55:26 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="windows-1251"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: Vitalij Satanivskij , "Justin T. Gibbs" , freebsd-current@freebsd.org, Borja Marcos , Dmitriy Makarov X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 14:55:11 -0000 I'm not clear what you rolled back there as r255173 has ntothing to do with this. Could you clarify Any errors recorded in /var/log/messages? Could you add code to record the none zero value of zio->io_error in l2arc_read_done as this may give some indication of the underlying issue. Additionally could always put a panic in that code path too and then create a dump so the details can be fully exhamined. In terms of the slowness thats going to be a side effect of the cache failures. Oh could you also confirm that the issue doesn't exist if you 1. Exclude r255753 2. Set vfs.zfs.max_auto_ashift=9 Regards Steve ----- Original Message ----- From: "Vitalij Satanivskij" To: "Steven Hartland" Cc: "Vitalij Satanivskij" ; "Dmitriy Makarov" ; "Justin T. Gibbs" ; "Borja Marcos" ; Sent: Wednesday, October 16, 2013 3:10 PM Subject: Re: ZFS secondarycache on SSD problem on r255173 > Yes > > We have 15 servers, all of them have problem while using with patch fo ashift, sh we rollback path (for r255173) > and all of them works for a week without that's problem's. Yesterday one of of servers was updated to stable/10 (beta1) > > wich include patch and after around 12 hours of works l2arc begin et errors like that > > kstat.zfs.misc.arcstats.l2_cksum_bad > kstat.zfs.misc.arcstats.l2_io_error > > > For now patch disabled in ower production. > > > Please note we have very heavy load on zfs pool so 90GB arc and 3x180Gb L2arc have very big hit's on it on it. > > > SSD used for cache's is intel ssd 530 series smart for all devices in in normal states's > no bad values on it. > > Steven Hartland wrote: > SH> Have you confirmed the ashift changes are the actual cause of this > SH> by backing out just those changes and retesting on the same hardware. > SH> > SH> Also worth checking your disks smart values to confirm there are no > SH> visible signs of HW errors. > SH> > SH> Regards > SH> Steve > SH> > SH> ----- Original Message ----- > SH> From: "Vitalij Satanivskij" > SH> To: "Dmitriy Makarov" > SH> Cc: "Steven Hartland" ; "Justin T. Gibbs" ; "Borja Marcos" ; > SH> > SH> Sent: Wednesday, October 16, 2013 9:01 AM > SH> Subject: Re: ZFS secondarycache on SSD problem on r255173 > SH> > SH> > SH> > Hello. > SH> > > SH> > Patch brocke cache functionality. > SH> > > SH> > Look at's Dmitriy's mail from Mon, 07 Oct 2013 21:09:06 +0300 > SH> > > SH> > With subject ZFS L2ARC - incorrect size and abnormal system load on r255173 > SH> > > SH> > As patch alredy in head and BETA it's not good. > SH> > > SH> > Yesterday we update one machine up to beta1 and forgot about patch. So 12 Hours and cache broken... :(( > SH> > > SH> > > SH> > > SH> > Dmitriy Makarov wrote: > SH> > DM> The attached patch by Steven Hartland fixes issue for me too. Thank you! > SH> > DM> > SH> > DM> > SH> > DM> --- Исходное сообщение --- > SH> > DM> От кого: "Steven Hartland" < killing@multiplay.co.uk > > SH> > DM> Дата: 18 сентября 2013, 01:53:10 > SH> > DM> > SH> > DM> ----- Original Message ----- > SH> > DM> From: "Justin T. Gibbs" < > SH> > DM> > SH> > DM> --- > SH> > DM> Дмитрий Макаров > SH> > DM> _______________________________________________ > SH> > DM> freebsd-current@freebsd.org mailing list > SH> > DM> http://lists.freebsd.org/mailman/listinfo/freebsd-current > SH> > DM> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > SH> > > SH> > SH> > SH> ================================================ > SH> This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the > event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any > information contained in it. > SH> > SH> In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 > SH> or return the E.mail to postmaster@multiplay.co.uk. > SH> > SH> _______________________________________________ > SH> freebsd-current@freebsd.org mailing list > SH> http://lists.freebsd.org/mailman/listinfo/freebsd-current > SH> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.