From owner-freebsd-scsi@FreeBSD.ORG Mon Jan 25 11:07:09 2010 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 926D11065697 for ; Mon, 25 Jan 2010 11:07:09 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 75D328FC1A for ; Mon, 25 Jan 2010 11:07:09 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o0PB79RV038888 for ; Mon, 25 Jan 2010 11:07:09 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o0PB78Kh038886 for freebsd-scsi@FreeBSD.org; Mon, 25 Jan 2010 11:07:08 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 25 Jan 2010 11:07:08 GMT Message-Id: <201001251107.o0PB78Kh038886@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2010 11:07:09 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/141934 scsi [cam] [patch] add support for SEAGATE DAT Scopion 130 o kern/140091 scsi [da] [patch] allow for da(4) large block transfer than o kern/138789 scsi [cam] [patch] cd(4) patch for drives/discs failing the o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o amd64/132394 scsi [isp] - bad underruns with QLogic qla2300 and amd64 o kern/132250 scsi [ciss] ciss driver does not support more then 15 drive o kern/132206 scsi [mpt] system panics on boot when mirroring and 2nd dri p kern/130735 scsi [cam] [patch] pass M_NOWAIT to the malloc() call insid o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 o kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus o kern/128452 scsi [sa] [panic] Accessing SCSI tape drive randomly crashe o kern/128245 scsi [scsi] "inquiry data fails comparison at DV1 step" [re o kern/127927 scsi [isp] isp(4) target driver crashes kernel when set up o kern/124667 scsi [amd] [panic] FreeBSD-7 kernel page faults at amd-scsi o kern/123674 scsi [ahc] ahc driver dumping f kern/123666 scsi [aac] attach fails with Adaptec SAS RAID 3805 controll o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc o kern/120487 scsi [sg] scsi_sg incompatible with scanners o kern/120247 scsi [mpt] FreeBSD 6.3 and LSI Logic 1030 = only 3.300MB/s o kern/119668 scsi [cam] [patch] certain errors are too verbose comparing o kern/114597 scsi [sym] System hangs at SCSI bus reset with dual HBAs o kern/110847 scsi [ahd] Tyan U320 onboard problem with more than 3 disks o kern/99954 scsi [ahc] reading from DVD failes on 6.x [regression] o kern/94838 scsi Kernel panic while mounting SD card with lock switch o o kern/92798 scsi [ahc] SCSI problem with timeouts o kern/90282 scsi [sym] SCSI bus resets cause loss of ch device o kern/76178 scsi [ahd] Problem with ahd and large SCSI Raid system o kern/74627 scsi [ahc] [hang] Adaptec 2940U2W Can't boot 5.3 s kern/61165 scsi [panic] kernel page fault after calling cam_send_ccb o kern/60641 scsi [sym] Sporadic SCSI bus resets with 53C810 under load o kern/60598 scsi wire down of scsi devices conflicts with config s kern/57398 scsi [mly] Current fails to install on mly(4) based RAID di o kern/52638 scsi [panic] SCSI U320 on SMP server won't run faster than o kern/44587 scsi dev/dpt/dpt.h is missing defines required for DPT_HAND o kern/40895 scsi wierd kernel / device driver bug o kern/39388 scsi ncr/sym drivers fail with 53c810 and more than 256MB m o kern/38828 scsi [dpt] [request] DPT PM2012B/90 doesn't work o kern/35234 scsi World access to /dev/pass? (for scanner) requires acce 39 problems total. From owner-freebsd-scsi@FreeBSD.ORG Tue Jan 26 16:16:33 2010 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B2C710656A3 for ; Tue, 26 Jan 2010 16:16:33 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) by mx1.freebsd.org (Postfix) with ESMTP id 014258FC26 for ; Tue, 26 Jan 2010 16:16:32 +0000 (UTC) Received: from elsa.codelab.cz (localhost.codelab.cz [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 98DCA19E019; Tue, 26 Jan 2010 17:16:31 +0100 (CET) Received: from [192.168.1.2] (r5bb235.net.upc.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 0E45619E046; Tue, 26 Jan 2010 17:16:29 +0100 (CET) Message-ID: <4B5F1553.4070709@quip.cz> Date: Tue, 26 Jan 2010 17:16:19 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1.7) Gecko/20100104 SeaMonkey/2.0.2 MIME-Version: 1.0 To: freebsd-scsi@freebsd.org References: 4B57263E.7040809@quip.cz Content-Type: text/plain; charset=ISO-8859-2; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: performance tuning of iSCSI and Dell MD3000i / gjournal problem X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jan 2010 16:16:33 -0000 I am CCing freebsd-performance@freebsd.org as somebody may be interested and not subscribed to freebsd-scsi. Just a note, I am still having performance problems with iSCSI and Dell MD3000i. I tried it with ZFS, but writing performance was even worse - about 3-5MB/s! Copying 33GB from local UFS partition do iSCSI ZFS partition takes almost 5 hours: ~/# cp -a /vol1/data /tank/vol2/data Terminated Usr: 0.224s Krnl: 131.862s Totl: 4:47:21.80s CPU: 0.7% swppd: 0 I/O: 271251+0 ~/# df -h /tank/vol2 Filesystem Size Used Avail Capacity Mounted on tank/vol2 49G 33G 15G 68% /tank/vol2 But ZFS read performance was impressive (thanks to prefetch enabled), about 88MB/s (up to 850Mbit/s on iSCSI interface bce1) # ifstat -i bce1 -b bce1 Kbps in Kbps out 0.00 0.00 726154.2 5020.94 721338.3 4000.65 674180.6 3658.37 693954.6 4227.52 679745.0 3619.76 693572.9 4170.39 544614.8 2812.59 579602.1 3021.56 752622.0 3921.31 854647.0 4654.90 da1 is ZFS partition over iSCSI # iostat -x -w 20 da0 da1 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 0.0 0.0 0.0 0.0 0 0.0 0 da1 1529.1 7.7 97860.8 56.7 9 16.4 99 Unfortunately there are some other problem with ZFS performance and Lighttpd. http://lists.freebsd.org/pipermail/freebsd-stable/2010-January/054384.html http://lists.freebsd.org/pipermail/freebsd-stable/2010-January/054385.html Then I reverted back to UFS + gjournal, but as I post in to freebsd-geom@ list, there is problem with gjournal timeout at boot time. So UFS and gjournal seems to be usable only on first / manual setup, but not in usual way with rc.d scripts. http://lists.freebsd.org/pipermail/freebsd-geom/2010-January/003872.html Then I choose to drop gjournal from my plan (but I am scared of really long fsck after crash / power failure). I have some problems with combination of GPT + gjournal + glabel so I destroyed the filesystem and start recreating it again and restore data from backup. *there were several system freezes caused by unkillable iscsi sessions in interaction with gjournal* The machine had to be rebooted from remote management (cold or warm reboot in Dell DRAC). But no kernel panic message. I was disappointed by terrible write performance again. UFS + SoftUpdates did only 4-7MB/s of write speed by rsync. # iostat -x -w 10 da0 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 0.4 107.4 6.4 6791.5 17 131.0 104 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 0.0 67.0 0.0 4285.8 18 215.6 100 # ifstat -i bce0,bce1 -b 10 bce0 bce1 Kbps in Kbps out Kbps in Kbps out 31542.66 1518.12 285.53 31415.25 34309.73 1652.56 306.90 34281.15 47235.61 2270.39 429.60 47887.96 30743.04 1483.37 267.57 30524.36 50871.20 2458.34 457.92 50449.77 63548.43 3023.20 581.71 64230.07 # iostat -w 5 tty mfid0 da0 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 1 319 15.47 2 0.04 62.90 71 4.35 1 0 0 0 98 0 100 0.00 0 0.00 60.22 62 3.66 1 0 0 0 99 0 63 16.00 1 0.02 63.86 66 4.15 1 0 0 0 98 0 39 22.65 35 0.78 57.97 73 4.13 2 0 1 0 96 0 63 19.18 21 0.40 63.90 92 5.74 1 0 0 0 98 It means 2-3 days to copy all data back to the storage! I stopped it, ad gjournal again and start rsync again. Much better results (28MB/s of write speed) but not as high as was mont ago when I started to play with iSCSI (it was about 60MB/s) This time it was: extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 0.7 447.9 11.1 28486.3 32 68.8 100 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 0.9 444.6 14.3 28367.0 23 69.4 99 # ifstat -i bce0,bce1 -b 10 bce0 bce1 Kbps in Kbps out Kbps in Kbps out 213487.4 8139.54 2201.14 237459.0 268473.5 10233.86 2354.66 250007.1 222936.1 8499.59 2179.59 235984.8 259017.9 9835.16 2094.63 221011.9 215187.9 8188.47 2205.16 237813.7 175749.2 6637.50 1888.68 207940.3 291358.4 11056.45 2353.09 251959.6 journal is on the local RAID1 (mfid0) with writing speed about 150MB/s so the data is fetched over network to the journal and then committed to iSCSI storage (da0) with higher speed than in case of SoftUpdates. But writing speed dropped after 5 hours of running rsync from 28MB/s to 8MB/s (and is still running) Now I am too tired and need some more hours of sleep... Miroslav Lachman PS: Thank you for your work Danny, I know it is not possible to you to help me if you do not have exactly the same HW setup. I will post you (and to freebsd-rc@) highly modified (and I hope improved) rc.d script for iSCSI with better handling of loading modules, mounting, stopping etc. From owner-freebsd-scsi@FreeBSD.ORG Wed Jan 27 12:49:50 2010 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40D71106568F for ; Wed, 27 Jan 2010 12:49:50 +0000 (UTC) (envelope-from freebsd-scsi@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id F06E68FC0A for ; Wed, 27 Jan 2010 12:49:48 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.50) id 1Na7L7-0007Mr-AY for freebsd-scsi@freebsd.org; Wed, 27 Jan 2010 13:49:45 +0100 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 27 Jan 2010 13:49:45 +0100 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 27 Jan 2010 13:49:45 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-scsi@freebsd.org From: Ivan Voras Date: Wed, 27 Jan 2010 13:49:33 +0100 Lines: 13 Message-ID: References: <4B5F1553.4070709@quip.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.5) Gecko/20100118 Thunderbird/3.0 In-Reply-To: <4B5F1553.4070709@quip.cz> Sender: news Cc: freebsd-performance@freebsd.org Subject: Re: performance tuning of iSCSI and Dell MD3000i / gjournal problem X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2010 12:49:50 -0000 On 01/26/10 17:16, Miroslav Lachman wrote: > I am CCing freebsd-performance@freebsd.org as somebody may be interested > and not subscribed to freebsd-scsi. > > Just a note, I am still having performance problems with iSCSI and Dell > MD3000i. > I tried it with ZFS, but writing performance was even worse - about > 3-5MB/s! Copying 33GB from local UFS partition do iSCSI ZFS partition > takes almost 5 hours: You've written a lot of information here. Firstly, is your problem related to iSCSI performance, ZFS performance or UFS performance? As it looks to me, you are mostly complaining about iSCSI performance, right?