Date: Wed, 10 Sep 2008 07:51:59 -0400 From: "Dennis R. Kolpanen" <kolpanen@kearfott.com> To: Danny Braniss <danny@cs.huji.ac.il> Cc: freebsd-scsi@freebsd.org Subject: Re: iscsi initiator - system hang Message-ID: <20080910115159.GA28900@qads.kearfott.com> In-Reply-To: <E1KdJFB-000GTJ-98@cs1.cs.huji.ac.il> References: <20080909211532.GA16009@qads.kearfott.com> <E1KdJFB-000GTJ-98@cs1.cs.huji.ac.il>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Sep 10, 2008 at 09:32:01AM +0300, Danny Braniss wrote: > > On a FreeBSD 7.0 system, certain commands issued against any of the > > three mounted iscsi drives causes the system to "hang". In this > > context, hang means: > > > > the system continues to respond to pings > > sendmail stops accepting connections > > imapd stops accepting connections > > sshd stops accepting connections > > shell sessions already established stop accepting commands > > login at the system console is not possible > > > > The problem has been caused, so far, by dump, restore, and pax. These > > commands work perfectly if they are directed against one of the > > internal drives and not the iscsi drives. The failures noted above do > > not normally happen immediately after issuing one of these commands. > > The problems seem to build over a period of minutes or tens of > > minutes. Note that the dump/restore/pax commands can take hours to > > run. > > > > Nothing is written to the system console or any of the log files > > indicating a problem. The only way to recover from the hang is by > > means of the hardware reset button. > > > > When the system was first being set up and no other users were on it, > > shutting down all but one of the CPUs by means of sysctl and > > "machdep.hlt_cpus" allowed restoring about 150gb to the three iscsi > > drives. Once the machine was placed into production, massive hardware > > problems on an old server required this to be done immediately, this > > trick no longer works. > > > > An overview of the hardware involved: > > > > dual, quad-core Intel Xeon processors > > 16 gb RAM FreeBSD 7.0 amd64 release > > NetworkAppliance FAS2020 SAN > > generic kernel > > > > A complete dmesg output can be provided if desired. > > > > By default, iscontrol creates the iscsi drives with the number of tags > > set to one. The performance of the iscsi drives with this default > > setting was quite poor. Based on a recommendation made on a mailing > > list some time ago, /etc/iscsi.conf was changed to set the tags to > > 128. This had a dramatic improvement on the iscsi performance. > > > > Testing on the system that was rushed into production is not really > > possible. However, within the next week or so, a nearly identical > > system should become available and this one could be used for testing. > > > > Any ideas on what could be wrong? Any solutions? > > > > maybe, but first, can you send me the output of 'sysctl net.iscsi'? > > danny > Danny, No problem. Here it is: net.iscsi.driver_version: 2.0.99 net.iscsi.isid: DIB00 net.iscsi.sessions: 1 net.iscsi.0.targetname: iqn.1992-08.com.netapp:sn.135029528 net.iscsi.0.targeaddress: 10.172.172.1 net.iscsi.0.stats: recv=1288005 sent=1287999 flags=0x0000011b pdus-alloc=0 pdus-max=220 cws=64 cmd=afa32 exp=afa32 max=afa71 stat=afa32 itt=afa32 net.iscsi.0.pid: 916 Thanks. Dennis
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080910115159.GA28900>