From owner-freebsd-scsi@freebsd.org Mon Aug 28 02:50:57 2017 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C160E03F7E for ; Mon, 28 Aug 2017 02:50:57 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: from mail-oi0-x22f.google.com (mail-oi0-x22f.google.com [IPv6:2607:f8b0:4003:c06::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F40DC65A93 for ; Mon, 28 Aug 2017 02:50:56 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: by mail-oi0-x22f.google.com with SMTP id k77so35539437oib.2 for ; Sun, 27 Aug 2017 19:50:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kev009.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=+RYuOkhkONu96Gb1zo0sDSvBRXP+l//xCop6qwpkTAA=; b=XiGKiWDb0HemgUUyYAVyn6jhIqzqI0fJSSSFpOInAr/CePSk9cj3UGHFRIWexyFau8 Lf65BQaHGxiumGbY4gO/A7LYv76CbIz/bFAcdvRV/UWjJCs+axGe5yuXSJSLdG5HVmkq u6Fo2iAbDxoTHTGr69H79JW4NRYvUi8XH99k4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=+RYuOkhkONu96Gb1zo0sDSvBRXP+l//xCop6qwpkTAA=; b=F7wF6P/bCYywm3j9rBFgvRufB7JJBLvKQlxGMboeeYqV6Au5nD1OgvQuFZkXR95zFF ZkdZFgYzEVtJ23bPGF9JJXMxeDdzvRH8ouc1Bi5bxS0VsAvCxBkXmERt5i2ZKmd1b/cF e4akQykSjbjVyjbu7OddZZG/ruQMuuXYVyTkZ9aZaXRpIDIuqv2y9fqLLzDLfa1SNZeq cBOYBwifBv+ryWoHB75R6o+3aDTva3xKOqRvE+rpChAegDppUDiNCiGnt1ARZMxp1IOd heR+DyyvrDIbo93S/30xDf64ehaxLtYDvwk/Fhc66ZTW7qMqAmKGiqMUNBWhW3lbO5WF 8PHw== X-Gm-Message-State: AHYfb5iEhWbt1IA8n7u4XWwF7D7OE0zZHIH1Z9m8PK7g/A51TLOa7Lmr DnXBPt2praPXa2R82S94diye7WU4uEeD X-Received: by 10.202.218.2 with SMTP id r2mr8238825oig.21.1503888655868; Sun, 27 Aug 2017 19:50:55 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.49.5 with HTTP; Sun, 27 Aug 2017 19:50:55 -0700 (PDT) In-Reply-To: <1112cc2edb666ce7fa9c72c11cdd284c@nitrology.com> References: <1112cc2edb666ce7fa9c72c11cdd284c@nitrology.com> From: Kevin Bowling Date: Sun, 27 Aug 2017 19:50:55 -0700 Message-ID: Subject: Re: Disk reordering on LSI SAS2008/mps(4) To: FreeBSD-scsi Cc: John Baldwin , Jason Wolfe Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Aug 2017 02:50:57 -0000 Note that we only see this bug with EARLY_AP_STARTUP enabled Regards, On Fri, Aug 25, 2017 at 2:11 PM, Jason Wolfe wrote: > Attachments are useful. > > > On 2017-08-25 13:58, Jason Wolfe wrote: > >> Hi! >> >> We've been having an issue where we see some disk reordering on boot >> on HEAD from mid July on LSI controllers, maybe 5% of the time. We >> brought mps current as of r322364 with no change behavior. >> >> I have a few logs attached with various debug output. In all cases >> I've seen the pass ordering to be proper, and cam does try to resolve >> the da ordering, but the device it tries to reassign to is already >> taken. Attached is the full output, and listing some relevant bits >> below for the casual reader. Being that the functionality in scsi_da.c >> has been fairly static, and it's attempting to reassign, it seems more >> likely we are running into something in mps here. The targets always >> look to be proper. >> >> The various settings of hw.mps.use_phy_num (-1/0/1) don't change the >> behavior, and neither does hw.mps.enable_ssu=0. We have machines over >> various FW versions (15/16) that see the issue. I'm wondering if the >> fact that we see this issue over soft reboots means that the firmware >> isn't coming into play. To confirm, we are booting from the >> controller, so the LSI BIOS is enabled. >> >> mps0@pci0:3:0:0: class=0x010700 card=0x040015d9 chip=0x00721000 >> rev=0x03 hdr=0x00 >> vendor = 'LSI Logic / Symbios Logic' >> device = 'SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]' >> class = mass storage >> subclass = SAS >> >> reorder-verbose.txt: >> boot_verbose="YES" >> hw.mps.debug_level="71" >> >> da0 at mps0 bus 0 scbus0 target 17 lun 0 >> cam_periph_alloc: attempt to re-allocate valid device da0 rejected >> flags 0x102 refcount 4 >> da1 at mps0 bus 0 scbus0 target 8 lun 0 >> daasync: Unable to attach to new device due to status 0x6 >> da2 at mps0 bus 0 scbus0 target 9 lun 0 >> ... >> da8 at mps0 bus 0 scbus0 target 15 lun 0 >> da9 at mps0 bus 0 scbus0 target 16 lun 0 >> da10 at mps0 bus 0 scbus0 target 18 lun 0 >> da11 at mps0 bus 0 scbus0 target 19 lun 0 >> >> pass0 at mps0 bus 0 scbus0 target 8 lun 0 >> pass1 at mps0 bus 0 scbus0 target 9 lun 0 >> ... >> pass9 at mps0 bus 0 scbus0 target 17 lun 0 >> pass10 at mps0 bus 0 scbus0 target 18 lun 0 >> pass11 at mps0 bus 0 scbus0 target 19 lun 0 >> >> >> >> >> reorder-mps-mapping.txt: >> hw.mps.debug_level="583" >> >> da0 at mps0 bus 0 scbus0 target 19 lun 0 >> da1 at mps0 bus 0 scbus0 target 8 lun 0 >> da2 at mps0 bus 0 scbus0 target 9 lun 0 >> ... >> da9 at mps0 bus 0 scbus0 target 16 lun 0 >> da10 at mps0 bus 0 scbus0 target 17 lun 0 >> da11 at mps0 bus 0 scbus0 target 18 lun 0 >> cam_periph_alloc: attempt to re-allocate valid device da0 rejected >> flags 0x106 refcount 6 >> daasync: Unable to attach to new device due to status 0x6 >> >> ses0: da1,pass0: Element descriptor: 'Slot 01' >> ses0: da1,pass0: SAS Device Slot Element: 1 Phys at Slot 0 >> ses0: da0,pass11: Element descriptor: 'Slot 12' >> ses0: da0,pass11: SAS Device Slot Element: 1 Phys at Slot 11 >> >> >> Luckily we have found a way to fairly easily repro it over a few >> hours, so we are open to any suggestions. >> >> Thanks! >> Jason > > > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" >