From owner-freebsd-amd64@FreeBSD.ORG Tue Jan 9 14:30:55 2007 Return-Path: X-Original-To: freebsd-amd64@FreeBSD.org Delivered-To: freebsd-amd64@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E3F7E16A4AB; Tue, 9 Jan 2007 14:30:55 +0000 (UTC) (envelope-from sven@dmv.com) Received: from smtp-gw-cl-c.dmv.com (smtp-gw-cl-c.dmv.com [216.240.97.41]) by mx1.freebsd.org (Postfix) with ESMTP id B545213C469; Tue, 9 Jan 2007 14:30:55 +0000 (UTC) (envelope-from sven@dmv.com) Received: from mail-gw-cl-a.dmv.com (mail-gw-cl-a.dmv.com [216.240.97.38]) by smtp-gw-cl-c.dmv.com (8.12.10/8.12.10) with ESMTP id l09EUs0F006724; Tue, 9 Jan 2007 09:30:54 -0500 (EST) (envelope-from sven@dmv.com) Received: from lanshark.dmv.com (lanshark.dmv.com [216.240.97.46]) by mail-gw-cl-a.dmv.com (8.12.9/8.12.9) with ESMTP id l09EUrXi052697; Tue, 9 Jan 2007 09:30:54 -0500 (EST) (envelope-from sven@dmv.com) From: Sven Willenberger To: Bruce Evans In-Reply-To: <20070109124826.M79616@delplex.bde.org> References: <1168211205.22629.6.camel@lanshark.dmv.com> <20070108154433.C75042@delplex.bde.org> <1168271935.23549.10.camel@lanshark.dmv.com> <20070109124826.M79616@delplex.bde.org> Content-Type: text/plain Date: Tue, 09 Jan 2007 09:37:05 -0500 Message-Id: <1168353425.29047.8.camel@lanshark.dmv.com> Mime-Version: 1.0 X-Mailer: Evolution 2.6.3 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.39 X-Scanned-By: MIMEDefang 2.48 on 216.240.97.38 Cc: stable@FreeBSD.org, freebsd-amd64@FreeBSD.org Subject: Re: Panic in 6.2-PRERELEASE with bge on amd64 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Jan 2007 14:30:56 -0000 On Tue, 2007-01-09 at 12:50 +1100, Bruce Evans wrote: > On Mon, 8 Jan 2007, Sven Willenberger wrote: > > > On Mon, 2007-01-08 at 16:06 +1100, Bruce Evans wrote: > >> On Sun, 7 Jan 2007, Sven Willenberger wrote: > > >>> The short and dirty of the dump: > >>> ... > >>> --- trap 0xc, rip = 0xffffffff801d5f17, rsp = 0xffffffffb371ab50, rbp = 0xffffffffb371aba0 --- > >>> bge_rxeof() at bge_rxeof+0x3b7 > >> > >> What is the instruction here? > > > > I will do my best to ferret out the information you need. For the > > bge_rxeof() at bge_rxeof+0x3b7 line, the instruction is: > > > > 0xffffffff801d5f17 : mov %r15,0x28(%r14) > > ... > >> Looks like a null pointer panic anyway. I guess the instruction is > >> movl to/from 0x28(%reg) where %reg is a null pointer. > >> > > > > from the above lines, apparently %r14 is null then. > > Yes. It's a bit suprising that the access is a write. > > >>> ... > >>> #8 0xffffffff801db818 in bge_intr (xsc=0x0) at /usr/src/sys/dev/bge/if_bge.c:2707 > >> > >> What is the statement here? It presumably follow a null pointer and only > >> the exprssion for the pointer is interesting. xsc is already null but > >> that is probably a bug in gdb, or the result of excessive optimization. > >> Compiling kernels with -O2 has little effect except to break debugging. > > > > the block of code from if_bge.c: > > > > 2705 if (ifp->if_drv_flags & IFF_DRV_RUNNING) { > > 2706 /* Check RX return ring producer/consumer. */ > > 2707 bge_rxeof(sc); > > 2708 > > 2709 /* Check TX ring producer/consumer. */ > > 2710 bge_txeof(sc); > > 2711 } > > Oops. I should have asked for the statment in bge_rxeof(). #7 0xffffffff801d5f17 in bge_rxeof (sc=0xffffffff8836b000) at /usr/src/sys/dev/bge/if_bge.c:2528 2528 m->m_pkthdr.len = m->m_len = cur_rx->bge_len - ETHER_CRC_LEN; (where m is defined as: 2449 struct mbuf *m = NULL; ) > > > By default -O2 is passed to CC (I don't use any custom make flags other > > than and only define CPUTYPE in my /etc/make.conf). > > -O2 is unfortunately the default for COPTFLAGS for most arches in > sys/conf/kern.pre.mk. All of my machines and most FreeBSD cluster > machines override this default in /etc/make.conf. > > With the override overridden for RELENG_6 amd64, gcc inlines bge_rxeof(), > so your environment must be a little different to get even the above > ifo. I think gdb can show the correct line numbers but not the call > frames (since there is no call). ddb and the kernel stack trace can > only show the call frames for actual calls. > > With -O1, I couldn't find any instruction similar to the mov to the > null pointer + 28. 28 is a popular offset in mbufs If you have a suggestion for an /etc/make.conf line, I can recompile the kernel accordingly assuming it still panics or locks up after the change of interface noted below. > > > The short of it is that this interface sees pretty much non-stop traffic > > as this is a mailserver (final destination) and is constantly being > > delivered to (direct disk access) and mail being retrieved (remote > > machine(s) with nfs mounted mail spools. If a momentary down of the > > interface is enough to completely panic the driver and then the kernel, > > this hardly seems "robust" if, in fact, this is what is happening. So > > the question arises as to what would be causing the down/up of the > > interface; I could start looking at the cable, the switch it's connected > > to and ... any other ideas? (I don't have watchdog enabled or anything > > like that, for example). > > I don't think down/up can occur in normal operation, since it takes ioctls > or a watchdog timeout to do it. Maybe some ioctls other than a full > down/up can cause problems... bge_init() is called for the following > ioctls: > - mtu changes > - some near down/up (possibly only these) > Suspend/resume and of course detach/attach do much the same things as > down/up. > > BTW, I added some sysctls and found it annoying to have to do down/up > to make the sysctls take effect. Sysctls in several other NIC drivers > require the same, since doing a full reinitialization is easiest. > Since I am tuning using sysctls, I got used to doing down/up too much. > > Similarly for the mtu ioctl. I think a full reinitialization is used > for mtu changes mainly in cases the change switches on/off support for > jumbo buffers. Then there is a lot of buffer reallocation to be > done, and interfaces have to be stopped to ensure that the bufferes > being deallocated are not in use, etc. > > Bruce As this was connected to a gigE switch with mtu left at 1500 I supposed it is possible that perhaps some mtu discovery/change may have been happening on the switch but that seems a bit out in left field. For now I am using the fxp interface connected to the same switch to see if the issue continues (the change of interface was driven by a hard lockup yesterday where I could not even type anything on the term). Sven