Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 12 Apr 2010 10:52:42 -0400
From:      John Baldwin <jhb@freebsd.org>
To:        freebsd-current@freebsd.org
Cc:        pyunyh@gmail.com, Jack Vogel <jfvogel@gmail.com>, Mike Tancsa <mike@sentex.net>
Subject:   Re: LOR on em in HEAD ( was Re: em driver regression
Message-ID:  <201004121052.42350.jhb@freebsd.org>
In-Reply-To: <z2r2a41acea1004091209maca81e47td7f03d7c343b3ec9@mail.gmail.com>
References:  <201004081313.o38DD4JM041821@lava.sentex.ca> <201004091900.o39J0b0k051687@lava.sentex.ca> <z2r2a41acea1004091209maca81e47td7f03d7c343b3ec9@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Friday 09 April 2010 3:09:24 pm Jack Vogel wrote:
> Someone else also pointed this out. I'm dubious about its claim.
> This happens because there is an RX lock taken in rxeof, its held
> thru the call into the stack, it then encounters another lock there
> and hence this complaint. I've had the RX hold as it is for a long
> while and would rather not have to give it up, can someone look
> at it and advise?

I've seen it happen with igb.  I suspect it is a transitive lock order.  That 
is, you probably never have the UDP lock acquired before an em/igb RX lock.  
However, if you have an em/igb adapter TX lock held when you acquire an em/igb 
RX lock in one place, and in if_start() you acquire the TX lock while the UDP 
lock is held, that can trigger the LOR.  Specifically, those two paths would 
give you these two orders:

TX -> RX
UDP -> TX

which implies the order

UDP -> RX

(lock order relationsips are transitive, just like a > b and b > c implies
a > c).

However, I haven't been able to track down what the raw orders are that might 
lead to this transitive order.  Attilio added some sysctls to dump all the raw 
lock orders in one of the debug.witness sysctls.  You can also try hardcoding 
the 'RX -> UDP' order using WITNESS_DEFINEORDER() before any of the em/igb 
RX/TX locks are acquired to see what different LOR is triggered.  If that LOR 
looks valid then you can keep hardcoding valid orders until you find the 
invalid one.

> On Fri, Apr 9, 2010 at 12:00 PM, Mike Tancsa <mike@sentex.net> wrote:
> 
> > While testing an i5 box with HEAD checked out from this morning, bringing
> > up the second NIC generated this LOR on the console
> >
> > em1: link state changed to UP
> > lock order reversal:
> >  1st 0xc5dc7c10 em1:rx(1) (em1:rx(1)) @
> > /usr/HEAD/src/sys/modules/em/../../dev/e1000/if_em.c:4089
> >  2nd 0xc0f7e88c udp (udp) @ /usr/HEAD/src/sys/netinet/udp_usrreq.c:454
> > KDB: stack backtrace:
> > db_trace_self_wrapper(c0cb4d33,c5b72a70,c08e4d65,c08d50db,c0cb7d58,...) at
> > db_trace_self_wrapper+0x26
> > kdb_backtrace(c08d50db,c0cb7d58,c5d31a98,c5d2cb60,c5b72acc,...) at
> > kdb_backtrace+0x29
> > _witness_debugger(c0cb7d58,c0f7e88c,c0c9cf0b,c5d2cb60,c0cd04ca,...) at
> > _witness_debugger+0x25
> > witness_checkorder(c0f7e88c,1,c0cd04ca,1c6,0,...) at
> > witness_checkorder+0x839
> > _rw_rlock(c0f7e88c,c0cd04ca,1c6,c5d33088,c5e8be24,...) at _rw_rlock+0x9c
> > udp_input(c67faa00,14,0,c5e8bd80,c0dfa860,...) at udp_input+0x246
> > ip_input(c67faa00,c5f2f380,c5b72be8,c07463b6,c0dfa860,...) at
> > ip_input+0x606
> > netisr_dispatch_src(1,0,c67faa00,c5b72c20,c0954bc1,...) at
> > netisr_dispatch_src+0xcd
> > netisr_dispatch(1,c67faa00,c6018c00,c6018c00,c6852800,...) at
> > netisr_dispatch+0x20
> > ether_demux(c6018c00,c67faa00,3,0,3,...) at ether_demux+0x1a1
> > ether_input(c6018c00,c67faa00,c11a0e17,ff9,64,...) at ether_input+0x365
> > em_rxeof(c5e8bd80,109,c6016180,0,c5b72cc8,...) at em_rxeof+0x13c
> > em_msix_rx(c5dc7c00,c5b72cc8,c088eb14,c0e133c0,c60342b8,...) at
> > em_msix_rx+0x25
> > intr_event_execute_handlers(c5d807f8,c6034280,c0cacd7e,533,c60342f0,...) 
at
> > intr_event_execute_handlers+0x125
> > ithread_loop(c603b4a0,c5b72d38,c0cacaed,343,c5d807f8,...) at
> > ithread_loop+0x9f
> > fork_exit(c0877800,c603b4a0,c5b72d38) at fork_exit+0xb8
> > fork_trampoline() at fork_trampoline+0x8
> > --- trap 0, eip = 0, esp = 0xc5b72d70, ebp = 0 ---
> >
> > 0(i5b)# uname -a
> > FreeBSD i5b.sentex.ca 9.0-CURRENT FreeBSD 9.0-CURRENT #2: Fri Apr  9
> > 11:56:25 EDT 2010     
mdtancsa@ich10.sentex.ca:/usr/HEAD/obj/usr/HEAD/src/sys/GENERIC
> >  i386
> > 0(i5b)#
> >
> > em0@pci0:0:25:0:        class=0x020000 card=0x34ec8086 chip=0x10ef8086
> > rev=0x05 hdr=0x00
> >    vendor     = 'Intel Corporation'
> >    class      = network
> >    subclass   = ethernet
> >    cap 01[c8] = powerspec 2  supports D0 D3  current D0
> >    cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
> >    cap 13[e0] = PCI Advanced Features: FLR TP
> > em1@pci0:3:0:0: class=0x020000 card=0x34ec8086 chip=0x10d38086 rev=0x00
> > hdr=0x00
> >    vendor     = 'Intel Corporation'
> >    device     = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
> >    class      = network
> >    subclass   = ethernet
> >    cap 01[c8] = powerspec 2  supports D0 D3  current D0
> >    cap 05[d0] = MSI supports 1 message, 64 bit
> >    cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
> >    cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled
> >
> >
> > 0(i5b)#  vmstat -i
> > interrupt                          total       rate
> > irq4: uart0                         6156          3
> > irq8: rtc                         224879        127
> > irq21: ehci0                        2662          1
> > irq23: ehci1                        3674          2
> > cpu0: timer                      1754210        998
> > irq256: em0                        10778          6
> > irq257: em1                          331          0
> > irq258: em1                            4          0
> > irq260: em1                            4          0
> > irq261: em1                            8          0
> > irq262: ahci0                         69          0
> > cpu3: timer                      1753938        998
> > cpu2: timer                      1753932        998
> > cpu1: timer                      1753886        998
> > Total                            7264531       4134
> > 0(i5b)#
> >
> > CPU: Intel(R) Core(TM) i5 CPU         750  @ 2.67GHz (2666.65-MHz 686-
class
> > CPU)
> >  Origin = "GenuineIntel"  Id = 0x106e5  Family = 6  Model = 1e  Stepping =
> > 5
> >
> >  
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> >
> >  
Features2=0x98e3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT>
> >  AMD Features=0x28100000<NX,RDTSCP,LM>
> >  AMD Features2=0x1<LAHF>
> >  TSC: P-state invariant
> > real memory  = 4294967296 (4096 MB)
> > avail memory = 2577711104 (2458 MB)
> > ACPI APIC Table: <INTEL  S3420GPC>
> > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
> >
> >
> >
> >
> > --------------------------------------------------------------------
> > Mike Tancsa,                                      tel +1 519 651 3400
> > Sentex Communications,                            mike@sentex.net
> > Providing Internet since 1994                    www.sentex.net
> > Cambridge, Ontario Canada                         www.sentex.net/mike
> >
> >
> _______________________________________________
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
> 

-- 
John Baldwin



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201004121052.42350.jhb>