From owner-freebsd-stable@FreeBSD.ORG Sat Aug 16 12:53:11 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F2D91065673 for ; Sat, 16 Aug 2008 12:53:11 +0000 (UTC) (envelope-from alson+ml@alm.flutnet.org) Received: from hatert.nijmegen.internl.net (mailrelay1.nijmegen.internl.net [217.149.192.44]) by mx1.freebsd.org (Postfix) with ESMTP id 26F858FC13 for ; Sat, 16 Aug 2008 12:53:10 +0000 (UTC) (envelope-from alson+ml@alm.flutnet.org) Received: from smtp20.nijmegen.internl.net by hatert.nijmegen.internl.net via smtp20.nijmegen.internl.net [217.149.192.18] with ESMTP for id m7GCHM4w007678 (8.13.6/2.04); Sat, 16 Aug 2008 14:17:22 +0200 (MEST) Received: from tafi.alm.flutnet.org (tafi.dsl.alm.flutnet.org [145.99.245.99]) by smtp20.nijmegen.internl.net (8.13.8/2.04) with ESMTP id m7GCHJ7R019615 for ; Sat, 16 Aug 2008 14:17:20 +0200 (CEST) Received: from localhost (localhost.alm.flutnet.org [127.0.0.1]) by tafi.alm.flutnet.org (Postfix) with ESMTP id C4C6B2844B for ; Sat, 16 Aug 2008 14:17:19 +0200 (CEST) X-Virus-Scanned: amavisd-new at alm.flutnet.org Received: from tafi.alm.flutnet.org ([127.0.0.1]) by localhost (tafi.alm.flutnet.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1NYxH48YgwoX for ; Sat, 16 Aug 2008 14:17:07 +0200 (CEST) Received: by tafi.alm.flutnet.org (Postfix, from userid 1000) id 000022844A; Sat, 16 Aug 2008 14:17:06 +0200 (CEST) Date: Sat, 16 Aug 2008 14:17:06 +0200 From: Alson van der Meulen To: freebsd-stable@freebsd.org Message-ID: <20080816121706.GA3040@waalsdorp.nl> Mail-Followup-To: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Mutt/1.5.18 (2008-05-17) Subject: Panic in nfs/ffs after upgrade from 6.2 to 6.3-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Aug 2008 12:53:11 -0000 Hello, This file server was upgraded from 6.2-RELEASE-p$something to 6.3-RELEASE-p3 (current RELENG_6_3 sources via csup) on August, 11 and rebooted to the new kernel on August, 12. The school is closed due to holidays. The only load is mail delivery to Maildirs via NFS from another server (just 50 mailboxes or so, none of them very busy) and nightly backups. I went there on August, 13 to perform some maintenance on unrelated systems. Performed a Windows installation (+ assorted desktop software) that fetched the installation files from Samba running on this server. This procedure downloads a total of ~5G in bursts via a 100mbit link. After most of this was over (I think it wast just installing patches at that point), the file server crashed. The crash: [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:= Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] [...] Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid =3D 0; apic id =3D 00 fault virtual address =3D 0x1c fault code =3D supervisor write, page not present instruction pointer =3D 0x20:0xc06fd937 stack pointer =3D 0x28:0xe78e78ec frame pointer =3D 0x28:0xe78e7914 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, def32 1, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 847 (nfsd) trap number =3D 12 panic: page fault cpuid =3D 1 KDB: stack backtrace: kdb_backtrace(100,c532f900,28,e78e78ac,c,...) at kdb_backtrace+0x29 panic(c0976f0c,c09cfe95,0,fffff,c532e49b,...) at panic+0x114 trap_fatal(e78e78ac,1c,c532f900,0,c,...) at trap_fatal+0x2ce trap_pfault(e78e78ac,0,1c) at trap_pfault+0x1f7 trap(e78e0008,c06a0028,e78e0028,200012,d9156118,...) at trap+0x325 calltrap() at calltrap+0x5 --- trap 0xc, eip =3D 0xc06fd937, esp =3D 0xe78e78ec, ebp =3D 0xe78e7914 --- getnewbuf(0,0,4000,4000) at getnewbuf+0x1bb getblk(c5964cc0,0,0,4000,0,...) at getblk+0x360 cluster_read(c5964cc0,46e5,0,0,0,...) at cluster_read+0xde ffs_read(e78e7b08) at ffs_read+0x25f VOP_READ_APV(c0a65560,e78e7b08) at VOP_READ_APV+0x38 nfsrv_read(c69b3800,c520d900,c532f900,e78e7c98,0,...) at nfsrv_read+0xb16 nfssvc_nfsd(c532f900) at nfssvc_nfsd+0x435 nfssvc(c532f900,e78e7d04) at nfssvc+0x1c0 syscall(3b,3b,3b,1,0,...) at syscall+0x2bf Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (155, FreeBSD ELF32, nfssvc), eip =3D 0x280bdf17, esp =3D 0xbfb= feb0c, ebp =3D 0xbfbfeb28 --- Uptime: 12h24m16s Dumping 1023 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 1023MB (261840 pages) 1007 991 975 959 943 927 911 895 879 863 8= 47 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 = 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255= 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=3Dr" (td)); (kgdb) list *0xc06fd937 0xc06fd937 is in getnewbuf (atomic.h:149). 144 static __inline int 145 atomic_cmpset_int(volatile u_int *dst, u_int exp, u_int src) 146 { 147 int res =3D exp; 148=09 149 __asm __volatile ( 150 " " __XSTRING(MPLOCKED) " " 151 " cmpxchgl %2,%1 ; " 152 " setz %%al ; " 153 " movzbl %%al,%0 ; " (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc06b25fe in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:4= 09 #2 0xc06b2955 in panic (fmt=3D0xc0976f0c "%s") at /usr/src/sys/kern/kern_s= hutdown.c:565 #3 0xc0924c86 in trap_fatal (frame=3D0xe78e78ac, eva=3D28) at /usr/src/sys= /i386/i386/trap.c:838 #4 0xc092498f in trap_pfault (frame=3D0xe78e78ac, usermode=3D0, eva=3D28) = at /usr/src/sys/i386/i386/trap.c:745 #5 0xc0924585 in trap (frame=3D {tf_fs =3D -410124280, tf_es =3D -1066794968, tf_ds =3D -410124248, t= f_edi =3D 2097170, tf_esi =3D -652910312, tf_ebp =3D -410093292, tf_isp =3D= -410093352, tf_ebx =3D 0, tf_edx =3D -1062624684, tf_ecx =3D -986515200, t= f_eax =3D 4, tf_trapno =3D 12, tf_err =3D 2, tf_eip =3D -1066411721, tf_cs = =3D 32, tf_eflags =3D 66182, tf_esp =3D 1, tf_ss =3D 4}) at /usr/src/sys/i3= 86/i386/trap.c:435 #6 0xc090ec9a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc06fd937 in getnewbuf (slpflag=3D0, slptimeo=3D0, size=3D16384, maxsi= ze=3D16384) at atomic.h:149 #8 0xc06fefdc in getblk (vp=3D0xc5964cc0, blkno=3D0, size=3D16384, slpflag= =3D0, slptimeo=3D0, flags=3D0) at /usr/src/sys/kern/vfs_bio.c:2516 #9 0xc0702a8a in cluster_read (vp=3D0xc5964cc0, filesize=3D18149, lblkno= =3D0, size=3D16384, cred=3D0x0, totread=3D8192, seqcount=3D5,=20 bpp=3D0x4) at /usr/src/sys/kern/vfs_cluster.c:118 #10 0xc081213b in ffs_read (ap=3D0x4) at /usr/src/sys/ufs/ffs/ffs_vnops.c:5= 03 #11 0xc09368a0 in VOP_READ_APV (vop=3D0x4, a=3D0xc0a9a254) at vnode_if.c:643 #12 0xc07ad086 in nfsrv_read (nfsd=3D0xc69b3800, slp=3D0xc520d900, td=3D0xc= 532f900, mrq=3D0xe78e7c98) at vnode_if.h:343 #13 0xc07bd329 in nfssvc_nfsd (td=3D0x4) at /usr/src/sys/nfsserver/nfs_sysc= alls.c:474 #14 0xc07bcb08 in nfssvc (td=3D0xc532f900, uap=3D0xe78e7d04) at /usr/src/sy= s/nfsserver/nfs_syscalls.c:181 #15 0xc0924fcb in syscall (frame=3D {tf_fs =3D 59, tf_es =3D 59, tf_ds =3D 59, tf_edi =3D 1, tf_esi =3D 0= , tf_ebp =3D -1077941464, tf_isp =3D -410092188, tf_ebx =3D 4, tf_edx =3D 6= 72460376, tf_ecx =3D 25, tf_eax =3D 155, tf_trapno =3D 12, tf_err =3D 2, tf= _eip =3D 671866647, tf_cs =3D 51, tf_eflags =3D 662, tf_esp =3D -1077941492= , tf_ss =3D 59}) at /usr/src/sys/i386/i386/trap.c:984 #16 0xc090ecef in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s= :200 #17 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) I have the crash dump and kernel build files still available, so let me know if I can provide any other useful output from the dump. The crash is in the NFS server part of the kernel, which is funny because NFS is only used for mail delivery. Between 0am and 11.59pm, 40 mails very delivered from the mailserver to maildirs on NFS. There were also a few clients checking their mailboxes via IMAP. The server booted again at 15.25, these are the imapd logs from around that time: Aug 13 15:22:48 eraser imapd-ssl: LOGIN, user=3DXXX, ip=3D[::ffff:x.x.x.x],= port=3D[63802], protocol=3DIMAP Aug 13 15:22:48 eraser imapd-ssl: LOGIN, user=3DXXX, ip=3D[::ffff:x.x.x.x],= port=3D[63803], protocol=3DIMAP Aug 13 15:22:49 eraser imapd-ssl: LOGOUT, user=3DXXX, ip=3D[::ffff:x.x.x.x]= , headers=3D0, body=3D0, rcvd=3D96, sent=3D470, time=3D1, starttls=3D1 Aug 13 15:22:49 eraser imapd-ssl: LOGOUT, user=3DXXX, ip=3D[::ffff:x.x.x.x]= , headers=3D0, body=3D0, rcvd=3D252, sent=3D944, time=3D1, starttls=3D1 Aug 13 15:22:49 eraser imapd-ssl: LOGIN, user=3DXXX, ip=3D[::ffff:x.x.x.x],= port=3D[63806], protocol=3DIMAP Aug 13 15:22:49 eraser imapd-ssl: LOGOUT, user=3DXXX, ip=3D[::ffff:x.x.x.x]= , headers=3D0, body=3D0, rcvd=3D38, sent=3D273, time=3D0, starttls=3D1 Aug 13 15:22:52 eraser imapd-ssl: LOGIN, user=3DXXX, ip=3D[::ffff:x.x.x.x],= port=3D[63808], protocol=3DIMAP Aug 13 15:22:52 eraser imapd-ssl: LOGOUT, user=3DXXX, ip=3D[::ffff:x.x.x.x]= , headers=3D0, body=3D0, rcvd=3D38, sent=3D273, time=3D0, starttls=3D1 Aug 13 15:22:53 eraser imapd-ssl: LOGIN, user=3DXXX, ip=3D[::ffff:x.x.x.x],= port=3D[63811], protocol=3DIMAP Aug 13 15:22:53 eraser imapd-ssl: LOGOUT, user=3DXXX, ip=3D[::ffff:x.x.x.x]= , headers=3D0, body=3D0, rcvd=3D106, sent=3D392, time=3D0, starttls=3D1 Aug 13 15:22:54 eraser imapd-ssl: LOGIN, user=3DXXX, ip=3D[::ffff:x.x.x.x],= port=3D[63812], protocol=3DIMAP Aug 13 15:26:35 eraser imapd-ssl: LOGIN, user=3DYYY, ip=3D[::ffff:x.x.x.x],= port=3D[64582], protocol=3DIMAP Aug 13 15:26:35 eraser imapd-ssl: LOGOUT, user=3DYYY, ip=3D[::ffff:x.x.x.x]= , headers=3D0, body=3D0, rcvd=3D106, sent=3D394, time=3D0, starttls=3D1 Aug 13 15:26:50 eraser imapd-ssl: DISCONNECTED, user=3DXXX, ip=3D[::ffff:x.= x.x.x], headers=3D0, body=3D0, rcvd=3D135, sent=3D10825, time=3D236, startt= ls=3D1 Aug 13 15:27:55 eraser imapd-ssl: LOGIN, user=3DXXX, ip=3D[::ffff:x.x.x.x],= port=3D[64975], protocol=3DIMAP Aug 13 15:27:55 eraser imapd-ssl: LOGOUT, user=3DXXX, ip=3D[::ffff:x.x.x.x]= , headers=3D0, body=3D0, rcvd=3D106, sent=3D392, time=3D1, starttls=3D1 The six almost simultaneous logins from user XXX look suspicious and =66rom the timestamp seem to be just prior to the crash, but shouldn't lead to the crash of the nfs server. The lack of logout after the 15:22:54 one suggests that this was the point that the file server went offline. The only directory that was mounted at that time was in /export (/dev/mirror/gm1a): alson@damaged:~$ gmirror status Name Status Components mirror/gm0 DEGRADED ad4s1 mirror/gm1 COMPLETE ad8s1 ad10s1 The server had been up for about 250 days with similar (and usually significantly heavier) load with 6.2-STABLE before the upgrade. The evening after the crash, I received the following message from smartmontools (not sure if it's related): Aug 13 21:19:30 damaged smartd[881]: Device: /dev/ad10, 4 Currently unreada= ble (pending) sectors Aug 13 21:19:30 damaged smartd[881]: Device: /dev/ad10, 4 Offline uncorrect= able sectors ad10 is part of the mirror, but I don't think this should be a problem since (a) the disk should have corrected it and (b) gmirror should have dropped the disk and try to read from ad8 if it had returned a read error. The server has been stable since, but hasn't seen much usage either. I have no idea if I can reproduce it. My first thought was disk corruption, but this doesn't appear to be the cas= e: # fsck -f=20 [...] ** /dev/mirror/gm1a ** Last Mounted on /export ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups FREE BLK COUNT(S) WRONG IN SUPERBLK SALVAGE? [yn] y 953252 files, 47080232 used, 44658220 free (159804 frags, 5562302 blocks, 0= =2E2% fragmentation) ***** FILE SYSTEM MARKED CLEAN ***** ***** FILE SYSTEM WAS MODIFIED ***** dmesg is attached below. regards, Alson Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.3-RELEASE-p3 #10: Mon Aug 11 22:01:47 CEST 2008 root@damaged.example.com:/usr/obj/usr/src/sys/DAMAGED Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (3000.12-MHz 686-class CPU) Origin =3D "GenuineIntel" Id =3D 0xf43 Stepping =3D 3 Features=3D0xbfebfbff Features2=3D0x649d AMD Features=3D0x20000000 Logical CPUs per core: 2 real memory =3D 1073545216 (1023 MB) avail memory =3D 1037168640 (989 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) hptrr: HPT RocketRAID controller driver v1.1 (Aug 11 2008 22:01:29) acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 acpi_throttle0: on cpu0 cpu1: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 28.0 on pci0 pci4: on pcib1 pcib2: irq 16 at device 28.4 on pci0 pci3: on pcib2 bge0: mem 0xfeaf0000-0xfeafffff irq= 16 at device 0.0 on pci3 miibus0: on bge0 brgphy0: on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000ba= seT-FDX, auto bge0: Ethernet address: 00:18:f3:2a:e0:cf pcib3: irq 17 at device 28.5 on pci0 pci2: on pcib3 bge1: mem 0xfe9f0000-0xfe9fffff irq= 17 at device 0.0 on pci2 miibus1: on bge1 brgphy1: on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000ba= seT-FDX, auto bge1: Ethernet address: 00:18:f3:2a:e0:58 uhci0: port 0xd880-0xd89f irq 16 at device = 29.0 on pci0 uhci0: [GIANT-LOCKED] usb0: on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1: port 0xdc00-0xdc1f irq 17 at device = 29.1 on pci0 uhci1: [GIANT-LOCKED] usb1: on uhci1 usb1: USB revision 1.0 uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0xe000-0xe01f irq 18 at device = 29.2 on pci0 uhci2: [GIANT-LOCKED] usb2: on uhci2 usb2: USB revision 1.0 uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub2: 2 ports with 2 removable, self powered uhci3: port 0xe080-0xe09f irq 19 at device = 29.3 on pci0 uhci3: [GIANT-LOCKED] usb3: on uhci3 usb3: USB revision 1.0 uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xfebff800-0xfebffbf= f irq 16 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub4: 8 ports with 8 removable, self powered pcib4: at device 30.0 on pci0 pci1: on pcib4 puc0: port 0xcc00-0xcc1f,0xc880-0xc887,0xc800-0x= c807 irq 20 at device 1.0 on pci1 sio4: on puc0 sio4: type 16550A sio4: unable to activate interrupt in fast mode - using normal mode sio5: on puc0 sio5: type 16550A sio5: unable to activate interrupt in fast mode - using normal mode pci1: at device 2.0 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177= ,0x376,0xffa0-0xffaf at device 31.1 on pci0 ata0: on atapci0 ata1: on atapci0 atapci1: port 0xec00-0xec07,0xe880-0xe883,0xe800-0x= e807,0xe480-0xe483,0xe400-0xe41f mem 0xfebffc00-0xfebfffff irq 19 at device= 31.2 on pci0 atapci1: AHCI Version 01.10 controller with 4 ports detected ata2: on atapci1 ata3: on atapci1 ata4: on atapci1 ata5: on atapci1 ichsmb0: port 0x400-0x41f at device= 31.3 on pci0 ichsmb0: [GIANT-LOCKED] smbus0: on ichsmb0 smb0: on smbus0 acpi_button0: on acpi0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on= acpi0 fdc0: [FAST] ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppbus0: on ppc0 plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acp= i0 sio0: type 16550A, console pmtimer0 on isa0 orm0: at iomem 0xc0000-0xc7fff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=3D0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec hptrr: no controller detected. acd0: CDROM at ata0-master PIO4 ad4: 35304MB at ata2-master SATA150 ad8: 190782MB at ata4-master SATA150 GEOM_MIRROR: Device gm0 created (id=3D1495539501). GEOM_MIRROR: Device gm0: provider ad4s1 detected. ad10: 190782MB at ata5-master SATA150 GEOM_MIRROR: Device gm1 created (id=3D1615731132). GEOM_MIRROR: Device gm1: provider ad8s1 detected. SMP: AP CPU #1 Launched! GEOM_MIRROR: Device gm1: provider ad10s1 detected. GEOM_MIRROR: Device gm1: provider ad8s1 activated. GEOM_MIRROR: Device gm1: provider mirror/gm1 launched. GEOM_MIRROR: Device gm1: rebuilding provider ad10s1. Root mount waiting for: GMIRROR Root mount waiting for: GMIRROR Root mount waiting for: GMIRROR Root mount waiting for: GMIRROR GEOM_MIRROR: Force device gm0 start due to timeout. GEOM_MIRROR: Device gm0: provider ad4s1 activated. GEOM_MIRROR: Device gm0: provider mirror/gm0 launched. Trying to mount root from ufs:/dev/mirror/gm0a vlan2: link state changed to UP vlan1: link state changed to UP bge1: link state changed to UP GEOM_MIRROR: Device gm1: rebuilding provider ad10s1 finished. GEOM_MIRROR: Device gm1: provider ad10s1 activated.