Date: Fri, 20 Dec 2013 11:48:35 +0200 From: Konstantin Belousov <kostikbel@gmail.com> To: Sean Bruno <seanbru@yahoo-inc.com> Cc: "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org>, Peter Wemm <peter@wemm.org> Subject: Re: 10.0 BETA 3 with redports kernel panic Message-ID: <20131220094835.GR59496@kib.kiev.ua> In-Reply-To: <1387492541.27693.5.camel@powernoodle.corp.yahoo.com> References: <1387217065.1407.3.camel@powernoodle.corp.yahoo.com> <1387219558.1407.6.camel@powernoodle.corp.yahoo.com> <20131217120019.GD59496@kib.kiev.ua> <1387285472.2372.2.camel@powernoodle.corp.yahoo.com> <1387473915.2494.0.camel@powernoodle.corp.yahoo.com> <20131219180833.GN59496@kib.kiev.ua> <1387479064.2494.5.camel@powernoodle.corp.yahoo.com> <CAGE5yCqhmRSM6oyw=FRZq59LniLsYaN%2BEog=GEPn3-ruuQk9EQ@mail.gmail.com> <CAGE5yCp9msPAy4HZ4TGRXws%2BxjLQ8iChbPQGj539qHJKhq2UJQ@mail.gmail.com> <1387492541.27693.5.camel@powernoodle.corp.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--fB0WXPwaL5nVAEzU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Dec 19, 2013 at 02:35:41PM -0800, Sean Bruno wrote: > On Thu, 2013-12-19 at 11:20 -0800, Peter Wemm wrote: > > On Thu, Dec 19, 2013 at 10:59 AM, Peter Wemm <peter@wemm.org> wrote: > > > On Thu, Dec 19, 2013 at 10:51 AM, Sean Bruno <sean_bruno@yahoo.com> w= rote: > > >> On Thu, 2013-12-19 at 20:08 +0200, Konstantin Belousov wrote: > > >>> On Thu, Dec 19, 2013 at 09:25:15AM -0800, Sean Bruno wrote: > > >>> > On Tue, 2013-12-17 at 05:04 -0800, Sean Bruno wrote: > > >>> > > On Tue, 2013-12-17 at 14:00 +0200, Konstantin Belousov wrote: > > >>> > > > On Mon, Dec 16, 2013 at 10:45:58AM -0800, Sean Bruno wrote: > > >>> > > > > On Mon, 2013-12-16 at 10:04 -0800, Sean Bruno wrote: > > >>> > > > > > > What is the source line for memrw+0x195 ? > > >>> > > > > > > > >>> > > > > > My apologies for the delay on this. Its been frustrating= getting a > > >>> > > > > > crashdump on these machines due to their very large tmpfs= usage. > > >>> > > > > > Currently, I am dumping a crash of 13+GB to a third HD th= at we had > > >>> > > > > > installed for this purpose. > > >>> > > > > > > > >>> > > > > > The machines are still running RC3 of 10.0r. > > >>> > > > > > > > >>> > > > > > I will attempt to get the requested information shortly. > > >>> > > > > > > > >>> > > > > > sean > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > I've updated http://people.freebsd.org/~sbruno/redbuild_pan= ic.txt > > >>> > > > > > > >>> > > > > It looks like its dying in uiomove() ? > > >>> > > > > > >>> > > > I believe I already posted the following patch, with no feedb= ack. > > >>> > > > > > >>> > > > diff --git a/sys/amd64/amd64/mem.c b/sys/amd64/amd64/mem.c > > >>> > > > index abbbb21..e371499 100644 > > >>> > > > --- a/sys/amd64/amd64/mem.c > > >>> > > > +++ b/sys/amd64/amd64/mem.c > > >>> > > > @@ -98,7 +98,11 @@ memrw(struct cdev *dev, struct uio *uio, i= nt flags) > > >>> > > > kmemphys: > > >>> > > > o =3D v & PAGE_MASK; > > >>> > > > c =3D min(uio->uio_resid, (u_int)(PAG= E_SIZE - o)); > > >>> > > > - error =3D uiomove((void *)PHYS_TO_DMA= P(v), (int)c, uio); > > >>> > > > + v =3D PHYS_TO_DMAP(v); > > >>> > > > + if (v < DMAP_MIN_ADDRESS || v >=3D DM= AP_MAX_ADDRESS || > > >>> > > > + pmap_kextract(v) =3D=3D 0) > > >>> > > > + return (EFAULT); > > >>> > > > + error =3D uiomove((void *)v, (int)c, = uio); > > >>> > > > continue; > > >>> > > > } > > >>> > > > else if (dev2unit(dev) =3D=3D CDEV_MINOR_KMEM= ) { > > >>> > > > > >>> > > Will begin testing immediately > > >>> > > > > >>> > > sean > > >>> > > > >>> > > > >>> > Huh ... both machines panic'd this morning. It'll take 30 minute= s or so > > >>> > to get a crash dump, but it looks like its still in the same plac= e. > > >>> > > > >>> > db> whe > > >>> > Tracing pid 489 tid 101801 td 0xfffff80322946490 > > >>> > kdb_enter() at kdb_enter+0x3e/frame 0xfffffe1839d26220 > > >>> > panic() at panic+0x175/frame 0xfffffe1839d262a0 > > >>> > vm_fault_hold() at vm_fault_hold+0x14ed/frame 0xfffffe1839d26500 > > >>> > vm_fault() at vm_fault+0x77/frame 0xfffffe1839d26540 > > >>> > trap_pfault() at trap_pfault+0x19b/frame 0xfffffe1839d265f0 > > >>> > trap() at trap+0x5e6/frame 0xfffffe1839d26810 > > >>> > calltrap() at calltrap+0x8/frame 0xfffffe1839d26810 > > >>> > --- trap 0xc, rip =3D 0xffffffff80cae47b, rsp =3D 0xfffffe1839d26= 8d0, rbp =3D > > >>> > 0xfffffe1839d26920 --- > > >>> > copyout() at copyout+0x3b/frame 0xfffffe1839d26920 > > >>> > memrw() at memrw+0x1b6/frame 0xfffffe1839d26960 > > >>> > giant_read() at giant_read+0x7a/frame 0xfffffe1839d269a0 > > >>> > devfs_read_f() at devfs_read_f+0xea/frame 0xfffffe1839d26a00 > > >>> > dofileread() at dofileread+0x7b/frame 0xfffffe1839d26a40 > > >>> > kern_readv() at kern_readv+0x65/frame 0xfffffe1839d26a90 > > >>> > sys_read() at sys_read+0x63/frame 0xfffffe1839d26ae0 > > >>> > amd64_syscall() at amd64_syscall+0x357/frame 0xfffffe1839d26bf0 > > >>> > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe1839d26bf0 > > >>> > --- syscall (3, FreeBSD ELF64, sys_read), rip =3D 0x800b750aa, rs= p =3D > > >>> > 0x7fffffffd068, rbp =3D 0x7fffffffd0b0 --- > > >>> > db> call doadump > > >>> > > > >>> > > >>> I need to see exact panic and trap messages, as well as I need to k= now > > >>> the source line for memrw+0x1b6 in the patched kernel. > > >> > > >> Here is the panic/trap and the requested display. Peter suspects th= at > > >> part of the failure is the use of DMAP_MAX_ADDR and not dmaplimit in > > >> this and other comparisons. Patch attached that contains your > > >> modifications and his. > > >> > > >> bcc peter@ > > >> > > >> > > >> panic: vm_fault: fault on nofault entry, addr: fffffe0327240000 > > >> cpuid =3D 16 > > >> KDB: stack backtrace: > > >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > > >> 0xfffffe1839d26170 > > >> kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe1839d26220 > > >> panic() at panic+0x155/frame 0xfffffe1839d262a0 > > >> vm_fault_hold() at vm_fault_hold+0x14ed/frame 0xfffffe1839d26500 > > >> vm_fault() at vm_fault+0x77/frame 0xfffffe1839d26540 > > >> trap_pfault() at trap_pfault+0x19b/frame 0xfffffe1839d265f0 > > >> trap() at trap+0x5e6/frame 0xfffffe1839d26810 > > >> calltrap() at calltrap+0x8/frame 0xfffffe1839d26810 > > >> --- trap 0xc, rip =3D 0xffffffff80cae47b, rsp =3D 0xfffffe1839d268d0= , rbp =3D > > >> 0xfffffe1839d26920 --- > > >> copyout() at copyout+0x3b/frame 0xfffffe1839d26920 > > >> memrw() at memrw+0x1b6/frame 0xfffffe1839d26960 > > >> giant_read() at giant_read+0x7a/frame 0xfffffe1839d269a0 > > >> devfs_read_f() at devfs_read_f+0xea/frame 0xfffffe1839d26a00 > > >> dofileread() at dofileread+0x7b/frame 0xfffffe1839d26a40 > > >> kern_readv() at kern_readv+0x65/frame 0xfffffe1839d26a90 > > >> sys_read() at sys_read+0x63/frame 0xfffffe1839d26ae0 > > >> amd64_syscall() at amd64_syscall+0x357/frame 0xfffffe1839d26bf0 > > >> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe1839d26bf0 > > >> --- syscall (3, FreeBSD ELF64, sys_read), rip =3D 0x800b750aa, rsp = =3D > > >> 0x7fffffffd068, rbp =3D 0x7fffffffd0b0 --- > > >> KDB: enter: panic > > >> > > >> > > >> (kgdb) whe > > >> #0 doadump (textdump=3D-2127435168) at pcpu.h:219 > > >> #1 0xffffffff80342e25 in db_fncall (dummy1=3D<value optimized out>,= dummy2=3D<value optimized out>, dummy3=3D<value optimized out>, dummy4=3D<= value optimized out>) > > >> at /usr/src/sys/ddb/db_command.c:578 > > >> #2 0xffffffff80342b0d in db_command (cmd_table=3D<value optimized o= ut>) at /usr/src/sys/ddb/db_command.c:449 > > >> #3 0xffffffff80342884 in db_command_loop () at /usr/src/sys/ddb/db_= command.c:502 > > >> #4 0xffffffff803451f0 in db_trap (type=3D<value optimized out>, cod= e=3D0) at /usr/src/sys/ddb/db_main.c:231 > > >> #5 0xffffffff808fad33 in kdb_trap (type=3D3, code=3D0, tf=3D<value = optimized out>) at /usr/src/sys/kern/subr_kdb.c:656 > > >> #6 0xffffffff80cb0277 in trap (frame=3D0xfffffe1839d26150) at /usr/= src/sys/amd64/amd64/trap.c:579 > > >> #7 0xffffffff80c96ef2 in calltrap () at /usr/src/sys/amd64/amd64/ex= ception.S:232 > > >> #8 0xffffffff808fa4ee in kdb_enter (why=3D0xffffffff80f07ff2 "panic= ", msg=3D<value optimized out>) at cpufunc.h:63 > > >> #9 0xffffffff808c1eb5 in panic (fmt=3D<value optimized out>) at /us= r/src/sys/kern/kern_shutdown.c:747 > > >> #10 0xffffffff80b299ed in vm_fault_hold (map=3D0xfffff80002000000, v= addr=3D<value optimized out>, fault_type=3D1 '\001', fault_flags=3D0, m_hol= d=3D0x0) at /usr/src/sys/vm/vm_fault.c:279 > > >> #11 0xffffffff80b284b7 in vm_fault (map=3D0xfffff80002000000, vaddr= =3D<value optimized out>, fault_type=3D1 '\001', fault_flags=3D0) at /usr/s= rc/sys/vm/vm_fault.c:224 > > >> #12 0xffffffff80cb08cb in trap_pfault (frame=3D0xfffffe1839d26820, u= sermode=3D0) at /usr/src/sys/amd64/amd64/trap.c:775 > > >> #17 0xffffffff80c9e746 in memrw (dev=3D<value optimized out>, uio=3D= <value optimized out>, flags=3D<value optimized out>) at /usr/src/sys/amd64= /amd64/mem.c:105 > > >> #18 0xffffffff8087323a in giant_read (dev=3D0xfffff80011302e00, uio= =3D0xfffffe1839d26ab0, ioflag=3D0) at /usr/src/sys/kern/kern_conf.c:444 > > >> #19 0xffffffff807b670a in devfs_read_f (fp=3D0xfffff80033711a50, uio= =3D0xfffffe1839d26ab0, cred=3D<value optimized out>, flags=3D0, td=3D0xffff= f80322946490) > > >> at /usr/src/sys/fs/devfs/devfs_vnops.c:1193 > > >> #20 0xffffffff809117eb in dofileread (td=3D0xfffff80322946490, fd=3D= 4, fp=3D0xfffff80033711a50, auio=3D0xfffffe1839d26ab0, offset=3D<value opti= mized out>, flags=3D0) at file.h:295 > > >> #21 0xffffffff80911525 in kern_readv (td=3D0xfffff80322946490, fd=3D= 4, auio=3D0xfffffe1839d26ab0) at /usr/src/sys/kern/sys_generic.c:256 > > >> #22 0xffffffff809114b3 in sys_read (td=3D<value optimized out>, uap= =3D<value optimized out>) at /usr/src/sys/kern/sys_generic.c:171 > > >> #23 0xffffffff80cb1017 in amd64_syscall (td=3D0xfffff80322946490, tr= aced=3D0) at subr_syscall.c:134 > > >> #24 0xffffffff80c971db in Xfast_syscall () at /usr/src/sys/amd64/amd= 64/exception.S:391 > > >> #25 0x0000000800b750aa in ?? () > > >> Previous frame inner to this frame (corrupt stack?) > > >> Current language: auto; currently minimal > > >> (kgdb) p memrw+0x1b6 > > >> $1 =3D (int (*)(struct cdev *, struct uio *, int)) 0xffffffff80c9e74= 6 <memrw+438> > > >> (kgdb) f 17 > > >> #17 0xffffffff80c9e746 in memrw (dev=3D<value optimized out>, uio=3D= <value optimized out>, flags=3D<value optimized out>) at /usr/src/sys/amd64= /amd64/mem.c:105 > > >> 105 error =3D uiomove((void *)v, (int)c,= uio); > > >> (kgdb) list > > >> 100 c =3D min(uio->uio_resid, (u_int)(PA= GE_SIZE - o)); > > >> 101 v =3D PHYS_TO_DMAP(v); > > >> 102 if (v < DMAP_MIN_ADDRESS || v >=3D D= MAP_MAX_ADDRESS || > > >> 103 pmap_kextract(v) =3D=3D 0) > > >> 104 return (EFAULT); > > >> 105 error =3D uiomove((void *)v, (int)c,= uio); > > >> 106 continue; > > >> 107 } > > >> 108 else if (dev2unit(dev) =3D=3D CDEV_MINOR_KME= M) { > > >> 109 v =3D uio->uio_offset; > > >> > > >> > > >> > > >> Index: sys/amd64/amd64/mem.c > > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > >> --- sys/amd64/amd64/mem.c (revision 258554) > > >> +++ sys/amd64/amd64/mem.c (working copy) > > >> @@ -98,7 +98,11 @@ > > >> kmemphys: > > >> o =3D v & PAGE_MASK; > > >> c =3D min(uio->uio_resid, (u_int)(PAGE_SIZE = - o)); > > >> - error =3D uiomove((void *)PHYS_TO_DMAP(v), (= int)c, > > >> uio); > > >> + v =3D PHYS_TO_DMAP(v); > > >> + if (v < DMAP_MIN_ADDRESS || v >=3D > > >> DMAP_MAX_ADDRESS || > > >> + pmap_kextract(v) =3D=3D 0) > > >> + return (EFAULT); > > >> + error =3D uiomove((void *)v, (int)c, uio); > > >> continue; > > >> } > > >> else if (dev2unit(dev) =3D=3D CDEV_MINOR_KMEM) { > > >> Index: sys/amd64/amd64/pmap.c > > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > >> --- sys/amd64/amd64/pmap.c (revision 258554) > > >> +++ sys/amd64/amd64/pmap.c (working copy) > > >> @@ -1870,7 +1870,7 @@ > > >> pd_entry_t pde; > > >> vm_paddr_t pa; > > >> > > >> - if (va >=3D DMAP_MIN_ADDRESS && va < DMAP_MAX_ADDRESS) { > > >> + if (va >=3D DMAP_MIN_ADDRESS && va < dmaplimit) { > > >> pa =3D DMAP_TO_PHYS(va); > > >> } else { > > >> pde =3D *vtopde(va); > > >> @@ -3308,7 +3308,7 @@ > > >> */ > > >> if ((oldpde & PG_A) =3D=3D 0 || (mpte =3D vm_page_al= loc(NULL, > > >> pmap_pde_pindex(va), (va >=3D DMAP_MIN_ADDRESS &= & va < > > >> - DMAP_MAX_ADDRESS ? VM_ALLOC_INTERRUPT : > > >> VM_ALLOC_NORMAL) | > > >> + dmaplimit ? VM_ALLOC_INTERRUPT : VM_ALLOC_NORMAL= ) | > > >> VM_ALLOC_NOOBJ | VM_ALLOC_WIRED)) =3D=3D NULL) { > > >> SLIST_INIT(&free); > > >> pmap_remove_pde(pmap, pde, trunc_2mpage(va), > > >> &free, > > >> @@ -6117,7 +6117,7 @@ > > >> vm_offset_t base, offset; > > >> > > >> /* If we gave a direct map region in pmap_mapdev, do nothing= */ > > >> - if (va >=3D DMAP_MIN_ADDRESS && va < DMAP_MAX_ADDRESS) > > >> + if (va >=3D DMAP_MIN_ADDRESS && va < dmaplimit) > > >> return; > > >> base =3D trunc_page(va); > > >> offset =3D va & PAGE_MASK; > > >> > > >> > > > > > > Specifically, pmap_kextract(v) is nothing more than a repeat of the > > > if() that kib added to mem.c. pmap_kextract() doesn't check to see if > > > it is attempting to access beyond the end of the instantiated part of > > > the direct map region. pmap_kextract(invalid_address) returns a value > > > even between dmaplimit and DMAP_MAX_ADDRESS - and that'll lead to a > > > fault. > >=20 > > The patch is wrong, as you found out. :) > >=20 > > --=20 > > Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com; KI= 6FJV > > Yes, I know, gmail sucks now. If you see this then I forgot. Habits > > are hard to break. >=20 >=20 >=20 > Yah, ACPI does not like this in the slightest. What does ACPI not like ? The panic below does not look related. The patch only changed the code path for usermode access to /dev/mem and /dev/kmem. Anyway, at the end is the patch which should be better after Peter' diagnostic of the cause. Please give it a try. For me, kgdb /dev/mem still worked with the patch applied. >=20 > KDB: debugger backends: ddb > KDB: current backend: ddb > kernel trap 12 with interrupts disabled >=20 >=20 > Fatal trap 12: page fault while in kernel mode > cpuid =3D 0; apic id =3D 00 > fault virtual address =3D 0x378 > fault code =3D supervisor read data, page not present > instruction pointer =3D 0x20:0xffffffff808a9b31 > stack pointer =3D 0x28:0xffffffff81a90b50 > frame pointer =3D 0x28:0xffffffff81a90bd0 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D resume, IOPL =3D 0 > current process =3D 0 () > [ thread pid 0 tid 0 ] > Stopped at __mtx_lock_sleep+0x1b1: movl 0x378(%rax),%ecx > db> bt > Tracing pid 0 tid 0 td 0xffffffff81527500 > __mtx_lock_sleep() at __mtx_lock_sleep+0x1b1/frame 0xffffffff81a90bd0 > vmem_xfree() at vmem_xfree+0x42/frame 0xffffffff81a90c10 > acpi_find_table() at acpi_find_table+0x274/frame 0xffffffff81a90c60 > madt_probe() at madt_probe+0x10/frame 0xffffffff81a90c70 > apic_init() at apic_init+0x53/frame 0xffffffff81a90c90 > mi_startup() at mi_startup+0x118/frame 0xffffffff81a90cb0 > btext() at btext+0x2c >=20 diff --git a/sys/amd64/amd64/mem.c b/sys/amd64/amd64/mem.c index abbbb21..e371499 100644 --- a/sys/amd64/amd64/mem.c +++ b/sys/amd64/amd64/mem.c @@ -98,7 +98,11 @@ memrw(struct cdev *dev, struct uio *uio, int flags) kmemphys: o =3D v & PAGE_MASK; c =3D min(uio->uio_resid, (u_int)(PAGE_SIZE - o)); - error =3D uiomove((void *)PHYS_TO_DMAP(v), (int)c, uio); + v =3D PHYS_TO_DMAP(v); + if (v < DMAP_MIN_ADDRESS || v >=3D DMAP_MAX_ADDRESS || + pmap_kextract(v) =3D=3D 0) + return (EFAULT); + error =3D uiomove((void *)v, (int)c, uio); continue; } else if (dev2unit(dev) =3D=3D CDEV_MINOR_KMEM) { diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c index 014020b..2569699 100644 --- a/sys/amd64/amd64/pmap.c +++ b/sys/amd64/amd64/pmap.c @@ -1869,7 +1872,7 @@ pmap_kextract(vm_offset_t va) pd_entry_t pde; vm_paddr_t pa; =20 - if (va >=3D DMAP_MIN_ADDRESS && va < DMAP_MAX_ADDRESS) { + if (va >=3D DMAP_MIN_ADDRESS && va < dmaplimit) { pa =3D DMAP_TO_PHYS(va); } else { pde =3D *vtopde(va); --fB0WXPwaL5nVAEzU Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJStBJyAAoJEJDCuSvBvK1B1ggQAJWbltMcLhCkAbp1hP5hvGyI jKGMRvoW4usAiSCEKSvOYleZjWi0kRlJ8aVLuoNJTEXzet23xDZPuulyKw64SuSh RVQ66F044FLcLFH9xI8ZNoND+Wa/RokEhUnXPLnRZ+XV26QGuEgWEhBTekpfrXDS N5gnCIFoD2VGqVwsY5AldFjxC1cs7vmA+NUYe/3CvHqL66LoMfdz3sDnqrN+12dm CRNl/n2SiHDnHuQe5lmBL5lOSjJIsv2LoQxhZ1DdoZv/u82ScySk6mdtPssSLC57 aBSqOC76RVpG+cgjpe6LKeCvV/6LrgTdCcHkyS2ILiVckzcUlBf2VjkvWaHVPtnD UWLSltReZEEmdtyE+ZJNba+naEzCrGHjFkgYAqpnOBydq08k5QFWx6ckQT5NdDO3 OlUsOglb/bctRPFcgJ4WNGLfhkd8a5+IYedP1nhOT875MbVn5B5i7KIk2A6OeZ5I qqClICoMVIRdhmuzK0oio6T4fCDevUqb8Cv+nCf1uLhFr57x0YAvW0DJ3uOGbkGN IRVYrF8GfComnpEkT6HlfDdTIymiaGlAW0cU6Nph4fNyjlHZ3Z1G5Coio+bonH4G DVH6mnMNXHuBO4eB9+YWJDDUVtGiP3n6nPiMQ8Sg29JzywCQm6zW3v06HbzD6bSu klvzN6zPkxrW309YhQR1 =YZKU -----END PGP SIGNATURE----- --fB0WXPwaL5nVAEzU--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20131220094835.GR59496>