Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 13 Jan 2005 09:41:58 -0800 (PST)
From:      Doug White <dwhite@gumbysoft.com>
To:        Matthew Sullivan <matthew@uq.edu.au>
Cc:        freebsd-current@freebsd.org
Subject:   Re: Fatal Trap 12: Page fault while in kernel mode (racoon/amd64/5.3-RELEASE-p4)
Message-ID:  <20050113093955.P12838@carver.gumbysoft.com>
In-Reply-To: <41E5F22A.6010607@uq.edu.au>
References:  <41E44CD0.1000008@uq.edu.au> <41E5F22A.6010607@uq.edu.au>

next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 13 Jan 2005, Matthew Sullivan wrote:

> First if this is the incorrect mailing list for these type of posts
> please let me know or I'll never be able to post to the correct location...

You're in the right spot.

> Further to my last I have finally located a null modem cable and got it
> installed... A little fiddling later and we have DDB taking over...
>
> root@desperado:~# racoon

Hm, null pointer+offset dereference. Are you using IPSEC or FAST_IPSEC in
your kernel?  When did you grab the sources last?

> Fatal trap 12: page fault while in kernel mode
> fault virtual address   = 0x39
> fault code              = supervisor write, page not present
> instruction pointer     = 0x8:0xffffffff80307a70
> stack pointer           = 0x10:0xffffffff94eb4860
> frame pointer           = 0x10:0xffffffff94eb4960
> code segment            = base 0x0, limit 0xfffff, type 0x1b
>                         = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags        = interrupt enabled, resume, IOPL = 0
> current process         = 454 (racoon)
> [thread 100081]
> Stopped at      keydb_newsecasvar+0x100:        decl    %ecx
>
> db> where
> keydb_newsecasvar() at keydb_newsecasvar+0x100
> raw_usend() at raw_usend+0x60
> key_send() at key_send+0xa
> sosend() at sosend+0x626
> kern_sendit() at kern_sendit+0x113
> sendit() at sendit+0x5f
> sendto() at sendto+0x4d
> syscall() at syscall+0x50c
> Xfast_syscall() at Xfast_syscall+0xa8
> --- syscall (133, FreeBSD ELF64, sendto), rip = 0x800a63da8, rsp =
> 0x7fffffffec38, rbp = 0x2 ---
> db> show reg
> cs                 0x8
> ss                0x10
> rax         0xffffffff94eb4890
> rcx         0xffffffff94eb493f
> rdx         0xffffffff94eb4970
> rbx         0xffffffff80307a6d  keydb_newsecasvar+0xfd
> rsp         0xffffffff94eb4860
> rbp         0xffffffff94eb4960
> rsi              0x280
> rdi         0xffffff001cce7c00
> r8                0xa0
> r9          0xffffff00151807b0
> r10         0xffffffff80513980  key_usrreqs
> r11         0xffffffff94eb4a10
> r12               0x39
> r13                  0
> r14                  0
> r15         0xffffff00164aa678
> rip         0xffffffff80307a70  keydb_newsecasvar+0x100
> rflags         0x10202
> dr0                  0
> dr1                  0
> dr2                  0
> dr3                  0
> dr4         0xffff0ff0
> dr5              0x400
> dr6         0xffff0ff0
> dr7              0x400
> keydb_newsecasvar+0x100:        decl    %ecx
> db> show all procs/m
>   pid   proc     uarea   uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
>   454 ffffff001555e5d0 ffffffff94ec0000    0   453   454 0004002 [CPU 0]
> racoon
>   453 ffffff001555e2e8 ffffffff94ebf000    0   443   453 0004002 [SLPQ
> wait 0xffffff001555e2e8][SLP] bash
>   452 ffffff001ccc62e8 ffffffff93cd3000    0     1     1 0004000 [SLPQ
> siodcd 0xffffff000096dc00][SLP] getty
>   451 ffffff001555e8b8 ffffffff94ec1000    0     1   451 0004002 [SLPQ
> ttyin 0xffffff0000956010][SLP] getty
>   450 ffffff001ccc6000 ffffffff93cd2000    0     1   450 0004002 [SLPQ
> ttyin 0xffffff0000956410][SLP] getty
>   449 ffffff001cce32e8 ffffffff93c92000    0     1   449 0004002 [SLPQ
> ttyin 0xffffff000096c010][SLP] getty
>   448 ffffff00152f98b8 ffffffff94e80000    0     1   448 0004002 [SLPQ
> ttyin 0xffffff0000954410][SLP] getty
>   447 ffffff001555eba0 ffffffff94ec2000    0     1   447 0004002 [SLPQ
> ttyin 0xffffff0000954810][SLP] getty
>   446 ffffff00152f92e8 ffffffff94e7e000    0     1   446 0004002 [SLPQ
> ttyin 0xffffff000096d410][SLP] getty
>   445 ffffff00152f95d0 ffffffff94e7f000    0     1   445 0004002 [SLPQ
> ttyin 0xffffff0000971c10][SLP] getty
>   444 ffffff00152f9000 ffffffff94e7d000    0     1   444 0004002 [SLPQ
> ttyin 0xffffff000096c810][SLP] getty
>   443 ffffff001ccc68b8 ffffffff93cd5000    0     1   443 0004102 [SLPQ
> wait 0xffffff001ccc68b8][SLP] login
>   442 ffffff00152f9ba0 ffffffff94e81000    0   441    53 0004002 [SLPQ
> nanslp 0xffffffff8053ba40][SLP] sleep
>   441 ffffff001cce35d0 ffffffff93c93000    0   438    53 0000002 [SLPQ
> wait 0xffffff001cce35d0][SLP] sh
>   439 ffffff001555e000 ffffffff94e82000    0     1    53 0004002 [SLPQ
> piperd 0xffffff001722ab40][SLP] logger
>   438 ffffff001ccc6ba0 ffffffff93cd6000    0     1    53 0000002 [SLPQ
> wait 0xffffff001ccc6ba0][SLP] sh
>   403 ffffff00154a1000 ffffffff94ec3000    0     1   403 0000000 [SLPQ
> nanslp 0xffffffff8053ba40][SLP] cron
>   390 ffffff001cda75d0 ffffffff93c00000   25     1   390 0000100 [SLPQ
> pause 0xffffff001cda7640][SLP] sendmail
>   386 ffffff001cda7ba0 ffffffff93c02000    0     1   386 0000100 [SLPQ
> select 0xffffffff80542030][SLP] sendmail
>   380 ffffff001cce38b8 ffffffff93cd0000    0     1   380 0000100 [SLPQ
> select 0xffffffff80542030][SLP] sshd
>   273 ffffff001cce3ba0 ffffffff93cd1000    0     1   273 0000000 [SLPQ
> select 0xffffffff80542030][SLP] syslogd
>   253 ffffff001cce3000 ffffffff93c91000    0     1   253 0000000 [SLPQ
> select 0xffffffff80542030][SLP] devd
>   181 ffffff001cda78b8 ffffffff93c01000    0     1   181 0000000 [SLPQ
> pause 0xffffff001cda7928][SLP] adjkerntz
>    52 ffffff001ccc65d0 ffffffff93cd4000    0     0     0 0000204 [SLPQ -
> 0xffffffff93cacbe4][SLP] schedcpu
>    51 ffffff001cda2000 ffffffff93bb8000    0     0     0 0000204 [SLPQ
> syncer 0xffffffff8053b720][SLP] syncer
>    50 ffffff001cda22e8 ffffffff93bb9000    0     0     0 0000204 [SLPQ
> vlruwt 0xffffff001cda22e8][SLP] vnlru
>    49 ffffff001cda25d0 ffffffff93bba000    0     0     0 0000204 [SLPQ
> psleep 0xffffffff8054295c][SLP] bufdaemon
>    48 ffffff001cda28b8 ffffffff93bbb000    0     0     0 000020c [SLPQ
> pgzero 0xffffffff80556dd4][SLP] pagezero
>    47 ffffff001cda2ba0 ffffffff93bbc000    0     0     0 0000204 [SLPQ
> psleep 0xffffffff80556e3c][SLP] vmdaemon
>    46 ffffff001cd83000 ffffffff93bbd000    0     0     0 0000204 [SLPQ
> psleep 0xffffffff80556dec][SLP] pagedaemon
>    45 ffffff001cd832e8 ffffffff93bfa000    0     0     0 0000204 [IWAIT]
> swi0: sio
>    44 ffffff001cd835d0 ffffffff93bfb000    0     0     0 0000204 [SLPQ -
> 0xffffff0000811848][SLP] fdc0
>    43 ffffff001cd838b8 ffffffff93bfc000    0     0     0 0000204 [SLPQ
> tzpoll 0xffffffff8052e568][SLP] acpi_thermal
>     9 ffffff001cd83ba0 ffffffff93bfd000    0     0     0 0000204 [SLPQ
> actask 0xffffffff8052e620][SLP] acpi_task2
>     8 ffffff001cda7000 ffffffff93bfe000    0     0     0 0000204 [SLPQ
> actask 0xffffffff8052e620][SLP] acpi_task1
>     7 ffffff001cd92000 ffffffff93b72000    0     0     0 0000204 [SLPQ
> actask 0xffffffff8052e620][SLP] acpi_task0
>    42 ffffff001cd922e8 ffffffff93b73000    0     0     0 0000204 [IWAIT]
> swi6: task queue
>    41 ffffff001cd925d0 ffffffff93b74000    0     0     0 0000204 [IWAIT]
> swi6:+
>     6 ffffff001cd928b8 ffffffff93b75000    0     0     0 0000204 [SLPQ -
> 0xffffff0000835b80][SLP] thread taskq
>    40 ffffff001cd92ba0 ffffffff93b76000    0     0     0 0000204 [IWAIT]
> swi6:+
>     5 ffffff001cdb7000 ffffffff93bb3000    0     0     0 0000204 [SLPQ -
> 0xffffff0000835d00][SLP] kqueue taskq
>    39 ffffff001cdb72e8 ffffffff93bb4000    0     0     0 0000204 [IWAIT]
> swi6: acpitaskq
>    38 ffffff001cdb75d0 ffffffff93bb5000    0     0     0 0000204 [SLPQ -
> 0xffffffff8052eb00][SLP] yarrow
>     4 ffffff001cdb78b8 ffffffff93bb6000    0     0     0 0000204 [SLPQ -
> 0xffffffff80532988][SLP] g_down
>     3 ffffff001cdb7ba0 ffffffff93bb7000    0     0     0 0000204 [SLPQ -
> 0xffffffff80532980][SLP] g_up
>     2 ffffff001cd8e2e8 ffffffff93b2d000    0     0     0 0000204 [SLPQ -
> 0xffffffff80532970][SLP] g_event
>    37 ffffff001cd8e5d0 ffffffff93b2e000    0     0     0 0000204 [IWAIT]
> swi4: vm
>    36 ffffff001cd8e8b8 ffffffff93b2f000    0     0     0 000020c [RUNQ]
> swi5: clock sio
>    35 ffffff001cd8eba0 ffffffff93b30000    0     0     0 0000204 [IWAIT]
> swi1: net
>    34 ffffff001cda5000 ffffffff93b6d000    0     0     0 0000204 [IWAIT]
> irq23:
>    33 ffffff001cda52e8 ffffffff93b6e000    0     0     0 0000204 [IWAIT]
> irq22:
>    32 ffffff001cda55d0 ffffffff93b6f000    0     0     0 0000204 [IWAIT]
> irq21:
>    31 ffffff001cda58b8 ffffffff93b70000    0     0     0 0000204 [IWAIT]
> irq20:
>    30 ffffff001cda5ba0 ffffffff93b71000    0     0     0 0000204 [RUNQ]
> irq19: sis0 sis1
>    29 ffffff001cda38b8 ffffffff93b07000    0     0     0 0000204 [IWAIT]
> irq18:
>    28 ffffff001cda3ba0 ffffffff93b08000    0     0     0 0000204 [IWAIT]
> irq17: atapci1
>    27 ffffff001cd8c000 ffffffff93b09000    0     0     0 0000204 [IWAIT]
> irq16:
>    26 ffffff001cd8c2e8 ffffffff93b28000    0     0     0 0000204 [IWAIT]
> irq15: ata1
>    25 ffffff001cd8c5d0 ffffffff93b29000    0     0     0 0000204 [IWAIT]
> irq14: ata0
>    24 ffffff001cd8c8b8 ffffffff93b2a000    0     0     0 0000204 [IWAIT]
> irq13:
>    23 ffffff001cd8cba0 ffffffff93b2b000    0     0     0 0000204 [IWAIT]
> irq12:
>    22 ffffff001cd8e000 ffffffff93b2c000    0     0     0 0000204 [IWAIT]
> irq11:
>    21 ffffff001cddd2e8 ffffffff93ae2000    0     0     0 0000204 [IWAIT]
> irq10:
>    20 ffffff001cddd5d0 ffffffff93ae3000    0     0     0 0000204 [IWAIT]
> irq9: acpi0
>    19 ffffff001cddd8b8 ffffffff93b02000    0     0     0 0000204 [IWAIT]
> irq8: rtc
>    18 ffffff001cdddba0 ffffffff93b03000    0     0     0 0000204 [IWAIT]
> irq7: ppc0
>    17 ffffff001cda3000 ffffffff93b04000    0     0     0 0000204 [IWAIT]
> irq6: fdc0
>    16 ffffff001cda32e8 ffffffff93b05000    0     0     0 0000204 [IWAIT]
> irq5:
>    15 ffffff001cda35d0 ffffffff93b06000    0     0     0 0000204 [IWAIT]
> irq4: sio0
>    14 ffffff001cdd6000 ffffffff93aa0000    0     0     0 0000204 [IWAIT]
> irq3: sio1
>    13 ffffff001cdd62e8 ffffffff93add000    0     0     0 0000204 [IWAIT]
> irq0: clk
>    12 ffffff001cdd65d0 ffffffff93ade000    0     0     0 0000204 [IWAIT]
> irq1: atkbd0
>    11 ffffff001cdd68b8 ffffffff93adf000    0     0     0 000020c [Can
> run] idle
>     1 ffffff001cdd6ba0 ffffffff93ae0000    0     0     1 0004200 [SLPQ
> wait 0xffffff001cdd6ba0][SLP] init
>    10 ffffff001cddd000 ffffffff93ae1000    0     0     0 0000204 [SLPQ
> ktrace 0xffffffff80538370][SLP] ktrace
>     0 ffffffff80532b00 ffffffff80679000    0     0     0 0000200 [SLPQ
> sched 0xffffffff80532b00][SLP] swapper
>
>
> I'm going to have to put this machine into production within the next 7
> days so any help would be really great, also any extra info anyone
> requires is available.  As I said in my last this is 100% reproducable.
> Dumps are not available - calling panic will lock the system solid.
> Calling boot(0) seems to work fine though...
>
> Regards,
>
>

-- 
Doug White                    |  FreeBSD: The Power to Serve
dwhite@gumbysoft.com          |  www.FreeBSD.org



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050113093955.P12838>