From owner-freebsd-bugs@FreeBSD.ORG Mon Jan 23 08:10:15 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1780D106566B for ; Mon, 23 Jan 2012 08:10:15 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id E6B038FC12 for ; Mon, 23 Jan 2012 08:10:14 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q0N8AEcv010197 for ; Mon, 23 Jan 2012 08:10:14 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q0N8AEHo010191; Mon, 23 Jan 2012 08:10:14 GMT (envelope-from gnats) Resent-Date: Mon, 23 Jan 2012 08:10:14 GMT Resent-Message-Id: <201201230810.q0N8AEHo010191@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, "Eugene M. Zheganin" Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E746A106566B for ; Mon, 23 Jan 2012 08:08:36 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id D4ED98FC08 for ; Mon, 23 Jan 2012 08:08:36 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q0N88aMW021034 for ; Mon, 23 Jan 2012 08:08:36 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id q0N88a0I021033; Mon, 23 Jan 2012 08:08:36 GMT (envelope-from nobody) Message-Id: <201201230808.q0N88a0I021033@red.freebsd.org> Date: Mon, 23 Jan 2012 08:08:36 GMT From: "Eugene M. Zheganin" To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/164400: immediate crash after the start of ipsec processing X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jan 2012 08:10:15 -0000 >Number: 164400 >Category: kern >Synopsis: immediate crash after the start of ipsec processing >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Jan 23 08:10:13 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Eugene M. Zheganin >Release: 9.0-RELEASE >Organization: RealService LLC >Environment: FreeBSD 9.0-RELEASE FreeBSD 9.0-RELEASE #1: Sun Jan 22 21:59:51 MSK 2012 emz@moscow-alpha:/usr/obj/usr/src/sys/MOSCOW amd64 >Description: There's a HA-cluster of 2 freebsd running ipsec+gre and a butch of tunnels to branch offices with OSPF. Both were running 8.2-RELEASE and/or different versions of 8.2-STABLE. Since this setup wasn't that stable I was constantly upgrading one of the nodes to a recent -STABLE. After a December -STABLE first node was crashing every hour, so I decided to test it with the memtest86+ (4.20). One week of running memtest gave no errors. So I upgraded to 9.0 and built a debug kernel with WITNESS/INVARIANTS and stuff. OSPF is served by quagga. ISAKMP is served by security/ipsec-tools. Now I have an immediate crash after a carp(4) with public address for gre tunnels and ipsec processing is switching to MASTER. Without it it runs fine. The crash is reproduceable (at least I tested 3 times and got 3 craches). BTs are identical ( I compared each screen first and last line, so only first set of screens is referenced here). This host has an ipkvm (and YES, I can give access to it if needed, it lives on a public address, you will need a working java browser plugin to use it), so here are the screens of this trap happening: http://tech.norma.perm.ru/files/screen01.jpeg http://tech.norma.perm.ru/files/screen02.jpeg http://tech.norma.perm.ru/files/screen03.jpeg http://tech.norma.perm.ru/files/screen04.jpeg http://tech.norma.perm.ru/files/screen05.jpeg Furthermore, when running with WITNESS on this node produces the following output immediately after loading a set of pf rules: http://tech.hq.norma.perm.ru/files/lor.txt This output is always the same too. ipsec.conf: [emz@:~]> cat /etc/ipsec.conf spdflush; # # Moscow, Schelkovskoye, Megaton # spdadd 94.159.37.114 89.250.210.69 gre -P out ipsec esp/transport/94.159.37.114-89.250.210.69/require ah/transport/94.159.37.114-89.250.210.69/require; spdadd 89.250.210.69 94.159.37.114 gre -P in ipsec esp/transport/89.250.210.69-94.159.37.114/require ah/transport/89.250.210.69-94.159.37.114/require; # # Moscow, Pervomayskaya, MGTS # spdadd 109.252.242.9 94.159.37.114 gre -P in ipsec esp/transport/109.252.242.9-94.159.37.114/require ah/transport/109.252.242.9-94.159.37.114/require; spdadd 94.159.37.114 109.252.242.9 gre -P out ipsec esp/transport/94.159.37.114-109.252.242.9/require ah/transport/94.159.37.114-109.252.242.9/require; # # Moscow, Privolnaya, 70 # spdadd 82.142.171.58 94.159.37.114 gre -P in ipsec esp/transport/82.142.171.58-94.159.37.114/require ah/transport/82.142.171.58-94.159.37.114/require; spdadd 94.159.37.114 82.142.171.58 gre -P out ipsec esp/transport/94.159.37.114-82.142.171.58/require ah/transport/94.159.37.114-82.142.171.58/require; # # Moscow, Lermontovsky, 2 # spdadd 79.120.78.118 94.159.37.114 gre -P in ipsec esp/transport/79.120.78.118-94.159.37.114/require ah/transport/79.120.78.118-94.159.37.114/require; spdadd 94.159.37.114 79.120.78.118 gre -P out ipsec esp/transport/94.159.37.114-79.120.78.118/require ah/transport/94.159.37.114-79.120.78.118/require; # # Moscow, Tashkentskaya 12-20 # spdadd 79.120.80.66 94.159.37.114 gre -P in ipsec esp/transport/79.120.80.66-94.159.37.114/require ah/transport/79.120.80.66-94.159.37.114/require; spdadd 94.159.37.114 79.120.80.66 gre -P out ipsec esp/transport/94.159.37.114-79.120.80.66/require ah/transport/94.159.37.114-79.120.80.66/require; # # Moscow, Baykalskaya 50/7 # spdadd 213.33.223.158 94.159.37.114 gre -P in ipsec esp/transport/213.33.223.158-94.159.37.114/require ah/transport/213.33.223.158-94.159.37.114/require; spdadd 94.159.37.114 213.33.223.158 gre -P out ipsec esp/transport/94.159.37.114-213.33.223.158/require ah/transport/94.159.37.114-213.33.223.158/require; # # Moscow, Bratyslavskaya 15-1 # #spdadd 212.46.203.106 94.159.37.114 gre -P in ipsec esp/transport/212.46.203.106-94.159.37.114/require ah/transport/212.46.203.106-94.159.37.114/require; #spdadd 94.159.37.114 212.46.203.106 gre -P out ipsec esp/transport/94.159.37.114-212.46.203.106/require ah/transport/94.159.37.114-212.46.203.106/require; #add 212.46.203.106 94.159.37.114 esp 0x10001 -m transport -E des-cbc 0xffffffffffffffff; #add 94.159.37.114 212.46.203.106 esp 0x10002 -m transport -E des-cbc 0xffffffffffffffff; #add 212.46.203.106 94.159.37.114 ah 0x10003 -m transport -A keyed-md5 "xxxxxxxxxxxxxxxx"; #add 94.159.37.114 212.46.203.106 ah 0x10004 -m transport -A keyed-md5 "xxxxxxxxxxxxxxxx"; racoon.conf: [emz@:~]# cat /usr/local/etc/racoon/racoon.conf path pre_shared_key "/usr/local/etc//racoon/psk.txt"; # "padding" defines some padding parameters. You should not touch these. padding { maximum_length 20;<># maximum padding length. randomize off; # enable randomize length. strict_check off; # enable strict check. exclusive_tail off; # extract last one octet. } # if no listen directive is specified, racoon will listen on all # available interface addresses. listen { isakmp 94.159.37.114 [500]; strict_address; # requires that all addresses must be bound. } # Specify various default timers. timer { # These value can be changed per remote node. counter 5; # maximum trying count to send. interval 20 sec; # maximum interval to resend. persend 1; # the number of packets per send. # maximum time to wait for completing each phase. phase1 30 sec; phase2 15 sec; } # # Kosm65, wizard, PiC # remote 89.250.210.69 { exchange_mode main; lifetime time 1 hour; my_identifier address 94.159.37.114; peers_identifier address 89.250.210.69; passive off; proposal_check obey; dpd_delay 20; proposal { encryption_algorithm des; hash_algorithm md5; authentication_method pre_shared_key; dh_group modp768; } } # # Moscow, Permovayskaya10/5, MGTS # remote 109.252.242.9 { exchange_mode main; lifetime time 1 hour; my_identifier address 94.159.37.114; peers_identifier address 109.252.242.9; passive off; proposal_check obey; dpd_delay 20; proposal { encryption_algorithm des; hash_algorithm md5; authentication_method pre_shared_key; dh_group modp768; } } # # Moscow, Privolnaya, 70 # remote 82.142.171.58 { exchange_mode main; lifetime time 1 hour; my_identifier address 94.159.37.114; peers_identifier address 82.142.171.58; passive off; proposal_check obey; dpd_delay 20; proposal { encryption_algorithm des; hash_algorithm md5; authentication_method pre_shared_key; dh_group modp768; } } # # Moscow, Lermontovsky, 2 # remote 79.120.78.118 { exchange_mode main; lifetime time 1 hour; my_identifier address 94.159.37.114; peers_identifier address 79.120.78.118; passive off; proposal_check obey; dpd_delay 20; proposal { encryption_algorithm des; hash_algorithm md5; authentication_method pre_shared_key; dh_group modp768; } } # # Moscow, Baykalskaya 50/7 # remote 213.33.223.158 { exchange_mode main; lifetime time 1 hour; my_identifier address 94.159.37.114; peers_identifier address 213.33.223.158; passive off; proposal_check obey; dpd_delay 20; proposal { encryption_algorithm des; hash_algorithm md5; authentication_method pre_shared_key; dh_group modp768; } } # # Moscow, Tashkentskaya 12-20 # remote 79.120.80.66 { exchange_mode main; lifetime time 1 hour; my_identifier address 94.159.37.114; peers_identifier address 79.120.80.66; passive off; proposal_check obey; dpd_delay 20; proposal { encryption_algorithm des; hash_algorithm md5; authentication_method pre_shared_key; dh_group modp768; } } # # Moscow, Bratyslavskaya 15-1 # #remote 212.46.203.106 { # exchange_mode main; # lifetime time 1 hour; # my_identifier address 94.159.37.114; # peers_identifier address 212.46.203.106; # passive off; # proposal_check obey; # dpd_delay 20; # proposal { # encryption_algorithm des; # hash_algorithm md5; # authentication_method pre_shared_key; # dh_group modp768; # } #} # # kosm65, wizard, PiC # sainfo address 89.250.210.69 [500] 47 address 94.159.37.114 [500] 47 { pfs_group modp768; lifetime time 60 min; encryption_algorithm blowfish; authentication_algorithm hmac_md5; compression_algorithm deflate; } sainfo address 94.159.37.114 [500] 47 address 89.250.210.69 [500] 47 { pfs_group modp768; lifetime time 60 min; encryption_algorithm blowfish; authentication_algorithm hmac_md5; compression_algorithm deflate; } # # Moscow, Pervomayskaya, 10/5, MGTS # sainfo address 94.159.37.114 [500] 47 address 109.252.242.9 [500] 47 { pfs_group modp768; lifetime time 60 min; encryption_algorithm des; authentication_algorithm hmac_sha1, non_auth; compression_algorithm deflate; } sainfo address 109.252.242.9 [500] 47 address 94.159.37.114 [500] 47 { pfs_group modp768; lifetime time 60 min; encryption_algorithm des; authentication_algorithm hmac_sha1, non_auth; compression_algorithm deflate; } # # Moscow, Privolnaya, 70 # sainfo address 94.159.37.114 [500] 47 address 82.142.171.58 [500] 47 { pfs_group modp768; lifetime time 60 min; encryption_algorithm des; authentication_algorithm hmac_sha1, non_auth; compression_algorithm deflate; } sainfo address 82.142.171.58 [500] 47 address 94.159.37.114 [500] 47 { pfs_group modp768; lifetime time 60 min; encryption_algorithm des; authentication_algorithm hmac_sha1, non_auth; compression_algorithm deflate; } # # Moscow, Lermontovsky, 2 # sainfo address 94.159.37.114 [500] 47 address 79.120.78.118 [500] 47 { pfs_group modp768; lifetime time 60 min; encryption_algorithm des; authentication_algorithm hmac_sha1, non_auth; compression_algorithm deflate; } sainfo address 79.120.78.118 [500] 47 address 94.159.37.114 [500] 47 { pfs_group modp768; lifetime time 60 min; encryption_algorithm des; authentication_algorithm hmac_sha1, non_auth; compression_algorithm deflate; } # # Moscow, Tashkentskaya 12-20 # sainfo address 94.159.37.114 [500] 47 address 79.120.80.66 [500] 47 { pfs_group modp768; lifetime time 60 min; encryption_algorithm des; authentication_algorithm hmac_sha1, non_auth; compression_algorithm deflate; } sainfo address 79.120.80.66 [500] 47 address 94.159.37.114 [500] 47 { pfs_group modp768; lifetime time 60 min; encryption_algorithm des; authentication_algorithm hmac_sha1, non_auth; compression_algorithm deflate; } # # Moscow, Baykalskaya 50/7 # sainfo address 94.159.37.114 [500] 47 address 213.33.223.158 [500] 47 { pfs_group modp768; lifetime time 60 min; encryption_algorithm des; authentication_algorithm hmac_sha1, non_auth; compression_algorithm deflate; } sainfo address 213.33.223.158 [500] 47 address 94.159.37.114 [500] 47 { pfs_group modp768; lifetime time 60 min; encryption_algorithm des; authentication_algorithm hmac_sha1, non_auth; compression_algorithm deflate; } # # Moscow, Bratyslavskaya 15-1 # #sainfo address 94.159.37.114 [500] 47 address 212.46.203.106 [500] 47 { # pfs_group modp768; # lifetime time 60 min; # encryption_algorithm des; # authentication_algorithm hmac_sha1, non_auth; # compression_algorithm deflate; #} #sainfo address 212.46.203.106 [500] 47 address 94.159.37.114 [500] 47 { # pfs_group modp768; # lifetime time 60 min; # encryption_algorithm des; # authentication_algorithm hmac_sha1, non_auth; # compression_algorithm deflate; #} >How-To-Repeat: Get a FreeBSD 9.0, get an ipsec setup, get a bunch of gre tunnels, get a quagga (trap mentions its ospfd), and probably get a carp(4) interface (not sure if it will trap without; at least I did not test it without a carp(4)). >Fix: >Release-Note: >Audit-Trail: >Unformatted: