Date: Mon, 23 Jan 2012 10:34:43 GMT From: "Eugene M. Zheganin" <eugene@zhegan.in> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/164402: pf crashes with a particular set of rules when first matching packet arrives Message-ID: <201201231034.q0NAYhQa090227@red.freebsd.org> Resent-Message-ID: <201201231040.q0NAe6Qp054365@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 164402
>Category: kern
>Synopsis: pf crashes with a particular set of rules when first matching packet arrives
>Confidential: no
>Severity: critical
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Mon Jan 23 10:40:06 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator: Eugene M. Zheganin
>Release: 9.0-RELEASE
>Organization:
RealService LLC
>Environment:
FreeBSD taiga 9.0-RELEASE FreeBSD 9.0-RELEASE #1: Mon Jan 23 13:36:16 YEKT 2012 emz@taiga:/usr/obj/usr/src/sys/TAIGA amd64
>Description:
This is a router with numerous ISPs connected.
This router is running pf with a set of rules and some kind of route-to/reply-to rules to make it answer from each ISP source address via corresponding ISP channel.
This router is known to hang quite frequently. It's previous incarnation running on another machine used to hang too (8.2-R, 8.2-S), I was suspecting the hardware, so I moved it here and installed a 9.0. This is an IBM x3560 machine, with all last firmware/BIOS fixes from IBM applied.
Eventually I managed to figure the exact set of rules which makes it to hang.
Set of rules is attached below. This router is running INVARIANTS/WITNESS/stuff kernel, but still it hangs, not traps. Sometimes though, it may eventually show a very large amount of kernel messages at enormous speed, after that it remains unresponsive to keyboard, to DDB entering and stuff until rebooted.
Now I should mention the exact place in the set of rules which makes it hang. This is the place:
===Cut===
1. # http, server
2. # outer world
3. pass in on $oif reply-to ($oif $picgw) proto tcp from !$picnet to $oip port { 80, 443 }
4. #pass in on $oif proto tcp from any to $oip port { 80, 443 } no state
5. #pass out on $asif route-to ($oif $picgw) proto tcp from $oip port { 80, 443 } to any no state
6. #pass out on $oif proto tcp from $oip port { 80, 443 } to any no state
7. # our servers on this link
8. pass in on $oif proto tcp from $picnet to $oip port { 80, 443 }
9. pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port { 80, 443 }
10. pass in on $asif proto tcp from any to $asip port { 80, 443 }
===Cut===
If I comment out line 3, and uncomment lines 4,5,6 (ofc with pfctl -f /etc/pf.myrules) the system will hang on first matching packets. This reproduceable (I tried 6 to 8 times, got the exact behaviour).
By the way, the reason of initiate line no.3 commenting was the fact that running with reply-to rule diminishes the speed from 100Mbit to average 6Kbytes/sec (and this is weird, as you can see there are no queues).
A reference to a screen capture of these messages is attached below:
http://tech.norma.perm.ru/files/pf-panic03.jpeg
This server is equipped with an ipkvm, I can give access to it if needed.
Also I can provide further information if needed/any possible help.
pf set of rules:
[emz@taiga:/etc]# cat pf.taiga
iifs = "{" vlan1 vlan2 vlan5 vlan9 vlan10 vlan12 vlan15 vlan19 "}"
oif = "vlan104"
oip = "89.250.210.67"
oif2 = "vlan818"
oip2 = "86.109.196.3"
oip3 = "86.109.196.5"
asif = "vlan23"
asip = "128.127.144.3"
picgw = "89.250.210.65"
picnet = "89.250.210.64/28"
syngw = "86.109.196.1"
defgw = "128.127.144.5"
localpubwifiifs = "{" vlan11 vlan21 "}"
table <localpubwifinets> { 192.168.8.192/26, 192.168.9.0/26 }
rdpip = "192.168.3.16"
oip6if = "gif0"
oip6ip = "2001:470:1f08:14c0::2"
tunbroker = "216.66.80.26"
iip6if = "vlan22"
vpnpool = "192.168.248.0/26"
hqmbxip = "192.168.3.32"
table <publicwifinets> { 192.168.8.192/26, 192.168.93.32/27, 192.168.93.64/27, 192.168.93.128/27, 192.168.93.96/27, 192.168.9.0/26, 192.168.93.192/27, 192.168.93.224/27, 192.168.93.160/27 }
table <rfc1918> { 192.168.0.0/16, 172.16.0.0/16, 10.0.0.0/8, 224.0.0.0/8, fd00::/16, fe80::16 }
table <moscowrsa> { 94.159.37.117, 94.159.37.118 }
no rdr on $oif proto tcp from 192.168.0.0/16 to <moscowrsa> port { 80, 443 }
rdr on $oif2 proto tcp from !<rfc1918> to $oip3 port 443 -> $hqmbxip port 443
rdr on $oif proto tcp from !<rfc1918> to $oip port 3389 -> $rdpip port 3389
rdr on $asif proto tcp from !<rfc1918> to $asip port 3389 -> $rdpip port 3389
rdr on $iifs proto tcp from 192.168.0.0/16 to !192.168.0.0/16 port { 80, 443 } -> 127.0.0.1 port 3129
no nat on $asif proto gre all
nat on $oif proto { tcp, udp, icmp } from 192.168.0.0/16 to !192.168.0.0/16 -> $oif
nat on $oif2 proto { tcp, udp, icmp } from 192.168.0.0/16 to !192.168.0.0/16 -> $oif2
nat on $asif proto { tcp, udp, icmp } from 192.168.0.0/16 to !192.168.0.0/16 -> $asif
# default blocking
block log all
# localhost
pass quick on lo0 no state
# internal interfaces
pass quick on $iifs no state
# icmp
pass quick proto icmp from any to any
pass quick proto icmp6 from any to any
# AS ospf
pass quick on $asif proto ospf no state
# carp
pass quick proto vrrp from any to 224.0.0.18
pass in quick on carp0 proto { icmp, tcp, udp } from any to any
pass out quick on carp0 proto { icmp, tcp, udp } from any to any
# rdp
pass in on $oif reply-to ($oif $picgw) proto tcp from !<rfc1918> to $rdpip
pass in on $asif proto tcp from !<rfc1918> to $asip port 3389
pass in on $asif proto tcp from !<rfc1918> to $rdpip port 3389
# redirect to exchange, SSL
pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from !<rfc1918> to $oip3 port 443
pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from !<rfc1918> to $hqmbxip port 443
# smtp
# blocking end-client smtp outside
block log on $iifs proto tcp from <rfc1918> to !<rfc1918>
# server
# outer world
pass in on $oif reply-to ($oif $picgw) proto tcp from any to $oip port 25
pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port 25
pass in on $asif proto tcp from any to $asip port 25
# client
pass out on $oif proto tcp from $oip to any port 25
pass out on $oif2 proto tcp from $oip2 to any port 25
pass out on $asif proto tcp from $asip to any port 25
# ssh, server
# outer world
pass in on $oif reply-to ($oif $picgw) proto tcp from !$picnet to $oip port 22
# our servers on this link
pass in on $oif proto tcp from $picnet to $oip port 22
pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port 22
pass in on $asif proto tcp from any to $asip port 22
# ssh, client
pass out on $oif proto tcp from $oip to any port 22
pass out on $oif2 proto tcp from $oip2 to any port 22
pass out on $asif proto tcp from $asip to any port 22
# http, server
# outer world
pass in on $oif reply-to ($oif $picgw) proto tcp from !$picnet to $oip port { 80, 443 }
#pass in on $oif proto tcp from any to $oip port { 80, 443 } no state
#pass out on $asif route-to ($oif $picgw) proto tcp from $oip port { 80, 443 } to any no state
#pass out on $oif proto tcp from $oip port { 80, 443 } to any no state
# our servers on this link
pass in on $oif proto tcp from $picnet to $oip port { 80, 443 }
pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port { 80, 443 }
pass in on $asif proto tcp from any to $asip port { 80, 443 }
# http, client
pass out on $oif proto tcp from $oip to any port { 80, 443 }
pass out on $oif2 proto tcp from $oip2 to any port { 80, 443 }
pass out on $asif proto tcp from $asip to any port { 80, 443 }
# ftp, client
pass out on $oif proto tcp from $oip to any port 21
pass out on $oif2 proto tcp from $oip2 to any port 21
pass out on $asif proto tcp from $asip to any port 21
# ftp, client, data, passive
pass in on $oif proto tcp from any port 20 to $oip
pass in on $oif2 proto tcp from any port 20 to $oip2
pass in on $asif proto tcp from any port 20 to $asip
# ftp, server
# data
pass in on $oif reply-to ($oif $picgw) proto tcp from any to $oip port 21
pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port 21
pass in on $asif proto tcp from any to $asip port 21
# ftp, server, data, passive
pass in on $oif reply-to ($oif $picgw) proto tcp from any to $oip port 49152:65535
pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port 49152:65535
pass in on $asif proto tcp from any to $asip port 49152:65535
# ftp, server, data, active
pass out on $oif route-to ($oif $picgw) proto tcp from $oip port 20 to any
pass out on $oif2 route-to ($oif2 $syngw) proto tcp from $oip port 20 to any
pass out on $asif proto tcp from $oip port 20 to any
# dns, zone transfers, client
pass out on $oif proto tcp from $oip to any port 53
pass out on $oif2 proto tcp from $oip2 to any port 53
pass out on $asif proto tcp from $asip to any port 53
# dns, zone transfers, server
pass in on $oif reply-to ($oif $picgw) proto tcp from any to $oip port 53
pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port 53
pass in on $asif proto tcp from any to $asip port 53
# dns, server
pass in on $oif reply-to ($oif $picgw) proto udp from any to $oip port 53
pass in on $oif2 reply-to ($oif2 $syngw) proto udp from any to $oip2 port 53
pass in on $asif proto udp from any to $asip port 53
# dns, client
pass out on $oif proto udp from $oip to any port 53
pass out on $oif2 proto udp from $oip2 to any port 53
pass out on $asif proto udp from $asip to any port 53
# ntpd
pass out on $oif proto udp from $oip to any port 123
pass out on $oif2 proto udp from $oip2 to any port 123
pass out on $asif proto udp from $asip to any port 123
# mpd, l2tp
pass in on $oif reply-to ($oif $picgw) proto udp from any to $oip port 1701
pass in on $oif2 reply-to ($oif2 $syngw) proto udp from any to $oip2 port 1701
pass in on $asif reply-to ($asif $defgw) proto udp from any to $asip port 1701
# mpd, pptp
# control
pass in on $oif reply-to ($oif $picgw) proto tcp from any to $oip port 1723
pass in on $oif2 reply-to ($oif2 $syngw) proto tcp from any to $oip2 port 1723
pass in on $asif proto tcp from any to $asip port 1723
# gre
pass in on $oif reply-to ($oif $picgw) proto gre from any to $oip no state
pass out on $asif route-to ($oif $picgw) proto gre from $oip to any no state
pass out on $oif proto gre from $oip to any no state
pass in on $oif2 reply-to ($oif2 $syngw) proto gre from any to $oip2
pass out on $asif route-to ($oif2 $syngw) proto gre from $oip2 to any
pass out on $oif2 route-to ($oif2 $syngw) proto gre from $oip2 to any
pass in on $asif proto gre from any to $asip no state
pass out on $asif proto gre from $asip to any no state
# group interfaces
pass on vpn inet proto { icmp, tcp, udp } all no state
# cvsup
pass out on $oif proto tcp from $oip to any port 5999
pass out on $oif2 proto tcp from $oip2 to any port 5999
pass out on $asif proto tcp from $asip to any port 5999
# cvs
pass out on $oif proto tcp from $oip to any port 2401
pass out on $oif2 proto tcp from $oip2 to any port 2401
pass out on $asif proto tcp from $asip to any port 2401
# rdp
pass out on $oif proto tcp from $oip to any port 3389
pass out on $oif2 proto tcp from $oip2 to any port 3389
pass out on $asif proto tcp from $asip to any port 3389
# Mdm
pass out on $oif proto tcp from $oip to 195.161.112.97 port 4434
pass out on $oif2 proto tcp from $oip2 to 195.161.112.97 port 4434
pass out on $asif proto tcp from $asip to 195.161.112.97 port 4434
# unicredit
pass out on $oif proto tcp from $oip to any port 7235
pass out on $oif2 proto tcp from $oip2 to any port 7235
pass out on $asif proto tcp from $asip to any port 7235
# some other bank
pass out on $oif proto tcp from $oip to 212.120.165.203 port 110
pass out on $oif2 proto tcp from $oip2 to 212.120.165.203 port 110
pass out on $asif proto tcp from $asip to 212.120.165.203 port 110
# sber
pass out on $oif proto tcp from $oip to 94.51.87.70 port { 664, 666, 668, 670 }
pass out on $oif2 proto tcp from $oip2 to 94.51.87.70 port { 664, 666, 668, 670 }
pass out on $asif proto tcp from $asip to 94.51.87.70 port { 664, 666, 668, 670 }
pass out on $oif proto tcp from $oip to 194.186.207.124 port 9443
pass out on $oif2 proto tcp from $oip2 to 194.186.207.124 port 9443
pass out on $asif proto tcp from $asip to 194.186.207.124 port 9443
# synterra billing
pass on $oif proto tcp from $oip to 217.115.85.6 port 8443
pass on $oif2 proto tcp from $oip2 to 217.115.85.6 port 8443
pass on $asif proto tcp from $asip to 217.115.85.6 port 8443
# irc
pass out on $oif proto tcp from $oip to any port { 6667, 6669 }
pass out on $oif2 proto tcp from $oip2 to any port { 6667, 6669 }
pass out on $asif proto tcp from $asip to any port { 6667, 6669 }
# public wifi
# local net
pass in on $localpubwifiifs proto { tcp, udp, icmp } from <localpubwifinets> to <localpubwifinets> tag pubwifi
pass out on $localpubwifiifs proto { tcp, udp, icmp } from <localpubwifinets> to <localpubwifinets> tag pubwifi
pass in on $localpubwifiifs proto udp from <localpubwifinets> to <localpubwifinets> port { 67, 68 } tag pubwifi
pass in on $localpubwifiifs proto udp from 0.0.0.0 port 68 to 255.255.255.255 port 67 tag pubwifi
pass in on $localpubwifiifs proto { tcp, udp, icmp } from <localpubwifinets> to !<rfc1918> tag pubwifi
# other networks
pass in on $iifs proto { tcp, udp, icmp } from <pubwifinets> to !<rfc1918> tag pubwifi
pass in on $oif proto { tcp, udp, icmp } from !<rfc1918> to <pubwifinets> tag pubwifi
pass in on $asif proto { tcp, udp, icmp } from !<rfc1918> to <pubwifinets> tag pubwifi
# turn loose the swans
pass out on $oif all tagged pubwifi
pass out on $asif all tagged pubwifi
# razor
pass out on $oif proto tcp from $oip to any port 2703
pass out on $oif2 proto tcp from $oip2 to any port 2703
pass out on $asif proto tcp from $asip to any port 2703
# kassy
pass out on $oif proto tcp from $oip to 212.33.228.45 port 5000
pass out on $oif2 proto tcp from $oip2 to 212.33.228.45 port 5000
pass out on $asif proto tcp from $asip to 212.33.228.45 port 5000
# jabber
pass out on $oif proto tcp from $oip to any port { 5222, 5223 }
pass out on $oif2 proto tcp from $oip2 to any port { 5222, 5223 }
pass out on $asif proto tcp from $asip to any port { 5222, 5223 }
# squid passive ftp
pass out on $oif proto tcp from $oip to any user squid
pass out on $oif2 proto tcp from $oip2 to any user squid
pass out on $asif proto tcp from $asip to any user squid
# moscow rsa
pass out on $oif proto tcp from $oip to <moscowrsa> port { 1044, 1045, 2000 }
pass out on $oif proto udp from $oip to <moscowrsa> port 2000
pass out on $oif2 proto tcp from $oip2 to <moscowrsa> port { 1044, 1045, 2000 }
pass out on $oif2 proto udp from $oip2 to <moscowrsa> port 2000
pass out on $asif proto tcp from $asip to <moscowrsa> port { 1044, 1045, 2000 }
pass out on $asif proto udp from $asip to <moscowrsa> port 2000
# ipv6
# internal interface
pass on $iip6if all no state
# gif traffic
# rule below breaks everything
pass out on $asif route-to ($oif $picgw) proto ipv6 from $oip to $tunbroker no state
pass out on $oif proto ipv6 from $oip to $tunbroker no state
pass in on $oif proto ipv6 from $tunbroker to $oip no state
# icmp
pass on $oip6if inet6 proto icmp6 from any to any no state
## client services
# dns
pass out on $oip6if inet6 proto udp from $oip6ip to any port 53
# http
pass out on $oip6if inet6 proto tcp from $oip6ip to any port { 80, 443 }
# our server
pass in on $oip6if inet6 proto tcp from any to $oip6ip port { 80, 443 }
## server
pass in on $oip6if inet6 proto udp from any to $oip6ip port 53
>How-To-Repeat:
Get a FreeBSD 9.0 server with numerous ISP, build a pf set of rules and a policy-based routing.
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201201231034.q0NAYhQa090227>