From owner-freebsd-pf@FreeBSD.ORG Wed Mar 25 09:26:27 2009 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B15EA1065692 for ; Wed, 25 Mar 2009 09:26:27 +0000 (UTC) (envelope-from sebster@sebster.com) Received: from mail.sebster.com (mail.sebster.com [193.46.80.82]) by mx1.freebsd.org (Postfix) with SMTP id E840C8FC19 for ; Wed, 25 Mar 2009 09:26:26 +0000 (UTC) (envelope-from sebster@sebster.com) Received: (qmail 4420 invoked from network); 25 Mar 2009 08:59:43 -0000 Received: from unknown (HELO ?192.168.1.34?) (sebster@195.240.254.51) by 10.0.98.3 with SMTP; 25 Mar 2009 08:59:43 -0000 Message-ID: <49C9F27F.3010505@sebster.com> Date: Wed, 25 Mar 2009 09:59:43 +0100 From: Sebastiaan van Erk User-Agent: Thunderbird 2.0.0.21 (X11/20090318) MIME-Version: 1.0 To: freebsd-pf@freebsd.org Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms030107090002090101050409" Subject: state mismatch/connection issues X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2009 09:26:28 -0000 This is a cryptographically signed message in MIME format. --------------ms030107090002090101050409 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi, I'm running FreeBSD-7.0 RELEASE with the following patch to the kernel (I know it's integrated in the latest patchlevels which you get when you do freebsd-update, but since I'm still getting state-mismatches WITH the patch I'm holding off on the upgrade until I have more information as to the nature of the problem): *** net/pf.c 2007/09/07 21:34:10 1.558 --- net/pf.c 2007/09/18 19:45:59 1.559 *************** pf_test_state_tcp(struct pf_state **state, int directi *** 3730,3735 **** --- 3730,3751 ---- REASON_SET(reason, PFRES_SYNPROXY); return (PF_SYNPROXY_DROP); } + } + + if (((th->th_flags & (TH_SYN|TH_ACK)) == TH_SYN) && + dst->state >= TCPS_FIN_WAIT_2 && + src->state >= TCPS_FIN_WAIT_2) { + if (pf_status.debug >= PF_DEBUG_MISC) { + printf("pf: state reuse "); + pf_print_state(*state); + pf_print_flags(th->th_flags); + printf("\n"); + } + /* XXX make sure it's the same direction ?? */ + (*state)->src.state = (*state)->dst.state = TCPS_CLOSED; + pf_unlink_state(*state); + *state = NULL; + return (PF_DROP); } if (src->wscale && dst->wscale && !(th->th_flags & TH_SYN)) { The problem I'm having is that I get intermittent connection refused/operation not permitted to another machine on the local network. When I do pfctl -s info I see *huge* numbers of state mismatches: Status: Enabled for 94 days 01:27:40 Debug: Urgent State Table Total Rate current entries 398 searches 986228319 121.4/s inserts 104049508 12.8/s removals 104049110 12.8/s Counters match 107482262 13.2/s bad-offset 0 0.0/s fragment 0 0.0/s short 0 0.0/s normalize 42 0.0/s memory 3125235 0.4/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 0 0.0/s proto-cksum 13919 0.0/s state-mismatch 3039814 0.4/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s This is causing serious problems at them moment. It seems that the state problems occur in certain small time windows (my nagios starts reporting that every service is connection refused/operation not permitted, which is about 20 services). Then I get 20 recovery messages. The firewall rules are trivially simple, $ext_if has 2 ips and $int_if has one: interfaces = "{" $ext_if "," $int_if "}" scrub in all set skip on lo0 antispoof for $interfaces inet block out log quick on $ext_if from !$ext_ip1 to any block in quick on $ext_if from any to 255.255.255.255 block log all pass in quick inet proto icmp all icmp-type $icmp_types pass in quick on $int_if from $int_net to any pass out quick on $int_if from any to $int_net pass out on $ext_if proto tcp all pass out on $ext_if proto { udp, icmp } all pass in on $ext_if proto tcp from any to $ext_ip1 port $tcp_services1 pass in on $ext_if proto tcp from any to $ext_ip2 port $tcp_services2 Does anybody have any idea what's going on and where I can look? This is a production server so it's seriously influencing the quality of the hosted services. :-( Regards, Sebastiaan --------------ms030107090002090101050409 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJUTCC AwMwggJsoAMCAQICEFN8DarMNuuKJDEtfs0UaqUwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UE BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMT I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA4MDYzMDEzNTE1N1oX DTA5MDYzMDEzNTE1N1owaDEQMA4GA1UEBBMHdmFuIEVyazETMBEGA1UEKhMKU2ViYXN0aWFh bjEbMBkGA1UEAxMSU2ViYXN0aWFhbiB2YW4gRXJrMSIwIAYJKoZIhvcNAQkBFhNzZWJzdGVy QHNlYnN0ZXIuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsJDDAeYHVmH/ GVxi+bhFx27dmg++9BdhPJfk8k041sqEqq7oXnR2GT54quY3Ac7A1BuOM2JvoICraGmjud4y b3EanRnqGIK6iH+VAhhTlV/Owrb2Qm1e13DLxwLp1SocSQl4IrEbF9Y5H3ASdIrE0iFqkpju nPiiHeNhz3LaI5ipjiluKYoH+F6gPx8njHoaDxPePCkSLg4r0IA0afLM74LVZxCRBZEfyRZS J6VVUJefKlz91dWSzR/3xSw/rO4u9Ds/Zh7VBUKy3K+YFryHxRpUek0gSepE1b70Q39L9Sqd M/NZqMvFpwrqgW2Zh2Nh8nqRge90maR4ypBzz3GzLwIDAQABozAwLjAeBgNVHREEFzAVgRNz ZWJzdGVyQHNlYnN0ZXIuY29tMAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEFBQADgYEAS1Sk NMgDVzb0ktO9tPPacV0KdKhTYOHcICVmuDEe2sFHOkjLAI1iAKp640pqJEVqvRnfRcCFJ9hK koPjjVZ+ui2rVmJWBG6FSloLRS/YYED4tUAw6DQhK61UOpjkpQxjCdm+5bHG/2ZgJAda1j0x uiN822+xFkcaW/5PQgxSRxcwggMDMIICbKADAgECAhBTfA2qzDbriiQxLX7NFGqlMA0GCSqG SIb3DQEBBQUAMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAo UHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBD QTAeFw0wODA2MzAxMzUxNTdaFw0wOTA2MzAxMzUxNTdaMGgxEDAOBgNVBAQTB3ZhbiBFcmsx EzARBgNVBCoTClNlYmFzdGlhYW4xGzAZBgNVBAMTElNlYmFzdGlhYW4gdmFuIEVyazEiMCAG CSqGSIb3DQEJARYTc2Vic3RlckBzZWJzdGVyLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEP ADCCAQoCggEBALCQwwHmB1Zh/xlcYvm4Rcdu3ZoPvvQXYTyX5PJNONbKhKqu6F50dhk+eKrm NwHOwNQbjjNib6CAq2hpo7neMm9xGp0Z6hiCuoh/lQIYU5VfzsK29kJtXtdwy8cC6dUqHEkJ eCKxGxfWOR9wEnSKxNIhapKY7pz4oh3jYc9y2iOYqY4pbimKB/heoD8fJ4x6Gg8T3jwpEi4O K9CANGnyzO+C1WcQkQWRH8kWUielVVCXnypc/dXVks0f98UsP6zuLvQ7P2Ye1QVCstyvmBa8 h8UaVHpNIEnqRNW+9EN/S/UqnTPzWajLxacK6oFtmYdjYfJ6kYHvdJmkeMqQc89xsy8CAwEA AaMwMC4wHgYDVR0RBBcwFYETc2Vic3RlckBzZWJzdGVyLmNvbTAMBgNVHRMBAf8EAjAAMA0G CSqGSIb3DQEBBQUAA4GBAEtUpDTIA1c29JLTvbTz2nFdCnSoU2Dh3CAlZrgxHtrBRzpIywCN YgCqeuNKaiRFar0Z30XAhSfYSpKD441Wfrotq1ZiVgRuhUpaC0Uv2GBA+LVAMOg0ISutVDqY 5KUMYwnZvuWxxv9mYCQHWtY9MbojfNtvsRZHGlv+T0IMUkcXMIIDPzCCAqigAwIBAgIBDTAN BgkqhkiG9w0BAQUFADCB0TELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTES MBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3dGUgQ29uc3VsdGluZzEoMCYGA1UE CxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEkMCIGA1UEAxMbVGhhd3RlIFBl cnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkBFhxwZXJzb25hbC1mcmVlbWFpbEB0 aGF3dGUuY29tMB4XDTAzMDcxNzAwMDAwMFoXDTEzMDcxNjIzNTk1OVowYjELMAkGA1UEBhMC WkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1Ro YXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMIGfMA0GCSqGSIb3DQEBAQUAA4GN ADCBiQKBgQDEpjxVc1X7TrnKmVoeaMB1BHCd3+n/ox7svc31W/Iadr1/DDph8r9RzgHU5VAK MNcCY1osiRVwjt3J8CuFWqo/cVbLrzwLB+fxH5E2JCoTzyvV84J3PQO+K/67GD4Hv0CAAmTX p6a7n2XRxSpUhQ9IBH+nttE8YQRAHmQZcmC3+wIDAQABo4GUMIGRMBIGA1UdEwEB/wQIMAYB Af8CAQAwQwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVBl cnNvbmFsRnJlZW1haWxDQS5jcmwwCwYDVR0PBAQDAgEGMCkGA1UdEQQiMCCkHjAcMRowGAYD VQQDExFQcml2YXRlTGFiZWwyLTEzODANBgkqhkiG9w0BAQUFAAOBgQBIjNFQg+oLLswNo2as Zw9/r6y+whehQ5aUnX9MIbj4Nh+qLZ82L8D0HFAgk3A8/a3hYWLD2ToZfoSxmRsAxRoLgnSe JVCUYsfbJ3FXJY3dqZw5jowgT2Vfldr394fWxghOrvbqNOUQGls1TXfjViF4gtwhGTXeJLHT HUb/XV9lTzGCA3EwggNtAgEBMHYwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBD b25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFp bCBJc3N1aW5nIENBAhBTfA2qzDbriiQxLX7NFGqlMAkGBSsOAwIaBQCgggHQMBgGCSqGSIb3 DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA5MDMyNTA4NTk0M1owIwYJKoZI hvcNAQkEMRYEFGsiiFdvEXGmJFJ5WvjOttxTZqRDMF8GCSqGSIb3DQEJDzFSMFAwCwYJYIZI AWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUr DgMCBzANBggqhkiG9w0DAgIBKDCBhQYJKwYBBAGCNxAEMXgwdjBiMQswCQYDVQQGEwJaQTEl MCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3Rl IFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEFN8DarMNuuKJDEtfs0UaqUwgYcGCyqG SIb3DQEJEAILMXigdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRp bmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3Vp bmcgQ0ECEFN8DarMNuuKJDEtfs0UaqUwDQYJKoZIhvcNAQEBBQAEggEAILsayitdUh9jm+tv hc0UqLlT9lm5q8wydEZeDatDlTbOeC6mypqbVVVjj4CgSkvxgRopDzj3GTmNHcTvSOdms1Na Q+zV7ILniwkscSM3S4aS80kiznA3z4QM50KdSlwQMRcP9xCXhQ5SinvTlFrD0KzfrwzMjP99 RcMJOm+dbAhrIEeCf37/YkVgIEvBN6O3sidzmji0MGkIEbCSxCqefoavu0fIs1/uKkWQt7zw wfC9GyYaxuDf7O/L62cPlom6Uc7BEzoysxA2Q6RwbZL4/uvOR+00Wmy9tCq99l8YvWP3T6Hm 37A/0mnNPmaRtYWgpfPz3Fxt/cltzsX61/2CpgAAAAAAAA== --------------ms030107090002090101050409--