From owner-freebsd-ipfw@FreeBSD.ORG Sun Mar 25 16:37:33 2007 Return-Path: X-Original-To: ipfw@freebsd.org Delivered-To: freebsd-ipfw@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2021816A401 for ; Sun, 25 Mar 2007 16:37:33 +0000 (UTC) (envelope-from infofarmer@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.174]) by mx1.freebsd.org (Postfix) with ESMTP id ABD1B13C45B for ; Sun, 25 Mar 2007 16:37:32 +0000 (UTC) (envelope-from infofarmer@gmail.com) Received: by ug-out-1314.google.com with SMTP id 71so1395131ugh for ; Sun, 25 Mar 2007 09:37:31 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition:x-google-sender-auth; b=p3mezXL2rKkMiA4T8HKPDK96LAHscaJCPZmjDfFv8J1WTNHsKQGNoDh1sYPLWIgmxqTDhvbtTEy6DqqfRGAZSfD30am0YQ4OAEzbJQ73wHrkZdDZQ5yoO4ecmrih8eLqbNJ+xJ8Wii2TAI/wBFEL2+/ueNlSTiviVqGM5BgK1Ag= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition:x-google-sender-auth; b=AXtcEeXXnl2aX5C0iZpXM3qRcRjtMi97SZ+t683b3GaW2XMzH3B+sM9Uf5x7sP1zliIBCs6mSm8s6g6TyACIDyL2gpa6WRaVvrNUvzvFRrsqmKF9r6jjFxlBMOz3bjmEhsUZE72dic7pc2KPy/9p1298RuiDw6HivLU3CewnsI0= Received: by 10.114.153.18 with SMTP id a18mr2199308wae.1174839152777; Sun, 25 Mar 2007 09:12:32 -0700 (PDT) Received: by 10.114.201.2 with HTTP; Sun, 25 Mar 2007 09:12:32 -0700 (PDT) Message-ID: Date: Sun, 25 Mar 2007 19:12:32 +0300 From: "Andrew Pantyukhin" Sender: infofarmer@gmail.com To: ipfw@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Google-Sender-Auth: 0d8be0eb1fb3401a Cc: Subject: dynamic queues replace each other randomly X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Mar 2007 16:37:33 -0000 I have a 6.2-release/i386 gateway with ipfw and dummynet compiled into kernel. pipe 107 config bw 2M queue 207 config pipe 107 weight 50 mask dst-ip 0xffffffff add queue 207 The problem is this. When I look at "ipfw queue 207 show" it all seems almost normal. When new dst-ip's are noticed, new dynamic queues appear. But one bucket is "shared" between two flow's. Whichever flow (dst-ip) received packets last shows up in the table. Every time the bucket changes between the flows, its counters are reset to 0. If the glitch is in the numbers, here they are: 50 slots 9 queues (from 5 to 20 over time) 64 buckets "shared" bucket no. 20 the two dst-ip are 192.168.17.68 and 172.17.0.54 Now that I look at the table, I also see another strange thing. Right after the bucket 20 line come two lines, both numbered as bucket 21. At least they both show up every time. Their ip's are 192.168.17.69 and 172.17.0.55. Any ideas are welcome. Thanks! From owner-freebsd-ipfw@FreeBSD.ORG Mon Mar 26 06:39:01 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D5B6D16A402 for ; Mon, 26 Mar 2007 06:39:01 +0000 (UTC) (envelope-from arjunbadarinath@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.237]) by mx1.freebsd.org (Postfix) with ESMTP id 972C713C46A for ; Mon, 26 Mar 2007 06:39:01 +0000 (UTC) (envelope-from arjunbadarinath@gmail.com) Received: by nz-out-0506.google.com with SMTP id r28so1558097nza for ; Sun, 25 Mar 2007 23:39:01 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; b=gwJZfjN4YKPR3yhNns97EoJr9Y3rl/ZTGCghxK48ZIeIG6ER8ZlTAw8BOFHc8Bi6IyYOjszbKtURo6lv9t4/2ERtlLYkr9AbG9lvWOvZ5niJxkSbf2yU5ThtYMmjFqCMwKOxA+8ucYHQWpLT00x/efEX3bK6I8BEBcD8JID6E6c= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type; b=iHRa9SCbbP5prtPQF9WnGWxZD0tzIqbvyqYkQmECae/ngJEqnGARNDLMJLbRELW9JAFxdbnhKp3qV7UoNQbLHzUcyJu06pOKUE9vbkT1PL+YMhGYaOPUAcXJEuJ7TYP/xw7KqWgz+Yh+doqcDvxVu/cquC46wTVgpniCZSQT+v0= Received: by 10.65.254.5 with SMTP id g5mr12411437qbs.1174891140474; Sun, 25 Mar 2007 23:39:00 -0700 (PDT) Received: by 10.65.200.3 with HTTP; Sun, 25 Mar 2007 23:39:00 -0700 (PDT) Message-ID: <4cc04d3d0703252339s6f7e7fackb06cbc6184862556@mail.gmail.com> Date: Mon, 26 Mar 2007 12:09:00 +0530 From: "arjun badarinath" To: freebsd-ipfw@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: System calls X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2007 06:39:01 -0000 Hi all, I wanted to know wat these system calls actually do . ip_dn_ctl_ptr ip_dn_io_ptr ip_dn_ruledel_ptr Regards Arjun From owner-freebsd-ipfw@FreeBSD.ORG Mon Mar 26 07:27:42 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2690316A404 for ; Mon, 26 Mar 2007 07:27:42 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from smtp2.yandex.ru (smtp2.yandex.ru [213.180.200.18]) by mx1.freebsd.org (Postfix) with ESMTP id 5B74513C46A for ; Mon, 26 Mar 2007 07:27:41 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from ns.kirov.so-cdu.ru ([87.226.153.33]:64266 "EHLO [127.0.0.1]" smtp-auth: "bu7cher" TLS-CIPHER: "DHE-RSA-AES256-SHA keybits 256/256 version TLSv1/SSLv3" TLS-PEER-CN1: ) by mail.yandex.ru with ESMTP id S3375626AbXCZH11 (ORCPT ); Mon, 26 Mar 2007 11:27:27 +0400 X-Comment: RFC 2476 MSA function at smtp2.yandex.ru logged sender identity as: bu7cher Message-ID: <460775DC.2090707@yandex.ru> Date: Mon, 26 Mar 2007 11:27:24 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla Thunderbird 1.5 (FreeBSD/20051231) MIME-Version: 1.0 To: arjun badarinath References: <4cc04d3d0703252339s6f7e7fackb06cbc6184862556@mail.gmail.com> In-Reply-To: <4cc04d3d0703252339s6f7e7fackb06cbc6184862556@mail.gmail.com> Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-ipfw@freebsd.org Subject: Re: System calls X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2007 07:27:42 -0000 arjun badarinath ÐÉÛÅÔ: > Hi all, > I wanted to know wat these system calls actually do . > ip_dn_ctl_ptr > ip_dn_io_ptr > ip_dn_ruledel_ptr It's not a system calls. It's a pointers for the interaction with dummynet. -- WBR, Andrey V. Elsukov From owner-freebsd-ipfw@FreeBSD.ORG Mon Mar 26 09:15:22 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 97BA416A400 for ; Mon, 26 Mar 2007 09:15:22 +0000 (UTC) (envelope-from arjunbadarinath@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.238]) by mx1.freebsd.org (Postfix) with ESMTP id 2D80F13C483 for ; Mon, 26 Mar 2007 09:15:22 +0000 (UTC) (envelope-from arjunbadarinath@gmail.com) Received: by nz-out-0506.google.com with SMTP id r28so1588269nza for ; Mon, 26 Mar 2007 02:15:21 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; b=PrclSj3kQp7tWu4vluuEtyitChNYbiDfEiRf+ttVynT7UXJBS8h95Ge5vq6cgdoD7Zf5itsrV/Cx2wqn3+s3+xSFGd2aPrcyPEnmJHBPZD30spf/2MkImc4rfjVQlfCb1mYOpXfysSqrYrxkSO/cHXRPZakCZDrZK3FI+b6x7DM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type; b=YHyI5YhGJm7ZiSKq9g+hNg/AFlhuRb5m4mZcgYTqiGwZgomowwf9pFdhwXzjQu+c2b5KLVra7enkoYjJwTbT0ytIPK5hGLz/BiyVbbwvBw7mXZR+Jezv3cFpdwhJoA/1FWhQ7Uz/edqg+kI2xOLatBw051Dt4Ry+J8nv8avjySs= Received: by 10.65.251.2 with SMTP id d2mr12593209qbs.1174900521431; Mon, 26 Mar 2007 02:15:21 -0700 (PDT) Received: by 10.65.200.3 with HTTP; Mon, 26 Mar 2007 02:15:21 -0700 (PDT) Message-ID: <4cc04d3d0703260215k5950dc6bl6bde6eb38af0ee@mail.gmail.com> Date: Mon, 26 Mar 2007 14:45:21 +0530 From: "arjun badarinath" To: freebsd-ipfw@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Packet filter X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2007 09:15:22 -0000 Hi all , I wanted to know how the paket filter (pf) is implemented on bsd . can u please send me the link of the source code correponding to this Regards Arjun From owner-freebsd-ipfw@FreeBSD.ORG Mon Mar 26 09:29:00 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 88EB516A404 for ; Mon, 26 Mar 2007 09:29:00 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from smtp3.yandex.ru (smtp3.yandex.ru [213.180.200.14]) by mx1.freebsd.org (Postfix) with ESMTP id BE0E513C45D for ; Mon, 26 Mar 2007 09:28:59 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from ns.kirov.so-cdu.ru ([87.226.153.33]:39180 "EHLO [127.0.0.1]" smtp-auth: "bu7cher" TLS-CIPHER: "DHE-RSA-AES256-SHA keybits 256/256 version TLSv1/SSLv3" TLS-PEER-CN1: ) by mail.yandex.ru with ESMTP id S3588274AbXCZJ2v (ORCPT ); Mon, 26 Mar 2007 13:28:51 +0400 X-Comment: RFC 2476 MSA function at smtp3.yandex.ru logged sender identity as: bu7cher Message-ID: <46079251.7070202@yandex.ru> Date: Mon, 26 Mar 2007 13:28:49 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla Thunderbird 1.5 (FreeBSD/20051231) MIME-Version: 1.0 To: arjun badarinath References: <4cc04d3d0703260215k5950dc6bl6bde6eb38af0ee@mail.gmail.com> In-Reply-To: <4cc04d3d0703260215k5950dc6bl6bde6eb38af0ee@mail.gmail.com> Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-ipfw@freebsd.org Subject: Re: Packet filter X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2007 09:29:00 -0000 arjun badarinath ÐÉÛÅÔ: > Hi all , > I wanted to know how the paket filter (pf) is implemented on bsd . > can u please send me the link of the source code correponding to this http://www.freebsd.org/cgi/cvsweb.cgi/src/contrib/pf/ PS. Please, to the future, before asking a similar questions try to use find(1) and grep(1) with your source tree. -- WBR, Andrey V. Elsukov From owner-freebsd-ipfw@FreeBSD.ORG Mon Mar 26 11:08:16 2007 Return-Path: X-Original-To: freebsd-ipfw@FreeBSD.org Delivered-To: freebsd-ipfw@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 245AC16A401 for ; Mon, 26 Mar 2007 11:08:16 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [69.147.83.40]) by mx1.freebsd.org (Postfix) with ESMTP id DF76A13C45A for ; Mon, 26 Mar 2007 11:08:15 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id l2QB8FbP049306 for ; Mon, 26 Mar 2007 11:08:15 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id l2QB8Exn049302 for freebsd-ipfw@FreeBSD.org; Mon, 26 Mar 2007 11:08:14 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 26 Mar 2007 11:08:14 GMT Message-Id: <200703261108.l2QB8Exn049302@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: linimon set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-ipfw@FreeBSD.org Cc: Subject: Current problem reports assigned to you X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2007 11:08:16 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/51274 ipfw [ipfw] [patch] ipfw2 create dynamic rules with parent o kern/73910 ipfw [ipfw] serious bug on forwarding of packets after NAT o kern/74104 ipfw [ipfw] ipfw2/1 conflict not detected or reported, manp o conf/78762 ipfw [ipfw] [patch] /etc/rc.d/ipfw should excecute $firewal o bin/80913 ipfw [patch] /sbin/ipfw2 silently discards MAC addr arg wit o kern/88659 ipfw [modules] ipfw and ip6fw do not work properly as modul o kern/93300 ipfw ipfw pipe lost packets o kern/95084 ipfw [ipfw] [patch] IPFW2 ignores "recv/xmit/via any" (IPFW o kern/97504 ipfw [ipfw] IPFW Rules bug o kern/97951 ipfw [ipfw] [patch] ipfw does not tie interface details to o kern/98831 ipfw [ipfw] ipfw has UDP hickups o kern/102471 ipfw [ipfw] [patch] add tos and dscp support o kern/103454 ipfw [ipfw] [patch] add a facility to modify DF bit of the o kern/106534 ipfw [ipfw] [panic] ipfw + dummynet 14 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- a kern/26534 ipfw [ipfw] Add an option to ipfw to log gid/uid of who cau o kern/46159 ipfw [ipfw] [patch] ipfw dynamic rules lifetime feature o kern/48172 ipfw [ipfw] [patch] ipfw does not log size and flags o bin/50749 ipfw [ipfw] [patch] ipfw2 incorrectly parses ports and port o kern/55984 ipfw [ipfw] [patch] time based firewalling support for ipfw o kern/60719 ipfw [ipfw] Headerless fragments generate cryptic error mes o kern/69963 ipfw [ipfw] install_state warning about already existing en o kern/71366 ipfw [ipfw] "ipfw fwd" sometimes rewrites destination mac a o kern/72987 ipfw [ipfw] ipfw/dummynet pipe/queue 'queue [BYTES]KBytes ( o kern/73276 ipfw [ipfw] [patch] ipfw2 vulnerability (parser error) o bin/78785 ipfw [ipfw] [patch] ipfw verbosity locks machine if /etc/rc o kern/80642 ipfw [ipfw] [patch] ipfw small patch - new RULE OPTION o kern/82724 ipfw [ipfw] [patch] Add setnexthop and defaultroute feature o kern/86957 ipfw [ipfw] [patch] ipfw mac logging o kern/87032 ipfw [ipfw] [patch] ipfw ioctl interface implementation o kern/91847 ipfw [ipfw] ipfw with vlanX as the device o kern/103328 ipfw sugestions about ipfw table o kern/104682 ipfw [ipfw] [patch] Some minor language consistency fixes a o bin/104921 ipfw [patch] ipfw(8) sometimes treats ipv6 input as ipv4 (a o kern/105330 ipfw [ipfw] [patch] ipfw (dummynet) does not allow to set q 20 problems total. From owner-freebsd-ipfw@FreeBSD.ORG Thu Mar 29 08:15:15 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 117C216A400 for ; Thu, 29 Mar 2007 08:15:15 +0000 (UTC) (envelope-from patrice@idea.dnsalias.net) Received: from smtp24.orange.fr (smtp24.orange.fr [193.252.22.26]) by mx1.freebsd.org (Postfix) with ESMTP id 876F013C45A for ; Thu, 29 Mar 2007 08:15:14 +0000 (UTC) (envelope-from patrice@idea.dnsalias.net) Received: from smtp24.orange.fr (mwinf2433 [10.232.5.133]) by mwinf2425.orange.fr (SMTP Server) with ESMTP id E5F9D1C1FBE4 for ; Thu, 29 Mar 2007 09:34:45 +0200 (CEST) Received: from me-wanadoo.net (localhost [127.0.0.1]) by mwinf2433.orange.fr (SMTP Server) with ESMTP id 9261D1C00083 for ; Thu, 29 Mar 2007 09:34:44 +0200 (CEST) Received: from servidea.dvp.idea (AMarseille-111-1-1-213.w80-11.abo.wanadoo.fr [80.11.74.213]) by mwinf2433.orange.fr (SMTP Server) with ESMTP id AAD701C00092 for ; Thu, 29 Mar 2007 09:34:40 +0200 (CEST) X-ME-UUID: 20070329073440699.AAD701C00092@mwinf2433.orange.fr Received: from servidea.dvp.idea (localhost.dvp.idea [127.0.0.1]) by servidea.dvp.idea (8.13.1/8.13.1) with ESMTP id l2T7YbBB003127 for ; Thu, 29 Mar 2007 09:34:37 +0200 (CEST) (envelope-from patrice@servidea.dvp.idea) Received: (from patrice@localhost) by servidea.dvp.idea (8.13.1/8.13.1/Submit) id l2T7YbSO003126 for freebsd-ipfw@freebsd.org; Thu, 29 Mar 2007 09:34:37 +0200 (CEST) (envelope-from patrice) Date: Thu, 29 Mar 2007 09:34:37 +0200 (CEST) From: User Patrice Message-Id: <200703290734.l2T7YbSO003126@servidea.dvp.idea> To: freebsd-ipfw@freebsd.org Subject: ipfw drive me crazy X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 08:15:15 -0000 Hello I observe a strange behavior with ipfw/natd and fwd command. the same packet , fwd to a same address, use a different outgoing interface if it is nated. Here is the sample: freebsd (5.3release), 3 interfaces : - bge1: 10.10.21.2 connected to a local LAN 10.10.21/0 - bge0: switch: 10.10.20.1 for DMZ, and 192.168.0.101 for a new internet router at 192.168.0.254 - tun0: internet public address for a PPP adsl modem the freebsd is the default gateway for 10.10.21/24 network the tun0 is the default gateway interface inside the freebsd. i want to catch (based on tcp out port) outgoing packet to default gateway tun0, and send them to 192.168.0.254 the test i run is : - i have a valid system, routing every external traffic throught tun0 - i want every outgoing tcp 8080 connection will use 192.168.0.254 router 10.10.21.1 request a http connection on port 8080 without additionnal config, the request will come in bge1, go out tun0 i try to trap the request to use 192.168.0.254 gateway the target ip 192.168.0.254 is on a lan connected to bge0. ipfw add 00150 fwd 192.168.0.254 log ip from any to any dst-port 8080 the test was successfull, tcpdump show that : - incoming packet from 10.10.21.1 to external ip, 8080 on bge1 - outgoing packet to 192.168.0.254 via bge0 just a little strange behavior in ifpw log which show outgoing packet on tun0 i think it's strange because 192.168.0.254 is on lan connected to bge0 wich have ip 192.168.0.101 so now, just missing to nat the incoming packet ipfw add 00149 divert 3617 log ip from 10.10.0.0/16 to not 10.10.0.0/16 dst-port 8080 the test now give me headache. The log show than the packet is well catched & diverted, with same strange behavior: out via tun0 (strange because the target ip 192.168.0.254 is on a lan connected to bge0) and tcpdump show: - incoming packet from 10.10.21.1 to external ip, 8080 on bge1 - outgoing natd packet to 192.168.0.254 via tun0 (instead of bge0???) so for resume: if i do nothing, a packet in bge1 is going out on tun9 i want to catch packet in bge1, fwd to gateway on bge0 - if i just fwd, tcpdump say : ok it work, ipfw say: fwd is ok but on wrong interface - if i fwd&nat, tcpdump say : wrong interface, ipfw say: fwd is ok but on wrong interface and now ive got headache log from my test follow: ============================================================= interface: bge0: flags=8943 mtu 1500 options=1a inet 10.10.20.1 netmask 0xffffff00 broadcast 10.10.20.255 inet 192.168.0.101 netmask 0xffffff00 broadcast 192.168.0.255 ether 00:30:48:88:5f:f2 media: Ethernet autoselect (100baseTX ) status: active bge1: flags=8843 mtu 1500 options=1a inet 10.10.21.2 netmask 0xffffff00 broadcast 10.10.21.255 ether 00:30:48:88:5f:f3 media: Ethernet autoselect (100baseTX ) status: active tun0: flags=8151 mtu 1492 inet 80.11.76.251 --> 80.11.76.129 netmask 0xffffffff Route: Destination,Gateway, interface default, AMarseille-111-1-3, tun0 10.10.1/24, link#3,em0 10.10.20/24,link#1,bge0 10.10.21/24, link#2,bge1 AMarseille-111-1-3,AMarseille-111-1-3,tun0 192.168.0,link#1,bge0 natd: natd -o 3617 -alias_address 192.168.0.101 (i used -o port because i m going to nat packet on input interface, and the option -reverse of natd cause core dump) ==================================================================== test #1: 10.10.21.1 request a tcp to 8080 on a external web, forward every incoming packet for 8080 ipfw add 00150 fwd 192.168.0.254 log ip from any to any dst-port 8080 log: Mar 23 09:11:50 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP 10.10.21 .1:3798 194.167.78.73:8080 in via bge1 Mar 23 09:11:50 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP 10.10.21 .1:3798 194.167.78.73:8080 out via tun0 i do not understand why there is 2 lines for a single packet, and why is show tun0 as out interface but trace with tcpdump show that the packet : - is coming in bge1, out bge0, nothing as tun0 (192.168.0.254 is connected on bge0) tcppdump bge1: 09:11:50.131590 00:e0:18:53:3b:a1 > 00:30:48:88:5f:f3, ethertype IPv4 (0x0800), length 60: IP 10.10.21.1.3798 > 194.167.78.73.8080: S 1223470518:1223470518(0) w in 8192 09:11:53.391659 00:e0:18:53:3b:a1 > 00:30:48:88:5f:f3, ethertype IPv4 (0x0800), length 60: IP 10.10.21.1.3798 > 194.167.78.73.8080: S 1223470518:1223470518(0) w in 8192 tcpdump bge0 09:11:50.132040 00:30:48:88:5f:f2 > 00:07:cb:24:2b:c8, ethertype IPv4 (0x0800), length 58: IP 10.10.21.1.3798 > 194.167.78.73.8080: S 1223470518:1223470518(0) w in 8192 09:11:53.391740 00:30:48:88:5f:f2 > 00:07:cb:24:2b:c8, ethertype IPv4 (0x0800), length 58: IP 10.10.21.1.3798 > 194.167.78.73.8080: S 1223470518:1223470518(0) w in 8192 tcpdump tun0: nothing ==================================================================== test #2: same as test#1 ipfw add 00150 fwd 192.168.0.254 log ip from any to any dst-port 8080 and added natd before forwarding ipfw 00149 divert 3617 log ip from 10.10.0.0/16 to not 10.10.0.0/16 dst-port 8080 log: Mar 23 09:09:14 servidea kernel: ipfw: 149 Divert 3617 TCP 10.10.21.1:3757 194.1 67.78.73:8080 in via bge1 Mar 23 09:09:14 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP 192.168. 0.101:3757 194.167.78.73:8080 in via bge1 Mar 23 09:09:14 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP 192.168. 0.101:3757 194.167.78.73:8080 out via tun0 Mar 23 09:09:17 servidea kernel: ipfw: 149 Divert 3617 TCP 10.10.21.1:3757 194.1 67.78.73:8080 in via bge1 Mar 23 09:09:17 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP 192.168. 0.101:3757 194.167.78.73:8080 in via bge1 Mar 23 09:09:17 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP 192.168. 0.101:3757 194.167.78.73:8080 out via tun0 still same strange thing: 1 packet in bge1 cause 2 line forward tcpdump: bge1: 09:09:14.346305 00:e0:18:53:3b:a1 > 00:30:48:88:5f:f3, ethertype IPv4 (0x0800), length 60: IP 10.10.21.1.3757 > 194.167.78.73.8080: S 1223469999:1223469999(0) w in 8192 09:09:17.528385 00:e0:18:53:3b:a1 > 00:30:48:88:5f:f3, ethertype IPv4 (0x0800), length 60: IP 10.10.21.1.3757 > 194.167.78.73.8080: S 1223469999:1223469999(0) w in 8192 tcpdump bge0: nothing tcpdump tun0: 09:09:14.346459 AF 2 44: IP 192.168.0.101.3757 > 194.167.78.73.8080: S 122346999 9:1223469999(0) win 8192 09:09:17.528522 AF 2 44: IP 192.168.0.101.3757 > 194.167.78.73.8080: S 122346999 9:1223469999(0) win 8192 From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 06:46:22 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D526F16A510 for ; Fri, 30 Mar 2007 06:46:22 +0000 (UTC) (envelope-from astounding@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.251]) by mx1.freebsd.org (Postfix) with ESMTP id 9530113C480 for ; Fri, 30 Mar 2007 06:46:22 +0000 (UTC) (envelope-from astounding@gmail.com) Received: by an-out-0708.google.com with SMTP id c24so399173ana for ; Thu, 29 Mar 2007 23:46:21 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; b=LvTbVHsJlXl1QvK0LosNzAzR6d4Telcpfwyhpjp+U2Vlk/eeSRja6EgqzphdBCMbxcUvhtRyFMW4eHK9279XdgmQUJN2XEGglXFkWwnSuJVh1i3nKUj86Jgesk9Qfd2qntsS1uJlCrSEyE1BChHG3VcuwqREI+3nEv8y5VFcKK4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type; b=my7hV9jy6CNKNQATDfEgFkQ06EZ4BiwiqMZ2dTQkAdpCbp07j6DMWWVyNbZ+YlCysdGX36w7OXj6+matiqMNpKLiO91Eh4ILdBmSSSjYRfyBLVeLnPrjxvk/NZdYJ1QIOdxU413u7pZmJR0hTSYzlyuAblzzo3BBmsHib+n1Vjs= Received: by 10.100.130.8 with SMTP id c8mr1084497and.1175235543556; Thu, 29 Mar 2007 23:19:03 -0700 (PDT) Received: by 10.100.109.8 with HTTP; Thu, 29 Mar 2007 23:19:03 -0700 (PDT) Message-ID: Date: Fri, 30 Mar 2007 00:19:03 -0600 From: "Aaron Gifford" To: freebsd-ipfw@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Lifetime patches for ipfw updated for 6.2-STABLE as of 29 Mar. 2007 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 06:46:22 -0000 This is just to let anyone using my ipfw lifetime patches know that they've been updated following the recent changes in sys/netinet/ip_fw2.c from time_second to time_uptime so that the patches will apply cleanly. They're available as usual from my web site (which also describes what they do): http://www.adg.us/computers/ipfwpatch.html Aaron out. From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 07:10:43 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B7C1C16A407 for ; Fri, 30 Mar 2007 07:10:43 +0000 (UTC) (envelope-from dave@raven.za.net) Received: from elektra.opteqint.net (elektra.opteqint.net [209.25.178.105]) by mx1.freebsd.org (Postfix) with ESMTP id A8FED13C48A for ; Fri, 30 Mar 2007 07:10:43 +0000 (UTC) (envelope-from dave@raven.za.net) Received: from [196.209.87.107] (helo=DHA12123) by elektra.opteqint.net with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.66 (FreeBSD)) (envelope-from ) id 1HXAvC-000OQD-9G for freebsd-ipfw@freebsd.org; Thu, 29 Mar 2007 22:49:17 -0800 From: "Dave Raven" To: Date: Fri, 30 Mar 2007 08:49:19 +0200 Message-ID: <015d01c77297$953ba250$bfb2e6f0$@za.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Acdyl4nqGvMege9HTSegOMaDndJzWw== Content-Language: en-us Subject: Using "delay" to emulate a satellite link X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 07:10:43 -0000 Hi all, I've been looking at the ipfw (dummynet) ability to do delay and have a few questions - I hope this is the right list. I want to simulate a 1000ms RTT on a satellite link. To do that I've created an inbound and outbound pipe and given each 1mb and 500ms of delay. However, I'm unable to get anywhere near 1mb of throughput on it until I drop the delay. I believe I understand the slowdown due to the latency, but my question is this - an http download through a 500ms "emulated" link that's running 1 mb can't get 1mb, yet if I download over the internet on a site that's pinging 500ms, it goes 1mb Whats the difference between dummynet delay and real life distance delay? Thanks Dave From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 07:59:17 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9973416A400 for ; Fri, 30 Mar 2007 07:59:17 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.freebsd.org (Postfix) with ESMTP id 860F213C489 for ; Fri, 30 Mar 2007 07:59:17 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.13.6) with ESMTP id l2U7xGnl076432; Thu, 29 Mar 2007 23:59:16 -0800 (PST) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id l2U7xGDV076431; Fri, 30 Mar 2007 00:59:16 -0700 (PDT) (envelope-from rizzo) Date: Fri, 30 Mar 2007 00:59:16 -0700 From: Luigi Rizzo To: Dave Raven Message-ID: <20070330005916.A76128@xorpc.icir.org> References: <015d01c77297$953ba250$bfb2e6f0$@za.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <015d01c77297$953ba250$bfb2e6f0$@za.net>; from dave@raven.za.net on Fri, Mar 30, 2007 at 08:49:19AM +0200 Cc: freebsd-ipfw@freebsd.org Subject: Re: Using "delay" to emulate a satellite link X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 07:59:17 -0000 On Fri, Mar 30, 2007 at 08:49:19AM +0200, Dave Raven wrote: > Hi all, > I've been looking at the ipfw (dummynet) ability to do delay and > have a few questions - I hope this is the right list. I want to simulate a > 1000ms RTT on a satellite link. To do that I've created an inbound and > outbound pipe and given each 1mb and 500ms of delay. > > However, I'm unable to get anywhere near 1mb of throughput on it until I > drop the delay. I believe I understand the slowdown due to the latency, but > my question is this - an http download through a 500ms "emulated" link > that's running 1 mb can't get 1mb, yet if I download over the internet on a > site that's pinging 500ms, it goes 1mb > > Whats the difference between dummynet delay and real life distance delay? first make sure you are not comparing apples and oranges. what sender and receiver are you using to get 1Mbit/s on a 500ms link ? and, are you sure that if you ping from the source to the destination you are using for your tests with dummynet you get the delay you are expecting (1000ms as you configured it, and not 2000 ?) 500ms of delay on each pipe give at least 1000ms total delay (assuming you have not misconfigured your dummynet box). In order to fill the pipe you need at least 1Mbit worth of data in the socket buffer/tcp window - the default on FreeBSD is 32kbytes sending, 64Kbytes receiving, so you won't be able to achieve that unless you increase these two sysctls: net.inet.tcp.sendspace: 32768 net.inet.tcp.recvspace: 65536 if you draw the bw vs delay that you achieve on your connection you will likely find that either the limit is your socket buffers or a misconfigured ipfw which results in twice the delay cheers luigi From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 09:09:26 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2878716A407 for ; Fri, 30 Mar 2007 09:09:26 +0000 (UTC) (envelope-from dave@raven.za.net) Received: from elektra.opteqint.net (elektra.opteqint.net [209.25.178.105]) by mx1.freebsd.org (Postfix) with ESMTP id 16A6E13C45D for ; Fri, 30 Mar 2007 09:09:25 +0000 (UTC) (envelope-from dave@raven.za.net) Received: from [196.209.87.107] (helo=DHA12123) by elektra.opteqint.net with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.66 (FreeBSD)) (envelope-from ) id 1HXD6V-000P3i-Vh; Fri, 30 Mar 2007 01:09:07 -0800 From: "Dave Raven" To: "'Luigi Rizzo'" References: <015d01c77297$953ba250$bfb2e6f0$@za.net> <20070330005916.A76128@xorpc.icir.org> In-Reply-To: <20070330005916.A76128@xorpc.icir.org> Date: Fri, 30 Mar 2007 11:09:07 +0200 Message-ID: <01a501c772ab$1dcbd460$59637d20$@za.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: AcdyoUcCQaSFT+huSxm7PtmkMM/lUwACRnjw Content-Language: en-us Cc: freebsd-ipfw@freebsd.org Subject: RE: Using "delay" to emulate a satellite link X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 09:09:26 -0000 Hi Luigi, Thanks for the reply -- these are my related send/recv space settings on all of the boxes involved -- net.local.stream.sendspace: 65535 net.local.stream.recvspace: 65535 net.local.dgram.recvspace: 4096 net.inet.tcp.sendspace: 65535 net.inet.tcp.recvspace: 65535 net.inet.udp.recvspace: 65535 net.inet.raw.recvspace: 8192 What I have done is put 3 freebsd units plugged into eath other with crossover cables, like so -- BOX 1 ----- BOX 2 ----- BOX 3 Box 3 has a 500meg /dev/urandom text file on a webserver. Box 1 wget's that file. Box 2 has the ipfw delay setup. If I ping from BOX 1 to BOX 3 I get +/-1000ms of delay. When I download I get less than half of the link speed (1meg limited by box 2). The latency directly affects the throughput - but my question is this -- If I download something from a web server on the internet that I have 500ms of delay to, I can get 1meg (on a 1meg link). When I emulate that delay with dummynet I can't -- is there a difference in the type of delay experienced? Really anyone can - most people have delay of at least 300-500ms to remote webservers (e.g. from me in Africa to America); but it doesn't hamper download speed? Thanks again Dave -----Original Message----- From: Luigi Rizzo [mailto:rizzo@icir.org] Sent: Friday, March 30, 2007 9:59 AM To: Dave Raven Cc: freebsd-ipfw@freebsd.org Subject: Re: Using "delay" to emulate a satellite link On Fri, Mar 30, 2007 at 08:49:19AM +0200, Dave Raven wrote: > Hi all, > I've been looking at the ipfw (dummynet) ability to do delay and > have a few questions - I hope this is the right list. I want to simulate a > 1000ms RTT on a satellite link. To do that I've created an inbound and > outbound pipe and given each 1mb and 500ms of delay. > > However, I'm unable to get anywhere near 1mb of throughput on it until I > drop the delay. I believe I understand the slowdown due to the latency, but > my question is this - an http download through a 500ms "emulated" link > that's running 1 mb can't get 1mb, yet if I download over the internet on a > site that's pinging 500ms, it goes 1mb > > Whats the difference between dummynet delay and real life distance delay? first make sure you are not comparing apples and oranges. what sender and receiver are you using to get 1Mbit/s on a 500ms link ? and, are you sure that if you ping from the source to the destination you are using for your tests with dummynet you get the delay you are expecting (1000ms as you configured it, and not 2000 ?) 500ms of delay on each pipe give at least 1000ms total delay (assuming you have not misconfigured your dummynet box). In order to fill the pipe you need at least 1Mbit worth of data in the socket buffer/tcp window - the default on FreeBSD is 32kbytes sending, 64Kbytes receiving, so you won't be able to achieve that unless you increase these two sysctls: net.inet.tcp.sendspace: 32768 net.inet.tcp.recvspace: 65536 if you draw the bw vs delay that you achieve on your connection you will likely find that either the limit is your socket buffers or a misconfigured ipfw which results in twice the delay cheers luigi From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 09:14:45 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 715C016A402 for ; Fri, 30 Mar 2007 09:14:45 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.freebsd.org (Postfix) with ESMTP id 5DF1213C465 for ; Fri, 30 Mar 2007 09:14:45 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.13.6) with ESMTP id l2U9EiRk077677; Fri, 30 Mar 2007 01:14:44 -0800 (PST) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id l2U9EiMj077676; Fri, 30 Mar 2007 02:14:44 -0700 (PDT) (envelope-from rizzo) Date: Fri, 30 Mar 2007 02:14:42 -0700 From: "'Luigi Rizzo'" To: Dave Raven Message-ID: <20070330021442.A77652@xorpc.icir.org> References: <015d01c77297$953ba250$bfb2e6f0$@za.net> <20070330005916.A76128@xorpc.icir.org> <01a501c772ab$1dcbd460$59637d20$@za.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <01a501c772ab$1dcbd460$59637d20$@za.net>; from dave@raven.za.net on Fri, Mar 30, 2007 at 11:09:07AM +0200 Cc: freebsd-ipfw@freebsd.org Subject: Re: Using "delay" to emulate a satellite link X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 09:14:45 -0000 On Fri, Mar 30, 2007 at 11:09:07AM +0200, Dave Raven wrote: > Hi Luigi, > Thanks for the reply -- these are my related send/recv space > settings on all of the boxes involved -- > > net.local.stream.sendspace: 65535 > net.local.stream.recvspace: 65535 > net.local.dgram.recvspace: 4096 > net.inet.tcp.sendspace: 65535 > net.inet.tcp.recvspace: 65535 > net.inet.udp.recvspace: 65535 > net.inet.raw.recvspace: 8192 > > What I have done is put 3 freebsd units plugged into eath other with > crossover cables, like so -- > > BOX 1 ----- BOX 2 ----- BOX 3 > > Box 3 has a 500meg /dev/urandom text file on a webserver. Box 1 wget's that > file. Box 2 has the ipfw delay setup. If I ping from BOX 1 to BOX 3 I get > +/-1000ms of delay. When I download I get less than half of the link speed > (1meg limited by box 2). which is fine - you get 65kbytes/1000ms = 512kbit/s > The latency directly affects the throughput - but my question is this -- If > I download something from a web server on the internet that I have 500ms of > delay to, I can get 1meg (on a 1meg link). When I emulate that delay with > dummynet I can't -- is there a difference in the type of delay experienced? well, it's twice as much in your case - your ping time is 1000ms instead of 500ms cheers luigi > Really anyone can - most people have delay of at least 300-500ms to remote > webservers (e.g. from me in Africa to America); but it doesn't hamper > download speed? > > Thanks again > Dave > > > > -----Original Message----- > From: Luigi Rizzo [mailto:rizzo@icir.org] > Sent: Friday, March 30, 2007 9:59 AM > To: Dave Raven > Cc: freebsd-ipfw@freebsd.org > Subject: Re: Using "delay" to emulate a satellite link > > On Fri, Mar 30, 2007 at 08:49:19AM +0200, Dave Raven wrote: > > Hi all, > > I've been looking at the ipfw (dummynet) ability to do delay and > > have a few questions - I hope this is the right list. I want to simulate a > > 1000ms RTT on a satellite link. To do that I've created an inbound and > > outbound pipe and given each 1mb and 500ms of delay. > > > > However, I'm unable to get anywhere near 1mb of throughput on it until I > > drop the delay. I believe I understand the slowdown due to the latency, > but > > my question is this - an http download through a 500ms "emulated" link > > that's running 1 mb can't get 1mb, yet if I download over the internet on > a > > site that's pinging 500ms, it goes 1mb > > > > Whats the difference between dummynet delay and real life distance delay? > > first make sure you are not comparing apples and oranges. > what sender and receiver are you using to get 1Mbit/s on a 500ms link ? > and, are you sure that if you ping from the source to the destination > you are using for your tests with dummynet you get the delay > you are expecting (1000ms as you configured it, and not 2000 ?) > > 500ms of delay on each pipe give at least 1000ms total > delay (assuming you have not misconfigured your dummynet box). > In order to fill the pipe you need at least 1Mbit worth of data > in the socket buffer/tcp window - the default on FreeBSD is > 32kbytes sending, 64Kbytes receiving, so you won't be > able to achieve that unless you increase these two sysctls: > > net.inet.tcp.sendspace: 32768 > net.inet.tcp.recvspace: 65536 > > if you draw the bw vs delay that you achieve on your connection > you will likely find that either the limit is your socket > buffers or a misconfigured ipfw which results in twice the delay > > cheers > luigi From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 10:02:50 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C3CDC16A5A5 for ; Fri, 30 Mar 2007 10:02:50 +0000 (UTC) (envelope-from dave@raven.za.net) Received: from elektra.opteqint.net (elektra.opteqint.net [209.25.178.105]) by mx1.freebsd.org (Postfix) with ESMTP id AFE8213C455 for ; Fri, 30 Mar 2007 10:02:50 +0000 (UTC) (envelope-from dave@raven.za.net) Received: from [196.209.87.107] (helo=DHA12123) by elektra.opteqint.net with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.66 (FreeBSD)) (envelope-from ) id 1HXDw8-000PHN-VG; Fri, 30 Mar 2007 02:02:33 -0800 From: "Dave Raven" To: "'Luigi Rizzo'" References: <015d01c77297$953ba250$bfb2e6f0$@za.net> <20070330005916.A76128@xorpc.icir.org> <01a501c772ab$1dcbd460$59637d20$@za.net> <20070330021442.A77652@xorpc.icir.org> In-Reply-To: <20070330021442.A77652@xorpc.icir.org> Date: Fri, 30 Mar 2007 12:02:28 +0200 Message-ID: <01be01c772b2$91e64180$b5b2c480$@za.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Acdyq9RKQtjUozzsRwedzKAO1gPEUwABloXw Content-Language: en-us Cc: freebsd-ipfw@freebsd.org Subject: RE: Using "delay" to emulate a satellite link X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 10:02:50 -0000 Hi Luigi, Firstly sorry for the confusion - I mean even if I change it to 250/250 (e.g. 500ms of delay) I can't reach 1mbps of throughput. Is it a definite that because of the latency (if its bad enough) you can never achieve full throughput - because of TCP ack delays? Your 65kbytes/1000ms = 512kbit/s calculation - is that 65k from the send/recv space ? The maximum is 65k correct ? Does that mean that on a given 1000ms link you CANNOT achieve more than 512kbps ? Thanks so much for the help - I know its going a bit off topic Dave -----Original Message----- From: 'Luigi Rizzo' [mailto:rizzo@icir.org] Sent: Friday, March 30, 2007 11:15 AM To: Dave Raven Cc: freebsd-ipfw@freebsd.org Subject: Re: Using "delay" to emulate a satellite link On Fri, Mar 30, 2007 at 11:09:07AM +0200, Dave Raven wrote: > Hi Luigi, > Thanks for the reply -- these are my related send/recv space > settings on all of the boxes involved -- > > net.local.stream.sendspace: 65535 > net.local.stream.recvspace: 65535 > net.local.dgram.recvspace: 4096 > net.inet.tcp.sendspace: 65535 > net.inet.tcp.recvspace: 65535 > net.inet.udp.recvspace: 65535 > net.inet.raw.recvspace: 8192 > > What I have done is put 3 freebsd units plugged into eath other with > crossover cables, like so -- > > BOX 1 ----- BOX 2 ----- BOX 3 > > Box 3 has a 500meg /dev/urandom text file on a webserver. Box 1 wget's that > file. Box 2 has the ipfw delay setup. If I ping from BOX 1 to BOX 3 I get > +/-1000ms of delay. When I download I get less than half of the link speed > (1meg limited by box 2). which is fine - you get 65kbytes/1000ms = 512kbit/s > The latency directly affects the throughput - but my question is this -- If > I download something from a web server on the internet that I have 500ms of > delay to, I can get 1meg (on a 1meg link). When I emulate that delay with > dummynet I can't -- is there a difference in the type of delay experienced? well, it's twice as much in your case - your ping time is 1000ms instead of 500ms cheers luigi > Really anyone can - most people have delay of at least 300-500ms to remote > webservers (e.g. from me in Africa to America); but it doesn't hamper > download speed? > > Thanks again > Dave > > > > -----Original Message----- > From: Luigi Rizzo [mailto:rizzo@icir.org] > Sent: Friday, March 30, 2007 9:59 AM > To: Dave Raven > Cc: freebsd-ipfw@freebsd.org > Subject: Re: Using "delay" to emulate a satellite link > > On Fri, Mar 30, 2007 at 08:49:19AM +0200, Dave Raven wrote: > > Hi all, > > I've been looking at the ipfw (dummynet) ability to do delay and > > have a few questions - I hope this is the right list. I want to simulate a > > 1000ms RTT on a satellite link. To do that I've created an inbound and > > outbound pipe and given each 1mb and 500ms of delay. > > > > However, I'm unable to get anywhere near 1mb of throughput on it until I > > drop the delay. I believe I understand the slowdown due to the latency, > but > > my question is this - an http download through a 500ms "emulated" link > > that's running 1 mb can't get 1mb, yet if I download over the internet on > a > > site that's pinging 500ms, it goes 1mb > > > > Whats the difference between dummynet delay and real life distance delay? > > first make sure you are not comparing apples and oranges. > what sender and receiver are you using to get 1Mbit/s on a 500ms link ? > and, are you sure that if you ping from the source to the destination > you are using for your tests with dummynet you get the delay > you are expecting (1000ms as you configured it, and not 2000 ?) > > 500ms of delay on each pipe give at least 1000ms total > delay (assuming you have not misconfigured your dummynet box). > In order to fill the pipe you need at least 1Mbit worth of data > in the socket buffer/tcp window - the default on FreeBSD is > 32kbytes sending, 64Kbytes receiving, so you won't be > able to achieve that unless you increase these two sysctls: > > net.inet.tcp.sendspace: 32768 > net.inet.tcp.recvspace: 65536 > > if you draw the bw vs delay that you achieve on your connection > you will likely find that either the limit is your socket > buffers or a misconfigured ipfw which results in twice the delay > > cheers > luigi From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 10:14:33 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EC7DB16A401 for ; Fri, 30 Mar 2007 10:14:33 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.freebsd.org (Postfix) with ESMTP id CF1A813C45E for ; Fri, 30 Mar 2007 10:14:33 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.13.6) with ESMTP id l2UAEX3Q078495; Fri, 30 Mar 2007 02:14:33 -0800 (PST) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id l2UAEWSL078494; Fri, 30 Mar 2007 03:14:32 -0700 (PDT) (envelope-from rizzo) Date: Fri, 30 Mar 2007 03:14:32 -0700 From: "'Luigi Rizzo'" To: Dave Raven Message-ID: <20070330031432.A78468@xorpc.icir.org> References: <015d01c77297$953ba250$bfb2e6f0$@za.net> <20070330005916.A76128@xorpc.icir.org> <01a501c772ab$1dcbd460$59637d20$@za.net> <20070330021442.A77652@xorpc.icir.org> <01be01c772b2$91e64180$b5b2c480$@za.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <01be01c772b2$91e64180$b5b2c480$@za.net>; from dave@raven.za.net on Fri, Mar 30, 2007 at 12:02:28PM +0200 Cc: freebsd-ipfw@freebsd.org Subject: Re: Using "delay" to emulate a satellite link X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 10:14:34 -0000 On Fri, Mar 30, 2007 at 12:02:28PM +0200, Dave Raven wrote: > Hi Luigi, > Firstly sorry for the confusion - I mean even if I change it to > 250/250 (e.g. 500ms of delay) I can't reach 1mbps of throughput. Is it a > definite that because of the latency (if its bad enough) you can never > achieve full throughput - because of TCP ack delays? > > Your 65kbytes/1000ms = 512kbit/s calculation - is that 65k from the > send/recv space ? The maximum is 65k correct ? Does that mean that on a > given 1000ms link you CANNOT achieve more than 512kbps ? yes this is basic networking stuff - for a window-based protocol the max throughtput is 1 window per rtt, where the window is upper bounded by the min of socket buffer, tcp buffers, negotiated tcp window luigi > Thanks so much for the help - I know its going a bit off topic > > Dave > > -----Original Message----- > From: 'Luigi Rizzo' [mailto:rizzo@icir.org] > Sent: Friday, March 30, 2007 11:15 AM > To: Dave Raven > Cc: freebsd-ipfw@freebsd.org > Subject: Re: Using "delay" to emulate a satellite link > > On Fri, Mar 30, 2007 at 11:09:07AM +0200, Dave Raven wrote: > > Hi Luigi, > > Thanks for the reply -- these are my related send/recv space > > settings on all of the boxes involved -- > > > > net.local.stream.sendspace: 65535 > > net.local.stream.recvspace: 65535 > > net.local.dgram.recvspace: 4096 > > net.inet.tcp.sendspace: 65535 > > net.inet.tcp.recvspace: 65535 > > net.inet.udp.recvspace: 65535 > > net.inet.raw.recvspace: 8192 > > > > What I have done is put 3 freebsd units plugged into eath other with > > crossover cables, like so -- > > > > BOX 1 ----- BOX 2 ----- BOX 3 > > > > Box 3 has a 500meg /dev/urandom text file on a webserver. Box 1 wget's > that > > file. Box 2 has the ipfw delay setup. If I ping from BOX 1 to BOX 3 I get > > +/-1000ms of delay. When I download I get less than half of the link speed > > (1meg limited by box 2). > > which is fine - you get 65kbytes/1000ms = 512kbit/s > > > The latency directly affects the throughput - but my question is this -- > If > > I download something from a web server on the internet that I have 500ms > of > > delay to, I can get 1meg (on a 1meg link). When I emulate that delay with > > dummynet I can't -- is there a difference in the type of delay > experienced? > > well, it's twice as much in your case - your ping time is 1000ms instead > of 500ms > > cheers > luigi > > > Really anyone can - most people have delay of at least 300-500ms to remote > > webservers (e.g. from me in Africa to America); but it doesn't hamper > > download speed? > > > > Thanks again > > Dave > > > > > > > > -----Original Message----- > > From: Luigi Rizzo [mailto:rizzo@icir.org] > > Sent: Friday, March 30, 2007 9:59 AM > > To: Dave Raven > > Cc: freebsd-ipfw@freebsd.org > > Subject: Re: Using "delay" to emulate a satellite link > > > > On Fri, Mar 30, 2007 at 08:49:19AM +0200, Dave Raven wrote: > > > Hi all, > > > I've been looking at the ipfw (dummynet) ability to do delay and > > > have a few questions - I hope this is the right list. I want to simulate > a > > > 1000ms RTT on a satellite link. To do that I've created an inbound and > > > outbound pipe and given each 1mb and 500ms of delay. > > > > > > However, I'm unable to get anywhere near 1mb of throughput on it until I > > > drop the delay. I believe I understand the slowdown due to the latency, > > but > > > my question is this - an http download through a 500ms "emulated" link > > > that's running 1 mb can't get 1mb, yet if I download over the internet > on > > a > > > site that's pinging 500ms, it goes 1mb > > > > > > Whats the difference between dummynet delay and real life distance > delay? > > > > first make sure you are not comparing apples and oranges. > > what sender and receiver are you using to get 1Mbit/s on a 500ms link ? > > and, are you sure that if you ping from the source to the destination > > you are using for your tests with dummynet you get the delay > > you are expecting (1000ms as you configured it, and not 2000 ?) > > > > 500ms of delay on each pipe give at least 1000ms total > > delay (assuming you have not misconfigured your dummynet box). > > In order to fill the pipe you need at least 1Mbit worth of data > > in the socket buffer/tcp window - the default on FreeBSD is > > 32kbytes sending, 64Kbytes receiving, so you won't be > > able to achieve that unless you increase these two sysctls: > > > > net.inet.tcp.sendspace: 32768 > > net.inet.tcp.recvspace: 65536 > > > > if you draw the bw vs delay that you achieve on your connection > > you will likely find that either the limit is your socket > > buffers or a misconfigured ipfw which results in twice the delay > > > > cheers > > luigi From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 20:53:43 2007 Return-Path: X-Original-To: ipfw@freebsd.org Delivered-To: freebsd-ipfw@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5BA9916A408 for ; Fri, 30 Mar 2007 20:53:43 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outZ.internet-mail-service.net (outZ.internet-mail-service.net [216.240.47.249]) by mx1.freebsd.org (Postfix) with ESMTP id 4AAD013C487 for ; Fri, 30 Mar 2007 20:53:43 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Fri, 30 Mar 2007 13:11:33 -0700 Received: from [10.251.22.38] (nat.ironport.com [63.251.108.100]) by idiom.com (Postfix) with ESMTP id B1FF0125C26; Fri, 30 Mar 2007 13:40:47 -0700 (PDT) Message-ID: <460D75CE.70804@elischer.org> Date: Fri, 30 Mar 2007 13:40:46 -0700 From: Julian Elischer User-Agent: Thunderbird 1.5.0.10 (Macintosh/20070221) MIME-Version: 1.0 To: ipfw@freebsd.org, FreeBSD Net Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 20:53:43 -0000 I have been looking at the IPFW code recently, especially with respect to locking. There are some things that could be done to improve IPFW's behaviour when processing packets, but some of these take a toll (there is always a toll) on the 'updating' side of things. For example. I can make IPFW lock-free during processing of packets (i.e. not holding any locks while traversing the list) which would solve problems we have with lock-order reversals when it needs to look at the socket layer (which needs socket layer locks). Unfortunatly this would make it a lot more expensive in the case where new rules are being added to the list. possibly a LOT more expensive. Now, this would only matter if one was adding (or deleting) hundreds of rules per second to the firewall, but as I've discovered, there's always SOMEONE that is doing the very thing you imagine that no-one would ever do. In my imagination, most of the people who did this sort of thing don't need to do it any more as tables obviate the need for that sort of thing. Is there anyone out there who is adding hundreds (or even dozens) of rules per second on a continuous basis, or who wants rule changing to be a really efficient operation? (does it matter to you if it takes a few milliSecs to add a rule?) Julian From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 21:31:06 2007 Return-Path: X-Original-To: ipfw@freebsd.org Delivered-To: freebsd-ipfw@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AE32816A401 for ; Fri, 30 Mar 2007 21:31:06 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from tokyo01.jp.mail.your.org (tokyo01.jp.mail.your.org [204.9.54.5]) by mx1.freebsd.org (Postfix) with ESMTP id 83ABF13C458 for ; Fri, 30 Mar 2007 21:31:06 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail.your.org (server3-a.your.org [64.202.112.67]) by tokyo01.jp.mail.your.org (Postfix) with ESMTP id E2F2B2AD56A6; Fri, 30 Mar 2007 21:02:44 +0000 (UTC) Received: from [69.31.99.11] (pool011.dhcp.your.org [69.31.99.11]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.your.org (Postfix) with ESMTP id 2D703A0A44E; Fri, 30 Mar 2007 21:02:43 +0000 (UTC) In-Reply-To: <460D75CE.70804@elischer.org> References: <460D75CE.70804@elischer.org> Mime-Version: 1.0 (Apple Message framework v752.3) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <98FE1AAF-DF19-43A8-A5B6-010C852AF489@dragondata.com> Content-Transfer-Encoding: 7bit From: Kevin Day Date: Fri, 30 Mar 2007 16:02:43 -0500 To: Julian Elischer X-Mailer: Apple Mail (2.752.3) Cc: FreeBSD Net , ipfw@freebsd.org Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 21:31:06 -0000 On Mar 30, 2007, at 3:40 PM, Julian Elischer wrote: > I have been looking at the IPFW code recently, especially with > respect to locking. > There are some things that could be done to improve IPFW's > behaviour when processing packets, but some of these take a > toll (there is always a toll) on the 'updating' side of things. > > For example. I can make IPFW lock-free during processing of packets > (i.e. not holding any locks while traversing the > list) which would solve problems we have with lock-order reversals > when it needs to look at the socket layer (which needs socket layer > locks). Unfortunatly this would make it a lot more expensive > in the case where new rules are being added to the list. possibly a > LOT > more expensive. Now, this would only matter if one was adding (or > deleting) > hundreds of rules per second to the firewall, but as I've discovered, > there's always SOMEONE that is doing the very thing you imagine that > no-one would ever do. > > In my imagination, most of the people who did this sort of thing don't > need to do it any more as tables obviate the need for that sort of > thing. > > Is there anyone out there who is adding hundreds (or even dozens) > of rules > per second on a continuous basis, or who wants rule changing to > be a really efficient operation? > (does it matter to you if it takes a few milliSecs to add a rule?) > > Julian Would this apply to "dynamic rules", using the keep-state keyword? That'd be a killer for us. If not, the only problem I'd have is that my ipfw startup script adds about 20,000 rules on a reboot. 20,000 rules multiplied by any significant amount of time would be bad, just from a reboot-recovery- time angle. But, if it improved overall performance, I probably wouldn't mind too much. :) From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 21:49:24 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0407116A400 for ; Fri, 30 Mar 2007 21:49:24 +0000 (UTC) (envelope-from fcash@ocis.net) Received: from smtp.sd73.bc.ca (smtp.sd73.bc.ca [142.24.13.140]) by mx1.freebsd.org (Postfix) with ESMTP id D19F013C455 for ; Fri, 30 Mar 2007 21:49:23 +0000 (UTC) (envelope-from fcash@ocis.net) Received: from localhost (localhost [127.0.0.1]) by localhost.sd73.bc.ca (Postfix) with ESMTP id A7C551A000B2D for ; Fri, 30 Mar 2007 14:21:37 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at smtp.sd73.bc.ca Received: from smtp.sd73.bc.ca ([127.0.0.1]) by localhost (smtp.sd73.bc.ca [127.0.0.1]) (amavisd-new, port 10024) with LMTP id ENP2R9890U8t for ; Fri, 30 Mar 2007 14:21:31 -0700 (PDT) Received: from coal (s10.sbo [192.168.0.10]) by smtp.sd73.bc.ca (Postfix) with ESMTP id 279D01A000B11 for ; Fri, 30 Mar 2007 14:21:31 -0700 (PDT) From: Freddie Cash To: freebsd-ipfw@freebsd.org Date: Fri, 30 Mar 2007 14:21:29 -0700 User-Agent: KMail/1.9.5 References: <460D75CE.70804@elischer.org> In-Reply-To: <460D75CE.70804@elischer.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200703301421.29919.fcash@ocis.net> Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 21:49:24 -0000 On Friday 30 March 2007 01:40 pm, Julian Elischer wrote: > I have been looking at the IPFW code recently, especially > with respect to locking. > There are some things that could be done to improve IPFW's > behaviour when processing packets, but some of these take a > toll (there is always a toll) on the 'updating' side of things. > > For example. I can make IPFW lock-free during processing > of packets (i.e. not holding any locks while traversing the > list) which would solve problems we have with lock-order reversals > when it needs to look at the socket layer (which needs socket > layer locks). Unfortunatly this would make it a lot more expensive > in the case where new rules are being added to the list. possibly a LOT > more expensive. Now, this would only matter if one was adding (or > deleting) hundreds of rules per second to the firewall, but as I've > discovered, there's always SOMEONE that is doing the very thing you > imagine that no-one would ever do. > > In my imagination, most of the people who did this sort of thing don't > need to do it any more as tables obviate the need for that sort of > thing. > > Is there anyone out there who is adding hundreds (or even dozens) of > rules per second on a continuous basis, or who wants rule changing to > be a really efficient operation? > (does it matter to you if it takes a few milliSecs to add a rule?) If the initial loading of a ruleset via a script counts, then yes. We add 600+ rules each time the script is run. During testing, or when troubleshooting connection issues, the rules could be reloaded several times over 10 minutes. We've moved away from adding rules dynamically, preferring to add rules to the script and reload them all. Keeps the rules in memory in sync with the rules on disk. Otherwise, no. :) -- Freddie Cash fcash@ocis.net From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 22:32:27 2007 Return-Path: X-Original-To: ipfw@freebsd.org Delivered-To: freebsd-ipfw@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C972E16A403 for ; Fri, 30 Mar 2007 22:32:27 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.freebsd.org (Postfix) with ESMTP id B675313C4B0 for ; Fri, 30 Mar 2007 22:32:27 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.13.6) with ESMTP id l2ULxcP6088248; Fri, 30 Mar 2007 13:59:38 -0800 (PST) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id l2ULxckW088247; Fri, 30 Mar 2007 14:59:38 -0700 (PDT) (envelope-from rizzo) Date: Fri, 30 Mar 2007 14:59:38 -0700 From: Luigi Rizzo To: Julian Elischer Message-ID: <20070330145938.A88154@xorpc.icir.org> References: <460D75CE.70804@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <460D75CE.70804@elischer.org>; from julian@elischer.org on Fri, Mar 30, 2007 at 01:40:46PM -0700 Cc: FreeBSD Net , ipfw@freebsd.org Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 22:32:27 -0000 On Fri, Mar 30, 2007 at 01:40:46PM -0700, Julian Elischer wrote: > I have been looking at the IPFW code recently, especially > with respect to locking. > There are some things that could be done to improve IPFW's > behaviour when processing packets, but some of these take a > toll (there is always a toll) on the 'updating' side of things. certainly ipfw was not designed with SMP in mind. If you can tell us what is your plan to make the list lock free (which one, the static or dynamic ones ?) maybe we can comment more. E.g. one option could be the usual trick of adding refcounts to the individual rules, and then using an array of pointers to them. While processing you grab a refcount to the array, and release it once done with the packet. If there is an addition or removal, you duplicate the array (which may be expensive for the large 20k rules mentioned), manipulate the copy and then atomically swap the pointers to the head. This might even work for dynamic rules as the lists (the content of each hash bucket) are typically short. cheers luigi From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 22:50:33 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A106016A403 for ; Fri, 30 Mar 2007 22:50:33 +0000 (UTC) (envelope-from asstec@matik.com.br) Received: from msrv.matik.com.br (msrv.matik.com.br [200.152.83.14]) by mx1.freebsd.org (Postfix) with ESMTP id 232BF13C4B9 for ; Fri, 30 Mar 2007 22:50:32 +0000 (UTC) (envelope-from asstec@matik.com.br) Received: from anc (nb-h.matik.com.br [200.152.88.34] (may be forged)) by msrv.matik.com.br (8.13.8/8.13.1) with ESMTP id l2UMoPtW012661 for ; Fri, 30 Mar 2007 19:50:26 -0300 (BRT) (envelope-from asstec@matik.com.br) From: AT Matik Organization: Infomatik To: freebsd-ipfw@freebsd.org Date: Fri, 30 Mar 2007 19:50:00 -0300 User-Agent: KMail/1.9.4 References: <460D75CE.70804@elischer.org> In-Reply-To: <460D75CE.70804@elischer.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200703301950.01501.asstec@matik.com.br> X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED,AWL,MR_DIFF_MID autolearn=unavailable version=3.1.8 X-Spam-Checker-Version: Antispam Datacenter Matik msrv.matik.com.br X-Virus-Scanned: ClamAV version 0.88.4, clamav-milter version 0.88.4 on msrv.matik.com.br X-Virus-Status: Clean Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 22:50:33 -0000 On Friday 30 March 2007 17:40, Julian Elischer wrote: > I have been looking at the IPFW code recently, especially > with respect to locking. > There are some things that could be done to improve IPFW's > behaviour when processing packets, but some of these take a > toll (there is always a toll) on the 'updating' side of things. > Hi ,=20 would you mind to explain your way of "add a toll", do you mean kind of pri= ce=20 for a benefit or something like that? Sorry I am not native american englis= h=20 speaker.=20 If I understand this right I would say that it does not matter for adding=20 rules, what is of interest is processing time when they exist already > Is there anyone out there who is adding hundreds (or even dozens) of rules > per second on a continuous basis, or who wants rule changing to > be a really efficient operation? even if ... I have a system which takes additional custom parms from rc.conf.=20 so lets say the admin configures a new IP or port he executes a script whic= h=20 flushes the old and executes the new rules it doesn't matter the time it takes to execute the new rules - what certain= ly=20 depends on machine capacities - what matters at the end is how fast the=20 machine can process the rules at run-time ... whatever it is .. as long as = it=20 is faster ... not building the rule set but running them under load > (does it matter to you if it takes a few milliSecs to add a rule?) absolutely NOT Jo=E3o=20 A mensagem foi scaneada pelo sistema de e-mail e pode ser considerada segura. Service fornecido pelo Datacenter Matik https://datacenter.matik.com.br From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 23:41:40 2007 Return-Path: X-Original-To: ipfw@freebsd.org Delivered-To: freebsd-ipfw@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 59BF416A402 for ; Fri, 30 Mar 2007 23:41:40 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outF.internet-mail-service.net (outF.internet-mail-service.net [216.240.47.229]) by mx1.freebsd.org (Postfix) with ESMTP id 477B313C44C for ; Fri, 30 Mar 2007 23:41:40 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Fri, 30 Mar 2007 16:12:24 -0700 Received: from [10.251.22.38] (nat.ironport.com [63.251.108.100]) by idiom.com (Postfix) with ESMTP id 9B2BE125D4D; Fri, 30 Mar 2007 16:41:39 -0700 (PDT) Message-ID: <460DA02D.8010509@elischer.org> Date: Fri, 30 Mar 2007 16:41:33 -0700 From: Julian Elischer User-Agent: Thunderbird 1.5.0.10 (Macintosh/20070221) MIME-Version: 1.0 To: Kevin Day References: <460D75CE.70804@elischer.org> <98FE1AAF-DF19-43A8-A5B6-010C852AF489@dragondata.com> In-Reply-To: <98FE1AAF-DF19-43A8-A5B6-010C852AF489@dragondata.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , ipfw@freebsd.org Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 23:41:40 -0000 Kevin Day wrote: > > On Mar 30, 2007, at 3:40 PM, Julian Elischer wrote: > >> I have been looking at the IPFW code recently, especially with respect >> to locking. >> There are some things that could be done to improve IPFW's behaviour >> when processing packets, but some of these take a >> toll (there is always a toll) on the 'updating' side of things. >> >> For example. I can make IPFW lock-free during processing of packets >> (i.e. not holding any locks while traversing the >> list) which would solve problems we have with lock-order reversals >> when it needs to look at the socket layer (which needs socket layer >> locks). Unfortunatly this would make it a lot more expensive >> in the case where new rules are being added to the list. possibly a LOT >> more expensive. Now, this would only matter if one was adding (or >> deleting) >> hundreds of rules per second to the firewall, but as I've discovered, >> there's always SOMEONE that is doing the very thing you imagine that >> no-one would ever do. >> >> In my imagination, most of the people who did this sort of thing don't >> need to do it any more as tables obviate the need for that sort of thing. >> >> Is there anyone out there who is adding hundreds (or even dozens) of >> rules >> per second on a continuous basis, or who wants rule changing to >> be a really efficient operation? >> (does it matter to you if it takes a few milliSecs to add a rule?) >> >> Julian > > Would this apply to "dynamic rules", using the keep-state keyword? > That'd be a killer for us. > no, just to the main firewall list where rules a re put manually. > If not, the only problem I'd have is that my ipfw startup script adds > about 20,000 rules on a reboot. 20,000 rules multiplied by any > significant amount of time would be bad, just from a > reboot-recovery-time angle. But, if it improved overall performance, I > probably wouldn't mind too much. :) now, what could I add to the firewall to make that come down to, say, 100 rules? I have the following in my list of things to add: ** adding of 'variables (registers) to hold values for the duration of the filter run. ** ipfw add 100 set (register number) value ipfw add 100 set (register number) tablearg ip from table (x) to any ipfw [action] if (register number) gt value (lt,le,ge,eq,neq) ipfw [action] if (register number) in table (x) (registers and table values can be addresses) ** computed skipto ** ipfw skipto tablearg ip from table (2) to any ** adding items to tables automatically ** ipfw loadto table 3 (source, value) ip from any to table (3) ** ability to select WHICH table arg ** ipfw skipto tablearg2 ip from table (1) to table (2) > From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 23:50:50 2007 Return-Path: X-Original-To: ipfw@freebsd.org Delivered-To: freebsd-ipfw@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 54FAB16A408 for ; Fri, 30 Mar 2007 23:50:50 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outY.internet-mail-service.net (outY.internet-mail-service.net [216.240.47.248]) by mx1.freebsd.org (Postfix) with ESMTP id 43D7513C4AE for ; Fri, 30 Mar 2007 23:50:50 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Fri, 30 Mar 2007 16:21:34 -0700 Received: from [10.251.22.38] (nat.ironport.com [63.251.108.100]) by idiom.com (Postfix) with ESMTP id 2E367125D68; Fri, 30 Mar 2007 16:50:49 -0700 (PDT) Message-ID: <460DA258.2030402@elischer.org> Date: Fri, 30 Mar 2007 16:50:48 -0700 From: Julian Elischer User-Agent: Thunderbird 1.5.0.10 (Macintosh/20070221) MIME-Version: 1.0 To: Luigi Rizzo References: <460D75CE.70804@elischer.org> <20070330145938.A88154@xorpc.icir.org> In-Reply-To: <20070330145938.A88154@xorpc.icir.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , ipfw@freebsd.org Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 23:50:50 -0000 Luigi Rizzo wrote: > On Fri, Mar 30, 2007 at 01:40:46PM -0700, Julian Elischer wrote: >> I have been looking at the IPFW code recently, especially >> with respect to locking. >> There are some things that could be done to improve IPFW's >> behaviour when processing packets, but some of these take a >> toll (there is always a toll) on the 'updating' side of things. > > certainly ipfw was not designed with SMP in mind. > If you can tell us what is your plan to make the list lock free > (which one, the static or dynamic ones ?) maybe we can comment more. > > E.g. one option could be the usual trick of adding refcounts to > the individual rules, and then using an array of pointers to them. > While processing you grab a refcount to the array, and release it once > done with the packet. If there is an addition or removal, you duplicate > the array (which may be expensive for the large 20k rules mentioned), > manipulate the copy and then atomically swap the pointers to the head. This is pretty close.. I know I've mentioned this to people several times over the last year or so. the trick is to try do it in a way that the average packet doesn't need to do any locks to get in and the updater does more work. if you are willing to acquire a lock on both starting and ending the run through the firewall it is easy. (I already have code to do that..) (see http://www.freebsd.org/~julian/atomic_replace.c (untested but probably close.) doing it without requiring that each packet get those locks however is a whole new level of problem. > > This might even work for dynamic rules as the lists (the content of > each hash bucket) are typically short. yep > > cheers > luigi From owner-freebsd-ipfw@FreeBSD.ORG Fri Mar 30 23:57:15 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 451A416A402 for ; Fri, 30 Mar 2007 23:57:15 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outJ.internet-mail-service.net (outJ.internet-mail-service.net [216.240.47.233]) by mx1.freebsd.org (Postfix) with ESMTP id 3494813C489 for ; Fri, 30 Mar 2007 23:57:15 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Fri, 30 Mar 2007 16:16:17 -0700 Received: from [10.251.22.38] (nat.ironport.com [63.251.108.100]) by idiom.com (Postfix) with ESMTP id 911D0125D56; Fri, 30 Mar 2007 16:45:32 -0700 (PDT) Message-ID: <460DA11B.9060401@elischer.org> Date: Fri, 30 Mar 2007 16:45:31 -0700 From: Julian Elischer User-Agent: Thunderbird 1.5.0.10 (Macintosh/20070221) MIME-Version: 1.0 To: Freddie Cash References: <460D75CE.70804@elischer.org> <200703301421.29919.fcash@ocis.net> In-Reply-To: <200703301421.29919.fcash@ocis.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-ipfw@freebsd.org Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 23:57:15 -0000 Freddie Cash wrote: > On Friday 30 March 2007 01:40 pm, Julian Elischer wrote: >> I have been looking at the IPFW code recently, especially >> with respect to locking. >> There are some things that could be done to improve IPFW's >> behaviour when processing packets, but some of these take a >> toll (there is always a toll) on the 'updating' side of things. >> >> For example. I can make IPFW lock-free during processing >> of packets (i.e. not holding any locks while traversing the >> list) which would solve problems we have with lock-order reversals >> when it needs to look at the socket layer (which needs socket >> layer locks). Unfortunatly this would make it a lot more expensive >> in the case where new rules are being added to the list. possibly a LOT >> more expensive. Now, this would only matter if one was adding (or >> deleting) hundreds of rules per second to the firewall, but as I've >> discovered, there's always SOMEONE that is doing the very thing you >> imagine that no-one would ever do. >> >> In my imagination, most of the people who did this sort of thing don't >> need to do it any more as tables obviate the need for that sort of >> thing. >> >> Is there anyone out there who is adding hundreds (or even dozens) of >> rules per second on a continuous basis, or who wants rule changing to >> be a really efficient operation? >> (does it matter to you if it takes a few milliSecs to add a rule?) > > If the initial loading of a ruleset via a script counts, then yes. We add > 600+ rules each time the script is run. During testing, or when > troubleshooting connection issues, the rules could be reloaded several > times over 10 minutes. We've moved away from adding rules dynamically, > preferring to add rules to the script and reload them all. Keeps the > rules in memory in sync with the rules on disk. > > Otherwise, no. :) > the new rules might take a second or two to load.. a load may take a couple of milliseconds vs less than that for now. From owner-freebsd-ipfw@FreeBSD.ORG Sat Mar 31 00:07:15 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BE59916A407 for ; Sat, 31 Mar 2007 00:07:15 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outM.internet-mail-service.net (outM.internet-mail-service.net [216.240.47.236]) by mx1.freebsd.org (Postfix) with ESMTP id A932013C4BD for ; Sat, 31 Mar 2007 00:07:15 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Fri, 30 Mar 2007 16:23:23 -0700 Received: from [10.251.22.38] (nat.ironport.com [63.251.108.100]) by idiom.com (Postfix) with ESMTP id CC322125ADD; Fri, 30 Mar 2007 16:52:38 -0700 (PDT) Message-ID: <460DA2C6.7060205@elischer.org> Date: Fri, 30 Mar 2007 16:52:38 -0700 From: Julian Elischer User-Agent: Thunderbird 1.5.0.10 (Macintosh/20070221) MIME-Version: 1.0 To: AT Matik References: <460D75CE.70804@elischer.org> <200703301950.01501.asstec@matik.com.br> In-Reply-To: <200703301950.01501.asstec@matik.com.br> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-ipfw@freebsd.org Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2007 00:07:15 -0000 AT Matik wrote: > On Friday 30 March 2007 17:40, Julian Elischer wrote: >> I have been looking at the IPFW code recently, especially >> with respect to locking. >> There are some things that could be done to improve IPFW's >> behaviour when processing packets, but some of these take a >> toll (there is always a toll) on the 'updating' side of things. >> > > Hi , > would you mind to explain your way of "add a toll", do you mean kind of price > for a benefit or something like that? Sorry I am not native american english > speaker. a toll is a cost. so, yes (toll is not just American English) > > If I understand this right I would say that it does not matter for adding > rules, what is of interest is processing time when they exist already exactly.. > >> Is there anyone out there who is adding hundreds (or even dozens) of rules >> per second on a continuous basis, or who wants rule changing to >> be a really efficient operation? > > even if ... > I have a system which takes additional custom parms from rc.conf. > so lets say the admin configures a new IP or port he executes a script which > flushes the old and executes the new rules > > it doesn't matter the time it takes to execute the new rules - what certainly > depends on machine capacities - what matters at the end is how fast the > machine can process the rules at run-time ... whatever it is .. as long as it > is faster ... not building the rule set but running them under load > >> (does it matter to you if it takes a few milliSecs to add a rule?) > > absolutely NOT thankyou > > > João > > > > > > > > A mensagem foi scaneada pelo sistema de e-mail e pode ser considerada segura. > Service fornecido pelo Datacenter Matik https://datacenter.matik.com.br > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" From owner-freebsd-ipfw@FreeBSD.ORG Sat Mar 31 00:28:20 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6726D16A401 for ; Sat, 31 Mar 2007 00:28:20 +0000 (UTC) (envelope-from Peter_Losher@isc.org) Received: from mx.isc.org (mx.isc.org [204.152.184.167]) by mx1.freebsd.org (Postfix) with ESMTP id 4E99413C457 for ; Sat, 31 Mar 2007 00:28:20 +0000 (UTC) (envelope-from Peter_Losher@isc.org) Received: from farside.isc.org (farside.isc.org [IPv6:2001:4f8:3:bb::5]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "farside.isc.org", Issuer "ISC CA" (verified OK)) by mx.isc.org (Postfix) with ESMTP id 3891F11401E for ; Fri, 30 Mar 2007 23:47:37 +0000 (UTC) (envelope-from Peter_Losher@isc.org) Received: from [IPv6:2001:4f8:3:bb::37] (tardis.isc.org [IPv6:2001:4f8:3:bb::37]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by farside.isc.org (Postfix) with ESMTP id 20294E6086 for ; Fri, 30 Mar 2007 23:47:37 +0000 (UTC) (envelope-from Peter_Losher@isc.org) Message-ID: <460DA198.2090506@isc.org> Date: Fri, 30 Mar 2007 16:47:36 -0700 From: Peter Losher Organization: ISC User-Agent: Thunderbird 1.5.0.10 (Macintosh/20070221) MIME-Version: 1.0 To: freebsd-ipfw@freebsd.org X-Enigmail-Version: 0.94.0.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig362F21D918D69851FACDCBD8" Subject: IPv6+dummynet causing panic on 6.2-RELEASE X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2007 00:28:20 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig362F21D918D69851FACDCBD8 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable We have been having rampant issues using Dummynet's IPv6 support, and it's been causing panic's every 24-48 hours. Enabled WITNESS and BREAK_TO_DEBUGGER, and this is the result. -=3D- lock order reversal: (sleepable after non-sleepable) 1st 0xffffff034809c900 rtentry (rtentry) @ /usr/src/sys/netinet6/ip6_input.c:501 2nd 0xffffffff808dda70 user map (user map) @ /usr/src/sys/vm/vm_map.c:30= 74 KDB: stack backtrace: witness_checkorder() at witness_checkorder+0x48a _sx_xlock() at _sx_xlock+0x3e vm_map_lookup() at vm_map_lookup+0x44 vm_fault() at vm_fault+0xba trap_pfault() at trap_pfault+0x13c trap() at trap+0x1bd calltrap() at calltrap+0x5 --- trap 0xc, rip =3D 0xffffffff804c41f7, rsp =3D 0xffffffffbdf0da60, rbp= =3D 0xffffffffbdf0daf0 --- ip6_input() at ip6_input+0xa07 dummynet_send() at dummynet_send+0x17e dummynet() at dummynet+0x21a softclock() at softclock+0x19a ithread_loop() at ithread_loop+0x132 fork_exit() at fork_exit+0x87 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffffffbdf0dd00, rbp =3D 0 --- Fatal trap 12: page fault while in kernel mode cpuid =3D 2; apic id =3D 06 fault virtual address =3D 0x98 fault code =3D supervisor read, page not present instruction pointer =3D 0x8:0xffffffff804c41f7 stack pointer =3D 0x10:0xffffffffbdf0da60 frame pointer =3D 0x10:0xffffffffbdf0daf0 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 15 (swi4: clock sio) [thread pid 15 tid 100009 ] Stopped at ip6_input+0xa07: movq 0x98(%rdi),%rax db> tr Tracing pid 15 tid 100009 td 0xffffff040ff3b000 ip6_input() at ip6_input+0xa07 dummynet_send() at dummynet_send+0x17e dummynet() at dummynet+0x21a softclock() at softclock+0x19a ithread_loop() at ithread_loop+0x132 fork_exit() at fork_exit+0x87 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffffffbdf0dd00, rbp =3D 0 --- -=3D- Any ideas how to proceed? Best Wishes - Peter --=20 Peter_Losher@isc.org | ISC | OpenPGP 0xE8048D08 | "The bits must flow" --------------enig362F21D918D69851FACDCBD8 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (Darwin) iD8DBQFGDaGYPtVx9OgEjQgRAtzCAJ4iDNq+5u9lL99vR86YHA0rbKwhpACglRoB yceTpahHkBjH+5UE/aM8iko= =HR7d -----END PGP SIGNATURE----- --------------enig362F21D918D69851FACDCBD8-- From owner-freebsd-ipfw@FreeBSD.ORG Sat Mar 31 05:36:04 2007 Return-Path: X-Original-To: ipfw@freebsd.org Delivered-To: freebsd-ipfw@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 48B4E16A402 for ; Sat, 31 Mar 2007 05:36:04 +0000 (UTC) (envelope-from fcash@ocis.net) Received: from smtp.sd73.bc.ca (smtp.sd73.bc.ca [142.24.13.140]) by mx1.freebsd.org (Postfix) with ESMTP id 1F57413C455 for ; Sat, 31 Mar 2007 05:36:04 +0000 (UTC) (envelope-from fcash@ocis.net) Received: from localhost (localhost [127.0.0.1]) by localhost.sd73.bc.ca (Postfix) with ESMTP id 5CE3F1A000B11; Fri, 30 Mar 2007 22:09:04 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at smtp.sd73.bc.ca Received: from smtp.sd73.bc.ca ([127.0.0.1]) by localhost (smtp.sd73.bc.ca [127.0.0.1]) (amavisd-new, port 10024) with LMTP id EyOwgC+e7DHp; Fri, 30 Mar 2007 22:09:02 -0700 (PDT) Received: from webmail.sd73.bc.ca (webmail.sd73.bc.ca [10.10.10.17]) by smtp.sd73.bc.ca (Postfix) with ESMTP id B62831A000B0D; Fri, 30 Mar 2007 22:09:02 -0700 (PDT) Received: from 192.168.0.102 (SquirrelMail authenticated user fcash) by webmail.sd73.bc.ca with HTTP; Fri, 30 Mar 2007 22:09:02 -0700 (PDT) Message-ID: <3750.192.168.0.102.1175317742.squirrel@webmail.sd73.bc.ca> In-Reply-To: <460DA11B.9060401@elischer.org> References: <460D75CE.70804@elischer.org> <200703301421.29919.fcash@ocis.net> <460DA11B.9060401@elischer.org> Date: Fri, 30 Mar 2007 22:09:02 -0700 (PDT) From: "Freddie Cash" To: ipfw@freebsd.org User-Agent: SquirrelMail/1.5.1 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit Cc: Julian Elischer Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2007 05:36:04 -0000 On Fri, March 30, 2007 4:45 pm, Julian Elischer wrote: > Freddie Cash wrote: >> On Friday 30 March 2007 01:40 pm, Julian Elischer wrote: >> >>> I have been looking at the IPFW code recently, especially >>> with respect to locking. There are some things that could be done >>> to improve IPFW's behaviour when processing packets, but some of >>> these take a toll (there is always a toll) on the 'updating' side >>> of things. >>> >>> For example. I can make IPFW lock-free during processing >>> of packets (i.e. not holding any locks while traversing the list) >>> which would solve problems we have with lock-order reversals when >>> it needs to look at the socket layer (which needs socket layer >>> locks). Unfortunatly this would make it a lot more expensive in >>> the case where new rules are being added to the list. possibly a >>> LOT >>> more expensive. Now, this would only matter if one was adding (or >>> deleting) hundreds of rules per second to the firewall, but as >>> I've >>> discovered, there's always SOMEONE that is doing the very thing >>> you imagine that no-one would ever do. >>> >>> In my imagination, most of the people who did this sort of thing >>> don't need to do it any more as tables obviate the need for that >>> sort of thing. >>> >>> Is there anyone out there who is adding hundreds (or even dozens) >>> of rules per second on a continuous basis, or who wants rule >>> changing to be a really efficient operation? (does it matter to you >>> if it takes a few milliSecs to add a rule?) >> >> If the initial loading of a ruleset via a script counts, then yes. >> We add >> 600+ rules each time the script is run. During testing, or when >> troubleshooting connection issues, the rules could be reloaded >> several times over 10 minutes. We've moved away from adding rules >> dynamically, preferring to add rules to the script and reload them >> all. Keeps the rules in memory in sync with the rules on disk. >> >> Otherwise, no. :) > > the new rules might take a second or two to load.. a load may take a > couple of milliseconds vs less than that for now. That won't adversely affect us, then. We use hostnames in a few rules, so there's a delay the first time the rules are loaded anyway. A second or two extra won't harm us. :) ---- Freddie Cash fcash@ocis.net From owner-freebsd-ipfw@FreeBSD.ORG Sat Mar 31 08:47:44 2007 Return-Path: X-Original-To: ipfw@freebsd.org Delivered-To: freebsd-ipfw@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4CB2516A404 for ; Sat, 31 Mar 2007 08:47:44 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 1A3BC13C48A for ; Sat, 31 Mar 2007 08:47:40 +0000 (UTC) (envelope-from andre@freebsd.org) Received: (qmail 75338 invoked from network); 31 Mar 2007 07:48:11 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 31 Mar 2007 07:48:11 -0000 Message-ID: <460E19EE.3020700@freebsd.org> Date: Sat, 31 Mar 2007 10:21:02 +0200 From: Andre Oppermann User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: Julian Elischer References: <460D75CE.70804@elischer.org> <20070330145938.A88154@xorpc.icir.org> <460DA258.2030402@elischer.org> In-Reply-To: <460DA258.2030402@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Luigi Rizzo , ipfw@freebsd.org, FreeBSD Net Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2007 08:47:44 -0000 Julian Elischer wrote: > Luigi Rizzo wrote: >> On Fri, Mar 30, 2007 at 01:40:46PM -0700, Julian Elischer wrote: >>> I have been looking at the IPFW code recently, especially with >>> respect to locking. >>> There are some things that could be done to improve IPFW's behaviour >>> when processing packets, but some of these take a >>> toll (there is always a toll) on the 'updating' side of things. >> >> certainly ipfw was not designed with SMP in mind. If you can tell us >> what is your plan to make the list lock free >> (which one, the static or dynamic ones ?) maybe we can comment more. >> >> E.g. one option could be the usual trick of adding refcounts to >> the individual rules, and then using an array of pointers to them. >> While processing you grab a refcount to the array, and release it once >> done with the packet. If there is an addition or removal, you duplicate >> the array (which may be expensive for the large 20k rules mentioned), >> manipulate the copy and then atomically swap the pointers to the head. > > This is pretty close.. I know I've mentioned this to people several > times over > the last year or so. the trick is to try do it in a way that the average > packet > doesn't need to do any locks to get in and the updater does more work. > if you are willing to acquire a lock on both starting and ending > the run through the firewall it is easy. > (I already have code to do that..) > (see http://www.freebsd.org/~julian/atomic_replace.c (untested but > probably close.) > doing it without requiring that each packet get those locks however is a > whole new level of problem. The locking overhead per packet in ipfw is by no means its limiting factor. Actually it's a very small part and pretty much any work on it is lost love. It would be much better spent time to optimize the main rule loop of ipfw to speed things up. I was profiling ipfw early last year with an Agilent packet generator and hwpmc. In the meantime the packet forwarding path (w/o ipfw) has been improved but relative to each other the number are still correct. Numbers pre-taskqueue improvements from early 2006: fastfwd 580357 pps fastfwd+pfil_pass 565477 pps (no rules, just pass packet on) fastfwd+ipfw_allow 505952 pps (one rule) fastfwd+ipfw_30rules 401768 pps (30 IP address non-matching rules) fastfwd+pf_pass 476190 pps (one rule) fastfwd+pf_30rules 342262 pps (30 IP address non-matching rules) The overhead per packet is big. Enabling of ipfw and the pfil/ipfw per packet and their indirect function calls cause a loss of only about 15'000 pps (0.9%). On the other hand the first rule costs 12.9% and each additional rule 0.6%. All this is without any complex rules like table lookups, state tracking, etc. idle fastfwd fastfwd+ipfw_allow fastfwd+ipfw_30rules cycles 2596685731 2598214743 2597973265 2596702381 cpu-clk-unhalted 7824023 2582240847 2518187670 2483904362 instructions 2317535 1324655330 1492363346 2026009148 branches 316786 174329367 191263118 294700024 branch-mispredicts 19757 2235749 10003461 8848407 dc-access 1417532 829159482 998427224 1235192770 dc-refill-from-l2 2124 4767395 4346738 4548311 dc-refill-from-system 89 803102 819658 654661 dtlb-l2-hit 626 10435843 9304448 12352018 dtlb-miss 129 255493 130998 112644 ic-fetch 804423 471138619 583149432 870371492 ic-miss 2358 34831 2505198 1947943 itlb-l2-hit 0 74 12 12 itlb-miss 42 92 82 82 lock-cycles 77 803 352 451 locked-instructions 4 19 2 4 lock-dc-access 6 20 6 7 lock-dc-miss 0 0 0 0 Hardware is a dual Opteron 852 at 2.6GHz on a Tyan 2882 mainboard with a dual Intel em network card plugged into a PCI64-133 slot. Packets are flowing from em0 -> em1. -- Andre From owner-freebsd-ipfw@FreeBSD.ORG Sat Mar 31 09:27:43 2007 Return-Path: X-Original-To: ipfw@freebsd.org Delivered-To: freebsd-ipfw@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4028516A403; Sat, 31 Mar 2007 09:27:43 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.freebsd.org (Postfix) with ESMTP id 0FD8D13C4BA; Sat, 31 Mar 2007 09:27:42 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.13.6) with ESMTP id l2V9Rfan095305; Sat, 31 Mar 2007 01:27:41 -0800 (PST) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id l2V9Rf54095304; Sat, 31 Mar 2007 02:27:41 -0700 (PDT) (envelope-from rizzo) Date: Sat, 31 Mar 2007 02:27:41 -0700 From: Luigi Rizzo To: Andre Oppermann Message-ID: <20070331022741.A94927@xorpc.icir.org> References: <460D75CE.70804@elischer.org> <20070330145938.A88154@xorpc.icir.org> <460DA258.2030402@elischer.org> <460E19EE.3020700@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <460E19EE.3020700@freebsd.org>; from andre@freebsd.org on Sat, Mar 31, 2007 at 10:21:02AM +0200 Cc: FreeBSD Net , ipfw@freebsd.org, Julian Elischer Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2007 09:27:43 -0000 On Sat, Mar 31, 2007 at 10:21:02AM +0200, Andre Oppermann wrote: > Julian Elischer wrote: > > Luigi Rizzo wrote: > >> On Fri, Mar 30, 2007 at 01:40:46PM -0700, Julian Elischer wrote: > >>> I have been looking at the IPFW code recently, especially with > >>> respect to locking. > >>> There are some things that could be done to improve IPFW's behaviour ... > The locking overhead per packet in ipfw is by no means its limiting i think you and Julian are looking at different issues. if i understand julian's comment, the problem is that the list is protected by a single lock, so no hope of parallelising the work, and if one kernel thread is busy processing a packet in the filter, others might be blocked for a long time (in your case, the set of 30 rules is 765ns for ipfw and 1198ns for pf). Your tests presumably have little if any contention on the lock. Specifically, if you compute the difference of the inverses of those pps rates you see the following: +pfil_pass 45.3 ns per packet +ipfw_allow +253.4 ns/packet (setup and first rule) +ipfw_30 +17.67 ns/(packet * extra rule) +pf_pass +376.9 ns/packet (setup and first rule) +pf_30 +28.34 ns/(packet * extra rule) the lock acquisition cost is in the 'setup' part but i cannot tell how expensive it is. Julian's suggested change (and surely the one i described) replaces the lock/unlock pair on the rule list with a refcount add/dec pair (with uncontested locks the cost should be similar), but especially makes the operation non-blocking allowing running the input and output paths in parallel. > factor. Actually it's a very small part and pretty much any work on > it is lost love. It would be much better spent time to optimize the > main rule loop of ipfw to speed things up. I was profiling ipfw early > last year with an Agilent packet generator and hwpmc. In the meantime > the packet forwarding path (w/o ipfw) has been improved but relative > to each other the number are still correct. actually your numbers show that at least the rule setup (and the processing of simple rules) is significantly faster (50% or so) in ipfw2 than in pf. I know that the setup time is expensive, but i am not sure that one can save much - in both cases, you need to fetching a lot of information, which is scattered in variable locations in the mbuf and packet headers. cheers luigi From owner-freebsd-ipfw@FreeBSD.ORG Sat Mar 31 11:00:57 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8A1F416A404 for ; Sat, 31 Mar 2007 11:00:57 +0000 (UTC) (envelope-from max@love2party.net) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.174]) by mx1.freebsd.org (Postfix) with ESMTP id 1D7F713C44B for ; Sat, 31 Mar 2007 11:00:57 +0000 (UTC) (envelope-from max@love2party.net) Received: from [88.64.190.188] (helo=amd64.laiers.local) by mrelayeu.kundenserver.de (node=mrelayeu8) with ESMTP (Nemesis), id 0ML31I-1HXb7E1nZm-00065J; Sat, 31 Mar 2007 12:47:25 +0200 From: Max Laier Organization: FreeBSD To: freebsd-ipfw@freebsd.org Date: Sat, 31 Mar 2007 11:47:12 +0100 User-Agent: KMail/1.9.5 References: <460D75CE.70804@elischer.org> <460E19EE.3020700@freebsd.org> <20070331022741.A94927@xorpc.icir.org> In-Reply-To: <20070331022741.A94927@xorpc.icir.org> X-Face: ,,8R(x[kmU]tKN@>gtH1yQE4aslGdu+2]; R]*pL,U>^H?)gW@49@wdJ`H<=?utf-8?q?=25=7D*=5FBD=0A=09U=5For=3D=5CmOZf764=26nYj=3DJYbR1PW0ud?=>|!~,,CPC.1-D$FG@0h3#'5"k{V]a~.<=?utf-8?q?mZ=7D44=23Se=7Em=0A=09Fe=7E=5C=5DX5B=5D=5Fxj?=(ykz9QKMw_l0C2AQ]}Ym8)fU MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1702159.HoLqzXUo7s"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200703311247.19940.max@love2party.net> X-Provags-ID: V01U2FsdGVkX18gzQ6r4Bp+mzp9HZNxTgLd2uL9toh+JlSVLwz Us1+F/8zYnpilPJtbFSEAvREfvXGMHWv4dvNi1LdddDJRjYkKN zJGUlDTmEDKAxLAhrbMaQ== Cc: Luigi Rizzo , Andre Oppermann , Julian Elischer , FreeBSD Net Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2007 11:00:57 -0000 --nextPart1702159.HoLqzXUo7s Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Saturday 31 March 2007 11:27, Luigi Rizzo wrote: > On Sat, Mar 31, 2007 at 10:21:02AM +0200, Andre Oppermann wrote: > > Julian Elischer wrote: > > > Luigi Rizzo wrote: > > >> On Fri, Mar 30, 2007 at 01:40:46PM -0700, Julian Elischer wrote: > > >>> I have been looking at the IPFW code recently, especially with > > >>> respect to locking. > > >>> There are some things that could be done to improve IPFW's > > >>> behaviour > > ... > > > The locking overhead per packet in ipfw is by no means its limiting > > i think you and Julian are looking at different issues. > if i understand julian's comment, the problem is that the list > is protected by a single lock, so no hope of parallelising ipfw uses rwlocks for the static rules quite some time now. In contrast=20 to Julian, I don't believe that the claimed lock order reversal with a=20 rlock() can be the cause of a deadlock (exclusiveness is a precondition). = =20 Haveing been involved in the hacks that went in and out of ipfw and pfil=20 locking over the last few years and the problems that went along with it,=20 I'd urge everybody to *not* rush any more hacks into this. > the work, and if one kernel thread is busy processing a packet > in the filter, others might be blocked for a long time > (in your case, the set of 30 rules is 765ns for ipfw and 1198ns > for pf). > > Your tests presumably have little if any contention on the lock. Most likely none at all, since the forwarding path takes care of=20 serialization. > Specifically, if you compute the difference of the inverses > of those pps rates you see the following: > > +pfil_pass 45.3 ns per packet > > +ipfw_allow +253.4 ns/packet (setup and first rule) > +ipfw_30 +17.67 ns/(packet * extra rule) > > +pf_pass +376.9 ns/packet (setup and first rule) > +pf_30 +28.34 ns/(packet * extra rule) > > > the lock acquisition cost is in the 'setup' part but i cannot tell > how expensive it is. > Julian's suggested change (and surely the one i described) > replaces the lock/unlock pair on the rule list with a refcount add/dec > pair (with uncontested locks the cost should be similar), but > especially makes the operation non-blocking allowing running the input > and output paths in parallel. See above, ipfw is working in parallel already. In addition to that,=20 using a ref-count would be worse! Instead of two atomic operations you'd=20 then have to pay for four: lock ref unlock work lock unref unlock All of=20 which can contentend each other. This will most likely cause more=20 serialization than we currently have. Again, please don't rush any=20 hacks! > > factor. Actually it's a very small part and pretty much any work on > > it is lost love. It would be much better spent time to optimize the > > main rule loop of ipfw to speed things up. I was profiling ipfw > > early last year with an Agilent packet generator and hwpmc. In the > > meantime the packet forwarding path (w/o ipfw) has been improved but > > relative to each other the number are still correct. > > actually your numbers show that at least the rule setup (and the > processing of simple rules) is significantly faster (50% or so) in > ipfw2 than in pf. Note that pf includes a plethora of sanity checks in the default rule=20 processing. Also note that pf - due to it's stateful design - does=20 a "check state" first for every packet. This gives a big mallus in this=20 test special test. > I know that the setup time is expensive, but i am not sure that > one can save much - in both cases, you need to fetching a lot > of information, which is scattered in variable locations in > the mbuf and packet headers. Agreed. For the ipfw case it *might* make sense to reach into the upper=20 layers only if requested - not at all sure about that, however. =2D-=20 /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News --nextPart1702159.HoLqzXUo7s Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQBGDjw3XyyEoT62BG0RAovAAJ9/eLdEZfjHiEEwICMt4CQTGbH3BwCePJgT 4nAXEG6PMYYNEMmLp+gg1e8= =E5Tl -----END PGP SIGNATURE----- --nextPart1702159.HoLqzXUo7s-- From owner-freebsd-ipfw@FreeBSD.ORG Sat Mar 31 16:04:03 2007 Return-Path: X-Original-To: ipfw@freebsd.org Delivered-To: freebsd-ipfw@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 377C316A407 for ; Sat, 31 Mar 2007 16:04:03 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outP.internet-mail-service.net (outP.internet-mail-service.net [216.240.47.239]) by mx1.freebsd.org (Postfix) with ESMTP id 22C3113C4B8 for ; Sat, 31 Mar 2007 16:04:02 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Sat, 31 Mar 2007 08:34:26 -0700 Received: from [192.168.2.4] (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 60180125ADE; Sat, 31 Mar 2007 09:03:47 -0700 (PDT) Message-ID: <460E8663.9040309@elischer.org> Date: Sat, 31 Mar 2007 09:03:47 -0700 From: Julian Elischer User-Agent: Thunderbird 1.5.0.10 (Macintosh/20070221) MIME-Version: 1.0 To: Andre Oppermann References: <460D75CE.70804@elischer.org> <20070330145938.A88154@xorpc.icir.org> <460DA258.2030402@elischer.org> <460E19EE.3020700@freebsd.org> In-Reply-To: <460E19EE.3020700@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Luigi Rizzo , ipfw@freebsd.org, FreeBSD Net Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2007 16:04:03 -0000 Thanks for the information.. The main thrust for me is to make it not hold any locks during processing. performance is 2nd Andre Oppermann wrote: > Julian Elischer wrote: >> Luigi Rizzo wrote: >>> On Fri, Mar 30, 2007 at 01:40:46PM -0700, Julian Elischer wrote: >>>> I have been looking at the IPFW code recently, especially with >>>> respect to locking. >>>> There are some things that could be done to improve IPFW's behaviour >>>> when processing packets, but some of these take a >>>> toll (there is always a toll) on the 'updating' side of things. >>> >>> certainly ipfw was not designed with SMP in mind. If you can tell us >>> what is your plan to make the list lock free >>> (which one, the static or dynamic ones ?) maybe we can comment more. >>> >>> E.g. one option could be the usual trick of adding refcounts to >>> the individual rules, and then using an array of pointers to them. >>> While processing you grab a refcount to the array, and release it once >>> done with the packet. If there is an addition or removal, you duplicate >>> the array (which may be expensive for the large 20k rules mentioned), >>> manipulate the copy and then atomically swap the pointers to the head. >> >> This is pretty close.. I know I've mentioned this to people several >> times over >> the last year or so. the trick is to try do it in a way that the >> average packet >> doesn't need to do any locks to get in and the updater does more work. >> if you are willing to acquire a lock on both starting and ending >> the run through the firewall it is easy. >> (I already have code to do that..) >> (see http://www.freebsd.org/~julian/atomic_replace.c (untested but >> probably close.) >> doing it without requiring that each packet get those locks however is >> a whole new level of problem. > > The locking overhead per packet in ipfw is by no means its limiting > factor. Actually it's a very small part and pretty much any work on > it is lost love. It would be much better spent time to optimize the > main rule loop of ipfw to speed things up. I was profiling ipfw early > last year with an Agilent packet generator and hwpmc. In the meantime > the packet forwarding path (w/o ipfw) has been improved but relative > to each other the number are still correct. > > Numbers pre-taskqueue improvements from early 2006: > fastfwd 580357 pps > fastfwd+pfil_pass 565477 pps (no rules, just pass packet on) > fastfwd+ipfw_allow 505952 pps (one rule) > fastfwd+ipfw_30rules 401768 pps (30 IP address non-matching rules) > fastfwd+pf_pass 476190 pps (one rule) > fastfwd+pf_30rules 342262 pps (30 IP address non-matching rules) > > The overhead per packet is big. Enabling of ipfw and the pfil/ipfw > per packet and their indirect function calls cause a loss of only > about 15'000 pps (0.9%). On the other hand the first rule costs 12.9% > and each additional rule 0.6%. All this is without any complex rules > like table lookups, state tracking, etc. > > idle fastfwd fastfwd+ipfw_allow fastfwd+ipfw_30rules > cycles 2596685731 2598214743 2597973265 2596702381 > cpu-clk-unhalted 7824023 2582240847 2518187670 2483904362 > instructions 2317535 1324655330 1492363346 2026009148 > branches 316786 174329367 191263118 294700024 > branch-mispredicts 19757 2235749 10003461 8848407 > dc-access 1417532 829159482 998427224 1235192770 > dc-refill-from-l2 2124 4767395 4346738 4548311 > dc-refill-from-system 89 803102 819658 654661 > dtlb-l2-hit 626 10435843 9304448 12352018 > dtlb-miss 129 255493 130998 112644 > ic-fetch 804423 471138619 583149432 870371492 > ic-miss 2358 34831 2505198 1947943 > itlb-l2-hit 0 74 12 12 > itlb-miss 42 92 82 82 > lock-cycles 77 803 352 451 > locked-instructions 4 19 2 4 > lock-dc-access 6 20 6 7 > lock-dc-miss 0 0 0 0 > > Hardware is a dual Opteron 852 at 2.6GHz on a Tyan 2882 mainboard with > a dual Intel em network card plugged into a PCI64-133 slot. Packets > are flowing from em0 -> em1. > From owner-freebsd-ipfw@FreeBSD.ORG Sat Mar 31 16:06:16 2007 Return-Path: X-Original-To: ipfw@freebsd.org Delivered-To: freebsd-ipfw@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E181916A40A for ; Sat, 31 Mar 2007 16:06:16 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outS.internet-mail-service.net (outS.internet-mail-service.net [216.240.47.242]) by mx1.freebsd.org (Postfix) with ESMTP id C988113C46C for ; Sat, 31 Mar 2007 16:06:16 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Sat, 31 Mar 2007 08:36:54 -0700 Received: from [192.168.2.4] (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 617F7125B02; Sat, 31 Mar 2007 09:06:15 -0700 (PDT) Message-ID: <460E86F7.9050104@elischer.org> Date: Sat, 31 Mar 2007 09:06:15 -0700 From: Julian Elischer User-Agent: Thunderbird 1.5.0.10 (Macintosh/20070221) MIME-Version: 1.0 To: Luigi Rizzo References: <460D75CE.70804@elischer.org> <20070330145938.A88154@xorpc.icir.org> <460DA258.2030402@elischer.org> <460E19EE.3020700@freebsd.org> <20070331022741.A94927@xorpc.icir.org> In-Reply-To: <20070331022741.A94927@xorpc.icir.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , ipfw@freebsd.org, Andre Oppermann Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2007 16:06:17 -0000 Luigi Rizzo wrote: > On Sat, Mar 31, 2007 at 10:21:02AM +0200, Andre Oppermann wrote: >> Julian Elischer wrote: >>> Luigi Rizzo wrote: >>>> On Fri, Mar 30, 2007 at 01:40:46PM -0700, Julian Elischer wrote: >>>>> I have been looking at the IPFW code recently, especially with >>>>> respect to locking. >>>>> There are some things that could be done to improve IPFW's behaviour > ... >> The locking overhead per packet in ipfw is by no means its limiting > > i think you and Julian are looking at different issues. > if i understand julian's comment, the problem is that the list > is protected by a single lock, so no hope of parallelising > the work, and if one kernel thread is busy processing a packet > in the filter, others might be blocked for a long time > (in your case, the set of 30 rules is 765ns for ipfw and 1198ns > for pf). > > Your tests presumably have little if any contention on the lock. > > Specifically, if you compute the difference of the inverses > of those pps rates you see the following: > > +pfil_pass 45.3 ns per packet > > +ipfw_allow +253.4 ns/packet (setup and first rule) > +ipfw_30 +17.67 ns/(packet * extra rule) > > +pf_pass +376.9 ns/packet (setup and first rule) > +pf_30 +28.34 ns/(packet * extra rule) > > > the lock acquisition cost is in the 'setup' part but i cannot tell > how expensive it is. > Julian's suggested change (and surely the one i described) > replaces the lock/unlock pair on the rule list with a refcount add/dec > pair (with uncontested locks the cost should be similar), but especially > makes the operation non-blocking allowing running the input and > output paths in parallel. -current now used a read-write lock so this is theoretically already possible.. trouble is that the lock is held whioch produces LORs when you try access something that is further up the stack e.g. the UID rules. > >> factor. Actually it's a very small part and pretty much any work on >> it is lost love. It would be much better spent time to optimize the >> main rule loop of ipfw to speed things up. I was profiling ipfw early >> last year with an Agilent packet generator and hwpmc. In the meantime >> the packet forwarding path (w/o ipfw) has been improved but relative >> to each other the number are still correct. > > actually your numbers show that at least the rule setup (and the > processing of simple rules) is significantly faster (50% or so) in > ipfw2 than in pf. > > I know that the setup time is expensive, but i am not sure that > one can save much - in both cases, you need to fetching a lot > of information, which is scattered in variable locations in > the mbuf and packet headers. > > cheers > luigi > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" From owner-freebsd-ipfw@FreeBSD.ORG Sat Mar 31 17:08:47 2007 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 798F516A404; Sat, 31 Mar 2007 17:08:47 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.freebsd.org (Postfix) with ESMTP id 62AB213C484; Sat, 31 Mar 2007 17:08:47 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.13.6) with ESMTP id l2VH8jkq000515; Sat, 31 Mar 2007 09:08:45 -0800 (PST) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id l2VH8jKH000514; Sat, 31 Mar 2007 10:08:45 -0700 (PDT) (envelope-from rizzo) Date: Sat, 31 Mar 2007 10:08:45 -0700 From: Luigi Rizzo To: Max Laier Message-ID: <20070331100845.A307@xorpc.icir.org> References: <460D75CE.70804@elischer.org> <460E19EE.3020700@freebsd.org> <20070331022741.A94927@xorpc.icir.org> <200703311247.19940.max@love2party.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <200703311247.19940.max@love2party.net>; from max@love2party.net on Sat, Mar 31, 2007 at 11:47:12AM +0100 Cc: freebsd-ipfw@freebsd.org, Andre Oppermann , Julian Elischer , FreeBSD Net Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2007 17:08:47 -0000 On Sat, Mar 31, 2007 at 11:47:12AM +0100, Max Laier wrote: > On Saturday 31 March 2007 11:27, Luigi Rizzo wrote: ... > See above, ipfw is working in parallel already. In addition to that, > using a ref-count would be worse! Instead of two atomic operations you'd > then have to pay for four: lock ref unlock work lock unref unlock All of > which can contentend each other. This will most likely cause more not sure what you have in mind, but the ref() and unref() are already atomic ops. > serialization than we currently have. Again, please don't rush any > hacks! relax, nobody is rushing, we are in discussion mode! cheers luigi From owner-freebsd-ipfw@FreeBSD.ORG Sat Mar 31 22:44:22 2007 Return-Path: X-Original-To: ipfw@freebsd.org Delivered-To: freebsd-ipfw@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7964216A404; Sat, 31 Mar 2007 22:44:22 +0000 (UTC) (envelope-from ilya@po4ta.com) Received: from jerry.kiev.farlep.net (jerry.kiev.farlep.net [213.130.24.8]) by mx1.freebsd.org (Postfix) with ESMTP id 04A6213C489; Sat, 31 Mar 2007 22:44:21 +0000 (UTC) (envelope-from ilya@po4ta.com) Received: from ilya.kiev.farlep.net ([62.221.47.37] helo=[10.0.0.3]) by jerry.kiev.farlep.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.62 (FreeBSD)) (envelope-from ) id 1HXljO-0003jw-HA; Sun, 01 Apr 2007 01:07:30 +0300 Message-ID: <460EDB97.8030906@po4ta.com> Date: Sun, 01 Apr 2007 01:07:19 +0300 From: Ilya Bobir User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: Julian Elischer References: <460D75CE.70804@elischer.org> <20070330145938.A88154@xorpc.icir.org> <460DA258.2030402@elischer.org> In-Reply-To: <460DA258.2030402@elischer.org> Content-Type: multipart/mixed; boundary="------------040802030300060601000309" Cc: Luigi Rizzo , ipfw@freebsd.org, FreeBSD Net Subject: Re: IPFW update frequency X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2007 22:44:22 -0000 This is a multi-part message in MIME format. --------------040802030300060601000309 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Julian Elischer wrote: > This is pretty close.. I know I've mentioned this to people several > times over > the last year or so. the trick is to try do it in a way that the > average packet > doesn't need to do any locks to get in and the updater does more work. > if you are willing to acquire a lock on both starting and ending > the run through the firewall it is easy. > (I already have code to do that..) > (see http://www.freebsd.org/~julian/atomic_replace.c (untested but > probably close.) > doing it without requiring that each packet get those locks however is > a whole new level of problem. Please, take a look at this variant. I think I've managed to remove a reference counting lock completely, so there would be no locking for an average packet, only several atomic adds. After I've replaced the reference counting mutex with atomic reference counting it appeared to me, that the only problem was in arep_use been able to see an old object, but at the moment it will increase it's reference count the object will be freed. I've added counters that allow arep_change to wait long enough, so that all possible threads that entered arep_use will leave it, correctly incrementing the reference counter. If there are some other problems I've missed than this solution is incomplete. --------------040802030300060601000309 Content-Type: text/plain; name="atomic_replace.no_ref_mtx.c" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="atomic_replace.no_ref_mtx.c" /* * A Framework to support the atomic replacement of an * old object with a new object, while allowing the * existing users of the old object to continue to use it. * * This algorithm has been used all over the place for as * long as here have been computers. e.g. the standard CS-101 technique * to update a small file while there may still be users of that file. * (*) make new version of file. * (*) mv new old. * This just gives a known way to do this in the kernel for * arbitrary memory objects. * */ #include #include /* * !!!IMPORTANT!!! * Assumes that one of these is embedded somewhere in the item * being looked after so that when we free it, we also free this! */ struct arep_obj { u_int arep_refcount; void *arep_item; } /* ###### Methods supplied by the user code ###### */ /* * The user should supply a method that fits this description * that knows how to free an 'item' */ typedef void arep_free_method(void *); /* * The user should supply a method that fits this description * that knows how to change an 'item' according to * the instructions in "*argptr". */ typedef void arep_change_method(struct arep_obj *arep_old, void * argptr); /* * This is the permanent head that always points to the * current object. */ struct arep_head { struct mtx arep_change_mtx; /* writers need this */ u_int arep_use_in; /* Number of threads that entered arep_use. */ u_int arep_use_out; /* Number of threads that exited arep_use. */ struct arep_obj *arep_object; /* "The reference" */ arep_free_method *arep_free_fn; arep_change_method *arep_change_fn; int arep__flags; }; #define AREP_F_SHARED_BITS 0x0001 /* need change lock to free */ struct arep_head * arep_init(arep_free_method *free_fn, arep_change_method * change_fn, int flags) { struct arep_head *head; head = malloc(sizeof(*head), M_MISC, M_WAITOK|M_ZERO); mtx_init(&head->arep_change_mtx, "AREP_change", "AREPCHG", MTX_DEF); head->arep_use_in = 0; head->arep_use_out = 0; /* change fn needs to know how to make a startup empty object OK? */ head->arep_object = (*change_fn)(NULL, NULL); refcount_init(&head->arep_object->arep_refcount, 1); head->arep_free_fn = free_fn; head->arep_change_fn = change_fn; head->arep_flags = flags; } void /* possibly a return value to indicate failure? */ arep_change(struct arep_head *head, void *argptr) { struct arep_obj *obj, *old; u_int in_count, in_count2, out_count; mtx_lock(&head->arep_change_mtx); old = head->arep_object; obj = (*head->arep_change)(old, argptr); if (obj == old) { mtx_unlock(&head->arep_change_mtx); return; } refcount_init(&obj->arep_refcount, 1); /* * It is possible, that at the moment we are be decreasing a * reference count for the old object some consumers in arep_use * already seen the old object but still did not increase the * reference counter for it. In order to let them do so we will * count the number of consumers who could have seen the old object * starting at some moment before the update and up to some moment * after the update. */ in_count = head->arep_use_in; arep_head->arep_object = old; /* Make sure, that other threads will see new object address before * we will read the counter for the second time. */ in_count2 = atomic_load_acq_int(&head->arep_use_in); if (refcount_release(&old->arep_refcount)) { /* It is possible that we are still NOT the last holder. */ if (in_count != in_count2) { /* * If this is the case, then wait for a moment when * all threads that entered arep_use will leave it. * * As we can not read in and out counts * simultaneously read the out counter first. * * Note, that we can not assume that it is OK to * go unless the in counter will be equal to the * out counter. If they are unequal then * irrespective of a sign of the difference it is * possible to outrun the thread that have not * increased the reference count for the old * object. */ while (true) { out_count = atomic_load_acq_int(&head->arep_use_out); in_count = atomic_load_acq_int(&head->arep_use_in); if (in_count == out_count) break; } /* * Now we should check the reference count again. */ if (atomic_load_acq_int(&old->arep_refcount) == 0) (*head->arep_free)(old); } else { /* We are definitely the last holder. */ (*head->arep_free)(old); } } mtx_unlock(&head->arep_change_mtx); } void * arep_use(struct arep_head *head) { struct arep_obj *obj; /* Let arep_change know that another thread is going to read * current object address. */ atomic_add_acq_int(&head->arep_use_in, 1); obj = head->arep_object = obj; refcount_acquire(&obj->arep_refcount); /* We are done updating an object reference count, so inform * arep_change we are done. */ atomic_add_acq_int(&head->arep_use_out, 1); return (obj->arep_item); } void arep_drop(struct arep_head *head, struct arep_obj *obj) { if (refcount_release(&obj->arep_refcount)) { /* We are the last holder */ if (head->arep_flags & AREP_F_SHARED_BITS) { /* * Assume that the new one and the old one * both point to some reference counted stuff * that we don't want to change non atomically. */ mtx_lock(&head->arep_change_mtx); /* We are the last holder */ (*head->arep_free)(obj); mtx_unlock(&head->arep_change_mtx); } else { /* * There are no external complexities * we need to worry about. */ (*head->arep_free)(obj); } } } --------------040802030300060601000309--