From owner-freebsd-net@FreeBSD.ORG Wed Jul 9 12:23:52 2003 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C061337B401 for ; Wed, 9 Jul 2003 12:23:52 -0700 (PDT) Received: from ecserv7.uwaterloo.ca (ecserv7.uwaterloo.ca [129.97.50.127]) by mx1.FreeBSD.org (Postfix) with ESMTP id D321B43F3F for ; Wed, 9 Jul 2003 12:23:51 -0700 (PDT) (envelope-from kw3wong@engmail.uwaterloo.ca) Received: from ecserv7.uwaterloo.ca (localhost.uwaterloo.ca [127.0.0.1]) h69JNrNp071112; Wed, 9 Jul 2003 15:23:53 -0400 (EDT) (envelope-from kw3wong@engmail.uwaterloo.ca) Received: (from www@localhost) by ecserv7.uwaterloo.ca (8.12.6p2/8.12.6/Submit) id h69JNqTK071111; Wed, 9 Jul 2003 15:23:52 -0400 (EDT) X-Authentication-Warning: ecserv7.uwaterloo.ca: www set sender to kw3wong@engmail.uwaterloo.ca using -f Received: from 129.97.106.24 ( [129.97.106.24])with HTTP; Wed, 9 Jul 2003 15:23:52 -0400 Message-ID: <1057778632.3f0c6bc8af474@www.nexusmail.uwaterloo.ca> Date: Wed, 9 Jul 2003 15:23:52 -0400 From: kw3wong@engmail.uwaterloo.ca To: freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.1 / FreeBSD-4.6.2 X-Originating-IP: 129.97.106.24 cc: dsze@engmail.uwaterloo.ca Subject: Question about bridging code X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2003 19:23:53 -0000 Hi guys, My first attempts at hacking FreeBSD kernel code has not been very fruitful, so I'm hoping someone with more experience and knowhow might be able to point out the mistakes that I'm making. Firstly, let me explain what I'm trying to do. I'm currently working on a University project that performs some type of transformation (compression, security, string replacement, etc) on packets as they pass through the system. The current setup has the FreeBSD machine configured as a router, and the transformation is performed on packets that are routed. This is done via divert sockets and everything is fine and dandy, we're getting great results from this setup. However, what we want to do next is to have the machine setup as a ethernet bridge instead, and the transformation is to be performed on the bridged packets. Unfortunately, as most of you probably know, divert sockets do not work with bridges as of yet. So I've been trying to add a somewhat hack-ish support for divert sockets over bridges. The concession that I'm making is that instead of diverting ip packets, I'll be diverting ethernet frames. In userspace my program will reattach the ethernet headers back onto the packet before passing it back to the divert socket. A second concession is that when I sendto the divert socket, the sin_zero in the sockaddr must contain the source network adaptor name. All these concessions are necessary (I think) as I would otherwise not know how to output the data in a ip-less bridge. So here is what my code changes involved so far. BTW, I'm using FreeBSD 4.8 1) Removed the check in ipfw_chk (ip_fw2.c) for whether it is layer2 or not. This allows briged packets to still match the ipfw2 divert rules 2) In bridge.c at function bdg_forward, after the ip_fw_chk_ptr (and after the check for dummynet, around line 974), the following code fragment is added if (i != 0 && (i & IP_FW_PORT_DYNT_FLAG) == 0) { struct mbuf *m; /* Need to determine whether this is an IP. If not just forward */ if (ntohs(eh->ether_type) != ETHERTYPE_IP) goto forward; if ( shared ) { int j = min(m0->m_pkthdr.len + ETHER_HDR_LEN, max_protohdr) ; m0 = m_pullup(m0, j) ; if (m0 == NULL) return NULL; } if (shared == 0 && once ) { /* no need to copy */ m = m0 ; m0 = NULL ; /* original is gone */ } else { m = m_copypacket(m0, M_DONTWAIT); if (m == NULL) { printf("bdg_forward: sorry, m_copypacket failed!\n"); return m0 ; /* the original is still there... */ } } if ( (void *)(eh + 1) == (void *)m->m_data) { m->m_data -= ETHER_HDR_LEN ; m->m_len += ETHER_HDR_LEN ; m->m_pkthdr.len += ETHER_HDR_LEN ; bdg_predict++; } else { M_PREPEND(m, ETHER_HDR_LEN, M_DONTWAIT); if (m == NULL) { printf("M_PREPEND failed\n"); /* Should probably return original instead of NULL */ /* return NULL; */ return m0; } bcopy(&save_eh, mtod(m, struct ether_header *), ETHER_HDR_LEN); } divert_packet(m, 1, i & 0xffff, args.divert_rule); return NULL; } This allows me to divert the ethernet frames to userspace. 3) To allow me to inject ethernet frames back into the system via divert sockets, I've modified div_output so that it will call ether_output_frame. The following are my changes to div_output, which is added before ip_output is called: /* rcvif is copied from sin_zero, and is required to be valid for the current system to work */ if (m->m_pkthdr.rcvif != NULL && BDG_USED(m->m_pkthdr.rcvif)) { if (m->m_len < sizeof(struct ether_header)) { /* XXX error in the caller. */ error = EINVAL; goto cantsend; } return ether_output_frame(m->m_pkthdr.rcvif, m); } 4) In userspace for testing purposes, I have a program that simply reads from the divert socket, and writes back out to it - here's the core snippet of the code. while (true) { sstBytes = ::recvfrom(nFD, kpucInPacket, sizeof(kpucInPacket), 0, (struct sockaddr *) &SockAddr, &AddrLen); if (sstBytes == -1) ::err(errno, "recvfrom"); ::bcopy(SockAddr.sin_zero, SockAddrSend.sin_zero, sizeof(SockAddr.sin_zero)); int nSendBytes = ::sendto(nSendFD, (void*)kpucInPacket, sstBytes, 0, (struct sockaddr *) &SockAddrSend, sizeof(SockAddrSend)); if (nSendBytes != sstBytes) ::err(errno, "sendto"); } Now I understand I'm breaking lots of abstractions/layers, but I do plan to clean that up a bit later. And I also understand that perhaps no one else in the world needs this functionality - although I can see a couple of other possible applications for it. The changes does seem to work, I'm able to receive the ethernet frame and also reinject it via the divert sockets - ping, ftp, etc. all work over the bridge when my test program is running. However, I'm finding that I'm losing/leaking mbufs. sbdrop will complain and panic that the sb_cc doesn't match up with what the mbuf chains says - usually the sb_cc will be larger by a couple of hundred bytes. Furthermore, a netstat -m will show that I have mbufs allocated to socket names and address even after the termination of the diverting program. This only seem to happen when I transfer over ftp a really large file (>100M) at high speed (full line speed of a 100Mbps network). Ping and ftping small files do not seem to cause the mbuf leakage. So my question is, does anyone see where I might be losing the mbufs - is there some mbufs that must be freed or not freed that I'm not aware of? I've never worked on the FreeBSD kernel before, so I'm not sure 100% sure how to correctly manage the mbufs. Any advise, tips, discussion, anything will be highly appreciated! =) If anyone needs any more clarification/information, just ask and I'll try my best to explain myself better. Thanks!! Bernie ---------------------------------------- This mail sent through www.mywaterloo.ca