From owner-freebsd-questions@FreeBSD.ORG Thu Nov 11 04:53:12 2004 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A2A1D16A4CE for ; Thu, 11 Nov 2004 04:53:12 +0000 (GMT) Received: from thewordnerd.thewordnerd.info (dsl093-216-130.aus1.dsl.speakeasy.net [66.93.216.130]) by mx1.FreeBSD.org (Postfix) with ESMTP id 506E643D1F for ; Thu, 11 Nov 2004 04:53:12 +0000 (GMT) (envelope-from nolan@thewordnerd.info) Received: from nolan by thewordnerd.thewordnerd.info with local (Exim 3.36 #1 (Debian)) id 1CREEp-00021L-00 for ; Mon, 08 Nov 2004 12:27:35 -0600 To: questions@freebsd.org User-Agent: SEMI/1.14.6 (Maruoka) FLIM/1.14.6 (Marutamachi) APEL/10.6 Emacs/21.3 (i386-pc-linux-gnu) (with unibyte mode) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Message-Id: From: "Nolan J. Darilek" Date: Mon, 08 Nov 2004 12:27:35 -0600 Subject: Jails, pf and larger packets X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Nov 2004 04:53:12 -0000 I'm having a bit of a problem with jails on my FreeBSD 5.2.1 system. Specifically, since I switched from my ipfirewall/natd-based firewall to pf, jails can't seen to send packets of greater than 46 bytes of data to other systems. My network is set up as follows. The firewall/host box, running 5.2.1, is on 192.168.0.1. On 192.168.0.2 I have a jail running numerous services, and on 192.168.0.3 I have a shell account jail for screen sessions and such. The internal LAN is also served by DHCP, and my desktop Debian box is at 0.100. The problem first surfaced when I was using esd on the shell server to stream and play audio on my desktop for IM events. I ran pftcpdump on pflog0 but saw no messages about blocked packets. Though I can't traceroute out of the jail to see where my packets are going, traceroutes from my desktop seem to indicate that jails appear to bridge, so it seems odd that anything on the firewall could have any effect on traffic within the LAN to the jail. I then ran strace on the esdplay process, and received the following edited output: socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 4 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [0], 4) = 0 connect(4, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 . . . write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16 write(4, "\0\0\0\0", 4) = 4 read(4, "\0\0\0\0", 4) = 4 . . . write(4, "\0\0\0\0", 4) = 4 write(4, "\0\0\0\0", 4) = 4 write(4, "\0\0\0\0", 4) = 4 write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 128) = 128 setsockopt(4, SOL_SOCKET, SO_SNDBUF, [0], 4) = 0 setsockopt(4, SOL_SOCKET, SO_RCVBUF, [0], 4) = 0 . . . write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = -1 EPERM (Operation not permitted) close(4) = 0 . . . Could a firewall cause a permission denied error like this? I'd think not, but I don't know much about networking, and the only major change I recall making was switching from ipfirewall/natd to pf for everything (which, incidentally, works perfectly except for this issue.) I then tried to create a simple test case, trying different permutations of that. I wrote a simple client and server. The server binds to the IP address and simply listens for a single string, and the client sends a string of the specified length. I can send data TO the jails from my desktop just fine, but I can reproduce the EPERM when sending data from jails to the desktop. Interestingly enough, the error begins when I send packets of 47 bytes or greater. It's been awhile since I've done TCP/IP; does the header happen to be 18 bytes? If so, is there something preventing packets of greater than 64 bytes from leaving jails? I see nothing in logs and have consulted the manpages, though it's certainly possible that I've missed something. And, again, this worked fine before I switched to pf. I'd rather not switch back to ipfirewall if I can avoid it, as I hope to use queueing soon and, in general, like the look of pf rules a bit more. Is there anything else that might be responsible for this failure, and what else can I offer to help you help me? Thanks.