Date: Fri, 8 Jan 2010 17:05:16 -0600 From: "M.Keith.Thompson " <m.keith.thompson@gmail.com> To: J65nko <j65nko@gmail.com> Cc: "freebsd-pf@freebsd.org" <freebsd-pf@freebsd.org> Subject: Re: ftp problem Message-ID: <BC848642-A55F-4EAD-9BA2-4DD04460F177@GMail.com> In-Reply-To: <19861fba1001081450w4d9096bdt83c8024b6e58af55@mail.gmail.com> References: <fc2243911001060809m5417b810vf2ed40c8a969fb5f@mail.gmail.com> <7731938b1001060923n5de4b511of07b8c63cff4e011@mail.gmail.com> <fc2243911001060957n16f906f7m703c696b970e8c3c@mail.gmail.com> <2cf1d0681001071216p6b516e9egcf7401f2b38e3c3d@mail.gmail.com> <19861fba1001071237ncc440d5u1ab280d2aaf0c72f@mail.gmail.com> <fc2243911001071319q4bbbc4a6o6b58031bba59c39b@mail.gmail.com> <19861fba1001072018g115a0bccrf9510a38454cc9db@mail.gmail.com> <fc2243911001080551n251851b5kb56a6c5f216c4a14@mail.gmail.com> <fc2243911001081250n4471d9d8kf5bcf691d2cbd50d@mail.gmail.com> <19861fba1001081450w4d9096bdt83c8024b6e58af55@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Remember the problem is only with one client and files over 98K.
Hundreds of other clients transfer multi-megabyte files no problem.
On Jan 8, 2010, at 4:50 PM, J65nko <j65nko@gmail.com> wrote:
> On Fri, Jan 8, 2010 at 9:50 PM, M. Keith Thompson
> <m.keith.thompson@gmail.com> wrote:
>> It looks like it was a tcp windowing problem. The command: "sysctl
>> -w
>> sysctl net.inet.tcp.rfc1323=0"
>> fixed the problem.
>
> This only fixes a symptom. :) There is something wrong with your
> ruleset.
>
>>>> # Allow ftp
>>>> pass in quick on $ext_if proto tcp from any to $ext_IP port 21
>>>> keep state
>>>> pass in quick on $ext_if proto tcp from any to $ext_IP port >
>>>> 49151 keep state
>>>> pass in quick on $ext_if proto tcp from any port > 10000 to $ext_IP
>>>> port 20 keep state
>
> With this active ftp rule you had before, the HP client could initiate
> and transfer over an active ftp data channel,
> while this rule does not allow this at all.
> There shouldn't be any active ftp datachannel communication possible
> because port 20 is the source port and not the destination port.
>
> I tried the following pf.conf , with a similar ftp-data rule , on a
> local FreeBSD 7.2 box in my network
>
> EXT_IF=fxp0
>
> # net.inet.ip.portrange.hilast: 65535
> # net.inet.ip.portrange.hifirst: 49152
> PASSIVE='49152:65535'
>
> set skip on lo0
>
> # default policy
> block log all
>
> # outgoing DNS requests
>
> pass out quick on $EXT_IF inet proto udp from $EXT_IF:network to any
> port domain keep state
> pass out quick on $EXT_IF inet proto tcp from $EXT_IF:network to any
> port domain keep state flags S/SA
>
> # incoming SSH
>
> pass in quick on $EXT_IF inet proto tcp from $EXT_IF:network to port
> 2022 keep state flags S/SA
>
> # incoming FTP
>
> # ftp command channel
> pass in quick on $EXT_IF inet proto tcp from any to $EXT_IF port ftp
> keep state flags S/SA
>
> # ftp data channel (passive)
> pass in quick on $EXT_IF inet proto tcp from any to $EXT_IF port
> $PASSIVE keep state flags S/SA
>
> # ftp data channel (active)
> # wrong direction!!!
> pass in quick on $EXT_IF inet proto tcp from any to $EXT_IF port
> ftp-data keep state flags S/SA
>
> Here I also erroneously used port 20 as destination port.
>
> An active ftp command to upload a file aborts with
>
> 200 EPRT command successful.
> 425 Can't build data connection: Operation not permitted.
>
> A pflog0 tcpdump shows the reason:
>
> tcpdump -eni pflog0 -s0 -v
> tcpdump: WARNING: pflog0: no IPv4 address assigned
> tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file),
> capture size 65535 bytes
> 22:16:42.551221 rule 0/0(match): block out on fxp0: (tos 0x8, ttl 64,
> id 2766, offset 0, flags [DF], proto TCP (6), length 60)
> 192.168.222.244.20 > 192.168.222.20.62316: S, cksum 0xc222 (correct),
> 488073960:488073960(0) win 65535 <mss 1460,nop,wscale
> 3,sackOK,timestamp 6278055 0>
>
> So this rule set blocks the first packet of the three-way TCP
> handshake.
>
> 192.168.222.244.20 > 192.168.222.20.62316.
> or
> server.20 > client.62316
>
> Your previous ruleset somehow created state or allowed the server to
> initiate the data channel to the client somewhere else in your rule
> set. It didn't create state on the initial packet of the TCP
> handshake, where the TCP window size is negotiated. Just like the
> Daniel Hartmeier article states, this a caused tcp window scaling
> issue.
>
> Remember that in pf the last matching rule wins.
> You only can defeat this behaviour by using the 'quick' keyword.
>
> After adding the following corrected rule the transfer proceeds
> without any blockage
>
> # correct direction/source port
> pass out quick on $EXT_IF inet proto tcp from any port ftp-data to
> any keep state flags S/SA
>
> The rules as parsed are now:
>
> # pfctl -vvf /etc/pf.conf
> No ALTQ support in kernel
> ALTQ related functions disabled
> Loaded 696 passive OS fingerprints
> EXT_IF = "fxp0"
> PASSIVE = "49152:65535"
> set skip on { lo0 }
> @0 block drop log all
> @1 pass out quick on fxp0 inet proto udp from 192.168.222.0/24 to any
> port = domain keep state
> @2 pass out quick on fxp0 inet proto tcp from 192.168.222.0/24 to any
> port = domain flags S/SA keep state
> @3 pass in quick on fxp0 inet proto tcp from 192.168.222.0/24 to any
> port = down flags S/SA keep state
> @4 pass in quick on fxp0 inet proto tcp from any to 192.168.222.244
> port = ftp flags S/SA keep state
> @5 pass in quick on fxp0 inet proto tcp from any to 192.168.222.244
> port 49152:65535 flags S/SA keep state
> @6 pass in quick on fxp0 inet proto tcp from any to 192.168.222.244
> port = ftp-data flags S/SA keep state
> @7 pass out quick on fxp0 inet proto tcp from any port = ftp-data to
> any flags S/SA keep state
>
> Rule 6 is the rule with the wrong direction/port specification.
> Rule 7 is the corrected one
>
> pflog0 doesn't show any errors:
> tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file),
> capture size 65535 bytes
> ^C
> 0 packets captured
> 0 packets received by filter
> 0 packets dropped by kernel
>
> And a quick state check still showed some remnants of the connections
>
> # pfctl -ss
> No ALTQ support in kernel
> ALTQ related functions disabled
> [snip]
> all tcp 192.168.222.244:21 <- 192.168.222.20:13497
> FIN_WAIT_2:FIN_WAIT_2
> all tcp 192.168.222.244:20 -> 192.168.222.20:55246
> FIN_WAIT_2:FIN_WAIT_2
> all tcp 192.168.222.244:20 -> 192.168.222.20:59354
> FIN_WAIT_2:FIN_WAIT_2
>
> The first state is the ftp command channel client:13497 > server:21
> The last two are the active data channels: server:20 > client:55246,
> and server.20 > client.59354
>
> Don't get confused, the last octet of the client IP (192.168.222.20)
> happens to be the same as
> the ftp-data port number. ;)
>
> A repeat of the transfer shows the following verbose state
>
> # pfctl -vvss
>
> all tcp 192.168.222.244:20 -> 192.168.222.20:64857
> ESTABLISHED:ESTABLISHED
> [1427346948 + 65536] wscale 3 [200059941 + 37224] wscale 0
> age 00:00:02, expires in 24:00:00, 13716:20572 pkts,
> 713240:30853060 bytes, rule 7
> id: 4b47883900000033 creatorid: c576b1b2
>
> By using this command you should be able to figure out which actual
> rule, here rule 7, creates state for the active ftp data channel.
>
> The same when the transfer has finished
>
> all tcp 192.168.222.244:20 -> 192.168.222.20:64857
> FIN_WAIT_2:FIN_WAIT_2
> [1427346950 + 65534] wscale 3 [259693437 + 66608] wscale 0
> age 00:00:59, expires in 00:00:38, 41209:61758 pkts,
> 2142876:92628227 bytes, rule 7
> id: 4b47883900000033 creatorid: c576b1b2
>
> Happy rule/state hunting
>
> Adriaan
>
>
> all tcp 192.168.222.244:20 -> 192.168.222.20:64857
> FIN_WAIT_2:FIN_WAIT_2
> [1427346950 + 65534] wscale 3 [259693437 + 66608] wscale 0
> age 00:00:59, expires in 00:00:38, 41209:61758 pkts,
> 2142876:92628227 bytes, rule 7
> id: 4b47883900000033 creatorid: c576b1b2
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BC848642-A55F-4EAD-9BA2-4DD04460F177>