Date: Sun, 9 Oct 2011 12:09:25 -0700 From: Michael Sierchio <kudzu@tenebras.com> To: freebsd_user@guice.ath.cx Cc: freebsd-questions@freebsd.org Subject: Re: System randomly not logging complete bi-directional traffic. Message-ID: <CAHu1Y71eunNb-4=sx9ChDs94op1=03C1Rg_wv9B2XeJ8kwcGOQ@mail.gmail.com> In-Reply-To: <aabce9271135c2d37c3ca429ae25a497.squirrel@wtp1.ath.cx> References: <aabce9271135c2d37c3ca429ae25a497.squirrel@wtp1.ath.cx>
next in thread | previous in thread | raw e-mail | index | archive | help
Sorry to have missed your prior post - please include the entire ruleset. Thanks. On Sun, Oct 9, 2011 at 10:28 AM, <freebsd_user@guice.ath.cx> wrote: > freebsd-questions@freebsd.org > # > # > # FreeBSD_7-4 RELEASE > # Our hardware is pristine > # > # What is described herein are regular, yet random occurrences; we need h= elp. > > We have already performed a reinstall of FreeBSD_7-4 RELEASE (and the > daemons in question); the issue remains. Below, is part of a conversation > with an httpd whereby the packets (entire conversations) are randomly > 'not' being logged and/or seen by either the httpd nor ipfw (logging > enabled), yet both tshark and tcpdump are capturing everything. > > To be perfectly clear, httpd and ipfw (randomly) will not see/log anythin= g > of an 'entire conversation'. =A0It is not like it drops certain packets o= f a > conversation; they (httpd/ipfw) either see and log everything during a > conversation, or, 'do not see' and 'do not log' any packet associated wit= h > a given conversation; all the while tshark and tcpdump are capturing > everything (bidirectional); hence the connection is real. > > The capture below was witnessed by both tshark and tcpdump, but not logge= d > via the httpd or the following ipfw rule: > > $cmd 00029 deny log logamount 0 ip from "table(1)" to me 80 > > The above ipfw rule functions properly from "table(1)" which contains -- > ip.ip.ip.ip/32 -- one (1) ip per line. > > The names (below) were changed to protect the innocent; yeah right. > > Internet Protocol Version 4, Src: ex.ter.nal.ip (ex.ter.nal.ip), Dst: > in.ter.nal.ip (in.ter.nal.ip) > =A0 =A0Version: 4 > =A0 =A0Header length: 20 bytes > =A0 =A0Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00= : > Not-ECT (Not ECN-Capable Transport)) > =A0 =A0 =A0 =A00000 00.. =3D Differentiated Services Codepoint: Default (= 0x00) .... > ..00 =3D Explicit Congestion Notification: Not-ECT (Not > ECN-Capable Transport) (0x00) > =A0 =A0Total Length: 60 > =A0 =A0Identification: 0x8ce5 (36069) > =A0 =A0Flags: 0x02 (Don't Fragment) > =A0 =A0 =A0 =A00... .... =3D Reserved bit: Not set > =A0 =A0 =A0 =A0.1.. .... =3D Don't fragment: Set > =A0 =A0 =A0 =A0..0. .... =3D More fragments: Not set > =A0 =A0Fragment offset: 0 > =A0 =A0Time to live: 251 > =A0 =A0Protocol: TCP (6) > =A0 =A0Header checksum: 0x9102 [correct] > =A0 =A0 =A0 =A0[Good: True] > =A0 =A0 =A0 =A0[Bad: False] > =A0 =A0Source: ex.ter.nal.ip (ex.ter.nal.ip) > =A0 =A0Destination: in.ter.nal.ip (in.ter.nal.ip) > Transmission Control Protocol, Src Port: 46463 (46463), Dst Port: http > (80), Seq: 0, Len: 0 > =A0 =A0Source port: 46463 (46463) > =A0 =A0Destination port: http (80) > =A0 =A0[Stream index: 19] > =A0 =A0Sequence number: 0 =A0 =A0(relative sequence number) > =A0 =A0Header length: 40 bytes > =A0 =A0Flags: 0x02 (SYN) > =A0 =A0 =A0 =A0000. .... .... =3D Reserved: Not set > =A0 =A0 =A0 =A0...0 .... .... =3D Nonce: Not set > =A0 =A0 =A0 =A0.... 0... .... =3D Congestion Window Reduced (CWR): Not se= t > =A0 =A0 =A0 =A0.... .0.. .... =3D ECN-Echo: Not set > =A0 =A0 =A0 =A0.... ..0. .... =3D Urgent: Not set > =A0 =A0 =A0 =A0.... ...0 .... =3D Acknowledgement: Not set > =A0 =A0 =A0 =A0.... .... 0... =3D Push: Not set > =A0 =A0 =A0 =A0.... .... .0.. =3D Reset: Not set > =A0 =A0 =A0 =A0.... .... ..1. =3D Syn: Set > =A0 =A0 =A0 =A0 =A0 =A0[Expert Info (Chat/Sequence): Connection establish= request > (SYN): server port http] > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0[Message: Connection establish request (SY= N): server port > http] > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0[Severity level: Chat] > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0[Group: Sequence] > =A0 =A0 =A0 =A0.... .... ...0 =3D Fin: Not set > =A0 =A0Window size value: 5840 > =A0 =A0[Calculated window size: 5840] > =A0 =A0Checksum: 0xe7f8 [validation disabled] > =A0 =A0 =A0 =A0[Good Checksum: False] > =A0 =A0 =A0 =A0[Bad Checksum: False] > =A0 =A0Options: (20 bytes) > =A0 =A0 =A0 =A0Maximum segment size: 1460 bytes > =A0 =A0 =A0 =A0TCP SACK Permitted Option: True > =A0 =A0 =A0 =A0Timestamps: TSval 309029146, TSecr 0 > =A0 =A0 =A0 =A0 =A0 =A0Kind: Timestamp (8) > =A0 =A0 =A0 =A0 =A0 =A0Length: 10 > =A0 =A0 =A0 =A0 =A0 =A0Timestamp value: 309029146 > =A0 =A0 =A0 =A0 =A0 =A0Timestamp echo reply: 0 > =A0 =A0 =A0 =A0No-Operation (NOP) > =A0 =A0 =A0 =A0Window scale: 7 (multiply by 128) > =A0 =A0 =A0 =A0 =A0 =A0Kind: Window Scale (3) > =A0 =A0 =A0 =A0 =A0 =A0Length: 3 > =A0 =A0 =A0 =A0 =A0 =A0Shift count: 7 > =A0 =A0 =A0 =A0 =A0 =A0[Multiplier: 128] > =A0 =A0Frame Number: 51 > =A0 =A0Frame Length: 74 bytes (592 bits) > =A0 =A0Capture Length: 74 bytes (592 bits) > =A0 =A0[Frame is marked: False] > =A0 =A0[Frame is ignored: False] > =A0 =A0[Protocols in frame: eth:ip:tcp] > Ethernet II, Src: Router_cf:gr:f0 (11:52:c3:fd:dd:f0), Dst: Goe_40:84:21 > (00:15:18:40:28:41) > =A0 =A0Destination: Goe_40:84:21 (00:15:18:40:28:41) > =A0 =A0 =A0 =A0Address: Goe_40:84:21 (00:15:18:40:28:41) > =A0 =A0 =A0 =A0.... ...0 .... .... .... .... =3D IG bit: Individual addre= ss > (unicast) > =A0 =A0 =A0 =A0.... ..0. .... .... .... .... =3D LG bit: Globally unique = address > (factory default) > =A0 =A0Source: Router_cf:gr:f0 (11:52:c3:fd:dd:f0) > =A0 =A0 =A0 =A0Address: Router_cf:gr:f0 (11:52:c3:fd:dd:f0) > =A0 =A0 =A0 =A0.... ...0 .... .... .... .... =3D IG bit: Individual addre= ss > (unicast) > =A0 =A0 =A0 =A0.... ..0. .... .... .... .... =3D LG bit: Globally unique = address > (factory default) > =A0 =A0Type: IP (0x0800) > Internet Protocol Version 4, Src: ex.ter.nal.ip (ex.ter.nal.ip), Dst: > in.ter.nal.ip (in.ter.nal.ip) > =A0 =A0Version: 4 > =A0 =A0Header length: 20 bytes > =A0 =A0Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00= : > Not-ECT (Not ECN-Capable Transport)) > =A0 =A0 =A0 =A00000 00.. =3D Differentiated Services Codepoint: Default (= 0x00) .... > ..00 =3D Explicit Congestion Notification: Not-ECT (Not > ECN-Capable Transport) (0x00) > =A0 =A0Total Length: 60 > =A0 =A0Identification: 0x8ce5 (36069) > =A0 =A0Flags: 0x02 (Don't Fragment) > =A0 =A0 =A0 =A00... .... =3D Reserved bit: Not set > =A0 =A0 =A0 =A0.1.. .... =3D Don't fragment: Set > =A0 =A0 =A0 =A0..0. .... =3D More fragments: Not set > =A0 =A0Fragment offset: 0 > =A0 =A0Time to live: 251 > =A0 =A0Protocol: TCP (6) > =A0 =A0Header checksum: 0x9102 [correct] > =A0 =A0 =A0 =A0[Good: True] > =A0 =A0 =A0 =A0[Bad: False] > =A0 =A0Source: ex.ter.nal.ip (ex.ter.nal.ip) > =A0 =A0Destination: in.ter.nal.ip (in.ter.nal.ip) > Transmission Control Protocol, Src Port: 46463 (46463), Dst Port: http > (80), Seq: 0, Len: 0 > =A0 =A0Source port: 46463 (46463) > =A0 =A0Destination port: http (80) > =A0 =A0[Stream index: 19] > =A0 =A0Sequence number: 0 =A0 =A0(relative sequence number) > =A0 =A0Header length: 40 bytes > =A0 =A0Flags: 0x02 (SYN) > =A0 =A0 =A0 =A0000. .... .... =3D Reserved: Not set > =A0 =A0 =A0 =A0...0 .... .... =3D Nonce: Not set > =A0 =A0 =A0 =A0.... 0... .... =3D Congestion Window Reduced (CWR): Not se= t > =A0 =A0 =A0 =A0.... .0.. .... =3D ECN-Echo: Not set > =A0 =A0 =A0 =A0.... ..0. .... =3D Urgent: Not set > =A0 =A0 =A0 =A0.... ...0 .... =3D Acknowledgement: Not set > =A0 =A0 =A0 =A0.... .... 0... =3D Push: Not set > =A0 =A0 =A0 =A0.... .... .0.. =3D Reset: Not set > =A0 =A0 =A0 =A0.... .... ..1. =3D Syn: Set > =A0 =A0 =A0 =A0 =A0 =A0[Expert Info (Chat/Sequence): Connection establish= request > (SYN): server port http] > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0[Message: Connection establish request (SY= N): server port > http] > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0[Severity level: Chat] > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0[Group: Sequence] > =A0 =A0 =A0 =A0.... .... ...0 =3D Fin: Not set > =A0 =A0Window size value: 5840 > =A0 =A0[Calculated window size: 5840] > =A0 =A0Checksum: 0xe7f8 [validation disabled] > =A0 =A0 =A0 =A0[Good Checksum: False] > =A0 =A0 =A0 =A0[Bad Checksum: False] > =A0 =A0Options: (20 bytes) > =A0 =A0 =A0 =A0Maximum segment size: 1460 bytes > =A0 =A0 =A0 =A0TCP SACK Permitted Option: True > =A0 =A0 =A0 =A0Timestamps: TSval 309029146, TSecr 0 > =A0 =A0 =A0 =A0 =A0 =A0Kind: Timestamp (8) > =A0 =A0 =A0 =A0 =A0 =A0Length: 10 > =A0 =A0 =A0 =A0 =A0 =A0Timestamp value: 309029146 > =A0 =A0 =A0 =A0 =A0 =A0Timestamp echo reply: 0 > =A0 =A0 =A0 =A0No-Operation (NOP) > =A0 =A0 =A0 =A0Window scale: 7 (multiply by 128) > =A0 =A0 =A0 =A0 =A0 =A0Kind: Window Scale (3) > =A0 =A0 =A0 =A0 =A0 =A0Length: 3 > =A0 =A0 =A0 =A0 =A0 =A0Shift count: 7 > =A0 =A0 =A0 =A0 =A0 =A0[Multiplier: 128] > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > We first noticed this logging issue with Apache22 and thought httpd was > the culprit; we installed another httpd and the problem remained. =A0At t= hat > point, with two (2) httpds' seemingly offering the same results we decide= d > to enable ipfw and its logging feature. Shortly thereafter we notice ipfw > was also randomly 'not seeing' the exact same traffic, destined for port > 80, that neither httpd's were also 'not seeing', yet Tshark and tcpdump > are in fact, capturing these packets. > > We would like help to troubleshoot this concern rather than blindly > replacing things such as syslogd. > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.o= rg" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHu1Y71eunNb-4=sx9ChDs94op1=03C1Rg_wv9B2XeJ8kwcGOQ>