Date: Tue, 11 Oct 2011 11:53:16 -0500 From: Larry Rosenman <ler@lerctr.org> To: Jack Vogel <jfvogel@gmail.com> Cc: freebsd-stable@freebsd.org, "Vogel, Jack" <jack.vogel@intel.com>, Jeremy Chadwick <freebsd@jdc.parodius.com>, John Baldwin <jhb@freebsd.org> Subject: Re: rsync corrupted MAC Message-ID: <6590d023-cd29-4d98-a3b5-161aa14bd006@email.android.com> In-Reply-To: <69ffedd5-d750-44f4-b934-61a4abe3c59c@email.android.com> References: <69ffedd5-d750-44f4-b934-61a4abe3c59c@email.android.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Not sure when it broke. I rebuilt the 9.0 server as 9.0, and ran the script=
and it started giving this.
--
Sent from my Android phone with K-9 Mail.=
Please excuse my brevity.
Jack Vogel <jfvogel@gmail.com> wrote:
Oh, I se=
e. So, did you have a previous working state?
Jack
On Tue, Oct 11, 2011=
at 12:06 AM, Larry Rosenman <ler@lerctr.org> wrote:
They are not local to=
each other. See the diagram. They are across the internet from each other.=
--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.
=
Jack Vogel <jfvogel@gmail.com> wrote:
Well, for a start I'd get both inte=
rfaces at the same speed, sounds like a hardware
issue of some sort, cable =
or switch maybe?
Jack
On Mon, Oct 10, 2011 at 5:42 PM, Larry Rosenman <l=
er@lerctr.org> wrote:
On Mon, 10 Oct 2011, Jeremy Chadwick wrote:
On Mon,=
Oct 10, 2011 at 04:15:25PM -0500, Larry Rosenman wrote:
On 10/10/2011 3:5=
7 PM, Louis Mamakos wrote:
On Oct 10, 2011, at 2:38 PM, Larry Rosenman wro=
te:
On 10/10/2011 10:47 AM, John Baldwin wrote:
On Sunday, October 09, 20=
11 5:06:26 pm Larry Rosenman wrote:
Any ideas on which side or what might =
be broke here?
ler/MAIL-ARCHIVE/2008/12/INBOX
Corrupted MAC on input.
Disc=
onnecting: Packet corrupt
rsync: connection unexpectedly closed (33845045 b=
ytes received so far)
[receiver]
rsync error: error in rsync protocol dat=
a stream (code 12) at io.c(605)
[receiver=3D3.0.9]
rsync: connection unex=
pectedly closed (1450 bytes received so far)
[generator]
rsync error: une=
xplained error (code 255) at io.c(605) [generator=3D3.0.9]
I've had somewh=
at similar issues (ssh getting corruption in its data stream)
when a NIC in=
my netbook was corrupting packet data when it ran at 1G (it
worked fine at=
10/100). Pyun eventually fixed the issue by applying enough
workarounds (=
it was likely a hardware bug in the NIC's chipset). However, it
wasn't eas=
y to debug unfortunately. :(
Any ideas on where to start?
from the 8.2 bo=
x (tbh.lerctr.org in the script):
8.2->PIX->Provider->Internet->Motorola S=
BG6580 (Time-Warner)->Trendnet TEG-160WS Gig switch->9.0 box (borg.lerctr.o=
rg).
So, where do I start?
I'd turn off IP / TCP / UDP checksum offloadin=
g on your NIC if it supports it, and see if you are getting network layer c=
hecksum errors. If the IP checksum is wrong, then it happened on the last =
hops between the NIC and memory or across the previous network hop.
Good=
idea, but, it didn't show ANY errors on EITHER side (both are
em nics).
N=
ext?
$ ifconfig em0
em0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAS=
T> metric 0 mtu 1500
options=3D2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWC=
SUM,WOL_MAGIC>
ether 00:30:48:2e:99:ba
inet 192.147.25.65 net=
mask 0xffffff00 broadcast 192.147.25.255
inet6 fe80::230:48ff:fe2e:9=
9ba%em0 prefixlen 64 scopeid 0x1
inet 192.147.25.45 netmask 0xffffff=
00 broadcast 192.147.25.255
inet 192.147.25.11 netmask 0xffffff00 br=
oadcast 192.147.25.255
nd6 options=3D3<PERFORMNUD,ACCEPT_RTADV>
=
media: Ethernet autoselect (100baseTX <full-duplex>)
status: acti=
ve
$
$ uname -a
FreeBSD thebighonker.lerctr.org 8.2-STABLE FreeBSD 8.2-STAB=
LE #45:
Sat Oct 8 10:57:43 CDT 2011
root@thebighonker.lerctr.org:/usr/obj/=
usr/src/sys/THEBIGHONKER
amd64
$
$ ifconfig em0
em0: flags=3D8843<UP,BRO=
ADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=3D2088<V=
LAN_MTU,VLAN_HWCSUM,WOL_MAGIC>
ether 00:30:48:8e:9f:f3
inet 1=
92.168.200.4 netmask 0xffffff00 broadcast 192.168.200.255
inet6 fe80=
::230:48ff:fe8e:9ff3%em0 prefixlen 64 scopeid 0x1
nd6 options=3D29<P=
ERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (100=
0baseT <full-duplex>)
status: active
$ uname -a
FreeBSD borg.lerctr.=
org 9.0-BETA3 FreeBSD 9.0-BETA3 #1: Sun Oct 9
10:03:42 CDT 2011
root@borg.=
lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64
$
Can you please provi=
de output from the following commands executed on
the machine showing the p=
roblem? The above commands show nothing
useful, other than the fact that o=
ne machine is at 100/full and the
other is at 1000/full (I don't know your =
network setup). Commands:
* netstat -inbd -I em0
* sysctl -a dev.em.0
* I=
ssue command "sysctl dev.em.0.debug=3D1", then type "dmesg" and
provide al=
l of the new output you will see at the bottom that
pertains to the NIC
I=
f you Google this problem, you will find that the majority of the time
it's=
caused by NIC drivers acting oddly.
Also, I believe the em(4) driver in 9=
.x is slightly different than on
8.x, so I'm CC'ing Jack Vogel here.
fro=
m 9.0:
Name Mtu Network Address Ipkts Ierrs Idrop =
Ibytes Opkts Oerrs Obytes Coll Drop
em0 1500 <Link#1> 00:=
30:48:8e:9f:f3 69776975 0 0 59660392277 52592789 0 104743924118=
0 0 em0 1500 192.168.200.0 192.168.200.4 69759773 - =
- 58681934612 96397272 - 104003761109 - - em0 1500 fe80::230:=
48f fe80::230:48ff:fe 0 - - 0 3 - =
248 - -
dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.=
3
dev.em.0.%driver: em
dev.em.0.%location: slot=3D0 function=3D0
dev.em.0.%=
pnpinfo: vendor=3D0x8086 device=3D0x1096 subvendor=3D0x15d9 subdevice=3D0x0=
000 class=3D0x020000
dev.em.0.%parent: pci6
dev.em.0.nvm: -1
dev.em.0.debug=
: -1
dev.em.0.rx_int_delay: 0
dev.em.0.tx_int_delay: 66
dev.em.0.rx_abs_int=
_delay: 66
dev.em.0.tx_abs_int_delay: 66
dev.em.0.rx_processing_limit: 100
=
dev.em.0.flow_control: 3
dev.em.0.eee_control: 0
dev.em.0.link_irq: 0
dev.e=
m.0.mbuf_alloc_fail: 0
dev.em.0.cluster_alloc_fail: 0
dev.em.0.dropped: 0
d=
ev.em.0.tx_dma_fail: 21755
dev.em.0.rx_overruns: 0
dev.em.0.watchdog_timeou=
ts: 0
dev.em.0.device_control: 1851969
dev.em.0.rx_control: 67141634
dev.em=
.0.fc_high_water: 30720
dev.em.0.fc_low_water: 29220
dev.em.0.queue0.txd_he=
ad: 136
dev.em.0.queue0.txd_tail: 136
dev.em.0.queue0.tx_irq: 0
dev.em.0.qu=
eue0.no_desc_avail: 0
dev.em.0.queue0.rxd_head: 2
dev.em.0.queue0.rxd_tail:=
1
dev.em.0.queue0.rx_irq: 0
dev.em.0.mac_stats.excess_coll: 0
dev.em.0.mac=
_stats.single_coll: 0
dev.em.0.mac_stats.multiple_coll: 0
dev.em.0.mac_stat=
s.late_coll: 0
dev.em.0.mac_stats.collision_count: 0
dev.em.0.mac_stats.sym=
bol_errors: 0
dev.em.0.mac_stats.sequence_errors: 0
dev.em.0.mac_stats.defe=
r_count: 0
dev.em.0.mac_stats.missed_packets: 0
dev.em.0.mac_stats.recv_no_=
buff: 0
dev.em.0.mac_stats.recv_undersize: 0
dev.em.0.mac_stats.recv_fragme=
nted: 0
dev.em.0.mac_stats.recv_oversize: 0
dev.em.0.mac_stats.recv_jabber:=
0
dev.em.0.mac_stats.recv_errs: 0
dev.em.0.mac_stats.crc_errs: 0
dev.em.0.=
mac_stats.alignment_errs: 0
dev.em.0.mac_stats.coll_ext_errs: 0
dev.em.0.ma=
c_stats.xon_recvd: 0
dev.em.0.mac_stats.xon_txd: 0
dev.em.0.mac_stats.xoff_=
recvd: 0
dev.em.0.mac_stats.xoff_txd: 0
dev.em.0.mac_stats.total_pkts_recvd=
: 69774324
dev.em.0.mac_stats.good_pkts_recvd: 69774324
dev.em.0.mac_stats.=
bcast_pkts_recvd: 28156
dev.em.0.mac_stats.mcast_pkts_recvd: 1758
dev.em.0.=
mac_stats.rx_frames_64: 54177
dev.em.0.mac_stats.rx_frames_65_127: 30157358=
dev.em.0.mac_stats.rx_frames_128_255: 1092948
dev.em.0.mac_stats.rx_frames=
_256_511: 125295
dev.em.0.mac_stats.rx_frames_512_1023: 128081
dev.em.0.mac=
_stats.rx_frames_1024_1522: 38216465
dev.em.0.mac_stats.good_octets_recvd: =
59938624047
dev.em.0.mac_stats.good_octets_txd: 106613836902
dev.em.0.mac_s=
tats.total_pkts_txd: 96250538
dev.em.0.mac_stats.good_pkts_txd: 96250538
de=
v.em.0.mac_stats.bcast_pkts_txd: 2989
dev.em.0.mac_stats.mcast_pkts_txd: 0
=
dev.em.0.mac_stats.tx_frames_64: 7551
dev.em.0.mac_stats.tx_frames_65_127: =
26727682
dev.em.0.mac_stats.tx_frames_128_255: 227574
dev.em.0.mac_stats.tx=
_frames_256_511: 167383
dev.em.0.mac_stats.tx_frames_512_1023: 302141
dev.e=
m.0.mac_stats.tx_frames_1024_1522: 68818207
dev.em.0.mac_stats.tso_txd: 172=
44234
dev.em.0.mac_stats.tso_ctx_fail: 0
dev.em.0.interrupts.asserts: 65945=
396
dev.em.0.interrupts.rx_pkt_timer: 8917
dev.em.0.interrupts.rx_abs_timer=
: 0
dev.em.0.interrupts.tx_pkt_timer: 1461
dev.em.0.interrupts.tx_abs_timer=
: 1951
dev.em.0.interrupts.tx_queue_empty: 0
dev.em.0.interrupts.tx_queue_m=
in_thresh: 0
dev.em.0.interrupts.rx_desc_min_thresh: 0
dev.em.0.interrupts.=
rx_overrun: 0
Interface is RUNNING and INACTIVE
em0: hw tdh =3D 221, hw td=
t =3D 221
em0: hw rdh =3D 467, hw rdt =3D 466
em0: Tx Queue Status =3D 0
em=
0: TX descriptors avail =3D 1024
em0: Tx Descriptors avail failure =3D 0
em=
0: RX discarded packets =3D 0
em0: RX Next to Check =3D 467
em0: RX Next to=
Refresh =3D 466
$
from 8.2:
Name Mtu Network Address =
Ipkts Ierrs Idrop Ibytes Opkts Oerrs Obytes Coll Drop
em0 =
1500 <Link#1> 00:30:48:2e:99:ba 276150 0 0 42614583 285=
398 0 207023352 0 0 em0 1500 192.147.25.0/ 192.147.25.65 =
318062 - - 48331105 285363 - 203026287 - - em0 =
1500 fe80::230:48f fe80::230:48ff:fe 0 - - 0 =
1 - 96 - - em0 1500 192.147.25.0/ 192.147.25.45 =
25071 - - 1782211 0 - 0 - - em0 =
1500 192.147.25.0/ 192.147.25.11 38433 - - 2742827 =
0 - 0 - -
dev.em.0.%desc: Intel(R) PRO/1000 Legac=
y Network Connection 1.0.3
dev.em.0.%driver: em
dev.em.0.%location: slot=3D=
2 function=3D0
dev.em.0.%pnpinfo: vendor=3D0x8086 device=3D0x1079 subvendor=
=3D0x15d9 subdevice=3D0x117a class=3D0x020000
dev.em.0.%parent: pci3
dev.em=
.0.nvm: -1
dev.em.0.rx_int_delay: 0
dev.em.0.tx_int_delay: 66
dev.em.0.rx_a=
bs_int_delay: 66
dev.em.0.tx_abs_int_delay: 66
dev.em.0.rx_processing_limit=
: 100
dev.em.0.flow_control: 3
dev.em.0.mbuf_alloc_fail: 0
dev.em.0.cluster=
_alloc_fail: 0
dev.em.0.dropped: 0
dev.em.0.tx_dma_fail: 0
dev.em.0.tx_desc=
_fail1: 0
dev.em.0.tx_desc_fail2: 0
dev.em.0.rx_overruns: 0
dev.em.0.watchd=
og_timeouts: 0
dev.em.0.device_control: 1089471041
dev.em.0.rx_control: 327=
70
dev.em.0.fc_high_water: 47104
dev.em.0.fc_low_water: 45604
dev.em.0.fifo=
_workaround: 0
dev.em.0.fifo_reset: 0
dev.em.0.txd_head: 73
dev.em.0.txd_ta=
il: 75
dev.em.0.rxd_head: 130
dev.em.0.rxd_tail: 129
dev.em.0.mac_stats.exc=
ess_coll: 0
dev.em.0.mac_stats.single_coll: 0
dev.em.0.mac_stats.multiple_c=
oll: 0
dev.em.0.mac_stats.late_coll: 0
dev.em.0.mac_stats.collision_count: =
0
dev.em.0.mac_stats.symbol_errors: 0
dev.em.0.mac_stats.sequence_errors: 0=
dev.em.0.mac_stats.defer_count: 0
dev.em.0.mac_stats.missed_packets: 0
dev=
.em.0.mac_stats.recv_no_buff: 0
dev.em.0.mac_stats.recv_undersize: 0
dev.em=
.0.mac_stats.recv_fragmented: 0
dev.em.0.mac_stats.recv_oversize: 0
dev.em.=
0.mac_stats.recv_jabber: 0
dev.em.0.mac_stats.recv_errs: 0
dev.em.0.mac_sta=
ts.crc_errs: 0
dev.em.0.mac_stats.alignment_errs: 0
dev.em.0.mac_stats.coll=
_ext_errs: 0
dev.em.0.mac_stats.xon_recvd: 0
dev.em.0.mac_stats.xon_txd: 0
=
dev.em.0.mac_stats.xoff_recvd: 0
dev.em.0.mac_stats.xoff_txd: 0
dev.em.0.ma=
c_stats.total_pkts_recvd: 276318
dev.em.0.mac_stats.good_pkts_recvd: 276318=
dev.em.0.mac_stats.bcast_pkts_recvd: 8
dev.em.0.mac_stats.mcast_pkts_recvd=
: 0
dev.em.0.mac_stats.rx_frames_64: 9012
dev.em.0.mac_stats.rx_frames_65_1=
27: 205540
dev.em.0.mac_stats.rx_frames_128_255: 44078
dev.em.0.mac_stats.r=
x_frames_256_511: 3166
dev.em.0.mac_stats.rx_frames_512_1023: 3934
dev.em.0=
.mac_stats.rx_frames_1024_1522: 10588
dev.em.0.mac_stats.good_octets_recvd:=
43761022
dev.em.0.mac_stats.good_octets_txd: 208238998
dev.em.0.mac_stats.=
total_pkts_txd: 285534
dev.em.0.mac_stats.good_pkts_txd: 285534
dev.em.0.ma=
c_stats.bcast_pkts_txd: 22
dev.em.0.mac_stats.mcast_pkts_txd: 3
dev.em.0.ma=
c_stats.tx_frames_64: 6172
dev.em.0.mac_stats.tx_frames_65_127: 75983
dev.e=
m.0.mac_stats.tx_frames_128_255: 53030
dev.em.0.mac_stats.tx_frames_256_511=
: 23216
dev.em.0.mac_stats.tx_frames_512_1023: 1472
dev.em.0.mac_stats.tx_f=
rames_1024_1522: 125661
dev.em.0.mac_stats.tso_txd: 0
dev.em.0.mac_stats.ts=
o_ctx_fail: 0
$ sudo sysctl dev.em.0.debug=3D1
sysctl: unknown oid 'dev.em=
.0.debug'
$
--
Larry Rosenman http://www.lerctr.org/~=
ler
Phone: +1 512-248-2683 E-Mail: ler@lerctr.org
US Mail: =
430 Valona Loop, Round Rock, TX 78681-3893
________________________________=
_______________
freebsd-stable@freebsd.org mailing list
http://lists.freebs=
d.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "fre=
ebsd-stable-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6590d023-cd29-4d98-a3b5-161aa14bd006>