From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 13 14:50:25 2014 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 67E409C8; Thu, 13 Feb 2014 14:50:25 +0000 (UTC) Received: from mho-01-ewr.mailhop.org (mho-03-ewr.mailhop.org [204.13.248.66]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 259A31B9C; Thu, 13 Feb 2014 14:50:24 +0000 (UTC) Received: from c-24-8-230-52.hsd1.co.comcast.net ([24.8.230.52] helo=damnhippie.dyndns.org) by mho-01-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1WDxcG-000KrL-Ri; Thu, 13 Feb 2014 14:50:17 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id s1DEoCtp075195; Thu, 13 Feb 2014 07:50:12 -0700 (MST) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 24.8.230.52 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX18JYactPpInJZASldI6+v6A Subject: Re: Receiving jumbo frames From: Ian Lepore To: Viktor Penkoff In-Reply-To: References: Content-Type: text/plain; charset="koi8-r" Date: Thu, 13 Feb 2014 07:50:12 -0700 Message-ID: <1392303012.1145.81.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by damnhippie.dyndns.org id s1DEoCtp075195 Cc: freebsd-hackers@FreeBSD.org, freebsd-arm X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Feb 2014 14:50:25 -0000 On Thu, 2014-02-13 at 11:05 +0200, Viktor Penkoff wrote: > Hi, folks! I'm writing an extension functionality to not-yet published > network driver. > I'm receiving the typical ethernet frames without problems. Considering= the > datasheet of the device, > I'm capable of receiving jumbo frames. When I try to do that, e.g. to s= end > jumbo frame of 8000 bytes, I'm receiving only a limited count of them = - > 105, then the kernel crashes with the following message: > "panic: vm_fault: fault on nofault entry, addr: cfcec000". > I have inspected a kernel dump with kgdb and the problem occurs at the > function bus_dmamap_sync. >=20 > The line that the panic happens is in a standard invoking receive > interrupt function. Packet data is stored in memory buffers, with any > single packet spanning multiple buffers if necessary. The buffers are > allocated by the CPU and are managed through chained descriptor lists. > When I'm sending jumbo packets, e.g. 8008 bytes in size each, due to li= ve > debugging with jtag and kgdb, I've got the following information for th= e > mbuf: >=20 > % ping -I eth0 -c 1 -s 8000 -M dont A.B.C.D -v -p ff >=20 > {m_hdr =3D {mh_next =3D 0x0, mh_nextpkt =3D 0x0, mh_data =3D 0xc3d41800= "", > mh_len =3D 2048, mh_flags =3D 3, mh_type =3D 1, pad =3D "\000"}, M_= dat =3D { > MH =3D {MH_pkthdr =3D {rcvif =3D 0x0, header =3D 0x0, len =3D 2048,= flowid =3D 0, > csum_flags =3D 0, csum_data =3D 0, tso_segsz =3D 0, PH_vt =3D { > vt_vtag =3D 0, vt_nrecs =3D 0}, tags =3D {slh_first =3D 0x0}, > dsa_tag =3D {0, 0}}, MH_dat =3D {MH_ext =3D {ext_buf =3D 0xc3d4= 1800 "", > ext_free =3D 0, ext_arg1 =3D 0x0, ext_arg2 =3D 0x0, ext_size = =3D 2048, > ref_cnt =3D 0xc3d3f244, ext_type =3D 6}, >=20 > In this case, the given mbuf must occupy ~4 descriptors, IMO. Following= the > next pointer in the descriptor list shows that the CPU has > an ownership. Nevertheless, =D4he byte count of the received frame show= s that > 8056 bytes are received. >=20 > More interesting is the fact that an echo reply is given back. But aft= er a > constant count of received packets, the kernel hangs. >=20 > Some background information: > To enable the jumbo frame, one must set the appropriate register. > At the software level, a ring buffer with the descriptors is implemente= d. > The device, under which the driver runs, is arm-based. > The freebsd version is 8.0. The device uses SerialDMA queues for > transmitting and receiving. > To receive packets, the CPU must perform the following: > 1. Prepare a linked list of descriptors > 2. Configure a given queue with the address of the first descriptor= in > the list, > 3. enable SerialDMA; >=20 > With the transmission - I don't have any problems. The logic is the sam= e as > by the reception of packets - ring buffer with descriptors. >=20 > Any ideas what can cause this type of crashes? You mention FreeBSD 8.0, that's pretty old. There have been many fixes to the arm busdma code since then. We still use 8.2 at work, and while we know there are problems with the busdma code that have been fixed in later revisions, they aren't affecting us so we leave well enough alone. We don't use jumbo frames though. At the very least you should apply the changeset r236086 to your code, although it's not all that likely to fix your problem because the mbufs you're working with should be aligned on 2k boundaries and never require a partial cacheline flush. Are you allocating the buffers with bus_dmamem_alloc() then attaching them to mbuf chains as external buffers? Or are you using one of the mbuf allocation routines? When interfacing with the busdma code, are you using bus_dmamap_load_mbuf()? Posting the full backtrace from the panic might help. More information about the arm chip/system you're using might be helpful too. (I've added the freebsd-arm mailing list to the CC, this is pretty likely to be an arm-specific problem rather than a general freebsd thing.) -- Ian