From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 05:33:03 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5A90F16A52F for ; Sun, 3 Feb 2008 05:33:03 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outD.internet-mail-service.net (outD.internet-mail-service.net [216.240.47.227]) by mx1.freebsd.org (Postfix) with ESMTP id A4D0113C474 for ; Sun, 3 Feb 2008 05:32:59 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Sat, 02 Feb 2008 21:32:58 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id E13641270B9; Sat, 2 Feb 2008 21:32:57 -0800 (PST) Message-ID: <47A55209.9040106@elischer.org> Date: Sat, 02 Feb 2008 21:32:57 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Robert Watson References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> <20080202224923.T66602@fledge.watson.org> In-Reply-To: <20080202224923.T66602@fledge.watson.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Kris Kennaway , Alexander Motin , freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 05:33:03 -0000 Robert Watson wrote: > be a good time to try to revalidate that. Basically, the goal would be > to make the pcpu cache FIFO as much as possible as that maximizes the you mean FILO or LIFO right? > chances that the newly allocated object already has lines in the cache. > It's a fairly trivial tweak to the UMA allocation code. > > Robert N M Watson > Computer Laboratory > University of Cambridge From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 12:04:58 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 48CE716A46C; Sun, 3 Feb 2008 12:04:58 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id F09D913C458; Sun, 3 Feb 2008 12:04:57 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 87B21207F; Sun, 3 Feb 2008 13:04:49 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.2/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 65D77207E; Sun, 3 Feb 2008 13:04:49 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 405358449D; Sun, 3 Feb 2008 13:04:49 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Julian Elischer References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> <20080202224923.T66602@fledge.watson.org> <47A55209.9040106@elischer.org> Date: Sun, 03 Feb 2008 13:04:48 +0100 In-Reply-To: <47A55209.9040106@elischer.org> (Julian Elischer's message of "Sat\, 02 Feb 2008 21\:32\:57 -0800") Message-ID: <86ejbumejj.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Kris Kennaway , Robert Watson , Alexander Motin , freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 12:04:58 -0000 Julian Elischer writes: > Robert Watson writes: > > be a good time to try to revalidate that. Basically, the goal would > > be to make the pcpu cache FIFO as much as possible as that maximizes > > the chances that the newly allocated object already has lines in the > > cache. It's a fairly trivial tweak to the UMA allocation code. > you mean FILO or LIFO right? Uh, no. You want to reuse the last-freed object, as it is most likely to still be in cache. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 12:58:17 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1998116A421 for ; Sun, 3 Feb 2008 12:58:17 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id D69BF13C455 for ; Sun, 3 Feb 2008 12:58:16 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 2B4A92084 for ; Sun, 3 Feb 2008 13:58:08 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.2/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id AA7C62083 for ; Sun, 3 Feb 2008 13:58:07 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 98484844B3; Sun, 3 Feb 2008 13:58:07 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: hackers@freebsd.org Date: Sun, 03 Feb 2008 13:58:07 +0100 Message-ID: <8663x6mc2o.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Subject: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 12:58:17 -0000 I've been trying to figure out why some periodic scripts consume so much memory. I've narrowed it down to sort(1). At first, I thought the scripts were using it inefficiently, feeding it more data than was really needed. Then I discovered this: des@ds4 ~% (sleep 10 | sort) & (sleep 5 ; top -o res | grep sort) [1] 66024 66024 des 1 -8 5 54796K 52680K piperd 1 0:00 0.88% sort That's right - sort(1) consumes 50+ MB of memory doing *nothing*. (roughly half that on a 32-bit box) Something is rotten in the state of GNU... DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 13:13:12 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0719416A418 for ; Sun, 3 Feb 2008 13:13:12 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: from palm.hoeg.nl (mx0.hoeg.nl [IPv6:2001:610:652::211]) by mx1.freebsd.org (Postfix) with ESMTP id C5BC313C46B for ; Sun, 3 Feb 2008 13:13:11 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: by palm.hoeg.nl (Postfix, from userid 1000) id 599091CC4E; Sun, 3 Feb 2008 14:13:22 +0100 (CET) Date: Sun, 3 Feb 2008 14:13:22 +0100 From: Ed Schouten To: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= Message-ID: <20080203131322.GK1179@hoeg.nl> References: <8663x6mc2o.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="aqWxf8ydqYKP8htK" Content-Disposition: inline In-Reply-To: <8663x6mc2o.fsf@ds4.des.no> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: hackers@freebsd.org Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 13:13:12 -0000 --aqWxf8ydqYKP8htK Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable * Dag-Erling Sm=F8rgrav wrote: > I've been trying to figure out why some periodic scripts consume so much > memory. I've narrowed it down to sort(1). >=20 > At first, I thought the scripts were using it inefficiently, feeding it > more data than was really needed. Then I discovered this: >=20 > des@ds4 ~% (sleep 10 | sort) & (sleep 5 ; top -o res | grep sort) > [1] 66024 > 66024 des 1 -8 5 54796K 52680K piperd 1 0:00 0.88% sort >=20 > That's right - sort(1) consumes 50+ MB of memory doing *nothing*. >=20 > (roughly half that on a 32-bit box) >=20 > Something is rotten in the state of GNU... On my i386 box it spends 27M, but when I replace `sort' with `sed', without any arguments, it's only 1.4 MB. I tried this on RELENG_6. I can also reproduce this on Linux. --=20 Ed Schouten WWW: http://g-rave.nl/ --aqWxf8ydqYKP8htK Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEARECAAYFAkelvfIACgkQ52SDGA2eCwVMMgCfSr+69jompdlAkN3rkIc3UZIA iqEAn3YNc1ShXVZ6JkvgKXSddX7aOHZs =8rpC -----END PGP SIGNATURE----- --aqWxf8ydqYKP8htK-- From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 14:12:47 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3F19F16A418 for ; Sun, 3 Feb 2008 14:12:47 +0000 (UTC) (envelope-from wundram@beenic.net) Received: from mail.beenic.net (mail.beenic.net [83.246.72.40]) by mx1.freebsd.org (Postfix) with ESMTP id 0C40C13C45B for ; Sun, 3 Feb 2008 14:12:46 +0000 (UTC) (envelope-from wundram@beenic.net) Received: from phoenix (hnvr-4dbbaf92.pool.einsundeins.de [77.187.175.146]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.beenic.net (Postfix) with ESMTP id DAC49A44529 for ; Sun, 3 Feb 2008 14:49:04 +0100 (CET) From: "Heiko Wundram (Beenic)" Organization: Beenic Networks GmbH To: hackers@freebsd.org Date: Sun, 3 Feb 2008 14:50:18 +0100 User-Agent: KMail/1.9.7 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200802031450.19420.wundram@beenic.net> Cc: Subject: getaddrinfo() spec doesn't match behaviour X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 14:12:47 -0000 Hey all! Before I go post this as a PR (or go about fixing the libc code), I just wanted to ask whether this is a known issue (and I simply haven't been able to find it), or if it's simply my stupidity that makes this fail. Basically, I have the following code: addrinfo hints; addrinfo* res; // Fill in hints structure. memset(&hints,0,sizeof(hints)); hints.ai_flags = AI_PASSIVE | AI_V4MAPPED; hints.ai_family = SERV.m_ipv6 ? AF_INET6 : AF_INET; hints.ai_socktype = SOCK_STREAM; hints.ai_protocol = IPPROTO_TCP; // Query address info. if( ( rv = getaddrinfo(bindaddr,port,&hints,&res) ) ) { if( rv == EAI_SYSTEM ) throw OSError("Failed resolving service and/or bind address"); throw ValueError(gai_strerror(rv)); } Now, according to the man-page of getaddrinfo() and my previous knowledge, this should work fine (for some bindaddr and port coming into this function as const char*), but, alas, it doesn't, and fails with EAI_BADFLAGS. Tracing through the libc code for getaddrinfo(), the cause of this failing is pretty obvious: hints.ai_flags is logically anded with AI_MASK at the beginning of the function, and AI_MASK (at least in my local netdb.h header) does not contain the flag AI_V4MAPPED. In case the result of that is non-zero (which it is due to me specifying AI_V4MAPPED), the function returns EAI_BADFLAGS. After that, getaddrinfo() does some checks on AI_V4MAPPED and AI_ALL (masking the respective flags in case the ai_family isn't AF_INET6), which are basically superfluous (as they can never be set anyway due to AI_MASK), but I didn't find any other reference to the flag in the whole of lib/libc/net/getaddrinfo.c, so I actually don't know whether the cause of this failing is that it simply isn't implemented (completely), or there's any other reason for AI_MASK not to contain these flags that I haven't grasped so far. If anyone out there can shed a hint on this, I'd be grateful, even if it's just the fact that my netdb.h installation is broken. Thanks! -- Heiko Wundram Product & Application Development From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 15:23:34 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C33C916A421 for ; Sun, 3 Feb 2008 15:23:34 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 7F39713C46B for ; Sun, 3 Feb 2008 15:23:34 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 88EAF207E; Sun, 3 Feb 2008 16:23:26 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.2/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 0253D2049; Sun, 3 Feb 2008 16:23:25 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id D3791844B3; Sun, 3 Feb 2008 16:23:25 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Ed Schouten References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> <20080203151550.GA67020@owl.midgard.homeip.net> Date: Sun, 03 Feb 2008 16:23:25 +0100 In-Reply-To: <20080203151550.GA67020@owl.midgard.homeip.net> (Erik Trulsson's message of "Sun\, 3 Feb 2008 16\:15\:50 +0100") Message-ID: <86prvekqs2.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: hackers@freebsd.org Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 15:23:34 -0000 Erik Trulsson writes: > Yep, it seems that GNU sort allocates a quite large buffer by default when > the size of the input is unknown (such as when it reads input from stdin.) > A quick check in the source code indicates that it tries to size this buf= fer > according to how much memory the system has (and according to any limits = set > on how much memory the process is allowed to use.) Uh, OK. This scaling doesn't seem to work correctly. It seems to allocate 27 MB on 32-bit machines and 54 MB on 64-bit machines, regardless of memory size. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 15:31:03 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AEC2216A417 for ; Sun, 3 Feb 2008 15:31:03 +0000 (UTC) (envelope-from erikt@midgard.homeip.net) Received: from ch-smtp01.sth.basefarm.net (ch-smtp01.sth.basefarm.net [80.76.149.212]) by mx1.freebsd.org (Postfix) with ESMTP id 7025C13C4E1 for ; Sun, 3 Feb 2008 15:31:03 +0000 (UTC) (envelope-from erikt@midgard.homeip.net) Received: from c83-253-25-183.bredband.comhem.se ([83.253.25.183]:55047 helo=falcon.midgard.homeip.net) by ch-smtp01.sth.basefarm.net with esmtp (Exim 4.68) (envelope-from ) id 1JLgZX-0005Jg-4n for hackers@freebsd.org; Sun, 03 Feb 2008 16:15:55 +0100 Received: (qmail 72769 invoked from network); 3 Feb 2008 16:15:50 +0100 Received: from owl.midgard.homeip.net (10.1.5.7) by falcon.midgard.homeip.net with ESMTP; 3 Feb 2008 16:15:50 +0100 Received: (qmail 67211 invoked by uid 1001); 3 Feb 2008 16:15:50 +0100 Date: Sun, 3 Feb 2008 16:15:50 +0100 From: Erik Trulsson To: Ed Schouten Message-ID: <20080203151550.GA67020@owl.midgard.homeip.net> Mail-Followup-To: Ed Schouten , Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= , hackers@freebsd.org References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <20080203131322.GK1179@hoeg.nl> User-Agent: Mutt/1.5.17 (2007-11-01) X-Originating-IP: 83.253.25.183 X-Scan-Result: No virus found in message 1JLgZX-0005Jg-4n. X-Scan-Signature: ch-smtp01.sth.basefarm.net 1JLgZX-0005Jg-4n 2add0a7beedd1263bb69711c98ee2a3a Cc: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= , hackers@freebsd.org Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 15:31:03 -0000 On Sun, Feb 03, 2008 at 02:13:22PM +0100, Ed Schouten wrote: > * Dag-Erling Sm=F8rgrav wrote: > > I've been trying to figure out why some periodic scripts consume so much > > memory. I've narrowed it down to sort(1). > >=20 > > At first, I thought the scripts were using it inefficiently, feeding it > > more data than was really needed. Then I discovered this: > >=20 > > des@ds4 ~% (sleep 10 | sort) & (sleep 5 ; top -o res | grep sort) > > [1] 66024 > > 66024 des 1 -8 5 54796K 52680K piperd 1 0:00 0.88% sort > >=20 > > That's right - sort(1) consumes 50+ MB of memory doing *nothing*. > >=20 > > (roughly half that on a 32-bit box) > >=20 > > Something is rotten in the state of GNU... >=20 > On my i386 box it spends 27M, but when I replace `sort' with `sed', > without any arguments, it's only 1.4 MB. I tried this on RELENG_6. I can > also reproduce this on Linux. >=20 Yep, it seems that GNU sort allocates a quite large buffer by default when the size of the input is unknown (such as when it reads input from stdin.) A quick check in the source code indicates that it tries to size this buffer according to how much memory the system has (and according to any limits set on how much memory the process is allowed to use.) The size of this buffer can be controlled with the --buffer-size option to sort(1). --=20 Erik Trulsson ertr1013@student.uu.se From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 15:31:43 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C651316A46C for ; Sun, 3 Feb 2008 15:31:43 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 8C45613C507 for ; Sun, 3 Feb 2008 15:31:43 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id EB11E2082; Sun, 3 Feb 2008 16:31:34 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.2/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id DCCE1207F; Sun, 3 Feb 2008 16:31:34 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id C4688844B3; Sun, 3 Feb 2008 16:31:34 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Ed Schouten References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> <20080203151550.GA67020@owl.midgard.homeip.net> <86prvekqs2.fsf@ds4.des.no> Date: Sun, 03 Feb 2008 16:31:34 +0100 In-Reply-To: <86prvekqs2.fsf@ds4.des.no> ("Dag-Erling =?utf-8?Q?Sm=C3=B8rg?= =?utf-8?Q?rav=22's?= message of "Sun\, 03 Feb 2008 16\:23\:25 +0100") Message-ID: <86lk62kqeh.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: hackers@freebsd.org Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 15:31:43 -0000 Dag-Erling Sm=C3=B8rgrav writes: > Erik Trulsson writes: > > Yep, it seems that GNU sort allocates a quite large buffer by default w= hen > > the size of the input is unknown (such as when it reads input from stdi= n.) > > A quick check in the source code indicates that it tries to size this b= uffer > > according to how much memory the system has (and according to any limit= s set > > on how much memory the process is allowed to use.) > Uh, OK. This scaling doesn't seem to work correctly. It seems to > allocate 27 MB on 32-bit machines and 54 MB on 64-bit machines, > regardless of memory size. Looking at the code, it seems to go to extreme lengths to get it absolutely wrong. For instance, if hw.physmem / 8 > hw.usermem, it will pick the former, which means it's pretty much guaranteed to either fail or hose your system (or both). In the immortal words of Blazing Star: YOU FAIL IT Count this as a vote for ditching GNU sort in favor of a BSD-licensed implementation (from {Net,Open}BSD for instance). DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 15:37:36 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DCCFC16A417 for ; Sun, 3 Feb 2008 15:37:36 +0000 (UTC) (envelope-from ws@au.dyndns.ws) Received: from ipmail05.adl2.internode.on.net (ipmail05.adl2.internode.on.net [203.16.214.145]) by mx1.freebsd.org (Postfix) with ESMTP id 78DB213C459 for ; Sun, 3 Feb 2008 15:37:35 +0000 (UTC) (envelope-from ws@au.dyndns.ws) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgAAAI9opUeWZWdv/2dsb2JhbAAIq3c X-IronPort-AV: E=Sophos;i="4.25,298,1199626200"; d="scan'208";a="47243121" Received: from ppp103-111.static.internode.on.net (HELO [192.168.1.131]) ([150.101.103.111]) by ipmail05.adl2.internode.on.net with ESMTP; 04 Feb 2008 01:52:17 +1030 From: Wayne Sierke To: Ed Schouten In-Reply-To: <20080203131322.GK1179@hoeg.nl> References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> Content-Type: text/plain; charset=ISO-8859-1 Date: Mon, 04 Feb 2008 01:52:15 +1030 Message-Id: <1202052136.5350.137.camel@predator-ii.buffyverse> Mime-Version: 1.0 X-Mailer: Evolution 2.12.3 FreeBSD GNOME Team Port Content-Transfer-Encoding: 8bit Cc: Dag-Erling =?ISO-8859-1?Q?Sm=F8rgrav?= , hackers@freebsd.org Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 15:37:36 -0000 On Sun, 2008-02-03 at 14:13 +0100, Ed Schouten wrote: > * Dag-Erling Smørgrav wrote: > > I've been trying to figure out why some periodic scripts consume so much > > memory. I've narrowed it down to sort(1). > > > > At first, I thought the scripts were using it inefficiently, feeding it > > more data than was really needed. Then I discovered this: > > > > des@ds4 ~% (sleep 10 | sort) & (sleep 5 ; top -o res | grep sort) > > [1] 66024 > > 66024 des 1 -8 5 54796K 52680K piperd 1 0:00 0.88% sort > > > > That's right - sort(1) consumes 50+ MB of memory doing *nothing*. > > > > (roughly half that on a 32-bit box) > > > > Something is rotten in the state of GNU... > > On my i386 box it spends 27M, but when I replace `sort' with `sed', > without any arguments, it's only 1.4 MB. I tried this on RELENG_6. I can > also reproduce this on Linux. > %uname -vm FreeBSD 7.0-PRERELEASE #1: Fri Jan 25 01:08:47 CST 2008 root@freebsd7stable-1.vmware:/usr/obj/usr/src/sys/GENERIC-KTR-0x2000 i386 %( sleep 10 | sort ) & ( sleep 5 ; top -n 150 | grep sort ) [2] 38158 38158 ws 1 -8 0 29760K 736K piperd 0:00 0.00% sort %su - # ( sleep 10 | sort ) & ( sleep 5 ; top -n 150 | grep sort ) [2] 38165 38165 root 1 -8 0 29760K 732K piperd 0:00 0.00% sort $ uname -vm FreeBSD 6.3-PRERELEASE #1: Fri Dec 28 17:49:43 CST 2007 root@predator-ii.buffyverse:/usr/obj/usr/src/sys/LILLITH-IV i386 $ (sleep 10 | sort) & (sleep 5; top -n 150 | grep sort) 68953 ws 1 -8 0 26988K 660K piperd 0:00 0.00% sort $ su - # (sleep 10 | sort) & (sleep 5; top -n 150 | grep sort) [1] 68981 68981 root 1 -8 0 26988K 660K piperd 0:00 0.00% sort Next one is Ubuntu 7.04 $ uname -a Linux developer 2.6.20-16-generic #2 SMP Thu Jun 7 20:19:32 UTC 2007 i686 GNU/Linux $ (sleep 10 | sort) & (sleep 5; ps aux | grep -E "^USER|sort$") [9] 10523 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND ws 10526 0.0 0.0 28436 676 pts/4 S 01:29 0:00 sort (I had to change the 'top' incantation to pick up the 'sort' process on these busy boxes.) Wayne From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 17:16:50 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3F37216A418 for ; Sun, 3 Feb 2008 17:16:50 +0000 (UTC) (envelope-from erikt@midgard.homeip.net) Received: from ch-smtp02.sth.basefarm.net (ch-smtp02.sth.basefarm.net [80.76.149.213]) by mx1.freebsd.org (Postfix) with ESMTP id BFE9913C467 for ; Sun, 3 Feb 2008 17:16:49 +0000 (UTC) (envelope-from erikt@midgard.homeip.net) Received: from c83-253-25-183.bredband.comhem.se ([83.253.25.183]:60963 helo=falcon.midgard.homeip.net) by ch-smtp02.sth.basefarm.net with esmtp (Exim 4.68) (envelope-from ) id 1JLiSW-00024Y-7r for hackers@freebsd.org; Sun, 03 Feb 2008 18:16:48 +0100 Received: (qmail 73413 invoked from network); 3 Feb 2008 18:16:44 +0100 Received: from owl.midgard.homeip.net (10.1.5.7) by falcon.midgard.homeip.net with ESMTP; 3 Feb 2008 18:16:44 +0100 Received: (qmail 67970 invoked by uid 1001); 3 Feb 2008 18:16:44 +0100 Date: Sun, 3 Feb 2008 18:16:44 +0100 From: Erik Trulsson To: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= Message-ID: <20080203171644.GA67839@owl.midgard.homeip.net> Mail-Followup-To: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= , Ed Schouten , hackers@freebsd.org References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> <20080203151550.GA67020@owl.midgard.homeip.net> <86prvekqs2.fsf@ds4.des.no> <86lk62kqeh.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <86lk62kqeh.fsf@ds4.des.no> User-Agent: Mutt/1.5.17 (2007-11-01) X-Originating-IP: 83.253.25.183 X-Scan-Result: No virus found in message 1JLiSW-00024Y-7r. X-Scan-Signature: ch-smtp02.sth.basefarm.net 1JLiSW-00024Y-7r 42ecd9198509d9ba64d2151713342fd8 Cc: hackers@freebsd.org, Ed Schouten Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 17:16:50 -0000 On Sun, Feb 03, 2008 at 04:31:34PM +0100, Dag-Erling Sm=F8rgrav wrote: > Dag-Erling Sm=F8rgrav writes: > > Erik Trulsson writes: > > > Yep, it seems that GNU sort allocates a quite large buffer by default= when > > > the size of the input is unknown (such as when it reads input from st= din.) > > > A quick check in the source code indicates that it tries to size this= buffer > > > according to how much memory the system has (and according to any lim= its set > > > on how much memory the process is allowed to use.) > > > > Uh, OK. This scaling doesn't seem to work correctly. It seems to > > allocate 27 MB on 32-bit machines and 54 MB on 64-bit machines, > > regardless of memory size. I said it *tries* to the size the buffer according the amount of memory available. I didn't say it succeded in doing so, or that it even made a good attempty at it. Those 27MB/54MB is probably because it hits some kind of limit. On a machine having only 64MB RAM, sort(1) "only" allocated 21MB adress spa= ce. I suspect the scaling algorithm was designed for older machines which rarely, if ever, had more than maybe 64MB RAM (and usually less than that), and that little thought was given to multi-gigabyte machines like those common today. >=20 > Looking at the code, it seems to go to extreme lengths to get it > absolutely wrong. For instance, if hw.physmem / 8 > hw.usermem, it will > pick the former, which means it's pretty much guaranteed to either fail > or hose your system (or both). >=20 > In the immortal words of Blazing Star: YOU FAIL IT >=20 > Count this as a vote for ditching GNU sort in favor of a BSD-licensed > implementation (from {Net,Open}BSD for instance). >=20 If any such implementation was a true drop-in replacement of GNU sort (supporting all the same options etc.) and did not have noticably worse performance, then I certainly would not raise any objections to that. --=20 Erik Trulsson ertr1013@student.uu.se From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 17:45:57 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3E4D116A418 for ; Sun, 3 Feb 2008 17:45:57 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: from palm.hoeg.nl (mx0.hoeg.nl [IPv6:2001:610:652::211]) by mx1.freebsd.org (Postfix) with ESMTP id 0A70F13C455 for ; Sun, 3 Feb 2008 17:45:57 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: by palm.hoeg.nl (Postfix, from userid 1000) id 75A171CE19; Sun, 3 Feb 2008 18:46:04 +0100 (CET) Date: Sun, 3 Feb 2008 18:46:04 +0100 From: Ed Schouten To: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= Message-ID: <20080203174604.GM1179@hoeg.nl> References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> <20080203151550.GA67020@owl.midgard.homeip.net> <86prvekqs2.fsf@ds4.des.no> <86lk62kqeh.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="2YJj5f1P6Th4nBRw" Content-Disposition: inline In-Reply-To: <86lk62kqeh.fsf@ds4.des.no> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: hackers@freebsd.org Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 17:45:57 -0000 --2YJj5f1P6Th4nBRw Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable * Dag-Erling Sm=F8rgrav wrote: > Count this as a vote for ditching GNU sort in favor of a BSD-licensed > implementation (from {Net,Open}BSD for instance). I just looked at the OpenBSD implementation and I can see it already misses one option that some people will miss, namely numeric sorting (-g). I don't know anything about NetBSD's implementation or how hard it is to add. --=20 Ed Schouten WWW: http://g-rave.nl/ --2YJj5f1P6Th4nBRw Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEARECAAYFAkel/dwACgkQ52SDGA2eCwUVYQCeIPFi5NlggFPf3DBZ5FrhH7Mr KFgAmwUm6L4B4vcAWnI4fUs2N+svzKSQ =+WPF -----END PGP SIGNATURE----- --2YJj5f1P6Th4nBRw-- From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 19:14:14 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 95D3616A421 for ; Sun, 3 Feb 2008 19:14:14 +0000 (UTC) (envelope-from ume@mahoroba.org) Received: from ameno.mahoroba.org (ent.mahoroba.org [IPv6:2001:2f0:104:8010::1]) by mx1.freebsd.org (Postfix) with ESMTP id 5E36513C4D3 for ; Sun, 3 Feb 2008 19:14:14 +0000 (UTC) (envelope-from ume@mahoroba.org) Received: from kasuga.mahoroba.org (IDENT:vOERczfgeRNjqtt4jKx5C76SbT39FMFw0fRiMhilKkEab/rpJuGbkdH2y4s/zVor@kasuga.mahoroba.org [IPv6:2001:2f0:104:8010:20b:97ff:fe2e:b521]) (user=ume mech=CRAM-MD5 bits=0) by ameno.mahoroba.org (8.13.8/8.13.8) with ESMTP/inet6 id m13JDmMA038735 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 4 Feb 2008 04:13:53 +0900 (JST) (envelope-from ume@mahoroba.org) Date: Mon, 04 Feb 2008 04:13:47 +0900 Message-ID: From: Hajimu UMEMOTO To: "Heiko Wundram (Beenic)" In-Reply-To: <200802031450.19420.wundram@beenic.net> References: <200802031450.19420.wundram@beenic.net> User-Agent: xcite1.57> Wanderlust/2.14.0 (Africa) SEMI/1.14.6 (Maruoka) FLIM/1.14.8 (=?ISO-8859-4?Q?Shij=F2?=) APEL/10.7 Emacs/22.1 (i386-pc-freebsd) MULE/5.0 (SAKAKI) X-Operating-System: FreeBSD 6.3-RELEASE X-PGP-Key: http://www.imasy.or.jp/~ume/publickey.asc X-PGP-Fingerprint: 1F00 0B9E 2164 70FC 6DC5 BF5F 04E9 F086 BF90 71FE Organization: Internet Mutual Aid Society, YOKOHAMA MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-3.0 (ameno.mahoroba.org [IPv6:2001:2f0:104:8010::1]); Mon, 04 Feb 2008 04:13:53 +0900 (JST) X-Virus-Scanned: by amavisd-new X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.2.4 X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on ameno.mahoroba.org Cc: hackers@freebsd.org Subject: Re: getaddrinfo() spec doesn't match behaviour X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 19:14:14 -0000 Hi, >>>>> On Sun, 3 Feb 2008 14:50:18 +0100 >>>>> "Heiko Wundram (Beenic)" said: wundram> hints.ai_flags is logically anded with AI_MASK at the beginning of the wundram> function, and AI_MASK (at least in my local netdb.h header) does not contain wundram> the flag AI_V4MAPPED. In case the result of that is non-zero (which it is due wundram> to me specifying AI_V4MAPPED), the function returns EAI_BADFLAGS. wundram> After that, getaddrinfo() does some checks on AI_V4MAPPED and AI_ALL (masking wundram> the respective flags in case the ai_family isn't AF_INET6), which are wundram> basically superfluous (as they can never be set anyway due to AI_MASK), but I wundram> didn't find any other reference to the flag in the whole of wundram> lib/libc/net/getaddrinfo.c, so I actually don't know whether the cause of wundram> this failing is that it simply isn't implemented (completely), or there's any wundram> other reason for AI_MASK not to contain these flags that I haven't grasped so wundram> far. Since the part is incomplete support of AI_ALL and AI_V4MAPPED, I've just nuked it. http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libc/net/getaddrinfo.c#rev1.87 Sincerely, -- Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan ume@mahoroba.org ume@{,jp.}FreeBSD.org http://www.imasy.org/~ume/ From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 21:15:22 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1727216A46E; Sun, 3 Feb 2008 21:15:22 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 360B713C459; Sun, 3 Feb 2008 21:15:21 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A62EE8.2000700@FreeBSD.org> Date: Sun, 03 Feb 2008 22:15:20 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Stefan Lambrev References: <4795CC13.7080601@moneybookers.com> <4795FE54.9090606@moneybookers.com> <86lk6i0vzk.fsf@ds4.des.no> <479605E2.6070709@moneybookers.com> <47964356.6030602@moneybookers.com> <479647FB.3070909@FreeBSD.org> <47970EE2.5000400@moneybookers.com> <479754E6.1060101@moneybookers.com> <9bbcef730801230802n5c52832bk60c6afc47be578f4@mail.gmail.com> <47976AD4.3020203@moneybookers.com> In-Reply-To: <47976AD4.3020203@moneybookers.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: gettimeofday() in hping X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 21:15:22 -0000 Stefan Lambrev wrote: > I run from host A : hping --flood -p 22 -S 10.3.3.2 > and systat -ifstat on host B to see the traffic that is generated > (I do not want to run this monitoring on the flooder host as it will > effect his performance) OK, I finally got time to look at this. Firstly, this is quite an inefficient program. It performs 5 syscalls for each packet that it sends: 2391 initial thread CALL sendto(0x3,0x61b050,0x28,0,0x5232c0,0x10) 2391 initial thread GIO fd 3 wrote 40 bytes 0x0000 4500 2800 7491 0000 4006 0000 0a00 0004 0a00 0001 3a96 0016 1865 a781 39d8 12aa 5002 0200 52c9 |E.(.t...@...........:....e..9...P...R.| 0x0026 0000 |..| 2391 initial thread RET sendto 40/0x28 2391 initial thread CALL sigaction(SIGALRM,0x7fffffffe6b0,0x7fffffffe690) 2391 initial thread RET sigaction 0 2391 initial thread CALL setitimer(0,0x7fffffffe6c0,0x7fffffffe6a0) 2391 initial thread RET setitimer 0 2391 initial thread CALL gettimeofday(0x7fffffffe680,0) 2391 initial thread RET gettimeofday 0 2391 initial thread CALL gettimeofday(0x7fffffffe680,0) 2391 initial thread RET gettimeofday 0 Here is a further litany of some of the ways in which this software is terrible: * It does not attempt to increase the socket buffer size (as we have already discussed), but * It also doesn't cope with the possibility that the packet may not be sent because the send buffer is full. * With every packet sent in flood mode it sets a timer for 1 second in the future even though we have told it not to send packets once a second but as fast as possible * We also set the signal handler with each packet sent, instead of setting it once and leaving it. * We call gettimeofday twice for each packet, once to retrieve the second timestamp and once to retrieve the microseconds. This is only for the purpose of computing the RTT. However, we can only keep track of 400 in-flight packets, which means this is also useless in flood mode. * The suspend handler does not work * This does not strike me as quality software :) Fixing all of the above I can send at about 13MB/sec (timecounter is not relevant any more). The CPU is spending about 75% of the time in the kernel, so Kris From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 21:16:15 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D27C116A420; Sun, 3 Feb 2008 21:16:15 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id BE7A413C457; Sun, 3 Feb 2008 21:16:14 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A62F1D.8070507@FreeBSD.org> Date: Sun, 03 Feb 2008 22:16:13 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Kris Kennaway References: <4795CC13.7080601@moneybookers.com> <4795FE54.9090606@moneybookers.com> <86lk6i0vzk.fsf@ds4.des.no> <479605E2.6070709@moneybookers.com> <47964356.6030602@moneybookers.com> <479647FB.3070909@FreeBSD.org> <47970EE2.5000400@moneybookers.com> <479754E6.1060101@moneybookers.com> <9bbcef730801230802n5c52832bk60c6afc47be578f4@mail.gmail.com> <47976AD4.3020203@moneybookers.com> <47A62EE8.2000700@FreeBSD.org> In-Reply-To: <47A62EE8.2000700@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Ivan Voras , Stefan Lambrev Subject: Re: gettimeofday() in hping X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 21:16:15 -0000 Kris Kennaway wrote: > Stefan Lambrev wrote: > >> I run from host A : hping --flood -p 22 -S 10.3.3.2 >> and systat -ifstat on host B to see the traffic that is generated >> (I do not want to run this monitoring on the flooder host as it will >> effect his performance) > > OK, I finally got time to look at this. Firstly, this is quite an > inefficient program. It performs 5 syscalls for each packet that it sends: > > 2391 initial thread CALL sendto(0x3,0x61b050,0x28,0,0x5232c0,0x10) > 2391 initial thread GIO fd 3 wrote 40 bytes > 0x0000 4500 2800 7491 0000 4006 0000 0a00 0004 0a00 0001 3a96 > 0016 1865 a781 39d8 12aa 5002 0200 52c9 > |E.(.t...@...........:....e..9...P...R.| > 0x0026 0000 |..| > > 2391 initial thread RET sendto 40/0x28 > 2391 initial thread CALL sigaction(SIGALRM,0x7fffffffe6b0,0x7fffffffe690) > 2391 initial thread RET sigaction 0 > 2391 initial thread CALL setitimer(0,0x7fffffffe6c0,0x7fffffffe6a0) > 2391 initial thread RET setitimer 0 > 2391 initial thread CALL gettimeofday(0x7fffffffe680,0) > 2391 initial thread RET gettimeofday 0 > 2391 initial thread CALL gettimeofday(0x7fffffffe680,0) > 2391 initial thread RET gettimeofday 0 > > Here is a further litany of some of the ways in which this software is > terrible: > > * It does not attempt to increase the socket buffer size (as we have > already discussed), but > > * It also doesn't cope with the possibility that the packet may not be > sent because the send buffer is full. > > * With every packet sent in flood mode it sets a timer for 1 second in > the future even though we have told it not to send packets once a second > but as fast as possible > > * We also set the signal handler with each packet sent, instead of > setting it once and leaving it. > > * We call gettimeofday twice for each packet, once to retrieve the > second timestamp and once to retrieve the microseconds. This is only > for the purpose of computing the RTT. However, we can only keep track > of 400 in-flight packets, which means this is also useless in flood mode. > > * The suspend handler does not work > > * This does not strike me as quality software :) > > Fixing all of the above I can send at about 13MB/sec (timecounter is not > relevant any more). The CPU is spending about 75% of the time in the > kernel, so that is the next place to look. [hit send too soon] Kris From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 22:01:27 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 340E916A419; Sun, 3 Feb 2008 22:01:27 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id EAAA613C448; Sun, 3 Feb 2008 22:01:25 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A639B5.2020909@FreeBSD.org> Date: Sun, 03 Feb 2008 23:01:25 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Kris Kennaway References: <4795CC13.7080601@moneybookers.com> <4795FE54.9090606@moneybookers.com> <86lk6i0vzk.fsf@ds4.des.no> <479605E2.6070709@moneybookers.com> <47964356.6030602@moneybookers.com> <479647FB.3070909@FreeBSD.org> <47970EE2.5000400@moneybookers.com> <479754E6.1060101@moneybookers.com> <9bbcef730801230802n5c52832bk60c6afc47be578f4@mail.gmail.com> <47976AD4.3020203@moneybookers.com> <47A62EE8.2000700@FreeBSD.org> <47A62F1D.8070507@FreeBSD.org> In-Reply-To: <47A62F1D.8070507@FreeBSD.org> Content-Type: multipart/mixed; boundary="------------050704020506030104040903" Cc: freebsd-hackers@freebsd.org, Ivan Voras , Stefan Lambrev Subject: Re: gettimeofday() in hping X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 22:01:27 -0000 This is a multi-part message in MIME format. --------------050704020506030104040903 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Kris Kennaway wrote: >> Fixing all of the above I can send at about 13MB/sec (timecounter is >> not relevant any more). The CPU is spending about 75% of the time in >> the kernel, so > that is the next place to look. [hit send too soon] Actually 15MB/sec once I disable all kernel debugging. This is identical to Linux 2.6.24 on the same hardware. The patch I use to fix hping brain-damage is attached. Kris --------------050704020506030104040903 Content-Type: text/plain; x-mac-type="0"; x-mac-creator="0"; name="braindeath.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="braindeath.diff" diff -ru work/hping3-20051105/opensockraw.c work~/hping3-20051105/opensockraw.c --- opensockraw.c.orig 2003-09-01 00:22:06.000000000 +0000 +++ opensockraw.c 2008-02-03 19:45:28.000000000 +0000 @@ -17,7 +17,7 @@ int open_sockraw() { - int s; + int s, t, val; s = socket(AF_INET, SOCK_RAW, IPPROTO_RAW); if (s == -1) { @@ -25,5 +25,12 @@ return -1; } + val = 262144; + t = setsockopt(s, SOL_SOCKET, SO_SNDBUF, &val, sizeof(val)); + if (t == -1) { + perror("[open_sockraw] setsockopt()"); + return -1; + } + return s; } diff -ru work/hping3-20051105/sendip.c work~/hping3-20051105/sendip.c --- sendip.c.orig 2004-04-09 23:38:56.000000000 +0000 +++ sendip.c 2008-02-03 19:50:35.000000000 +0000 @@ -110,7 +110,7 @@ result = sendto(sockraw, packet, packetsize, 0, (struct sockaddr*)&remote, sizeof(remote)); - if (result == -1 && errno != EINTR && !opt_rand_dest && !opt_rand_source) { + if (result == -1 && errno != ENOBUFS && errno != EINTR && !opt_rand_dest && !opt_rand_source) { perror("[send_ip] sendto"); if (close(sockraw) == -1) perror("[ipsender] close(sockraw)"); diff -ru work/hping3-20051105/sendtcp.c work~/hping3-20051105/sendtcp.c --- sendtcp.c.orig 2003-09-01 00:22:06.000000000 +0000 +++ sendtcp.c 2008-02-03 20:30:51.000000000 +0000 @@ -85,8 +85,10 @@ packet_size); #endif +#if 0 /* adds this pkt in delaytable */ delaytable_add(sequence, src_port, time(NULL), get_usec(), S_SENT); +#endif /* send packet */ send_ip_handler(packet+PSEUDOHDR_SIZE, packet_size); --- send.c.orig 2003-08-31 17:23:53.000000000 +0000 +++ send.c 2008-02-03 21:58:59.000000000 +0000 @@ -63,6 +63,8 @@ } } +static int sigalarm_handler = 0; + /* The signal handler for SIGALRM will send the packets */ void send_packet (int signal_id) { @@ -79,12 +81,15 @@ else send_tcp(); sent_pkt++; - Signal(SIGALRM, send_packet); + if (!opt_flood && !sigalarm_handler) { + Signal(SIGALRM, send_packet); + sigalarm_handler = 1; + } if (count != -1 && count == sent_pkt) { /* count reached? */ Signal(SIGALRM, print_statistics); alarm(COUNTREACHED_TIMEOUT); - } else if (!opt_listenmode) { + } else if (!opt_listenmode && !opt_flood) { if (opt_waitinusec == FALSE) alarm(sending_wait); else --------------050704020506030104040903-- From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 22:10:24 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D35316A468 for ; Sun, 3 Feb 2008 22:10:24 +0000 (UTC) (envelope-from sandiegobiker@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.191]) by mx1.freebsd.org (Postfix) with ESMTP id E3F2613C4F9 for ; Sun, 3 Feb 2008 22:10:23 +0000 (UTC) (envelope-from sandiegobiker@gmail.com) Received: by rv-out-0910.google.com with SMTP id g13so1473025rvb.43 for ; Sun, 03 Feb 2008 14:10:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; bh=qOXawWEj+w5q8vLjfteCM1W4qvhUzeZAksX7JVW8BL0=; b=B2QlJxflvp2eVso8yjNtMYGfSK5f4srgWlc89fxnC/8WACU1IJeerdC75i24/LRh26MhLK0c60ajCPoZZ0OF6LDm9Z1JBJ7Hn2H9KdMaKprC/F7msA4sq4C2B+6nUJeCSuH9+xPdJfIG4WYc5GG8YYRTPxuhmCs20Ps/sE4mB10= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=GxbzeND5kzyD2fdUhqRGm5mlf057Sd30swxFeeZMdYYMPjmGCqNS0KOoxBs2lkEnSkct0SXEUdOrWl6WGUFmL5s03VQrmji5HNXet6+Xl8NjEy4IFgI1V+WM+EozAeTm/XY30MuXiq2W+iUHz3+pHfGqrbDAlnMAnmo42SAK+HI= Received: by 10.141.20.7 with SMTP id x7mr4202302rvi.61.1202074921104; Sun, 03 Feb 2008 13:42:01 -0800 (PST) Received: by 10.141.35.8 with HTTP; Sun, 3 Feb 2008 13:42:01 -0800 (PST) Message-ID: <27cb3ada0802031342w62c96ba8i933f712c74de4988@mail.gmail.com> Date: Sun, 3 Feb 2008 13:42:01 -0800 From: "Len Gross" To: "freebsd-hackers@freebsd.org" MIME-Version: 1.0 X-Mailman-Approved-At: Sun, 03 Feb 2008 22:26:56 +0000 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Netgraph, usleep, and moving to Kernel X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 22:10:24 -0000 I've had some good success with implementing a custom MAC protocol using Netgraph. The current implementation runs in userland, connects to the Kernel iface and kernel ethernet nodes. It uses a polling loop with usleep, All very cool. This is just background as the question really has to do with usleeop in userland or kernel mode. In testing with nothing else running on the box. it seems to reveal that if I ask for 100 usec of "sleep time," I get about 1000 usec (1 msec). I have set hz to 10,000 in kern.clockrate and Ticks is at 100. If I set hz to 20,000 there is no improvement and Ticks seems to "reset itsefl" to 50. Box is an old Pentium with about 500 Mhz clock. I also get this behaviour on "test programs" that just call usleep. 1) Is there a way to get finer timing with usleep or an equivalent call? 2) When I move the userland into a "true" Netgraph Kernel node will I see different behaviour? Thanks in advance. -- Len From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 23:20:45 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C55416A46C for ; Sun, 3 Feb 2008 23:20:45 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outN.internet-mail-service.net (outN.internet-mail-service.net [216.240.47.237]) by mx1.freebsd.org (Postfix) with ESMTP id C9CFE13C478 for ; Sun, 3 Feb 2008 23:20:44 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Sun, 03 Feb 2008 15:20:43 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id 0F79D1270BD; Sun, 3 Feb 2008 15:20:43 -0800 (PST) Message-ID: <47A64C4A.5020002@elischer.org> Date: Sun, 03 Feb 2008 15:20:42 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: =?UTF-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> <20080202224923.T66602@fledge.watson.org> <47A55209.9040106@elischer.org> <86ejbumejj.fsf@ds4.des.no> In-Reply-To: <86ejbumejj.fsf@ds4.des.no> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-hackers@freebsd.org, Kris Kennaway , Robert Watson , Alexander Motin , freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 23:20:45 -0000 Dag-Erling Smørgrav wrote: > Julian Elischer writes: >> Robert Watson writes: >>> be a good time to try to revalidate that. Basically, the goal would >>> be to make the pcpu cache FIFO as much as possible as that maximizes >>> the chances that the newly allocated object already has lines in the >>> cache. It's a fairly trivial tweak to the UMA allocation code. >> you mean FILO or LIFO right? > > Uh, no. You want to reuse the last-freed object, as it is most likely > to still be in cache. > > DES exactly.. FILO or LIFO (last in First out.) From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 3 23:40:15 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6042316A41B; Sun, 3 Feb 2008 23:40:15 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id 2DCF013C45A; Sun, 3 Feb 2008 23:40:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona 1.7.0 Received: from [212.86.226.226] (account mav@alkar.net HELO [192.168.3.2]) by cmail.optima.ua (CommuniGate Pro SMTP 5.1.14) with ESMTPA id 72717087; Mon, 04 Feb 2008 01:40:13 +0200 Message-ID: <47A650DA.8020908@FreeBSD.org> Date: Mon, 04 Feb 2008 01:40:10 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Kris Kennaway , Robert Watson References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> In-Reply-To: <47A4F1AF.9090306@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Memory allocation performance X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 23:40:15 -0000 Kris Kennaway wrote: > You can look at the raw output from pmcstat, which is a collection of > instruction pointers that you can feed to e.g. addr2line to find out > exactly where in those functions the events are occurring. This will > often help to track down the precise causes. Thanks to the hint, it was interesting hunting, but it shown nothing. It hits into very simple lines like: bucket = cache->uc_freebucket; cache->uc_allocs++; if (zone->uz_ctor != NULL) { cache->uc_frees++; and so on. There is no loops, there is no inlines or macroses. Nothing! And the only hint about it is a huge number of "p4-resource-stall"s in those lines. I have no idea what exactly does it means, why does it happens mostly here and how to fight it. I would probably agreed that it might be some profiler fluctuation, but performance benefits I have got from self-made uma calls caching look very real. :( Robert Watson wrote: > There was, FYI, a report a few years ago that there was a measurable > improvement from allocating off the free bucket rather than maintaining > separate alloc and free buckets. It sounded good at the time but I was > never able to reproduce the benefits in my test environment. Now might > be a good time to try to revalidate that. Basically, the goal would be > to make the pcpu cache FIFO as much as possible as that maximizes the > chances that the newly allocated object already has lines in the cache. > It's a fairly trivial tweak to the UMA allocation code. I have tried this, but have not found a difference. May be it gives some benefits, but not in this situation. In this situation profiling shows delays in allocator itself, so as soon as allocator does not touches data objects itself it probably more speaks about management structure's memory caching then about objects caching. I have got one more crazy idea that memory containing zones may have some special hardware or configuration features, like "noncaching" or something alike. That could explain slowdown in accessing it. But as I can't prove it, it just one more crazy theory. :( -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 00:13:48 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 096FD16A419; Mon, 4 Feb 2008 00:13:48 +0000 (UTC) (envelope-from sam@errno.com) Received: from ebb.errno.com (ebb.errno.com [69.12.149.25]) by mx1.freebsd.org (Postfix) with ESMTP id B7F2113C447; Mon, 4 Feb 2008 00:13:47 +0000 (UTC) (envelope-from sam@errno.com) Received: from Macintosh-2.local ([10.0.0.196]) (authenticated bits=0) by ebb.errno.com (8.13.6/8.12.6) with ESMTP id m140Dkr4020918 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 3 Feb 2008 16:13:47 -0800 (PST) (envelope-from sam@errno.com) Message-ID: <47A658BB.1080104@errno.com> Date: Sun, 03 Feb 2008 16:13:47 -0800 From: Sam Leffler Organization: Errno Consulting User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Kris Kennaway References: <4795CC13.7080601@moneybookers.com> <4795FE54.9090606@moneybookers.com> <86lk6i0vzk.fsf@ds4.des.no> <479605E2.6070709@moneybookers.com> <47964356.6030602@moneybookers.com> <479647FB.3070909@FreeBSD.org> <47970EE2.5000400@moneybookers.com> <479754E6.1060101@moneybookers.com> <9bbcef730801230802n5c52832bk60c6afc47be578f4@mail.gmail.com> <47976AD4.3020203@moneybookers.com> <47A62EE8.2000700@FreeBSD.org> In-Reply-To: <47A62EE8.2000700@FreeBSD.org> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DCC--Metrics: ebb.errno.com; whitelist Cc: freebsd-hackers@freebsd.org, Ivan Voras , Stefan Lambrev Subject: Re: gettimeofday() in hping X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 00:13:48 -0000 Kris Kennaway wrote: > Stefan Lambrev wrote: > >> I run from host A : hping --flood -p 22 -S 10.3.3.2 >> and systat -ifstat on host B to see the traffic that is generated >> (I do not want to run this monitoring on the flooder host as it will >> effect his performance) > > OK, I finally got time to look at this. Firstly, this is quite an > inefficient program. It performs 5 syscalls for each packet that it sends: > > 2391 initial thread CALL sendto(0x3,0x61b050,0x28,0,0x5232c0,0x10) > 2391 initial thread GIO fd 3 wrote 40 bytes > 0x0000 4500 2800 7491 0000 4006 0000 0a00 0004 0a00 0001 3a96 > 0016 1865 a781 39d8 12aa 5002 0200 52c9 > |E.(.t...@...........:....e..9...P...R.| > 0x0026 0000 |..| > > 2391 initial thread RET sendto 40/0x28 > 2391 initial thread CALL sigaction(SIGALRM,0x7fffffffe6b0,0x7fffffffe690) > 2391 initial thread RET sigaction 0 > 2391 initial thread CALL setitimer(0,0x7fffffffe6c0,0x7fffffffe6a0) > 2391 initial thread RET setitimer 0 > 2391 initial thread CALL gettimeofday(0x7fffffffe680,0) > 2391 initial thread RET gettimeofday 0 > 2391 initial thread CALL gettimeofday(0x7fffffffe680,0) > 2391 initial thread RET gettimeofday 0 > > Here is a further litany of some of the ways in which this software is > terrible: > > * It does not attempt to increase the socket buffer size (as we have > already discussed), but > > * It also doesn't cope with the possibility that the packet may not be > sent because the send buffer is full. > > * With every packet sent in flood mode it sets a timer for 1 second in > the future even though we have told it not to send packets once a second > but as fast as possible > > * We also set the signal handler with each packet sent, instead of > setting it once and leaving it. > > * We call gettimeofday twice for each packet, once to retrieve the > second timestamp and once to retrieve the microseconds. This is only > for the purpose of computing the RTT. However, we can only keep track > of 400 in-flight packets, which means this is also useless in flood mode. > > * The suspend handler does not work > > * This does not strike me as quality software :) > > Fixing all of the above I can send at about 13MB/sec (timecounter is not > relevant any more). The CPU is spending about 75% of the time in the > kernel, so The "doesn't cope with the possibility that ... the send buffer is full" issue is classic linux-specific mis-behaviour. On linux the process will block when the default qdisc finds the device q is stopped (due to being full). I remember cursing iperf for this. Sam From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 02:40:44 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B03EF16A417; Mon, 4 Feb 2008 02:40:44 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail18.syd.optusnet.com.au (mail18.syd.optusnet.com.au [211.29.132.199]) by mx1.freebsd.org (Postfix) with ESMTP id 20C7A13C455; Mon, 4 Feb 2008 02:40:43 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c211-30-219-213.carlnfd3.nsw.optusnet.com.au (c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213]) by mail18.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m142eVi6014054 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 4 Feb 2008 13:40:33 +1100 Date: Mon, 4 Feb 2008 13:40:31 +1100 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Alexander Motin In-Reply-To: <47A650DA.8020908@FreeBSD.org> Message-ID: <20080204133437.T63947@delplex.bde.org> References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> <47A650DA.8020908@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Mailman-Approved-At: Mon, 04 Feb 2008 04:51:22 +0000 Cc: freebsd-hackers@freebsd.org, Robert Watson , freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Memory allocation performance X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 02:40:44 -0000 On Mon, 4 Feb 2008, Alexander Motin wrote: > Kris Kennaway wrote: >> You can look at the raw output from pmcstat, which is a collection of >> instruction pointers that you can feed to e.g. addr2line to find out >> exactly where in those functions the events are occurring. This will often >> help to track down the precise causes. > > Thanks to the hint, it was interesting hunting, but it shown nothing. It hits > into very simple lines like: > bucket = cache->uc_freebucket; > cache->uc_allocs++; > if (zone->uz_ctor != NULL) { > cache->uc_frees++; > and so on. > There is no loops, there is no inlines or macroses. Nothing! And the only > hint about it is a huge number of "p4-resource-stall"s in those lines. I have > no idea what exactly does it means, why does it happens mostly here and how > to fight it. Try profiling it one another type of CPU, to get different performance counters but hopefully not very different stalls. If the other CPU doesn't stall at all, put another black mark against P4 and delete your copies of it :-). Bruce From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 08:53:10 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B89A16A417 for ; Mon, 4 Feb 2008 08:53:10 +0000 (UTC) (envelope-from samflanker@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.154]) by mx1.freebsd.org (Postfix) with ESMTP id C3F5313C43E for ; Mon, 4 Feb 2008 08:53:09 +0000 (UTC) (envelope-from samflanker@gmail.com) Received: by fg-out-1718.google.com with SMTP id 16so2063608fgg.35 for ; Mon, 04 Feb 2008 00:53:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:user-agent:mime-version:to:subject:content-type:content-transfer-encoding; bh=q0V8ERrOClY5BshST1lt3I0ApjGoSZCa5SblvUvhCZM=; b=koLDWOod89neQvawWzql9jOaXE8uX+34SNvIAgJPS5OW6Lkm2ngtxnJS2BU5DxzKoCSD+k+vdVuucKiU5mb7Smy5Jt7pjhi/LS14JdsKEXoFURweUnpiJv3Afgjp7IDhHLpFg5H9Ri0ZsE4lS1LPn/vwjbGtHW+DuLHCpY8c4fk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:content-type:content-transfer-encoding; b=Rt3a43zKVlYe0292apB6AViHdrRky6fTsdJxdml0E8sa9ai9GWFQz0hdPvRBzclDrVl89MZR1p0SF3QjuCwqbcJtXxD5VTRsL9CgFvb5ChaEeQD2QQdtY7cAhWbDijgqQDhjX4ui/nD7q1X8tP1PiWaQZMjMOAAAxZ1EgAuweZM= Received: by 10.86.76.16 with SMTP id y16mr6407158fga.23.1202115188484; Mon, 04 Feb 2008 00:53:08 -0800 (PST) Received: from ?192.168.12.166? ( [213.152.137.38]) by mx.google.com with ESMTPS id e20sm7639444fga.1.2008.02.04.00.53.06 (version=SSLv3 cipher=RC4-MD5); Mon, 04 Feb 2008 00:53:07 -0800 (PST) Message-ID: <47A6D272.2000406@gmail.com> Date: Mon, 04 Feb 2008 11:53:06 +0300 From: sam User-Agent: Thunderbird 2.0.0.4 (Windows/20070604) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: SCHED_ULE trouble after ugrade 6.2-RELEASE -> 6.3-RELEASE X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 08:53:10 -0000 hi all my trouble description: --------------part of dmesg----------- CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2392.26-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf27 Stepping = 7 Features=0xbfebfbff Features2=0x400 Logical CPUs per core: 2 --------------------------------------- 1. using FreeBSD 6.2-RELEASE i386 SMP with SCHED_ULE description of boot process: system is booting normal 2. using FreeBSD 6.3-RELEASE i386 SMP with SCHED_ULE description of boot process: system is full stop on boot process message: ------------------------------------------------- Timecounters tick every 1.000 msec Waiting 5 seconds for SCSI devices to settle ------------------------------------------------- 3. using FreeBSD 6.3-STABLE i386 SMP with SCHED_ULE description of boot process: system is full stop on boot process message: ------------------------------------------------- Timecounters tick every 1.000 msec Waiting 5 seconds for SCSI devices to settle ------------------------------------------------- 4. using FreeBSD 6.3-RELEASE i386 SMP with SCHED_4BSD description of boot process: system is booting normal please any solution /Vladimir Ermakov From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 08:56:22 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 81D1D16A419 for ; Mon, 4 Feb 2008 08:56:22 +0000 (UTC) (envelope-from samflanker@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.155]) by mx1.freebsd.org (Postfix) with ESMTP id 0574C13C4EF for ; Mon, 4 Feb 2008 08:56:21 +0000 (UTC) (envelope-from samflanker@gmail.com) Received: by fg-out-1718.google.com with SMTP id 16so2064369fgg.35 for ; Mon, 04 Feb 2008 00:56:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding; bh=e3cnLE+lRLFrZPyFczPn+ZE+wYquhDCOWfXxBZQoMtw=; b=kVBryjCmONUEq2JDzAMw8B6PQSZ56VA+tpi0P+tPI0qxIYaqU+IGoeqsruezpJ8Mxz2VtbavDdRu7gQDyd3KSjXiHCttefSgcRFbv/0NpYUfsJDYYNZPpIuf3PTqvZilMoABB41SnPs1AzN2GjUa2YhhDDOovyqkNiag0DSpl4s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding; b=HwPF1dg58bV8ZAJQBp6trXuo7JEBXkEQdBoW0ZKYAGUXASLlbLtEqyarJKBaAvJyubhbroa6MSgIVoKxplblfnuzAlQYt1WiEz6GYlWO7oemnewTb16n3esgRnQ3s+BUoDb4UFqBupW1l0YClgMOppeLb3iOKBE9Du9gVmYoDTU= Received: by 10.86.62.3 with SMTP id k3mr6401085fga.24.1202115380678; Mon, 04 Feb 2008 00:56:20 -0800 (PST) Received: from ?192.168.12.166? ( [213.152.137.38]) by mx.google.com with ESMTPS id l19sm9689261fgb.0.2008.02.04.00.56.18 (version=SSLv3 cipher=RC4-MD5); Mon, 04 Feb 2008 00:56:19 -0800 (PST) Message-ID: <47A6D335.2000902@gmail.com> Date: Mon, 04 Feb 2008 11:56:21 +0300 From: sam User-Agent: Thunderbird 2.0.0.4 (Windows/20070604) MIME-Version: 1.0 To: Eygene Ryabinkin References: <47A2E647.20503@gmail.com> <20080201105111.GC15719@server.vk2pj.dyndns.org> <47A2FD4F.4090609@gmail.com> <1201867184.2975.30.camel@localhost> <47A318F3.6040206@gmail.com> <8BK9hGY47q693Ae/THmLm0+ESYM@lAQxaF3PeR+wzAYKIeRVhQKEwXM> In-Reply-To: <8BK9hGY47q693Ae/THmLm0+ESYM@lAQxaF3PeR+wzAYKIeRVhQKEwXM> Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Cc: Tom Evans , freebsd-hackers@freebsd.org Subject: Re: double start of scripts in /usr/local/etc/rc.d X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 08:56:22 -0000 Eygene Ryabinkin wrote: >>>> >>>> >>> And is your /usr/X11R6 symlinked to /usr/local? >>> >>> >> have this symlink >> > > Then remove /usr/X11R6/etc from the local_startup variable > (in /etc/rc.conf and/or in /etc/defaults/rc.conf) and enjoy single > startup of scripts ;)) > > Seems like you had updated your system to Xorg 7.2, but forgot > to run /usr/ports/Tools/scripts/mergebase.sh or mergebase.sh > failed to remove /usr/X11R6/etc from local_startup. > thx iam deleted this symlink /Vladimir Ermakov From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 09:00:31 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A8A316A41B for ; Mon, 4 Feb 2008 09:00:31 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.delphij.net (delphij-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:2c9::2]) by mx1.freebsd.org (Postfix) with ESMTP id C3A0A13C4EC for ; Mon, 4 Feb 2008 09:00:30 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.geekcn.org (unknown [202.108.54.204]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tarsier.delphij.net (Postfix) with ESMTP id B23EC28448 for ; Mon, 4 Feb 2008 17:00:28 +0800 (CST) Received: from localhost (unknown [202.108.54.204]) by tarsier.geekcn.org (Postfix) with ESMTP id 760EEEC289E; Mon, 4 Feb 2008 17:00:28 +0800 (CST) X-Virus-Scanned: amavisd-new at geekcn.org Received: from tarsier.geekcn.org ([202.108.54.204]) by localhost (mail.geekcn.org [202.108.54.204]) (amavisd-new, port 10024) with ESMTP id Bfx6sLzfp2vd; Mon, 4 Feb 2008 17:00:22 +0800 (CST) Received: from charlie.delphij.net (c-67-161-39-180.hsd1.ca.comcast.net [67.161.39.180]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTP id BE868EC285A; Mon, 4 Feb 2008 17:00:21 +0800 (CST) DomainKey-Signature: a=rsa-sha1; s=default; d=delphij.net; c=nofws; q=dns; h=message-id:date:from:reply-to:organization:user-agent: mime-version:to:cc:subject:references:in-reply-to: x-enigmail-version:openpgp:content-type:content-transfer-encoding; b=XDxonHsbq5utc1YVoCgneJzwhfckunG7lWYbopFBiCXIC/07/OuzsL+HA/uLzOump wgbmli+8M5ijPrapVVFQQ== Message-ID: <47A6D420.5050700@delphij.net> Date: Mon, 04 Feb 2008 01:00:16 -0800 From: Xin LI Organization: The FreeBSD Project User-Agent: Thunderbird 2.0.0.9 (X11/20080122) MIME-Version: 1.0 To: sam References: <47A6D272.2000406@gmail.com> In-Reply-To: <47A6D272.2000406@gmail.com> X-Enigmail-Version: 0.95.5 OpenPGP: id=18EDEBA0; url=http://www.delphij.net/delphij.asc Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: SCHED_ULE trouble after ugrade 6.2-RELEASE -> 6.3-RELEASE X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: d@delphij.net List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 09:00:31 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 sam wrote: > hi all > my trouble description: Do NOT use ULE on 6.x. ULE has been revamped heavily on 7.0 and the version on 6.x is old, and is known to contain some bugs. Cheers, - -- Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHptQgi+vbBBjt66ARAvJvAJ4stGN+IC4kAyJLyJGDa7mfkNA58gCggj1V ErCk1Yiw8lU6UM1BACTbA2M= =pkR+ -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 09:40:25 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B38216A419 for ; Mon, 4 Feb 2008 09:40:25 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 1C0C413C461 for ; Mon, 4 Feb 2008 09:40:24 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender:X-Spam-Status:Subject; b=qfeCtEq/+SGjBOfaSdFaG+wiIUW0+1CYr00X6EyzaJYTKYKqhn8YLr/PDK7r0ICx6F8GG0egNz/NuaGvPdUmAa9perPa41CsuB9llbKvRsFRLe8WohiAQdkVOUgSPHvpJCK/gZyYQrGCpB7pHVmUr7+8P2sKdyWtcVGc96mJ7CI=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1JLxV4-0009Ly-RH; Mon, 04 Feb 2008 12:20:26 +0300 Date: Mon, 4 Feb 2008 12:20:25 +0300 From: Eygene Ryabinkin To: sam Message-ID: References: <47A2E647.20503@gmail.com> <20080201105111.GC15719@server.vk2pj.dyndns.org> <47A2FD4F.4090609@gmail.com> <1201867184.2975.30.camel@localhost> <47A318F3.6040206@gmail.com> <8BK9hGY47q693Ae/THmLm0+ESYM@lAQxaF3PeR+wzAYKIeRVhQKEwXM> <47A6D335.2000902@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <47A6D335.2000902@gmail.com> Sender: rea-fbsd@codelabs.ru X-Spam-Status: No, score=-1.9 required=4.0 tests=ALL_TRUSTED,AWL,BAYES_50 Cc: Tom Evans , freebsd-hackers@freebsd.org Subject: Re: double start of scripts in /usr/local/etc/rc.d X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 09:40:25 -0000 Mon, Feb 04, 2008 at 11:56:21AM +0300, sam wrote: >> Then remove /usr/X11R6/etc from the local_startup variable >> (in /etc/rc.conf and/or in /etc/defaults/rc.conf) and enjoy single >> startup of scripts ;)) > > thx > iam deleted this symlink No, no, no: you should not remove the symlink itself. By the immortal words of Kris Kennaway in /usr/ports/UPDATING: ----- When the merge operation completes successfully, the /usr/X11R6 directory hierarchy will be removed and replaced by a symlink to /usr/local. This symlink is necessary because some binary ports (and some remaining source ports) have hard-coded references to /usr/X11R6. ----- So you can hit the problem of hard-coded references. Just remove the string /usr/X11R6/etc from startup configuration as was described above. -- Eygene From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 10:34:21 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 53A9716A421; Mon, 4 Feb 2008 10:34:21 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id F2A8813C455; Mon, 4 Feb 2008 10:34:20 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 49DAE1B10F2F; Mon, 4 Feb 2008 11:34:19 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 6E9341B10F2C; Mon, 4 Feb 2008 11:34:16 +0100 (CET) Message-ID: <47A6EA28.6000902@moneybookers.com> Date: Mon, 04 Feb 2008 12:34:16 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Kris Kennaway References: <4795CC13.7080601@moneybookers.com> <4795FE54.9090606@moneybookers.com> <86lk6i0vzk.fsf@ds4.des.no> <479605E2.6070709@moneybookers.com> <47964356.6030602@moneybookers.com> <479647FB.3070909@FreeBSD.org> <47970EE2.5000400@moneybookers.com> <479754E6.1060101@moneybookers.com> <9bbcef730801230802n5c52832bk60c6afc47be578f4@mail.gmail.com> <47976AD4.3020203@moneybookers.com> <47A62EE8.2000700@FreeBSD.org> <47A62F1D.8070507@FreeBSD.org> <47A639B5.2020909@FreeBSD.org> In-Reply-To: <47A639B5.2020909@FreeBSD.org> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: gettimeofday() in hping X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 10:34:21 -0000 Greetings, Kris Kennaway wrote: > Kris Kennaway wrote: > >>> Fixing all of the above I can send at about 13MB/sec (timecounter is >>> not relevant any more). The CPU is spending about 75% of the time >>> in the kernel, so >> that is the next place to look. [hit send too soon] > > Actually 15MB/sec once I disable all kernel debugging. This is > identical to Linux 2.6.24 on the same hardware. The patch I use to > fix hping brain-damage is attached. > > Kris Indeed this patch highly improve results (x2) under FreeBSD. On my hardware the max speeds under FreeBSD i still little slower compared to Linux (something like 19 vs 19.5 MB/s) but the speed is quite more stable under FreeBSD (under linux the average speed seems slower) I didn't tested the patch under linux but will do soon. Thanks for the help :) BTW 262144 is little high as this match the default value in FreeBSD, I tested with twice smaller buffer and do not see performance lost. Kris if you do not mind I'll write to hping developers to adopt this patch, and if no response from them I can try to reach the port maintainer, so we have this patched in ports? -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 10:35:50 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34EF616A419; Mon, 4 Feb 2008 10:35:50 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id DB39913C45D; Mon, 4 Feb 2008 10:35:49 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id D88242082; Mon, 4 Feb 2008 11:35:41 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.2/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 5EBB7207F; Mon, 4 Feb 2008 11:35:41 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 3A628844A1; Mon, 4 Feb 2008 11:35:41 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Julian Elischer References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> <20080202224923.T66602@fledge.watson.org> <47A55209.9040106@elischer.org> <86ejbumejj.fsf@ds4.des.no> <47A64C4A.5020002@elischer.org> Date: Mon, 04 Feb 2008 11:35:41 +0100 In-Reply-To: <47A64C4A.5020002@elischer.org> (Julian Elischer's message of "Sun\, 03 Feb 2008 15\:20\:42 -0800") Message-ID: <86prvdggaq.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Kris Kennaway , Robert Watson , Alexander Motin , freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 10:35:50 -0000 Julian Elischer writes: > Dag-Erling Sm=C3=B8rgrav writes: > > Julian Elischer writes: > > > you mean FILO or LIFO right? > > Uh, no. You want to reuse the last-freed object, as it is most > > likely to still be in cache. > exactly.. FILO or LIFO (last in First out.) Clearly, I must have written the above in an acute state of caffeine deprivation. You are perfectly correct. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 10:55:01 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D60A716A41A; Mon, 4 Feb 2008 10:55:01 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 70A5013C4F4; Mon, 4 Feb 2008 10:54:57 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A6EEFE.8040809@FreeBSD.org> Date: Mon, 04 Feb 2008 11:54:54 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Stefan Lambrev References: <4795CC13.7080601@moneybookers.com> <4795FE54.9090606@moneybookers.com> <86lk6i0vzk.fsf@ds4.des.no> <479605E2.6070709@moneybookers.com> <47964356.6030602@moneybookers.com> <479647FB.3070909@FreeBSD.org> <47970EE2.5000400@moneybookers.com> <479754E6.1060101@moneybookers.com> <9bbcef730801230802n5c52832bk60c6afc47be578f4@mail.gmail.com> <47976AD4.3020203@moneybookers.com> <47A62EE8.2000700@FreeBSD.org> <47A62F1D.8070507@FreeBSD.org> <47A639B5.2020909@FreeBSD.org> <47A6EA28.6000902@moneybookers.com> In-Reply-To: <47A6EA28.6000902@moneybookers.com> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: gettimeofday() in hping X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 10:55:01 -0000 Stefan Lambrev wrote: > Greetings, > > Kris Kennaway wrote: >> Kris Kennaway wrote: >> >>>> Fixing all of the above I can send at about 13MB/sec (timecounter is >>>> not relevant any more). The CPU is spending about 75% of the time >>>> in the kernel, so >>> that is the next place to look. [hit send too soon] >> >> Actually 15MB/sec once I disable all kernel debugging. This is >> identical to Linux 2.6.24 on the same hardware. The patch I use to >> fix hping brain-damage is attached. >> >> Kris > Indeed this patch highly improve results (x2) under FreeBSD. On my > hardware the max speeds under FreeBSD i still little slower compared to > Linux (something like 19 vs 19.5 MB/s) > but the speed is quite more stable under FreeBSD (under linux the > average speed seems slower) > > I didn't tested the patch under linux but will do soon. > Thanks for the help :) > > BTW 262144 is little high as this match the default value in FreeBSD, I > tested with twice smaller buffer and do not see performance lost. > Kris if you do not mind I'll write to hping developers to adopt this > patch, and if no response from them I can try to reach the port > maintainer, so we have this patched in ports? > My patch is a bit crude, because I didnt have the courage to digest the entire structure of the code, so it may be slightly wrong with respect to other operating modes (the #if 0 part is wrong, but that code needs to be rewritten). Some version of it should be included in the software, but perhaps after renaming the patch file ;-) Kris From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 10:58:11 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1128016A417 for ; Mon, 4 Feb 2008 10:58:11 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.191]) by mx1.freebsd.org (Postfix) with ESMTP id 512C613C455 for ; Mon, 4 Feb 2008 10:58:10 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: by rv-out-0910.google.com with SMTP id g13so1611276rvb.43 for ; Mon, 04 Feb 2008 02:58:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=McNgwRxl8gxIMwjY62j63n59NqNgnyAFb0mbvJO1/oY=; b=bdGdrcId1raPKh+nFSGcHcrZ6fL5RPKUvVRsp0kPAOhNlRiYMdtKOx9QPgDC6rgX7JJ8jJyf1SLb+tSJ3GzV3j0l+1D+5WYfbOWIgk8f+3L/T064JODe7PpmxkjzABEOngQeVDn0PVeSNDrufCQb4yzKPStuBcy0N2UXMWN7Vdo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=Yej5Kvre4EaA+iV6gj0hUvyLe7baKb/n8iSYRFnvPZTtpCDCDZfiBTiec/h5zJr80xLlhNVUBWvRQ7y9RAWtGhPwdD8EJOHRO2lt1LqXRZ9kJ+hFA3CESZL4HfcNzHDAbQEyXrqtCsgm+fyRcInLKurtYtOl5g2/F0J47yedykI= Received: by 10.141.79.12 with SMTP id g12mr803859rvl.29.1202122689504; Mon, 04 Feb 2008 02:58:09 -0800 (PST) Received: by 10.140.252.21 with HTTP; Mon, 4 Feb 2008 02:58:09 -0800 (PST) Message-ID: <9bbcef730802040258v3fb42cd7hb70432c0da56aed3@mail.gmail.com> Date: Mon, 4 Feb 2008 11:58:09 +0100 From: "Ivan Voras" Sender: ivoras@gmail.com To: "Stefan Lambrev" In-Reply-To: <47A6EA28.6000902@moneybookers.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4795CC13.7080601@moneybookers.com> <47970EE2.5000400@moneybookers.com> <479754E6.1060101@moneybookers.com> <9bbcef730801230802n5c52832bk60c6afc47be578f4@mail.gmail.com> <47976AD4.3020203@moneybookers.com> <47A62EE8.2000700@FreeBSD.org> <47A62F1D.8070507@FreeBSD.org> <47A639B5.2020909@FreeBSD.org> <47A6EA28.6000902@moneybookers.com> X-Google-Sender-Auth: 869e75cca96b4df8 Cc: freebsd-hackers@freebsd.org Subject: Re: gettimeofday() in hping X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 10:58:11 -0000 On 04/02/2008, Stefan Lambrev wrote: > Kris if you do not mind I'll write to hping developers to adopt this > patch, and if no response from them I can try to reach the port > maintainer, so we have this patched in ports? As we're in a "the whole world is a Linux" situation, would it be helpful to open a page on our wiki to enumerate differences like this, that would help people write and port applications to FreeBSD? I know the sort of advice given by Kris should be obvious to professional developers, but history shows that it is not and that people rather read blogs than manuals. From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 12:27:22 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6222E16A419; Mon, 4 Feb 2008 12:27:22 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 106CD13C4D1; Mon, 4 Feb 2008 12:27:21 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 7ACEA1B10F3B; Mon, 4 Feb 2008 13:27:20 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id DE2731B10F2C; Mon, 4 Feb 2008 13:27:17 +0100 (CET) Message-ID: <47A704A5.1050001@moneybookers.com> Date: Mon, 04 Feb 2008 14:27:17 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Ivan Voras References: <4795CC13.7080601@moneybookers.com> <47970EE2.5000400@moneybookers.com> <479754E6.1060101@moneybookers.com> <9bbcef730801230802n5c52832bk60c6afc47be578f4@mail.gmail.com> <47976AD4.3020203@moneybookers.com> <47A62EE8.2000700@FreeBSD.org> <47A62F1D.8070507@FreeBSD.org> <47A639B5.2020909@FreeBSD.org> <47A6EA28.6000902@moneybookers.com> <9bbcef730802040258v3fb42cd7hb70432c0da56aed3@mail.gmail.com> In-Reply-To: <9bbcef730802040258v3fb42cd7hb70432c0da56aed3@mail.gmail.com> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: gettimeofday() in hping X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 12:27:22 -0000 Greetings, Ivan Voras wrote: > On 04/02/2008, Stefan Lambrev wrote: > > >> Kris if you do not mind I'll write to hping developers to adopt this >> patch, and if no response from them I can try to reach the port >> maintainer, so we have this patched in ports? >> > > As we're in a "the whole world is a Linux" situation, would it be > helpful to open a page on our wiki to enumerate differences like this, > that would help people write and port applications to FreeBSD? I know > the sort of advice given by Kris should be obvious to professional > developers, but history shows that it is not and that people rather > read blogs than manuals. > That's the reason I started to dislike Linux last few years - because it's "defacto" standart in open source/unix like OS world, and their dev team act very much like MS dev team :( In the wiki there is a page about how to avoid "linuxism". I think we have to mention this there. It was the first place where I looked how to speed up hping. I think the info about gettimeofday (and replacements) and the ENOBUFS situation have to be explained little more there. I do not expect developers, that write programs for linux to read the FreeBSD wiki, but it will help people like me - not very experienced in coding, to produce patches/ports without bothering too much advanced developers :) It will be good if there are some examples too. -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 13:13:38 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4308816A41A for ; Mon, 4 Feb 2008 13:13:38 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 2D8F213C4E8 for ; Mon, 4 Feb 2008 13:13:37 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 00C821CC031; Mon, 4 Feb 2008 04:58:32 -0800 (PST) Date: Mon, 4 Feb 2008 04:58:31 -0800 From: Jeremy Chadwick To: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= Message-ID: <20080204125831.GA4052@eos.sc1.parodius.com> References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> <20080203151550.GA67020@owl.midgard.homeip.net> <86prvekqs2.fsf@ds4.des.no> <86lk62kqeh.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <86lk62kqeh.fsf@ds4.des.no> User-Agent: Mutt/1.5.16 (2007-06-09) Cc: hackers@freebsd.org, Ed Schouten Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 13:13:38 -0000 On Sun, Feb 03, 2008 at 04:31:34PM +0100, Dag-Erling Smørgrav wrote: > Dag-Erling Smørgrav writes: > > Erik Trulsson writes: > > > Yep, it seems that GNU sort allocates a quite large buffer by default when > > > the size of the input is unknown (such as when it reads input from stdin.) > > > A quick check in the source code indicates that it tries to size this buffer > > > according to how much memory the system has (and according to any limits set > > > on how much memory the process is allowed to use.) > > Uh, OK. This scaling doesn't seem to work correctly. It seems to > > allocate 27 MB on 32-bit machines and 54 MB on 64-bit machines, > > regardless of memory size. I've looked at the code, as has a peer of mine. As you said: the code shows that when no files are specified (e.g. read off a pipe), sort will make some assumptions regarding the initial buffer size to read data into. The buffer size allocated in that case is fairly large, rather than basing it off of the first line off stdin; it looks like this is done to save CPU time in the long run (otherwise you'd have to rellocate more later and take a hit; initbuf() is responsible for that). I think being able to select which implementation you want (less memory with more potential CPU overhead, vs. more memory with less potential CPU overhead) would be best. Regarding Erik's concern over how much is allocated on 32-bit vs. 64-bit: two things come to mind: 1) There was a recent discussion about this on freebsd-am64, specifically in regards to increased VSZ of processes on amd64 vs. i386 with shared libraries: http://lists.freebsd.org/pipermail/freebsd-amd64/2007-September/010254.html 2) int/long on amd64 are obviously 64-bit in size, while on i386 they're 32-bit (e.g. half the size). 27*2=54, so possibly that's where the excess comes from? > Looking at the code, it seems to go to extreme lengths to get it > absolutely wrong. For instance, if hw.physmem / 8 > hw.usermem, it will > pick the former, which means it's pretty much guaranteed to either fail > or hose your system (or both). Can you expand on this? Looking at the code, it doesn't appear that's possible. The code in question is default_sort_size(), which is used when no -S or --buffer-size argument is specified. > Count this as a vote for ditching GNU sort in favor of a BSD-licensed > implementation (from {Net,Open}BSD for instance). In this specific case, I think you're bashing GNU just because you feel like it. Come on man... =/ -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 13:52:43 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E8FDE16A4E2; Mon, 4 Feb 2008 13:52:43 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 6DAC313C455; Mon, 4 Feb 2008 13:52:43 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 75A352085; Mon, 4 Feb 2008 14:52:35 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.2/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 511732084; Mon, 4 Feb 2008 14:52:35 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 3127F844A1; Mon, 4 Feb 2008 14:52:35 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Jeremy Chadwick References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> <20080203151550.GA67020@owl.midgard.homeip.net> <86prvekqs2.fsf@ds4.des.no> <86lk62kqeh.fsf@ds4.des.no> <20080204125831.GA4052@eos.sc1.parodius.com> Date: Mon, 04 Feb 2008 14:52:35 +0100 In-Reply-To: <20080204125831.GA4052@eos.sc1.parodius.com> (Jeremy Chadwick's message of "Mon\, 4 Feb 2008 04\:58\:31 -0800") Message-ID: <86d4rcg76k.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: hackers@freebsd.org, Ed Schouten Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 13:52:44 -0000 Jeremy Chadwick writes: > As you said: the code shows that when no files are specified (e.g. read > off a pipe), sort will make some assumptions regarding the initial > buffer size to read data into. The buffer size allocated in that case > is fairly large, rather than basing it off of the first line off stdin; > it looks like this is done to save CPU time in the long run (otherwise > you'd have to rellocate more later and take a hit; initbuf() is > responsible for that). Oh give me a break, the self-starting exponential algorithm for growth of dynamically allocated buffers has been known for decades. In case any GNU sort developers are reading this, here it comes, free of charge: static char *buf =3D NULL; static size_t bufsz =3D 0; static size_t buflen =3D 0; int buf_append(const char *str) { size_t len; len =3D strlen(str); if (buflen + len + 1 > bufsz) { size_t nbufsz =3D bufsz; char *nbuf; while (buflen + len + 1 > nbufsz) nbufsz =3D nbufsz * 2 + 1; nbuf =3D realloc(buf, nbufsz); if (nbuf =3D=3D NULL) return (-1); buf =3D nbuf; bufsz =3D nbufsz; } memcpy(buf + buflen, str, len); buf[buflen + len] =3D '\0'; buflen +=3D len; return (0); } With a good allocator - and depending to some extent on the memory usage pattern of the rest of your program - if you jump-start it by initally allocating 16 kB or so (and setting bufsz accordingly), realloc() will never need to copy anything - but even in the worst case, the amortized cost will be O(2n), IIRC. This is practically unnoticeable next to the cost of the sorting algorithm itself, which will be O(n log n) at best and O(n*n) at worst. > > Looking at the code, it seems to go to extreme lengths to get it > > absolutely wrong. For instance, if hw.physmem / 8 > hw.usermem, it will > > pick the former, which means it's pretty much guaranteed to either fail > > or hose your system (or both). > Can you expand on this? Looking at the code, it doesn't appear that's > possible. The code in question is default_sort_size(), which is used > when no -S or --buffer-size argument is specified. I looked at how it computes the cap, which is MAX(total / 8, avail) - in other words, never mind what's actually feasible, I want more! More! More, I say! > > Count this as a vote for ditching GNU sort in favor of a BSD-licensed > > implementation (from {Net,Open}BSD for instance). > In this specific case, I think you're bashing GNU just because you feel > like it. Come on man... =3D/ No, I'm bashing GNU because it's bloated crap, as this example clearly shows. It wouldn't be the first time a BSD rewrite of a GNU tool ended up performing better; see for instance bsdtar. Besides, the FreeBSD project has a tradition of replacing GNU tools with BSD-licensed equivalents as long as no functionality is lost. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 14:01:37 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1A68216A421; Mon, 4 Feb 2008 14:01:37 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 8014513C468; Mon, 4 Feb 2008 14:01:36 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 0EEFA1B10F39; Mon, 4 Feb 2008 15:01:35 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 860071B10EE0; Mon, 4 Feb 2008 15:01:28 +0100 (CET) Message-ID: <47A71AB7.6000209@moneybookers.com> Date: Mon, 04 Feb 2008 16:01:27 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Kris Kennaway References: <4795CC13.7080601@moneybookers.com> <4795FE54.9090606@moneybookers.com> <86lk6i0vzk.fsf@ds4.des.no> <479605E2.6070709@moneybookers.com> <47964356.6030602@moneybookers.com> <479647FB.3070909@FreeBSD.org> <47970EE2.5000400@moneybookers.com> <479754E6.1060101@moneybookers.com> <9bbcef730801230802n5c52832bk60c6afc47be578f4@mail.gmail.com> <47976AD4.3020203@moneybookers.com> <47A62EE8.2000700@FreeBSD.org> <47A62F1D.8070507@FreeBSD.org> <47A639B5.2020909@FreeBSD.org> <47A6EA28.6000902@moneybookers.com> In-Reply-To: <47A6EA28.6000902@moneybookers.com> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: gettimeofday() in hping X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 14:01:37 -0000 Greetings, Stefan Lambrev wrote: > Greetings, > > Kris Kennaway wrote: >> Kris Kennaway wrote: >> >>>> Fixing all of the above I can send at about 13MB/sec (timecounter >>>> is not relevant any more). The CPU is spending about 75% of the >>>> time in the kernel, so >>> that is the next place to look. [hit send too soon] >> >> Actually 15MB/sec once I disable all kernel debugging. This is >> identical to Linux 2.6.24 on the same hardware. The patch I use to >> fix hping brain-damage is attached. >> >> Kris > Indeed this patch highly improve results (x2) under FreeBSD. On my > hardware the max speeds under FreeBSD i still little slower compared > to Linux (something like 19 vs 19.5 MB/s) > but the speed is quite more stable under FreeBSD (under linux the > average speed seems slower) > > I didn't tested the patch under linux but will do soon. > Thanks for the help :) > > BTW 262144 is little high as this match the default value in FreeBSD, > I tested with twice smaller buffer and do not see performance lost. > Kris if you do not mind I'll write to hping developers to adopt this > patch, and if no response from them I can try to reach the port > maintainer, so we have this patched in ports? > I have results from hping + patches with different counters in freebsd and linux: Linux - ACPI em1 in 22.257 MB/s 22.479 MB/s 15.384 GB out 13.735 MB/s 14.247 MB/s 12.432 GB Linux -- TSC em1 in 22.334 MB/s 22.445 MB/s 18.599 GB out 13.567 MB/s 14.176 MB/s 14.379 GB Linux - jiffies em1 in 22.119 MB/s 22.268 MB/s 20.078 GB out 13.962 MB/s 14.058 MB/s 15.270 GB FreeBSD - ACPI em1 in 13.157 MB/s 13.162 MB/s 23.697 GB out 13.150 MB/s 13.153 MB/s 17.976 GB FreeBSD - TSC em1 in 18.624 MB/s 18.832 MB/s 25.507 GB out 17.008 MB/s 17.041 MB/s 19.681 GB FreeBSD - i8254 em1 in 6.763 MB/s 6.763 MB/s 26.005 GB out 6.756 MB/s 6.758 MB/s 20.151 GB FreeBSD - dummy em1 in 18.705 MB/s 18.796 MB/s 27.014 GB out 17.217 MB/s 17.225 MB/s 21.082 GB It is weird that changing time counter in linux does not affect performance which may mean, that time counter is not changed at all (always TSC used?) ... But here are few others interesting findings 1) patch increase performance in linux too. 2) when flooding from linux the numbers of returned packets are less. 3) on both linux and freebsd the max combined speed is 35MB/s which is something like 630,000 packets/s Does anyone ever succeed to transfer without error more then 630kpps (300kpps incoming) with em driver and intel gigabit card? -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 14:41:22 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 024E616A418 for ; Mon, 4 Feb 2008 14:41:22 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id A261513C4D9 for ; Mon, 4 Feb 2008 14:41:21 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender:X-Spam-Status:Subject; b=ksSmH6stS60MPYniO+FrE9FUUxu7rXEIMI4ZPfUfuKmcAqQ1MLwjac0Hk7C0Rb79gtcXY3hbR1h3vuYohPgy3FW6ueBI+UfUgziXbFLIfgORcDF22Psbv7ex7EpTLyifTT8yv10+jolbzFgpprCAIKq/BEYN6Fm1GZrHSgZ83p0=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1JM2Vb-000BFF-Fh; Mon, 04 Feb 2008 17:41:19 +0300 Date: Mon, 4 Feb 2008 17:41:18 +0300 From: Eygene Ryabinkin To: cristi@roedu.net Message-ID: References: <4c65bf9e-80e5-4c0b-acef-d9914a57679f@i12g2000prf.googlegroups.com> <47728D80.5010507@zirakzigil.org> <475A8637.9020507@zirakzigil.org> <20071212155244.GJ1442@daemon.grid.kiae.ru> <4760114D.6020203@zirakzigil.org> <47950952.6040909@roedu.net> <41549.193.226.5.33.1201087601.squirrel@mail.roedu.net> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <41549.193.226.5.33.1201087601.squirrel@mail.roedu.net> Sender: rea-fbsd@codelabs.ru X-Spam-Status: No, score=-1.9 required=4.0 tests=ALL_TRUSTED,AWL,BAYES_40 Cc: freebsd-hackers@freebsd.org, Giulio Ferro , nork@freebsd.org Subject: Re: Synaptics X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 14:41:22 -0000 Cristian, good day. Wed, Jan 23, 2008 at 01:26:41PM +0200, cristi@roedu.net wrote: > >> I just ran into the same problem. In xorg.conf I explicitly told the > >> synaptics driver to use "psm" and "/dev/psm0", but the error message > >> would > >> suggest that it uses "event". > >> > >> Also, I tried to change the source code of the synaptics driver > >> (synaptics.c) and hard-coded "psm" as the only driver, no matter what > >> xorg.conf says. Synaptics still would not start, but this time > >> complaining > >> that no device was specified. Please note that I had "Device" in my > >> xorg.conf, but the error suggests that the driver ignored it. > > Suppose I use the attached xorg.conf file, at some point, > /var/log/Xorg.0.log shows the following error: > > (II) Synaptics touchpad driver version 0.14.6 (1406) > Synaptics_Touchpad no synaptics event device found (checked 10 nodes) > Synaptics_Touchpad The /dev/input/event* device nodes seem to be missing > (EE) xf86OpenSerial: No Device specified. > Synaptics driver unable to open device > (EE) PreInit failed for input device "Synaptics_Touchpad" > (II) UnloadModule: "synaptics" > > As you said, it looks like synaptics is trying to use the "auto" protocol, > although the configuration file tells it to use "psm". > > Now, if I put the attached patch in x11-drivers/synaptics/files, using the > same xorg.conf, synaptics will fail like this: > > (II) Synaptics touchpad driver version 0.14.6 (1406) > (EE) xf86OpenSerial: No Device specified. > Synaptics driver unable to open device > (EE) PreInit failed for input device "Synaptics_Touchpad" > (II) UnloadModule: "synaptics" > > It almost looks as if I have to hardcode the device too, because synaptics > certainly ignores my options. This is a sort of 'ping' mail, sorry. To the point: I had reproduced the problem and will start looking into it once this message will fly from my mailserver. Stay tuned ;)) -- Eygene From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 15:04:43 2008 Return-Path: Delivered-To: hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1A26816A46C for ; Mon, 4 Feb 2008 15:04:43 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id CAA2E13C4E3 for ; Mon, 4 Feb 2008 15:04:42 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 871E72082; Mon, 4 Feb 2008 16:04:35 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.2/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 75E682049; Mon, 4 Feb 2008 16:04:35 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 5C2108449D; Mon, 4 Feb 2008 16:04:35 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Ed Schouten References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> <20080203151550.GA67020@owl.midgard.homeip.net> <86prvekqs2.fsf@ds4.des.no> <86lk62kqeh.fsf@ds4.des.no> <20080204145534.GA40490@VARK.MIT.EDU> Date: Mon, 04 Feb 2008 16:04:35 +0100 In-Reply-To: <20080204145534.GA40490@VARK.MIT.EDU> (David Schultz's message of "Mon\, 4 Feb 2008 09\:55\:34 -0500") Message-ID: <86myqgepa4.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: hackers@FreeBSD.ORG Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 15:04:43 -0000 David Schultz writes: > We had been using a BSD-licensed sort(1), but ache@ changed it back to > GNU sort several years ago. Anyone know why? If I had to guess I'd say > i18n [...] That is my recollection as well. We would do well to take a look at the code (and CVS logs) from NetBSD; they may have solved the i18n issues by now. There may also be a few missing features: someone mentioned -g, which is apparently similar to yet somehow different from -n, in weird and wonderful ways which the authors did not bother documenting. Last I checked, it also (rather surprisingly) lacked -u (unique), which is required by POSIX. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 15:07:13 2008 Return-Path: Delivered-To: hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D4AAD16A420 for ; Mon, 4 Feb 2008 15:07:13 +0000 (UTC) (envelope-from das@FreeBSD.ORG) Received: from VARK.MIT.EDU (VARK.MIT.EDU [18.95.3.179]) by mx1.freebsd.org (Postfix) with ESMTP id 949E713C478 for ; Mon, 4 Feb 2008 15:07:13 +0000 (UTC) (envelope-from das@FreeBSD.ORG) Received: from VARK.MIT.EDU (localhost [127.0.0.1]) by VARK.MIT.EDU (8.14.2/8.14.1) with ESMTP id m14EtYtC040591; Mon, 4 Feb 2008 09:55:34 -0500 (EST) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by VARK.MIT.EDU (8.14.2/8.14.1/Submit) id m14EtYVs040590; Mon, 4 Feb 2008 09:55:34 -0500 (EST) (envelope-from das@FreeBSD.ORG) Date: Mon, 4 Feb 2008 09:55:34 -0500 From: David Schultz To: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= Message-ID: <20080204145534.GA40490@VARK.MIT.EDU> Mail-Followup-To: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= , Ed Schouten , hackers@FreeBSD.ORG References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> <20080203151550.GA67020@owl.midgard.homeip.net> <86prvekqs2.fsf@ds4.des.no> <86lk62kqeh.fsf@ds4.des.no> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <86lk62kqeh.fsf@ds4.des.no> Cc: hackers@FreeBSD.ORG, Ed Schouten Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 15:07:13 -0000 On Sun, Feb 03, 2008, Dag-Erling Smørgrav wrote: > Dag-Erling Smørgrav writes: > > Erik Trulsson writes: > > > Yep, it seems that GNU sort allocates a quite large buffer by default when > > > the size of the input is unknown (such as when it reads input from stdin.) > > > A quick check in the source code indicates that it tries to size this buffer > > > according to how much memory the system has (and according to any limits set > > > on how much memory the process is allowed to use.) > > Uh, OK. This scaling doesn't seem to work correctly. It seems to > > allocate 27 MB on 32-bit machines and 54 MB on 64-bit machines, > > regardless of memory size. > > Looking at the code, it seems to go to extreme lengths to get it > absolutely wrong. For instance, if hw.physmem / 8 > hw.usermem, it will > pick the former, which means it's pretty much guaranteed to either fail > or hose your system (or both). > > In the immortal words of Blazing Star: YOU FAIL IT > > Count this as a vote for ditching GNU sort in favor of a BSD-licensed > implementation (from {Net,Open}BSD for instance). We had been using a BSD-licensed sort(1), but ache@ changed it back to GNU sort several years ago. Anyone know why? If I had to guess I'd say i18n, but that's not very hard to deal with these days given strcoll(3). That said, I'm unaware of any technical differences between the two. From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 15:10:43 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4DCC116A469 for ; Mon, 4 Feb 2008 15:10:43 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 3BC1513C467 for ; Mon, 4 Feb 2008 15:10:43 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from zion.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by elvis.mu.org (Postfix) with ESMTP id CEF501A4D86; Mon, 4 Feb 2008 07:10:42 -0800 (PST) From: John Baldwin To: freebsd-hackers@freebsd.org Date: Mon, 4 Feb 2008 10:03:22 -0500 User-Agent: KMail/1.9.7 References: <47A4FF5F.9010604@gmail.com> In-Reply-To: <47A4FF5F.9010604@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200802041003.22658.jhb@freebsd.org> Cc: "Aryeh M. Friedman" Subject: Re: /dev/dsp disappeared after power outage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 15:10:43 -0000 On Saturday 02 February 2008 06:40:15 pm Aryeh M. Friedman wrote: > I just had a power outage and when it came back /dev/dsp0.0 was > missing from the devices. the kern module loaded fine and detected > the card correctly (according to dmesg, sysctl and /dev/sndstat) but > neither the above or /dev/pcm exists. After rebooting the problem > remains. Any ideas how to fix it? Nothing to fix. This is how devfs device cloning works. -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 15:14:10 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C87B716A469 for ; Mon, 4 Feb 2008 15:14:10 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from www.pkgsrc-box.org (www.ostsee-abc.de [62.206.222.50]) by mx1.freebsd.org (Postfix) with ESMTP id 782D513C501 for ; Mon, 4 Feb 2008 15:14:10 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from britannica.bec.de (www.pkgsrc-box.org [127.0.0.1]) by www.pkgsrc-box.org (Postfix) with ESMTP id 7B827E7A3E0; Mon, 4 Feb 2008 15:14:06 +0000 (UTC) Received: by britannica.bec.de (Postfix, from userid 1000) id 20038175CA; Mon, 4 Feb 2008 16:02:43 +0100 (CET) Date: Mon, 4 Feb 2008 16:02:43 +0100 From: Joerg Sonnenberger To: freebsd-hackers@freebsd.org, hackers@FreeBSD.ORG Message-ID: <20080204150243.GA3944@britannica.bec.de> Mail-Followup-To: freebsd-hackers@freebsd.org, hackers@FreeBSD.ORG References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> <20080203151550.GA67020@owl.midgard.homeip.net> <86prvekqs2.fsf@ds4.des.no> <86lk62kqeh.fsf@ds4.des.no> <20080204145534.GA40490@VARK.MIT.EDU> <86myqgepa4.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <86myqgepa4.fsf@ds4.des.no> User-Agent: Mutt/1.5.16 (2007-06-09) Cc: Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 15:14:10 -0000 On Mon, Feb 04, 2008 at 04:04:35PM +0100, Dag-Erling Sm?rgrav wrote: > Last I checked, it also (rather surprisingly) lacked -u (unique), > which is required by POSIX. That must have been before the import into src/usr.bin/sort in 2000. Joerg From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 15:38:33 2008 Return-Path: Delivered-To: hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D891A16A417 for ; Mon, 4 Feb 2008 15:38:33 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from www.pkgsrc-box.org (www.ostsee-abc.de [62.206.222.50]) by mx1.freebsd.org (Postfix) with ESMTP id AB7CA13C458 for ; Mon, 4 Feb 2008 15:38:32 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from britannica.bec.de (www.pkgsrc-box.org [127.0.0.1]) by www.pkgsrc-box.org (Postfix) with ESMTP id 7B827E7A3E0; Mon, 4 Feb 2008 15:14:06 +0000 (UTC) Received: by britannica.bec.de (Postfix, from userid 1000) id 20038175CA; Mon, 4 Feb 2008 16:02:43 +0100 (CET) Date: Mon, 4 Feb 2008 16:02:43 +0100 From: Joerg Sonnenberger To: freebsd-hackers@freebsd.org, hackers@FreeBSD.ORG Message-ID: <20080204150243.GA3944@britannica.bec.de> Mail-Followup-To: freebsd-hackers@freebsd.org, hackers@FreeBSD.ORG References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> <20080203151550.GA67020@owl.midgard.homeip.net> <86prvekqs2.fsf@ds4.des.no> <86lk62kqeh.fsf@ds4.des.no> <20080204145534.GA40490@VARK.MIT.EDU> <86myqgepa4.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <86myqgepa4.fsf@ds4.des.no> User-Agent: Mutt/1.5.16 (2007-06-09) Cc: Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 15:38:33 -0000 On Mon, Feb 04, 2008 at 04:04:35PM +0100, Dag-Erling Sm?rgrav wrote: > Last I checked, it also (rather surprisingly) lacked -u (unique), > which is required by POSIX. That must have been before the import into src/usr.bin/sort in 2000. Joerg From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 16:10:28 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3347016A417 for ; Mon, 4 Feb 2008 16:10:28 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id 2730C13C4F9 for ; Mon, 4 Feb 2008 16:10:26 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id 2EFD243DEC6; Mon, 4 Feb 2008 17:47:44 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Sdig71qR3LVa; Mon, 4 Feb 2008 17:47:44 +0200 (EET) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 3466343DDA5; Mon, 4 Feb 2008 17:47:43 +0200 (EET) Message-ID: <47A7339E.4070708@icyb.net.ua> Date: Mon, 04 Feb 2008 17:47:42 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20080123) MIME-Version: 1.0 To: Remko Lodder , scottl@FreeBSD.org, freebsd-fs@freebsd.org References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> In-Reply-To: <47A2F404.7010208@icyb.net.ua> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Pav Lucistnik , Bruce Evans Subject: fs/udf: vm pages "overlap" while reading large dir X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 16:10:28 -0000 More on the problem with reading big directories on UDF. First, some sleuthing. I came to believe that the problem is caused by some larger change in vfs/vm/buf area. It seems that now VMIO is applied to more vnode types than before. In particular it seems that now vnodes for devices have non-NULL v_object (or maybe this is about directory vnodes, I am not sure). Also it seems that the problem should happen for any directory with size larger than four 2048-bytes sectors (I think that any directory with > 300 files would definitely qualify). After some code reading and run-time debugging, here are some facts about udf directory reading: 1. bread-ing is done via device vnode (as opposed to directory vnodes), as a consequence udf_strategy is not involved in directory reading 2. the device vnode has a non-NULL v_object, this means that VMIO is used 3. it seems that the code assumed that non-VM buffers are used 4. bread is done on as much data as possible, up to MAXBSIZE [and even slightly more in some cases] (i.e. code determines directory data size and attempts to read in everything in one go) 5. physical sector number adjusted to DEV_BSIZE (512 bytes) sectors is passed to bread - this is because of #1 above and peculiarities of buf system 6. directory reading and file reading are quite different, because the latter does reading via file vnode Let's a consider a simple case of "directory reading" 5 * PAGE_SIZE (4K) bytes starting from a physical sector N with N%2 = 0 from media with sector size of 2K (most common CD/DVD case): 1. we want to read 12 KB = 3 pages = 6 sectors starting from sector N, N%2 = 0 2. 4*N is passed as a sector number to bread 3. bo_bsize of the corresponding bufobj is a media sector size, i.e. 2K 4. actual read in bread will happen from b_iooffset of 4*N*DEV_BSIZE = N*2K, i.e. correct offset within the device 5. for a fresh read getblk will be called 6. getblk will set b_offset to blkno*bo_bsize=4*N*2K, i.e. 4 times the correct byte offset within the device 7. allocbuf will allocate pages using B_VMIO case 8. allocbuf calculates base page index as follows: pi = b_offset/PAGE_SIZE this means the following: sectors N and N+1 will be bound to a page with index 4*N*2K/4K = 2*N sectors N+2 and N+3 will be bound to the next page 2*N +1 sectors N+4 and N+5 is tied to the next page 2*N + 2 Now let's consider a "directory read" of 2 sectors (1 page) starting at physical sector N+1. Using the same calculations as above, we see that the sector will be tied to a page 2*(N+1) = 2*N + 2. And this page already contains valid cached data from the previous read, *but* it contains data from sectors N+4 and N+5 instead of N+1 and N+2. So, we see, that because of b_offset being 4 times the actual byte offset, we get incorrect page indexing. Namely, sector X gets associated with different pages depending on what sector is used as a starting sector for bread. If bread starts at sector N, then data of sector N+1 is associated with (second half of) page with index 2*N; but if bread starts at sector N+1 then it gets associated with (the first half of) page with index 2*(N+1). Previously, before VMIO change, data for all reads was put into separate buffers that did not share anything between them. So the problem was limited only to wasting memory with duplicate data, but no actual over-runs did happen. Now we have the over-runs because the VM pages are shared between the buffers of the same vnode. One obvious solution is to limit bread size to 2*PAGE_SIZE = 4 * sector_size. In this case, as before, we would waste some memory on duplicate data but we would avoid page overruns. If we limit bread size even more, to 1 sector, then we would not have any duplicate data at all. But there would still be some resource waste - each page would correspond to one sector, so 4K page would have only 2K of valid data and the other half in each page is unused. Another solution, which to me seems to be better, is to do "usual" reading - pass a directory vnode and 2048-byte sector offset to bread. In this case udf_strategy will get called for bklno translation, all offsets and indexes will be correct and everything will work perfectly as it is the case for all other filesystems. The only overhead in this case comes from the need to handle UDF_INVALID_BMAP case (where data is embedded into file entry). So it means that we have to call bmap_internal to see if we have that special case and hanlde it, and if the case is not special we would call bread on a directory vnode which means that bmap_internal would be called again in udf_strategy. One optimization that can be done in this case is to create a ligher version of bmap_internal that would merely check for the special case and wouldn't do anything else. I am attaching a patch based on the ideas above. It fixes the problem for me and doesn't seem to create any new ones :-) About the patch: hunk #1 fixes a copy/paste hunk #2 should fixes strategy to not set junk b_blkno in case of udf_bmap_internal failure and also replaces divisions/multiplications with bit-shifts hunk #3 really limits max size in bread to MAXBSIZE and also makes bread to be done a directory vnode BTW, maybe it's a good idea to further limit size of bread() anyway. Like always reading 1 sector. Anyway, this should make any difference only in minority of cases and I am not sure about performance effects (I do not expect difference in anything else other than performance). on 01/02/2008 12:27 Andriy Gapon said the following: > on 01/02/2008 12:00 Andriy Gapon said the following: >> ---- a different, under-debugged problem ----- >> BTW, on some smaller directories (but still large ones) I get some very >> strange problems with reading a directory too. It seems like some bad >> interaction between udf and buffer cache system. I added a lot of >> debugging prints and the problems looks like the following: >> >> read starting at physical sector (2048-byte one) N, size is ~20K, N%4=0 >> bread(4 * N, some_big_size) >> correct data is read >> repeat the above couple dozen times >> read starting at physical sector (2048-byte one) N+1, size is ~20K >> bread(4 * (N+1), some_big_size) >> data is read from physical sector N+4 (instead of N+1) >> >> I remember that Bruce Evance warned me that something like this could >> happen but I couldn't understand him, because I don't understand >> VM/buffer subsystem. I'll try to dig up the email. >> > > Sorry for the flood - additional info: > if I limit max read size in udf_readatoffset() to 2048 (instead of > MAXBSIZE), then large directories can be read OK. Seems like something > with overlapping buffers, maybe? > > BTW, here's how I created test environment for the described issues (in > tcsh): > > mkdir /tmp/bigdir > > cd /tmp/bigdir > > set i=1 > > while ($i < NNNNN) > touch file.$i > set i=`expr $i + 1` > end > > cd /tmp > > mkisofs -udf -o test.iso bigdir > > mdconfig -a -t vnode -f test.iso -S 2048 -u 0 > > mount_udf /dev/md0 /mnt > > ls -l /mnt > -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 16:19:52 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4A7C016A417; Mon, 4 Feb 2008 16:19:52 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 0360413C448; Mon, 4 Feb 2008 16:19:51 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 3DCCF2083; Mon, 4 Feb 2008 17:19:43 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.2/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 30731207E; Mon, 4 Feb 2008 17:19:43 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 1717984493; Mon, 4 Feb 2008 17:19:43 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: freebsd-hackers@freebsd.org References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> <20080203151550.GA67020@owl.midgard.homeip.net> <86prvekqs2.fsf@ds4.des.no> <86lk62kqeh.fsf@ds4.des.no> <20080204145534.GA40490@VARK.MIT.EDU> <86myqgepa4.fsf@ds4.des.no> <20080204150243.GA3944@britannica.bec.de> Date: Mon, 04 Feb 2008 17:19:43 +0100 In-Reply-To: <20080204150243.GA3944@britannica.bec.de> (Joerg Sonnenberger's message of "Mon\, 4 Feb 2008 16\:02\:43 +0100") Message-ID: <864pcoelsw.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: hackers@FreeBSD.ORG Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 16:19:52 -0000 Joerg Sonnenberger writes: > On Mon, Feb 04, 2008 at 04:04:35PM +0100, Dag-Erling Sm?rgrav wrote: > > Last I checked, it also (rather surprisingly) lacked -u (unique), > > which is required by POSIX. > That must have been before the import into src/usr.bin/sort in 2000. Oops - I looked at the man page, not the code, and misread the SYNOPSIS. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 16:19:52 2008 Return-Path: Delivered-To: hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4A7C016A417; Mon, 4 Feb 2008 16:19:52 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 0360413C448; Mon, 4 Feb 2008 16:19:51 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 3DCCF2083; Mon, 4 Feb 2008 17:19:43 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.2/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 30731207E; Mon, 4 Feb 2008 17:19:43 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 1717984493; Mon, 4 Feb 2008 17:19:43 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: freebsd-hackers@freebsd.org References: <8663x6mc2o.fsf@ds4.des.no> <20080203131322.GK1179@hoeg.nl> <20080203151550.GA67020@owl.midgard.homeip.net> <86prvekqs2.fsf@ds4.des.no> <86lk62kqeh.fsf@ds4.des.no> <20080204145534.GA40490@VARK.MIT.EDU> <86myqgepa4.fsf@ds4.des.no> <20080204150243.GA3944@britannica.bec.de> Date: Mon, 04 Feb 2008 17:19:43 +0100 In-Reply-To: <20080204150243.GA3944@britannica.bec.de> (Joerg Sonnenberger's message of "Mon\, 4 Feb 2008 16\:02\:43 +0100") Message-ID: <864pcoelsw.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: hackers@FreeBSD.ORG Subject: Re: sort(1) memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 16:19:52 -0000 Joerg Sonnenberger writes: > On Mon, Feb 04, 2008 at 04:04:35PM +0100, Dag-Erling Sm?rgrav wrote: > > Last I checked, it also (rather surprisingly) lacked -u (unique), > > which is required by POSIX. > That must have been before the import into src/usr.bin/sort in 2000. Oops - I looked at the man page, not the code, and misread the SYNOPSIS. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 16:24:20 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7A82E16A468; Mon, 4 Feb 2008 16:24:20 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 26F8C13C467; Mon, 4 Feb 2008 16:24:19 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender:X-Spam-Status:Subject; b=Ln/ot7Tcs78WEAR99yLZ6vbOISjdYjKwczrY4iqs9W863TG56Zq3qHWXr6ToPi9moPblpw+z42kmXsaPaigRiWEe6lGJTN4fMmRJZGNbVawUmHNuOiCpphUpWV8pTID2+3qkOiiQoFENiUnH1MP/X2elSQSW7aiCkQFL2OsAuOE=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1JM47H-000Bpx-0d; Mon, 04 Feb 2008 19:24:19 +0300 Date: Mon, 4 Feb 2008 19:24:17 +0300 From: Eygene Ryabinkin To: cristi@roedu.net Message-ID: References: <4c65bf9e-80e5-4c0b-acef-d9914a57679f@i12g2000prf.googlegroups.com> <47728D80.5010507@zirakzigil.org> <475A8637.9020507@zirakzigil.org> <20071212155244.GJ1442@daemon.grid.kiae.ru> <4760114D.6020203@zirakzigil.org> <47950952.6040909@roedu.net> <41549.193.226.5.33.1201087601.squirrel@mail.roedu.net> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="pvezYHf7grwyp3Bc" Content-Disposition: inline In-Reply-To: Sender: rea-fbsd@codelabs.ru X-Spam-Status: No, score=-1.9 required=4.0 tests=ALL_TRUSTED,AWL,BAYES_50 Cc: freebsd-hackers@freebsd.org, Giulio Ferro , nork@freebsd.org Subject: Re: Synaptics X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 16:24:20 -0000 --pvezYHf7grwyp3Bc Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Me again. Mon, Feb 04, 2008 at 05:41:18PM +0300, Eygene Ryabinkin wrote: > This is a sort of 'ping' mail, sorry. To the point: I had reproduced > the problem and will start looking into it once this message will > fly from my mailserver. Stay tuned ;)) OK, things should be better with the attached patch. I was not able to fully test the resulting Synaptics driver, since I have no Synaptics beast at my amd64 machine ;)) But with the provided patch, my Synaptics driver tries to search for the psm device, as I told him, so, please, give it a try. Must be patched with 'patch -p1' and one should be in the port directory. -- Eygene --pvezYHf7grwyp3Bc Content-Type: text/x-diff; charset=koi8-r Content-Disposition: attachment; filename="synaptics-amd64-ARCH.patch" >From 083c1be4c91da739436f2b1e509a96512ac05867 Mon Sep 17 00:00:00 2001 From: Eygene Ryabinkin Date: Mon, 4 Feb 2008 19:17:43 +0300 Subject: [PATCH] Fix compilation at amd64 by overriding the ARCH variable properly. FreeBSD make sets ARCH variable to 'amd64' [1] and invokes GNU make. It inherits the ARCH variable and refuses to set it via ordinary '=' operator. So we must force ARCH assignments. [1] Try 'make -V ARCH' in the port directory. Signed-off-by: Eygene Ryabinkin --- files/patch-Makefile | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/files/patch-Makefile b/files/patch-Makefile index 1ae3cbe..90870c5 100644 --- a/files/patch-Makefile +++ b/files/patch-Makefile @@ -1,5 +1,5 @@ ---- Makefile.orig Sun Jul 16 00:58:26 2006 -+++ Makefile Sun Aug 13 10:47:35 2006 +--- Makefile.orig 2006-07-15 19:58:26.000000000 +0400 ++++ Makefile 2008-02-04 19:11:33.000000000 +0300 @@ -12,14 +12,14 @@ MANDIR = $(DESTDIR)$(PREFIX)/man @@ -8,7 +8,8 @@ + ARCH = $(shell uname -m) endif ifeq ($(ARCH),amd64) - ARCH = x86_64 +- ARCH = x86_64 ++ override ARCH = x86_64 endif ifeq ($(ARCH),x86_64) ARCH_DEFINES = -D__x86_64__ -D_XSERVER64 -- 1.5.3.8 --pvezYHf7grwyp3Bc-- From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 16:27:07 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 64EE116A4E6 for ; Mon, 4 Feb 2008 16:27:07 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id B4C5B13C4E7 for ; Mon, 4 Feb 2008 16:27:06 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id 9178F43D7AD; Mon, 4 Feb 2008 17:56:22 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id whg4EB8ETErA; Mon, 4 Feb 2008 17:56:22 +0200 (EET) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 9ABA143C8F2; Mon, 4 Feb 2008 17:56:21 +0200 (EET) Message-ID: <47A735A4.3060506@icyb.net.ua> Date: Mon, 04 Feb 2008 17:56:20 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20080123) MIME-Version: 1.0 To: Remko Lodder , scottl@FreeBSD.org, freebsd-fs@freebsd.org References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> In-Reply-To: <47A2F404.7010208@icyb.net.ua> Content-Type: multipart/mixed; boundary="------------070300020503030807040707" Cc: freebsd-hackers@freebsd.org, Pav Lucistnik , Bruce Evans Subject: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 16:27:07 -0000 This is a multi-part message in MIME format. --------------070300020503030807040707 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit More on the problem with reading big directories on UDF. First, some sleuthing. I came to believe that the problem is caused by some larger change in vfs/vm/buf area. It seems that now VMIO is applied to more vnode types than before. In particular it seems that now vnodes for devices have non-NULL v_object (or maybe this is about directory vnodes, I am not sure). Also it seems that the problem should happen for any directory with size larger than four 2048-bytes sectors (I think that any directory with > 300 files would definitely qualify). After some code reading and run-time debugging, here are some facts about udf directory reading: 1. bread-ing is done via device vnode (as opposed to directory vnodes), as a consequence udf_strategy is not involved in directory reading 2. the device vnode has a non-NULL v_object, this means that VMIO is used 3. it seems that the code assumed that non-VM buffers are used 4. bread is done on as much data as possible, up to MAXBSIZE [and even slightly more in some cases] (i.e. code determines directory data size and attempts to read in everything in one go) 5. physical sector number adjusted to DEV_BSIZE (512 bytes) sectors is passed to bread - this is because of #1 above and peculiarities of buf system 6. directory reading and file reading are quite different, because the latter does reading via file vnode Let's a consider a simple case of "directory reading" 5 * PAGE_SIZE (4K) bytes starting from a physical sector N with N%2 = 0 from media with sector size of 2K (most common CD/DVD case): 1. we want to read 12 KB = 3 pages = 6 sectors starting from sector N, N%2 = 0 2. 4*N is passed as a sector number to bread 3. bo_bsize of the corresponding bufobj is a media sector size, i.e. 2K 4. actual read in bread will happen from b_iooffset of 4*N*DEV_BSIZE = N*2K, i.e. correct offset within the device 5. for a fresh read getblk will be called 6. getblk will set b_offset to blkno*bo_bsize=4*N*2K, i.e. 4 times the correct byte offset within the device 7. allocbuf will allocate pages using B_VMIO case 8. allocbuf calculates base page index as follows: pi = b_offset/PAGE_SIZE this means the following: sectors N and N+1 will be bound to a page with index 4*N*2K/4K = 2*N sectors N+2 and N+3 will be bound to the next page 2*N +1 sectors N+4 and N+5 is tied to the next page 2*N + 2 Now let's consider a "directory read" of 2 sectors (1 page) starting at physical sector N+1. Using the same calculations as above, we see that the sector will be tied to a page 2*(N+1) = 2*N + 2. And this page already contains valid cached data from the previous read, *but* it contains data from sectors N+4 and N+5 instead of N+1 and N+2. So, we see, that because of b_offset being 4 times the actual byte offset, we get incorrect page indexing. Namely, sector X gets associated with different pages depending on what sector is used as a starting sector for bread. If bread starts at sector N, then data of sector N+1 is associated with (second half of) page with index 2*N; but if bread starts at sector N+1 then it gets associated with (the first half of) page with index 2*(N+1). Previously, before VMIO change, data for all reads was put into separate buffers that did not share anything between them. So the problem was limited only to wasting memory with duplicate data, but no actual over-runs did happen. Now we have the over-runs because the VM pages are shared between the buffers of the same vnode. One obvious solution is to limit bread size to 2*PAGE_SIZE = 4 * sector_size. In this case, as before, we would waste some memory on duplicate data but we would avoid page overruns. If we limit bread size even more, to 1 sector, then we would not have any duplicate data at all. But there would still be some resource waste - each page would correspond to one sector, so 4K page would have only 2K of valid data and the other half in each page is unused. Another solution, which to me seems to be better, is to do "usual" reading - pass a directory vnode and 2048-byte sector offset to bread. In this case udf_strategy will get called for bklno translation, all offsets and indexes will be correct and everything will work perfectly as it is the case for all other filesystems. The only overhead in this case comes from the need to handle UDF_INVALID_BMAP case (where data is embedded into file entry). So it means that we have to call bmap_internal to see if we have that special case and hanlde it, and if the case is not special we would call bread on a directory vnode which means that bmap_internal would be called again in udf_strategy. One optimization that can be done in this case is to create a ligher version of bmap_internal that would merely check for the special case and wouldn't do anything else. I am attaching a patch based on the ideas above. It fixes the problem for me and doesn't seem to create any new ones :-) About the patch: hunk #1 fixes a copy/paste hunk #2 should fixes strategy to not set junk b_blkno in case of udf_bmap_internal failure and also replaces divisions/multiplications with bit-shifts hunk #3 really limits max size in bread to MAXBSIZE and also makes bread to be done a directory vnode BTW, maybe it's a good idea to further limit size of bread() anyway. Like always reading 1 sector. Anyway, this should make any difference only in minority of cases and I am not sure about performance effects (I do not expect difference in anything else other than performance). on 01/02/2008 12:27 Andriy Gapon said the following: > on 01/02/2008 12:00 Andriy Gapon said the following: >> ---- a different, under-debugged problem ----- >> BTW, on some smaller directories (but still large ones) I get some very >> strange problems with reading a directory too. It seems like some bad >> interaction between udf and buffer cache system. I added a lot of >> debugging prints and the problems looks like the following: >> >> read starting at physical sector (2048-byte one) N, size is ~20K, N%4=0 >> bread(4 * N, some_big_size) >> correct data is read >> repeat the above couple dozen times >> read starting at physical sector (2048-byte one) N+1, size is ~20K >> bread(4 * (N+1), some_big_size) >> data is read from physical sector N+4 (instead of N+1) >> >> I remember that Bruce Evance warned me that something like this could >> happen but I couldn't understand him, because I don't understand >> VM/buffer subsystem. I'll try to dig up the email. >> > > Sorry for the flood - additional info: > if I limit max read size in udf_readatoffset() to 2048 (instead of > MAXBSIZE), then large directories can be read OK. Seems like something > with overlapping buffers, maybe? > > BTW, here's how I created test environment for the described issues (in > tcsh): > > mkdir /tmp/bigdir > > cd /tmp/bigdir > > set i=1 > > while ($i < NNNNN) > touch file.$i > set i=`expr $i + 1` > end > > cd /tmp > > mkisofs -udf -o test.iso bigdir > > mdconfig -a -t vnode -f test.iso -S 2048 -u 0 > > mount_udf /dev/md0 /mnt > > ls -l /mnt > -- Andriy Gapon --------------070300020503030807040707 Content-Type: text/x-patch; name="udf_vnops.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="udf_vnops.diff" --- udf_vnops.c.orig 2008-01-29 23:50:49.000000000 +0200 +++ udf_vnops.c 2008-02-03 23:36:15.000000000 +0200 @@ -340,7 +340,7 @@ #define lblkno(udfmp, loc) ((loc) >> (udfmp)->bshift) #define blkoff(udfmp, loc) ((loc) & (udfmp)->bmask) -#define lblktosize(imp, blk) ((blk) << (udfmp)->bshift) +#define lblktosize(udfmp, blk) ((blk) << (udfmp)->bshift) static int udf_read(struct vop_read_args *ap) @@ -802,7 +802,6 @@ int maxsize; daddr_t sector; struct bufobj *bo; - int multiplier; bp = a->a_bp; vp = a->a_vp; @@ -813,21 +812,17 @@ * Files that are embedded in the fentry don't translate well * to a block number. Reject. */ - if (udf_bmap_internal(node, bp->b_lblkno * node->udfmp->bsize, - §or, &maxsize)) { + if (udf_bmap_internal(node, lblktosize(node->udfmp, bp->b_lblkno), §or, &maxsize)) { clrbuf(bp); bp->b_blkno = -1; + bufdone(bp); + return (0); } + /* udf_bmap_internal gives sector numbers, bio works with device blocks */ + bp->b_blkno = sector << (node->udfmp->bshift - DEV_BSHIFT); + } - /* bmap gives sector numbers, bio works with device blocks */ - multiplier = node->udfmp->bsize / DEV_BSIZE; - bp->b_blkno = sector * multiplier; - } - if ((long)bp->b_blkno == -1) { - bufdone(bp); - return (0); - } bo = node->udfmp->im_bo; bp->b_iooffset = dbtob(bp->b_blkno); BO_STRATEGY(bo, bp); @@ -1061,6 +1056,7 @@ udfmp = node->udfmp; *bp = NULL; + /* this call is only to handle UDF_INVALID_BMAP case, its results are not used for common case */ error = udf_bmap_internal(node, offset, §or, &max_size); if (error == UDF_INVALID_BMAP) { /* @@ -1078,9 +1074,8 @@ /* Adjust the size so that it is within range */ if (*size == 0 || *size > max_size) *size = max_size; - *size = min(*size, MAXBSIZE); - - if ((error = udf_readlblks(udfmp, sector, *size + (offset & udfmp->bmask), bp))) { + *size = min(*size, MAXBSIZE - blkoff(udfmp, offset)); + if ((error = bread(node->i_vnode, lblkno(udfmp, offset), (*size + blkoff(udfmp, offset) + udfmp->bmask) & ~udfmp->bmask, NOCRED, bp))) { printf("warning: udf_readlblks returned error %d\n", error); /* note: *bp may be non-NULL */ return (error); --------------070300020503030807040707-- From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 16:08:19 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B096216A419; Mon, 4 Feb 2008 16:08:19 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id 0202613C467; Mon, 4 Feb 2008 16:08:17 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from phobos.samsco.home (phobos.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.8/8.13.8) with ESMTP id m14FtEvB054627; Mon, 4 Feb 2008 08:55:15 -0700 (MST) (envelope-from scottl@samsco.org) Message-ID: <47A73562.4010309@samsco.org> Date: Mon, 04 Feb 2008 08:55:14 -0700 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.11) Gecko/20071128 SeaMonkey/1.1.7 MIME-Version: 1.0 To: Andriy Gapon References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A7339E.4070708@icyb.net.ua> In-Reply-To: <47A7339E.4070708@icyb.net.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=5.4 tests=ALL_TRUSTED autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on pooker.samsco.org X-Mailman-Approved-At: Mon, 04 Feb 2008 17:12:49 +0000 Cc: freebsd-fs@FreeBSD.org, freebsd-hackers@FreeBSD.org, Remko Lodder , Pav Lucistnik , Bruce Evans Subject: Re: fs/udf: vm pages "overlap" while reading large dir X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 16:08:19 -0000 Andriy Gapon wrote: > More on the problem with reading big directories on UDF. > > First, some sleuthing. I came to believe that the problem is caused by > some larger change in vfs/vm/buf area. It seems that now VMIO is applied > to more vnode types than before. In particular it seems that now vnodes > for devices have non-NULL v_object (or maybe this is about directory > vnodes, I am not sure). > > Also it seems that the problem should happen for any directory with size > larger than four 2048-bytes sectors (I think that any directory with > > 300 files would definitely qualify). > > After some code reading and run-time debugging, here are some facts > about udf directory reading: > 1. bread-ing is done via device vnode (as opposed to directory vnodes), > as a consequence udf_strategy is not involved in directory reading > 2. the device vnode has a non-NULL v_object, this means that VMIO is used > 3. it seems that the code assumed that non-VM buffers are used > 4. bread is done on as much data as possible, up to MAXBSIZE [and even > slightly more in some cases] (i.e. code determines directory data size > and attempts to read in everything in one go) > 5. physical sector number adjusted to DEV_BSIZE (512 bytes) sectors is > passed to bread - this is because of #1 above and peculiarities of buf > system > 6. directory reading and file reading are quite different, because the > latter does reading via file vnode > > Let's a consider a simple case of "directory reading" 5 * PAGE_SIZE (4K) > bytes starting from a physical sector N with N%2 = 0 from media with > sector size of 2K (most common CD/DVD case): > 1. we want to read 12 KB = 3 pages = 6 sectors starting from sector N, > N%2 = 0 > 2. 4*N is passed as a sector number to bread > 3. bo_bsize of the corresponding bufobj is a media sector size, i.e. 2K > 4. actual read in bread will happen from b_iooffset of 4*N*DEV_BSIZE = > N*2K, i.e. correct offset within the device > 5. for a fresh read getblk will be called > 6. getblk will set b_offset to blkno*bo_bsize=4*N*2K, i.e. 4 times the > correct byte offset within the device > 7. allocbuf will allocate pages using B_VMIO case > 8. allocbuf calculates base page index as follows: pi = b_offset/PAGE_SIZE > this means the following: > sectors N and N+1 will be bound to a page with index 4*N*2K/4K = 2*N > sectors N+2 and N+3 will be bound to the next page 2*N +1 > sectors N+4 and N+5 is tied to the next page 2*N + 2 > > Now let's consider a "directory read" of 2 sectors (1 page) starting at > physical sector N+1. > Using the same calculations as above, we see that the sector will be > tied to a page 2*(N+1) = 2*N + 2. > And this page already contains valid cached data from the previous read, > *but* it contains data from sectors N+4 and N+5 instead of N+1 and N+2. > > So, we see, that because of b_offset being 4 times the actual byte > offset, we get incorrect page indexing. Namely, sector X gets associated > with different pages depending on what sector is used as a starting > sector for bread. If bread starts at sector N, then data of sector N+1 > is associated with (second half of) page with index 2*N; but if bread > starts at sector N+1 then it gets associated with (the first half of) > page with index 2*(N+1). > > Previously, before VMIO change, data for all reads was put into separate > buffers that did not share anything between them. So the problem was > limited only to wasting memory with duplicate data, but no actual > over-runs did happen. Now we have the over-runs because the VM pages are > shared between the buffers of the same vnode. > > One obvious solution is to limit bread size to 2*PAGE_SIZE = 4 * > sector_size. In this case, as before, we would waste some memory on > duplicate data but we would avoid page overruns. If we limit bread size > even more, to 1 sector, then we would not have any duplicate data at > all. But there would still be some resource waste - each page would > correspond to one sector, so 4K page would have only 2K of valid data > and the other half in each page is unused. > > Another solution, which to me seems to be better, is to do "usual" > reading - pass a directory vnode and 2048-byte sector offset to bread. > In this case udf_strategy will get called for bklno translation, all > offsets and indexes will be correct and everything will work perfectly > as it is the case for all other filesystems. > The only overhead in this case comes from the need to handle > UDF_INVALID_BMAP case (where data is embedded into file entry). So it > means that we have to call bmap_internal to see if we have that special > case and hanlde it, and if the case is not special we would call bread > on a directory vnode which means that bmap_internal would be called > again in udf_strategy. > One optimization that can be done in this case is to create a ligher > version of bmap_internal that would merely check for the special case > and wouldn't do anything else. > > I am attaching a patch based on the ideas above. It fixes the problem > for me and doesn't seem to create any new ones :-) > About the patch: > hunk #1 fixes a copy/paste > hunk #2 should fixes strategy to not set junk b_blkno in case of > udf_bmap_internal failure and also replaces divisions/multiplications > with bit-shifts > hunk #3 really limits max size in bread to MAXBSIZE and also makes bread > to be done a directory vnode > > BTW, maybe it's a good idea to further limit size of bread() anyway. > Like always reading 1 sector. Anyway, this should make any difference > only in minority of cases and I am not sure about performance effects (I > do not expect difference in anything else other than performance). > I think you forgot to attach the patch =-) This is an excellent write-up, and I think you're on the right track. It's been nearly 8 years since I wrote most of this code, so my memory is a bit fuzzy, but I think the reason that I used the device vnode is because I was having trouble with overlapping buffers when using the directory vnode, so this was the easiest way to avoid that at the time. Since it sounds like this is no longer a viable solution, I definitely support your ideas. Scott From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 17:45:57 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D26CB16A419 for ; Mon, 4 Feb 2008 17:45:57 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.184]) by mx1.freebsd.org (Postfix) with ESMTP id 5914B13C45D for ; Mon, 4 Feb 2008 17:45:57 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: by rv-out-0910.google.com with SMTP id g13so1675556rvb.43 for ; Mon, 04 Feb 2008 09:45:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=d9uFmZKCjlVUA0kASZPqP/1R3sdJrKJtzcrQ3SBMThU=; b=rWIdmfVNTXmX0D9vFqoNmfrRP52X657GrfnqNnSiC5uCU7EQZPpN2hYU3Uuvb3XTzo6k0/CDHemo+fZ5AmQ41JisRay5ISjXFAlWt5scz75tcMpFTUnaAZjWBkKqbVUCngRL7CC1XEKAg712ToIBOofmPlUCKzZSQR/iLeCBa9E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=GbV/gKVvsExPCr1y0p8ataocMt7pepdQVG61VNAG4/flFEkBaxNoefOku1jt9wRjwLdNgsbgNRXnTUEGio0fStWSk+MEr7yY6vHCgBPH94FsozuQlciAFpmA8AhPYTpEF4D+O0gW03LgyUN3W1h0/hfrXZ3y5b6BNJrn0UG1KyM= Received: by 10.141.132.8 with SMTP id j8mr4930234rvn.7.1202147156496; Mon, 04 Feb 2008 09:45:56 -0800 (PST) Received: by 10.140.252.21 with HTTP; Mon, 4 Feb 2008 09:45:56 -0800 (PST) Message-ID: <9bbcef730802040945o20b7ee8v51fdc16e43a86c0@mail.gmail.com> Date: Mon, 4 Feb 2008 18:45:56 +0100 From: "Ivan Voras" Sender: ivoras@gmail.com To: "Alexander Leidinger" In-Reply-To: <20080204181905.5sri1eptwg0oog4g@webmail.leidinger.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4795CC13.7080601@moneybookers.com> <9bbcef730801230802n5c52832bk60c6afc47be578f4@mail.gmail.com> <47976AD4.3020203@moneybookers.com> <47A62EE8.2000700@FreeBSD.org> <47A62F1D.8070507@FreeBSD.org> <47A639B5.2020909@FreeBSD.org> <47A6EA28.6000902@moneybookers.com> <9bbcef730802040258v3fb42cd7hb70432c0da56aed3@mail.gmail.com> <47A704A5.1050001@moneybookers.com> <20080204181905.5sri1eptwg0oog4g@webmail.leidinger.net> X-Google-Sender-Auth: 5064f5583e510902 Cc: freebsd-hackers@freebsd.org, Stefan Lambrev Subject: Re: gettimeofday() in hping X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 17:45:57 -0000 On 04/02/2008, Alexander Leidinger wrote: > Ivan just change the page today. Feel free to suggest more specific things. Yes, I've added notes on syscall caching on Linux and socket buffer semantics. I've also linked to the page from Wikipedia article on FreeBSD - maybe it will help it to be noticed. From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 18:36:57 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0CF4516A473 for ; Mon, 4 Feb 2008 18:36:57 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outC.internet-mail-service.net (outC.internet-mail-service.net [216.240.47.226]) by mx1.freebsd.org (Postfix) with ESMTP id DDBED13C465 for ; Mon, 4 Feb 2008 18:36:56 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Mon, 04 Feb 2008 10:36:55 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id 315561270B2; Mon, 4 Feb 2008 10:36:55 -0800 (PST) Message-ID: <47A75B47.2040604@elischer.org> Date: Mon, 04 Feb 2008 10:36:55 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Andriy Gapon References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> In-Reply-To: <47A735A4.3060506@icyb.net.ua> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Bruce Evans , freebsd-hackers@freebsd.org, scottl@FreeBSD.org, freebsd-fs@freebsd.org, Pav Lucistnik , Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 18:36:57 -0000 Andriy Gapon wrote: > More on the problem with reading big directories on UDF. You do realise that you have now made yourself the official maintainer of the UDF file system by submitting a competent and insightful analysis of the problem? > > First, some sleuthing. I came to believe that the problem is caused by > some larger change in vfs/vm/buf area. It seems that now VMIO is applied > to more vnode types than before. In particular it seems that now vnodes > for devices have non-NULL v_object (or maybe this is about directory > vnodes, I am not sure). > > Also it seems that the problem should happen for any directory with size > larger than four 2048-bytes sectors (I think that any directory with > > 300 files would definitely qualify). > > After some code reading and run-time debugging, here are some facts > about udf directory reading: > 1. bread-ing is done via device vnode (as opposed to directory vnodes), > as a consequence udf_strategy is not involved in directory reading > 2. the device vnode has a non-NULL v_object, this means that VMIO is used > 3. it seems that the code assumed that non-VM buffers are used > 4. bread is done on as much data as possible, up to MAXBSIZE [and even > slightly more in some cases] (i.e. code determines directory data size > and attempts to read in everything in one go) > 5. physical sector number adjusted to DEV_BSIZE (512 bytes) sectors is > passed to bread - this is because of #1 above and peculiarities of buf > system > 6. directory reading and file reading are quite different, because the > latter does reading via file vnode > > Let's a consider a simple case of "directory reading" 5 * PAGE_SIZE (4K) > bytes starting from a physical sector N with N%2 = 0 from media with > sector size of 2K (most common CD/DVD case): > 1. we want to read 12 KB = 3 pages = 6 sectors starting from sector N, > N%2 = 0 > 2. 4*N is passed as a sector number to bread > 3. bo_bsize of the corresponding bufobj is a media sector size, i.e. 2K > 4. actual read in bread will happen from b_iooffset of 4*N*DEV_BSIZE = > N*2K, i.e. correct offset within the device > 5. for a fresh read getblk will be called > 6. getblk will set b_offset to blkno*bo_bsize=4*N*2K, i.e. 4 times the > correct byte offset within the device > 7. allocbuf will allocate pages using B_VMIO case > 8. allocbuf calculates base page index as follows: pi = b_offset/PAGE_SIZE > this means the following: > sectors N and N+1 will be bound to a page with index 4*N*2K/4K = 2*N > sectors N+2 and N+3 will be bound to the next page 2*N +1 > sectors N+4 and N+5 is tied to the next page 2*N + 2 > > Now let's consider a "directory read" of 2 sectors (1 page) starting at > physical sector N+1. > Using the same calculations as above, we see that the sector will be > tied to a page 2*(N+1) = 2*N + 2. > And this page already contains valid cached data from the previous read, > *but* it contains data from sectors N+4 and N+5 instead of N+1 and N+2. > > So, we see, that because of b_offset being 4 times the actual byte > offset, we get incorrect page indexing. Namely, sector X gets associated > with different pages depending on what sector is used as a starting > sector for bread. If bread starts at sector N, then data of sector N+1 > is associated with (second half of) page with index 2*N; but if bread > starts at sector N+1 then it gets associated with (the first half of) > page with index 2*(N+1). > > Previously, before VMIO change, data for all reads was put into separate > buffers that did not share anything between them. So the problem was > limited only to wasting memory with duplicate data, but no actual > over-runs did happen. Now we have the over-runs because the VM pages are > shared between the buffers of the same vnode. > > One obvious solution is to limit bread size to 2*PAGE_SIZE = 4 * > sector_size. In this case, as before, we would waste some memory on > duplicate data but we would avoid page overruns. If we limit bread size > even more, to 1 sector, then we would not have any duplicate data at > all. But there would still be some resource waste - each page would > correspond to one sector, so 4K page would have only 2K of valid data > and the other half in each page is unused. > > Another solution, which to me seems to be better, is to do "usual" > reading - pass a directory vnode and 2048-byte sector offset to bread. > In this case udf_strategy will get called for bklno translation, all > offsets and indexes will be correct and everything will work perfectly > as it is the case for all other filesystems. > The only overhead in this case comes from the need to handle > UDF_INVALID_BMAP case (where data is embedded into file entry). So it > means that we have to call bmap_internal to see if we have that special > case and hanlde it, and if the case is not special we would call bread > on a directory vnode which means that bmap_internal would be called > again in udf_strategy. > One optimization that can be done in this case is to create a ligher > version of bmap_internal that would merely check for the special case > and wouldn't do anything else. > > I am attaching a patch based on the ideas above. It fixes the problem > for me and doesn't seem to create any new ones :-) > About the patch: > hunk #1 fixes a copy/paste > hunk #2 should fixes strategy to not set junk b_blkno in case of > udf_bmap_internal failure and also replaces divisions/multiplications > with bit-shifts > hunk #3 really limits max size in bread to MAXBSIZE and also makes bread > to be done a directory vnode > > BTW, maybe it's a good idea to further limit size of bread() anyway. > Like always reading 1 sector. Anyway, this should make any difference > only in minority of cases and I am not sure about performance effects (I > do not expect difference in anything else other than performance). > > on 01/02/2008 12:27 Andriy Gapon said the following: >> on 01/02/2008 12:00 Andriy Gapon said the following: >>> ---- a different, under-debugged problem ----- >>> BTW, on some smaller directories (but still large ones) I get some very >>> strange problems with reading a directory too. It seems like some bad >>> interaction between udf and buffer cache system. I added a lot of >>> debugging prints and the problems looks like the following: >>> >>> read starting at physical sector (2048-byte one) N, size is ~20K, N%4=0 >>> bread(4 * N, some_big_size) >>> correct data is read >>> repeat the above couple dozen times >>> read starting at physical sector (2048-byte one) N+1, size is ~20K >>> bread(4 * (N+1), some_big_size) >>> data is read from physical sector N+4 (instead of N+1) >>> >>> I remember that Bruce Evance warned me that something like this could >>> happen but I couldn't understand him, because I don't understand >>> VM/buffer subsystem. I'll try to dig up the email. >>> >> Sorry for the flood - additional info: >> if I limit max read size in udf_readatoffset() to 2048 (instead of >> MAXBSIZE), then large directories can be read OK. Seems like something >> with overlapping buffers, maybe? >> >> BTW, here's how I created test environment for the described issues (in >> tcsh): >> >> mkdir /tmp/bigdir >> >> cd /tmp/bigdir >> >> set i=1 >> >> while ($i < NNNNN) >> touch file.$i >> set i=`expr $i + 1` >> end >> >> cd /tmp >> >> mkisofs -udf -o test.iso bigdir >> >> mdconfig -a -t vnode -f test.iso -S 2048 -u 0 >> >> mount_udf /dev/md0 /mnt >> >> ls -l /mnt >> > > > > ------------------------------------------------------------------------ > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 20:01:19 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ED52B16A41B for ; Mon, 4 Feb 2008 20:01:19 +0000 (UTC) (envelope-from xistence@0x58.com) Received: from mailexchange.osnn.net (1e.66.5646.static.theplanet.com [70.86.102.30]) by mx1.freebsd.org (Postfix) with SMTP id 9F54313C44B for ; Mon, 4 Feb 2008 20:01:19 +0000 (UTC) (envelope-from xistence@0x58.com) Received: (qmail 31148 invoked by uid 0); 4 Feb 2008 19:57:29 -0000 Received: from unknown (HELO ?168.158.222.24?) (xistence@0x58.com@168.158.222.24) by mailexchange.osnn.net with SMTP; 4 Feb 2008 19:57:29 -0000 Cc: FreeBSD Hackers Message-Id: From: Bert JW Regeer In-Reply-To: <200802041003.22658.jhb@freebsd.org> Content-Type: multipart/signed; boundary=Apple-Mail-1-509741225; micalg=sha1; protocol="application/pkcs7-signature" Mime-Version: 1.0 (Apple Message framework v915) Date: Mon, 4 Feb 2008 13:00:40 -0700 References: <47A4FF5F.9010604@gmail.com> <200802041003.22658.jhb@freebsd.org> X-Mailer: Apple Mail (2.915) X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: /dev/dsp disappeared after power outage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 20:01:20 -0000 --Apple-Mail-1-509741225 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit On Feb 4, 2008, at 08:03 , John Baldwin wrote: > On Saturday 02 February 2008 06:40:15 pm Aryeh M. Friedman wrote: >> I just had a power outage and when it came back /dev/dsp0.0 was >> missing from the devices. the kern module loaded fine and detected >> the card correctly (according to dmesg, sysctl and /dev/sndstat) but >> neither the above or /dev/pcm exists. After rebooting the problem >> remains. Any ideas how to fix it? > > Nothing to fix. This is how devfs device cloning works. > > -- > John Baldwin Nothing to fix? The sound card that is correctly detected by the kernel module is not being created in /dev, ONLY after he had a power outage. It is not even coming back when he reboots the machine. I don't have any suggestions, I just don't believe "Nothing to fix" is the right answer. Bert JW Regeer --Apple-Mail-1-509741225-- From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 20:14:06 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B05BC16A41A for ; Mon, 4 Feb 2008 20:14:06 +0000 (UTC) (envelope-from gregoryd.freebsd@free.fr) Received: from smtp8-g19.free.fr (smtp8-g19.free.fr [212.27.42.65]) by mx1.freebsd.org (Postfix) with ESMTP id 7251A13C4E5 for ; Mon, 4 Feb 2008 20:14:06 +0000 (UTC) (envelope-from gregoryd.freebsd@free.fr) Received: from smtp8-g19.free.fr (localhost [127.0.0.1]) by smtp8-g19.free.fr (Postfix) with ESMTP id 38313180492; Mon, 4 Feb 2008 21:14:05 +0100 (CET) Received: from imp6-g19.free.fr (imp6-g19.free.fr [212.27.42.6]) by smtp8-g19.free.fr (Postfix) with ESMTP id 577871815AF; Mon, 4 Feb 2008 21:13:56 +0100 (CET) Received: by imp6-g19.free.fr (Postfix, from userid 33) id E2BDA428C; Mon, 4 Feb 2008 21:13:49 +0100 (CET) Received: from mar44-4-88-161-154-223.fbx.proxad.net (mar44-4-88-161-154-223.fbx.proxad.net [88.161.154.223]) by imp.free.fr (IMP) with HTTP for ; Mon, 04 Feb 2008 21:13:49 +0100 Message-ID: <1202156029.47a771fda5d1c@imp.free.fr> Date: Mon, 04 Feb 2008 21:13:49 +0100 From: gregoryd.freebsd@free.fr To: d@delphij.net, Xin LI References: <47A6D272.2000406@gmail.com> <47A6D420.5050700@delphij.net> In-Reply-To: <47A6D420.5050700@delphij.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.8 X-Originating-IP: 88.161.154.223 Cc: freebsd-hackers@freebsd.org Subject: Re: SCHED_ULE trouble after ugrade 6.2-RELEASE -> 6.3-RELEASE X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 20:14:06 -0000 Quoting Xin LI : > Do NOT use ULE on 6.x. ULE has been revamped heavily on 7.0 and the > version on 6.x is old, and is known to contain some bugs. Is it particular to 6.x *SMP* systems ? I've been using 6.3-R with ULE for about a fortnight without any trouble (on a pentium4-m UP). I was drivent to switch to ULE because I suffered awful slow-downs, mouse jerkiness and all: when compiling kernel or a port, for instance, mouse would not respond with fluidity in firefox, switching tabs would take many seconds, and watching TV with vlc showed many cut-offs and fixed pictures (not related to network traffic or bandwidth limitations). The transition to ULE made them disappear (still occasional mouse jerkiness though). I suppose the enhancements/corrections of 7.x ULE won't be MFC'd ? gregory delfly From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 20:21:10 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F1DFA16A418 for ; Mon, 4 Feb 2008 20:21:10 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.delphij.net (delphij-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:2c9::2]) by mx1.freebsd.org (Postfix) with ESMTP id 99B5C13C46A for ; Mon, 4 Feb 2008 20:21:10 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.geekcn.org (unknown [202.108.54.204]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tarsier.delphij.net (Postfix) with ESMTP id 816142844D for ; Tue, 5 Feb 2008 04:21:09 +0800 (CST) Received: from localhost (unknown [202.108.54.204]) by tarsier.geekcn.org (Postfix) with ESMTP id 2A8BEEC2BFF; Tue, 5 Feb 2008 04:21:09 +0800 (CST) X-Virus-Scanned: amavisd-new at geekcn.org Received: from tarsier.geekcn.org ([202.108.54.204]) by localhost (mail.geekcn.org [202.108.54.204]) (amavisd-new, port 10024) with ESMTP id t2NfvQwX8pxn; Tue, 5 Feb 2008 04:21:04 +0800 (CST) Received: from charlie.delphij.net (71.5.7.139.ptr.us.xo.net [71.5.7.139]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTP id 9C621EC2BF9; Tue, 5 Feb 2008 04:21:03 +0800 (CST) DomainKey-Signature: a=rsa-sha1; s=default; d=delphij.net; c=nofws; q=dns; h=message-id:date:from:reply-to:organization:user-agent: mime-version:to:cc:subject:references:in-reply-to: x-enigmail-version:openpgp:content-type:content-transfer-encoding; b=baHp9OAuszTCy9sDVDFfsJAtiLeyzUyHf4O3DRCP9AfznVl2LJa02Vei3Md7F0gAE lTq8NB4h9MT1Qoy72b+eA== Message-ID: <47A773AD.1010108@delphij.net> Date: Mon, 04 Feb 2008 12:21:01 -0800 From: Xin LI Organization: The FreeBSD Project User-Agent: Thunderbird 2.0.0.9 (X11/20080122) MIME-Version: 1.0 To: gregoryd.freebsd@free.fr References: <47A6D272.2000406@gmail.com> <47A6D420.5050700@delphij.net> <1202156029.47a771fda5d1c@imp.free.fr> In-Reply-To: <1202156029.47a771fda5d1c@imp.free.fr> X-Enigmail-Version: 0.95.5 OpenPGP: id=18EDEBA0; url=http://www.delphij.net/delphij.asc Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, d@delphij.net Subject: Re: SCHED_ULE trouble after ugrade 6.2-RELEASE -> 6.3-RELEASE X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: d@delphij.net List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 20:21:11 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 gregoryd.freebsd@free.fr wrote: > Quoting Xin LI : > >> Do NOT use ULE on 6.x. ULE has been revamped heavily on 7.0 and the >> version on 6.x is old, and is known to contain some bugs. > > Is it particular to 6.x *SMP* systems ? Particular to *all* 6.x systems at the moment, possibly. > I've been using 6.3-R with ULE for about a fortnight without any trouble (on a > pentium4-m UP). > I was drivent to switch to ULE because I suffered awful slow-downs, mouse > jerkiness and all: when compiling kernel or a port, for instance, mouse would > not respond with fluidity in firefox, switching tabs would take many seconds, > and watching TV with vlc showed many cut-offs and fixed pictures (not related to > network traffic or bandwidth limitations). > The transition to ULE made them disappear (still occasional mouse jerkiness > though). > > I suppose the enhancements/corrections of 7.x ULE won't be MFC'd ? I can not speak for that, but my understanding is, no, it won't be MFC'ed. The performance enhancements on 7.x included a lot of factors, ULE is one of them, and there are also some other enhancements in the system, which could be not suitable for MFC due to ABI/KBI change. Cheers, - -- Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHp3Osi+vbBBjt66ARAud3AKCyFlIGeLTD5US/wbzWP6dwJclPZwCfRLF2 TyTZ5a+OHVX6zHEXKm+Oj0A= =F1PJ -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 20:21:52 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A774016A468; Mon, 4 Feb 2008 20:21:52 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id BFD2313C47E; Mon, 4 Feb 2008 20:21:51 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A773DF.3010409@FreeBSD.org> Date: Mon, 04 Feb 2008 21:21:51 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Stefan Lambrev References: <4795CC13.7080601@moneybookers.com> <4795FE54.9090606@moneybookers.com> <86lk6i0vzk.fsf@ds4.des.no> <479605E2.6070709@moneybookers.com> <47964356.6030602@moneybookers.com> <479647FB.3070909@FreeBSD.org> <47970EE2.5000400@moneybookers.com> <479754E6.1060101@moneybookers.com> <9bbcef730801230802n5c52832bk60c6afc47be578f4@mail.gmail.com> <47976AD4.3020203@moneybookers.com> <47A62EE8.2000700@FreeBSD.org> <47A62F1D.8070507@FreeBSD.org> <47A639B5.2020909@FreeBSD.org> <47A6EA28.6000902@moneybookers.com> <47A71AB7.6000209@moneybookers.com> In-Reply-To: <47A71AB7.6000209@moneybookers.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: gettimeofday() in hping X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 20:21:52 -0000 Stefan Lambrev wrote: > FreeBSD - ACPI > > em1 in 13.157 MB/s 13.162 MB/s 23.697 GB > out 13.150 MB/s 13.153 MB/s 17.976 GB > > FreeBSD - TSC > > em1 in 18.624 MB/s 18.832 MB/s 25.507 GB > out 17.008 MB/s 17.041 MB/s 19.681 GB > > FreeBSD - i8254 > > em1 in 6.763 MB/s 6.763 MB/s 26.005 GB > out 6.756 MB/s 6.758 MB/s 20.151 GB > > FreeBSD - dummy > > em1 in 18.705 MB/s 18.796 MB/s 27.014 GB > out 17.217 MB/s 17.225 MB/s 21.082 GB I forgot to mention I was using -a to spoof the return address so time is not wasted receiving packets (since your goal was to syn flood the server). Probably there are other (possibly also bogus) gettimeofday calls that are still present in hping when receiving packets. Kris From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 20:25:01 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A0C5016A473 for ; Mon, 4 Feb 2008 20:25:01 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id BD47D13C458; Mon, 4 Feb 2008 20:25:00 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A7749B.8070100@FreeBSD.org> Date: Mon, 04 Feb 2008 21:24:59 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: d@delphij.net References: <47A6D272.2000406@gmail.com> <47A6D420.5050700@delphij.net> <1202156029.47a771fda5d1c@imp.free.fr> <47A773AD.1010108@delphij.net> In-Reply-To: <47A773AD.1010108@delphij.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, gregoryd.freebsd@free.fr Subject: Re: SCHED_ULE trouble after ugrade 6.2-RELEASE -> 6.3-RELEASE X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 20:25:01 -0000 Xin LI wrote: > I can not speak for that, but my understanding is, no, it won't be > MFC'ed. The performance enhancements on 7.x included a lot of factors, > ULE is one of them, and there are also some other enhancements in the > system, which could be not suitable for MFC due to ABI/KBI change. Yes, there is no possibility of ULE 2.0 being merged to 6.x. Use it in 6.x if you dare, just don't complain to us if it breaks your system :-) i.e. if at any point you start experiencing problems, do not report them until you have verified that they persist with 4BSD also. Kris From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 20:35:16 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1C05516A474 for ; Mon, 4 Feb 2008 20:35:16 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from raven.customer.vol.cz (raven.customer.vol.cz [195.250.144.108]) by mx1.freebsd.org (Postfix) with ESMTP id 996AF13C4E3 for ; Mon, 4 Feb 2008 20:35:14 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from [192.168.0.23] (rb5dg130.net.upc.cz [89.176.238.130]) (authenticated bits=0) by raven.customer.vol.cz (8.14.1/8.14.1) with ESMTP id m14K7hQE000556; Mon, 4 Feb 2008 21:07:46 +0100 (CET) (envelope-from pav@FreeBSD.org) From: Pav Lucistnik To: Julian Elischer In-Reply-To: <47A75B47.2040604@elischer.org> References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> Content-Type: text/plain; charset=ISO-8859-2 Date: Mon, 04 Feb 2008 21:07:43 +0100 Message-Id: <1202155663.62432.0.camel@ikaros.oook.cz> Mime-Version: 1.0 X-Mailer: Evolution 2.12.3 FreeBSD GNOME Team Port Content-Transfer-Encoding: 8bit X-Spam-Score: -2.548 () AWL,BAYES_00 X-Scanned-By: MIMEDefang 2.63 on 195.250.144.108 X-Milter: Spamilter (Reciever: raven.customer.vol.cz; Sender-ip: 89.176.238.130; Sender-helo: [192.168.0.23]; ) Cc: Bruce Evans , freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, Andriy Gapon , freebsd-fs@FreeBSD.org, Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pav@FreeBSD.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 20:35:16 -0000 Julian Elischer pí¹e v po 04. 02. 2008 v 10:36 -0800: > Andriy Gapon wrote: > > More on the problem with reading big directories on UDF. > > You do realise that you have now made yourself the official > maintainer of the UDF file system by submitting a competent > and insightful analysis of the problem? Yay, and can you fix the sequential read performance while you're at it? Kthx! -- Pav Lucistnik Squish. Larger than the normal icky things, and twice as icky. From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 20:49:39 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4783F16A41A for ; Mon, 4 Feb 2008 20:49:39 +0000 (UTC) (envelope-from gregoryd.freebsd@free.fr) Received: from smtp8-g19.free.fr (smtp8-g19.free.fr [212.27.42.65]) by mx1.freebsd.org (Postfix) with ESMTP id 079FB13C442 for ; Mon, 4 Feb 2008 20:49:39 +0000 (UTC) (envelope-from gregoryd.freebsd@free.fr) Received: from smtp8-g19.free.fr (localhost [127.0.0.1]) by smtp8-g19.free.fr (Postfix) with ESMTP id CD83B17F515; Mon, 4 Feb 2008 21:49:37 +0100 (CET) Received: from imp6-g19.free.fr (imp6-g19.free.fr [212.27.42.6]) by smtp8-g19.free.fr (Postfix) with ESMTP id 959B317F875; Mon, 4 Feb 2008 21:49:34 +0100 (CET) Received: by imp6-g19.free.fr (Postfix, from userid 33) id 5EB4F43A1; Mon, 4 Feb 2008 21:49:27 +0100 (CET) Received: from mar44-4-88-161-154-223.fbx.proxad.net (mar44-4-88-161-154-223.fbx.proxad.net [88.161.154.223]) by imp.free.fr (IMP) with HTTP for ; Mon, 04 Feb 2008 21:49:27 +0100 Message-ID: <1202158167.47a77a572701f@imp.free.fr> Date: Mon, 04 Feb 2008 21:49:27 +0100 From: gregoryd.freebsd@free.fr To: Kris Kennaway References: <47A6D272.2000406@gmail.com> <47A6D420.5050700@delphij.net> <1202156029.47a771fda5d1c@imp.free.fr> <47A773AD.1010108@delphij.net> <47A7749B.8070100@FreeBSD.org> In-Reply-To: <47A7749B.8070100@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.8 X-Originating-IP: 88.161.154.223 Cc: freebsd-hackers@freebsd.org, d@delphij.net Subject: Re: SCHED_ULE trouble after ugrade 6.2-RELEASE -> 6.3-RELEASE X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 20:49:39 -0000 Quoting Kris Kennaway : > Yes, there is no possibility of ULE 2.0 being merged to 6.x. Use it in > 6.x if you dare, just don't complain to us if it breaks your system :-) All right, I won't :-) > i.e. if at any point you start experiencing problems, do not report them > until you have verified that they persist with 4BSD also. I can hear you. Then again, I'm in the slow process of converting people in my office to use FreeBSD instead of GNU/Linux: it's not going to be easy if 6.3 4BSD exhibits slownesses when compiling a kernel, and 6.3 ULE might prove not that stable :-\ (I've not encountered any problem until now, though, and I'm "touching wood" as my granny says) By the way, why still include ULE in 6.x if it is to be avoided ? gregory From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 21:01:44 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ADB8116A419 for ; Mon, 4 Feb 2008 21:01:44 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id CE5AA13C46E; Mon, 4 Feb 2008 21:01:43 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A77D36.6040102@FreeBSD.org> Date: Mon, 04 Feb 2008 22:01:42 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: gregoryd.freebsd@free.fr References: <47A6D272.2000406@gmail.com> <47A6D420.5050700@delphij.net> <1202156029.47a771fda5d1c@imp.free.fr> <47A773AD.1010108@delphij.net> <47A7749B.8070100@FreeBSD.org> <1202158167.47a77a572701f@imp.free.fr> In-Reply-To: <1202158167.47a77a572701f@imp.free.fr> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, d@delphij.net Subject: Re: SCHED_ULE trouble after ugrade 6.2-RELEASE -> 6.3-RELEASE X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 21:01:44 -0000 gregoryd.freebsd@free.fr wrote: > Quoting Kris Kennaway : > >> Yes, there is no possibility of ULE 2.0 being merged to 6.x. Use it in >> 6.x if you dare, just don't complain to us if it breaks your system :-) > > All right, I won't :-) > >> i.e. if at any point you start experiencing problems, do not report them >> until you have verified that they persist with 4BSD also. > > I can hear you. > Then again, I'm in the slow process of converting people in my office to use > FreeBSD instead of GNU/Linux: it's not going to be easy if 6.3 4BSD exhibits > slownesses when compiling a kernel, and 6.3 ULE might prove not that stable :-\ > (I've not encountered any problem until now, though, and I'm "touching wood" as > my granny says) > > By the way, why still include ULE in 6.x if it is to be avoided ? Typically we don't remove even experimental (even broken) code in stable branches in case it is still useful to someone despite the problems. Try 7.0 instead. Kris From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 22:14:22 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5125D16A417 for ; Mon, 4 Feb 2008 22:14:22 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from acme.spoerlein.net (acme.spoerlein.net [217.172.44.86]) by mx1.freebsd.org (Postfix) with ESMTP id C0E4013C468 for ; Mon, 4 Feb 2008 22:14:21 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from roadrunner.spoerlein.net (e180166101.adsl.alicedsl.de [85.180.166.101]) by acme.spoerlein.net (8.14.1/8.14.1) with ESMTP id m14Lr38P070237 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 4 Feb 2008 22:53:04 +0100 (CET) (envelope-from uspoerlein@gmail.com) Received: from roadrunner.spoerlein.net (localhost [127.0.0.1]) by roadrunner.spoerlein.net (8.14.2/8.14.2) with ESMTP id m14LqpRq006484 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 4 Feb 2008 22:52:51 +0100 (CET) (envelope-from uspoerlein@gmail.com) Received: (from uqs@localhost) by roadrunner.spoerlein.net (8.14.2/8.14.2/Submit) id m14LqoxP006483; Mon, 4 Feb 2008 22:52:50 +0100 (CET) (envelope-from uspoerlein@gmail.com) X-Authentication-Warning: roadrunner.spoerlein.net: uqs set sender to uspoerlein@gmail.com using -f Date: Mon, 4 Feb 2008 22:52:50 +0100 From: Ulrich Spoerlein To: Bert JW Regeer Message-ID: <20080204215250.GA1526@roadrunner.spoerlein.net> Mail-Followup-To: Bert JW Regeer , FreeBSD Hackers References: <47A4FF5F.9010604@gmail.com> <200802041003.22658.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.17 (2007-11-01) Cc: FreeBSD Hackers Subject: Re: /dev/dsp disappeared after power outage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 22:14:22 -0000 On Mon, 04.02.2008 at 13:00:40 -0700, Bert JW Regeer wrote: > On Feb 4, 2008, at 08:03 , John Baldwin wrote: > > On Saturday 02 February 2008 06:40:15 pm Aryeh M. Friedman wrote: > >> I just had a power outage and when it came back /dev/dsp0.0 was > >> missing from the devices. the kern module loaded fine and detected > >> the card correctly (according to dmesg, sysctl and /dev/sndstat) but > >> neither the above or /dev/pcm exists. After rebooting the problem > >> remains. Any ideas how to fix it? > > > > Nothing to fix. This is how devfs device cloning works. > > Nothing to fix? The sound card that is correctly detected by the kernel > module is not being created in /dev, ONLY after he had a power outage. It is > not even coming back when he reboots the machine. > > I don't have any suggestions, I just don't believe "Nothing to fix" is the > right answer. Sigh, AFAIK dev cloning works by creating the device nodes when open()ed. Using 'ls /dev/dsp*' will not open() any devices, so nothing is created. He should use 'ls /dev/dsp0 /dev/dsp0.0' and then the devices should appear. Try it for yourself, do 'ls /dev/dsp*' then 'ls /dev/dsp.8' Not that anything usefull can be done with ls(1) to get sound :) Cheers, Ulrich Spoerlein -- It is better to remain silent and be thought a fool, than to speak, and remove all doubt. From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 4 22:15:19 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65B2416A421 for ; Mon, 4 Feb 2008 22:15:19 +0000 (UTC) (envelope-from xistence@0x58.com) Received: from mailexchange.osnn.net (1e.66.5646.static.theplanet.com [70.86.102.30]) by mx1.freebsd.org (Postfix) with SMTP id 1B19513C4E3 for ; Mon, 4 Feb 2008 22:15:19 +0000 (UTC) (envelope-from xistence@0x58.com) Received: (qmail 39579 invoked by uid 0); 4 Feb 2008 22:11:30 -0000 Received: from unknown (HELO ?168.158.222.24?) (xistence@0x58.com@168.158.222.24) by mailexchange.osnn.net with SMTP; 4 Feb 2008 22:11:30 -0000 Message-Id: <954C22ED-D865-4067-9FDE-4002EB445AED@0x58.com> From: Bert JW Regeer To: Ulrich Spoerlein In-Reply-To: <20080204215250.GA1526@roadrunner.spoerlein.net> Content-Type: multipart/signed; boundary=Apple-Mail-2-517790282; micalg=sha1; protocol="application/pkcs7-signature" Mime-Version: 1.0 (Apple Message framework v915) Date: Mon, 4 Feb 2008 15:14:49 -0700 References: <47A4FF5F.9010604@gmail.com> <200802041003.22658.jhb@freebsd.org> <20080204215250.GA1526@roadrunner.spoerlein.net> X-Mailer: Apple Mail (2.915) X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: FreeBSD Hackers Subject: Re: /dev/dsp disappeared after power outage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 22:15:19 -0000 --Apple-Mail-2-517790282 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit On Feb 4, 2008, at 14:52 , Ulrich Spoerlein wrote: > On Mon, 04.02.2008 at 13:00:40 -0700, Bert JW Regeer wrote: >> On Feb 4, 2008, at 08:03 , John Baldwin wrote: >>> On Saturday 02 February 2008 06:40:15 pm Aryeh M. Friedman wrote: >>>> I just had a power outage and when it came back /dev/dsp0.0 was >>>> missing from the devices. the kern module loaded fine and >>>> detected >>>> the card correctly (according to dmesg, sysctl and /dev/sndstat) >>>> but >>>> neither the above or /dev/pcm exists. After rebooting the problem >>>> remains. Any ideas how to fix it? >>> >>> Nothing to fix. This is how devfs device cloning works. >> >> Nothing to fix? The sound card that is correctly detected by the >> kernel >> module is not being created in /dev, ONLY after he had a power >> outage. It is >> not even coming back when he reboots the machine. >> >> I don't have any suggestions, I just don't believe "Nothing to fix" >> is the >> right answer. > > Sigh, > > AFAIK dev cloning works by creating the device nodes when open()ed. > Using 'ls /dev/dsp*' will not open() any devices, so nothing is > created. > He should use 'ls /dev/dsp0 /dev/dsp0.0' and then the devices should > appear. > > Try it for yourself, do 'ls /dev/dsp*' then 'ls /dev/dsp.8' > > Not that anything usefull can be done with ls(1) to get sound :) > > > Cheers, > Ulrich Spoerlein > -- > It is better to remain silent and be thought a fool, > than to speak, and remove all doubt. I just booted up my desktop machine at home, I don't have sound enabled by default, so I loaded the module that is required. Before the module was loaded: ls -lah /dev/dsp* ls: No match. After the module was loaded (I just load snd_driver). Nothing else was executed after the module was loaded. ls -lah /dev/dsp* crw-rw-rw- 1 root wheel 0, 106 Jan 26 05:24 /dev/dsp0.0 crw-rw-rw- 1 root wheel 0, 109 Jan 26 05:24 /dev/dsp0.1 crw-rw-rw- 1 root wheel 0, 112 Jan 26 05:24 /dev/dsp0.2 crw-rw-rw- 1 root wheel 0, 115 Jan 26 05:24 /dev/dsp0.3 crw-rw-rw- 1 root wheel 0, 118 Jan 26 05:24 /dev/dsp0.4 crw-rw-rw- 1 root wheel 0, 122 Jan 26 05:24 /dev/dsp0.5 crw-rw-rw- 1 root wheel 0, 107 Jan 26 05:24 /dev/dspW0.0 crw-rw-rw- 1 root wheel 0, 110 Jan 26 05:24 /dev/dspW0.1 crw-rw-rw- 1 root wheel 0, 113 Jan 26 05:24 /dev/dspW0.2 crw-rw-rw- 1 root wheel 0, 116 Jan 26 05:24 /dev/dspW0.3 crw-rw-rw- 1 root wheel 0, 119 Jan 26 05:24 /dev/dspW0.4 crw-rw-rw- 1 root wheel 0, 123 Jan 26 05:24 /dev/dspW0.5 crw-rw-rw- 1 root wheel 0, 121 Jan 26 05:24 /dev/dspr0.4 So what gives? Bert JW Regeer --Apple-Mail-2-517790282-- From owner-freebsd-hackers@FreeBSD.ORG Tue Feb 5 09:26:21 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from miki (localhost [IPv6:::1]) by hub.freebsd.org (Postfix) with SMTP id 4F6C216A417; Tue, 5 Feb 2008 09:26:17 +0000 (UTC) (envelope-from ariff@FreeBSD.org) Date: Tue, 5 Feb 2008 17:25:46 +0800 From: Ariff Abdullah To: xistence@0x58.com, aryeh.friedman@gmail.com Message-Id: <20080205172546.7caea944.ariff@FreeBSD.org> In-Reply-To: <954C22ED-D865-4067-9FDE-4002EB445AED@0x58.com> References: <47A4FF5F.9010604@gmail.com> <200802041003.22658.jhb@freebsd.org> <20080204215250.GA1526@roadrunner.spoerlein.net> <954C22ED-D865-4067-9FDE-4002EB445AED@0x58.com> Organization: FreeBSD X-Mailer: /usr/local/lib/ruby/1.8/net/smtp.rb Mime-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="PGP-SHA1"; boundary="Signature=_Tue__5_Feb_2008_17_25_46_+0800_maaMw6fEYT/btAl5" Cc: freebsd-hackers@freebsd.org Subject: Re: /dev/dsp disappeared after power outage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 09:26:21 -0000 --Signature=_Tue__5_Feb_2008_17_25_46_+0800_maaMw6fEYT/btAl5 Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, 4 Feb 2008 15:14:49 -0700 Bert JW Regeer wrote: >=20 > On Feb 4, 2008, at 14:52 , Ulrich Spoerlein wrote: >=20 > > On Mon, 04.02.2008 at 13:00:40 -0700, Bert JW Regeer wrote: > >> On Feb 4, 2008, at 08:03 , John Baldwin wrote: > >>> On Saturday 02 February 2008 06:40:15 pm Aryeh M. Friedman > >wrote: >>> I just had a power outage and when it came back > >/dev/dsp0.0 was >>> missing from the devices. the kern module > >loaded fine and >>> detected > >>>> the card correctly (according to dmesg, sysctl and > >/dev/sndstat) >>> but > >>>> neither the above or /dev/pcm exists. After rebooting the > >problem >>> remains. Any ideas how to fix it? > >>> > >>> Nothing to fix. This is how devfs device cloning works. > >> > >> Nothing to fix? The sound card that is correctly detected by the=20 > > > >> kernel > >> module is not being created in /dev, ONLY after he had a power =20 > >> outage. It is > >> not even coming back when he reboots the machine. > >> > >> I don't have any suggestions, I just don't believe "Nothing to > >fix" > is the > >> right answer. > > > > Sigh, > > > > AFAIK dev cloning works by creating the device nodes when > > open()ed. Using 'ls /dev/dsp*' will not open() any devices, so > > nothing is created. > > He should use 'ls /dev/dsp0 /dev/dsp0.0' and then the devices > > should appear. > > > > Try it for yourself, do 'ls /dev/dsp*' then 'ls /dev/dsp.8' > > > > Not that anything usefull can be done with ls(1) to get sound :) > > > > > > Cheers, > > Ulrich Spoerlein > > --=20 > > It is better to remain silent and be thought a fool, > > than to speak, and remove all doubt. >=20 >=20 > I just booted up my desktop machine at home, I don't have sound =20 > enabled by default, so I loaded the module that is required. Before=20 >=20 > the module was loaded: >=20 > ls -lah /dev/dsp* > ls: No match. >=20 > After the module was loaded (I just load snd_driver). Nothing else > was executed after the module was loaded. >=20 > ls -lah /dev/dsp* > crw-rw-rw- 1 root wheel 0, 106 Jan 26 05:24 /dev/dsp0.0 > crw-rw-rw- 1 root wheel 0, 109 Jan 26 05:24 /dev/dsp0.1 > crw-rw-rw- 1 root wheel 0, 112 Jan 26 05:24 /dev/dsp0.2 > crw-rw-rw- 1 root wheel 0, 115 Jan 26 05:24 /dev/dsp0.3 > crw-rw-rw- 1 root wheel 0, 118 Jan 26 05:24 /dev/dsp0.4 > crw-rw-rw- 1 root wheel 0, 122 Jan 26 05:24 /dev/dsp0.5 > crw-rw-rw- 1 root wheel 0, 107 Jan 26 05:24 /dev/dspW0.0 > crw-rw-rw- 1 root wheel 0, 110 Jan 26 05:24 /dev/dspW0.1 > crw-rw-rw- 1 root wheel 0, 113 Jan 26 05:24 /dev/dspW0.2 > crw-rw-rw- 1 root wheel 0, 116 Jan 26 05:24 /dev/dspW0.3 > crw-rw-rw- 1 root wheel 0, 119 Jan 26 05:24 /dev/dspW0.4 > crw-rw-rw- 1 root wheel 0, 123 Jan 26 05:24 /dev/dspW0.5 > crw-rw-rw- 1 root wheel 0, 121 Jan 26 05:24 /dev/dspr0.4 ^^^^^^^^^^^^ Yours is obviously 6.x . Beginning with 7.x and beyond, nothing will be created by default, just like what the original poster being experiencing. Refer to sound(4). John and Ulrich are correct. Nothing to fix. >=20 > So what gives? >=20 > Bert JW Regeer >=20 -- Ariff Abdullah FreeBSD ... Recording in stereo is obviously too advanced and confusing for us idiot ***** users :P ........ --Signature=_Tue__5_Feb_2008_17_25_46_+0800_maaMw6fEYT/btAl5 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEARECAAYFAkeoK5oACgkQlr+deMUwTNpgDgCg0Hu0RRn5elugZkOaaAtvMgHH oUAAoK9yyTXJ18MLdXD1yQAUD7BIr1oj =dX4l -----END PGP SIGNATURE----- --Signature=_Tue__5_Feb_2008_17_25_46_+0800_maaMw6fEYT/btAl5-- From owner-freebsd-hackers@FreeBSD.ORG Tue Feb 5 09:59:03 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BD1A016A418 for ; Tue, 5 Feb 2008 09:59:03 +0000 (UTC) (envelope-from usleepless@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.231]) by mx1.freebsd.org (Postfix) with ESMTP id 7821013C46B for ; Tue, 5 Feb 2008 09:59:03 +0000 (UTC) (envelope-from usleepless@gmail.com) Received: by wx-out-0506.google.com with SMTP id i29so2099176wxd.7 for ; Tue, 05 Feb 2008 01:59:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=qy5llm9zCn1PSSAu/jaGd/gn1qotfQxmOSah+A0ShPw=; b=JqsC1Ixb9ZLn0/Bn+J7+QzixjY+4kmNqXNCO55Pv+6a36bqTGvYkW4XTHzdpDOle4E81UE03b1OvE1JwKsxyn+q/CMQoDKbh4Pnc/unGaTKuVT34xV5BPxP28+a7m17sd4tJ8/mIPGOp0jaUxXoYE07z7KDDyA7jbXln92fGjJ0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=k8caHNMOsZWLs8z1CxgDEmOwsMG1iTz41DUjWs1RaRE47VRQpVREG0MytrdKiyyB/G2B9Uhzj/h9k85bEq04R5/bCdOQaZu1+99rG7Kvzn1MBzcfjPeABaZ9etfiBMuGI0Q7abT9YihXO9lKOzSgrwHfnrzdDNNqOh3nxBYTiyA= Received: by 10.142.73.8 with SMTP id v8mr4096171wfa.68.1202205541801; Tue, 05 Feb 2008 01:59:01 -0800 (PST) Received: by 10.142.178.14 with HTTP; Tue, 5 Feb 2008 01:59:01 -0800 (PST) Message-ID: Date: Tue, 5 Feb 2008 10:59:01 +0100 From: usleepless@gmail.com To: "Ariff Abdullah" In-Reply-To: <20080205172546.7caea944.ariff@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <47A4FF5F.9010604@gmail.com> <200802041003.22658.jhb@freebsd.org> <20080204215250.GA1526@roadrunner.spoerlein.net> <954C22ED-D865-4067-9FDE-4002EB445AED@0x58.com> <20080205172546.7caea944.ariff@FreeBSD.org> Cc: aryeh.friedman@gmail.com, xistence@0x58.com, freebsd-hackers@freebsd.org Subject: Re: /dev/dsp disappeared after power outage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 09:59:03 -0000 Hi All, On Feb 5, 2008 10:25 AM, Ariff Abdullah wrote: > > On Mon, 4 Feb 2008 15:14:49 -0700 > Bert JW Regeer wrote: > > > > On Feb 4, 2008, at 14:52 , Ulrich Spoerlein wrote: > > > > > On Mon, 04.02.2008 at 13:00:40 -0700, Bert JW Regeer wrote: > > >> On Feb 4, 2008, at 08:03 , John Baldwin wrote: > > >>> On Saturday 02 February 2008 06:40:15 pm Aryeh M. Friedman > > >wrote: >>> I just had a power outage and when it came back > > >/dev/dsp0.0 was >>> missing from the devices. the kern module > > >loaded fine and >>> detected > > >>>> the card correctly (according to dmesg, sysctl and > > >/dev/sndstat) >>> but > > >>>> neither the above or /dev/pcm exists. After rebooting the > > >problem >>> remains. Any ideas how to fix it? > > >>> > > >>> Nothing to fix. This is how devfs device cloning works. > > >> > > >> Nothing to fix? The sound card that is correctly detected by the > > > > > >> kernel > > >> module is not being created in /dev, ONLY after he had a power > > >> outage. It is > > >> not even coming back when he reboots the machine. > > >> > > >> I don't have any suggestions, I just don't believe "Nothing to > > >fix" > is the > > >> right answer. > > > > > > Sigh, > > > > > > AFAIK dev cloning works by creating the device nodes when > > > open()ed. Using 'ls /dev/dsp*' will not open() any devices, so > > > nothing is created. > > > He should use 'ls /dev/dsp0 /dev/dsp0.0' and then the devices > > > should appear. > > > > > > Try it for yourself, do 'ls /dev/dsp*' then 'ls /dev/dsp.8' > > > > > > Not that anything usefull can be done with ls(1) to get sound :) > > > > > > > > > Cheers, > > > Ulrich Spoerlein > > > -- > > > It is better to remain silent and be thought a fool, > > > than to speak, and remove all doubt. > > > > > > I just booted up my desktop machine at home, I don't have sound > > enabled by default, so I loaded the module that is required. Before > > > > the module was loaded: > > > > ls -lah /dev/dsp* > > ls: No match. > > > > After the module was loaded (I just load snd_driver). Nothing else > > was executed after the module was loaded. > > > > ls -lah /dev/dsp* > > crw-rw-rw- 1 root wheel 0, 106 Jan 26 05:24 /dev/dsp0.0 > > crw-rw-rw- 1 root wheel 0, 109 Jan 26 05:24 /dev/dsp0.1 > > crw-rw-rw- 1 root wheel 0, 112 Jan 26 05:24 /dev/dsp0.2 > > crw-rw-rw- 1 root wheel 0, 115 Jan 26 05:24 /dev/dsp0.3 > > crw-rw-rw- 1 root wheel 0, 118 Jan 26 05:24 /dev/dsp0.4 > > crw-rw-rw- 1 root wheel 0, 122 Jan 26 05:24 /dev/dsp0.5 > > crw-rw-rw- 1 root wheel 0, 107 Jan 26 05:24 /dev/dspW0.0 > > crw-rw-rw- 1 root wheel 0, 110 Jan 26 05:24 /dev/dspW0.1 > > crw-rw-rw- 1 root wheel 0, 113 Jan 26 05:24 /dev/dspW0.2 > > crw-rw-rw- 1 root wheel 0, 116 Jan 26 05:24 /dev/dspW0.3 > > crw-rw-rw- 1 root wheel 0, 119 Jan 26 05:24 /dev/dspW0.4 > > crw-rw-rw- 1 root wheel 0, 123 Jan 26 05:24 /dev/dspW0.5 > > crw-rw-rw- 1 root wheel 0, 121 Jan 26 05:24 /dev/dspr0.4 > ^^^^^^^^^^^^ > Yours is obviously 6.x . > > Beginning with 7.x and beyond, nothing will be created by default, > just like what the original poster being experiencing. Refer to > sound(4). > > > John and Ulrich are correct. Nothing to fix. although it might be expected behaviour, i am struggling with this as well: if i don't "touch" the dsp device upon boot, ekiga doesn't detect the proper soundcard ( my builtin one ). it selects the USB speakers, which have created a dsp(1)-device. so i ended up writing a startup script which touches dsp0. i find this a strange situation. regards, usleep From owner-freebsd-hackers@FreeBSD.ORG Tue Feb 5 14:40:17 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 43E4F16A498; Tue, 5 Feb 2008 14:40:17 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id A74CC13C4E8; Tue, 5 Feb 2008 14:40:16 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id C63B443DC32; Tue, 5 Feb 2008 16:40:15 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id F77-1TpguA4B; Tue, 5 Feb 2008 16:40:15 +0200 (EET) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 46AF743D33F; Tue, 5 Feb 2008 16:40:14 +0200 (EET) Message-ID: <47A8754C.5010607@icyb.net.ua> Date: Tue, 05 Feb 2008 16:40:12 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20080123) MIME-Version: 1.0 To: pav@FreeBSD.org References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> <1202155663.62432.0.camel@ikaros.oook.cz> In-Reply-To: <1202155663.62432.0.camel@ikaros.oook.cz> Content-Type: multipart/mixed; boundary="------------060803080903090508090302" Cc: Bruce Evans , freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, freebsd-fs@FreeBSD.org, Julian Elischer , Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 14:40:17 -0000 This is a multi-part message in MIME format. --------------060803080903090508090302 Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: 8bit on 04/02/2008 22:07 Pav Lucistnik said the following: > Julian Elischer pí¹e v po 04. 02. 2008 v 10:36 -0800: >> Andriy Gapon wrote: >>> More on the problem with reading big directories on UDF. >> You do realise that you have now made yourself the official >> maintainer of the UDF file system by submitting a competent >> and insightful analysis of the problem? > > Yay, and can you fix the sequential read performance while you're at it? > Kthx! > Pav, this was almost trivial :-) See the attached patch, first hunk is just for consistency. The code was borrowed from cd9660, only field/variable names are adjusted. But there is another issue that I also mentioned in the email about directory reading. It is UDF_INVALID_BMAP case of udf_bmap_internal, i.e. the case when file data is embedded into a file entry. This is a special case that needs to be handled differently. udf_readatoffset() handles it, but the latest udf_read code doesn't. I have a real UDF filesystem where this type of allocation is used for small files and those files can not be read. This is described in Part 4, section 14.6.8 of ECMA-167. -- Andriy Gapon --------------060803080903090508090302 Content-Type: text/x-patch; name="udf_ra.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="udf_ra.patch" --- udf_vnops.c.orig 2008-01-29 23:50:49.000000000 +0200 +++ udf_vnops.c 2008-02-05 01:30:23.000000000 +0200 @@ -851,7 +846,7 @@ udf_bmap(struct vop_bmap_args *a) if (a->a_runb) *a->a_runb = 0; - error = udf_bmap_internal(node, a->a_bn * node->udfmp->bsize, &lsector, + error = udf_bmap_internal(node, a->a_bn << node->udfmp->bshift, &lsector, &max_size); if (error) return (error); @@ -859,9 +854,27 @@ udf_bmap(struct vop_bmap_args *a) /* Translate logical to physical sector number */ *a->a_bnp = lsector << (node->udfmp->bshift - DEV_BSHIFT); - /* Punt on read-ahead for now */ - if (a->a_runp) - *a->a_runp = 0; + /* + * Determine maximum number of readahead blocks following the + * requested block. + */ + if (a->a_runp) { + off_t fsize; + int nblk; + + fsize = le64toh(node->fentry->inf_len); + nblk = (fsize >> node->udfmp->bshift) - (a->a_bn + 1); + if (nblk <= 0) + *a->a_runp = 0; + else if (nblk >= (MAXBSIZE >> node->udfmp->bshift)) + *a->a_runp = (MAXBSIZE >> node->udfmp->bshift) - 1; + else + *a->a_runp = nblk; + } + + if (a->a_runb) { + *a->a_runb = 0; + } return (0); } --- udf_vfsops.c.orig 2007-03-13 03:50:24.000000000 +0200 +++ udf_vfsops.c 2008-02-05 01:29:10.000000000 +0200 @@ -330,6 +330,11 @@ udf_mountfs(struct vnode *devvp, struct bo = &devvp->v_bufobj; + if (devvp->v_rdev->si_iosize_max != 0) + mp->mnt_iosize_max = devvp->v_rdev->si_iosize_max; + if (mp->mnt_iosize_max > MAXPHYS) + mp->mnt_iosize_max = MAXPHYS; + /* XXX: should be M_WAITOK */ MALLOC(udfmp, struct udf_mnt *, sizeof(struct udf_mnt), M_UDFMOUNT, M_NOWAIT | M_ZERO); --------------060803080903090508090302-- From owner-freebsd-hackers@FreeBSD.ORG Tue Feb 5 14:52:45 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 021AE16A4CC; Tue, 5 Feb 2008 14:52:45 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id 9DEF213C459; Tue, 5 Feb 2008 14:52:44 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id 7FA8243F66F; Tue, 5 Feb 2008 16:52:43 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5ZvYvl+ATS2F; Tue, 5 Feb 2008 16:52:43 +0200 (EET) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 7A32C43F569; Tue, 5 Feb 2008 16:52:42 +0200 (EET) Message-ID: <47A87838.6060300@icyb.net.ua> Date: Tue, 05 Feb 2008 16:52:40 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20080123) MIME-Version: 1.0 To: Julian Elischer References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> In-Reply-To: <47A75B47.2040604@elischer.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Bruce Evans , freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, freebsd-fs@FreeBSD.org, Pav Lucistnik , Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 14:52:45 -0000 on 04/02/2008 20:36 Julian Elischer said the following: > Andriy Gapon wrote: >> More on the problem with reading big directories on UDF. > > You do realise that you have now made yourself the official > maintainer of the UDF file system by submitting a competent > and insightful analysis of the problem? I feel like I have to reply to this :-) First of all, thank you. But unfortunately my knowledge of core kernel is quite limited, I pick up some bits here and there by reading the code, but this is too little and sometimes my deductions are wrong. Another thing is very limited time. I do like fixing code, but I rarely have time and when I have it must be something very interesting and important to me. This time it is interesting :-) Because I actually use UDF quite a lot, I use DVD-RAM media for various archival and data transfer purposes, and sometimes there is some unusual case. I also try to keep up with FreeBSD development as much as possible, so sometimes there are new issues after upgrades. So, thank you again, but please think of me as of occasional patch submitter. -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Tue Feb 5 18:16:23 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23C2516A469 for ; Tue, 5 Feb 2008 18:16:23 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from raven.customer.vol.cz (raven.customer.vol.cz [195.250.144.108]) by mx1.freebsd.org (Postfix) with ESMTP id 70DF613C45B for ; Tue, 5 Feb 2008 18:16:22 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from [192.168.0.23] (rb5dg130.net.upc.cz [89.176.238.130]) (authenticated bits=0) by raven.customer.vol.cz (8.14.1/8.14.1) with ESMTP id m15IG9Qh021467; Tue, 5 Feb 2008 19:16:11 +0100 (CET) (envelope-from pav@FreeBSD.org) From: Pav Lucistnik To: Andriy Gapon In-Reply-To: <47A8754C.5010607@icyb.net.ua> References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> <1202155663.62432.0.camel@ikaros.oook.cz> <47A8754C.5010607@icyb.net.ua> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-a0as8OlXkhwaCBWRWkrr" Date: Tue, 05 Feb 2008 19:16:08 +0100 Message-Id: <1202235368.68281.12.camel@ikaros.oook.cz> Mime-Version: 1.0 X-Mailer: Evolution 2.12.3 FreeBSD GNOME Team Port X-Spam-Score: -2.57 () AWL,BAYES_00 X-Scanned-By: MIMEDefang 2.63 on 195.250.144.108 X-Milter: Spamilter (Reciever: raven.customer.vol.cz; Sender-ip: 89.176.238.130; Sender-helo: [192.168.0.23]; ) Cc: Bruce Evans , freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, freebsd-fs@FreeBSD.org, Julian Elischer , Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pav@FreeBSD.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 18:16:23 -0000 --=-a0as8OlXkhwaCBWRWkrr Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: quoted-printable Andriy Gapon p=ED=B9e v =FAt 05. 02. 2008 v 16:40 +0200: > > Yay, and can you fix the sequential read performance while you're at it= ? > > Kthx! > this was almost trivial :-) > See the attached patch, first hunk is just for consistency. > The code was borrowed from cd9660, only field/variable names are adjusted= . Omg looks good. Can't wait for it to bubble throught to 6-STABLE :) Thanks million for working on this. --=20 Pav Lucistnik What is the airspeed velocity of an unladen swallow? --=-a0as8OlXkhwaCBWRWkrr Content-Type: application/pgp-signature; name=signature.asc Content-Description: Toto je =?UTF-8?Q?digit=C3=A1ln=C4=9B?= =?ISO-8859-1?Q?_podepsan=E1?= =?UTF-8?Q?_=C4=8D=C3=A1st?= =?ISO-8859-1?Q?_zpr=E1vy?= -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEABECAAYFAkeop+UACgkQntdYP8FOsoJTEwCgnFMMuMEb3JCazBXQyIJvILD+ uJoAn0tWakjK81PqRferUhg2Z7fgOtsb =ysEA -----END PGP SIGNATURE----- --=-a0as8OlXkhwaCBWRWkrr-- From owner-freebsd-hackers@FreeBSD.ORG Tue Feb 5 20:43:55 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2643016A419; Tue, 5 Feb 2008 20:43:55 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id AF62813C447; Tue, 5 Feb 2008 20:43:54 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from phobos.samsco.home (phobos.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.8/8.13.8) with ESMTP id m15KhlRQ067518; Tue, 5 Feb 2008 13:43:48 -0700 (MST) (envelope-from scottl@samsco.org) Message-ID: <47A8CA83.1000405@samsco.org> Date: Tue, 05 Feb 2008 13:43:47 -0700 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.11) Gecko/20071128 SeaMonkey/1.1.7 MIME-Version: 1.0 To: Andriy Gapon References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> <1202155663.62432.0.camel@ikaros.oook.cz> <47A8754C.5010607@icyb.net.ua> In-Reply-To: <47A8754C.5010607@icyb.net.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.4 required=5.4 tests=ALL_TRUSTED autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on pooker.samsco.org X-Mailman-Approved-At: Tue, 05 Feb 2008 20:48:46 +0000 Cc: Bruce Evans , freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, freebsd-fs@FreeBSD.org, pav@FreeBSD.org, Julian Elischer , Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 20:43:55 -0000 Andriy Gapon wrote: > on 04/02/2008 22:07 Pav Lucistnik said the following: >> Julian Elischer píše v po 04. 02. 2008 v 10:36 -0800: >>> Andriy Gapon wrote: >>>> More on the problem with reading big directories on UDF. >>> You do realise that you have now made yourself the official >>> maintainer of the UDF file system by submitting a competent >>> and insightful analysis of the problem? >> Yay, and can you fix the sequential read performance while you're at it? >> Kthx! >> > > Pav, > > this was almost trivial :-) > See the attached patch, first hunk is just for consistency. > The code was borrowed from cd9660, only field/variable names are adjusted. > Your patch looks reasonable. Btw, for the same reason that read-ahead makes file reading much faster, I would not change directory reading to be 1 sector at a time (unless you also do read-ahead for it). > But there is another issue that I also mentioned in the email about > directory reading. It is UDF_INVALID_BMAP case of udf_bmap_internal, > i.e. the case when file data is embedded into a file entry. > This is a special case that needs to be handled differently. > udf_readatoffset() handles it, but the latest udf_read code doesn't. > I have a real UDF filesystem where this type of allocation is used for > small files and those files can not be read. Oh, so directory data can also follow this convention? Blah. Feel free to fix that too if you want =-) Scott From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 6 06:15:00 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A597E16A41A for ; Wed, 6 Feb 2008 06:15:00 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outF.internet-mail-service.net (outF.internet-mail-service.net [216.240.47.229]) by mx1.freebsd.org (Postfix) with ESMTP id A0A6213C461 for ; Wed, 6 Feb 2008 06:15:00 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Tue, 05 Feb 2008 22:15:00 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id 48E721270F8; Tue, 5 Feb 2008 22:14:59 -0800 (PST) Message-ID: <47A95065.3060703@elischer.org> Date: Tue, 05 Feb 2008 22:15:01 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Andriy Gapon References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> <1202155663.62432.0.camel@ikaros.oook.cz> <47A8754C.5010607@icyb.net.ua> In-Reply-To: <47A8754C.5010607@icyb.net.ua> Content-Type: text/plain; charset=ISO-8859-2; format=flowed Content-Transfer-Encoding: 8bit Cc: Bruce Evans , freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, freebsd-fs@FreeBSD.org, pav@FreeBSD.org, Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 06:15:00 -0000 Andriy Gapon wrote: > on 04/02/2008 22:07 Pav Lucistnik said the following: >> Julian Elischer pí¹e v po 04. 02. 2008 v 10:36 -0800: >>> Andriy Gapon wrote: >>>> More on the problem with reading big directories on UDF. >>> You do realise that you have now made yourself the official >>> maintainer of the UDF file system by submitting a competent >>> and insightful analysis of the problem? >> Yay, and can you fix the sequential read performance while you're at it? >> Kthx! >> > > Pav, > > this was almost trivial :-) Ok that settles it.. it's yours. > See the attached patch, first hunk is just for consistency. > The code was borrowed from cd9660, only field/variable names are adjusted. > > But there is another issue that I also mentioned in the email about > directory reading. It is UDF_INVALID_BMAP case of udf_bmap_internal, > i.e. the case when file data is embedded into a file entry. > This is a special case that needs to be handled differently. > udf_readatoffset() handles it, but the latest udf_read code doesn't. > I have a real UDF filesystem where this type of allocation is used for > small files and those files can not be read. > > This is described in Part 4, section 14.6.8 of ECMA-167. > > From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 6 14:34:31 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0908016A418 for ; Wed, 6 Feb 2008 14:34:31 +0000 (UTC) (envelope-from ws@au.dyndns.ws) Received: from ipmail05.adl2.internode.on.net (ipmail05.adl2.internode.on.net [203.16.214.145]) by mx1.freebsd.org (Postfix) with ESMTP id 77CE513C4D5 for ; Wed, 6 Feb 2008 14:34:30 +0000 (UTC) (envelope-from ws@au.dyndns.ws) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgAAAINSqUeWZWdv/2dsb2JhbAAIkX2bcg X-IronPort-AV: E=Sophos;i="4.25,312,1199626200"; d="scan'208";a="49383878" Received: from ppp103-111.static.internode.on.net (HELO [192.168.1.131]) ([150.101.103.111]) by ipmail05.adl2.internode.on.net with ESMTP; 07 Feb 2008 01:04:30 +1030 From: Wayne Sierke To: hackers Content-Type: text/plain Date: Thu, 07 Feb 2008 01:04:27 +1030 Message-Id: <1202308467.5350.220.camel@predator-ii.buffyverse> Mime-Version: 1.0 X-Mailer: Evolution 2.12.3 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Subject: Any interest in unused hardware being made available for remote developer use? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 14:34:31 -0000 I have a Compaq ML370 Proliant server here which is likely to remain unused for the foreseeable future. Some of its key specs are: - Dual 800MHz PIII/133MHz/256MB CPU - 1.25GB ECC Memory - 2 x 9G 10k SCSI-UW2 - 1 x 18G 10k SCSI-UW2 - 2 x Intel EtherExpress PRO/100 Ethernet I've been contemplating the idea of offering it for remote access, to those with a legitimate use for it in a FreeBSD-related development endeavour. There are numerous aspects about this that warrant careful consideration and for which I'll be seeking comment if it comes to it, however the first question is: Is there any interest, or likely to be? Thanks, Wayne From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 6 16:29:27 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89F5616A418; Wed, 6 Feb 2008 16:29:27 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id D8D9013C44B; Wed, 6 Feb 2008 16:29:26 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id AA36843C906; Wed, 6 Feb 2008 18:29:24 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pJDvGNN+uRPR; Wed, 6 Feb 2008 18:29:24 +0200 (EET) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id BA5EB43C20E; Wed, 6 Feb 2008 18:29:23 +0200 (EET) Message-ID: <47A9E062.3040409@icyb.net.ua> Date: Wed, 06 Feb 2008 18:29:22 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20080123) MIME-Version: 1.0 To: Scott Long , Poul-Henning Kamp References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A7339E.4070708@icyb.net.ua> <47A73562.4010309@samsco.org> In-Reply-To: <47A73562.4010309@samsco.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, freebsd-hackers@FreeBSD.org, Remko Lodder , Pav Lucistnik , Bruce Evans Subject: Re: fs/udf: vm pages "overlap" while reading large dir X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 16:29:27 -0000 on 04/02/2008 17:55 Scott Long said the following: > Andriy Gapon wrote: [snip] >> After some code reading and run-time debugging, here are some facts >> about udf directory reading: >> 1. bread-ing is done via device vnode (as opposed to directory vnodes), >> as a consequence udf_strategy is not involved in directory reading >> 2. the device vnode has a non-NULL v_object, this means that VMIO is used [strike out incorrect guess] >> 4. bread is done on as much data as possible, up to MAXBSIZE [and even >> slightly more in some cases] (i.e. code determines directory data size >> and attempts to read in everything in one go) >> 5. physical sector number adjusted to DEV_BSIZE (512 bytes) sectors is >> passed to bread - this is because of #1 above and peculiarities of buf >> system >> 6. directory reading and file reading are quite different, because the >> latter does reading via file vnode >> >> Let's a consider a simple case of "directory reading" 5 * PAGE_SIZE (4K) >> bytes starting from a physical sector N with N%2 = 0 from media with >> sector size of 2K (most common CD/DVD case): >> 1. we want to read 12 KB = 3 pages = 6 sectors starting from sector N, >> N%2 = 0 >> 2. 4*N is passed as a sector number to bread >> 3. bo_bsize of the corresponding bufobj is a media sector size, i.e. 2K >> 4. actual read in bread will happen from b_iooffset of 4*N*DEV_BSIZE = >> N*2K, i.e. correct offset within the device >> 5. for a fresh read getblk will be called >> 6. getblk will set b_offset to blkno*bo_bsize=4*N*2K, i.e. 4 times the >> correct byte offset within the device >> 7. allocbuf will allocate pages using B_VMIO case >> 8. allocbuf calculates base page index as follows: pi = b_offset/PAGE_SIZE >> this means the following: >> sectors N and N+1 will be bound to a page with index 4*N*2K/4K = 2*N >> sectors N+2 and N+3 will be bound to the next page 2*N +1 >> sectors N+4 and N+5 is tied to the next page 2*N + 2 [insert] note that bstrategy will get correct params as b_iooffset is set to blkno*DEV_BSIZE, i.e. correct byte offset. >> >> Now let's consider a "directory read" of 2 sectors (1 page) starting at >> physical sector N+1. >> Using the same calculations as above, we see that the sector will be >> tied to a page 2*(N+1) = 2*N + 2. >> And this page already contains valid cached data from the previous read, >> *but* it contains data from sectors N+4 and N+5 instead of N+1 and N+2. >> >> So, we see, that because of b_offset being 4 times the actual byte >> offset, we get incorrect page indexing. Namely, sector X gets associated >> with different pages depending on what sector is used as a starting >> sector for bread. If bread starts at sector N, then data of sector N+1 >> is associated with (second half of) page with index 2*N; but if bread >> starts at sector N+1 then it gets associated with (the first half of) >> page with index 2*(N+1). [snip] > I think you forgot to attach the patch =-) This is an excellent > write-up, and I think you're on the right track. It's been nearly 8 > years since I wrote most of this code, so my memory is a bit fuzzy, but > I think the reason that I used the device vnode is because I was having > trouble with overlapping buffers when using the directory vnode, so this > was the easiest way to avoid that at the time. Since it sounds like > this is no longer a viable solution, I definitely support your ideas. [additional analysis of the problem, with new facts that might give rise to a different solution] Small summary of the above long description. For directory reading fs/udf performs bread() on a (underlying) device vnode. It passes block number as if block size was 512 bytes (i.e. byte_offset_within_dev/512). On the other hand vm page index calculation code uses the following formula: (blkno*bo_bsize)/PAGE_SIZE. For CD/DVD devices bo_bsize is set to 2048. This results in page index overlap when reading sufficiently many blocks from adjacent starting blocks. That is, it seems that in "normal" cases page index gets calculated as byte_offset/PAGE_SIZE, but in this case page index gets to be 4*byte_offset/PAGE_SIZE. More details are in the quoted above text. With a lot of help from Bruce Evance I found this commit which seems to have made a big difference: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/vfs_bio.c.diff?r1=1.458;r2=1.459;f=h Before this change page index for device vnodes was always (blkno*512)/PAGE_SIZE. This is because of the vn_isdisk() case. udf_mountfs device vnode is passed to g_vfs_open() (in geom_vfs.c) and the latter has the following line of code: bo->bo_bsize = pp->sectorsize; Where pp is geom provider for the device in question. I am not sure if the mentioned above vfs_bio.c change broke something else in reading from device vnodes or if it fixed something for that. I am also not sure what would be the consequences of setting bo_bsize to 512 for vnodes of "disk" devices. It would most probably fix current code in UDF, but I am not sure if it might break anything else. Just wanted to let the more knowledgeable people know and make a decision. P.S. bo_bsize seems to be referenced only in a handful of files and in most of them it seems that it is related to "file" vnodes (as opposed to "device" vnodes). -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 6 16:34:30 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42BB916A419; Wed, 6 Feb 2008 16:34:30 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id CE1AE13C4E1; Wed, 6 Feb 2008 16:34:29 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id F2A4043F8F2; Wed, 6 Feb 2008 18:34:28 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pVY5er24lJp3; Wed, 6 Feb 2008 18:34:28 +0200 (EET) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id CC0CE43CA85; Wed, 6 Feb 2008 18:34:26 +0200 (EET) Message-ID: <47A9E191.8000607@icyb.net.ua> Date: Wed, 06 Feb 2008 18:34:25 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20080123) MIME-Version: 1.0 To: pav@FreeBSD.org References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> <1202155663.62432.0.camel@ikaros.oook.cz> <47A8754C.5010607@icyb.net.ua> <1202235368.68281.12.camel@ikaros.oook.cz> In-Reply-To: <1202235368.68281.12.camel@ikaros.oook.cz> Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: 8bit Cc: Bruce Evans , freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, freebsd-fs@FreeBSD.org, Julian Elischer , Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 16:34:30 -0000 on 05/02/2008 20:16 Pav Lucistnik said the following: > Andriy Gapon pí¹e v út 05. 02. 2008 v 16:40 +0200: > >>> Yay, and can you fix the sequential read performance while you're at it? >>> Kthx! > >> this was almost trivial :-) >> See the attached patch, first hunk is just for consistency. >> The code was borrowed from cd9660, only field/variable names are adjusted. > > Omg looks good. Can't wait for it to bubble throught to 6-STABLE :) > > Thanks million for working on this. > Actually the patch is not entirely correct. max_size returned from udf_bmap_internal should be used to calculate number of continuous sectors for read-ahead (as opposed to file size in the patch). -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 6 16:36:02 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ABC7D16A477; Wed, 6 Feb 2008 16:36:02 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id 4366D13C442; Wed, 6 Feb 2008 16:36:02 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id 6E34643E003; Wed, 6 Feb 2008 18:36:01 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2uP3KJXq51GZ; Wed, 6 Feb 2008 18:36:01 +0200 (EET) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 9BACA43C20E; Wed, 6 Feb 2008 18:36:00 +0200 (EET) Message-ID: <47A9E1EF.5020507@icyb.net.ua> Date: Wed, 06 Feb 2008 18:35:59 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20080123) MIME-Version: 1.0 To: Scott Long References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> <1202155663.62432.0.camel@ikaros.oook.cz> <47A8754C.5010607@icyb.net.ua> <47A8CA83.1000405@samsco.org> In-Reply-To: <47A8CA83.1000405@samsco.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, freebsd-fs@FreeBSD.org, pav@FreeBSD.org, Julian Elischer , Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 16:36:02 -0000 on 05/02/2008 22:43 Scott Long said the following: > Andriy Gapon wrote: > >> But there is another issue that I also mentioned in the email about >> directory reading. It is UDF_INVALID_BMAP case of udf_bmap_internal, >> i.e. the case when file data is embedded into a file entry. >> This is a special case that needs to be handled differently. >> udf_readatoffset() handles it, but the latest udf_read code doesn't. >> I have a real UDF filesystem where this type of allocation is used for >> small files and those files can not be read. > > Oh, so directory data can also follow this convention? Blah. Feel free > to fix that too if you want =-) Yes, this allocation type can be applied to both files and directories. Directories are already handled because of readatoffset() but udf_read needs to be made aware of this special case. -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 6 20:26:30 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 43C7116A419 for ; Wed, 6 Feb 2008 20:26:30 +0000 (UTC) (envelope-from gahr@gahr.ch) Received: from cpanel03.rubas-s03.net (cpanel03.rubas-s03.net [195.182.222.73]) by mx1.freebsd.org (Postfix) with ESMTP id D142913C461 for ; Wed, 6 Feb 2008 20:26:29 +0000 (UTC) (envelope-from gahr@gahr.ch) Received: from 80-218-191-236.dclient.hispeed.ch ([80.218.191.236] helo=gahrtop.localhost) by cpanel03.rubas-s03.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.68) (envelope-from ) id 1JMqqj-0004WK-4e for freebsd-hackers@freebsd.org; Wed, 06 Feb 2008 21:26:29 +0100 Message-ID: <47AA17E7.8050502@gahr.ch> Date: Wed, 06 Feb 2008 21:26:15 +0100 From: Pietro Cerutti User-Agent: Thunderbird 2.0.0.9 (X11/20071121) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org X-Enigmail-Version: 0.95.5 OpenPGP: id=9571F78E; url=http://www.gahr.ch/pgp Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="------------enig174309CD43744A713FE3671E" X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cpanel03.rubas-s03.net X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - gahr.ch X-Source: X-Source-Args: X-Source-Dir: Subject: C99 & Posix X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 20:26:30 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig174309CD43744A713FE3671E Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi hackers, is there any other source of information about the status of Posix and C99 conformity in FreeBSD than the two (outdated) pages below? http://www.freebsd.org/projects/c99/index.html http://people.freebsd.org/~schweikh/posix-utilities.html I would like to do something in this field, but I have no clue where to start from.. Thanks --=20 Pietro Cerutti PGP Public Key: http://gahr.ch/pgp --------------enig174309CD43744A713FE3671E Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHqhfrwMJqmJVx944RCvGsAJ9nsYbF3A4ylia3PQTVew7q4X6lowCfQ+mn jptPe5GB55heqWkDmwIg8lo= =N2X4 -----END PGP SIGNATURE----- --------------enig174309CD43744A713FE3671E-- From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 6 23:33:28 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1550B16A476; Wed, 6 Feb 2008 23:33:28 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id 5EA6C13C459; Wed, 6 Feb 2008 23:33:27 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id 66E62744013; Thu, 7 Feb 2008 01:33:25 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10027) with ESMTP id gHzvoAmT8B76; Thu, 7 Feb 2008 01:33:25 +0200 (EET) Received: from [10.74.70.239] (unknown [193.138.145.53]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 801A6744012; Thu, 7 Feb 2008 01:33:23 +0200 (EET) Message-ID: <47AA43B9.1040608@icyb.net.ua> Date: Thu, 07 Feb 2008 01:33:13 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20071208) MIME-Version: 1.0 To: pav@FreeBSD.org, scottl@FreeBSD.org References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> <1202155663.62432.0.camel@ikaros.oook.cz> <47A8754C.5010607@icyb.net.ua> <1202235368.68281.12.camel@ikaros.oook.cz> <47A9E191.8000607@icyb.net.ua> In-Reply-To: <47A9E191.8000607@icyb.net.ua> Content-Type: multipart/mixed; boundary="------------090706050706020808090509" Cc: freebsd-fs@FreeBSD.org, freebsd-hackers@FreeBSD.org Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 23:33:28 -0000 This is a multi-part message in MIME format. --------------090706050706020808090509 Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: 7bit on 06/02/2008 18:34 Andriy Gapon said the following: > > Actually the patch is not entirely correct. max_size returned from > udf_bmap_internal should be used to calculate number of continuous > sectors for read-ahead (as opposed to file size in the patch). > Attached is an updated patch. The most prominent changes from the previous version: 1. udf_read can handle files with data embedded into fentry (sysutils/udfclient can produce such things for small files). 2. above mentioned files can now be mmap-ed, so that you can not only cat them, but also cp them; btw, I believe that cp(1) should have logic to try to fallback to read if mmap() fails - remember kern/92040 ? :-) 3. udf_bmap uses max_size[*] returned by udf_bmap_internal to calculate a number of contiguous blocks for read-ahead; [*] - this is a number of contiguous bytes from a given offset within a file. Things that stay the same: 1. I still use bread() via directory vnode; I think it's better even if reading via device vnode would work correctly. 2. I still try to read as much data as possible in directory bread(), I think this is a sort of an ad-hoc read-ahead without all the heuristics and logic that cluster reading does; I think that this should be ok because we do not have random reading of directory, it's always sequential - either to read-in the whole directory or linearly search through it until wanted entry is found. Detailed description of the patch. udf_vnops.c: Hunk1 - this is "just in case" patch, reject files with unsupported strategy right away instead of deferring failure to udf_bmap_internal. Hunk2 - fix typo in existing macro, add new macro to check if a file has data embedded into fentry ("unusual file"). Hunk3 - for an unusual file just uiomove data from fentry. Hunk4 - cosmetic changes plus a fix for udf_bmap_internal errors not actually being honored; also added an additional printf, because udf_strategy should never really be called for unusual files. Hunk5 - return EOPNOTSUPP for unusual files, this is correct because we can not correctly bmap them and also this enables correct handling of this files in vm code (vnode_pager); also code for read-ahead calculation borrowed from cd9660. Hunk6- explain function of udf_bmap_internal call in this place. Hunk7 - some cosmetics; prevent size passed to bread from being greater than MAXBSIZE; do bread via directory vnode rather than device vnode (udf_readlblks was a macro-wrapper around bread). udf_vfsops.c: Hunk1 - this is borrowed from cd9660, apparently this data is needed for correct cluster reading. Couple of words about testing. udf in 7.0-RC1 plus this patch correctly handled everything I could throw at it - huge directories, unusual files, reading, mmaping. Using udfclientfs I wrote a directory on DVD-RAM UDF disk that contained 2G of files from ports distfiles. The files varied in size from about 100 of bytes to hundreds of megabytes. I watched systat -vmstat while copying this directory. With unpatched code some files would not be copied (small "unusual files"), KB/t value stayed at 2K (meaning reads always were sector sized), MB/s was about 1. With the patch all files were copied successfully and correctly (md5 verified), KB/t varied from 2K to 64K (apparently depending on size of currently copied file), MB/s varied from 1 to 4 which is not bad for these DVD-RAM disk and drive. If somebody asks I can produce a UDF image that would contain various test cases: large directory, unusual files, etc. Or you can give a try to udfclient/udfclientfs from sysutils/udfclient port. I hope this patch will be useful. -- Andriy Gapon --------------090706050706020808090509 Content-Type: text/x-patch; name="udf_latest2.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="udf_latest2.diff" --- udf_vnops.c.orig 2008-01-29 23:50:49.000000000 +0200 +++ udf_vnops.c 2008-02-07 00:12:34.000000000 +0200 @@ -168,6 +168,16 @@ udf_open(struct vop_open_args *ap) { struct udf_node *np = VTON(ap->a_vp); off_t fsize; + /* + * We currently do not support any other strategy type, so + * udf_bmap_internal, udf_bmap, udf_strategy would fail. + * I.e. we can not access content of this file anyway. + */ + if (le16toh(np->fentry->icbtag.strat_type) != 4) { + printf("Unsupported strategy type %d\n", le16toh(np->fentry->icbtag.strat_type)); + return ENODEV; + } + fsize = le64toh(np->fentry->inf_len); vnode_create_vobject(ap->a_vp, fsize, ap->a_td); return 0; @@ -340,7 +350,9 @@ udf_pathconf(struct vop_pathconf_args *a #define lblkno(udfmp, loc) ((loc) >> (udfmp)->bshift) #define blkoff(udfmp, loc) ((loc) & (udfmp)->bmask) -#define lblktosize(imp, blk) ((blk) << (udfmp)->bshift) +#define lblktosize(udfmp, blk) ((blk) << (udfmp)->bshift) + +#define HAS_EMBEDDED_DATA(node) ((le16toh((node)->fentry->icbtag.flags) & 0x7) == 3) static int udf_read(struct vop_read_args *ap) @@ -359,6 +371,26 @@ udf_read(struct vop_read_args *ap) return (0); if (uio->uio_offset < 0) return (EINVAL); + + if (HAS_EMBEDDED_DATA(node)) { + struct file_entry *fentry; + uint8_t *data; + + fentry = node->fentry; + data = &fentry->data[le32toh(fentry->l_ea)]; + fsize = le32toh(fentry->l_ad); + + n = uio->uio_resid; + diff = fsize - uio->uio_offset; + if (diff <= 0) + return (0); + if (diff < n) + n = diff; + + error = uiomove(data + uio->uio_offset, (int)n, uio); + return (error); + } + fsize = le64toh(node->fentry->inf_len); udfmp = node->udfmp; do { @@ -799,10 +831,10 @@ udf_strategy(struct vop_strategy_args *a struct buf *bp; struct vnode *vp; struct udf_node *node; + struct bufobj *bo; int maxsize; daddr_t sector; - struct bufobj *bo; - int multiplier; + int error; bp = a->a_bp; vp = a->a_vp; @@ -813,24 +845,23 @@ udf_strategy(struct vop_strategy_args *a * Files that are embedded in the fentry don't translate well * to a block number. Reject. */ - if (udf_bmap_internal(node, bp->b_lblkno * node->udfmp->bsize, - §or, &maxsize)) { + error = udf_bmap_internal(node, lblktosize(node->udfmp, bp->b_lblkno), §or, &maxsize); + if (error) { + if (error == UDF_INVALID_BMAP) + printf("udf_strategy called for file with data embedded in fentry\n"); clrbuf(bp); bp->b_blkno = -1; + bufdone(bp); + return (0); } - - /* bmap gives sector numbers, bio works with device blocks */ - multiplier = node->udfmp->bsize / DEV_BSIZE; - bp->b_blkno = sector * multiplier; - - } - if ((long)bp->b_blkno == -1) { - bufdone(bp); - return (0); + /* udf_bmap_internal gives sector numbers, bio works with device blocks */ + bp->b_blkno = sector << (node->udfmp->bshift - DEV_BSHIFT); } + bo = node->udfmp->im_bo; bp->b_iooffset = dbtob(bp->b_blkno); BO_STRATEGY(bo, bp); + return (0); } @@ -851,17 +882,43 @@ udf_bmap(struct vop_bmap_args *a) if (a->a_runb) *a->a_runb = 0; - error = udf_bmap_internal(node, a->a_bn * node->udfmp->bsize, &lsector, - &max_size); + /* + * UDF_INVALID_BMAP means data embedded into fentry, this is an internal + * error that should not be propagated to calling code. + * Most obvious mapping for this error is EOPNOTSUPP as we can not truly + * translate block numbers in this case. + * Incidentally, this return code will make vnode pager to use VOP_READ + * to get data for mmap-ed pages and udf_read knows how to do the right + * thing for this kind of files. + */ + error = udf_bmap_internal(node, a->a_bn << node->udfmp->bshift, &lsector, &max_size); + if (error == UDF_INVALID_BMAP) + return EOPNOTSUPP; if (error) return (error); /* Translate logical to physical sector number */ *a->a_bnp = lsector << (node->udfmp->bshift - DEV_BSHIFT); - /* Punt on read-ahead for now */ - if (a->a_runp) - *a->a_runp = 0; + /* + * Determine maximum number of readahead blocks following the + * requested block. + */ + if (a->a_runp) { + int nblk; + + nblk = (max_size >> node->udfmp->bshift) - 1; + if (nblk <= 0) + *a->a_runp = 0; + else if (nblk >= (MAXBSIZE >> node->udfmp->bshift)) + *a->a_runp = (MAXBSIZE >> node->udfmp->bshift) - 1; + else + *a->a_runp = nblk; + } + + if (a->a_runb) { + *a->a_runb = 0; + } return (0); } @@ -1060,7 +1117,10 @@ udf_readatoffset(struct udf_node *node, udfmp = node->udfmp; - *bp = NULL; + /* + * This call is made not only to detect UDF_INVALID_BMAP case, + * max_size is used as an ad-hoc read-ahead hint for "normal" case. + */ error = udf_bmap_internal(node, offset, §or, &max_size); if (error == UDF_INVALID_BMAP) { /* @@ -1078,9 +1138,10 @@ udf_readatoffset(struct udf_node *node, /* Adjust the size so that it is within range */ if (*size == 0 || *size > max_size) *size = max_size; - *size = min(*size, MAXBSIZE); + *size = min(*size, MAXBSIZE - blkoff(udfmp, offset)); - if ((error = udf_readlblks(udfmp, sector, *size + (offset & udfmp->bmask), bp))) { + *bp = NULL; + if ((error = bread(node->i_vnode, lblkno(udfmp, offset), (*size + blkoff(udfmp, offset) + udfmp->bmask) & ~udfmp->bmask, NOCRED, bp))) { printf("warning: udf_readlblks returned error %d\n", error); /* note: *bp may be non-NULL */ return (error); --- udf_vfsops.c.orig 2007-03-13 03:50:24.000000000 +0200 +++ udf_vfsops.c 2008-02-05 01:29:10.000000000 +0200 @@ -330,6 +330,11 @@ udf_mountfs(struct vnode *devvp, struct bo = &devvp->v_bufobj; + if (devvp->v_rdev->si_iosize_max != 0) + mp->mnt_iosize_max = devvp->v_rdev->si_iosize_max; + if (mp->mnt_iosize_max > MAXPHYS) + mp->mnt_iosize_max = MAXPHYS; + /* XXX: should be M_WAITOK */ MALLOC(udfmp, struct udf_mnt *, sizeof(struct udf_mnt), M_UDFMOUNT, M_NOWAIT | M_ZERO); --------------090706050706020808090509-- From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 7 00:15:39 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6FCA616A41A for ; Thu, 7 Feb 2008 00:15:39 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 23B8F13C459 for ; Thu, 7 Feb 2008 00:15:38 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JMuQQ-0004FR-L2 for freebsd-hackers@freebsd.org; Thu, 07 Feb 2008 00:15:34 +0000 Received: from 78-0-75-107.adsl.net.t-com.hr ([78.0.75.107]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 07 Feb 2008 00:15:34 +0000 Received: from ivoras by 78-0-75-107.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 07 Feb 2008 00:15:34 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Thu, 07 Feb 2008 01:15:27 +0100 Lines: 32 Message-ID: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig61BC6998166D2631200E4029" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 78-0-75-107.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) X-Enigmail-Version: 0.95.6 Sender: news Subject: Interesting development discipline model - PostgreSQL X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 00:15:39 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig61BC6998166D2631200E4029 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable I've just read an interesting post about a new model accepted for=20 developing PostgreSQL, and I'm finding parallels in how their release=20 8.3 was delayed similar to our 7.0. http://archives.postgresql.org/pgsql-hackers/2008-02/msg00193.php I also see some possible reasons why it couldn't work for FreeBSD: a=20 much larger group of developers, lots of 3rd party software integrated=20 into the base system, etc. but maybe it could be made to work at least=20 for ports? --------------enig61BC6998166D2631200E4029 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHqk2fldnAQVacBcgRAidLAKC2lViGdwnBAzv3CoAKbl02pMJcYQCfQSQ/ LezynLnPOBz8QOpq89NXvL8= =DPf2 -----END PGP SIGNATURE----- --------------enig61BC6998166D2631200E4029-- From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 7 20:35:51 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 174C216A473 for ; Thu, 7 Feb 2008 20:35:51 +0000 (UTC) (envelope-from gahr@gahr.ch) Received: from cpanel03.rubas-s03.net (cpanel03.rubas-s03.net [195.182.222.73]) by mx1.freebsd.org (Postfix) with ESMTP id C6A9E13C4E3 for ; Thu, 7 Feb 2008 20:35:50 +0000 (UTC) (envelope-from gahr@gahr.ch) Received: from 80-218-191-236.dclient.hispeed.ch ([80.218.191.236] helo=gahrtop.localhost) by cpanel03.rubas-s03.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.68) (envelope-from ) id 1JNDTJ-0002SN-NW for freebsd-hackers@freebsd.org; Thu, 07 Feb 2008 21:35:49 +0100 Message-ID: <47AB6B9E.1090605@gahr.ch> Date: Thu, 07 Feb 2008 21:35:42 +0100 From: Pietro Cerutti User-Agent: Thunderbird 2.0.0.9 (X11/20071121) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org X-Enigmail-Version: 0.95.5 OpenPGP: id=9571F78E; url=http://www.gahr.ch/pgp Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="------------enig59FB2DD22BBCDE5D31664558" X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cpanel03.rubas-s03.net X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - gahr.ch X-Source: X-Source-Args: X-Source-Dir: Subject: uma_zcreate and number of items to create X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 20:35:51 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig59FB2DD22BBCDE5D31664558 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi hackers, I was thinking about modifying the zone allocator to allow to specify the number of items to pre-allocate when a zone is first created. Before starting to hack it, I would like to know what you think about it. Do you find it useful or not? Thanks --=20 Pietro Cerutti PGP Public Key: http://gahr.ch/pgp --------------enig59FB2DD22BBCDE5D31664558 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHq2uiwMJqmJVx944RCl9ZAKCAr/cVDG8KQE6YBuPnyCWQ5DggvgCg1neH q2luRH/igJeLtqekE/JYFBQ= =H1rF -----END PGP SIGNATURE----- --------------enig59FB2DD22BBCDE5D31664558-- From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 7 20:49:11 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D34A216A419 for ; Thu, 7 Feb 2008 20:49:11 +0000 (UTC) (envelope-from freebsd.dev@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.181]) by mx1.freebsd.org (Postfix) with ESMTP id 7B04A13C461 for ; Thu, 7 Feb 2008 20:49:11 +0000 (UTC) (envelope-from freebsd.dev@gmail.com) Received: by py-out-1112.google.com with SMTP id u52so5107035pyb.10 for ; Thu, 07 Feb 2008 12:49:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; bh=k9ZOBUkusd1vfxt7v7xuHxrkf/qPaR24UIFm1woGNDE=; b=UDwCshOyENCHSRPQ7Wdic9JsMEJ2nYU3ouFxG4ppHZIMbVJFVTsQ31kLndu9EPd78S9gmBcHTA/5xdAcle7kObM8flVv8wdtF7e2GDvtW81z9Sr72CszYa6z6aMGBNJIviqilL4IXWQaNwfYtkgYdB/gsrdovC/278du8fv9prk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=ViulPDIl9IN3Uh2z6UzlrQvogcVq6SwldKaoJInlPBv7ozpOglxfH+uZcyh/gnWBEOxHS1P+BOXkdKTPUM8Kf3dlhlVwqoCW9BV5hvyv5m/SNTOrEisSjVIfygwUifB6CSWSLNvAPa4ClcAPntZneO05Cw+xczLLjVSgw2hykYg= Received: by 10.141.113.6 with SMTP id q6mr7906391rvm.36.1202415773948; Thu, 07 Feb 2008 12:22:53 -0800 (PST) Received: by 10.140.136.7 with HTTP; Thu, 7 Feb 2008 12:22:53 -0800 (PST) Message-ID: <50cd4e5f0802071222w1222d901o3ce8770b5f5725b4@mail.gmail.com> Date: Thu, 7 Feb 2008 14:22:53 -0600 From: "Biks N" To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: retrive data from mbuf chain X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 20:49:11 -0000 Hi, I am new to FreeBSD kernel programming. Currently I am trying to work on mbuf data manupulation. >From my understanding: data (payload) is stored into one or more mufs which are chained together through m_next pointer. Now, I need to retrive all data in mbuf chain ( mbufs linked by m_next). I am working ip_output() in netinet/ip_output.c Does there exist inbuilt function/macro to retrive all the data in mbuf chain? thanks in advance :) From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 7 21:12:02 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DBA7A16A418 for ; Thu, 7 Feb 2008 21:12:02 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from fk-out-0910.google.com (fk-out-0910.google.com [209.85.128.184]) by mx1.freebsd.org (Postfix) with ESMTP id 7008C13C459 for ; Thu, 7 Feb 2008 21:12:02 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: by fk-out-0910.google.com with SMTP id b27so3876062fka.11 for ; Thu, 07 Feb 2008 13:12:01 -0800 (PST) Received: by 10.82.158.12 with SMTP id g12mr21429959bue.18.1202418720276; Thu, 07 Feb 2008 13:12:00 -0800 (PST) Received: by 10.82.187.11 with HTTP; Thu, 7 Feb 2008 13:12:00 -0800 (PST) Message-ID: Date: Thu, 7 Feb 2008 23:12:00 +0200 From: "Vlad GALU" To: "Biks N" In-Reply-To: <50cd4e5f0802071222w1222d901o3ce8770b5f5725b4@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <50cd4e5f0802071222w1222d901o3ce8770b5f5725b4@mail.gmail.com> Cc: freebsd-hackers@freebsd.org Subject: Re: retrive data from mbuf chain X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 21:12:02 -0000 On 2/7/08, Biks N wrote: > Hi, > > I am new to FreeBSD kernel programming. > > Currently I am trying to work on mbuf data manupulation. > > >From my understanding: data (payload) is stored into one or more mufs > which are chained together through m_next pointer. > > Now, I need to retrive all data in mbuf chain ( mbufs linked by > m_next). I am working ip_output() in netinet/ip_output.c > > Does there exist inbuilt function/macro to retrive all the data in mbuf chain? I may be wrong on this one, but according to my understanding, a m_pullup() followed by a mtod() should do the trick. > > thanks in advance :) > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > -- Mahnahmahnah! From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 7 21:26:15 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2CD6316A41A for ; Thu, 7 Feb 2008 21:26:15 +0000 (UTC) (envelope-from sam@errno.com) Received: from ebb.errno.com (ebb.errno.com [69.12.149.25]) by mx1.freebsd.org (Postfix) with ESMTP id E6FDA13C469 for ; Thu, 7 Feb 2008 21:26:14 +0000 (UTC) (envelope-from sam@errno.com) Received: from trouble.errno.com (trouble.errno.com [10.0.0.248]) (authenticated bits=0) by ebb.errno.com (8.13.6/8.12.6) with ESMTP id m17LQEEw055552 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 7 Feb 2008 13:26:14 -0800 (PST) (envelope-from sam@errno.com) Message-ID: <47AB7775.2040000@errno.com> Date: Thu, 07 Feb 2008 13:26:13 -0800 From: Sam Leffler User-Agent: Thunderbird 2.0.0.9 (X11/20071125) MIME-Version: 1.0 To: Biks N References: <50cd4e5f0802071222w1222d901o3ce8770b5f5725b4@mail.gmail.com> In-Reply-To: <50cd4e5f0802071222w1222d901o3ce8770b5f5725b4@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DCC--Metrics: ebb.errno.com; whitelist Cc: freebsd-hackers@freebsd.org Subject: Re: retrive data from mbuf chain X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 21:26:15 -0000 Biks N wrote: > Hi, > > I am new to FreeBSD kernel programming. > > Currently I am trying to work on mbuf data manupulation. > > >From my understanding: data (payload) is stored into one or more mufs > which are chained together through m_next pointer. > > Now, I need to retrive all data in mbuf chain ( mbufs linked by > m_next). I am working ip_output() in netinet/ip_output.c > > Does there exist inbuilt function/macro to retrive all the data in mbuf chain? > man 9 mbuf; look for m_copydata. Sam From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 7 21:30:19 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BD3416A418 for ; Thu, 7 Feb 2008 21:30:19 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outG.internet-mail-service.net (outG.internet-mail-service.net [216.240.47.230]) by mx1.freebsd.org (Postfix) with ESMTP id 33D6013C4E7 for ; Thu, 7 Feb 2008 21:30:18 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Thu, 07 Feb 2008 13:30:18 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id B003B1270F7; Thu, 7 Feb 2008 13:30:17 -0800 (PST) Message-ID: <47AB786D.8070403@elischer.org> Date: Thu, 07 Feb 2008 13:30:21 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Vlad GALU References: <50cd4e5f0802071222w1222d901o3ce8770b5f5725b4@mail.gmail.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Biks N Subject: Re: retrive data from mbuf chain X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 21:30:19 -0000 Vlad GALU wrote: > On 2/7/08, Biks N wrote: >> Hi, >> >> I am new to FreeBSD kernel programming. >> >> Currently I am trying to work on mbuf data manupulation. >> >> >From my understanding: data (payload) is stored into one or more mufs >> which are chained together through m_next pointer. >> >> Now, I need to retrive all data in mbuf chain ( mbufs linked by >> m_next). I am working ip_output() in netinet/ip_output.c >> >> Does there exist inbuilt function/macro to retrive all the data in mbuf chain? > > I may be wrong on this one, but according to my understanding, a > m_pullup() followed by a mtod() should do the trick. depends what he wants to do.. check /sys/sys/mbuf.h and /sys/kern/uipc_mbuf.c for all sorts of methods for handling mbufs. > >> thanks in advance :) >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >> > > From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 7 21:38:06 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 264C616A420 for ; Thu, 7 Feb 2008 21:38:06 +0000 (UTC) (envelope-from freebsd.dev@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.189]) by mx1.freebsd.org (Postfix) with ESMTP id AE42A13C469 for ; Thu, 7 Feb 2008 21:38:05 +0000 (UTC) (envelope-from freebsd.dev@gmail.com) Received: by rv-out-0910.google.com with SMTP id g13so2582843rvb.43 for ; Thu, 07 Feb 2008 13:38:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=4aFgZE2EnBhM07GcAvTaGUe43xEKyfkBcMYTZFVR1N0=; b=UYZLNuIQRxXHYWlBvIhfkgpRXa0BsqSMwkeTovpb+u3YT7u0fZ27JespzQois59ls5FeIWg7+yorylCQd5o6zdHo67XH0QMJem0DiU6asHqFoyl78zCSJRqLAltQiCpqJnbQAUF4LF59XRbdmU3wepxpvCwAcvgKh80uVO9LgNc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=u8l5bYqo22mRLEuPlFy5IoD/sYJ6ARiMujUJ7Z43wuoYVJA5qLgYPUx2K5xBYoW66PJVHwdDmqe/PBPxCDyv9KE+iYFME6A6yrPxWAZB+9ccGC+BI9Xc4tLCBwzgWUl5VaOq9PH5JWm8J/doyJhTBuZj4I5X7uAUiSJqjLUn9dU= Received: by 10.141.18.14 with SMTP id v14mr7896592rvi.242.1202420285217; Thu, 07 Feb 2008 13:38:05 -0800 (PST) Received: by 10.140.136.7 with HTTP; Thu, 7 Feb 2008 13:38:05 -0800 (PST) Message-ID: <50cd4e5f0802071338h63b4fcbax9ceb9643aa5d7504@mail.gmail.com> Date: Thu, 7 Feb 2008 15:38:05 -0600 From: "Biks N" To: "Sam Leffler" In-Reply-To: <47AB7775.2040000@errno.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <50cd4e5f0802071222w1222d901o3ce8770b5f5725b4@mail.gmail.com> <47AB7775.2040000@errno.com> Cc: freebsd-hackers@freebsd.org Subject: Re: retrive data from mbuf chain X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 21:38:06 -0000 On Feb 7, 2008 3:26 PM, Sam Leffler wrote: > > Biks N wrote: > > Hi, > > > > I am new to FreeBSD kernel programming. > > > > Currently I am trying to work on mbuf data manupulation. > > > > >From my understanding: data (payload) is stored into one or more mufs > > which are chained together through m_next pointer. > > > > Now, I need to retrive all data in mbuf chain ( mbufs linked by > > m_next). I am working ip_output() in netinet/ip_output.c > > > > Does there exist inbuilt function/macro to retrive all the data in mbuf chain? > > > > man 9 mbuf; look for m_copydata. m_copydata(mbuf, offset, len, buf) Will it copy data from entire "mbuf chain" or just from the current mbuf ? Thanks for your help > > Sam > > From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 7 22:10:43 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 557E916A417 for ; Thu, 7 Feb 2008 22:10:43 +0000 (UTC) (envelope-from Vijay.Singh@netapp.com) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by mx1.freebsd.org (Postfix) with ESMTP id 1576513C457 for ; Thu, 7 Feb 2008 22:10:42 +0000 (UTC) (envelope-from Vijay.Singh@netapp.com) X-IronPort-AV: E=Sophos;i="4.25,317,1199692800"; d="scan'208";a="149881264" Received: from smtp2.corp.netapp.com ([10.57.159.114]) by mx2-out.netapp.com with ESMTP; 07 Feb 2008 13:41:16 -0800 Received: from svlexrs01.hq.netapp.com (svlexrs01.corp.netapp.com [10.57.156.158]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id m17LfG58004586; Thu, 7 Feb 2008 13:41:16 -0800 (PST) Received: from exsvlrb01.hq.netapp.com ([10.56.8.62]) by svlexrs01.hq.netapp.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 7 Feb 2008 13:41:16 -0800 Received: from exsvl03.hq.netapp.com ([10.56.8.64]) by exsvlrb01.hq.netapp.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 7 Feb 2008 13:41:15 -0800 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Thu, 7 Feb 2008 13:41:14 -0800 Message-ID: <8B3BF7A02B910640AB146A919AB36594054A71E1@exsvl03.hq.netapp.com> In-Reply-To: <50cd4e5f0802071338h63b4fcbax9ceb9643aa5d7504@mail.gmail.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: retrive data from mbuf chain Thread-Index: Achp0cqk8PYntFTFRfm031xNtQ9O1QAAFtpQ References: <50cd4e5f0802071222w1222d901o3ce8770b5f5725b4@mail.gmail.com><47AB7775.2040000@errno.com> <50cd4e5f0802071338h63b4fcbax9ceb9643aa5d7504@mail.gmail.com> From: "Singh, Vijay" To: "Biks N" , "Sam Leffler" X-OriginalArrivalTime: 07 Feb 2008 21:41:15.0139 (UTC) FILETIME=[2A284530:01C869D2] X-Mailman-Approved-At: Thu, 07 Feb 2008 22:25:02 +0000 Cc: freebsd-hackers@freebsd.org Subject: RE: retrive data from mbuf chain X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 22:10:43 -0000 >From the entire chain. best regards, Vijay =20 > -----Original Message----- > From: Biks N [mailto:freebsd.dev@gmail.com]=20 > Sent: Thursday, February 07, 2008 1:38 PM > To: Sam Leffler > Cc: freebsd-hackers@freebsd.org > Subject: Re: retrive data from mbuf chain >=20 > On Feb 7, 2008 3:26 PM, Sam Leffler wrote: > > > > Biks N wrote: > > > Hi, > > > > > > I am new to FreeBSD kernel programming. > > > > > > Currently I am trying to work on mbuf data manupulation. > > > > > > >From my understanding: data (payload) is stored into one or more=20 > > > >mufs > > > which are chained together through m_next pointer. > > > > > > Now, I need to retrive all data in mbuf chain ( mbufs linked by=20 > > > m_next). I am working ip_output() in netinet/ip_output.c > > > > > > Does there exist inbuilt function/macro to retrive all=20 > the data in mbuf chain? > > > > > > > man 9 mbuf; look for m_copydata. >=20 >=20 > m_copydata(mbuf, offset, len, buf) >=20 > Will it copy data from entire "mbuf chain" or just from the=20 > current mbuf ? >=20 > Thanks for your help >=20 >=20 > > > > Sam > > > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list=20 > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to=20 > "freebsd-hackers-unsubscribe@freebsd.org" >=20 From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 07:38:29 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8EB5116A419 for ; Fri, 8 Feb 2008 07:38:29 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: from palm.hoeg.nl (mx0.hoeg.nl [IPv6:2001:610:652::211]) by mx1.freebsd.org (Postfix) with ESMTP id 573A613C448 for ; Fri, 8 Feb 2008 07:38:29 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: by palm.hoeg.nl (Postfix, from userid 1000) id 2E3331CD2A; Fri, 8 Feb 2008 08:36:49 +0100 (CET) Date: Fri, 8 Feb 2008 08:36:49 +0100 From: Ed Schouten To: Pietro Cerutti Message-ID: <20080208073649.GB94844@hoeg.nl> References: <47AB6B9E.1090605@gahr.ch> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="7iMSBzlTiPOCCT2k" Content-Disposition: inline In-Reply-To: <47AB6B9E.1090605@gahr.ch> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: FreeBSD Hackers Subject: Re: uma_zcreate and number of items to create X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 07:38:29 -0000 --7iMSBzlTiPOCCT2k Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable * Pietro Cerutti wrote: > Before starting to hack it, I would like to know what you think about > it. Do you find it useful or not? I was thinking about this as well, because my TTY hackery could also use this to preallocate cblocks to store characters from interrupts, but then you could also just keep a free list to store them. You need to allocate them anyway, so the space is already wasted. Using your own per-{device,object,etc} free list should even be faster, because uma probably has some per-zone spinlock, while you may not need the locking in your specific situation (i.e. another lock already protects it). --=20 Ed Schouten WWW: http://g-rave.nl/ --7iMSBzlTiPOCCT2k Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEARECAAYFAkesBpEACgkQ52SDGA2eCwVumgCfdfAsco9CiceyPOcSX5KYKyIW Pq0AniyerMLAYY1VOHaiFDaqTYtZi7r7 =Ura5 -----END PGP SIGNATURE----- --7iMSBzlTiPOCCT2k-- From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 07:56:34 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C4C8616A46B for ; Fri, 8 Feb 2008 07:56:34 +0000 (UTC) (envelope-from gahr@gahr.ch) Received: from cpanel03.rubas-s03.net (cpanel03.rubas-s03.net [195.182.222.73]) by mx1.freebsd.org (Postfix) with ESMTP id B943713C46A for ; Fri, 8 Feb 2008 07:56:33 +0000 (UTC) (envelope-from gahr@gahr.ch) Received: from 80-218-191-236.dclient.hispeed.ch ([80.218.191.236] helo=gahrtop.localhost) by cpanel03.rubas-s03.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.68) (envelope-from ) id 1JNO64-000514-4v; Fri, 08 Feb 2008 08:56:32 +0100 Message-ID: <47AC0B28.8060508@gahr.ch> Date: Fri, 08 Feb 2008 08:56:24 +0100 From: Pietro Cerutti User-Agent: Thunderbird 2.0.0.9 (X11/20071121) MIME-Version: 1.0 To: Ed Schouten References: <47AB6B9E.1090605@gahr.ch> <20080208073649.GB94844@hoeg.nl> In-Reply-To: <20080208073649.GB94844@hoeg.nl> X-Enigmail-Version: 0.95.5 OpenPGP: id=9571F78E; url=http://www.gahr.ch/pgp Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="------------enig80D1382A4F579DCB3CAE7F14" X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cpanel03.rubas-s03.net X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - gahr.ch X-Source: X-Source-Args: X-Source-Dir: Cc: FreeBSD Hackers Subject: Re: uma_zcreate and number of items to create X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 07:56:34 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig80D1382A4F579DCB3CAE7F14 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Ed Schouten wrote: > Using your own per-{device,object,etc} free list should even be faster,= > because uma probably has some per-zone spinlock, while you may not need= > the locking in your specific situation (i.e. another lock already > protects it). Interesting point! --=20 Pietro Cerutti PGP Public Key: http://gahr.ch/pgp --------------enig80D1382A4F579DCB3CAE7F14 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEAREKAAYFAkesCywACgkQwMJqmJVx944obACgj+ZqNVpvio8KUhUUfd9qBW7D Fb0AoJK/ziPbqWZJa4edx7EVnverOze0 =BN6p -----END PGP SIGNATURE----- --------------enig80D1382A4F579DCB3CAE7F14-- From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 07:59:05 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA37816A417 for ; Fri, 8 Feb 2008 07:59:05 +0000 (UTC) (envelope-from sbruno@miralink.com) Received: from plato.miralink.com (mail.miralink.com [70.103.185.20]) by mx1.freebsd.org (Postfix) with ESMTP id 9C6B913C461 for ; Fri, 8 Feb 2008 07:59:05 +0000 (UTC) (envelope-from sbruno@miralink.com) Received: from localhost (localhost.localdomain [127.0.0.1]) by plato.miralink.com (Postfix) with ESMTP id 4843161B91B for ; Thu, 7 Feb 2008 23:59:05 -0800 (PST) Received: from plato.miralink.com ([127.0.0.1]) by localhost (plato.miralink.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 25976-06 for ; Thu, 7 Feb 2008 23:59:04 -0800 (PST) Received: from [10.47.1.58] (vpn.office.miralink.com [10.0.0.5]) by plato.miralink.com (Postfix) with ESMTP id AA28661B886 for ; Thu, 7 Feb 2008 23:59:04 -0800 (PST) Message-ID: <47AC0BC8.3030509@miralink.com> Date: Thu, 07 Feb 2008 23:59:04 -0800 From: Sean Bruno User-Agent: Thunderbird 2.0.0.9 (X11/20071115) MIME-Version: 1.0 To: FreeBSD Hackers Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DSPAM-Result: Innocent X-DSPAM-Processed: Thu Feb 7 23:59:05 2008 X-DSPAM-Confidence: 0.7879 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 47ac0bc972433377160525 X-DSPAM-Factors: 27, X-Virus-Scanned: amavisd-new at X-Spam-Status: No, score=-4.499 tagged_above=-10 required=6.6 autolearn=ham tests=[ALL_TRUSTED=-1.8, BAYES_00=-2.599, DSPAM_HAM=-0.1] X-Spam-Score: -4.499 X-Spam-Level: Subject: Question about http://torrents.freebsd.org:8080/index.html X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 07:59:05 -0000 Does it seem correct to all concerned that each release actually lists all files twice? There is a torrent for the entire release CD ISO set, and then there is a completely separate torrent for each CD ISO file. At least that is what it looks like to me. Is this correct? Sean From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 09:52:32 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3719316A41A for ; Fri, 8 Feb 2008 09:52:32 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 141B613C46B for ; Fri, 8 Feb 2008 09:52:32 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id A303C4B7E6; Fri, 8 Feb 2008 04:52:31 -0500 (EST) Date: Fri, 8 Feb 2008 09:52:31 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Biks N In-Reply-To: <50cd4e5f0802071222w1222d901o3ce8770b5f5725b4@mail.gmail.com> Message-ID: <20080208094836.P3375@fledge.watson.org> References: <50cd4e5f0802071222w1222d901o3ce8770b5f5725b4@mail.gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org Subject: Re: retrive data from mbuf chain X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 09:52:32 -0000 On Thu, 7 Feb 2008, Biks N wrote: > I am new to FreeBSD kernel programming. > > Currently I am trying to work on mbuf data manupulation. > >> From my understanding: data (payload) is stored into one or more mufs > which are chained together through m_next pointer. > > Now, I need to retrive all data in mbuf chain ( mbufs linked by m_next). I > am working ip_output() in netinet/ip_output.c > > Does there exist inbuilt function/macro to retrive all the data in mbuf > chain? netsmb and friends use a kernel library, mchain, to do marshalling and unmarashalling. This is generally not used by lower level protocol code because casting to a data structure via m_pull() and mtod() generally does the trick. However, for more complex protocols with conditional structuring, mchain can make life easier. mdchain(9) describes functions for pulling data out of mbuf chains, and mbchain(9) describes functions for constructing a chain. You'll need to declare a module dependency on mchain. NAME mdchain, md_initm, md_done, md_append_record, md_next_record, md_get_uint8, md_get_uint16, md_get_uint16be, md_get_uint16le, md_get_uint32, md_get_uint32be, md_get_uint32le, md_get_int64, md_get_int64be, md_get_int64le, md_get_mem, md_get_mbuf, md_get_uio -- set of functions to dissect an mbuf chain to various data types NAME mbchain, mb_init, mb_initm, mb_done, mb_detach, mb_fixhdr, mb_reserve, mb_put_uint8, mb_put_uint16be, mb_put_uint16le, mb_put_uint32be, mb_put_uint32le, mb_put_int64be, mb_put_int64le, mb_put_mem, mb_put_mbuf, mb_put_uio -- set of functions to build an mbuf chain from various data types I'm not sure if these routines introduce additional data copies or not, but if implementing an RPC-like protocol they presumably make life a lot easier. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 10:28:52 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F06216A417 for ; Fri, 8 Feb 2008 10:28:52 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 144C013C459 for ; Fri, 8 Feb 2008 10:28:52 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 6FA252083 for ; Fri, 8 Feb 2008 11:28:43 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.3/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id DB4A9207F for ; Fri, 8 Feb 2008 11:28:42 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id B4935844B6; Fri, 8 Feb 2008 11:28:42 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: hackers@freebsd.org Date: Fri, 08 Feb 2008 11:28:42 +0100 Message-ID: <86prv7agit.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Subject: nvi strangeness X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 10:28:52 -0000 As everybody knows, the ex/vi command for regexp substitution is [range] s[ubstitute] [/pattern/replace/] [options] [count] [flags] The man page does not document the syntax for ranges (nor for offsets or counts for that matter) Assuming the syntax is the same as for ed / sed, the following should replace every occurrence of the word wait with the word delta: :,s/\/delta/g where "," means "the entire file" That doesn't work, however, and neither does "1,"; only "X,Y" works, so if you want to substitute in the entire file, you first have to find out how long it is, then manually type in that number. ...or you could use the following (courtesy of roberto@): :%s/\/delta/g None of this is documented anywhere. I would really like nvi to understand ed-like ranges, and vi(1) to either document this or cross-reference ed(1). Going off on a tangent, I wonder why the nvi license is reproduced in the man page, as it doesn't seem to require it, and no other man page includes the license of the program it describes (with a single exception: groff) DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 11:31:02 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EFA3216A420 for ; Fri, 8 Feb 2008 11:31:02 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 5B31513C4D3; Fri, 8 Feb 2008 11:31:01 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47AC3D73.2010705@FreeBSD.org> Date: Fri, 08 Feb 2008 12:30:59 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Sean Bruno References: <47AC0BC8.3030509@miralink.com> In-Reply-To: <47AC0BC8.3030509@miralink.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Hackers Subject: Re: Question about http://torrents.freebsd.org:8080/index.html X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 11:31:03 -0000 Sean Bruno wrote: > Does it seem correct to all concerned that each release actually lists > all files twice? > > There is a torrent for the entire release CD ISO set, and then there is > a completely separate torrent for each CD ISO file. At least that is > what it looks like to me. > Is this correct? Yes. Many people like to download the whole thing in one torrent. It does split the aggregate swarm bandwidth, but it doesn't seem to be a problem in practice. Kris From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 11:33:31 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B95116A41A for ; Fri, 8 Feb 2008 11:33:31 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id F211613C469 for ; Fri, 8 Feb 2008 11:33:30 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender:X-Spam-Status:Subject; b=L6DgCcA3+YO4d9THAnI391jGS88PsxXcgTes7QM2m8yj6+Kky6CtfOd3/nq9kQzYgB6vOslOWQpu/ud49t2pqoMK860/sKCA2Eq4HHxrXnPzxnl3uPEUc2c6OSU0/wMs6B0ARp84lVhUFtGLOGDKLUExZ4aOaTqzz1qYQPxwVBo=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1JNRH8-000H8A-DU; Fri, 08 Feb 2008 14:20:10 +0300 Date: Fri, 8 Feb 2008 14:20:09 +0300 From: Eygene Ryabinkin To: Dag-Erling Sm??rgrav Message-ID: References: <86prv7agit.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <86prv7agit.fsf@ds4.des.no> Sender: rea-fbsd@codelabs.ru X-Spam-Status: No, score=-1.8 required=4.0 tests=ALL_TRUSTED,AWL,BAYES_50 Cc: hackers@freebsd.org Subject: Re: nvi strangeness X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 11:33:31 -0000 Dag-Erling, good day. Speaking for the system's nvi (not the ports one -- not using it). Fri, Feb 08, 2008 at 11:28:42AM +0100, Dag-Erling Sm??rgrav wrote: > As everybody knows, the ex/vi command for regexp substitution is > > [range] s[ubstitute] [/pattern/replace/] [options] [count] [flags] > > The man page does not document the syntax for ranges (nor for offsets or > counts for that matter) > > Assuming the syntax is the same as for ed / sed, the following should > replace every occurrence of the word wait with the word delta: > > :,s/\/delta/g > > where "," means "the entire file" Nope, "," means "from the current line to the current line". So :,s/anything/the other thing/g is equal to plain :s/anything/the other thing/g > That doesn't work, however, and neither does "1,"; only "X,Y" works, so > if you want to substitute in the entire file, you first have to find out > how long it is, then manually type in that number. > > ...or you could use the following (courtesy of roberto@): > > :%s/\/delta/g :1,$s/anything/the other thing/ will be the same as the ':%s...'. But ':%s...' is shorter to type ;)) Moreover, "1," will select lines from the first one to the current one. And, for example, ",+3" will select four lines, counting from the current and going down; "-2," will select three lines, counting from the current and going up. > None of this is documented anywhere. $ zcat /usr/share/doc/usd/13.viref/paper.ascii.gz | less and search for the 'percent sign'. -- Eygene From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 11:33:41 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B24316A47D for ; Fri, 8 Feb 2008 11:33:41 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id AAA8B13C4D9 for ; Fri, 8 Feb 2008 11:33:40 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender:X-Spam-Status:Subject; b=RaidSMrZYmNcxUPGHPzDecx2dcoPPPYhf4i7FGlRTMjheUfPEGOeBhUw+xcqHEd8Z3M4mFKBuf08S1Mt6Zo+tCvbLajCREf50CpDOjpbUtF2te8dKJg6qyDU7edNtmRrqibyUYTQTdAwf3+FjuQXCPvxZvYL8q2/nUhMegVN+BI=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1JNRJz-000H9Q-0R; Fri, 08 Feb 2008 14:23:07 +0300 Date: Fri, 8 Feb 2008 14:23:05 +0300 From: Eygene Ryabinkin To: Dag-Erling Sm??rgrav Message-ID: References: <86prv7agit.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: Sender: rea-fbsd@codelabs.ru X-Spam-Status: No, score=-1.8 required=4.0 tests=ALL_TRUSTED,AWL,BAYES_50 Cc: hackers@freebsd.org Subject: Re: nvi strangeness X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 11:33:41 -0000 Fri, Feb 08, 2008 at 02:20:09PM +0300, Eygene Ryabinkin wrote: > > None of this is documented anywhere. > > $ zcat /usr/share/doc/usd/13.viref/paper.ascii.gz | less > and search for the 'percent sign'. Hmm, it is much better to search for 'Ex Addressing' ;)) -- Eygene From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 12:07:47 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5D91416A419 for ; Fri, 8 Feb 2008 12:07:47 +0000 (UTC) (envelope-from xfb52@dial.pipex.com) Received: from mk-outboundfilter-1.mail.uk.tiscali.com (mk-outboundfilter-1.mail.uk.tiscali.com [212.74.114.37]) by mx1.freebsd.org (Postfix) with ESMTP id E3B3A13C45E for ; Fri, 8 Feb 2008 12:07:46 +0000 (UTC) (envelope-from xfb52@dial.pipex.com) X-Trace: 36593402/mk-outboundfilter-1.mail.uk.tiscali.com/PIPEX/$MX-ACCEPTED/pipex-infrastructure/62.241.163.6 X-SBRS: None X-RemoteIP: 62.241.163.6 X-IP-MAIL-FROM: xfb52@dial.pipex.com X-IP-BHB: Once X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao8CACLOq0c+8aMG/2dsb2JhbACsKA X-IP-Direction: IN Received: from astro.systems.pipex.net ([62.241.163.6]) by smtp.pipex.tiscali.co.uk with ESMTP; 08 Feb 2008 11:38:40 +0000 Received: from [192.168.23.2] (62-31-10-181.cable.ubr05.edin.blueyonder.co.uk [62.31.10.181]) by astro.systems.pipex.net (Postfix) with ESMTP id 7A2FFE00008A; Fri, 8 Feb 2008 11:38:39 +0000 (GMT) Message-ID: <47AC3F3C.7070909@dial.pipex.com> Date: Fri, 08 Feb 2008 11:38:36 +0000 From: Alex Zbyslaw User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-GB; rv:1.7.13) Gecko/20061205 X-Accept-Language: en MIME-Version: 1.0 To: =?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?= References: <86prv7agit.fsf@ds4.des.no> In-Reply-To: <86prv7agit.fsf@ds4.des.no> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Cc: hackers@freebsd.org Subject: Re: nvi strangeness X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 12:07:47 -0000 Dag-Erling Sm=F8rgrav wrote: >That doesn't work, however, and neither does "1,"; only "X,Y" works, so >if you want to substitute in the entire file, you first have to find out= >how long it is, then manually type in that number. > > =20 > Does 1,$ not work? That's what I always used in ed. --Alex From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 12:59:08 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 51EA516A419 for ; Fri, 8 Feb 2008 12:59:08 +0000 (UTC) (envelope-from cihan@enderunix.org) Received: from istanbul.enderunix.org (freefall.marmara.edu.tr [193.140.143.23]) by mx1.freebsd.org (Postfix) with ESMTP id 465BB13C447 for ; Fri, 8 Feb 2008 12:59:05 +0000 (UTC) (envelope-from cihan@enderunix.org) Received: (qmail 44343 invoked by uid 1040); 8 Feb 2008 12:31:09 -0000 Received: from unknown (HELO localhost) (cihan@85.105.200.118) by 0 with ESMTPA; 8 Feb 2008 12:31:08 -0000 Date: Fri, 8 Feb 2008 14:30:51 +0200 From: =?utf-8?B?Q2loYW4gS8O2bWXDp2/En2x1?= X-Mailer: The Bat! (v3.99.29) UNREG X-Priority: 3 (Normal) Message-ID: <1658547503.20080208143051@enderunix.org> To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: quoted-printable X-Antivirus: avast! (VPS 080207-0, 07.02.2008), Outbound message X-Antivirus-Status: Clean X-SMTP-Filter: SurGATE SMTP Filter Engine Release 1.0.1 http://www.endersys.com X-SurGATE-Result: Clean (Content eval: -4.40 points) Subject: vfork? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 12:59:08 -0000 hello list I have a question about vfork when I create a child process with vfork, child is copying parent process file descriptor array. this is time, after child copied parent file descriptor array, file's entry reference count in open file table is being incremented by one? thanks for answers --=20 Cihan K=F6me=E7o=F0lu, EnderUNIX SDT mailto:cihan@enderunix.org From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 14:46:43 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 91F9716A41B for ; Fri, 8 Feb 2008 14:46:43 +0000 (UTC) (envelope-from frank@exit.com) Received: from tinker.exit.com (tinker.exit.com [206.223.0.1]) by mx1.freebsd.org (Postfix) with ESMTP id 1051913C468 for ; Fri, 8 Feb 2008 14:46:42 +0000 (UTC) (envelope-from frank@exit.com) Received: from jill.exit.com (jill.exit.com [206.223.0.4]) by tinker.exit.com (8.14.1/8.14.1) with ESMTP id m18EJNqe075210; Fri, 8 Feb 2008 06:19:23 -0800 (PST) (envelope-from frank@exit.com) DKIM-Signature: v=1; a=rsa-sha1; c=simple/simple; d=exit.com; s=tinker; t=1202480363; bh=WIvgBEBwbFftDjCGtxu4unQQsg0=; h=X-Authentication-Warning:Subject:From:Reply-To:To:Cc:In-Reply-To: References:Content-Type:Content-Transfer-Encoding:Organization: Date:Message-Id:Mime-Version:X-Mailer; b=vw1witKBwRw1UQGKE7jq38lQo mGrrTz9lruA0S53sxRn9X34CpA8U3ac+bEO9H3ufQZ7VyMGYQd/tvhtMlOQL0a8tsJw LFgtpTXi6W/EOCNxO+smaOD3Zhf0yfmaBCsXva4Z6c8jbKVF2rWlQZPYELL6FOUijWm Na0CNyqBHK/k= Received: from jill.exit.com (localhost [127.0.0.1]) by jill.exit.com (8.14.1/8.14.1) with ESMTP id m18EJNPx093481; Fri, 8 Feb 2008 06:19:23 -0800 (PST) (envelope-from frank@exit.com) Received: (from frank@localhost) by jill.exit.com (8.14.1/8.14.2/Submit) id m18EJMrB093467; Fri, 8 Feb 2008 06:19:22 -0800 (PST) (envelope-from frank@exit.com) X-Authentication-Warning: jill.exit.com: frank set sender to frank@exit.com using -f From: Frank Mayhar To: Alex Zbyslaw In-Reply-To: <47AC3F3C.7070909@dial.pipex.com> References: <86prv7agit.fsf@ds4.des.no> <47AC3F3C.7070909@dial.pipex.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Organization: Exit Consulting Date: Fri, 08 Feb 2008 06:19:22 -0800 Message-Id: <1202480362.92566.1.camel@jill.exit.com> Mime-Version: 1.0 X-Mailer: Evolution 2.12.3 FreeBSD GNOME Team Port X-Virus-Scanned: ClamAV 0.92/5741/Fri Feb 8 03:47:08 2008 on tinker.exit.com X-Virus-Status: Clean Cc: Dag-Erling =?ISO-8859-1?Q?Sm=F8rgrav?= , hackers@freebsd.org Subject: Re: nvi strangeness X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: frank@exit.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 14:46:43 -0000 On Fri, 2008-02-08 at 11:38 +0000, Alex Zbyslaw wrote: > Dag-Erling Sm=F8rgrav wrote: >=20 > >That doesn't work, however, and neither does "1,"; only "X,Y" works, so > >if you want to substitute in the entire file, you first have to find out > >how long it is, then manually type in that number. > > > > =20 > > > Does 1,$ not work? That's what I always used in ed. Yes, it works. I use it all the time in nvi, in various forms. --=20 Frank Mayhar frank@exit.com http://www.exit.com/ Exit Consulting http://www.gpsclock.com/ http://www.exit.com/blog/frank/ http://www.zazzle.com/fmayhar* From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 17:30:28 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5FC3B16A417 for ; Fri, 8 Feb 2008 17:30:28 +0000 (UTC) (envelope-from sbruno@miralink.com) Received: from plato.miralink.com (mail.miralink.com [70.103.185.20]) by mx1.freebsd.org (Postfix) with ESMTP id 2993613C458 for ; Fri, 8 Feb 2008 17:30:28 +0000 (UTC) (envelope-from sbruno@miralink.com) Received: from localhost (localhost.localdomain [127.0.0.1]) by plato.miralink.com (Postfix) with ESMTP id BD1C261B833; Fri, 8 Feb 2008 09:30:27 -0800 (PST) Received: from plato.miralink.com ([127.0.0.1]) by localhost (plato.miralink.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 12634-07; Fri, 8 Feb 2008 09:30:26 -0800 (PST) Received: from iago.office.miralink.com (iago.office.miralink.com [10.0.0.40]) by plato.miralink.com (Postfix) with ESMTP id E2BE161B831; Fri, 8 Feb 2008 09:30:26 -0800 (PST) Message-ID: <47AC91B2.4020007@miralink.com> Date: Fri, 08 Feb 2008 09:30:26 -0800 From: Sean Bruno User-Agent: Thunderbird 2.0.0.9 (X11/20071115) MIME-Version: 1.0 To: Kris Kennaway References: <47AC0BC8.3030509@miralink.com> <47AC3D73.2010705@FreeBSD.org> In-Reply-To: <47AC3D73.2010705@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DSPAM-Result: Innocent X-DSPAM-Processed: Fri Feb 8 09:30:27 2008 X-DSPAM-Confidence: 0.9997 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 47ac91b3136941804284693 X-DSPAM-Factors: 27, X-Virus-Scanned: amavisd-new at X-Spam-Status: No, score=-4.499 tagged_above=-10 required=6.6 autolearn=ham tests=[ALL_TRUSTED=-1.8, BAYES_00=-2.599, DSPAM_HAM=-0.1] X-Spam-Score: -4.499 X-Spam-Level: Cc: FreeBSD Hackers Subject: Re: Question about http://torrents.freebsd.org:8080/index.html X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 17:30:28 -0000 Kris Kennaway wrote: > Sean Bruno wrote: >> Does it seem correct to all concerned that each release actually >> lists all files twice? >> >> There is a torrent for the entire release CD ISO set, and then there >> is a completely separate torrent for each CD ISO file. At least that >> is what it looks like to me. >> Is this correct? > > Yes. Many people like to download the whole thing in one torrent. It > does split the aggregate swarm bandwidth, but it doesn't seem to be a > problem in practice. > > Kris Thanks for anticipating my next question! :) <2cents> I just found it odd that that users couldn't just specify the file to download from the release torrent. It also makes it slightly(read almost negligible) harder to seed effectively as I would need to download more than one centralized torrent. Since I use the complete torrent for my purposes, I can only assist the folks who also download the entire torrent. Sean From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 19:24:24 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B27AC16A417 for ; Fri, 8 Feb 2008 19:24:24 +0000 (UTC) (envelope-from sbruno@miralink.com) Received: from plato.miralink.com (mail.miralink.com [70.103.185.20]) by mx1.freebsd.org (Postfix) with ESMTP id 896B213C45D for ; Fri, 8 Feb 2008 19:24:24 +0000 (UTC) (envelope-from sbruno@miralink.com) Received: from localhost (localhost.localdomain [127.0.0.1]) by plato.miralink.com (Postfix) with ESMTP id 4E5B061B967; Fri, 8 Feb 2008 11:24:22 -0800 (PST) Received: from plato.miralink.com ([127.0.0.1]) by localhost (plato.miralink.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 06853-07; Fri, 8 Feb 2008 11:24:21 -0800 (PST) Received: from iago.office.miralink.com (iago.office.miralink.com [10.0.0.40]) by plato.miralink.com (Postfix) with ESMTP id 7A8F261B831; Fri, 8 Feb 2008 11:24:21 -0800 (PST) Message-ID: <47ACAC65.4030601@miralink.com> Date: Fri, 08 Feb 2008 11:24:21 -0800 From: Sean Bruno User-Agent: Thunderbird 2.0.0.9 (X11/20071115) MIME-Version: 1.0 To: Lowell Gilbert References: <47AC0BC8.3030509@miralink.com> <47AC3D73.2010705@FreeBSD.org> <47AC91B2.4020007@miralink.com> <44r6fns1dz.fsf@be-well.ilk.org> In-Reply-To: <44r6fns1dz.fsf@be-well.ilk.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DSPAM-Result: Innocent X-DSPAM-Processed: Fri Feb 8 11:24:22 2008 X-DSPAM-Confidence: 0.9997 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 47acac6678541592213743 X-DSPAM-Factors: 27, X-Virus-Scanned: amavisd-new at X-Spam-Status: No, score=-4.499 tagged_above=-10 required=6.6 autolearn=ham tests=[ALL_TRUSTED=-1.8, BAYES_00=-2.599, DSPAM_HAM=-0.1] X-Spam-Score: -4.499 X-Spam-Level: Cc: freebsd-hackers@freebsd.org Subject: Re: Question about http://torrents.freebsd.org:8080/index.html X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 19:24:24 -0000 Lowell Gilbert wrote: > Sean Bruno writes: > > >> <2cents> >> I just found it odd that that users couldn't just specify the file to >> download from the release torrent. It also makes it slightly(read >> almost negligible) harder to seed effectively as I would need to >> download more than one centralized torrent. Since I use the complete >> torrent for my purposes, I can only assist the folks who also download >> the entire torrent. >> >> >> > > You *can* do it both ways. Right after a release, I often download > the "everything" torrent, then continue to seed that one while also > seeding the partial torrents. I use file links to avoid needing to > download the partial torrents separately. > Ah, that's a good idea. I'll do something like that tonight. That solution doesn't seem to be very obvious to me though. Not that I am an expert in this kind of stuff. /me drops it. Sean From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 20:32:56 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22FCF16A46C; Fri, 8 Feb 2008 20:32:56 +0000 (UTC) (envelope-from cristi@roedu.net) Received: from alpha.roedu.net (alpha.roedu.net [81.180.250.131]) by mx1.freebsd.org (Postfix) with ESMTP id 7FD7C13C4D1; Fri, 8 Feb 2008 20:32:55 +0000 (UTC) (envelope-from cristi@roedu.net) Received: from localhost (localhost [127.0.0.1]) by alpha.roedu.net (Postfix) with ESMTP id 906CF9B659; Fri, 8 Feb 2008 22:32:52 +0200 (EET) X-Virus-Scanned: amavisd-new at roedu.net Received: from alpha.roedu.net ([127.0.0.1]) by localhost (alpha.roedu.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qSlGIAQ7CTOr; Fri, 8 Feb 2008 22:32:51 +0200 (EET) Received: from [172.27.2.200] (c7.campus.utcluj.ro [193.226.6.226]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by alpha.roedu.net (Postfix) with ESMTP id 0D5BD9B305; Fri, 8 Feb 2008 22:32:51 +0200 (EET) Message-ID: <47ACBC73.8070309@roedu.net> Date: Fri, 08 Feb 2008 22:32:51 +0200 From: Cristian KLEIN User-Agent: Thunderbird 2.0.0.6 (X11/20071022) MIME-Version: 1.0 To: Eygene Ryabinkin References: <4c65bf9e-80e5-4c0b-acef-d9914a57679f@i12g2000prf.googlegroups.com> <47728D80.5010507@zirakzigil.org> <475A8637.9020507@zirakzigil.org> <20071212155244.GJ1442@daemon.grid.kiae.ru> <4760114D.6020203@zirakzigil.org> <47950952.6040909@roedu.net> <41549.193.226.5.33.1201087601.squirrel@mail.roedu.net> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Giulio Ferro , nork@freebsd.org Subject: Re: Synaptics X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 20:32:56 -0000 Eygene Ryabinkin a scris: > Cristian, good day. > > Wed, Jan 23, 2008 at 01:26:41PM +0200, cristi@roedu.net wrote: >>>> I just ran into the same problem. In xorg.conf I explicitly told the >>>> synaptics driver to use "psm" and "/dev/psm0", but the error message >>>> would >>>> suggest that it uses "event". >>>> >>>> Also, I tried to change the source code of the synaptics driver >>>> (synaptics.c) and hard-coded "psm" as the only driver, no matter what >>>> xorg.conf says. Synaptics still would not start, but this time >>>> complaining >>>> that no device was specified. Please note that I had "Device" in my >>>> xorg.conf, but the error suggests that the driver ignored it. >> Suppose I use the attached xorg.conf file, at some point, >> /var/log/Xorg.0.log shows the following error: >> >> (II) Synaptics touchpad driver version 0.14.6 (1406) >> Synaptics_Touchpad no synaptics event device found (checked 10 nodes) >> Synaptics_Touchpad The /dev/input/event* device nodes seem to be missing >> (EE) xf86OpenSerial: No Device specified. >> Synaptics driver unable to open device >> (EE) PreInit failed for input device "Synaptics_Touchpad" >> (II) UnloadModule: "synaptics" >> >> As you said, it looks like synaptics is trying to use the "auto" protocol, >> although the configuration file tells it to use "psm". >> >> Now, if I put the attached patch in x11-drivers/synaptics/files, using the >> same xorg.conf, synaptics will fail like this: >> >> (II) Synaptics touchpad driver version 0.14.6 (1406) >> (EE) xf86OpenSerial: No Device specified. >> Synaptics driver unable to open device >> (EE) PreInit failed for input device "Synaptics_Touchpad" >> (II) UnloadModule: "synaptics" >> >> It almost looks as if I have to hardcode the device too, because synaptics >> certainly ignores my options. > > This is a sort of 'ping' mail, sorry. To the point: I had reproduced > the problem and will start looking into it once this message will > fly from my mailserver. Stay tuned ;)) Thank you very much for 'pinging'. :) As the driver synaptics driver din not change for month, I speculate there might have been a change in the Xorg API, especially related to sucking configuration options. Is there any way I could help? From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 20:54:22 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DAD6516A417 for ; Fri, 8 Feb 2008 20:54:22 +0000 (UTC) (envelope-from aduane@juniper.net) Received: from exprod7og105.obsmtp.com (exprod7og105.obsmtp.com [64.18.2.163]) by mx1.freebsd.org (Postfix) with ESMTP id 7DA1013C46B for ; Fri, 8 Feb 2008 20:54:22 +0000 (UTC) (envelope-from aduane@juniper.net) Received: from source ([66.129.224.36]) by exprod7ob105.postini.com ([64.18.6.12]) with SMTP; Fri, 08 Feb 2008 12:54:22 PST Received: from proton.jnpr.net ([10.10.2.37]) by emailsmtp56.jnpr.net with Microsoft SMTPSVC(6.0.3790.3959); Fri, 8 Feb 2008 12:52:42 -0800 Received: from antipi.jnpr.net ([10.10.2.34]) by proton.jnpr.net with Microsoft SMTPSVC(6.0.3790.1830); Fri, 8 Feb 2008 15:52:24 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Fri, 8 Feb 2008 15:52:23 -0500 Message-ID: <0FCFCF6165E968449991746EB91D614DF7722E@antipi.jnpr.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: NOR flash drivers Thread-Index: AchqlIFCkxil66WhQzChZKsyjj229Q== From: "Andrew Duane" To: , X-OriginalArrivalTime: 08 Feb 2008 20:52:24.0699 (UTC) FILETIME=[81E45CB0:01C86A94] Cc: Subject: NOR flash drivers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 20:54:22 -0000 I may have asked this once before, but are any NOR flash drivers available in FreeBSD? I have support that I put into the bootstrap I'm building, interfaced to libstand, but it is read-only, and far from useful for a real UFS. -- Andrew Duane Juniper Networks 978-589-0551 10 Technology Park Dr aduane@juniper.net Westford, MA 01886-3418 =20 From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 20:55:53 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 617F216A47C for ; Fri, 8 Feb 2008 20:55:53 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outA.internet-mail-service.net (outA.internet-mail-service.net [216.240.47.224]) by mx1.freebsd.org (Postfix) with ESMTP id 45D8313C461 for ; Fri, 8 Feb 2008 20:55:53 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Fri, 08 Feb 2008 12:55:52 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id 8AC5412715B; Fri, 8 Feb 2008 12:55:51 -0800 (PST) Message-ID: <47ACC1DD.8010105@elischer.org> Date: Fri, 08 Feb 2008 12:55:57 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: =?UTF-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= References: <86prv7agit.fsf@ds4.des.no> In-Reply-To: <86prv7agit.fsf@ds4.des.no> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: hackers@freebsd.org Subject: Re: nvi strangeness X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 20:55:53 -0000 Dag-Erling Smørgrav wrote: > As everybody knows, the ex/vi command for regexp substitution is > > [range] s[ubstitute] [/pattern/replace/] [options] [count] [flags] > > The man page does not document the syntax for ranges (nor for offsets or > counts for that matter) > > Assuming the syntax is the same as for ed / sed, the following should > replace every occurrence of the word wait with the word delta: > > :,s/\/delta/g > > where "," means "the entire file" the symbol for end of file is $ so 1,$ > > That doesn't work, however, and neither does "1,"; only "X,Y" works, so > if you want to substitute in the entire file, you first have to find out > how long it is, then manually type in that number. > > ...or you could use the following (courtesy of roberto@): > > :%s/\/delta/g :g/pattern/s//replacement/g :1,$s/patter/replacement/g > > None of this is documented anywhere. > > I would really like nvi to understand ed-like ranges, and vi(1) to > either document this or cross-reference ed(1). > > Going off on a tangent, I wonder why the nvi license is reproduced in > the man page, as it doesn't seem to require it, and no other man page > includes the license of the program it describes (with a single > exception: groff) > > DES From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 21:18:10 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7026316A41A for ; Fri, 8 Feb 2008 21:18:10 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail06.syd.optusnet.com.au (mail06.syd.optusnet.com.au [211.29.132.187]) by mx1.freebsd.org (Postfix) with ESMTP id 029A413C448 for ; Fri, 8 Feb 2008 21:18:09 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from server.vk2pj.dyndns.org (c220-239-20-82.belrs4.nsw.optusnet.com.au [220.239.20.82]) by mail06.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m18LI3hw003359 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 9 Feb 2008 08:18:04 +1100 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.2/8.14.1) with ESMTP id m18LI20K087749; Sat, 9 Feb 2008 08:18:02 +1100 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.2/8.14.2/Submit) id m18LI2NY087748; Sat, 9 Feb 2008 08:18:02 +1100 (EST) (envelope-from peter) Date: Sat, 9 Feb 2008 08:18:02 +1100 From: Peter Jeremy To: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= Message-ID: <20080208211802.GJ4008@server.vk2pj.dyndns.org> References: <86prv7agit.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="m46qSNjkc66Ye11q" Content-Disposition: inline In-Reply-To: <86prv7agit.fsf@ds4.des.no> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.17 (2007-11-01) Cc: hackers@freebsd.org Subject: Re: nvi strangeness X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 21:18:10 -0000 --m46qSNjkc66Ye11q Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Feb 08, 2008 at 11:28:42AM +0100, Dag-Erling Sm=F8rgrav wrote: >As everybody knows, the ex/vi command for regexp substitution is > > [range] s[ubstitute] [/pattern/replace/] [options] [count] [flags] > >The man page does not document the syntax for ranges (nor for offsets or >counts for that matter) It definitely would be nice if vi(1) documented the various [foo] terms it uses and included explicit pointers to /usr/share/doc/{12.vi,13.viref} instead of just hinting that they exist. >Assuming the syntax is the same as for ed / sed, the following should >replace every occurrence of the word wait with the word delta: > >:,s/\/delta/g > >where "," means "the entire file" Actually, ',s/foo/bar/' is illegal in sed ('s/foo/bar/' means every line). That means the same syntax means three different things in three different editors. >I would really like nvi to understand ed-like ranges, and vi(1) to >either document this or cross-reference ed(1). Given that a bare comma means different things in ed and vi, I don't think that the meanings can be aligned because it would cause too much pain. Comparing the various behaviours, IMHO, the ed ',s/foo/bar/' behaviour is the anomoly: In vi, a null address in a range is the current line ('.' in ed). In both ed and sed, a null address in a range is illegal, except that two null addresses in ed refer to the entire file. --=20 Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. --m46qSNjkc66Ye11q Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHrMcK/opHv/APuIcRAmVEAJ96gAZ7Qie/QSXv4TsLRH0B6PnoEACeMswq 3u9/P01/gVmWFASwq1dYRiY= =U/tz -----END PGP SIGNATURE----- --m46qSNjkc66Ye11q-- From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 21:22:29 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1FE8816A421; Fri, 8 Feb 2008 21:22:29 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id B884313C46B; Fri, 8 Feb 2008 21:22:28 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.2/8.14.1) with ESMTP id m18LIeYO013120; Fri, 8 Feb 2008 14:18:41 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Fri, 08 Feb 2008 14:21:38 -0700 (MST) Message-Id: <20080208.142138.-1417611621.imp@bsdimp.com> To: aduane@juniper.net From: "M. Warner Losh" In-Reply-To: <0FCFCF6165E968449991746EB91D614DF7722E@antipi.jnpr.net> References: <0FCFCF6165E968449991746EB91D614DF7722E@antipi.jnpr.net> X-Mailer: Mew version 5.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, freebsd-embedded@freebsd.org Subject: Re: NOR flash drivers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 21:22:29 -0000 In message: <0FCFCF6165E968449991746EB91D614DF7722E@antipi.jnpr.net> "Andrew Duane" writes: : I may have asked this once before, but are any NOR flash drivers : available in FreeBSD? I have support that I put into the bootstrap I'm : building, interfaced to libstand, but it is read-only, and far from : useful for a real UFS. I believe that there's a dev/cfi floating around out there. I believe this supports NOR, but I get it and NAND confused. It is in the e500 branch of perforce. I think Marcel committed it. I don't know how close it is to working. Warner From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 22:08:16 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7821216A41A for ; Fri, 8 Feb 2008 22:08:16 +0000 (UTC) (envelope-from heliocentric@gmail.com) Received: from wr-out-0506.google.com (wr-out-0506.google.com [64.233.184.226]) by mx1.freebsd.org (Postfix) with ESMTP id 2A32113C442 for ; Fri, 8 Feb 2008 22:08:15 +0000 (UTC) (envelope-from heliocentric@gmail.com) Received: by wr-out-0506.google.com with SMTP id 68so3703268wri.3 for ; Fri, 08 Feb 2008 14:08:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=tYB6ipEdXPE3emjgn5cfChCBGYxxq3L7OYMmN2iYATg=; b=wxgGj7keH/izrgu9OdPlFUdIzB0+Xw4fxFXTWUp0LqMxYmDcQWiVBqLyquq66wVl3WjbnY+XYhqF7BpaYHzrxrYUfMLamyH1qlDQIBYaFjH0WP9muIpdsPHyb1NFUtVedNEH6rhHSdJ4vgCNecNFeMsuN6ORA8MjViiHrjlDJNQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=Kw8wlZTR8YKs6eiqPKB864F5QTMw9VrCeuzRJOW7vdMEQbFEO03VCqG6es1odexH8semKsu4QSuUPwqmL1rWCPXK0D8z/520+jG8BwWKf+wd3EMUAR0YvWwrxTirXid/lJjo//F7RD7lAL4/vu8tsr2bOo06mIFgvnYEVt99tWY= Received: by 10.142.50.15 with SMTP id x15mr7349076wfx.169.1202506988200; Fri, 08 Feb 2008 13:43:08 -0800 (PST) Received: by 10.142.144.2 with HTTP; Fri, 8 Feb 2008 13:43:08 -0800 (PST) Message-ID: Date: Fri, 8 Feb 2008 16:43:08 -0500 From: "Dylan Cochran" Sender: heliocentric@gmail.com To: "Adrian Penisoara" , freebsd-hackers@freebsd.org In-Reply-To: <78cb3d3f0802011424w46fb71fbp7169a5e60ee5718f@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <78cb3d3f0801201239x18c339a6sbc6a04d2b5670b81@mail.gmail.com> <4793C977.9000803@freebsd.org> <78cb3d3f0802011424w46fb71fbp7169a5e60ee5718f@mail.gmail.com> X-Google-Sender-Auth: b42e058b3e3f12bf Cc: Subject: Re: Squeezing out some 70 bytes out of the boot2 loader X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 22:08:16 -0000 ... > > > [1] I'm trying to get support for /boot being mounted as a separate FS > > and > > > as such I would need to have a "self-pointing" symlink (e.g. "boot -> ." > > ) You're going about this the wrong way; no need to modify the boot2 source, just place a boot.config in the root of the /boot, with a single line: '/loader', then everything works fine. From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 8 19:36:29 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E16416A46C for ; Fri, 8 Feb 2008 19:36:29 +0000 (UTC) (envelope-from lgusenet@be-well.ilk.org) Received: from be-well.ilk.org (dsl092-078-145.bos1.dsl.speakeasy.net [66.92.78.145]) by mx1.freebsd.org (Postfix) with ESMTP id 0347D13C4EC for ; Fri, 8 Feb 2008 19:36:28 +0000 (UTC) (envelope-from lgusenet@be-well.ilk.org) Received: by be-well.ilk.org (Postfix, from userid 1147) id 6542128430; Fri, 8 Feb 2008 14:18:16 -0500 (EST) To: freebsd-hackers@freebsd.org, Sean Bruno References: <47AC0BC8.3030509@miralink.com> <47AC3D73.2010705@FreeBSD.org> <47AC91B2.4020007@miralink.com> From: Lowell Gilbert Date: Fri, 08 Feb 2008 14:18:16 -0500 In-Reply-To: <47AC91B2.4020007@miralink.com> (Sean Bruno's message of "Fri\, 08 Feb 2008 09\:30\:26 -0800") Message-ID: <44r6fns1dz.fsf@be-well.ilk.org> User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailman-Approved-At: Fri, 08 Feb 2008 23:06:14 +0000 Cc: Subject: Re: Question about http://torrents.freebsd.org:8080/index.html X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 19:36:29 -0000 Sean Bruno writes: > <2cents> > I just found it odd that that users couldn't just specify the file to > download from the release torrent. It also makes it slightly(read > almost negligible) harder to seed effectively as I would need to > download more than one centralized torrent. Since I use the complete > torrent for my purposes, I can only assist the folks who also download > the entire torrent. > > You *can* do it both ways. Right after a release, I often download the "everything" torrent, then continue to seed that one while also seeding the partial torrents. I use file links to avoid needing to download the partial torrents separately. From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 9 06:30:04 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4494B16A41B; Sat, 9 Feb 2008 06:30:04 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id EEB0413C45A; Sat, 9 Feb 2008 06:30:03 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender:X-Spam-Status:Subject; b=NK2tEr93v6ZHPpmBUke+1z46Stnnkne+OasYgMeakXxYZGT/vrPXsNQlrvkvqBO3vgsvTDRkxMRxV7ZPNEzyQfN8C+RGgQ4cZ87kLQD94APsVG7fenQxUOmMvBv8rMbFkoK6bX2Z/mLDt5fj+ibLDbg8KjB5dRKPS76wEAasd5w=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1JNjDt-000Nto-7e; Sat, 09 Feb 2008 09:30:01 +0300 Date: Sat, 9 Feb 2008 09:29:59 +0300 From: Eygene Ryabinkin To: Cristian KLEIN Message-ID: References: <47728D80.5010507@zirakzigil.org> <475A8637.9020507@zirakzigil.org> <20071212155244.GJ1442@daemon.grid.kiae.ru> <4760114D.6020203@zirakzigil.org> <47950952.6040909@roedu.net> <41549.193.226.5.33.1201087601.squirrel@mail.roedu.net> <47ACBC73.8070309@roedu.net> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <47ACBC73.8070309@roedu.net> Sender: rea-fbsd@codelabs.ru X-Spam-Status: No, score=-1.8 required=4.0 tests=ALL_TRUSTED,AWL,BAYES_50 Cc: freebsd-hackers@freebsd.org, Giulio Ferro , nork@freebsd.org Subject: Re: Synaptics X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 06:30:04 -0000 Cristian, good day. Fri, Feb 08, 2008 at 10:32:51PM +0200, Cristian KLEIN wrote: > Thank you very much for 'pinging'. :) As the driver synaptics driver din > not change for month, I speculate there might have been a change in the > Xorg API, especially related to sucking configuration options. No, I had found the first problem and sent the message to the list shortly after the 'ping' mail. It has the patch attached. If you had not received it, it is strange, but anyway, here is the URL http://lists.freebsd.org/pipermail/freebsd-hackers/2008-February/023272.html Make mistakenly thought that your Synaptics driver should be compiled for i386, not for amd64. Patch will change the situation back to normal. > Is there any way I could help? Yes, please, try the mentioned patch and report back ;)) -- Eygene From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 9 19:26:37 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D869216A41B for ; Sat, 9 Feb 2008 19:26:37 +0000 (UTC) (envelope-from chuckr@chuckr.org) Received: from mail6.sea5.speakeasy.net (mail6.sea5.speakeasy.net [69.17.117.8]) by mx1.freebsd.org (Postfix) with ESMTP id BBD7913C442 for ; Sat, 9 Feb 2008 19:26:37 +0000 (UTC) (envelope-from chuckr@chuckr.org) Received: (qmail 9210 invoked from network); 9 Feb 2008 19:26:37 -0000 Received: from april.chuckr.org (chuckr@[66.92.151.30]) (envelope-sender ) by mail6.sea5.speakeasy.net (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 9 Feb 2008 19:26:37 -0000 Message-ID: <47ADFD2C.80009@chuckr.org> Date: Sat, 09 Feb 2008 14:21:16 -0500 From: Chuck Robey User-Agent: Thunderbird 2.0.0.6 (X11/20071107) MIME-Version: 1.0 To: FreeBSD-Hackers X-Enigmail-Version: 0.95.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: USB Graphic Tablets X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 19:26:37 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I just laid hands on a UC-Logic WP8060-TAB08 Graphic tablet, so as to make my work in gimp easier I got this one instead of a Wacom unit for the single obvious reason: $$. It seems to have all the features of the big boys, it's 8" by 6", 1024 intensity levels, chrome bumpers, dual carbs, etc. Anyhow, the probing returns, on my FreeBSD-current system: ums0: on uhub0 ums0: X report 0x0002 not supported device_attach: ums0 attach returned 6 and in /dev, there isn;'t any ums0 device to be found, and that error #6 is (appropriately enough) Device Not Configured. Well, my first question is, does there exist a tool for USB that let's you view the raw return from the usb probing? I want to see what this device is actually identifying itself as. Actual numeric vendor IDs and product IDs are the sort of thing I'm after. Something a whole lot more detailed that usbdevs, please. Even the dmesg listing (as I show above) hasn't got the numbers and tails I need. Secondly, if I find out (I've contacted a friendly fellow at the manufacturer) that this device works very much like, say, a Wacom device, is there any way to fake out the kernel and force it to think that the device I'm showing it is actually something else, some alias? If there isn't some fairly direct method of doing this, then could someone give me some sort of description of how I might go about sticking in a fake entry that works exactly as a present entry? I figure at least part of it is find out out that object file holds the intended alias target, and also mucking about with the usbdevs file, what else might be required? If it turns out that a completely new driver is required, then I need no immediate answer, because (like I said) I wrote the manufacturer, and I figure they'll take a few years to reply with technical details, and probably tell me how great their Windows driver is. Thanks for whatever I can get. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHrf0sz62J6PPcoOkRAn1dAJ4moCo3iBp4WlNF3/sVH14zKR23fgCePt7B xyKtj04Z/O7oCZKS5fxr2wY= =TE+n -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 9 19:49:59 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E9FF16A417 for ; Sat, 9 Feb 2008 19:49:59 +0000 (UTC) (envelope-from jhs@berklix.org) Received: from tower.berklix.org (tower.berklix.org [83.236.223.114]) by mx1.freebsd.org (Postfix) with ESMTP id 2466213C448 for ; Sat, 9 Feb 2008 19:49:58 +0000 (UTC) (envelope-from jhs@berklix.org) Received: from js.berklix.net (p549A4EF9.dip.t-dialin.net [84.154.78.249]) (authenticated bits=0) by tower.berklix.org (8.13.6/8.13.6) with ESMTP id m19Jnqp4065757; Sat, 9 Feb 2008 19:49:53 GMT (envelope-from jhs@berklix.org) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by js.berklix.net (8.13.8/8.13.8) with ESMTP id m19JpTlY034048; Sat, 9 Feb 2008 20:51:29 +0100 (CET) (envelope-from jhs@berklix.org) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.13.8/8.13.8) with ESMTP id m19JpJfj096492; Sat, 9 Feb 2008 20:51:24 +0100 (CET) (envelope-from jhs@fire.js.berklix.net) Message-Id: <200802091951.m19JpJfj096492@fire.js.berklix.net> To: Chuck Robey In-reply-to: <47ADFD2C.80009@chuckr.org> References: <47ADFD2C.80009@chuckr.org> Comments: In-reply-to Chuck Robey message dated "Sat, 09 Feb 2008 14:21:16 -0500." Date: Sat, 09 Feb 2008 20:51:19 +0100 From: "Julian H. Stacey" Cc: FreeBSD-Hackers Subject: Re: USB Graphic Tablets X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 19:49:59 -0000 Chuck Robey wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I just laid hands on a UC-Logic WP8060-TAB08 Graphic tablet, so as to make > my work in gimp easier I got this one instead of a Wacom unit for the > single obvious reason: $$. It seems to have all the features of the big > boys, it's 8" by 6", 1024 intensity levels, chrome bumpers, dual carbs, etc. > > Anyhow, the probing returns, on my FreeBSD-current system: > > ums0: on uhub0 > ums0: X report 0x0002 not supported > device_attach: ums0 attach returned 6 > > and in /dev, there isn;'t any ums0 device to be found, and that error #6 is > (appropriately enough) Device Not Configured. > > Well, my first question is, does there exist a tool for USB that let's you > view the raw return from the usb probing? I want to see what this device > is actually identifying itself as. Actual numeric vendor IDs and product > IDs are the sort of thing I'm after. Something a whole lot more detailed > that usbdevs, please. Even the dmesg listing (as I show above) hasn't got > the numbers and tails I need. Hi Chuck, On list sane-devel@lists.alioth.debian.org (A mostly Linux using crownd though not just, some BSD) They mostly seem to rec. some (free I assume) MS$ comparible .exe prog for snatching USB traffic to then analyse / develop for *IX drivers. I don't remember prog name, but http://lists.alioth.debian.org/mailman/listinfo/sane-devel Julian -- Julian Stacey. BSD Unix Linux Net Consultant, Munich. http://berklix.com From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 9 20:08:23 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A684016A418 for ; Sat, 9 Feb 2008 20:08:23 +0000 (UTC) (envelope-from yngwiie@bk.ru) Received: from k2smtpout01-02.prod.mesa1.secureserver.net (k2smtpout01-02.prod.mesa1.secureserver.net [64.202.189.89]) by mx1.freebsd.org (Postfix) with SMTP id 5F54613C45D for ; Sat, 9 Feb 2008 20:08:23 +0000 (UTC) (envelope-from yngwiie@bk.ru) Received: (qmail 22215 invoked from network); 9 Feb 2008 19:41:42 -0000 Received: from unknown (HELO vh.kytex.net) (208.109.92.156) by k2smtpout01-02.prod.mesa1.secureserver.net (64.202.189.89) with ESMTP; 09 Feb 2008 19:41:42 -0000 Received: (qmail 16030 invoked from network); 9 Feb 2008 21:21:42 +0200 Received: from unknown (HELO dune) (89.252.214.155) by ip-208-109-92-149.ip.secureserver.net with SMTP; 9 Feb 2008 21:21:42 +0200 Date: Sat, 9 Feb 2008 21:21:27 +0200 From: Ivan Georgiev To: hackers@freebsd.org Message-Id: <20080209212127.b1b4c4fb.yngwiie@bk.ru> Organization: Ivan Georgiev X-Mailer: Sylpheed 2.4.8 (GTK+ 2.12.7; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: Subject: quotactl returns double values X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 20:08:23 -0000 Hello, I was just playing, trying to see how quotactl works, but in all my tries the values returned are doubled, here is an example: #include #include #include #include #include int main(int argc, char* argv[]) { struct dqblk *my_st; const char* path = "/usr"; int uid; my_st = malloc(sizeof(struct dqblk)); uid = atoi(argv[1]); if((quotactl(path,QCMD(Q_GETQUOTA,USRQUOTA),uid,my_st)) < 0) { free(my_st); perror("error"); exit(-1); } printf("User Hard Limit : %d\n \ Actual Usage : %d\n", my_st->dqb_bhardlimit, \ my_st->dqb_curblocks); free(my_st); return 0; } the values stored in my_st->dqb_bhardlimit and my_st->dqb_curblocks are the real values times two, the hard limit on the user is 102400K, but my_st->dqb_bhardlimit holds 204800K. Could someone please tell me what i am doing wrong. Thanks in advance. Regards: Ivan From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 9 20:29:48 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3314816A421 for ; Sat, 9 Feb 2008 20:29:48 +0000 (UTC) (envelope-from chuckr@chuckr.org) Received: from mail2.sea5.speakeasy.net (mail2.sea5.speakeasy.net [69.17.117.4]) by mx1.freebsd.org (Postfix) with ESMTP id 132EA13C447 for ; Sat, 9 Feb 2008 20:29:47 +0000 (UTC) (envelope-from chuckr@chuckr.org) Received: (qmail 32731 invoked from network); 9 Feb 2008 20:29:47 -0000 Received: from april.chuckr.org (chuckr@[66.92.151.30]) (envelope-sender ) by mail2.sea5.speakeasy.net (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 9 Feb 2008 20:29:46 -0000 Message-ID: <47AE0BFA.5030007@chuckr.org> Date: Sat, 09 Feb 2008 15:24:26 -0500 From: Chuck Robey User-Agent: Thunderbird 2.0.0.6 (X11/20071107) MIME-Version: 1.0 To: "Julian H. Stacey" References: <47ADFD2C.80009@chuckr.org> <200802091951.m19JpJfj096492@fire.js.berklix.net> In-Reply-To: <200802091951.m19JpJfj096492@fire.js.berklix.net> X-Enigmail-Version: 0.95.5 OpenPGP: id=F3DCA0E9; url=http://pgp.mit.edu Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: FreeBSD-Hackers Subject: Re: USB Graphic Tablets X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 20:29:48 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Julian H. Stacey wrote: > Chuck Robey wrote: >> Well, my first question is, does there exist a tool for USB that let's you >> view the raw return from the usb probing? I want to see what this device >> is actually identifying itself as. Actual numeric vendor IDs and product >> IDs are the sort of thing I'm after. Something a whole lot more detailed >> that usbdevs, please. Even the dmesg listing (as I show above) hasn't got >> the numbers and tails I need. > > Hi Chuck, > On list sane-devel@lists.alioth.debian.org (A mostly Linux using > crownd though not just, some BSD) They mostly seem to rec. some > (free I assume) MS$ comparible .exe prog for snatching USB traffic > to then analyse / develop for *IX drivers. I don't remember prog name, > but http://lists.alioth.debian.org/mailman/listinfo/sane-devel > Julian Thanks much for the URL, I will definitely look it up. I'm a bit surprised that the SANE folks, who I took to be scanner-oriented, should be interested in graphic tablets. I know, from having used it in the days before I switched back to FreeBSD, that the verbose mode of the Linux usb lister gives all the info you could possibly ask for, I would really rather not use the Linux tool to do that, both for feelings of loyalty, and to be honest, my legs are too shot, and getting back and forth to my now moved Linux machine with the graphic pad is unfun. Maybe I will look at usbdevs and see if it's hackable to make it have a blabbermouth mode. Or, maybe the verbose mode of booting might be more verbose for USB devices? As far as the other side of things, to have a usb device that is identical to one on the list, except the mfr has moved the ident info, and have that driver be inaccessible to FreeBSD seems awfully straitlaced to me. Is it a feeling of the usb kernel developers involved that thjey should impose that level of control on folks? What, in the name of security, maybe? It seems that they might maybe protect it to some lenth, but to remove the abilityu to experiment easily, that sounds a bit too draconian. Is this really the sense of folks here, that no user-provoked experimentation should be allowed? -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHrgv6z62J6PPcoOkRAlKoAJ46/zBH/n0Mr2uaUnhF87IR1a8MsgCgm6Lz iaCPsQ1uPPcfGdl3/F+fJ4E= =QjY6 -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 9 20:40:46 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 101DF16A418 for ; Sat, 9 Feb 2008 20:40:46 +0000 (UTC) (envelope-from yngwiie@bk.ru) Received: from mx28.mail.ru (mx28.mail.ru [194.67.23.67]) by mx1.freebsd.org (Postfix) with ESMTP id C540113C4D1 for ; Sat, 9 Feb 2008 20:40:45 +0000 (UTC) (envelope-from yngwiie@bk.ru) Received: from mx27.mail.ru (mx27.mail.ru [194.67.23.23]) by mx28.mail.ru (mPOP.Fallback_MX) with ESMTP id 6C7D731C27C for ; Sat, 9 Feb 2008 23:02:22 +0300 (MSK) Received: from [89.252.214.155] (port=60536 helo=dune) by mx27.mail.ru with asmtp id 1JNvu0-000Inv-00 for freebsd-hackers@freebsd.org; Sat, 09 Feb 2008 23:02:20 +0300 Date: Sat, 9 Feb 2008 22:02:06 +0200 From: Ivan Georgiev To: freebsd-hackers@freebsd.org Message-Id: <20080209220206.695e5d9b.yngwiie@bk.ru> Organization: Ivan Georgiev X-Mailer: Sylpheed 2.4.8 (GTK+ 2.12.7; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: quotactl returns double values X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 20:40:46 -0000 Hello, I was just playing, trying to see how quotactl works, but in all my tries the values returned are doubled, here is an example: #include #include #include #include #include int main(int argc, char* argv[]) { struct dqblk *my_st; const char* path = "/usr"; int uid; my_st = malloc(sizeof(struct dqblk)); uid = atoi(argv[1]); if((quotactl(path,QCMD(Q_GETQUOTA,USRQUOTA),uid,my_st)) < 0) { free(my_st); perror("error"); exit(-1); } printf("User Hard Limit : %d\n \ Actual Usage : %d\n", my_st->dqb_bhardlimit, \ my_st->dqb_curblocks); free(my_st); return 0; } the values stored in my_st->dqb_bhardlimit and my_st->dqb_curblocks are the real values times two, the hard limit on the user is 102400K, but my_st->dqb_bhardlimit holds 204800K. Could someone please tell me what i am doing wrong. Thanks in advance. Regards: Ivan From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 9 22:04:04 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D35DC16A417 for ; Sat, 9 Feb 2008 22:04:04 +0000 (UTC) (envelope-from doconnor@gsoft.com.au) Received: from cain.gsoft.com.au (cain.gsoft.com.au [203.31.81.10]) by mx1.freebsd.org (Postfix) with ESMTP id 7311913C442 for ; Sat, 9 Feb 2008 22:04:04 +0000 (UTC) (envelope-from doconnor@gsoft.com.au) Received: from inchoate.gsoft.com.au (ppp121-45-81-160.lns10.adl6.internode.on.net [121.45.81.160]) (authenticated bits=0) by cain.gsoft.com.au (8.13.8/8.13.8) with ESMTP id m19M3sFk051386 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 10 Feb 2008 08:33:55 +1030 (CST) (envelope-from doconnor@gsoft.com.au) From: "Daniel O'Connor" To: freebsd-hackers@freebsd.org Date: Sun, 10 Feb 2008 08:33:20 +1030 User-Agent: KMail/1.9.7 References: <20080209220206.695e5d9b.yngwiie@bk.ru> In-Reply-To: <20080209220206.695e5d9b.yngwiie@bk.ru> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1693033.5eWXbDU3C1"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200802100833.36917.doconnor@gsoft.com.au> X-Spam-Score: -2.212 () BAYES_00,RDNS_DYNAMIC X-Scanned-By: MIMEDefang 2.63 on 203.31.81.10 Cc: Ivan Georgiev Subject: Re: quotactl returns double values X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 22:04:04 -0000 --nextPart1693033.5eWXbDU3C1 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Sun, 10 Feb 2008, Ivan Georgiev wrote: > the values stored in my_st->dqb_bhardlimit and my_st->dqb_curblocks > are the real values times two, the hard limit on the user is 102400K, > but my_st->dqb_bhardlimit holds 204800K. > > Could someone please tell me what i am doing wrong. =46rom my reading of the header I would say the quota is reported/set in=20 terms of the number of 512 byte disk blocks rather than in kilobytes. =2D-=20 Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C --nextPart1693033.5eWXbDU3C1 Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQBHriM45ZPcIHs/zowRAhG6AJ9ILUtn5hVT/xR/hbxpdQB9a0xcJgCdGjGL XIL5FstdmDX8xvHabdVt8pU= =1Du+ -----END PGP SIGNATURE----- --nextPart1693033.5eWXbDU3C1-- From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 9 22:15:25 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32D4D16A420 for ; Sat, 9 Feb 2008 22:15:25 +0000 (UTC) (envelope-from jhs@berklix.org) Received: from tower.berklix.org (tower.berklix.org [83.236.223.114]) by mx1.freebsd.org (Postfix) with ESMTP id A7FD513C4D5 for ; Sat, 9 Feb 2008 22:15:24 +0000 (UTC) (envelope-from jhs@berklix.org) Received: from js.berklix.net (p549A5434.dip.t-dialin.net [84.154.84.52]) (authenticated bits=0) by tower.berklix.org (8.13.6/8.13.6) with ESMTP id m19MFJSm066312; Sat, 9 Feb 2008 22:15:19 GMT (envelope-from jhs@berklix.org) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by js.berklix.net (8.13.8/8.13.8) with ESMTP id m19MGvCr034949; Sat, 9 Feb 2008 23:16:57 +0100 (CET) (envelope-from jhs@berklix.org) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.13.8/8.13.8) with ESMTP id m19MGgte098739; Sat, 9 Feb 2008 23:16:52 +0100 (CET) (envelope-from jhs@fire.js.berklix.net) Message-Id: <200802092216.m19MGgte098739@fire.js.berklix.net> To: Chuck Robey In-reply-to: <47AE0BFA.5030007@chuckr.org> References: <47ADFD2C.80009@chuckr.org> <200802091951.m19JpJfj096492@fire.js.berklix.net> <47AE0BFA.5030007@chuckr.org> Comments: In-reply-to Chuck Robey message dated "Sat, 09 Feb 2008 15:24:26 -0500." Date: Sat, 09 Feb 2008 23:16:42 +0100 From: "Julian H. Stacey" Cc: FreeBSD-Hackers Subject: Re: USB Graphic Tablets X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 22:15:25 -0000 Chuck Robey wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Julian H. Stacey wrote: > > Chuck Robey wrote: > >> Well, my first question is, does there exist a tool for USB that let's you > >> view the raw return from the usb probing? I want to see what this device > >> is actually identifying itself as. Actual numeric vendor IDs and product > >> IDs are the sort of thing I'm after. Something a whole lot more detailed > >> that usbdevs, please. Even the dmesg listing (as I show above) hasn't got > >> the numbers and tails I need. > > > > Hi Chuck, > > On list sane-devel@lists.alioth.debian.org (A mostly Linux using > > crownd though not just, some BSD) They mostly seem to rec. some > > (free I assume) MS$ comparible .exe prog for snatching USB traffic > > to then analyse / develop for *IX drivers. I don't remember prog name, > > but http://lists.alioth.debian.org/mailman/listinfo/sane-devel > > Julian > > Thanks much for the URL, I will definitely look it up. I'm a bit surprised > that the SANE folks, who I took to be scanner-oriented, should be > interested in graphic tablets. Err, they're not far as I know, but they often seem to want to grab byte streams on scanners to then analyse for Sane. I guess what one wants to later use the grabbed traffic for doesnt matter, so long as one has the traffic grabbed for analysis ? > Is it a feeling of the usb kernel developers involved that thjey .... Maybe current@ or usb@ might know more than hackers@ -- Julian Stacey. BSD Unix Linux Net Consultant, Munich. http://berklix.com From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 9 22:21:33 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C99D816A41A for ; Sat, 9 Feb 2008 22:21:33 +0000 (UTC) (envelope-from chuckr@chuckr.org) Received: from mail8.sea5.speakeasy.net (mail8.sea5.speakeasy.net [69.17.117.10]) by mx1.freebsd.org (Postfix) with ESMTP id A860613C461 for ; Sat, 9 Feb 2008 22:21:33 +0000 (UTC) (envelope-from chuckr@chuckr.org) Received: (qmail 17762 invoked from network); 9 Feb 2008 22:21:33 -0000 Received: from april.chuckr.org (chuckr@[66.92.151.30]) (envelope-sender ) by mail8.sea5.speakeasy.net (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 9 Feb 2008 22:21:32 -0000 Message-ID: <47AE2628.3070500@chuckr.org> Date: Sat, 09 Feb 2008 17:16:08 -0500 From: Chuck Robey User-Agent: Thunderbird 2.0.0.6 (X11/20071107) MIME-Version: 1.0 To: "Julian H. Stacey" References: <47ADFD2C.80009@chuckr.org> <200802091951.m19JpJfj096492@fire.js.berklix.net> <47AE0BFA.5030007@chuckr.org> <200802092216.m19MGgte098739@fire.js.berklix.net> In-Reply-To: <200802092216.m19MGgte098739@fire.js.berklix.net> X-Enigmail-Version: 0.95.5 OpenPGP: id=F3DCA0E9; url=http://pgp.mit.edu Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: FreeBSD-Hackers Subject: Re: USB Graphic Tablets X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 22:21:33 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Julian H. Stacey wrote: > Chuck Robey wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Julian H. Stacey wrote: >>> Chuck Robey wrote: >>>> Well, my first question is, does there exist a tool for USB that let's you >>>> view the raw return from the usb probing? I want to see what this device >>>> is actually identifying itself as. Actual numeric vendor IDs and product >>>> IDs are the sort of thing I'm after. Something a whole lot more detailed >>>> that usbdevs, please. Even the dmesg listing (as I show above) hasn't got >>>> the numbers and tails I need. >>> Hi Chuck, >>> On list sane-devel@lists.alioth.debian.org (A mostly Linux using >>> crownd though not just, some BSD) They mostly seem to rec. some >>> (free I assume) MS$ comparible .exe prog for snatching USB traffic >>> to then analyse / develop for *IX drivers. I don't remember prog name, >>> but http://lists.alioth.debian.org/mailman/listinfo/sane-devel >>> Julian >> Thanks much for the URL, I will definitely look it up. I'm a bit surprised >> that the SANE folks, who I took to be scanner-oriented, should be >> interested in graphic tablets. > > Err, they're not far as I know, but they often seem to want to grab > byte streams on scanners to then analyse for Sane. I guess what > one wants to later use the grabbed traffic for doesnt matter, so > long as one has the traffic grabbed for analysis ? Well ... OK, I'm having just a bit of a hard time seeing how that would help me in trying to get my graphic tablet working on FreeBSD for my dear old gimp, but I know darn well you're right more often than I am, so I will give a good look at that list anyhow. > >> Is it a feeling of the usb kernel developers involved that thjey .... > > Maybe current@ or usb@ might know more than hackers@ > Good suggestion, I'll act on it, thanks -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHriYoz62J6PPcoOkRAsfdAJ473iGX+wqmVv29WJ2O9Z8Qj7O3wgCePfH8 dd3BoXE2quZdCs2U2oTF3SA= =dYZc -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 9 23:10:21 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC3BB16A417 for ; Sat, 9 Feb 2008 23:10:21 +0000 (UTC) (envelope-from daniel@dgnetwork.com.br) Received: from mail.mastercabo.com.br (mail.mastercabo.com.br [200.179.179.14]) by mx1.freebsd.org (Postfix) with SMTP id A2E6713C455 for ; Sat, 9 Feb 2008 23:10:20 +0000 (UTC) (envelope-from daniel@dgnetwork.com.br) Received: (qmail 16018 invoked by uid 1008); 9 Feb 2008 22:43:38 -0000 X-Spam-Checker-Version: SpamAssassin 3.1.6-unknown (2006-10-03) on srvmail1 X-Spam-Level: X-Spam-Status: No, score=-1.5 required=4.7 tests=AWL,BAYES_00 autolearn=ham version=3.1.6-unknown Received: from unknown (HELO ?10.0.1.10?) (daniel@dgnetwork.com.br@200.243.216.68) by mail.mastercabo.com.br with SMTP; 9 Feb 2008 22:43:32 -0000 Message-ID: <47AE2AEC.5060604@dgnetwork.com.br> Date: Sat, 09 Feb 2008 20:36:28 -0200 From: =?ISO-8859-1?Q?Daniel_Dias_Gon=E7alves?= Organization: DGNET Network Solutions User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: DEVICE_POLLING IF_EM CPU usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: daniel@dgnetwork.com.br List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 23:10:21 -0000 Hi, I activated polling in 8 interfaces em0... em7 (ifconfig em* polling) to carry through performance tests, when activating, it had a significant reduction of CPU usage, load average measured of "0,50, 0,69, 0,52" for "0,43, 0,39, 0,21" and the CPU usage (SNMP Graphic) measured of 35% for 5%. Passed some hours I disactivated polling (ifconfig em* - polling) and CPU usage continues low. Somebody could explain this to me? Some information: FreeBSD 6.3-RELEASE CPU: Intel(R) Xeon(TM) CPU 3.20GHz (3200.13-MHz 686-class CPU) Logical CPUs per core: 2 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs SMP: AP CPU #1 Launched! Interfaces if_em: vendor = 'Intel Corporation' device = '82546EB Dual Port Gigabit Ethernet Controller' Thanks. Daniel From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 9 23:33:26 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8CB9616A46B for ; Sat, 9 Feb 2008 23:33:26 +0000 (UTC) (envelope-from jhs@berklix.org) Received: from tower.berklix.org (tower.berklix.org [83.236.223.114]) by mx1.freebsd.org (Postfix) with ESMTP id 08B0C13C448 for ; Sat, 9 Feb 2008 23:33:25 +0000 (UTC) (envelope-from jhs@berklix.org) Received: from js.berklix.net (p549A4892.dip.t-dialin.net [84.154.72.146]) (authenticated bits=0) by tower.berklix.org (8.13.6/8.13.6) with ESMTP id m19NXHJP066618; Sat, 9 Feb 2008 23:33:17 GMT (envelope-from jhs@berklix.org) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by js.berklix.net (8.13.8/8.13.8) with ESMTP id m19NYea2035425; Sun, 10 Feb 2008 00:33:11 +0100 (CET) (envelope-from jhs@berklix.org) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.13.8/8.13.8) with ESMTP id m19NYUax000451; Sun, 10 Feb 2008 00:34:35 +0100 (CET) (envelope-from jhs@fire.js.berklix.net) Message-Id: <200802092334.m19NYUax000451@fire.js.berklix.net> To: Chuck Robey In-reply-to: <47AE2628.3070500@chuckr.org> References: <47ADFD2C.80009@chuckr.org> <200802091951.m19JpJfj096492@fire.js.berklix.net> <47AE0BFA.5030007@chuckr.org> <200802092216.m19MGgte098739@fire.js.berklix.net> <47AE2628.3070500@chuckr.org> Comments: In-reply-to Chuck Robey message dated "Sat, 09 Feb 2008 17:16:08 -0500." Date: Sun, 10 Feb 2008 00:34:30 +0100 From: "Julian H. Stacey" Cc: FreeBSD-Hackers Subject: Re: USB Graphic Tablets X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 23:33:26 -0000 Chuck Robey wrote: > >>>> Well, my first question is, does there exist a tool for USB that let's you > >>>> view the raw return from the usb probing? I want to see what this device > >>>> is actually identifying itself as. Actual numeric vendor IDs and product > >>>> IDs are the sort of thing I'm after. Something a whole lot more detailed > >>>> that usbdevs, please. Even the dmesg listing (as I show above) hasn't got > >>>> the numbers and tails I need. > >>> Hi Chuck, > >>> On list sane-devel@lists.alioth.debian.org (A mostly Linux using > >>> crownd though not just, some BSD) They mostly seem to rec. some > >>> (free I assume) MS$ comparible .exe prog for snatching USB traffic > >>> to then analyse / develop for *IX drivers. I don't remember prog name, > >>> but http://lists.alioth.debian.org/mailman/listinfo/sane-devel > >>> Julian > >> Thanks much for the URL, I will definitely look it up. I'm a bit surprised > >> that the SANE folks, who I took to be scanner-oriented, should be > >> interested in graphic tablets. > > > > Err, they're not far as I know, but they often seem to want to grab > > byte streams on scanners to then analyse for Sane. I guess what > > one wants to later use the grabbed traffic for doesnt matter, so > > long as one has the traffic grabbed for analysis ? > > Well ... OK, I'm having just a bit of a hard time seeing how that would > help me in trying to get my graphic tablet working on FreeBSD for my dear > old gimp, Ah, sorry, forgot to say: Using a 2nd PC as traffic monitor. You'd issue probe from FreeBSD to USB device using whatever tools, & the (Gasp! Wash my mouth out with soap!) - MS.EXE prog running on a 2nd PC would trap a copy of traffic in each direction, synchronising the 2 as well I believe though never tried it. The 2nd monitoring (MS) PC uses 2 USB ports, one to copy FreeSBD PC to tablet device traffic, & the other port to copy tablet to FreeBSD traffic. ... & you have to make up a special USB cable, eg a male to female USB extender cable, with tapped copies of signal in each direction going to 2 extra USB connectors to 2 ports on 2nd monitor PC. Sane developers seem to do that a lot on new unknown USB scanners. > but I know darn well you're right more often than I am, so I > will give a good look at that list anyhow. Non tech flippancy: Chuckle, dreadful thought to be wise! .. BBC's Hitchkikers Guide To The Galaxy: To leader of bird people: "What do we call you ?" "Well, some call me the wise .. old .. bird" ponderously said by John Le Mesurier (Sergeant Wilson in BBC's 'Dad's Army') :-) -- Julian Stacey. BSD Unix Linux Net Consultant, Munich. http://berklix.com