From owner-freebsd-hackers@FreeBSD.ORG Sun May 11 01:18:03 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 292231065676; Sun, 11 May 2008 01:18:03 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.10]) by mx1.freebsd.org (Postfix) with ESMTP id B1B528FC13; Sun, 11 May 2008 01:18:02 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.14.3/8.14.3/NETPLEX) with ESMTP id m4B0s96I027040; Sat, 10 May 2008 20:54:09 -0400 (EDT) X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.0 (mail.netplex.net [204.213.176.10]); Sat, 10 May 2008 20:54:10 -0400 (EDT) Date: Sat, 10 May 2008 20:54:09 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: Robert Watson In-Reply-To: <20080510213323.E35578@fledge.watson.org> Message-ID: References: <4dcb5abd0805050540m292b319aw52aa2cb8ba018e12@mail.gmail.com> <481F0DB3.9070505@FreeBSD.org> <481F48EE.3050806@elischer.org> <481F4EED.2030300@FreeBSD.org> <4dcb5abd0805051132o77d68e36u3f0ad38630a02afd@mail.gmail.com> <481F6990.9010007@elischer.org> <20080510134833.J63808@fledge.watson.org> <20080510152718.788d638f@bhuda.mired.org> <20080510213323.E35578@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org, Mike Meyer , Carl Shapiro , Julian Elischer Subject: Re: binary compatibility query X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 May 2008 01:18:03 -0000 On Sat, 10 May 2008, Robert Watson wrote: > On Sat, 10 May 2008, Mike Meyer wrote: > >> On Sat, 10 May 2008 13:50:48 +0100 (BST) >> Robert Watson wrote: >>> On Mon, 5 May 2008, Julian Elischer wrote: >>>> basically if you rely only on the standard posix interfaces and don't do >>>> anything exotic then you will "probably" be safe. >>> For "raw" UNIX applications, this rule of thumb works well, but not for >>> applications that depend on third-party libraries, languages, or daemons. >> >> You can lose the dependency on third party libraries by compiling your >> application to a static binary. That will also help with the few breakages >> that occur with the system libraries, and with compatibility across major >> releases using the backwards compatX packages. > > However, if you do that, we'll probably shoot you in the foot by removing > kernel support for the thread library you are linking against. Since the > removal of KSE, it appears our ABI compatibility promise is at the libc or > libpthread layer, and not at the system call layer. We haven't done the > necessary compat work to let 6.x binaries using libkse work on an 8.x system > yet, but will presumably do that with a backport of libthr to a 6.x userspace > (i.e., 6.x library version). > > On almost all (serious) OS's I'm aware of, static linking for applications is > essentially forbidden for that reason -- vendors promise library > compatibility, not system call compatibility, and rely on being able to > change the implementations of system calls. I've brought this up before - we should seriously look at removing our static libraries, or at least make the statement that static binaries are not guaranteed to be ABI compatible. -- DE From owner-freebsd-hackers@FreeBSD.ORG Sun May 11 13:37:28 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE94F1065780 for ; Sun, 11 May 2008 13:37:28 +0000 (UTC) (envelope-from yanefbsd@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.181]) by mx1.freebsd.org (Postfix) with ESMTP id 7F2A18FC12 for ; Sun, 11 May 2008 13:37:28 +0000 (UTC) (envelope-from yanefbsd@gmail.com) Received: by wa-out-1112.google.com with SMTP id j4so2510086wah.3 for ; Sun, 11 May 2008 06:37:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:cc:message-id:from:to:in-reply-to:content-type:content-transfer-encoding:mime-version:subject:date:references:x-mailer; bh=9Man8vqgZWjftz+lYkt7CiI9vVvuCnxfpMFE1fEP2sM=; b=khAo95uT/6MCM7vwRhJqzRW5ZykWu3WYnuyVeF1xhwxribC5FTPAXEF0b+txDI+mQLMHypiLUt6z0wFx8RgV0xlseAoOT1+HdixLH9SUlbj3eb64jcqZf8Zlwv7MEiISVfsX92bouqg2hmSoCf6pDLP3/KlM78uL/968hiHSEa0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=cc:message-id:from:to:in-reply-to:content-type:content-transfer-encoding:mime-version:subject:date:references:x-mailer; b=plUBybi7qkSHy4k6DEfUBwEzVsS/jqO1q6hViF3dpR1wrv+aVgFbH7EKapNR8ARSpoTu605urBuH96Gl4qpcESRdiRRBIbm9o1g1D2l8iuCQ6DqBUVmNI5g0xOZzOaB0VE2oY2SOUHRJfJEN+7wWTNVwN/pXTH9etH/mhxKmROc= Received: by 10.114.194.1 with SMTP id r1mr6385299waf.205.1210513048033; Sun, 11 May 2008 06:37:28 -0700 (PDT) Received: from ?192.168.10.42? ( [99.155.199.64]) by mx.google.com with ESMTPS id k37sm12163601waf.31.2008.05.11.06.37.19 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 11 May 2008 06:37:20 -0700 (PDT) Message-Id: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> From: Garrett Cooper To: Joerg Sonnenberger In-Reply-To: <20080509124308.GA596@britannica.bec.de> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v919.2) Date: Sun, 11 May 2008 06:38:25 -0700 References: <20080509124308.GA596@britannica.bec.de> X-Mailer: Apple Mail (2.919.2) Cc: freebsd-hackers@freebsd.org Subject: Re: Adding .db support to pkg_tools X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 May 2008 13:37:28 -0000 On May 9, 2008, at 5:43 AM, Joerg Sonnenberger wrote: > On Fri, May 09, 2008 at 01:52:46PM +0200, Anders Nore wrote: >> I'm working on adding .db support to the pkg_tools( i.e. pkg_add, >> pkg_info, >> etc. ) as part of SoC 2008. The database api used is BerkeleyDB >> that comes >> with the base system (/usr/src/include/db.h). BerkeleyDB is not >> you're >> typical relational db, and can only save key/value pairs. The way I'm >> thinking of storing information to the .db is to name the keys as the >> directory names in /var/db/pkg. And save the +* files in the >> directories to >> the value element in the db, separated with a special character or >> similar. > > As one of the persons hacking on pkg_install in pkgsrc/NetBSD, I would > *strongly* advisy you against storing the files only in a bdb file. > The change of major and complete corruption with bdb185 is high, > consider pulling the plug in the middle of a long update. +1. BDB is quite easy to corrupt... -Garrett From owner-freebsd-hackers@FreeBSD.ORG Sun May 11 13:40:56 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BFFE0106566B for ; Sun, 11 May 2008 13:40:56 +0000 (UTC) (envelope-from yanefbsd@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.178]) by mx1.freebsd.org (Postfix) with ESMTP id 901B68FC1A for ; Sun, 11 May 2008 13:40:56 +0000 (UTC) (envelope-from yanefbsd@gmail.com) Received: by wa-out-1112.google.com with SMTP id j4so2511283wah.3 for ; Sun, 11 May 2008 06:40:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:cc:message-id:from:to:in-reply-to:content-type:content-transfer-encoding:mime-version:subject:date:references:x-mailer; bh=FWPA16tq8UTzqZrXF7Xlh01w41uX5fCoQXpku87ngwk=; b=wornT2DEOT1vZI0bcYdZgttdMwAz/X6rnjKDxzBPqvRdp0DjB9Nnxs7mGJWHTSsXntEPXCq8se5dklkD1jkBXCMgwwThyzDWZCenBfdclwPBCyuUcK9wXWoN+YIbVCjWJApMPuXKxRodEuYk73nCx3TY7KMlCtQnR6THAInJK+o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=cc:message-id:from:to:in-reply-to:content-type:content-transfer-encoding:mime-version:subject:date:references:x-mailer; b=wq9qCQBZAWbw+kben+FBoopIrU9/85/R+SLvePvhuVBfLocXpTgxwcBnqu31c0Av6kSGqgfylH4AcG5khlCcGFZ5Q3HksNErT6ovvTnT+Z+S+eYkZaGrsEv2PYTwxJ41vmHN4mRy+NgrC8QS/G8f/nKAXzw4hhGQJKdgRWnX0kU= Received: by 10.114.174.2 with SMTP id w2mr6434378wae.17.1210513256121; Sun, 11 May 2008 06:40:56 -0700 (PDT) Received: from ?192.168.10.42? ( [99.155.199.64]) by mx.google.com with ESMTPS id k14sm12206241waf.29.2008.05.11.06.40.54 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 11 May 2008 06:40:55 -0700 (PDT) Message-Id: From: Garrett Cooper To: d@delphij.net In-Reply-To: <482499AB.5000609@delphij.net> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v919.2) Date: Sun, 11 May 2008 06:42:00 -0700 References: <539c60b90805091103u2c8907a6q83cdca69fb45888e@mail.gmail.com> <482499AB.5000609@delphij.net> X-Mailer: Apple Mail (2.919.2) Cc: freebsd-hackers@freebsd.org, stevefranks@ieee.org Subject: Re: correct #define in source to specify FBSD vs. linux? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 May 2008 13:40:56 -0000 On May 9, 2008, at 11:36 AM, Xin LI wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Steve Franks wrote: > | Seems there is a more appropriate list for my earlier question to > | freebsd-questions: > | > | On and on I charge porting linux engineering tools. Major pita. I > | see a bunch of #ifdef __APPLE__ lines to pull in alternate headers; > | what's the equiv for FreeBSD? > > #ifdef __FreeBSD__? > > - -- > Xin LI http://www.delphij.net/ > FreeBSD - The Power to Serve! > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.9 (FreeBSD) > > iEYEARECAAYFAkgkmaoACgkQi+vbBBjt66C30ACeIbw6P7CuwErAIvzcUjX4d4Gk > 7G0An3+S2hM9YctAkKsWDVzkEoZIMhOH > =AzxS > -----END PGP SIGNATURE----- #ifdef __linux__ for Linux as well. See: http://lists.debian.org/debian-devel/2001/01/msg00971.html (same applies for all platforms I would assume).. -Garrett From owner-freebsd-hackers@FreeBSD.ORG Sun May 11 18:49:23 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0400E106564A for ; Sun, 11 May 2008 18:49:23 +0000 (UTC) (envelope-from bart@it-ss.be) Received: from mx20.it-ss.be (mx20.it-ss.be [195.28.164.230]) by mx1.freebsd.org (Postfix) with ESMTP id 72ECB8FC15 for ; Sun, 11 May 2008 18:49:22 +0000 (UTC) (envelope-from bart@it-ss.be) Received: from authrelay.it-ss.be (authrelay.it-ss.be [195.28.164.225]) by mx20.it-ss.be (8.13.8/8.13.8) with ESMTP id m4BGipkr003284 for ; Sun, 11 May 2008 18:44:51 +0200 Received: from bartwrkstxp (63.104-242-81.adsl-dyn.isp.belgacom.be [81.242.104.63]) (authenticated bits=0) by mx10.it-ss.be (8.13.6/8.13.6) with ESMTP id m4BGiklA032592 for ; Sun, 11 May 2008 18:44:48 +0200 Message-ID: <007001c8b386$528dbf70$020b000a@bartwrkstxp> From: "Bart Van Kerckhove" To: References: <4dcb5abd0805050540m292b319aw52aa2cb8ba018e12@mail.gmail.com><481F0DB3.9070505@FreeBSD.org> <481F48EE.3050806@elischer.org><481F4EED.2030300@FreeBSD.org><4dcb5abd0805051132o77d68e36u3f0ad38630a02afd@mail.gmail.com><481F6990.9010007@elischer.org><20080510134833.J63808@fledge.watson.org><20080510152718.788d638f@bhuda.mired.org><20080510213323.E35578@fledge.watson.org> Date: Sun, 11 May 2008 18:44:45 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Scanned-By: MIMEDefang 2.63 on 195.28.164.230 X-Scanned-By: MIMEDefang 2.63 on 195.28.164.225 Subject: Xenu PV X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 May 2008 18:49:23 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear list, I'm interested in checking out the work being done on XenU PV support in the freebsd kernel. However, probably because of some stupid oversight on my part, I can't seem to check out the source from eiter cvsup10 or 18 (as stated on http://wiki.freebsd.org/FreeBSD/Xen ). I've been asking around and sending mails for read access on p4, but to no avail. Could you guys give me a nudge in the right direction? How do I get to the sources that are being worked on? Cheers! With kind regards, Bart Van Kerckhove bart@it-ss.be http://friet.net/pgp.txt -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (MingW32) - WinPT 1.0.1 iD8DBQFIJyJ1CggVyEEzQEoRAtcHAJ9A/wIMtPz9wutZfKS1gyFZSEFPjQCeLjjS w8BgGgHRk607aNK+xqd/VAk= =Kbwb -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Sun May 11 19:34:02 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B2481065676 for ; Sun, 11 May 2008 19:34:02 +0000 (UTC) (envelope-from jos@catnook.com) Received: from lizzy.dyndns.org (209-204-188-132.dsl.static.sonic.net [209.204.188.132]) by mx1.freebsd.org (Postfix) with SMTP id 009AE8FC20 for ; Sun, 11 May 2008 19:34:01 +0000 (UTC) (envelope-from jos@catnook.com) Received: (qmail 62595 invoked by uid 1000); 11 May 2008 19:07:41 -0000 Date: Sun, 11 May 2008 12:07:41 -0700 From: Jos Backus To: freebsd-hackers@freebsd.org Message-ID: <20080511190741.GB47383@lizzy.catnook.local> Mail-Followup-To: freebsd-hackers@freebsd.org References: <20080509124308.GA596@britannica.bec.de> <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> User-Agent: Mutt/1.5.17 (2007-11-01) Subject: Re: Adding .db support to pkg_tools X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: jos@catnook.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 May 2008 19:34:02 -0000 On Sun, May 11, 2008 at 06:38:25AM -0700, Garrett Cooper wrote: > +1. BDB is quite easy to corrupt... If we're going to use a binary file format, please consider using SQLite instead. It has the right license, a nice API (transactions!) and is robust enough for yum to use it for a similar purpose. -- Jos Backus jos at catnook.com From owner-freebsd-hackers@FreeBSD.ORG Sun May 11 20:50:33 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E5D661065672; Sun, 11 May 2008 20:50:33 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from hosted.kievnet.com (hosted.kievnet.com [193.138.144.10]) by mx1.freebsd.org (Postfix) with ESMTP id 9AA588FC1A; Sun, 11 May 2008 20:50:33 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost ([127.0.0.1] helo=edge.pp.kiev.ua) by hosted.kievnet.com with esmtpa (Exim 4.62) (envelope-from ) id 1JvIV4-0005QA-MY; Sun, 11 May 2008 23:50:30 +0300 Message-ID: <48275C0C.2040601@icyb.net.ua> Date: Sun, 11 May 2008 23:50:20 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.12 (X11/20080320) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org References: <480E4269.2090604@icyb.net.ua> <480FBAB9.1000904@icyb.net.ua> <48103F36.6060707@icyb.net.ua> <200804240811.26183.jhb@freebsd.org> <4810FD1E.70602@icyb.net.ua> <20080425095009.GD18958@deviant.kiev.zoral.com.ua> <4811E6BC.4060306@icyb.net.ua> <20080425143646.GF18958@deviant.kiev.zoral.com.ua> In-Reply-To: <20080425143646.GF18958@deviant.kiev.zoral.com.ua> Content-Type: multipart/mixed; boundary="------------030709040303030500080608" Cc: Kostik Belousov Subject: Re: devctl (alike?) for devfs X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 May 2008 20:50:34 -0000 This is a multi-part message in MIME format. --------------030709040303030500080608 Content-Type: text/plain; charset=KOI8-U; format=flowed Content-Transfer-Encoding: 7bit Kostik, John, Warner, thank you for your guidance and suggestions. I am currently testing the patch attached and I am using a kernel with WITNESS and INVARIANTS enabled. Scope of my testing is plugging/unplugging of UMASS devices. I get CREATE notifications all right. I do not get any panics/complaints from the kernel, good. Unfortunately I do not get any DESTROY notifications either. Could you please look through the patch? Is there any control flow path that I missed or something even more obvious? I hope that we do not have any cdev leaks. I am testing the patch with RELENG_7 as of May 6. -- Andriy Gapon --------------030709040303030500080608 Content-Type: text/plain; name="devfs-notify.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="devfs-notify.diff" diff --git a/sys/kern/kern_conf.c b/sys/kern/kern_conf.c index 1db25f8..0245253 100644 --- a/sys/kern/kern_conf.c +++ b/sys/kern/kern_conf.c @@ -30,6 +30,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -99,6 +100,9 @@ dev_unlock_and_free(void) mtx_unlock(&devmtx); while ((cdp = TAILQ_FIRST(&cdp_free)) != NULL) { + if (!cold) + devctl_notify("DEVFS", cdp->cdp_c.si_name, "DESTROY", NULL); + TAILQ_REMOVE(&cdp_free, cdp, cdp_list); devfs_free(&cdp->cdp_c); } @@ -172,8 +176,12 @@ dev_rel(struct cdev *dev) flag = 1; } dev_unlock(); - if (flag) + if (flag) { + if (!cold) + devctl_notify("DEVFS", dev->si_name, "DESTROY", NULL); + devfs_free(dev); + } } struct cdevsw * @@ -706,6 +714,10 @@ make_dev_credv(int flags, struct cdevsw *devsw, int minornr, devfs_create(dev); clean_unrhdrl(devfs_inos); dev_unlock_and_free(); + + if (!cold) + devctl_notify("DEVFS", dev->si_name, "CREATE", NULL); + return (dev); } @@ -794,6 +806,10 @@ make_dev_alias(struct cdev *pdev, const char *fmt, ...) clean_unrhdrl(devfs_inos); dev_unlock(); dev_depends(pdev, dev); + + if (!cold) + devctl_notify("DEVFS", dev->si_name, "CREATE", NULL); + return (dev); } --------------030709040303030500080608-- From owner-freebsd-hackers@FreeBSD.ORG Sun May 11 21:48:44 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB0BB106564A; Sun, 11 May 2008 21:48:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay02.kiev.sovam.com (relay02.kiev.sovam.com [62.64.120.197]) by mx1.freebsd.org (Postfix) with ESMTP id 7BD658FC1C; Sun, 11 May 2008 21:48:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=skuns.kiev.zoral.com.ua) by relay02.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1JvJPN-000ABO-Qo; Mon, 12 May 2008 00:48:42 +0300 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by skuns.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4BLmawl083445 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 12 May 2008 00:48:36 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4BLmXo3036722; Mon, 12 May 2008 00:48:33 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m4BLmXEr036721; Mon, 12 May 2008 00:48:33 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 12 May 2008 00:48:33 +0300 From: Kostik Belousov To: Andriy Gapon Message-ID: <20080511214833.GB18958@deviant.kiev.zoral.com.ua> References: <480E4269.2090604@icyb.net.ua> <480FBAB9.1000904@icyb.net.ua> <48103F36.6060707@icyb.net.ua> <200804240811.26183.jhb@freebsd.org> <4810FD1E.70602@icyb.net.ua> <20080425095009.GD18958@deviant.kiev.zoral.com.ua> <4811E6BC.4060306@icyb.net.ua> <20080425143646.GF18958@deviant.kiev.zoral.com.ua> <48275C0C.2040601@icyb.net.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="GURrebd5m7w1Nnt/" Content-Disposition: inline In-Reply-To: <48275C0C.2040601@icyb.net.ua> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.4 X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on skuns.kiev.zoral.com.ua X-Scanner-Signature: 88d0fd334128e7d5bb9b9ebe4fef9048 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 2798 [May 08 2008] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: freebsd-hackers@freebsd.org Subject: Re: devctl (alike?) for devfs X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 May 2008 21:48:44 -0000 --GURrebd5m7w1Nnt/ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, May 11, 2008 at 11:50:20PM +0300, Andriy Gapon wrote: >=20 > Kostik, John, Warner, >=20 > thank you for your guidance and suggestions. > I am currently testing the patch attached and I am using a kernel with=20 > WITNESS and INVARIANTS enabled. > Scope of my testing is plugging/unplugging of UMASS devices. > I get CREATE notifications all right. > I do not get any panics/complaints from the kernel, good. > Unfortunately I do not get any DESTROY notifications either. >=20 > Could you please look through the patch? Is there any control flow path= =20 > that I missed or something even more obvious? > I hope that we do not have any cdev leaks. >=20 > I am testing the patch with RELENG_7 as of May 6. No, we do not have a leak, but we have somewhat non-obvious behaviour. The cdev structure is freed only after the last reference to cdev is gone. Typical holder of the reference is the devfs vnode. In the normal usage, the vnode is present until both the device is destroyed _and_ devfs_populate_loop() run is performed. This function actually reclaim the vnodes for destroyed devices, that causes last reference to cdev to be dropped and memory freed. The populate loop is called syncronously from the upper levels. The easiest method to trigger it is to do ls /dev, since it is called from the devfs_lookupx(). >=20 > --=20 > Andriy Gapon > diff --git a/sys/kern/kern_conf.c b/sys/kern/kern_conf.c > index 1db25f8..0245253 100644 > --- a/sys/kern/kern_conf.c > +++ b/sys/kern/kern_conf.c > @@ -30,6 +30,7 @@ __FBSDID("$FreeBSD$"); > #include > #include > #include > +#include > #include > #include > #include > @@ -99,6 +100,9 @@ dev_unlock_and_free(void) > mtx_unlock(&devmtx); > =20 > while ((cdp =3D TAILQ_FIRST(&cdp_free)) !=3D NULL) { > + if (!cold) > + devctl_notify("DEVFS", cdp->cdp_c.si_name, "DESTROY", NULL); > + > TAILQ_REMOVE(&cdp_free, cdp, cdp_list); > devfs_free(&cdp->cdp_c); > } > @@ -172,8 +176,12 @@ dev_rel(struct cdev *dev) > flag =3D 1; > } > dev_unlock(); > - if (flag) > + if (flag) { > + if (!cold) > + devctl_notify("DEVFS", dev->si_name, "DESTROY", NULL); > + > devfs_free(dev); > + } > } > =20 > struct cdevsw * > @@ -706,6 +714,10 @@ make_dev_credv(int flags, struct cdevsw *devsw, int = minornr, > devfs_create(dev); > clean_unrhdrl(devfs_inos); > dev_unlock_and_free(); > + > + if (!cold) > + devctl_notify("DEVFS", dev->si_name, "CREATE", NULL); > + > return (dev); > } > =20 > @@ -794,6 +806,10 @@ make_dev_alias(struct cdev *pdev, const char *fmt, .= ..) > clean_unrhdrl(devfs_inos); > dev_unlock(); > dev_depends(pdev, dev); > + > + if (!cold) > + devctl_notify("DEVFS", dev->si_name, "CREATE", NULL); > + > return (dev); > } > =20 --GURrebd5m7w1Nnt/ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkgnabAACgkQC3+MBN1Mb4ihSgCfRQjZSXKDLinsfMuGXTEx+kbZ JcgAoN3qatcTfVTT3ABTzjXn3TUm9Ijs =FBo+ -----END PGP SIGNATURE----- --GURrebd5m7w1Nnt/-- From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 01:45:39 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C643E1065670 for ; Mon, 12 May 2008 01:45:39 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.bluestop.org (unknown [IPv6:2001:41c8:1:548a::2]) by mx1.freebsd.org (Postfix) with ESMTP id 665058FC1C for ; Mon, 12 May 2008 01:45:39 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from [IPv6:2a01:348:10f:0:2849:d3b8:b35c:fd45] (unknown [IPv6:2a01:348:10f:0:2849:d3b8:b35c:fd45]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by muon.bluestop.org (Postfix) with ESMTP id F1834301B5; Mon, 12 May 2008 02:45:37 +0100 (BST) Message-ID: <4827A129.4000205@cran.org.uk> Date: Mon, 12 May 2008 02:45:13 +0100 From: Bruce Cran User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ollivier Robert References: <4818F72C.90300@cran.org.uk> <20080509173626.GC35511@keltia.freenix.fr> In-Reply-To: <20080509173626.GC35511@keltia.freenix.fr> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: hackers@freebsd.org Subject: Re: sshd patch to avoid DNS lookups when using 'UseDNS no' or -u0 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 01:45:39 -0000 Ollivier Robert wrote: > According to Bruce Cran: >> I've attached a patch which implements this (the change to loginrec.c >> reverts it back to the default OpenSSH code) and was wondering if someone >> could take a look at it. > > If you have not already done so, please use send-pr to record it in GNATS, > that will help not forgetting about it, thanks! The patch has been attached to PR bin/97499 - http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/97499 -- Bruce From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 03:09:38 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D3401065682 for ; Mon, 12 May 2008 03:09:38 +0000 (UTC) (envelope-from elijah.buck@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.169]) by mx1.freebsd.org (Postfix) with ESMTP id F11118FC20 for ; Mon, 12 May 2008 03:09:37 +0000 (UTC) (envelope-from elijah.buck@gmail.com) Received: by ug-out-1314.google.com with SMTP id q2so617958uge.37 for ; Sun, 11 May 2008 20:09:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; bh=bL5guok2zWY3pd49iJaWa62zQEjJlDT5PbjvOCroxyQ=; b=Uainsn7HkDEA8qkaA0s95nUIkMu9aCRYI6MEQ3v2tKvOmI/seXp+UNEPNRkgNBMSlXp/EQPwLfBqGOdlNVNV5Cpo1Zv87RL1fSmKKDyLL9JRnY6zQnDzCK7R3nrfebVzyUTVZh7JutiaCsTihzQTrEBN+v2RU3v0SirigJC6Mik= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=H07TkHuOvWaQrSKnUQOczx+oEo21siScC4CaKl0h3uwG6pon51cinWCd54zN9HACGirGkZSW1EZ27eCOzZxcnQpadxrYFNSYL1lrQLnZLTUX0PjSJwUErvaYaE4xoG4QXXQZYSoG/+jks5Y7KDxgrifJeoyrTODv4lwGs9wiD64= Received: by 10.66.223.2 with SMTP id v2mr4944014ugg.18.1210560222976; Sun, 11 May 2008 19:43:42 -0700 (PDT) Received: by 10.67.121.6 with HTTP; Sun, 11 May 2008 19:43:42 -0700 (PDT) Message-ID: <303eddcb0805111943t7c456c79l56f427c526d0b454@mail.gmail.com> Date: Sun, 11 May 2008 21:43:42 -0500 From: "Elijah Buck" To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: 4bsd fuzzy runq X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 03:09:38 -0000 Hi, I'm looking at the code for 4bsd fuzzy run queues in kern_switch.c The relevant bit: if (fuzz > 1) { int count = fuzz; int cpu = PCPU_GET(cpuid); struct thread *td2; td2 = td = TAILQ_FIRST(rqh); while (count-- && td2) { if (td->td_lastcpu == cpu) { td = td2; break; } td2 = TAILQ_NEXT(td2, td_runq); } ...return(td) The purpose of this code appears to be to look through the runq to a depth defined by fuzz for a thread that was last run on the current cpu. Here are the cases I see: 1.) if (td_lastcpu == cpu) on the first iteration then TAILQ_FIRST(rqh) is the selected thread (because of the break). 2.) if (td_lastcpu != cpu) on the first iteration then td is never set again, and so (td_lastcpu != cpu) is always true. So, again, TAILQ_FIRST(rqh) will be selected (because count will reach 0). Which doesn't seem right (since that's what the else clause does). What am I missing here? Thanks, Elijah From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 06:10:52 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A41591065674 for ; Mon, 12 May 2008 06:10:52 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outY.internet-mail-service.net (outy.internet-mail-service.net [216.240.47.248]) by mx1.freebsd.org (Postfix) with ESMTP id 8CC228FC43 for ; Mon, 12 May 2008 06:10:52 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Mon, 12 May 2008 12:44:40 -0700 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id A38CD2D600D; Sun, 11 May 2008 23:10:51 -0700 (PDT) Message-ID: <4827DF6B.6050101@elischer.org> Date: Sun, 11 May 2008 23:10:51 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Elijah Buck References: <303eddcb0805111943t7c456c79l56f427c526d0b454@mail.gmail.com> In-Reply-To: <303eddcb0805111943t7c456c79l56f427c526d0b454@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: 4bsd fuzzy runq X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 06:10:52 -0000 Elijah Buck wrote: > Hi, > > I'm looking at the code for 4bsd fuzzy run queues in kern_switch.c > > The relevant bit: > > if (fuzz > 1) { > int count = fuzz; > int cpu = PCPU_GET(cpuid); > struct thread *td2; > td2 = td = TAILQ_FIRST(rqh); > while (count-- && td2) { > if (td->td_lastcpu == cpu) { this should be td2. The bug is mine.. and I wondered why the code commited never had the same result as that I tested in another tree. > td = td2; > break; > } > td2 = TAILQ_NEXT(td2, td_runq); > } > ...return(td) > The purpose of this code appears to be to look through the runq to a depth > defined by fuzz for a thread that was last run on the current cpu. > > Here are the cases I see: > 1.) if (td_lastcpu == cpu) on the first iteration then TAILQ_FIRST(rqh) is > the selected thread (because of the break). > 2.) if (td_lastcpu != cpu) on the first iteration then td is never set > again, and so (td_lastcpu != cpu) is always true. So, again, > TAILQ_FIRST(rqh) will be selected (because count will reach 0). > > Which doesn't seem right (since that's what the else clause does). What am I > missing here? > > Thanks, > Elijah > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 08:54:33 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 47B1A106566B for ; Mon, 12 May 2008 08:54:33 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from mx0.awanti.com (mx0.awanti.com [91.190.112.18]) by mx1.freebsd.org (Postfix) with ESMTP id 0568A8FC1B for ; Mon, 12 May 2008 08:54:32 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from pstation (unknown [10.28.4.14]) by mx0.awanti.com (Postfix) with ESMTP id A127B4C1B5; Mon, 12 May 2008 12:35:42 +0400 (MSD) Date: Mon, 12 May 2008 12:38:06 +0400 From: Anthony Pankov X-Mailer: The Bat! (v1.51) Personal X-Priority: 3 (Normal) Message-ID: <15336578.20080512123806@mail.ru> To: Garrett Cooper , Joerg Sonnenberger In-Reply-To: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> References: <20080509124308.GA596@britannica.bec.de> <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Anthony Pankov List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 08:54:33 -0000 Please, can anybody explain what is the problem with BDB (1.86). Is there known caveats of using BDB? Is there some rules which guarantee from curruption or it is fully undesirable to use BDB under high load? It is important for me because of using BDB in my project. > On Fri, May 09, 2008 at 01:52:46PM +0200, Joerg Sonnenberger wrote: >> >> As one of the persons hacking on pkg_install in pkgsrc/NetBSD, I would >> *strongly* advisy you against storing the files only in a bdb file. >> The change of major and complete corruption with bdb185 is high, >> consider pulling the plug in the middle of a long update. >Sunday, May 11, 2008, 5:38:25 PM, you wrote: GC> +1. BDB is quite easy to corrupt... -- Best regards, Anthony mailto:ap00@mail.ru From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 10:19:21 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB2751065672 for ; Mon, 12 May 2008 10:19:21 +0000 (UTC) (envelope-from jonathan+freebsd-hackers@hst.org.za) Received: from hermes.hst.org.za (onix.hst.org.za [209.203.2.133]) by mx1.freebsd.org (Postfix) with ESMTP id D523F8FC16 for ; Mon, 12 May 2008 10:19:20 +0000 (UTC) (envelope-from jonathan+freebsd-hackers@hst.org.za) Received: from sysadmin.hst.org.za (sysadmin.int.dbn.hst.org.za [10.1.1.20]) (authenticated bits=0) by hermes.hst.org.za (8.13.8/8.13.8) with ESMTP id m4C9n2Fj017246 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NO) for ; Mon, 12 May 2008 11:49:02 +0200 (SAST) (envelope-from jonathan+freebsd-hackers@hst.org.za) From: Jonathan McKeown Organization: Health Systems Trust To: freebsd-hackers@freebsd.org User-Agent: KMail/1.7.2 References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> In-Reply-To: <15336578.20080512123806@mail.ru> MIME-Version: 1.0 Content-Disposition: inline X-UID: 2295 X-Length: 1299 Date: Mon, 12 May 2008 11:53:00 +0200 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> X-Spam-Score: -4.399 () ALL_TRUSTED,BAYES_00 X-Scanned-By: MIMEDefang 2.61 on 209.203.2.133 Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 10:19:22 -0000 On Monday 12 May 2008 10:38, Anthony Pankov wrote: > Please, can anybody explain what is the problem with BDB (1.86). > > Is there known caveats of using BDB? Is there some rules which > guarantee from curruption or it is fully undesirable to use BDB under > high load? > > It is important for me because of using BDB in my project. Interesting. I would have thought that the two processes "find out advantages and problems of proposed solutions" and "choose a solution" had a natural ordering other than the one you seem to be using. Jonathan From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 14:26:03 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 316B01065677 for ; Mon, 12 May 2008 14:26:03 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id D72D08FC1F for ; Mon, 12 May 2008 14:26:02 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id BFEA143CD0D for ; Mon, 12 May 2008 17:26:00 +0300 (EEST) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8-tRPtj0yZ2I for ; Mon, 12 May 2008 17:26:00 +0300 (EEST) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 7764143C668 for ; Mon, 12 May 2008 17:26:00 +0300 (EEST) Message-ID: <48285372.9060103@icyb.net.ua> Date: Mon, 12 May 2008 17:25:54 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.12 (X11/20080311) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Subject: openbsd solution to mounted umass removal X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 14:26:03 -0000 I've just came back from a good 2 week vacation and catching up on news. In release notes for OpenBSD 4.3 I see the following: http://openbsd.org/43.html Filesystems on USB devices are automatically dismounted if the device is disconnected. Does anybody have more [technical] details on this? Thanks. -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 14:34:38 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A9DA71065689 for ; Mon, 12 May 2008 14:34:38 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id 411068FC51 for ; Mon, 12 May 2008 14:34:38 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id 55B08744004; Mon, 12 May 2008 17:34:37 +0300 (EEST) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GbiKjsOze5xa; Mon, 12 May 2008 17:34:37 +0300 (EEST) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id E9305744003; Mon, 12 May 2008 17:34:36 +0300 (EEST) Message-ID: <4828557B.9000506@icyb.net.ua> Date: Mon, 12 May 2008 17:34:35 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.12 (X11/20080311) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org, freebsd-i386@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Subject: i386 cpu_reset_real: code/comment mismatch X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 14:34:38 -0000 This is not a real issue, just a code clarification. First a snippet from sys/i386/i386/vm_machdep.c, cpu_reset_real() /* * Attempt to force a reset via the Reset Control register at * I/O port 0xcf9. Bit 2 forces a system reset when it is * written as 1. Bit 1 selects the type of reset to attempt: * 0 selects a "soft" reset, and 1 selects a "hard" reset. We * try to do a "soft" reset first, and then a "hard" reset. */ outb(0xcf9, 0x2); outb(0xcf9, 0x6); I think that the comment is correct up to but not including the last sentence. Writing 0x2 sets bit 1 to 1 (thus selecting hard reset), and writing 0x6 sets both bits 2 and 1 to 1 (thus performing hard reset). So we always just do a hard reset, no trying of soft reset (would it even make sense to do the last line of the comment says). -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 15:14:13 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0AF88106564A for ; Mon, 12 May 2008 15:14:13 +0000 (UTC) (envelope-from andenore@start.no) Received: from osl1smout1.broadpark.no (osl1smout1.broadpark.no [80.202.4.58]) by mx1.freebsd.org (Postfix) with ESMTP id B9D318FC0C for ; Mon, 12 May 2008 15:14:12 +0000 (UTC) (envelope-from andenore@start.no) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; delsp=yes; charset=us-ascii Received: from osl1sminn1.broadpark.no ([80.202.4.59]) by osl1smout1.broadpark.no (Sun Java(tm) System Messaging Server 6.3-3.01 (built Jul 12 2007; 32bit)) with ESMTP id <0K0R003Y2GZNQ890@osl1smout1.broadpark.no> for freebsd-hackers@freebsd.org; Mon, 12 May 2008 17:14:11 +0200 (CEST) Received: from duckjen.nextgentel.no ([84.48.194.205]) by osl1sminn1.broadpark.no (Sun Java(tm) System Messaging Server 6.3-3.01 (built Jul 12 2007; 32bit)) with ESMTP id <0K0R0095MGZMM700@osl1sminn1.broadpark.no> for freebsd-hackers@freebsd.org; Mon, 12 May 2008 17:14:11 +0200 (CEST) Date: Mon, 12 May 2008 17:12:56 +0200 To: freebsd-hackers@freebsd.org From: Anders Nore References: <20080509124308.GA596@britannica.bec.de> <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <20080511190741.GB47383@lizzy.catnook.local> Message-id: In-reply-to: <20080511190741.GB47383@lizzy.catnook.local> User-Agent: Opera Mail/9.27 (FreeBSD) Cc: Subject: Re: Adding .db support to pkg_tools X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 15:14:13 -0000 On Sun, 11 May 2008 21:07:41 +0200, Jos Backus wrote: > On Sun, May 11, 2008 at 06:38:25AM -0700, Garrett Cooper wrote: >> +1. BDB is quite easy to corrupt... > > If we're going to use a binary file format, please consider using SQLite > instead. It has the right license, a nice API (transactions!) and is > robust > enough for yum to use it for a similar purpose. > One of the reasons for using BDB is that it is in the base system, SQLite however is not. From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 15:47:53 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BEBD21065680 for ; Mon, 12 May 2008 15:47:53 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id AE2A88FC0A for ; Mon, 12 May 2008 15:47:53 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 9FD4F1CC033; Mon, 12 May 2008 08:47:53 -0700 (PDT) Date: Mon, 12 May 2008 08:47:53 -0700 From: Jeremy Chadwick To: Anders Nore Message-ID: <20080512154753.GA16847@eos.sc1.parodius.com> References: <20080509124308.GA596@britannica.bec.de> <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <20080511190741.GB47383@lizzy.catnook.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.17 (2007-11-01) Cc: freebsd-hackers@freebsd.org Subject: Re: Adding .db support to pkg_tools X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 15:47:53 -0000 On Mon, May 12, 2008 at 05:12:56PM +0200, Anders Nore wrote: > On Sun, 11 May 2008 21:07:41 +0200, Jos Backus wrote: > >> On Sun, May 11, 2008 at 06:38:25AM -0700, Garrett Cooper wrote: >>> +1. BDB is quite easy to corrupt... >> >> If we're going to use a binary file format, please consider using SQLite >> instead. It has the right license, a nice API (transactions!) and is >> robust >> enough for yum to use it for a similar purpose. >> > > One of the reasons for using BDB is that it is in the base system, SQLite > however is not. Interesting discussion, and there's some irony in it (SQLite uses a very BDB-like a storage medium, despite being a relational database system). SQLite's license is compatible/friendly, its API is decent, the source is fairly small and binary footprint (disk and memory both) are small, and it includes tools for dumping databases to ASCII so developers aren't forced to write DB dumping tools and users don't have to resort to using hexdump. A few things worth pointing out: With regards to DB1.x being easily corrupted, this is true, and I have heard claims of it from a couple developer-folk I worked with many years ago at Best Internet. On the other hand, SQLite appears to have a history of this as well, although those bugs have been fixed each and every time: http://www.sqlite.org/cvstrac/wiki?p=DatabaseCorruption Secondly, the following FAQ entry and documentation from Mozilla is of concern, specifically the last paragraph of the FAQ entry, since there is ongoing work in the ports collection to support parallel building, which would surely use fork() or locks): http://www.sqlite.org/faq.html#q6 http://developer.mozilla.org/en/docs/Storage#SQLite_Locking Thirdly, there is the concern of disk space. There are a substantial number of users who have FreeBSD on embedded systems or systems with little or no space, who use binary packages (pkg_add -r), which of course still makes use of /var/db/pkg. SQLite DBs grow over time, and don't shrink in size unless you use auto-vacuum mode or execute the VACUUM query: http://www.sqlite.org/faq.html#q12 Fourthly, it doesn't sound as if storage SQLite databases on an NFS mount is a good idea, and I've a feeling some people have /var NFS mounted (diskless systems for example): http://www.sqlite.org/faq.html#q5 Finally, there's the concern about the SQLite file format changing. Sometimes they're backwards-compatible, other times not. http://www.sqlite.org/formatchng.html http://www.sqlite.org/fileformat.html Just stuff to consider. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 16:03:32 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 00FDB106564A for ; Mon, 12 May 2008 16:03:32 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from www.pkgsrc-box.org (www.ostsee-abc.de [62.206.222.50]) by mx1.freebsd.org (Postfix) with ESMTP id ADEF08FC18 for ; Mon, 12 May 2008 16:03:31 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from britannica.bec.de (www.pkgsrc-box.org [127.0.0.1]) by www.pkgsrc-box.org (Postfix) with ESMTP id 7F6CFE506A7 for ; Mon, 12 May 2008 16:03:26 +0000 (UTC) Received: by britannica.bec.de (Postfix, from userid 1000) id B0E3F16FC3; Mon, 12 May 2008 18:02:40 +0200 (CEST) Date: Mon, 12 May 2008 18:02:40 +0200 From: Joerg Sonnenberger To: freebsd-hackers@freebsd.org Message-ID: <20080512160240.GA19040@britannica.bec.de> Mail-Followup-To: freebsd-hackers@freebsd.org References: <20080509124308.GA596@britannica.bec.de> <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <20080511190741.GB47383@lizzy.catnook.local> <20080512154753.GA16847@eos.sc1.parodius.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080512154753.GA16847@eos.sc1.parodius.com> User-Agent: Mutt/1.5.17 (2007-11-01) Subject: Re: Adding .db support to pkg_tools X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 16:03:32 -0000 On Mon, May 12, 2008 at 08:47:53AM -0700, Jeremy Chadwick wrote: > Secondly, the following FAQ entry and documentation from Mozilla is of > concern, specifically the last paragraph of the FAQ entry, since there > is ongoing work in the ports collection to support parallel building, > which would surely use fork() or locks): I don't think this is applicable. > Thirdly, there is the concern of disk space. There are a substantial > number of users who have FreeBSD on embedded systems or systems with > little or no space, who use binary packages (pkg_add -r), which of > course still makes use of /var/db/pkg. SQLite DBs grow over time, and > don't shrink in size unless you use auto-vacuum mode or execute the > VACUUM query: It is trivial to recreate the database, so I don't think this is a concern either. I should point again that it will result in a non-trivial increase in size. > Fourthly, it doesn't sound as if storage SQLite databases on an NFS > mount is a good idea, and I've a feeling some people have /var NFS > mounted (diskless systems for example): Same applies to BDB and basically anything that depends on fcntl to work. Basically, as long as you have /var per host (you don't share /var, do you?), it will just work as the *local* locks are working. The problem with NFS comes form the fact that multiple hosts might not share them. Joerg From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 16:32:03 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65612106564A for ; Mon, 12 May 2008 16:32:03 +0000 (UTC) (envelope-from ale@FreeBSD.org) Received: from lab.alexdupre.com (cl-506.trn-01.it.sixxs.net [IPv6:2001:1418:100:1f9::2]) by mx1.freebsd.org (Postfix) with ESMTP id 982928FC1C for ; Mon, 12 May 2008 16:32:02 +0000 (UTC) (envelope-from ale@FreeBSD.org) Received: (qmail 79493 invoked from network); 12 May 2008 16:31:56 -0000 Received: from unknown (HELO athlon.alexdupre.com) (192.168.178.2) by lab.alexdupre.com with SMTP; 12 May 2008 16:31:56 -0000 Message-ID: <482870FC.5050806@FreeBSD.org> Date: Mon, 12 May 2008 18:31:56 +0200 From: Alex Dupre User-Agent: Thunderbird 2.0.0.14 (X11/20080504) MIME-Version: 1.0 To: Andriy Gapon References: <48285372.9060103@icyb.net.ua> In-Reply-To: <48285372.9060103@icyb.net.ua> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: openbsd solution to mounted umass removal X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 16:32:03 -0000 Andriy Gapon wrote: > Filesystems on USB devices are automatically dismounted if the device is > disconnected. > > Does anybody have more [technical] details on this? Modified files: sys/dev/usb : umass.c sys/scsi : sdvar.h sd.c sys/kern : vfs_default.c vfs_subr.c Log message: Allow to pull out an usb stick with ffs filesystem while mounted and a file is written onto the stick. Without these fixes the machine panics or hangs. The usb fix calls the callback when the stick is pulled out to free the associated buffers. Otherwise we have busy buffers for ever and the automatic unmount will panic. The change in the scsi layer prevents passing down further dirty buffers to usb after the stick has been deactivated. In vfs the automatic unmount has moved from the function vgonel() to vop_generic_revoke(). Both are called when the sd device's vnode is removed. In vgonel() the VXLOCK is already held which can cause a deadlock. So call dounmount() earlier. -- Alex Dupre From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 16:58:32 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CDC4C1065670 for ; Mon, 12 May 2008 16:58:32 +0000 (UTC) (envelope-from jos@catnook.com) Received: from lizzy.dyndns.org (209-204-188-132.dsl.static.sonic.net [209.204.188.132]) by mx1.freebsd.org (Postfix) with SMTP id 6FDC68FC22 for ; Mon, 12 May 2008 16:58:32 +0000 (UTC) (envelope-from jos@catnook.com) Received: (qmail 11381 invoked by uid 1000); 12 May 2008 16:58:53 -0000 Date: Mon, 12 May 2008 09:58:53 -0700 From: Jos Backus To: Anders Nore Message-ID: <20080512165853.GA11324@lizzy.catnook.local> Mail-Followup-To: Anders Nore , freebsd-hackers@freebsd.org References: <20080509124308.GA596@britannica.bec.de> <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <20080511190741.GB47383@lizzy.catnook.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.17 (2007-11-01) Cc: freebsd-hackers@freebsd.org Subject: Re: Adding .db support to pkg_tools X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: jos@catnook.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 16:58:32 -0000 On Mon, May 12, 2008 at 05:12:56PM +0200, Anders Nore wrote: > One of the reasons for using BDB is that it is in the base system, SQLite > however is not. I'm aware of that. But I believe that the pain and suffering of importing and maintaining SQLite in the base (that is, the cost) is outweighed by its richer feature set and higher-level API, facilitating easier tools development. -- Jos Backus jos at catnook.com From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 17:39:17 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1CAB21065670 for ; Mon, 12 May 2008 17:39:17 +0000 (UTC) (envelope-from fbsd06+V3=62d9fb26@mlists.homeunix.com) Received: from turtle-out.mxes.net (turtle-out.mxes.net [216.86.168.191]) by mx1.freebsd.org (Postfix) with ESMTP id B4FBC8FC0C for ; Mon, 12 May 2008 17:39:16 +0000 (UTC) (envelope-from fbsd06+V3=62d9fb26@mlists.homeunix.com) Received: from mxout-03.mxes.net (mxout-03.mxes.net [216.86.168.178]) by turtle-in.mxes.net (Postfix) with ESMTP id 43511163F74 for ; Mon, 12 May 2008 13:23:42 -0400 (EDT) Received: from gumby.homeunix.com. (unknown [87.81.140.128]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTP id 5CBB823E3FA for ; Mon, 12 May 2008 13:23:38 -0400 (EDT) Date: Mon, 12 May 2008 18:23:28 +0100 From: RW To: freebsd-hackers@freebsd.org Message-ID: <20080512182328.09a8a173@gumby.homeunix.com.> In-Reply-To: <4828557B.9000506@icyb.net.ua> References: <4828557B.9000506@icyb.net.ua> X-Mailer: Claws Mail 3.4.0 (GTK+ 2.12.9; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: i386 cpu_reset_real: code/comment mismatch X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 17:39:17 -0000 On Mon, 12 May 2008 17:34:35 +0300 Andriy Gapon wrote: > > This is not a real issue, just a code clarification. > > First a snippet from sys/i386/i386/vm_machdep.c, cpu_reset_real() > /* > * Attempt to force a reset via the Reset Control register at > * I/O port 0xcf9. Bit 2 forces a system reset when it is > * written as 1. Bit 1 selects the type of reset to attempt: > * 0 selects a "soft" reset, and 1 selects a "hard" reset. We > * try to do a "soft" reset first, and then a "hard" reset. > */ > outb(0xcf9, 0x2); > outb(0xcf9, 0x6); > > I think that the comment is correct up to but not including the last > sentence. Writing 0x2 sets bit 1 to 1 (thus selecting hard reset), and > writing 0x6 sets both bits 2 and 1 to 1 (thus performing hard reset). > So we always just do a hard reset, no trying of soft reset (would it > even make sense to do the last line of the comment says). > It looks to me as if the comment was added retrospectively by someone who got the two bits mixed-up when reading the source. If bits 1 and 2 were the other way around, it would be code for a soft-reset followed by a hard-reset. From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 18:33:08 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F0443106566B for ; Mon, 12 May 2008 18:33:08 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from mx0.awanti.com (mx0.awanti.com [91.190.112.18]) by mx1.freebsd.org (Postfix) with ESMTP id B45748FC15 for ; Mon, 12 May 2008 18:33:08 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from pstation (unknown [10.28.4.14]) by mx0.awanti.com (Postfix) with ESMTP id 00FF84C054 for ; Mon, 12 May 2008 22:33:06 +0400 (MSD) Date: Mon, 12 May 2008 22:35:31 +0400 From: Anthony Pankov X-Mailer: The Bat! (v1.51) Personal X-Priority: 3 (Normal) Message-ID: <1663320218.20080512223531@mail.ru> To: freebsd-hackers@freebsd.org In-Reply-To: <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Re[2]: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Anthony Pankov List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 18:33:09 -0000 So, can anyone make clear about BDB 1.86 (which is a part of base system). When 1. there is no need for SQL 2. processes are sharing db file in concurrent mode (key=>value pair) 3. reading/writing = 60%/40% the first idea is to use BDB. Because BDB: 1. do not need additional installation 2. is part of base system which mean it is mature, reliable and stable (otherwise why BDB is still a part of FreeBSD?) Discussion "Adding .db support to pkg_tools" reveal BDB ability to corrupt data. Can anyone suggest BDB alternative (not GPLed)? Monday, May 12, 2008, 1:53:00 PM, you wrote: JM> On Monday 12 May 2008 10:38, Anthony Pankov wrote: >> Please, can anybody explain what is the problem with BDB (1.86). >> >> Is there known caveats of using BDB? Is there some rules which >> guarantee from curruption or it is fully undesirable to use BDB under >> high load? >> >> It is important for me because of using BDB in my project. JM> Interesting. I would have thought that the two processes "find out advantages JM> and problems of proposed solutions" and "choose a solution" had a natural JM> ordering other than the one you seem to be using. JM> Jonathan -- Best regards, Anthony mailto:ap00@mail.ru From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 19:24:35 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 13A4D106566C for ; Mon, 12 May 2008 19:24:35 +0000 (UTC) (envelope-from mwm@mired.org) Received: from mired.org (five.mired.org [66.92.153.75]) by mx1.freebsd.org (Postfix) with ESMTP id 97F508FC16 for ; Mon, 12 May 2008 19:24:34 +0000 (UTC) (envelope-from mwm@mired.org) Received: (qmail 56465 invoked from network); 12 May 2008 15:25:00 -0400 Received: from unknown (HELO mbook.local) (192.168.195.193) by 0 with SMTP; 12 May 2008 15:25:00 -0400 Date: Mon, 12 May 2008 15:24:30 -0400 From: Mike Meyer To: Anthony Pankov Message-ID: <20080512152430.3720683e@mbook.local> In-Reply-To: <1663320218.20080512223531@mail.ru> References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> Organization: Meyer Consulting X-Mailer: Claws Mail 3.3.1 (GTK+ 2.12.9; i386-apple-darwin9.2.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 12 May 2008 20:09:33 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 19:24:35 -0000 On Mon, 12 May 2008 22:35:31 +0400 Anthony Pankov wrote: > Because BDB: > 1. do not need additional installation > 2. is part of base system which mean it is mature, reliable and stable BDB in the base system is mature, reliable and stable *for what it's used for in the base system.* So long as your requirements are covered by that usage, you'll be ok. The uses I know of for BDB in the base system all consist of databases of relatively small items that are changed infrequently, and usually with a locking mechanism. From what you've said, this doesn't describe your requirements. More importantly, from what other people are saying, your requirements are ones for which it's known that BDB is *not* reliable, or otherwise unsuitable. In particular, an effort is underway to allow parallel ports builds, which implies concurrent access to the database, which is a known source of problems for BDB. Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6AEB4106568A for ; Mon, 12 May 2008 20:45:39 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from bunrab.catwhisker.org (adsl-63-193-123-122.dsl.snfc21.pacbell.net [63.193.123.122]) by mx1.freebsd.org (Postfix) with ESMTP id 27D4E8FC1E for ; Mon, 12 May 2008 20:45:38 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from bunrab.catwhisker.org (localhost [127.0.0.1]) by bunrab.catwhisker.org (8.13.3/8.13.3) with ESMTP id m4CK91Kn098337 for ; Mon, 12 May 2008 13:09:01 -0700 (PDT) (envelope-from david@bunrab.catwhisker.org) Received: (from david@localhost) by bunrab.catwhisker.org (8.13.3/8.13.1/Submit) id m4CK91Io098336 for hackers@freebsd.org; Mon, 12 May 2008 13:09:01 -0700 (PDT) (envelope-from david) Date: Mon, 12 May 2008 13:09:01 -0700 From: David Wolfskill To: hackers@freebsd.org Message-ID: <20080512200901.GL66703@bunrab.catwhisker.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="R0UoHh4jwSc8qXKC" Content-Disposition: inline User-Agent: Mutt/1.4.2.1i Cc: Subject: [Patch] Using sysctl(8) to acquire info from different systems X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 20:45:39 -0000 --R0UoHh4jwSc8qXKC Content-Type: multipart/mixed; boundary="ZMT28BdW279F9lxY" Content-Disposition: inline --ZMT28BdW279F9lxY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I recently started using output from sysctl(8) to get information from some FreeBSD systems in what I hope is a low-impact way. I wanted to avoid installing or configuring any additional software on the machines; as they didn't have an SNMP daemon running (and I'm not the one who actually controls or runs them anyway), I was reluctant to recommend making such changes to production systems. But I did have a requirement to gather information as to resource consumption on the production machines, so I thought that sysctl(8) output might be of some use -- and it has (or so I believe). But I ran into one mild annoyance: Once I got a script working to collect data from one machine, and I then tried running the script (configured to collect information on the same OIDs), things were OK ... as long as the new system was running the same revision of FreeBSD. I realize that there will be changes in supported OIDs over time. But the current behavior of sysctl(8) is to (squawk unless -q is specified) and bail as soon as it sees a request to report on the value of an OID it doesn't know. In my case, I believe it would be useful to provide an ability to tell sysctl(8) to report on everything asked for that it does know, and ignore the OIDs it doesn't know. Is this percpetion so radical that I'm way off base? If so, please educate me as to why. Otherwise, I'll plan on filing a PR with the attached patch, which adds "-i" to sysctl(8)'s flags -- and which appears to work as described above: localhost(8.0-C)[1] uname -a FreeBSD localhost 8.0-CURRENT FreeBSD 8.0-CURRENT #769: Mon May 12 07:17:44= PDT 2008 root@g1-37.catwhisker.org:/common/S4/obj/usr/src/sys/CANARY = i386 localhost(8.0-C)[2] sysctl hw.ncpu kern.sched.quantum net.inet.ip.fw.enable hw.ncpu: 1 sysctl: unknown oid 'kern.sched.quantum' localhost(8.0-C)[3] sysctl -i hw.ncpu kern.sched.quantum net.inet.ip.fw.ena= ble hw.ncpu: 1 net.inet.ip.fw.enable: 1 localhost(8.0-C)[4] ^-i^-q sysctl -q hw.ncpu kern.sched.quantum net.inet.ip.fw.enable hw.ncpu: 1 localhost(8.0-C)[5]=20 Peace, david --=20 David H. Wolfskill david@catwhisker.org I submit that "conspiracy" would be an appropriate collective noun for cats. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --ZMT28BdW279F9lxY Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="sysctl.patch" Content-Transfer-Encoding: quoted-printable Index: sysctl.8 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /cvs/freebsd/src/sbin/sysctl/sysctl.8,v retrieving revision 1.64 diff -u -r1.64 sysctl.8 --- sysctl.8 28 Nov 2007 14:48:30 -0000 1.64 +++ sysctl.8 12 May 2008 19:14:08 -0000 @@ -82,6 +82,12 @@ is specified, or a variable is being set. .It Fl h Format output for human, rather than machine, readability. +.It Fl i +Ignore unknown OIDs. +The purpose is to make use of +.Nm +for collecting data from a variety of machines (not all of which +are necessarily running exactly the same software) easier. .It Fl N Show only variable names, not their values. This is particularly useful with shells that offer programmable Index: sysctl.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /cvs/freebsd/src/sbin/sysctl/sysctl.c,v retrieving revision 1.88 diff -u -r1.88 sysctl.c --- sysctl.c 15 Oct 2007 20:00:19 -0000 1.88 +++ sysctl.c 12 May 2008 19:07:34 -0000 @@ -58,8 +58,8 @@ #include #include =20 -static int aflag, bflag, dflag, eflag, hflag, Nflag, nflag, oflag; -static int qflag, xflag; +static int aflag, bflag, dflag, eflag, hflag, iflag, Nflag, nflag; +static int oflag, qflag, xflag; =20 static int oidfmt(int *, int, char *, u_int *); static void parse(char *); @@ -89,7 +89,7 @@ setbuf(stdout,0); setbuf(stderr,0); =20 - while ((ch =3D getopt(argc, argv, "AabdehNnoqwxX")) !=3D -1) { + while ((ch =3D getopt(argc, argv, "AabdehiNnoqwxX")) !=3D -1) { switch (ch) { case 'A': /* compatibility */ @@ -110,6 +110,9 @@ case 'h': hflag =3D 1; break; + case 'i': + iflag =3D 1; + break; case 'N': Nflag =3D 1; break; @@ -185,6 +188,8 @@ len =3D name2oid(bufp, mib); =20 if (len < 0) { + if (iflag) + return; if (qflag) exit(1); else --ZMT28BdW279F9lxY-- --R0UoHh4jwSc8qXKC Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (FreeBSD) iEYEARECAAYFAkgoo9wACgkQmprOCmdXAD2IygCfRJlKA3DUICvywPUGRU6jr8KP apwAnjtARZrbE24WP+q6w+7Y0xiKKHbQ =Vb7I -----END PGP SIGNATURE----- --R0UoHh4jwSc8qXKC-- From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 21:05:32 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF79B1065679 for ; Mon, 12 May 2008 21:05:32 +0000 (UTC) (envelope-from reg@openpave.org) Received: from mx3.ucdavis.edu (mx3.ucdavis.edu [128.120.32.33]) by mx1.freebsd.org (Postfix) with ESMTP id A62CF8FC15 for ; Mon, 12 May 2008 21:05:32 +0000 (UTC) (envelope-from reg@openpave.org) Received: from flint.openpave.org ([169.237.230.40]) by mx3.ucdavis.edu (8.13.7/8.13.1/it-defang-5.4.0) with ESMTP id m4CKjxoZ011965 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 12 May 2008 13:46:17 -0700 (PDT) Received: from sandy.local (flint.local [192.168.1.5]) by flint.openpave.org (8.14.2/8.14.2) with ESMTP id m4CKjwS9043620; Mon, 12 May 2008 20:45:58 GMT (envelope-from reg@sandy.local) Received: (from reg@localhost) by sandy.local (8.14.2/8.14.2/Submit) id m4CKjwOp043619; Mon, 12 May 2008 13:45:58 -0700 (PDT) (envelope-from reg) Date: Mon, 12 May 2008 13:45:58 -0700 From: Jeremy Lea To: Anders Nore Message-ID: <20080512204558.GA44114@flint.openpave.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on av8 X-Virus-Status: Clean X-Scanned-By: MIMEDefang 2.57 on 128.120.32.8 Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: Adding .db support to pkg_tools X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 21:05:32 -0000 Hi, On Fri, May 09, 2008 at 01:52:46PM +0200, Anders Nore wrote: > I'm working on adding .db support to the pkg_tools I hope that you know that you're stepping into a hotly debated part of the project... Good luck. My advice to you is to ignore any calls for a 'complete rewrite' and to focus on releasing small patches from your project rather than one big patch at the end. Back each patch up with performance numbers (I suggest pkg_add gnome2-x (from a local package source), pkg_version and pkg_delete -r glib-2.x). > The way I'm thinking of storing information to the .db is to name the > keys as the directory names in /var/db/pkg. And save the +* files in > the directories to the value element in the db, separated with a > special character or similar. I think this is a really bad idea. The problem with the tools is not with the files. It is that the files need to be parsed on each run, often recursively, and your solution would not help at all. The database(s) should just be a cache of the information stored in the files. This is what the existing databases used by portupgrade and NetBSD's tools do. They store the files and which ports installed them, so that it is easy to do conflict resolution, and some pkg_info operations. In addition, what you need is a cache which stores {pkg_name,pkg_origin} pairs, where both are unique keys with a 1:1 relationship, and then a {pkg_name,pkg_dep_name} directed graph, which stores the dependency network between ports (aslo replacing +REQUIRED_BY). Both can be reconstructed if needed from the +CONTENTS files. You could do both in BDB, but you'll be better off loading both into memory completely (as needed), using C to do the queries, and just maintaining the database when things change (on pkg_add or pkg_delete). SQLite would make this job easier in some ways, but hurt in others. You could use queries, which would make what was going on much more obvious, but then it would be slower. You also have to fight the anti-SQLite lobby. Ignore people who talk about disk space on embedded systems. They're just trolling. ;-) If they don't have the disk space for /var/db/pkg, then they don't have the space for the ports themselves. Besides, removing the redundant copy of +MTREE_DIRS for each package would save more space (and is as easy as checking the CVS revision of the new file in each package and updating a single master copy (even /etc/mtree/BSD.local.dist), if it is newer). With regards to locking, which someone raised, the package tools are a long way away from parallel modifications to /var/db/pkg. People are talking about parallel ports builds, but those have small windows where the package database is being updated, and those updates need to be sync'ed via a lock. Regards, -Jeremy -- FreeBSD - Because the best things in life are free... http://www.freebsd.org/ From owner-freebsd-hackers@FreeBSD.ORG Mon May 12 22:45:02 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34547106567A for ; Mon, 12 May 2008 22:45:02 +0000 (UTC) (envelope-from yanefbsd@gmail.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.168]) by mx1.freebsd.org (Postfix) with ESMTP id 05D938FC20 for ; Mon, 12 May 2008 22:45:01 +0000 (UTC) (envelope-from yanefbsd@gmail.com) Received: by wf-out-1314.google.com with SMTP id 28so2393356wfa.7 for ; Mon, 12 May 2008 15:45:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:in-reply-to:subject:x-priority:references:message-id:content-type:content-transfer-encoding:mime-version:date:cc:x-mailer; bh=hbsrSJAP1++z/C6WweHSJrVrS3F4RtSBNYKLL1S4a9o=; b=MP418SUA8MnrbSzbVyBTUDjaO10n78YfRip0fCaxMbaW89l4NnWK9A2VVqrX5XhXLqkPqWkCwgYjxm8eP9ptFfBUaCyeusgo1PIrx0pK3W0gClTPcZiWoNx2HaUpkI+Fkj/WM5wAfZvZMVumzIOjsPTeudD3DixFixSaHquLQnI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:in-reply-to:subject:x-priority:references:message-id:content-type:content-transfer-encoding:mime-version:date:cc:x-mailer; b=L12UzhUzMLpWldY7f8n2Ye/ybF3naV93MSLxhnUWpsiTkRevhyn41kTZUSzBPosvpzDuZidLApMsC2P4PQoDGthui/5KUlG097ORVn9JJw+QO/NdYbK+Uz5XCScEiaEkgibwimn9HEQ0IcigodncsFntVXMDcSK4Y+z0mPFrMG0= Received: by 10.142.217.17 with SMTP id p17mr3493746wfg.99.1210632301741; Mon, 12 May 2008 15:45:01 -0700 (PDT) Received: from ?192.168.10.42? ( [99.155.199.64]) by mx.google.com with ESMTPS id 28sm18810513wfd.4.2008.05.12.15.44.59 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 12 May 2008 15:45:00 -0700 (PDT) From: Garrett Cooper To: Anthony Pankov In-Reply-To: <15336578.20080512123806@mail.ru> X-Priority: 3 (Normal) References: <20080509124308.GA596@britannica.bec.de> <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> Message-Id: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v919.2) Date: Mon, 12 May 2008 15:46:07 -0700 X-Mailer: Apple Mail (2.919.2) Cc: freebsd-hackers@freebsd.org, Joerg Sonnenberger Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 22:45:02 -0000 On May 12, 2008, at 1:38 AM, Anthony Pankov wrote: > > Please, can anybody explain what is the problem with BDB (1.86). > > Is there known caveats of using BDB? Is there some rules which > guarantee from curruption or it is fully undesirable to use BDB under > high load? > > It is important for me because of using BDB in my project. > > >> On Fri, May 09, 2008 at 01:52:46PM +0200, Joerg Sonnenberger wrote: >>> >>> As one of the persons hacking on pkg_install in pkgsrc/NetBSD, I >>> would >>> *strongly* advisy you against storing the files only in a bdb file. >>> The change of major and complete corruption with bdb185 is high, >>> consider pulling the plug in the middle of a long update. > >> Sunday, May 11, 2008, 5:38:25 PM, you wrote: > > GC> +1. BDB is quite easy to corrupt... BDB isn't ATOMic, like SQL or other DB backends. -Garrett From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 02:18:07 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E05591065690 for ; Tue, 13 May 2008 02:18:07 +0000 (UTC) (envelope-from lidl@pix.net) Received: from rox.fddx.com (rox.fddx.com [71.241.230.59]) by mx1.freebsd.org (Postfix) with ESMTP id 7846D8FC1B for ; Tue, 13 May 2008 02:18:07 +0000 (UTC) (envelope-from lidl@pix.net) Received: from torb.pix.net (torb.pix.net [192.168.16.32]) by rox.fddx.com (8.13.8+Sun/8.13.8) with ESMTP id m4D1ukoW014278; Mon, 12 May 2008 21:56:46 -0400 (EDT) Message-ID: <4828F55E.6020407@pix.net> Date: Mon, 12 May 2008 21:56:46 -0400 From: Kurt Lidl User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213) MIME-Version: 1.0 To: Garrett Cooper References: <20080509124308.GA596@britannica.bec.de> <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on rox.fddx.com X-Virus-Status: Clean Cc: freebsd-hackers@freebsd.org, Joerg Sonnenberger , Anthony Pankov Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 02:18:08 -0000 Garrett Cooper wrote: > On May 12, 2008, at 1:38 AM, Anthony Pankov wrote: > >> >> Please, can anybody explain what is the problem with BDB (1.86). >> >> Is there known caveats of using BDB? Is there some rules which >> guarantee from curruption or it is fully undesirable to use BDB under >> high load? >> >> It is important for me because of using BDB in my project. >> >> >>> On Fri, May 09, 2008 at 01:52:46PM +0200, Joerg Sonnenberger wrote: >>>> >>>> As one of the persons hacking on pkg_install in pkgsrc/NetBSD, I would >>>> *strongly* advisy you against storing the files only in a bdb file. >>>> The change of major and complete corruption with bdb185 is high, >>>> consider pulling the plug in the middle of a long update. >> >>> Sunday, May 11, 2008, 5:38:25 PM, you wrote: >> >> GC> +1. BDB is quite easy to corrupt... > > BDB isn't ATOMic, like SQL or other DB backends. You mean ACID probably. And there are plenty of SQL databases that aren't ACID either. (e.g. Mysql 4.x, Mysql 5.x w/o the right kind of backing store) -Kurt From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 02:55:07 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EAE96106567B for ; Tue, 13 May 2008 02:55:07 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.29]) by mx1.freebsd.org (Postfix) with ESMTP id A5DA98FC14 for ; Tue, 13 May 2008 02:55:07 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by yw-out-2324.google.com with SMTP id 9so1433523ywe.13 for ; Mon, 12 May 2008 19:55:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; bh=YYVVyaJ6M1ZHEhi3C4LPXpJZEwQoVgCT5fDb0aDjFWQ=; b=TtFSpt4bXU5TQU+y12PYq4nos2e27SSL4Nu7pmRgmg8MdSyr2xCl0q/odsKpFRKHRYSCnze5Qwm9aJsKLr7lKWdcVu6d8Gg0Bc+DNCihjsZxo/OwB5qQ3UsgYWMR/bnIAs8ulpBhfalSE/T3Nr8W+HNrBI4i4r19h1j3AUF0r40= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=FhJFEneZxjC9D3UrhtBbWxaoXypfVcs7ysEHFTiQza1rWPn2eQ7xFe3jc/Z0p2i8IkestMEVyPj2aB7uPa9gmY79TH69n7pddBs7S/QHbftXnxi9puJAYjkiUVR97kyAIt1GdySU8Wpa1SZzfu/HE7ohyROBpUijRTID8SV8FkM= Received: by 10.150.79.42 with SMTP id c42mr9069982ybb.163.1210647307091; Mon, 12 May 2008 19:55:07 -0700 (PDT) Received: by 10.150.53.11 with HTTP; Mon, 12 May 2008 19:55:07 -0700 (PDT) Message-ID: <5f67a8c40805121955t3269b1d5we302dda607e21573@mail.gmail.com> Date: Mon, 12 May 2008 22:55:07 -0400 From: "Zaphod Beeblebrox" To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: ZFS and mmap() datapoint. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 02:55:08 -0000 I don't know exactly where this fits in the discussion, but I was using "ghost" v 14 to backup my XP box over SMB to a 7.0-RELEASE system using ZFS. After doing one backup, the second backup wouldn't proceed, so I erased the backup files and started fresh. The next attempt, I turned on the verify option in ghost. It claimed that the backup it was creating was corrupt. So I upgraded the FreeBSD machine to 7-STABLE. Started the backup again, same problem. So I checked into samba's options ... and it has an option that claims to turn off the mmap() usage of the daemon. It certainly slows the daemon, so it's doing something. Anyways... the backup was still corrupt. Obviously... this is testing at a high layer. Does the option in smb.conf turn off the use of mmap(). Is that moot because C libraries might use mmap() anyways? Is ghost the culprit here? I dunno. The ZFS array does handle linear reads and writes of large media files without detectable error. From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 04:20:42 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 955DD106566B for ; Tue, 13 May 2008 04:20:42 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from kientzle.com (h-66-166-149-50.snvacaid.covad.net [66.166.149.50]) by mx1.freebsd.org (Postfix) with ESMTP id 6B0F18FC2C for ; Tue, 13 May 2008 04:20:42 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from [10.0.0.128] (p54.kientzle.com [66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id m4D4Kftv098830; Mon, 12 May 2008 21:20:41 -0700 (PDT) (envelope-from kientzle@freebsd.org) Message-ID: <48291719.5010801@freebsd.org> Date: Mon, 12 May 2008 21:20:41 -0700 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060422 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Jeremy Lea References: <20080512204558.GA44114@flint.openpave.org> In-Reply-To: <20080512204558.GA44114@flint.openpave.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Anders Nore Subject: Re: Adding .db support to pkg_tools X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 04:20:42 -0000 > I think this is a really bad idea. The problem with the tools is not > with the files. It is that the files need to be parsed on each run, > often recursively, and your solution would not help at all. Parsing one file isn't expensive; parsing several hundred files to find one bit of information is expensive. > The database(s) should just be a cache of the information stored in the > files. Bingo! As long as the .db version can be easily recreated from scratch from the master data stored in the same files as always, it doesn't really matter if the BDB is occasionally corrupted, as long as it can be rebuilt fairly quickly. So, the key points are: * Use the .db file for lookups that can't be done quickly against the existing data. In particular, look for operations where the pkg tools have to scan a lot of files to verify a single fact. * Ensure the .db file can be deleted and rebuilt from scratch from the data in the regular files. Tim Kientzle From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 11:41:42 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DA20D1065672 for ; Tue, 13 May 2008 11:41:42 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from mx0.awanti.com (mx0.awanti.com [91.190.112.18]) by mx1.freebsd.org (Postfix) with ESMTP id A1A2D8FC17 for ; Tue, 13 May 2008 11:41:42 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from pstation (unknown [10.28.4.14]) by mx0.awanti.com (Postfix) with ESMTP id C53C34C194 for ; Tue, 13 May 2008 15:41:40 +0400 (MSD) Date: Tue, 13 May 2008 15:44:06 +0400 From: Anthony Pankov X-Mailer: The Bat! (v1.51) Personal X-Priority: 3 (Normal) Message-ID: <2117635718.20080513154406@mail.ru> To: freebsd-hackers@freebsd.org In-Reply-To: <20080512152430.3720683e@mbook.local> References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> MIME-Version: 1.0 Content-Type: text/plain; charset=Windows-1251 Content-Transfer-Encoding: 8bit Subject: Re[2]: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Anthony Pankov List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 11:41:42 -0000 Monday, Mike Meyer May 12, 2008, 11:24:30 PM, you wrote: MM> On Mon, 12 May 2008 22:35:31 +0400 Anthony Pankov wrote: >> Because BDB: >> 1. do not need additional installation >> 2. is part of base system which mean it is mature, reliable and stable MM> BDB in the base system is mature, reliable and stable *for what it's MM> used for in the base system.* So long as your requirements are covered MM> by that usage, you'll be ok. MM> The uses I know of for BDB in the base system all consist of databases MM> of relatively small items that are changed infrequently, and usually MM> with a locking mechanism. From what you've said, this doesn't describe MM> your requirements. MM> More importantly, from what other people are saying, your requirements MM> are ones for which it's known that BDB is *not* reliable, or otherwise MM> unsuitable. In particular, an effort is underway to allow parallel MM> ports builds, which implies concurrent access to the database, which MM> is a known source of problems for BDB. MM> Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BE0A31065674 for ; Tue, 13 May 2008 12:14:52 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id B46D88FC1B for ; Tue, 13 May 2008 12:14:52 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 8B0FD1CC033; Tue, 13 May 2008 05:14:52 -0700 (PDT) Date: Tue, 13 May 2008 05:14:52 -0700 From: Jeremy Chadwick To: Anthony Pankov Message-ID: <20080513121452.GA70860@eos.sc1.parodius.com> References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2117635718.20080513154406@mail.ru> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: freebsd-hackers@freebsd.org Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 12:14:52 -0000 On Tue, May 13, 2008 at 03:44:06PM +0400, Anthony Pankov wrote: > If concurrency is the only problem then: > 1. ?an data corruption be avoided? Or this is impossible? > 2. How? Use Sleepycat/Oracle DB instead? The libc DB1.x, despite being "mature", really should be deprecated in some manner. I'm sure there are others I've forgotten, but the only thing I can think of in the base system which relies on DB1.x is sendmail (which IMHO should really be removed from the base system and replaced with a small standalone mailer -- but that's been discussed in a previous thread in the past). Even "simple" ports like postgrey pull in db41, even though they could technically "work fine" with DB1.x. > If all BDB readers would use O_SHLOCK and all writers O_EXLOCK is it > guarantee for data integrity? The corruption I've seen in the past results in DB operations failing for no particular reason ("what do you mean those are all the records? No! I just inserted a bunch more!"). It turns out some of the data in the actual .db file is corrupt -- even when locking is used everywhere. It's as if the code has some weird bug where it'll write bogus data to the file for some reason. I'll ask you this: is there some particular reason you can't just write to a file yourself, using your own/documented file format? Why does DB have to be used? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 13:00:54 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9326A1065677 for ; Tue, 13 May 2008 13:00:54 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from www.pkgsrc-box.org (www.ostsee-abc.de [62.206.222.50]) by mx1.freebsd.org (Postfix) with ESMTP id 5ADDC8FC21 for ; Tue, 13 May 2008 13:00:54 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from britannica.bec.de (www.pkgsrc-box.org [127.0.0.1]) by www.pkgsrc-box.org (Postfix) with ESMTP id E7C92E506AA for ; Tue, 13 May 2008 13:00:41 +0000 (UTC) Received: by britannica.bec.de (Postfix, from userid 1000) id 2CF2416FC3; Tue, 13 May 2008 14:59:57 +0200 (CEST) Date: Tue, 13 May 2008 14:59:57 +0200 From: Joerg Sonnenberger To: freebsd-hackers@freebsd.org Message-ID: <20080513125956.GD397@britannica.bec.de> Mail-Followup-To: freebsd-hackers@freebsd.org References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2117635718.20080513154406@mail.ru> User-Agent: Mutt/1.5.17 (2007-11-01) Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 13:00:54 -0000 On Tue, May 13, 2008 at 03:44:06PM +0400, Anthony Pankov wrote: > 3. reading/writing = 60%/40% I don't know where you get those numbers from, but they feel *very* wrong from the perspective of someone who actually dealt a lot with those tools. Writing is only a very small part of the operations and typically done a single large batch. For the same reason a single reader/writer lock is perfective fine. Joerg From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 15:50:54 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07E7C106564A for ; Tue, 13 May 2008 15:50:54 +0000 (UTC) (envelope-from yanefbsd@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.183]) by mx1.freebsd.org (Postfix) with ESMTP id CEA2F8FC0A for ; Tue, 13 May 2008 15:50:53 +0000 (UTC) (envelope-from yanefbsd@gmail.com) Received: by wa-out-1112.google.com with SMTP id j4so3690832wah.3 for ; Tue, 13 May 2008 08:50:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:cc:message-id:from:to:in-reply-to:content-type:content-transfer-encoding:mime-version:subject:date:references:x-mailer; bh=hHNJlqPwsG+1L+lQDJ3Ykrjl1gQp7/qmCUkcIV/umqk=; b=w+jjm5sG9hAbCv43HZtHgCsEV02hJwN99YZSwEGuIDvLfopDVWzaRlFliApW17wJhm86JEgl0vvqapniMs+hQDFZa+y/Utj/3FhzrD2tNMqQMJK95+kig48UTczjNJcweiPini+6+Bl3FNWMNCfWmLW50vPAA5zcsEHww/NT4RM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=cc:message-id:from:to:in-reply-to:content-type:content-transfer-encoding:mime-version:subject:date:references:x-mailer; b=td4UNl8G5C6xLjvdKPxtbeVyHx5o7KtrHIsWxcCpBXiy5wF8BHR/7MaLOpwVLPOXLKycSxTf9g4znUiDHfcBJGJhIJ03HkdH77u7QVxgilnaLKRLVElD/IsQo2HYw1aEszccOPQEksVjyGrRic07eqzQ8YO0yg17tXscZTHz5Ow= Received: by 10.114.180.1 with SMTP id c1mr9355433waf.121.1210693853412; Tue, 13 May 2008 08:50:53 -0700 (PDT) Received: from ?192.168.10.42? ( [99.155.199.64]) by mx.google.com with ESMTPS id n20sm13109pof.0.2008.05.13.08.50.51 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 13 May 2008 08:50:52 -0700 (PDT) Message-Id: <48B3E83F-A165-4680-8F29-4EE6FB60D079@gmail.com> From: Garrett Cooper To: Kurt Lidl In-Reply-To: <4828F55E.6020407@pix.net> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v919.2) Date: Tue, 13 May 2008 08:51:59 -0700 References: <20080509124308.GA596@britannica.bec.de> <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <4828F55E.6020407@pix.net> X-Mailer: Apple Mail (2.919.2) Cc: freebsd-hackers@freebsd.org, Joerg Sonnenberger , Anthony Pankov Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 15:50:54 -0000 On May 12, 2008, at 6:56 PM, Kurt Lidl wrote: > Garrett Cooper wrote: >> On May 12, 2008, at 1:38 AM, Anthony Pankov wrote: >>> >>> Please, can anybody explain what is the problem with BDB (1.86). >>> >>> Is there known caveats of using BDB? Is there some rules which >>> guarantee from curruption or it is fully undesirable to use BDB >>> under >>> high load? >>> >>> It is important for me because of using BDB in my project. >>> >>> >>>> On Fri, May 09, 2008 at 01:52:46PM +0200, Joerg Sonnenberger wrote: >>>>> >>>>> As one of the persons hacking on pkg_install in pkgsrc/NetBSD, I >>>>> would >>>>> *strongly* advisy you against storing the files only in a bdb >>>>> file. >>>>> The change of major and complete corruption with bdb185 is high, >>>>> consider pulling the plug in the middle of a long update. >>> >>>> Sunday, May 11, 2008, 5:38:25 PM, you wrote: >>> >>> GC> +1. BDB is quite easy to corrupt... >> BDB isn't ATOMic, like SQL or other DB backends. > > You mean ACID probably. And there are plenty of SQL databases > that aren't ACID either. (e.g. Mysql 4.x, Mysql 5.x w/o the > right kind of backing store) > > -Kurt Yes, you're right. Atomicity is the A in ACID. -Garrett From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 16:03:07 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4532B106566C for ; Tue, 13 May 2008 16:03:07 +0000 (UTC) (envelope-from lidl@rox.fddx.com) Received: from rox.fddx.com (rox.fddx.com [71.241.230.59]) by mx1.freebsd.org (Postfix) with ESMTP id DA0F88FC19 for ; Tue, 13 May 2008 16:03:06 +0000 (UTC) (envelope-from lidl@rox.fddx.com) Received: from rox.fddx.com (localhost.pix.net [127.0.0.1]) by rox.fddx.com (8.13.8+Sun/8.13.8) with ESMTP id m4DFmfeA029152; Tue, 13 May 2008 11:48:41 -0400 (EDT) Received: (from lidl@localhost) by rox.fddx.com (8.13.8+Sun/8.13.8/Submit) id m4DFmf0V029151; Tue, 13 May 2008 11:48:41 -0400 (EDT) Date: Tue, 13 May 2008 11:48:41 -0400 From: "Kurt J. Lidl" To: Anthony Pankov Message-ID: <20080513154840.GB28842@pix.net> Mail-Followup-To: Anthony Pankov , freebsd-hackers@freebsd.org References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2117635718.20080513154406@mail.ru> User-Agent: Mutt/1.5.16 (2007-06-09) X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on rox.fddx.com X-Virus-Status: Clean Cc: freebsd-hackers@freebsd.org Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 16:03:07 -0000 On Tue, May 13, 2008 at 03:44:06PM +0400, Anthony Pankov wrote: > My requirements is > 1. there is no need for SQL > 2. processes are sharing db file in concurrent mode > 3. reading/writing = 60%/40% > > With BDB > clause 1 - satisfied > clause 3 - satisfied (databases of relatively small items that are > changed infrequently). > > Is there a problem with concurrent access? And, most important, is > this the ONLY problem? (I still don't understand because of blurred > mention of "data corruption".) > > If concurrency is the only problem then: > 1. Can data corruption be avoided? Or this is impossible? > 2. How? > > If all BDB readers would use O_SHLOCK and all writers O_EXLOCK is it > guarantee for data integrity? This is probably good enough, if your writers also do a fflush() before releasing the lock. If the writing process ever dies horribly, you can still have database corruption. You could turn the thing that does the actual I/O into the database into daemon that fullfils read/write requests from a unix domain socket. In fact, there is a sample application that does exactly that for the newer Sleepycat BDB code. It could probably be adapted or re-written for the 1.8x series code. If it were me, I'd just punt on using the 1.8x code, drop straight into using the BDB 4.6 codebase, and use their transactioning code. It's mature, robust, etc, etc, for a much large set of operations than the 1.8x code ever was. But there's still that licensing issue. -Kurt From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 16:03:07 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DD3FE1065670; Tue, 13 May 2008 16:03:07 +0000 (UTC) (envelope-from lidl@rox.fddx.com) Received: from rox.fddx.com (rox.fddx.com [71.241.230.59]) by mx1.freebsd.org (Postfix) with ESMTP id 842898FC1E; Tue, 13 May 2008 16:03:07 +0000 (UTC) (envelope-from lidl@rox.fddx.com) Received: from rox.fddx.com (localhost.pix.net [127.0.0.1]) by rox.fddx.com (8.13.8+Sun/8.13.8) with ESMTP id m4DFfdZN029027; Tue, 13 May 2008 11:41:39 -0400 (EDT) Received: (from lidl@localhost) by rox.fddx.com (8.13.8+Sun/8.13.8/Submit) id m4DFfd4r029026; Tue, 13 May 2008 11:41:39 -0400 (EDT) Date: Tue, 13 May 2008 11:41:39 -0400 From: "Kurt J. Lidl" To: Jeremy Chadwick Message-ID: <20080513154137.GA28842@pix.net> Mail-Followup-To: Jeremy Chadwick , Anthony Pankov , freebsd-hackers@freebsd.org References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> <20080513121452.GA70860@eos.sc1.parodius.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080513121452.GA70860@eos.sc1.parodius.com> User-Agent: Mutt/1.5.16 (2007-06-09) X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on rox.fddx.com X-Virus-Status: Clean Cc: freebsd-hackers@freebsd.org, Anthony Pankov Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 16:03:08 -0000 On Tue, May 13, 2008 at 05:14:52AM -0700, Jeremy Chadwick wrote: > On Tue, May 13, 2008 at 03:44:06PM +0400, Anthony Pankov wrote: > > If concurrency is the only problem then: > > 1. ?an data corruption be avoided? Or this is impossible? > > 2. How? > > Use Sleepycat/Oracle DB instead? The libc DB1.x, despite being > "mature", really should be deprecated in some manner. This catapults back into the arena of "stuff that isn't in the base system". Not to mention I'm not sure that the Oracle BDB license would allow bundling in the OS as a binary. I doubt it, but that's a different bikeshed to paint :-) > I'm sure there are others I've forgotten, but the only thing I can think > of in the base system which relies on DB1.x is sendmail (which IMHO > should really be removed from the base system and replaced with a small > standalone mailer -- but that's been discussed in a previous thread in > the past). Even "simple" ports like postgrey pull in db41, even though > they could technically "work fine" with DB1.x. Well, on a 7.0 machine, it looks like the following: /etc/mail/aliases.db /etc/{s,}pwd.db /usr/share/misc/{termcap,vgrindefs}.db And of course, nvi uses the db code's recno interface to do all its work. > > If all BDB readers would use O_SHLOCK and all writers O_EXLOCK is it > > guarantee for data integrity? > > The corruption I've seen in the past results in DB operations failing > for no particular reason ("what do you mean those are all the records? > No! I just inserted a bunch more!"). It turns out some of the data in > the actual .db file is corrupt -- even when locking is used everywhere. > It's as if the code has some weird bug where it'll write bogus data to > the file for some reason. There are known problems with certain keys corrupting the DB 1.8x series code. In fact, the "release" of the 1.86 was an attempt to solve this problem when the KerberosV people at MIT found a repeatable key insert sequence that would corrupt things. (Or at least that's what I remember, it was a long time ago, and I might have the details wrong.) -Kurt From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 16:13:54 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 467F11065677 for ; Tue, 13 May 2008 16:13:54 +0000 (UTC) (envelope-from romain@blogreen.org) Received: from smtp3-g19.free.fr (smtp3-g19.free.fr [212.27.42.29]) by mx1.freebsd.org (Postfix) with ESMTP id D544D8FC1D for ; Tue, 13 May 2008 16:13:53 +0000 (UTC) (envelope-from romain@blogreen.org) Received: from smtp3-g19.free.fr (localhost.localdomain [127.0.0.1]) by smtp3-g19.free.fr (Postfix) with ESMTP id 31BD917B548 for ; Tue, 13 May 2008 18:13:52 +0200 (CEST) Received: from marvin.blogreen.org (marvin.blogreen.org [82.247.213.140]) by smtp3-g19.free.fr (Postfix) with ESMTP id 1A58017B589 for ; Tue, 13 May 2008 18:13:51 +0200 (CEST) Received: by marvin.blogreen.org (Postfix, from userid 1001) id 9C9EF5C095; Tue, 13 May 2008 18:13:51 +0200 (CEST) Date: Tue, 13 May 2008 18:13:51 +0200 From: Romain =?iso-8859-1?Q?Tarti=E8re?= To: freebsd-hackers@freebsd.org Message-ID: <20080513161351.GA10097@marvin.blogreen.org> Mail-Followup-To: freebsd-hackers@freebsd.org Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ibTvN161/egqYuK8" Content-Disposition: inline User-Agent: Mutt/1.4.2.3i X-PGP-Key: http://romain.blogreen.org/pubkey.asc Subject: Adaptec RAID 3805 and FreeBSD 7.0 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 16:13:54 -0000 --ibTvN161/egqYuK8 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi! I am trying to have an Adaptec RAID 3805 controller working on FreeBSD 7.0. According to the release note [1], it's okay. Unfortunately, the driver fails to detect everything as expected. The following is displayed at boot-time: > aac0: mem 0xff200000-0xff3fffff irq 16 at device 14.0= on pci3 > aac0: couldn't allocate register window 0 > device_attach: aac0 attach returned 6 The CD provided with the controller include kernel modules for FreeBSD 5 and 6, but not 7 (I however tried to kldload it (Yes, I am _very_ optimistic by nature) but link_elf fails with "symbol msleep undefined"). The Adaptec website does not provide drivers for FreeBSD 7.0, but has a much more recent version of the driver than the one on the CD, so I fetched it, and tried to compile it (AACRAID 32- and 64-bit Driver Build 15727 for FreeBSD 6.1 and 6.2). I fixed the source the "lazy way", that is fixing that makes the compilation fail untill it pass. The result was not much better: > aacu0: mem 0xff200000-0xff3fffff irq 16 at device 14.= 0 on pci3 > aacu0: couldn't allocate register window 0 > device_attach: aacu0 attach returned 12 So I took the aac(4) source code of FreeBSD 6.2-STABLE, 7.0-STABLE and the version of Adaptec and tried to merge changes from 6.2 to 7.0 in the Adaptec code... The result is however almost the same :-( Has anybody tried / succeeded in using such a controller on FreeBSD 7.0-CURRENT? Has somebody this controller working on FreeBSD 6.3 (I can consider to downgrade to the legacy release if required since the machine is not yet in production)? Thank you! Romain References: 1. http://www.freebsd.org/releases/7.0R/hardware.html#DISK --=20 Romain Tarti=E8re http://romain.blogreen.org/ pgp: 8DAB A124 0DA4 7024 F82A E748 D8E9 A33F FF56 FF43 (ID: 0xFF56FF43) (plain text =3Dnon-HTML=3D PGP/GPG encrypted/signed e-mail much appreciated) --ibTvN161/egqYuK8 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkgpvj8ACgkQ2OmjP/9W/0OfewCdF93k/Zvo7en8Q4xlm0iNrhQg H6kAn3nxKlvYmduX8zU77sDDHk7yt9so =slzr -----END PGP SIGNATURE----- --ibTvN161/egqYuK8-- From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 19:05:04 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E5DF71065686; Tue, 13 May 2008 19:05:04 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from hosted.kievnet.com (hosted.kievnet.com [193.138.144.10]) by mx1.freebsd.org (Postfix) with ESMTP id 81F088FC12; Tue, 13 May 2008 19:05:04 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost ([127.0.0.1] helo=edge.pp.kiev.ua) by hosted.kievnet.com with esmtpa (Exim 4.62) (envelope-from ) id 1Jvzo5-0000Uv-QP; Tue, 13 May 2008 22:05:01 +0300 Message-ID: <4829E658.3000605@icyb.net.ua> Date: Tue, 13 May 2008 22:04:56 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.12 (X11/20080320) MIME-Version: 1.0 To: Kostik Belousov References: <480E4269.2090604@icyb.net.ua> <480FBAB9.1000904@icyb.net.ua> <48103F36.6060707@icyb.net.ua> <200804240811.26183.jhb@freebsd.org> <4810FD1E.70602@icyb.net.ua> <20080425095009.GD18958@deviant.kiev.zoral.com.ua> <4811E6BC.4060306@icyb.net.ua> <20080425143646.GF18958@deviant.kiev.zoral.com.ua> <48275C0C.2040601@icyb.net.ua> <20080511214833.GB18958@deviant.kiev.zoral.com.ua> In-Reply-To: <20080511214833.GB18958@deviant.kiev.zoral.com.ua> Content-Type: multipart/mixed; boundary="------------080908040200020205000508" Cc: freebsd-hackers@freebsd.org Subject: Re: devctl (alike?) for devfs X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 19:05:05 -0000 This is a multi-part message in MIME format. --------------080908040200020205000508 Content-Type: text/plain; charset=KOI8-U; format=flowed Content-Transfer-Encoding: 7bit on 12/05/2008 00:48 Kostik Belousov said the following: > No, we do not have a leak, but we have somewhat non-obvious behaviour. > > The cdev structure is freed only after the last reference to cdev is > gone. Typical holder of the reference is the devfs vnode. In the normal > usage, the vnode is present until both the device is destroyed _and_ > devfs_populate_loop() run is performed. This function actually reclaim > the vnodes for destroyed devices, that causes last reference to cdev to > be dropped and memory freed. > > The populate loop is called syncronously from the upper levels. The > easiest method to trigger it is to do ls /dev, since it is called from > the devfs_lookupx(). Thank you for the explanation! ls did trigger DESTROY notifications. But this arbitrary delay between device entry going away and notification about about it is a little bit "uncool". So I re-wrote the patch to post notifications about deleted devices with si_refcount > 0 in a fashion similar to how si_refcount=0 devices are freed. I put those cdev-s onto a special list (re-/ab-using si_siblings LIST link field) and bump their si_refcount to prevent them from getting destroyed before the notification is sent (it would need si_name). Then, in dev_unlock_and_free() I send notifications and call dev_rel on the devices. I am not entirely satisfied with the code: 1. I don't like the way I move LIST elements from one head to the other, this should be a macro in queue.h 2. I am not sure about the names I picked 3. I slightly don't like a fact that parent-child destroy notifications are sent in reverse order because of my simplistic LIST usage. -- Andriy Gapon --------------080908040200020205000508 Content-Type: text/plain; name="devfs-notify.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="devfs-notify.diff" diff --git a/sys/kern/kern_conf.c b/sys/kern/kern_conf.c index 1db25f8..0245253 100644 --- a/sys/kern/kern_conf.c +++ b/sys/kern/kern_conf.c @@ -30,6 +30,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -99,6 +100,9 @@ dev_unlock_and_free(void) mtx_unlock(&devmtx); while ((cdp = TAILQ_FIRST(&cdp_free)) != NULL) { + if (!cold) + devctl_notify("DEVFS", cdp->cdp_c.si_name, "DESTROY", NULL); + TAILQ_REMOVE(&cdp_free, cdp, cdp_list); devfs_free(&cdp->cdp_c); } @@ -172,8 +176,12 @@ dev_rel(struct cdev *dev) flag = 1; } dev_unlock(); - if (flag) + if (flag) { + if (!cold) + devctl_notify("DEVFS", dev->si_name, "DESTROY", NULL); + devfs_free(dev); + } } struct cdevsw * @@ -706,6 +714,10 @@ make_dev_credv(int flags, struct cdevsw *devsw, int minornr, devfs_create(dev); clean_unrhdrl(devfs_inos); dev_unlock_and_free(); + + if (!cold) + devctl_notify("DEVFS", dev->si_name, "CREATE", NULL); + return (dev); } @@ -794,6 +806,10 @@ make_dev_alias(struct cdev *pdev, const char *fmt, ...) clean_unrhdrl(devfs_inos); dev_unlock(); dev_depends(pdev, dev); + + if (!cold) + devctl_notify("DEVFS", dev->si_name, "CREATE", NULL); + return (dev); } --------------080908040200020205000508-- From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 19:09:34 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2EA511065673; Tue, 13 May 2008 19:09:34 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from hosted.kievnet.com (hosted.kievnet.com [193.138.144.10]) by mx1.freebsd.org (Postfix) with ESMTP id B82938FC15; Tue, 13 May 2008 19:09:33 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost ([127.0.0.1] helo=edge.pp.kiev.ua) by hosted.kievnet.com with esmtpa (Exim 4.62) (envelope-from ) id 1JvzsS-0000iw-Oj; Tue, 13 May 2008 22:09:32 +0300 Message-ID: <4829E76B.8010204@icyb.net.ua> Date: Tue, 13 May 2008 22:09:31 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.12 (X11/20080320) MIME-Version: 1.0 To: Kostik Belousov References: <480E4269.2090604@icyb.net.ua> <480FBAB9.1000904@icyb.net.ua> <48103F36.6060707@icyb.net.ua> <200804240811.26183.jhb@freebsd.org> <4810FD1E.70602@icyb.net.ua> <20080425095009.GD18958@deviant.kiev.zoral.com.ua> <4811E6BC.4060306@icyb.net.ua> <20080425143646.GF18958@deviant.kiev.zoral.com.ua> <48275C0C.2040601@icyb.net.ua> <20080511214833.GB18958@deviant.kiev.zoral.com.ua> <4829E658.3000605@icyb.net.ua> In-Reply-To: <4829E658.3000605@icyb.net.ua> Content-Type: multipart/mixed; boundary="------------000802000206000606030404" Cc: freebsd-hackers@freebsd.org Subject: Re: devctl (alike?) for devfs X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 19:09:34 -0000 This is a multi-part message in MIME format. --------------000802000206000606030404 Content-Type: text/plain; charset=KOI8-U; format=flowed Content-Transfer-Encoding: 7bit on 13/05/2008 22:04 Andriy Gapon said the following: > on 12/05/2008 00:48 Kostik Belousov said the following: >> No, we do not have a leak, but we have somewhat non-obvious behaviour. >> >> The cdev structure is freed only after the last reference to cdev is >> gone. Typical holder of the reference is the devfs vnode. In the normal >> usage, the vnode is present until both the device is destroyed _and_ >> devfs_populate_loop() run is performed. This function actually reclaim >> the vnodes for destroyed devices, that causes last reference to cdev to >> be dropped and memory freed. >> >> The populate loop is called syncronously from the upper levels. The >> easiest method to trigger it is to do ls /dev, since it is called from >> the devfs_lookupx(). > > Thank you for the explanation! ls did trigger DESTROY notifications. > But this arbitrary delay between device entry going away and > notification about about it is a little bit "uncool". > > So I re-wrote the patch to post notifications about deleted devices with > si_refcount > 0 in a fashion similar to how si_refcount=0 devices are > freed. I put those cdev-s onto a special list (re-/ab-using si_siblings > LIST link field) and bump their si_refcount to prevent them from getting > destroyed before the notification is sent (it would need si_name). > Then, in dev_unlock_and_free() I send notifications and call dev_rel on > the devices. > > I am not entirely satisfied with the code: > 1. I don't like the way I move LIST elements from one head to the other, > this should be a macro in queue.h > 2. I am not sure about the names I picked > 3. I slightly don't like a fact that parent-child destroy notifications > are sent in reverse order because of my simplistic LIST usage. > Oops! I attached the older patch. New patch is here. -- Andriy Gapon --------------000802000206000606030404 Content-Type: text/plain; name="devfs-notify.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="devfs-notify.diff" diff --git a/sys/kern/kern_conf.c b/sys/kern/kern_conf.c index 1db25f8..7106182 100644 --- a/sys/kern/kern_conf.c +++ b/sys/kern/kern_conf.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include #include @@ -63,6 +64,8 @@ static struct cdev_priv_list cdevp_free_list = TAILQ_HEAD_INITIALIZER(cdevp_free_list); static SLIST_HEAD(free_cdevsw, cdevsw) cdevsw_gt_post_list = SLIST_HEAD_INITIALIZER(); +static LIST_HEAD(cdev_notif, cdev) cdev_notif_list = + LIST_HEAD_INITIALIZER(cdev_notif_list); void dev_lock(void) @@ -82,8 +85,10 @@ dev_unlock_and_free(void) { struct cdev_priv_list cdp_free; struct free_cdevsw csw_free; + struct cdev_notif cd_notif; struct cdev_priv *cdp; struct cdevsw *csw; + struct cdev *cd; mtx_assert(&devmtx, MA_OWNED); @@ -95,10 +100,23 @@ dev_unlock_and_free(void) TAILQ_CONCAT(&cdp_free, &cdevp_free_list, cdp_list); csw_free = cdevsw_gt_post_list; SLIST_INIT(&cdevsw_gt_post_list); + /* XXX How to do the following nicely? */ + cd_notif = cdev_notif_list; + if (!LIST_EMPTY(&cd_notif)) + LIST_FIRST(&cd_notif)->si_siblings.le_prev = &LIST_FIRST(&cd_notif); + LIST_INIT(&cdev_notif_list); mtx_unlock(&devmtx); + while ((cd = LIST_FIRST(&cd_notif)) != NULL) { + if (!cold) + devctl_notify("DEVFS", cd->si_name, "DESTROY", NULL); + LIST_REMOVE(cd, si_siblings); + dev_rel(cd); + } while ((cdp = TAILQ_FIRST(&cdp_free)) != NULL) { + if (!cold) + devctl_notify("DEVFS", cdp->cdp_c.si_name, "DESTROY", NULL); TAILQ_REMOVE(&cdp_free, cdp, cdp_list); devfs_free(&cdp->cdp_c); } @@ -706,6 +724,10 @@ make_dev_credv(int flags, struct cdevsw *devsw, int minornr, devfs_create(dev); clean_unrhdrl(devfs_inos); dev_unlock_and_free(); + + if (!cold) + devctl_notify("DEVFS", dev->si_name, "CREATE", NULL); + return (dev); } @@ -794,6 +816,10 @@ make_dev_alias(struct cdev *pdev, const char *fmt, ...) clean_unrhdrl(devfs_inos); dev_unlock(); dev_depends(pdev, dev); + + if (!cold) + devctl_notify("DEVFS", dev->si_name, "CREATE", NULL); + return (dev); } @@ -861,6 +887,9 @@ destroy_devl(struct cdev *dev) if (dev->si_refcount > 0) { LIST_INSERT_HEAD(&dead_cdevsw.d_devs, dev, si_list); + /* (re-/ab-)use si_siblings for destroy notification list */ + LIST_INSERT_HEAD(&cdev_notif_list, dev, si_siblings); + dev_refl(dev); /* Avoid race with dev_rel() */ } else { dev_free_devlocked(dev); } --------------000802000206000606030404-- From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 19:16:24 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BADE21065676; Tue, 13 May 2008 19:16:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay02.kiev.sovam.com (relay02.kiev.sovam.com [62.64.120.197]) by mx1.freebsd.org (Postfix) with ESMTP id 52FBF8FC25; Tue, 13 May 2008 19:16:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=skuns.kiev.zoral.com.ua) by relay02.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1Jvzz4-0004Hj-JG; Tue, 13 May 2008 22:16:22 +0300 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by skuns.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4DJGCro088179 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 13 May 2008 22:16:12 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4DJGAuG082232; Tue, 13 May 2008 22:16:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m4DJGAw7082231; Tue, 13 May 2008 22:16:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 13 May 2008 22:16:10 +0300 From: Kostik Belousov To: Andriy Gapon Message-ID: <20080513191610.GK18958@deviant.kiev.zoral.com.ua> References: <480FBAB9.1000904@icyb.net.ua> <48103F36.6060707@icyb.net.ua> <200804240811.26183.jhb@freebsd.org> <4810FD1E.70602@icyb.net.ua> <20080425095009.GD18958@deviant.kiev.zoral.com.ua> <4811E6BC.4060306@icyb.net.ua> <20080425143646.GF18958@deviant.kiev.zoral.com.ua> <48275C0C.2040601@icyb.net.ua> <20080511214833.GB18958@deviant.kiev.zoral.com.ua> <4829E658.3000605@icyb.net.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="KWJGpjkfDfw3nKYG" Content-Disposition: inline In-Reply-To: <4829E658.3000605@icyb.net.ua> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.4 X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on skuns.kiev.zoral.com.ua X-Scanner-Signature: eacc24c597fa83a26e5f02f00d1ff66c X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Header: Not Detected X-SpamTest-Info: Profiles 2809 [May 13 2008] X-SpamTest-Info: helo_type=3 X-SpamTest-Method: none X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0278], KAS30/Release Cc: freebsd-hackers@freebsd.org Subject: Re: devctl (alike?) for devfs X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 19:16:24 -0000 --KWJGpjkfDfw3nKYG Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 13, 2008 at 10:04:56PM +0300, Andriy Gapon wrote: > on 12/05/2008 00:48 Kostik Belousov said the following: > >No, we do not have a leak, but we have somewhat non-obvious behaviour. > > > >The cdev structure is freed only after the last reference to cdev is > >gone. Typical holder of the reference is the devfs vnode. In the normal > >usage, the vnode is present until both the device is destroyed _and_ > >devfs_populate_loop() run is performed. This function actually reclaim > >the vnodes for destroyed devices, that causes last reference to cdev to > >be dropped and memory freed. > > > >The populate loop is called syncronously from the upper levels. The > >easiest method to trigger it is to do ls /dev, since it is called from > >the devfs_lookupx(). >=20 > Thank you for the explanation! ls did trigger DESTROY notifications. > But this arbitrary delay between device entry going away and=20 > notification about about it is a little bit "uncool". >=20 > So I re-wrote the patch to post notifications about deleted devices with= =20 > si_refcount > 0 in a fashion similar to how si_refcount=3D0 devices are= =20 > freed. I put those cdev-s onto a special list (re-/ab-using si_siblings= =20 > LIST link field) and bump their si_refcount to prevent them from getting= =20 > destroyed before the notification is sent (it would need si_name). > Then, in dev_unlock_and_free() I send notifications and call dev_rel on= =20 > the devices. >=20 > I am not entirely satisfied with the code: > 1. I don't like the way I move LIST elements from one head to the other,= =20 > this should be a macro in queue.h > 2. I am not sure about the names I picked > 3. I slightly don't like a fact that parent-child destroy notifications= =20 > are sent in reverse order because of my simplistic LIST usage. I looked at your previous patch, and it seems it is much simpler to do drop the devmtx once more then to try to abuse free lists. In the destroy_devl(), after the=20 while (dev->si_threadcount !=3D 0) { /* Use unique dummy wait ident */ msleep(&csw, &devmtx, PRIBIO, "devdrn", hz / 10); } loop, add mtx_unlock(&devmtx); if (!cold) devctl_notify("DEVFS", dev->si_name, "DESTROY", NULL); mtx_lock(&devmtx); --KWJGpjkfDfw3nKYG Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkgp6PkACgkQC3+MBN1Mb4imDACg0StKn5T1Pwa1YM47p9jMpV86 eZsAoNV5x65YeTg7CQtFHRYWdU5Eb8XD =9Goh -----END PGP SIGNATURE----- --KWJGpjkfDfw3nKYG-- From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 19:26:13 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 538DC1065674 for ; Tue, 13 May 2008 19:26:13 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from hosted.kievnet.com (hosted.kievnet.com [193.138.144.10]) by mx1.freebsd.org (Postfix) with ESMTP id 0F4468FC14 for ; Tue, 13 May 2008 19:26:13 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost ([127.0.0.1] helo=edge.pp.kiev.ua) by hosted.kievnet.com with esmtpa (Exim 4.62) (envelope-from ) id 1Jw08a-0001Ri-5W for freebsd-hackers@freebsd.org; Tue, 13 May 2008 22:26:12 +0300 Message-ID: <4829EB53.3010804@icyb.net.ua> Date: Tue, 13 May 2008 22:26:11 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.12 (X11/20080320) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: /dev/mem wrap-around behavior X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 19:26:13 -0000 This is just out of curiosity, not a real issue. I use i386 RELENG_7. I recently played with dd-ing memory contents out of /dev/mem and examining various stuff. I "intuitively" expected dd to reach EOF at 0xFFFFFFFF (i.e. maximum possible address for i386 non-PAE), but it continued on. Apparently this happens because off_t is somewhere assigned to caddr_t (or something like that) without any extra checks. So, I am curious if this intentional, unintentional or it just happened to be this way and nobody really cares. -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 20:40:23 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9870C1065670; Tue, 13 May 2008 20:40:23 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from hosted.kievnet.com (hosted.kievnet.com [193.138.144.10]) by mx1.freebsd.org (Postfix) with ESMTP id 493E08FC26; Tue, 13 May 2008 20:40:23 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost ([127.0.0.1] helo=edge.pp.kiev.ua) by hosted.kievnet.com with esmtpa (Exim 4.62) (envelope-from ) id 1Jw1IM-0003Z7-4l; Tue, 13 May 2008 23:40:22 +0300 Message-ID: <4829FCB5.8030308@icyb.net.ua> Date: Tue, 13 May 2008 23:40:21 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.12 (X11/20080320) MIME-Version: 1.0 To: Kostik Belousov References: <480FBAB9.1000904@icyb.net.ua> <48103F36.6060707@icyb.net.ua> <200804240811.26183.jhb@freebsd.org> <4810FD1E.70602@icyb.net.ua> <20080425095009.GD18958@deviant.kiev.zoral.com.ua> <4811E6BC.4060306@icyb.net.ua> <20080425143646.GF18958@deviant.kiev.zoral.com.ua> <48275C0C.2040601@icyb.net.ua> <20080511214833.GB18958@deviant.kiev.zoral.com.ua> <4829E658.3000605@icyb.net.ua> <20080513191610.GK18958@deviant.kiev.zoral.com.ua> In-Reply-To: <20080513191610.GK18958@deviant.kiev.zoral.com.ua> Content-Type: multipart/mixed; boundary="------------020209090703020207030305" Cc: freebsd-hackers@freebsd.org Subject: Re: devctl (alike?) for devfs X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 20:40:23 -0000 This is a multi-part message in MIME format. --------------020209090703020207030305 Content-Type: text/plain; charset=KOI8-U; format=flowed Content-Transfer-Encoding: 7bit on 13/05/2008 22:16 Kostik Belousov said the following: > I looked at your previous patch, and it seems it is much simpler to > do drop the devmtx once more then to try to abuse free lists. > In the destroy_devl(), after the > > while (dev->si_threadcount != 0) { > /* Use unique dummy wait ident */ > msleep(&csw, &devmtx, PRIBIO, "devdrn", hz / 10); > } > > loop, add > > mtx_unlock(&devmtx); > if (!cold) > devctl_notify("DEVFS", dev->si_name, "DESTROY", NULL); > mtx_lock(&devmtx); Thank you again! This is simply perfect. -- Andriy Gapon --------------020209090703020207030305 Content-Type: text/plain; name="devfs-notify.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="devfs-notify.diff" diff --git a/sys/kern/kern_conf.c b/sys/kern/kern_conf.c index 1db25f8..f90e469 100644 --- a/sys/kern/kern_conf.c +++ b/sys/kern/kern_conf.c @@ -30,6 +30,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -706,6 +707,10 @@ make_dev_credv(int flags, struct cdevsw *devsw, int minornr, devfs_create(dev); clean_unrhdrl(devfs_inos); dev_unlock_and_free(); + + if (!cold) + devctl_notify("DEVFS", dev->si_name, "CREATE", NULL); + return (dev); } @@ -794,6 +799,10 @@ make_dev_alias(struct cdev *pdev, const char *fmt, ...) clean_unrhdrl(devfs_inos); dev_unlock(); dev_depends(pdev, dev); + + if (!cold) + devctl_notify("DEVFS", dev->si_name, "CREATE", NULL); + return (dev); } @@ -842,6 +851,12 @@ destroy_devl(struct cdev *dev) msleep(&csw, &devmtx, PRIBIO, "devdrn", hz / 10); } + if (!cold) { + mtx_unlock(&devmtx); + devctl_notify("DEVFS", dev->si_name, "DESTROY", NULL); + mtx_lock(&devmtx); + } + dev->si_drv1 = 0; dev->si_drv2 = 0; bzero(&dev->__si_u, sizeof(dev->__si_u)); --------------020209090703020207030305-- From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 20:48:35 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17FB91065670 for ; Tue, 13 May 2008 20:48:35 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (five.mired.org [66.92.153.75]) by mx1.freebsd.org (Postfix) with ESMTP id AFC0E8FC2B for ; Tue, 13 May 2008 20:48:34 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 18044 invoked by uid 1001); 13 May 2008 16:49:01 -0400 Received: from bhuda.mired.org (bhuda [192.168.195.1]) by bhuda (tmda-ofmipd) with ESMTP; Tue, 13 May 2008 16:49:00 -0400 Date: Tue, 13 May 2008 16:49:00 -0400 To: Anthony Pankov Message-ID: <20080513164900.35f89e47@bhuda.mired.org> In-Reply-To: <2117635718.20080513154406@mail.ru> References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> Organization: Meyer Consulting X-Mailer: Claws Mail 3.4.0 (GTK+ 2.12.9; amd64-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Delivery-Agent: TMDA/1.1.12 (Macallan) From: Mike Meyer Cc: freebsd-hackers@freebsd.org Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 20:48:35 -0000 On Tue, 13 May 2008 15:44:06 +0400 Anthony Pankov wrote: > > Monday, Mike Meyer May 12, 2008, 11:24:30 PM, you wrote: > > MM> On Mon, 12 May 2008 22:35:31 +0400 Anthony Pankov wrote: > >> Because BDB: > >> 1. do not need additional installation > >> 2. is part of base system which mean it is mature, reliable and stable > > MM> BDB in the base system is mature, reliable and stable *for what it's > MM> used for in the base system.* So long as your requirements are covered > MM> by that usage, you'll be ok. > > MM> The uses I know of for BDB in the base system all consist of databases > MM> of relatively small items that are changed infrequently, and usually > MM> with a locking mechanism. From what you've said, this doesn't describe > MM> your requirements. > > MM> More importantly, from what other people are saying, your requirements > MM> are ones for which it's known that BDB is *not* reliable, or otherwise > MM> unsuitable. In particular, an effort is underway to allow parallel > MM> ports builds, which implies concurrent access to the database, which > MM> is a known source of problems for BDB. > > MM> > My requirements is > 1. there is no need for SQL > 2. processes are sharing db file in concurrent mode > 3. reading/writing = 60%/40% > > With BDB > clause 1 - satisfied > clause 3 - satisfied (databases of relatively small items that are > changed infrequently). Actually, I don't think you get #3, because my wording was poor. The things in the base system are databases of small items, where the *database* is changed infrequently. I think what you have is a database of small items where the *items* are changed infrequently, but the database itself changes frequently. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 21:49:36 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACBC31065676 for ; Tue, 13 May 2008 21:49:36 +0000 (UTC) (envelope-from james@mansionfamily.plus.com) Received: from ptb-relay02.plus.net (ptb-relay02.plus.net [212.159.14.213]) by mx1.freebsd.org (Postfix) with ESMTP id 457668FC24 for ; Tue, 13 May 2008 21:49:35 +0000 (UTC) (envelope-from james@mansionfamily.plus.com) Received: from [80.229.150.39] (helo=mansionfamily.plus.com) by ptb-relay02.plus.net with esmtp (Exim) id 1Jw1cx-0006Cg-74 for freebsd-hackers@freebsd.org; Tue, 13 May 2008 22:01:39 +0100 Received: from [192.168.0.87] ([192.168.0.87]:2306) by mansionfamily.plus.com with [XMail 1.22 ESMTP Server] id for from ; Tue, 13 May 2008 22:07:35 +0100 Message-ID: <482A02CD.7040308@mansionfamily.plus.com> Date: Tue, 13 May 2008 22:06:21 +0100 From: James Mansion Organization: MsgWare User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Jeremy Chadwick , Anthony Pankov , freebsd-hackers@freebsd.org References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> <20080513121452.GA70860@eos.sc1.parodius.com> <20080513154137.GA28842@pix.net> In-Reply-To: <20080513154137.GA28842@pix.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Plusnet-Relay: e8a5e4f0217e2da3162a7b5e88a0c4cc Cc: Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 21:49:36 -0000 Kurt J. Lidl wrote: > This catapults back into the arena of "stuff that isn't in the > base system". Not to mention I'm not sure that the Oracle BDB > license would allow bundling in the OS as a binary. I doubt it, > but that's a different bikeshed to paint :-) > Is the LGPL of QDBM and TokyoCabinet also a problem? Could even try grovelling with Mikio? (Partially joking there. I assume he chose LGPL because he wants it that way, but people have been known to change licenses for a base system - like this http://blogs.sun.com/aalok/entry/lzma_on_opensolaris) And is the objection to SQL such the sqlite is really out of the running? Anyway, in this case, would writing an RPC server to own the data kill the performance? It should be easier to write something that can save the database atomically and index it in-core. It could be started on demand and shut down after a short inactivity, a bit like tibco's rvd. > There are known problems with certain keys corrupting the DB 1.8x > series code. In fact, the "release" of the 1.86 was an attempt > to solve this problem when the KerberosV people at MIT found > a repeatable key insert sequence that would corrupt things. > (Or at least that's what I remember, it was a long time ago, and > I might have the details wrong.) > > Have to say its a little concerning that such 'mature' code is actually problematic. Particularly since I'm not aware of a non-LGPL alternative. Do you have anything by way of a pointer? Google didn't help me here. James From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 23:47:50 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9011E106566C; Tue, 13 May 2008 23:47:50 +0000 (UTC) (envelope-from msaad@datapipe.com) Received: from exchfe01.datapipe-corp.net (exchfe01.datapipe-corp.net [64.106.130.69]) by mx1.freebsd.org (Postfix) with ESMTP id 59B588FC0A; Tue, 13 May 2008 23:47:50 +0000 (UTC) (envelope-from msaad@datapipe.com) Received: from divide.lan (192.168.128.20) by exchfe01.datapipe-corp.net (64.106.130.71) with Microsoft SMTP Server id 8.0.783.2; Tue, 13 May 2008 19:37:37 -0400 Message-ID: <482A2639.7000401@datapipe.com> Date: Tue, 13 May 2008 19:37:29 -0400 From: Mark Saad User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: "freebsd-stable@freebsd.org" , Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: : Re: Socket leak (Was: Re: What triggers "No BufferSpace), ?Available" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: msaad@datapipe.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 23:47:50 -0000 Hello All This issue goes back some time, but I do not see a solution. Sorry about the cross post not sure where this belongs. Here is an overview of my issue which is similar and I hope someone can point me in the direction of a solution. I have experiencing an odd socket related issue on a few servers i manage. They are fairly large ftp servers for popular north american news agency. They handle 1000's of ftp transactions per hour. Currently they are running FreeBSD 6.3-Release-p1 . I have verified this happened on FreeBSD 6.1-Release 6.1-Stable 6.2-Release 6.2-Stable and 7.0-Release all 32bit installs and in both SMP an UP kernels. Oddly this issue did not happen on FreeBSD 4.x . I have a similar setup that has a 1400+ Day uptime running FreeBSD 4.x-Release. The issue is after 7 to 14 days the servers lock up and will not create any new tcp sockets. The system used proftp with mysql for authentication of the ftp accounts. The system is also running Apache 2.2.x , Postfix, Cyrus, clam-av, Diablo JDK 1.5 for Resin Appserver and daemontools . The only sysctls that seem to help are kern.ipc.maxsockets and kern.maxusers . Currently they are set to 65535 and 1024 . Changing kern.ipc.maxsockbuf did not have any effect I tried bumping this up to 2Meg, In any case I started work on logging everything we could think of to see what was happening. I started logging the values of kern.ipc.numopensockets and I noticed that something is leaking sockets. Here is a sample of the log 2008-04-29--15:04.10 ____ kern.ipc.numopensockets: 1501 2008-04-29--16:04.01 ____ kern.ipc.numopensockets: 1535 2008-04-29--17:04.00 ____ kern.ipc.numopensockets: 1617 2008-04-29--18:04.00 ____ kern.ipc.numopensockets: 1710 This continues until kern.ipc.maxsockets its reached or the box is rebooted. The other thing we looked at was the output from vmstat -z The first thing was the high amount of malloc 128 bucket failures 128 Bucket: 524, 0, 2489, 80, 8364, 23055239 I also logged the mbuf clusters, we never reached the max mbuf clusters Its almost like there are stale sockets. Here is a snapshot of the server now ewr# sockstat -4u |wc -l 139 ewr# sysctl kern.ipc.numopensockets kern.ipc.numopensockets: 13935 ewr# uptime 7:30PM up 6 days, 26 mins, 3 users, load averages: 0.18, 0.25, 0.17 My questions. 1. If I can not identify who / what is consuming all my tcp sockets what will happen if I double or triple the value of kern.ipc.maxsockets ? 2. Could this be an issue with a low kern.maxusers . Its currently set to 1024 . Also at times when I can not create a new socket I am not pinned on mbuf clusters . I was able to verify this in the past. 3. I installed a debugging kernel, which I built on the server. I was able to get a coredump of the server at the point in time we last had an issue. But I am not sure what I can do with this, kernel debugging is way beyond what I am capable of doing . Do I want to even pursue this ? 4. Does anyone have any system tunings you could recommend for a high volume ftp site ? What does ftp.freebsd.org have ? -- Mark Saad msaad@datapipe.com From owner-freebsd-hackers@FreeBSD.ORG Tue May 13 23:48:26 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 077AF1065674; Tue, 13 May 2008 23:48:26 +0000 (UTC) (envelope-from msaad@datapipe.com) Received: from exchfe01.datapipe-corp.net (exchfe01.datapipe-corp.net [64.106.130.69]) by mx1.freebsd.org (Postfix) with ESMTP id C50EC8FC16; Tue, 13 May 2008 23:48:25 +0000 (UTC) (envelope-from msaad@datapipe.com) Received: from divide.lan (192.168.128.20) by exchfe01.datapipe-corp.net (64.106.130.71) with Microsoft SMTP Server id 8.0.783.2; Tue, 13 May 2008 19:38:16 -0400 Message-ID: <482A2668.8070209@datapipe.com> Date: Tue, 13 May 2008 19:38:16 -0400 From: Mark Saad User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: "freebsd-stable@freebsd.org" , Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: : Re: Socket leak (Was: Re: What triggers "No BufferSpace), ?Available" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: msaad@datapipe.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2008 23:48:26 -0000 Hello All This issue goes back some time, but I do not see a solution. Sorry about the cross post not sure where this belongs. Here is an overview of my issue which is similar and I hope someone can point me in the direction of a solution. I have experiencing an odd socket related issue on a few servers i manage. They are fairly large ftp servers for popular north american news agency. They handle 1000's of ftp transactions per hour. Currently they are running FreeBSD 6.3-Release-p1 . I have verified this happened on FreeBSD 6.1-Release 6.1-Stable 6.2-Release 6.2-Stable and 7.0-Release all 32bit installs and in both SMP an UP kernels. Oddly this issue did not happen on FreeBSD 4.x . I have a similar setup that has a 1400+ Day uptime running FreeBSD 4.x-Release. The issue is after 7 to 14 days the servers lock up and will not create any new tcp sockets. The system used proftp with mysql for authentication of the ftp accounts. The system is also running Apache 2.2.x , Postfix, Cyrus, clam-av, Diablo JDK 1.5 for Resin Appserver and daemontools . The only sysctls that seem to help are kern.ipc.maxsockets and kern.maxusers . Currently they are set to 65535 and 1024 . Changing kern.ipc.maxsockbuf did not have any effect I tried bumping this up to 2Meg, In any case I started work on logging everything we could think of to see what was happening. I started logging the values of kern.ipc.numopensockets and I noticed that something is leaking sockets. Here is a sample of the log 2008-04-29--15:04.10 ____ kern.ipc.numopensockets: 1501 2008-04-29--16:04.01 ____ kern.ipc.numopensockets: 1535 2008-04-29--17:04.00 ____ kern.ipc.numopensockets: 1617 2008-04-29--18:04.00 ____ kern.ipc.numopensockets: 1710 This continues until kern.ipc.maxsockets its reached or the box is rebooted. The other thing we looked at was the output from vmstat -z The first thing was the high amount of malloc 128 bucket failures 128 Bucket: 524, 0, 2489, 80, 8364, 23055239 I also logged the mbuf clusters, we never reached the max mbuf clusters Its almost like there are stale sockets. Here is a snapshot of the server now ewr# sockstat -4u |wc -l 139 ewr# sysctl kern.ipc.numopensockets kern.ipc.numopensockets: 13935 ewr# uptime 7:30PM up 6 days, 26 mins, 3 users, load averages: 0.18, 0.25, 0.17 My questions. 1. If I can not identify who / what is consuming all my tcp sockets what will happen if I double or triple the value of kern.ipc.maxsockets ? 2. Could this be an issue with a low kern.maxusers . Its currently set to 1024 . Also at times when I can not create a new socket I am not pinned on mbuf clusters . I was able to verify this in the past. 3. I installed a debugging kernel, which I built on the server. I was able to get a coredump of the server at the point in time we last had an issue. But I am not sure what I can do with this, kernel debugging is way beyond what I am capable of doing . Do I want to even pursue this ? 4. Does anyone have any system tunings you could recommend for a high volume ftp site ? What does ftp.freebsd.org have ? -- Mark Saad msaad@datapipe.com From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 04:54:31 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B7FB106566B for ; Wed, 14 May 2008 04:54:31 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from smtp6.yandex.ru (smtp6.yandex.ru [213.180.200.197]) by mx1.freebsd.org (Postfix) with ESMTP id 9AF3A8FC1B for ; Wed, 14 May 2008 04:54:30 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from ns.kirov.so-cdu.ru ([77.72.136.145]:35308 "EHLO [127.0.0.1]" smtp-auth: "bu7cher" TLS-CIPHER: "DHE-RSA-AES256-SHA keybits 256/256 version TLSv1/SSLv3" TLS-PEER-CN1: ) by mail.yandex.ru with ESMTP id S5473784AbYENEPd (ORCPT ); Wed, 14 May 2008 08:15:33 +0400 X-Yandex-Spam: 1 X-Yandex-Front: smtp6 X-Yandex-TimeMark: 1210738533 X-MsgDayCount: 4 X-Comment: RFC 2476 MSA function at smtp6.yandex.ru logged sender identity as: bu7cher Message-ID: <482A6760.5020704@yandex.ru> Date: Wed, 14 May 2008 08:15:28 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla Thunderbird 1.5 (FreeBSD/20051231) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org References: <20080513161351.GA10097@marvin.blogreen.org> In-Reply-To: <20080513161351.GA10097@marvin.blogreen.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: Ed Maste , scottl@freebsd.org, romain@blogreen.org Subject: Re: Adaptec RAID 3805 and FreeBSD 7.0 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 04:54:31 -0000 Romain Tartière wrote: > I am trying to have an Adaptec RAID 3805 controller working on FreeBSD > 7.0. According to the release note [1], it's okay. Unfortunately, the > driver fails to detect everything as expected. The following is > displayed at boot-time: >> aac0: mem 0xff200000-0xff3fffff irq 16 at device 14.0 on pci3 >> aac0: couldn't allocate register window 0 >> device_attach: aac0 attach returned 6 > The result was not much better: >> aacu0: mem 0xff200000-0xff3fffff irq 16 at device 14.0 on pci3 >> aacu0: couldn't allocate register window 0 >> device_attach: aacu0 attach returned 12 > Has anybody tried / succeeded in using such a controller on FreeBSD > 7.0-CURRENT? Has somebody this controller working on FreeBSD 6.3 (I can > consider to downgrade to the legacy release if required since the > machine is not yet in production)? Did you try last 8.0-CURRENT snapshot? I remember that Ed made many changes in aac(4) driver in CURRENT.. If it will not work then fill problem report. -- WBR, Andrey V. Elsukov From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 06:51:11 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2D7FD106564A for ; Wed, 14 May 2008 06:51:11 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from mx0.awanti.com (mx0.awanti.com [91.190.112.18]) by mx1.freebsd.org (Postfix) with ESMTP id E0D3F8FC1A for ; Wed, 14 May 2008 06:51:10 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from pstation (unknown [10.28.4.14]) by mx0.awanti.com (Postfix) with ESMTP id 460714C030 for ; Wed, 14 May 2008 10:51:09 +0400 (MSD) Date: Wed, 14 May 2008 10:53:35 +0400 From: Anthony Pankov X-Mailer: The Bat! (v1.51) Personal X-Priority: 3 (Normal) Message-ID: <394875078.20080514105335@mail.ru> To: freebsd-hackers@freebsd.org In-Reply-To: <482A02CD.7040308@mansionfamily.plus.com> References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> <20080513121452.GA70860@eos.sc1.parodius.com> <20080513154137.GA28842@pix.net> <482A02CD.7040308@mansionfamily.plus.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Re[3]: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Anthony Pankov List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 06:51:11 -0000 Thanks everyone for responding. I'll use QDBM as most attractive from my point of view. Oracle BDB is to complex for my task and have drastic "free" license with unknown price for commercial use. Licensing is not a main issue for me now, but i'll beware it to be on the safe side. I think that FreeBSD (ideally) must have a few PRIMARY datastore tools which together cover most tasks and able for horizontal and vertical scaling: 1. BDB1 = QDBM 2. CDB 3. SQLite 4. Firebird 5. PostgreSQL I'm absolutely sure that tradition of programmers to write own datastore, memory list, etc has to be dropped. I think that some warning like 'may corrupt data' has to be committed to 'man db'. -- Best regards, Anthony mailto:ap00@mail.ru From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 07:17:34 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2502E1065674 for ; Wed, 14 May 2008 07:17:34 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail02.syd.optusnet.com.au (mail02.syd.optusnet.com.au [211.29.132.183]) by mx1.freebsd.org (Postfix) with ESMTP id A5DC08FC26 for ; Wed, 14 May 2008 07:17:32 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from server.vk2pj.dyndns.org (c122-106-215-175.belrs3.nsw.optusnet.com.au [122.106.215.175]) by mail02.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m4E7HT2L028378 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 14 May 2008 17:17:30 +1000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.2/8.14.2) with ESMTP id m4E7HT8D064782; Wed, 14 May 2008 17:17:29 +1000 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.2/8.14.2/Submit) id m4E7HS3X064781; Wed, 14 May 2008 17:17:28 +1000 (EST) (envelope-from peter) Date: Wed, 14 May 2008 17:17:28 +1000 From: Peter Jeremy To: James Mansion Message-ID: <20080514071728.GP64804@server.vk2pj.dyndns.org> References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> <20080513121452.GA70860@eos.sc1.parodius.com> <20080513154137.GA28842@pix.net> <482A02CD.7040308@mansionfamily.plus.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="8X7/QrJGcKSMr1RN" Content-Disposition: inline In-Reply-To: <482A02CD.7040308@mansionfamily.plus.com> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.17 (2007-11-01) Cc: freebsd-hackers@freebsd.org Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 07:17:34 -0000 --8X7/QrJGcKSMr1RN Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2008-May-13 22:06:21 +0100, James Mansion = wrote: >And is the objection to SQL such the sqlite is really out of the running? There is no specific objection to SQL. There is a general objection to adding more utilities to the base system unless a _very_ good case can be made for including them. So far, no-one has made a compelling reason to include SQLite (or other SQL engine) into the base system. SQLite does appear to be a potential candidate for the base system - the license is acceptable and it is relatively small. On the downside, SQLite is currently undergoing rapid development - at a rate much faster than FreeBSD can comfortably accomodate. >Anyway, in this case, would writing an RPC server to own the data kill the= =20 >performance? That depends on how everything is implemented. It doesn't really solve the underlying problem that a database is needed. --=20 Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. --8X7/QrJGcKSMr1RN Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkgqkggACgkQ/opHv/APuIeybwCgtHCVkEiyvRXp67nAKCi7OcJr YmIAnj/qBo7byXA1cbDstj5LuScAitiM =KmtC -----END PGP SIGNATURE----- --8X7/QrJGcKSMr1RN-- From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 07:24:11 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 013E81065676 for ; Wed, 14 May 2008 07:24:11 +0000 (UTC) (envelope-from yanefbsd@gmail.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.169]) by mx1.freebsd.org (Postfix) with ESMTP id C1CE68FC0C for ; Wed, 14 May 2008 07:24:10 +0000 (UTC) (envelope-from yanefbsd@gmail.com) Received: by wf-out-1314.google.com with SMTP id 28so2978093wfa.7 for ; Wed, 14 May 2008 00:24:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:cc:message-id:from:to:in-reply-to:content-type:content-transfer-encoding:mime-version:subject:date:references:x-mailer; bh=HUqk5bKcJGAYnsGUNEN0wJ5mFV2ODexX+pAoAfJUTGg=; b=xc1Y1OZhz4pCEPPK9yuTCwmGRbT2tfiqWAUEeGrH8Iy8S6Dybp/4B/Xzbvye9CY5WDcc13bzPELzEyf+rzRmHVGl9RDRu+VXY0XurY0cZ8zJfz7yZdVzpbn9gIkJpTymXm8N2dMXrh2UIkZksMhwBAJpPre33k3tGbdswsLNAfI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=cc:message-id:from:to:in-reply-to:content-type:content-transfer-encoding:mime-version:subject:date:references:x-mailer; b=xgdAVomDyttOTCpSz9tO/yTOFJVpqW/RVKU0Bitv10nzu7z23ykzkGQbBm9+BG4kBk3Wi81lfeKKYx0GFjIj8wXL8OnBxYFXT4OVD61CoYMG5YgOuKfkMcZbP8kDOji2hPv1sDHXEdcZykvWsD7N1bfYBNi1lHBYC5OTPOok5R4= Received: by 10.142.99.21 with SMTP id w21mr291990wfb.55.1210749850491; Wed, 14 May 2008 00:24:10 -0700 (PDT) Received: from ?192.168.10.42? ( [99.155.199.64]) by mx.google.com with ESMTPS id 30sm1773902wfc.5.2008.05.14.00.24.08 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 14 May 2008 00:24:09 -0700 (PDT) Message-Id: From: Garrett Cooper To: James Mansion In-Reply-To: <482A02CD.7040308@mansionfamily.plus.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v919.2) Date: Wed, 14 May 2008 00:25:16 -0700 References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> <20080513121452.GA70860@eos.sc1.parodius.com> <20080513154137.GA28842@pix.net> <482A02CD.7040308@mansionfamily.plus.com> X-Mailer: Apple Mail (2.919.2) Cc: freebsd-hackers@freebsd.org, Jeremy Chadwick , Anthony Pankov Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 07:24:11 -0000 On May 13, 2008, at 2:06 PM, James Mansion wrote: > Kurt J. Lidl wrote: >> This catapults back into the arena of "stuff that isn't in the >> base system". Not to mention I'm not sure that the Oracle BDB >> license would allow bundling in the OS as a binary. I doubt it, >> but that's a different bikeshed to paint :-) >> > Is the LGPL of QDBM and TokyoCabinet also a problem? Could even try > grovelling > with Mikio? (Partially joking there. I assume he chose LGPL because > he wants > it that way, but people have been known to change licenses for a > base system - like > this http://blogs.sun.com/aalok/entry/lzma_on_opensolaris) > > And is the objection to SQL such the sqlite is really out of the > running? > > Anyway, in this case, would writing an RPC server to own the data > kill the performance? > It should be easier to write something that can save the database > atomically and index > it in-core. It could be started on demand and shut down after a > short inactivity, a bit > like tibco's rvd. >> There are known problems with certain keys corrupting the DB 1.8x >> series code. In fact, the "release" of the 1.86 was an attempt >> to solve this problem when the KerberosV people at MIT found >> a repeatable key insert sequence that would corrupt things. >> (Or at least that's what I remember, it was a long time ago, and >> I might have the details wrong.) >> >> > Have to say its a little concerning that such 'mature' code is > actually problematic. > Particularly since I'm not aware of a non-LGPL alternative. > > Do you have anything by way of a pointer? Google didn't help me here. > > James Most of the complaints about other DBs is licensing related, but SQLite's complaint was also the fact that the past stability record was a bit rocky. HTH, -Garrett From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 08:44:53 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE7F9106566C; Wed, 14 May 2008 08:44:53 +0000 (UTC) (envelope-from doconnor@gsoft.com.au) Received: from cain.gsoft.com.au (cain.gsoft.com.au [203.31.81.10]) by mx1.freebsd.org (Postfix) with ESMTP id 35F6B8FC1A; Wed, 14 May 2008 08:44:52 +0000 (UTC) (envelope-from doconnor@gsoft.com.au) Received: from inchoate.gsoft.com.au (ppp121-45-62-238.lns11.adl2.internode.on.net [121.45.62.238]) (authenticated bits=0) by cain.gsoft.com.au (8.13.8/8.13.8) with ESMTP id m4E8PMMu079271 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Wed, 14 May 2008 17:55:23 +0930 (CST) (envelope-from doconnor@gsoft.com.au) From: "Daniel O'Connor" To: freebsd-hackers@freebsd.org Date: Wed, 14 May 2008 17:55:33 +0930 User-Agent: KMail/1.9.7 References: <20080512204558.GA44114@flint.openpave.org> <48291719.5010801@freebsd.org> In-Reply-To: <48291719.5010801@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart3648637.0ilReWUd7Z"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200805141755.35892.doconnor@gsoft.com.au> X-Spam-Score: -2.212 () BAYES_00,RDNS_DYNAMIC X-Scanned-By: MIMEDefang 2.63 on 203.31.81.10 Cc: Jeremy Lea , Tim Kientzle , Anders Nore Subject: Re: Adding .db support to pkg_tools X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 08:44:54 -0000 --nextPart3648637.0ilReWUd7Z Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Tue, 13 May 2008, Tim Kientzle wrote: > > I think this is a really bad idea. The problem with the tools is > > not with the files. It is that the files need to be parsed on each > > run, often recursively, and your solution would not help at all. > > Parsing one file isn't expensive; parsing several hundred files > to find one bit of information is expensive. > > > The database(s) should just be a cache of the information stored in > > the files. > > Bingo! As long as the .db version can be easily recreated > from scratch from the master data stored in the same files > as always, it doesn't really matter if the BDB is occasionally > corrupted, as long as it can be rebuilt fairly quickly. So long as you can tell it is corrupted.. It's also a drag from a user POV when the tool crashes because the DB is=20 hosed (seen in portupgrade a number of times) =2D-=20 Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C --nextPart3648637.0ilReWUd7Z Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.8 (FreeBSD) iD8DBQBIKqH/5ZPcIHs/zowRAicSAJ0bihPNmz3tKAJ7SYKOFoR9l6XQ6gCggK3/ DXBGeT+Dmxen4Gi/bHriK8Y= =QpYT -----END PGP SIGNATURE----- --nextPart3648637.0ilReWUd7Z-- From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 10:17:31 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10D1F106566B; Wed, 14 May 2008 10:17:31 +0000 (UTC) (envelope-from romain@blogreen.org) Received: from smtp3-g19.free.fr (smtp3-g19.free.fr [212.27.42.29]) by mx1.freebsd.org (Postfix) with ESMTP id C0BBD8FC22; Wed, 14 May 2008 10:17:30 +0000 (UTC) (envelope-from romain@blogreen.org) Received: from smtp3-g19.free.fr (localhost.localdomain [127.0.0.1]) by smtp3-g19.free.fr (Postfix) with ESMTP id DE7EE17B5A2; Wed, 14 May 2008 12:17:29 +0200 (CEST) Received: from marvin.blogreen.org (marvin.blogreen.org [82.247.213.140]) by smtp3-g19.free.fr (Postfix) with ESMTP id C23AC17B59A; Wed, 14 May 2008 12:17:29 +0200 (CEST) Received: by marvin.blogreen.org (Postfix, from userid 1001) id 9F8085C070; Wed, 14 May 2008 12:17:29 +0200 (CEST) Date: Wed, 14 May 2008 12:17:29 +0200 From: Romain =?iso-8859-1?Q?Tarti=E8re?= To: "Andrey V. Elsukov" Message-ID: <20080514101729.GB77349@marvin.blogreen.org> Mail-Followup-To: "Andrey V. Elsukov" , freebsd-hackers@freebsd.org, Ed Maste , scottl@freebsd.org, romain@blogreen.org References: <20080513161351.GA10097@marvin.blogreen.org> <482A6760.5020704@yandex.ru> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="U+BazGySraz5kW0T" Content-Disposition: inline In-Reply-To: <482A6760.5020704@yandex.ru> User-Agent: Mutt/1.4.2.3i X-PGP-Key: http://romain.blogreen.org/pubkey.asc Cc: freebsd-hackers@freebsd.org, scottl@freebsd.org, Ed Maste , romain@blogreen.org Subject: Re: Adaptec RAID 3805 and FreeBSD 7.0 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 10:17:31 -0000 --U+BazGySraz5kW0T Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, May 14, 2008 at 08:15:28AM +0400, Andrey V. Elsukov wrote: > Romain Tarti=E8re wrote: > > I am trying to have an Adaptec RAID 3805 controller working on > > FreeBSD 7.0. > Did you try last 8.0-CURRENT snapshot? Just tried with April snapshot. Same result as with 7.0-STABLE :-( . > I remember that Ed made many changes in aac(4) driver in CURRENT.. > If it will not work then fill problem report. PR kern/123666 sent. http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/123666 Regards, Romain --=20 Romain Tarti=E8re http://romain.blogreen.org/ pgp: 8DAB A124 0DA4 7024 F82A E748 D8E9 A33F FF56 FF43 (ID: 0xFF56FF43) (plain text =3Dnon-HTML=3D PGP/GPG encrypted/signed e-mail much appreciated) --U+BazGySraz5kW0T Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkgqvDkACgkQ2OmjP/9W/0NpKQCdEUF7nJQlluI4iyAFV0rMd5RD bN0An0L0doPERp3w2m10vdGd0BvDvUz9 =iXkP -----END PGP SIGNATURE----- --U+BazGySraz5kW0T-- From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 10:42:26 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1048E1065677 for ; Wed, 14 May 2008 10:42:26 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mu-out-0910.google.com (mu-out-0910.google.com [209.85.134.188]) by mx1.freebsd.org (Postfix) with ESMTP id 7EE078FC1E for ; Wed, 14 May 2008 10:42:25 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: by mu-out-0910.google.com with SMTP id i2so1996077mue.3 for ; Wed, 14 May 2008 03:42:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:to:subject:references:organization:from:date:in-reply-to:message-id:user-agent:mime-version:content-type; bh=lJxYYCOA8ozKraiiwEKnN1ZufYFpQsSdaObKtMxPa0o=; b=nDwXprmTbZps/aUwYL6+KMB/NAbWsB4wz1aIY4D1zvMSbTlkovviAdGx8NJuOSalFnjpzeANSe431W7/elmLK3EHyEYxv5eizhEqYyqX7iPOn5kDZxNZQ9N46RhKywThOwe/B5UE1MojvAx6g1Uhcf938AYGt4jCnCzlJUmJb0E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=to:subject:references:organization:from:date:in-reply-to:message-id:user-agent:mime-version:content-type; b=Gef2iQmf4e39oyCFFx59d9J7PMgpWAb0SKo9qR1LlzEjZg1VmWWby2U0LcswlC+qVqivGNobRStVmVU+IK725bBFOB4ScZs96gqthh7567dptd2zG1PYa7gfOYYp8EAJuQzo2ugDm/765GtQ/cUB//12SpKepNgoVMcw7vt/w5s= Received: by 10.125.144.13 with SMTP id w13mr195405mkn.12.1210760083391; Wed, 14 May 2008 03:14:43 -0700 (PDT) Received: from localhost ( [80.85.90.67]) by mx.google.com with ESMTPS id e20sm1017961fga.7.2008.05.14.03.14.42 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 14 May 2008 03:14:42 -0700 (PDT) To: References: <482A2639.7000401@datapipe.com> Organization: TOA Ukraine From: Mikolaj Golub Date: Wed, 14 May 2008 13:14:41 +0300 In-Reply-To: <482A2639.7000401@datapipe.com> (Mark Saad's message of "Tue\, 13 May 2008 19\:37\:29 -0400") Message-ID: <81zlqtfazy.fsf@zhuzha.ua1> User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: Socket leak X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 10:42:26 -0000 On Tue, 13 May 2008 19:37:29 -0400 Mark Saad wrote: MS> I started logging the values of kern.ipc.numopensockets and I noticed MS> that something is leaking sockets. Here is a sample of the log MS> 2008-04-29--15:04.10 ____ kern.ipc.numopensockets: 1501 MS> 2008-04-29--16:04.01 ____ kern.ipc.numopensockets: 1535 MS> 2008-04-29--17:04.00 ____ kern.ipc.numopensockets: 1617 MS> 2008-04-29--18:04.00 ____ kern.ipc.numopensockets: 1710 MS> This continues until kern.ipc.maxsockets its reached or the box is MS> rebooted. MS> The other thing we looked at was the output from vmstat -z MS> The first thing was the high amount of malloc 128 bucket failures MS> 128 Bucket: 524, 0, 2489, 80, 8364, 23055239 MS> I also logged the mbuf clusters, we never reached the max mbuf clusters MS> Its almost like there are stale sockets. Here is a snapshot of the server now MS> ewr# sockstat -4u |wc -l MS> 139 MS> ewr# sysctl kern.ipc.numopensockets MS> kern.ipc.numopensockets: 13935 MS> ewr# uptime MS> 7:30PM up 6 days, 26 mins, 3 users, load averages: 0.18, 0.25, 0.17 We had the same problem on one of hosts running 6.2-RELEASE-p11. The situation was complicated by the fact that I didn't have root access to the host and there were problems with getting more debugging or running tcpdump. Eventually, it appeared that problem was caused by proftpd. One of our clients connected to ftp server every five minutes looking for new file to download. When there was the file everything was good. But when there wasn't, some strange things happened. In proftpd logs we had: FTP session opened. mod_delay/0.5: delaying for 28 usecs user fake authenticated by mod_auth_pam.c USER fake: Login successful. Preparing to chroot to directory '/var/ftp/fake' Environment successfully chroot()ed. mod_delay/0.5: delaying for 621 usecs Entering Passive Mode (XX,YY,ZZ,213,241,70). FTP session closed. i.e. the client connected to server, had login successful, created new DATA connection in passive mode and then exited. But although proftpd reported that connection closed and proftpd process exited we still had this orphaned connection in our system reported by netstat in ESTABLISHED state. sockstat did not display this connections. Some of these connections could be in CLOSE_WAIT mode instead of ESTABLISHED. Such connection was seen by netstat for several hours and then disappeared but I suspect that the socket buffer was not freed and numopensockets counter did not decrease. Unfortunately, I did not managed to persuade admin to increase DebugLevel in proftpd.conf and run tcpdump to investigate more what was going on. It turned out that we had proftpd built for FREEBSD5_4: Compile-time Settings: Version: 1.3.0 Platform: FREEBSD5 (FREEBSD5_4) Built With: configure --localstatedir=/var/run --sysconfdir=/usr/local/etc --disable-sendfile --disable-ipv6 --with-modules=mod_ratio:mod_readme:mod_rewrite:mod_wrap:mod_ifsession --prefix=/usr/local i386-portbld-freebsd5.4 Upgrade to more recent proftpd built for proper platform resolved the problem. So I would recommend to look for process that could cause this leak. In my case careful investigation of netstat output history and comparing with sockstat output helped to find guilty. May be it would help to restart daemons one by one and see if sockets are freed. You can surely increase kern.ipc.maxsockets as workaround until you identify what causes the problem. -- Mikolaj Golub From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 10:59:48 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2257D106564A for ; Wed, 14 May 2008 10:59:48 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id CC9688FC17 for ; Wed, 14 May 2008 10:59:47 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 78F441B10F42; Wed, 14 May 2008 12:45:44 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, J_CHICKENPOX_34 autolearn=no version=3.2.4 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id CC5401B10EF9 for ; Wed, 14 May 2008 12:45:41 +0200 (CEST) Message-ID: <482AC2D5.1010800@moneybookers.com> Date: Wed, 14 May 2008 13:45:41 +0300 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.14 (X11/20080503) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on blah.cmotd.com X-Virus-Status: Clean Subject: 53 ECONNABORTED Software caused connection abort. A connection abort was caused internal to your host machine. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 10:59:48 -0000 Greetings, I'm running web balancer pound-2.3.2 on FreeBSD 6.x and I see strange error message: HTTP accept: Software caused connection abort. The code that triggers this is: for(lstn = listeners, i = 0; lstn; lstn = lstn->next, i++) { if(polls[i].revents & (POLLIN | POLLPRI)) { memset(&clnt_addr, 0, sizeof(clnt_addr)); clnt_length = sizeof(clnt_addr); if((clnt = accept(lstn->sock, (struct sockaddr *)&clnt_addr, (socklen_t *)&clnt_length)) < 0) { logmsg(LOG_WARNING, "HTTP accept: %s", strerror(errno)); ... Any ideas where should I look to understand better the error message and why errno 53 was returned? -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 12:19:36 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 545D41065670; Wed, 14 May 2008 12:19:36 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay01.kiev.sovam.com (relay01.kiev.sovam.com [62.64.120.200]) by mx1.freebsd.org (Postfix) with ESMTP id D9AC48FC0A; Wed, 14 May 2008 12:19:35 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=skuns.kiev.zoral.com.ua) by relay01.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1JwFxD-0004ES-NO; Wed, 14 May 2008 15:19:34 +0300 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by skuns.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4ECJOrR025014 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 14 May 2008 15:19:24 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4ECJMN6022032; Wed, 14 May 2008 15:19:22 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m4ECJKYs021688; Wed, 14 May 2008 15:19:20 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 14 May 2008 15:19:20 +0300 From: Kostik Belousov To: Andriy Gapon Message-ID: <20080514121920.GR18958@deviant.kiev.zoral.com.ua> References: <200804240811.26183.jhb@freebsd.org> <4810FD1E.70602@icyb.net.ua> <20080425095009.GD18958@deviant.kiev.zoral.com.ua> <4811E6BC.4060306@icyb.net.ua> <20080425143646.GF18958@deviant.kiev.zoral.com.ua> <48275C0C.2040601@icyb.net.ua> <20080511214833.GB18958@deviant.kiev.zoral.com.ua> <4829E658.3000605@icyb.net.ua> <20080513191610.GK18958@deviant.kiev.zoral.com.ua> <4829FCB5.8030308@icyb.net.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="XESQGuwklioZw9MS" Content-Disposition: inline In-Reply-To: <4829FCB5.8030308@icyb.net.ua> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.4 X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on skuns.kiev.zoral.com.ua X-Scanner-Signature: ba8373d0121182535360aa5673094e5a X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Header: Not Detected X-SpamTest-Info: Profiles 2815 [May 13 2008] X-SpamTest-Info: helo_type=3 X-SpamTest-Method: none X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0278], KAS30/Release Cc: freebsd-hackers@freebsd.org Subject: Re: devctl (alike?) for devfs X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 12:19:36 -0000 --XESQGuwklioZw9MS Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 13, 2008 at 11:40:21PM +0300, Andriy Gapon wrote: > on 13/05/2008 22:16 Kostik Belousov said the following: > >I looked at your previous patch, and it seems it is much simpler to > >do drop the devmtx once more then to try to abuse free lists. > >In the destroy_devl(), after the=20 > > > > while (dev->si_threadcount !=3D 0) { > > /* Use unique dummy wait ident */ > > msleep(&csw, &devmtx, PRIBIO, "devdrn", hz / 10); > > } > > > >loop, add > > > > mtx_unlock(&devmtx); > > if (!cold) > > devctl_notify("DEVFS", dev->si_name, "DESTROY", NULL); > > mtx_lock(&devmtx); >=20 > Thank you again! This is simply perfect. >=20 >=20 > --=20 > Andriy Gapon > diff --git a/sys/kern/kern_conf.c b/sys/kern/kern_conf.c > index 1db25f8..f90e469 100644 I do not like abusing the subsystem to communicate the data. What about this instead ? The created/destroyed cdev name is available in the $cdev variable, example clause for devd.conf looks like this: notify 1000 { system "DEVFS"; subsystem "CDEV"; action "echo $type $cdev >/dev/console"; }; diff --git a/sys/kern/kern_conf.c b/sys/kern/kern_conf.c index e911913..a5ed25c 100644 --- a/sys/kern/kern_conf.c +++ b/sys/kern/kern_conf.c @@ -30,6 +30,7 @@ __FBSDID("$FreeBSD: src/sys/kern/kern_conf.c,v 1.211 2008= /04/02 11:11:58 kib Exp #include #include #include +#include #include #include #include @@ -526,6 +527,37 @@ unit2minor(int unit) return ((unit & 0xff) | ((unit << 8) & ~0xffff)); } =20 +static void +notify(struct cdev *dev, const char *ev) +{ + static const char prefix[] =3D "cdev=3D"; + char *data; + int namelen; + + if (cold) + return; + namelen =3D strlen(dev->si_name); + data =3D malloc(namelen + sizeof(prefix), M_TEMP, M_WAITOK); + memcpy(data, prefix, sizeof(prefix) - 1); + memcpy(data + sizeof(prefix) - 1, dev->si_name, namelen + 1); + devctl_notify("DEVFS", "CDEV", ev, data); + free(data, M_TEMP); +} + +static void +notify_create(struct cdev *dev) +{ + + notify(dev, "CREATE"); +} + +static void +notify_destroy(struct cdev *dev) +{ + + notify(dev, "DESTROY"); +} + static struct cdev * newdev(struct cdevsw *csw, int y, struct cdev *si) { @@ -706,6 +738,9 @@ make_dev_credv(int flags, struct cdevsw *devsw, int min= ornr, devfs_create(dev); clean_unrhdrl(devfs_inos); dev_unlock_and_free(); + + notify_create(dev); + return (dev); } =20 @@ -794,6 +829,9 @@ make_dev_alias(struct cdev *pdev, const char *fmt, ...) clean_unrhdrl(devfs_inos); dev_unlock(); dev_depends(pdev, dev); + + notify_create(dev); + return (dev); } =20 @@ -842,6 +880,10 @@ destroy_devl(struct cdev *dev) msleep(&csw, &devmtx, PRIBIO, "devdrn", hz / 10); } =20 + mtx_unlock(&devmtx); + notify_destroy(dev); + mtx_lock(&devmtx); + dev->si_drv1 =3D 0; dev->si_drv2 =3D 0; bzero(&dev->__si_u, sizeof(dev->__si_u)); --XESQGuwklioZw9MS Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkgq2McACgkQC3+MBN1Mb4jx6gCgvO+ewO5SxOPPRpObZaAMUnM6 xF8An0AMuUWcaK1acNS353GEqIv2KpMc =oGoZ -----END PGP SIGNATURE----- --XESQGuwklioZw9MS-- From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 13:19:10 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 88645106564A; Wed, 14 May 2008 13:19:10 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id EE90D8FC14; Wed, 14 May 2008 13:19:09 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id 6D4D243F80C; Wed, 14 May 2008 16:19:07 +0300 (EEST) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id un0XGVnckL8X; Wed, 14 May 2008 16:19:07 +0300 (EEST) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 8874143E725; Wed, 14 May 2008 16:19:06 +0300 (EEST) Message-ID: <482AE6C9.7090505@icyb.net.ua> Date: Wed, 14 May 2008 16:19:05 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.12 (X11/20080311) MIME-Version: 1.0 To: Kostik Belousov References: <200804240811.26183.jhb@freebsd.org> <4810FD1E.70602@icyb.net.ua> <20080425095009.GD18958@deviant.kiev.zoral.com.ua> <4811E6BC.4060306@icyb.net.ua> <20080425143646.GF18958@deviant.kiev.zoral.com.ua> <48275C0C.2040601@icyb.net.ua> <20080511214833.GB18958@deviant.kiev.zoral.com.ua> <4829E658.3000605@icyb.net.ua> <20080513191610.GK18958@deviant.kiev.zoral.com.ua> <4829FCB5.8030308@icyb.net.ua> <20080514121920.GR18958@deviant.kiev.zoral.com.ua> In-Reply-To: <20080514121920.GR18958@deviant.kiev.zoral.com.ua> Content-Type: text/plain; charset=KOI8-U Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: devctl (alike?) for devfs X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 13:19:10 -0000 on 14/05/2008 15:19 Kostik Belousov said the following: > On Tue, May 13, 2008 at 11:40:21PM +0300, Andriy Gapon wrote: >> on 13/05/2008 22:16 Kostik Belousov said the following: >>> I looked at your previous patch, and it seems it is much simpler to >>> do drop the devmtx once more then to try to abuse free lists. >>> In the destroy_devl(), after the >>> >>> while (dev->si_threadcount != 0) { >>> /* Use unique dummy wait ident */ >>> msleep(&csw, &devmtx, PRIBIO, "devdrn", hz / 10); >>> } >>> >>> loop, add >>> >>> mtx_unlock(&devmtx); >>> if (!cold) >>> devctl_notify("DEVFS", dev->si_name, "DESTROY", NULL); >>> mtx_lock(&devmtx); >> Thank you again! This is simply perfect. >> >> >> -- >> Andriy Gapon > >> diff --git a/sys/kern/kern_conf.c b/sys/kern/kern_conf.c >> index 1db25f8..f90e469 100644 > > I do not like abusing the subsystem to communicate the data. > > What about this instead ? The created/destroyed cdev name is available > in the $cdev variable, example clause for devd.conf looks like this: Kostik, I do not have a strong opinion on the exact format of notification. Your code definitely looks cleaner and friendlier, so I am all for it. P.S. the only non-obvious thing in "strcat-ing" is that space for '\0' is accounted for in sizeof(prefix) whereas actual '\0' is copied from si_name. > notify 1000 { > system "DEVFS"; > subsystem "CDEV"; > action "echo $type $cdev >/dev/console"; > }; > > diff --git a/sys/kern/kern_conf.c b/sys/kern/kern_conf.c > index e911913..a5ed25c 100644 > --- a/sys/kern/kern_conf.c > +++ b/sys/kern/kern_conf.c > @@ -30,6 +30,7 @@ __FBSDID("$FreeBSD: src/sys/kern/kern_conf.c,v 1.211 2008/04/02 11:11:58 kib Exp > #include > #include > #include > +#include > #include > #include > #include > @@ -526,6 +527,37 @@ unit2minor(int unit) > return ((unit & 0xff) | ((unit << 8) & ~0xffff)); > } > > +static void > +notify(struct cdev *dev, const char *ev) > +{ > + static const char prefix[] = "cdev="; > + char *data; > + int namelen; > + > + if (cold) > + return; > + namelen = strlen(dev->si_name); > + data = malloc(namelen + sizeof(prefix), M_TEMP, M_WAITOK); > + memcpy(data, prefix, sizeof(prefix) - 1); > + memcpy(data + sizeof(prefix) - 1, dev->si_name, namelen + 1); > + devctl_notify("DEVFS", "CDEV", ev, data); > + free(data, M_TEMP); > +} > + > +static void > +notify_create(struct cdev *dev) > +{ > + > + notify(dev, "CREATE"); > +} > + > +static void > +notify_destroy(struct cdev *dev) > +{ > + > + notify(dev, "DESTROY"); > +} > + > static struct cdev * > newdev(struct cdevsw *csw, int y, struct cdev *si) > { > @@ -706,6 +738,9 @@ make_dev_credv(int flags, struct cdevsw *devsw, int minornr, > devfs_create(dev); > clean_unrhdrl(devfs_inos); > dev_unlock_and_free(); > + > + notify_create(dev); > + > return (dev); > } > > @@ -794,6 +829,9 @@ make_dev_alias(struct cdev *pdev, const char *fmt, ...) > clean_unrhdrl(devfs_inos); > dev_unlock(); > dev_depends(pdev, dev); > + > + notify_create(dev); > + > return (dev); > } > > @@ -842,6 +880,10 @@ destroy_devl(struct cdev *dev) > msleep(&csw, &devmtx, PRIBIO, "devdrn", hz / 10); > } > > + mtx_unlock(&devmtx); > + notify_destroy(dev); > + mtx_lock(&devmtx); > + > dev->si_drv1 = 0; > dev->si_drv2 = 0; > bzero(&dev->__si_u, sizeof(dev->__si_u)); -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 13:46:37 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 980B4106566C for ; Wed, 14 May 2008 13:46:37 +0000 (UTC) (envelope-from msaad@datapipe.com) Received: from exchfe01.datapipe-corp.net (exchfe01.datapipe-corp.net [64.106.130.69]) by mx1.freebsd.org (Postfix) with ESMTP id 65CA08FC13 for ; Wed, 14 May 2008 13:46:37 +0000 (UTC) (envelope-from msaad@datapipe.com) Received: from divide.lan (192.168.128.20) by exchfe01.datapipe-corp.net (64.106.130.71) with Microsoft SMTP Server id 8.0.783.2; Wed, 14 May 2008 09:46:36 -0400 Message-ID: <482AED3B.1020307@datapipe.com> Date: Wed, 14 May 2008 09:46:35 -0400 From: Mark Saad User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Mikolaj Golub , References: <482A2639.7000401@datapipe.com> <81zlqtfazy.fsf@zhuzha.ua1> In-Reply-To: <81zlqtfazy.fsf@zhuzha.ua1> Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Re: Socket leak X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: msaad@datapipe.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 13:46:37 -0000 Mikolaj Thanks for the input, did you change any of the options for TimeoutLinger or TimeoutIdle ? The Proftpd I am running is build for 6.3-RELEASE here are the build options Compile-time Settings: Version: 1.3.0a Platform: FREEBSD6 (FREEBSD6_3) Built With: configure CPPFLAGS=-DHAVE_OPENSSL --localstatedir=/var/run --disable-sendfile --disable-ipv6 --with-modules=mod_sql:mod_sql_mysql:mod_check_mysql:mod_check_digest --prefix=/usr/local --with-includes=/usr/local/include/mysql:/usr/include/openssl --with-libraries=/usr/local/lib/mysql This is not built from ports, one of the other S/A who works on this setup built proftpd with MySQL support, not sure why this is not from ports , from what I understand this mysql patch is not in ports. Mikolaj Golub wrote: > On Tue, 13 May 2008 19:37:29 -0400 Mark Saad wrote: > > MS> I started logging the values of kern.ipc.numopensockets and I noticed > MS> that something is leaking sockets. Here is a sample of the log > > MS> 2008-04-29--15:04.10 ____ kern.ipc.numopensockets: 1501 > MS> 2008-04-29--16:04.01 ____ kern.ipc.numopensockets: 1535 > MS> 2008-04-29--17:04.00 ____ kern.ipc.numopensockets: 1617 > MS> 2008-04-29--18:04.00 ____ kern.ipc.numopensockets: 1710 > > MS> This continues until kern.ipc.maxsockets its reached or the box is > MS> rebooted. > > MS> The other thing we looked at was the output from vmstat -z > MS> The first thing was the high amount of malloc 128 bucket failures > > MS> 128 Bucket: 524, 0, 2489, 80, 8364, 23055239 > > MS> I also logged the mbuf clusters, we never reached the max mbuf clusters > > MS> Its almost like there are stale sockets. Here is a snapshot of the server now > > MS> ewr# sockstat -4u |wc -l > MS> 139 > MS> ewr# sysctl kern.ipc.numopensockets > MS> kern.ipc.numopensockets: 13935 > > MS> ewr# uptime > MS> 7:30PM up 6 days, 26 mins, 3 users, load averages: 0.18, 0.25, 0.17 > > We had the same problem on one of hosts running 6.2-RELEASE-p11. The situation > was complicated by the fact that I didn't have root access to the host and > there were problems with getting more debugging or running tcpdump. > > Eventually, it appeared that problem was caused by proftpd. One of our clients > connected to ftp server every five minutes looking for new file to > download. When there was the file everything was good. But when there wasn't, > some strange things happened. In proftpd logs we had: > > FTP session opened. > mod_delay/0.5: delaying for 28 usecs > user fake authenticated by mod_auth_pam.c > USER fake: Login successful. > Preparing to chroot to directory '/var/ftp/fake' > Environment successfully chroot()ed. > mod_delay/0.5: delaying for 621 usecs > Entering Passive Mode (XX,YY,ZZ,213,241,70). > FTP session closed. > > i.e. the client connected to server, had login successful, created new DATA > connection in passive mode and then exited. But although proftpd reported that > connection closed and proftpd process exited we still had this orphaned > connection in our system reported by netstat in ESTABLISHED state. sockstat > did not display this connections. Some of these connections could be in > CLOSE_WAIT mode instead of ESTABLISHED. Such connection was seen by netstat > for several hours and then disappeared but I suspect that the socket buffer > was not freed and numopensockets counter did not decrease. > > Unfortunately, I did not managed to persuade admin to increase DebugLevel in > proftpd.conf and run tcpdump to investigate more what was going on. It turned > out that we had proftpd built for FREEBSD5_4: > > Compile-time Settings: > Version: 1.3.0 > Platform: FREEBSD5 (FREEBSD5_4) > Built With: > configure --localstatedir=/var/run --sysconfdir=/usr/local/etc --disable-sendfile --disable-ipv6 --with-modules=mod_ratio:mod_readme:mod_rewrite:mod_wrap:mod_ifsession --prefix=/usr/local i386-portbld-freebsd5.4 > > Upgrade to more recent proftpd built for proper platform resolved the problem. > > So I would recommend to look for process that could cause this leak. In my > case careful investigation of netstat output history and comparing with > sockstat output helped to find guilty. May be it would help to restart daemons > one by one and see if sockets are freed. > > You can surely increase kern.ipc.maxsockets as workaround until you identify > what causes the problem. > > -- > Mikolaj Golub > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > -- Mark Saad msaad@datapipe.com From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 13:50:54 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6C2291065671; Wed, 14 May 2008 13:50:54 +0000 (UTC) (envelope-from lidl@rox.fddx.com) Received: from rox.fddx.com (rox.fddx.com [71.241.230.59]) by mx1.freebsd.org (Postfix) with ESMTP id 286A88FC14; Wed, 14 May 2008 13:50:53 +0000 (UTC) (envelope-from lidl@rox.fddx.com) Received: from rox.fddx.com (localhost.pix.net [127.0.0.1]) by rox.fddx.com (8.13.8+Sun/8.13.8) with ESMTP id m4EDorKS021798; Wed, 14 May 2008 09:50:53 -0400 (EDT) Received: (from lidl@localhost) by rox.fddx.com (8.13.8+Sun/8.13.8/Submit) id m4EDoqBw021797; Wed, 14 May 2008 09:50:52 -0400 (EDT) Date: Wed, 14 May 2008 09:50:52 -0400 From: "Kurt J. Lidl" To: Garrett Cooper Message-ID: <20080514135051.GA21370@pix.net> Mail-Followup-To: Garrett Cooper , James Mansion , freebsd-hackers@freebsd.org, Jeremy Chadwick , Anthony Pankov References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> <20080513121452.GA70860@eos.sc1.parodius.com> <20080513154137.GA28842@pix.net> <482A02CD.7040308@mansionfamily.plus.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.16 (2007-06-09) X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on rox.fddx.com X-Virus-Status: Clean Cc: freebsd-hackers@freebsd.org, Jeremy Chadwick , Anthony Pankov , James Mansion Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 13:50:54 -0000 On Wed, May 14, 2008 at 12:25:16AM -0700, Garrett Cooper wrote: > Most of the complaints about other DBs is licensing related, but SQLite's > complaint was also the fact that the past stability record was a bit rocky. One other thing to watch for in SQLite is the lack of atomicity in updates. It's not ACID, just like BDB 1.8x isn't ACID. Without a write-ahead log, you cannot be sure that the data written actually made it to stable storage, and as such, you cannot be sure that your database didn't get corrupted when the process stops in a non-optimal way. I view SQLite as adding syntactic sugar to your data access language. It doesn't give you real tranactions, it doesn't give you a write-ahead log, it doesn't give you referential integrity. Heck, the last time I looked (admitted, a while ago), it didn't even enforce column type checking on tables. (Put this string in the INT column? No problem!) -Kurt From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 14:05:01 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F50D106567E; Wed, 14 May 2008 14:05:01 +0000 (UTC) (envelope-from lidl@rox.fddx.com) Received: from rox.fddx.com (rox.fddx.com [71.241.230.59]) by mx1.freebsd.org (Postfix) with ESMTP id B5F4C8FC29; Wed, 14 May 2008 14:05:00 +0000 (UTC) (envelope-from lidl@rox.fddx.com) Received: from rox.fddx.com (localhost.pix.net [127.0.0.1]) by rox.fddx.com (8.13.8+Sun/8.13.8) with ESMTP id m4EE50DQ022052; Wed, 14 May 2008 10:05:00 -0400 (EDT) Received: (from lidl@localhost) by rox.fddx.com (8.13.8+Sun/8.13.8/Submit) id m4EE4xKE022051; Wed, 14 May 2008 10:04:59 -0400 (EDT) Date: Wed, 14 May 2008 10:04:59 -0400 From: "Kurt J. Lidl" To: James Mansion Message-ID: <20080514140458.GB21370@pix.net> Mail-Followup-To: James Mansion , Jeremy Chadwick , Anthony Pankov , freebsd-hackers@freebsd.org References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> <20080513121452.GA70860@eos.sc1.parodius.com> <20080513154137.GA28842@pix.net> <482A02CD.7040308@mansionfamily.plus.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <482A02CD.7040308@mansionfamily.plus.com> User-Agent: Mutt/1.5.16 (2007-06-09) X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on rox.fddx.com X-Virus-Status: Clean Cc: freebsd-hackers@freebsd.org, Jeremy Chadwick , Anthony Pankov Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 14:05:01 -0000 On Tue, May 13, 2008 at 10:06:21PM +0100, James Mansion wrote: > Kurt J. Lidl wrote: >> There are known problems with certain keys corrupting the DB 1.8x >> series code. In fact, the "release" of the 1.86 was an attempt >> to solve this problem when the KerberosV people at MIT found >> a repeatable key insert sequence that would corrupt things. >> (Or at least that's what I remember, it was a long time ago, and >> I might have the details wrong.) >> > Have to say its a little concerning that such 'mature' code is actually > problematic. > Particularly since I'm not aware of a non-LGPL alternative. > > Do you have anything by way of a pointer? Google didn't help me here. This is somewhat alluded to here: http://www.mail-archive.com/kerberos@mit.edu/msg01560.html You might want to read the entire thread on that issue. There is the comment in the Oracle web pages about 1.86 vs 1.85: http://www.oracle.com/technology/software/products/berkeley-db/db/index.html "Do not upgrade to the 1.86 release other than to fix specific hash access method problems found in the 1.85 release." And in the "Berkeley DB: A Retrospective" paper (http://sites.computer.org/debull/A07Sept/seltzer.pdf), Margo notes: "Db-1.85 enjoyed widespread adoption after its release with 4.4BSD. In particular, both the MIT Kerberos project and the University of Michigan LDAP project incorporated it. A group at Harvard released a minor patch version db-1.86 for the Kerberos team in 1996." So, I think my recollection is correct. -Kurt From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 14:25:06 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0CC4E106566B for ; Wed, 14 May 2008 14:25:06 +0000 (UTC) (envelope-from xorquewasp@googlemail.com) Received: from fk-out-0910.google.com (fk-out-0910.google.com [209.85.128.184]) by mx1.freebsd.org (Postfix) with ESMTP id 8F4A88FC0C for ; Wed, 14 May 2008 14:25:05 +0000 (UTC) (envelope-from xorquewasp@googlemail.com) Received: by fk-out-0910.google.com with SMTP id k31so3095393fkk.11 for ; Wed, 14 May 2008 07:25:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:received:received:received:date:from:to:cc:subject:message-id:references:mime-version:content-type:content-disposition:in-reply-to; bh=mSJ3DnEFWxIPnj86pBrrwCNla7+HgjDfSkAUi0RzoBQ=; b=Mr09cTTQg1twdeE73cqhbXZFOfqDl/iRPO9luPGxYfYD7ryLkCjCGHejJ/CcaU1rg2JAKBHslueNzV/dYrELxQC7OhIFUOm5EsSduJ9PMwksehb9mQ1Ml9gfCWMj5FExflsfPNtWfoBYkmRQ3IPu43PlVTkxioK8KW6BJPggFjc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version:content-type:content-disposition:in-reply-to; b=mtSZTY+HNAiEcisnRKRxWxSKaQf/LZPKFzINavf1I+oRM3ojSyS762KDIqDHeMATfAMGr0eWHvZrOfZ2YnsQXaNjLiIWwWWVOD0htOmn60M/KUKrMg7bq7vkK9p5zwHJtHm6RJSUNz7bZo/Fv4JXyIE05+2K+YznlzI2Cpa3T+Y= Received: by 10.125.162.17 with SMTP id p17mr423828mko.67.1210773539502; Wed, 14 May 2008 06:58:59 -0700 (PDT) Received: from logik.internal.network ( [81.86.41.187]) by mx.google.com with ESMTPS id d4sm1372130fga.4.2008.05.14.06.58.44 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 14 May 2008 06:58:47 -0700 (PDT) Received: by logik.internal.network (Postfix, from userid 11001) id 5563B5F41; Wed, 14 May 2008 13:58:22 +0000 (UTC) Date: Wed, 14 May 2008 13:58:22 +0000 From: xorquewasp@googlemail.com To: "Kurt J. Lidl" Message-ID: <20080514135822.GA4873@logik.internal.network> References: <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> <20080513121452.GA70860@eos.sc1.parodius.com> <20080513154137.GA28842@pix.net> <482A02CD.7040308@mansionfamily.plus.com> <20080514135051.GA21370@pix.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080514135051.GA21370@pix.net> Cc: freebsd-hackers@freebsd.org Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 14:25:06 -0000 On 20080514 09:50:52, Kurt J. Lidl wrote: > Heck, the last time I looked (admitted, a while ago), it didn't > even enforce column type checking on tables. (Put this string in > the INT column? No problem!) They consider it a feature. Nobody knows why. From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 14:24:17 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A81901065670 for ; Wed, 14 May 2008 14:24:17 +0000 (UTC) (envelope-from mwm@mired.org) Received: from mired.org (five.mired.org [66.92.153.75]) by mx1.freebsd.org (Postfix) with ESMTP id 342538FC2B for ; Wed, 14 May 2008 14:24:16 +0000 (UTC) (envelope-from mwm@mired.org) Received: (qmail 43060 invoked from network); 14 May 2008 10:24:43 -0400 Received: from unknown (HELO mbook.local) (192.168.195.192) by 0 with SMTP; 14 May 2008 10:24:43 -0400 Date: Wed, 14 May 2008 10:24:10 -0400 From: Mike Meyer To: Peter Jeremy Message-ID: <20080514102410.639f16ef@mbook.local> In-Reply-To: <20080514071728.GP64804@server.vk2pj.dyndns.org> References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> <20080513121452.GA70860@eos.sc1.parodius.com> <20080513154137.GA28842@pix.net> <482A02CD.7040308@mansionfamily.plus.com> <20080514071728.GP64804@server.vk2pj.dyndns.org> Organization: Meyer Consulting X-Mailer: Claws Mail 3.3.1 (GTK+ 2.12.9; i386-apple-darwin9.2.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 14 May 2008 14:42:20 +0000 Cc: freebsd-hackers@freebsd.org, James Mansion Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 14:24:17 -0000 On Wed, 14 May 2008 17:17:28 +1000 Peter Jeremy wrote: > On 2008-May-13 22:06:21 +0100, James Mansion wrote: > >And is the objection to SQL such the sqlite is really out of the running? > > There is no specific objection to SQL. There is a general objection > to adding more utilities to the base system unless a _very_ good case > can be made for including them. So far, no-one has made a compelling > reason to include SQLite (or other SQL engine) into the base system. Just out of curiosity - there seems to be an unspoken assumption that the ports system can only use tools that are part of the base system. But this is clearly false - the ports system currently includes a couple of directories full of software that's not in the base system. Adding compiled code to those tools would mean that installing the ports system gets a bit more complex - you have to run "make install" after extracting the tarball. Is that so bad it's not going to happen? Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D47D01065674 for ; Wed, 14 May 2008 14:52:11 +0000 (UTC) (envelope-from antonfb@hesiod.org) Received: from paris.hesiod.org (jaanton-pt.tunnel.tserv3.fmt2.ipv6.he.net [IPv6:2001:470:1f04:d1::2]) by mx1.freebsd.org (Postfix) with ESMTP id 871E48FC19 for ; Wed, 14 May 2008 14:52:11 +0000 (UTC) (envelope-from antonfb@hesiod.org) Received: from atlas.hesiod.org (atlas.hesiod.org [192.168.1.9]) by paris.hesiod.org (8.14.2/8.14.2) with ESMTP id m4EEqAUM027152 for ; Wed, 14 May 2008 07:52:11 -0700 (PDT) (envelope-from antonfb@hesiod.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hesiod.org; s=dec07; t=1210776731; bh=++7EMRhSNE+JoENslt4nePVddVTbvICy6Sl2mbVkPm8=; h=DomainKey-Signature:Message-ID:Date:From:User-Agent:MIME-Version: To:Subject:References:In-Reply-To:Content-Type: Content-Transfer-Encoding; b=q3603vJHcregnmQctl+nHbdM9ewHvSoFGXo2q b+wFUyWwGowqHqc51GvcAdWnNOIH96hIok+jqd2s5gya8at2PvTKm3WA0kG17VpprDB GYrdCfyDiU1B6fjyt77iTr97OHDmX3MWNusiJ3NLLaOT1R77b4jwgZLac9xpnYOijX0 = DomainKey-Signature: a=rsa-sha1; s=dec07; d=hesiod.org; c=nofws; q=dns; h=message-id:date:from:user-agent:mime-version:to:subject: references:in-reply-to; b=jdk0sNLH0tHhsUpIMPeRgrwGlRe13xgdmPv87Q/S6UIGYdPOLXLpjuY9H9qaJaAgi YlwmIZbw4nzkldFyRhc0WwsakqrmkJj6M85NmGBnGZm7uTFdFAmvfHtU2oPyri6iFjK 5/ItuWjP2iB4FPWUDfOIxjmv5lRRVaxPbuszp/c= Message-ID: <482AFC9A.7050807@hesiod.org> Date: Wed, 14 May 2008 07:52:10 -0700 From: Jeff Anton User-Agent: Thunderbird 2.0.0.12 (X11/20080426) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org References: <20080514120011.9131F10656C4@hub.freebsd.org> In-Reply-To: <20080514120011.9131F10656C4@hub.freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: BDB corrupt - patches X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 14:52:11 -0000 Some years ago I mailed patches out to someone regarding Berkeley DB 1.85 btree problems. The two issues which come to mind are... 1) The page split position is improperly computed. This can cause corruption when a very full page has an item which is very large inserted onto it. The split then happens but because the split isn't in the right place the inserted item will not fit and I think it overruns where it should on the page. (I think 1.86 tried to fix this, but the fix is about recovering late i.e. when it sees an item doesn't fit, the proper fix is to split the page right in the first place.) 2) The record put code has a "last page put to" member which speeds up sequential inserts, however if that last page put to gets deleted, that variable is not cleared causing, if you are unlucky, a put might try to put data onto a deleted page which is then woven into both the tree and the free page list. This causes terrible problems with record scanning and eventual corruption. I'm going to have to dig up these fixes, but presuming I do, who should be alerted? Just file a bug? Recreation is extremely difficult. Jeff Anton From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 15:34:44 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F272106567B for ; Wed, 14 May 2008 15:34:44 +0000 (UTC) (envelope-from fabonacci@yahoo.com) Received: from n56.bullet.mail.sp1.yahoo.com (n56.bullet.mail.sp1.yahoo.com [98.136.44.52]) by mx1.freebsd.org (Postfix) with SMTP id 15C7C8FC21 for ; Wed, 14 May 2008 15:34:44 +0000 (UTC) (envelope-from fabonacci@yahoo.com) Received: from [216.252.122.217] by n56.bullet.mail.sp1.yahoo.com with NNFMP; 14 May 2008 15:21:38 -0000 Received: from [69.147.84.109] by t2.bullet.sp1.yahoo.com with NNFMP; 14 May 2008 15:21:38 -0000 Received: from [127.0.0.1] by omp207.mail.sp1.yahoo.com with NNFMP; 14 May 2008 15:21:38 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 303491.50155.bm@omp207.mail.sp1.yahoo.com Received: (qmail 78606 invoked by uid 60001); 14 May 2008 15:21:38 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=sBmkZYiJV42JiWff04DFGhIniIcIzZLEePwWVsobRVn+5cUhpwaORc+PLf7iDYF5nmEptZALNn05L8XL1IkdFKOUdPLoGBLGFD2NFswNKMTET782McEmz6sAHM5i745/yoei+/bCgQkJkYlqJx2X4+5VzWq403JcpLuZyqK1jNw=; X-YMail-OSG: QIDEDOAVM1lUg1im6vxnm0i_ZmKZwWnnTrQGlL5fGaz4aivA3ti8m_SQaJtQegDU1A2n4LtctGhv Received: from [219.128.146.102] by web46306.mail.sp1.yahoo.com via HTTP; Wed, 14 May 2008 08:21:37 PDT Date: Wed, 14 May 2008 08:21:37 -0700 (PDT) From: John Timony To: MIME-Version: 1.0 Message-ID: <163189.78306.qm@web46306.mail.sp1.yahoo.com> Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: A prob to Installing FreeBSD7.0 in VMWare 5.0 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 15:34:44 -0000 BlankBODY { MARGIN-TOP: 25px; FONT-SIZE: 10pt; MARGIN-LEFT: 25px; COLOR: #000000; =FONT-FAMILY: Arial, Helvetica}P.msoNormal { MARGIN-TOP: 0px; FONT-SIZE: 10pt; MARGIN-LEFT: 0px; COLOR: #ffffcc; =FONT-FAMILY: Helvetica, "Times New Roman"}LI.msoNormal { MARGIN-TOP: 0px; FONT-SIZE: 10pt; MARGIN-LEFT: 0px; COLOR: #ffffcc; =FONT-FAMILY: Helvetica, "Times New Roman"}Hi,I use the CD-ROM to =install FreeBSD7.O=20in VMWare 5.0,But when i start the Virtual Mathine,I get that ="elf32_loadimage=20failed"... By the way,,,I use a =FreeBSD7.0.iso=20mounted in CD-ROM... Any ideas? BlankBODY { MARGIN-TOP: 25px; FONT-SIZE: 10pt; MARGIN-LEFT: 25px; COLOR: #000000; FONT-FAMILY: Arial, Helvetica}P.msoNormal { MARGIN-TOP: 0px; FONT-SIZE: 10pt; MARGIN-LEFT: 0px; COLOR: #ffffcc; FONT-FAMILY: Helvetica, "Times New Roman"}LI.msoNormal { MARGIN-TOP: 0px; FONT-SIZE: 10pt; MARGIN-LEFT: 0px; COLOR: #ffffcc; FONT-FAMILY: Helvetica, "Times New Roman"}Hi,I use the CD-ROM to install FreeBSD7.O in VMWare 5.0,But when i start the Virtual Mathine,I get that "elf32_loadimage failed"... By the way,,,I use a FreeBSD7.0.iso mounted in CD-ROM... Any ideas? From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 15:48:53 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA62C106566B for ; Wed, 14 May 2008 15:48:53 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from kientzle.com (h-66-166-149-50.snvacaid.covad.net [66.166.149.50]) by mx1.freebsd.org (Postfix) with ESMTP id ACD9D8FC19 for ; Wed, 14 May 2008 15:48:53 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from [10.0.0.128] (p54.kientzle.com [66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id m4EFmrtv012137; Wed, 14 May 2008 08:48:53 -0700 (PDT) (envelope-from kientzle@freebsd.org) Message-ID: <482B09E5.5010302@freebsd.org> Date: Wed, 14 May 2008 08:48:53 -0700 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060422 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Jeff Anton References: <20080514120011.9131F10656C4@hub.freebsd.org> <482AFC9A.7050807@hesiod.org> In-Reply-To: <482AFC9A.7050807@hesiod.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: BDB corrupt - patches X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 15:48:53 -0000 Jeff Anton wrote: > Some years ago I mailed patches out to someone regarding > Berkeley DB 1.85 btree problems. The two issues which come > to mind are... > > 1) The page split position is improperly computed. ... > 2) The record put code has a "last page put to" member ... > > I'm going to have to dig up these fixes, but presuming > I do, who should be alerted? Just file a bug? Recreation > is extremely difficult. Definitely file a PR and include the patches. If nothing else, that will help ensure they don't get lost. When you do file that PR, notify this list. Given how heavily the base system uses DB 1.85, I think there will be a lot of interest in anything that improves the stability and reliability of that code. Cheers, Tim Kientzle From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 17:51:56 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 84977106566C for ; Wed, 14 May 2008 17:51:56 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.delphij.net (delphij-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:2c9::2]) by mx1.freebsd.org (Postfix) with ESMTP id 2FD228FC14 for ; Wed, 14 May 2008 17:51:56 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.geekcn.org (tarsier.geekcn.org [202.108.54.204]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tarsier.delphij.net (Postfix) with ESMTPS id 12D6928448 for ; Thu, 15 May 2008 01:51:55 +0800 (CST) Received: from localhost (tarsier.geekcn.org [202.108.54.204]) by tarsier.geekcn.org (Postfix) with ESMTP id C1119EB1246; Thu, 15 May 2008 01:51:52 +0800 (CST) X-Virus-Scanned: amavisd-new at geekcn.org Received: from tarsier.geekcn.org ([202.108.54.204]) by localhost (mail.geekcn.org [202.108.54.204]) (amavisd-new, port 10024) with ESMTP id 5Of189Tzv-Pv; Thu, 15 May 2008 01:51:47 +0800 (CST) Received: from charlie.delphij.net (71.5.7.139.ptr.us.xo.net [71.5.7.139]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTPSA id 060F6EB0A75; Thu, 15 May 2008 01:51:45 +0800 (CST) DomainKey-Signature: a=rsa-sha1; s=default; d=delphij.net; c=nofws; q=dns; h=message-id:date:from:reply-to:organization:user-agent: mime-version:to:cc:subject:references:in-reply-to: x-enigmail-version:openpgp:content-type:content-transfer-encoding; b=S+UaXX/KiTJl6QMGsqfZoFSNnB9AWzBbkTNLqrcRO0RvAab7A4mp48i5uxLtND0US tYMX+nBR8zcwYrYoZUP3w== Message-ID: <482B26AF.3050302@delphij.net> Date: Wed, 14 May 2008 10:51:43 -0700 From: Xin LI Organization: The FreeBSD Project User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jeff Anton References: <20080514120011.9131F10656C4@hub.freebsd.org> <482AFC9A.7050807@hesiod.org> In-Reply-To: <482AFC9A.7050807@hesiod.org> X-Enigmail-Version: 0.95.6 OpenPGP: id=18EDEBA0; url=http://www.delphij.net/delphij.asc Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: BDB corrupt - patches X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: d@delphij.net List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 17:51:56 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeff Anton wrote: [...] | I'm going to have to dig up these fixes, but presuming | I do, who should be alerted? Just file a bug? Recreation | is extremely difficult. I think Oracle is maintaining a webpage about their known bug/fixes here: http://www.oracle.com/technology/software/products/berkeley-db/db/index.html FreeBSD has applied a number of fixes IIRC, if it was a BerkeleyDB bug I'd suggest that you also give Oracle developers a headsup. Cheers, - -- ** Help China's quake relief at http://www.redcross.org.cn/ |>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkgrJq4ACgkQi+vbBBjt66CB3gCfQ/hhQXVsyItDVmbP/j4lbI4v 8k8AnRgqtHZeg7lw9aacSeXNb8uYo9Ae =dWBS -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 18:03:58 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7DC591065673; Wed, 14 May 2008 18:03:58 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (merlin.alerce.com [64.62.142.94]) by mx1.freebsd.org (Postfix) with ESMTP id 62D838FC1E; Wed, 14 May 2008 18:03:58 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id 13D6833C62; Wed, 14 May 2008 10:40:36 -0700 (PDT) Received: from postfix.alerce.com (w092.z064001164.sjc-ca.dsl.cnc.net [64.1.164.92]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by merlin.alerce.com (Postfix) with ESMTP id C785833C5B; Wed, 14 May 2008 10:40:35 -0700 (PDT) Received: by postfix.alerce.com (Postfix, from userid 501) id 07FCB42CABF; Wed, 14 May 2008 10:40:25 -0700 (PDT) From: George Hartzell MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18475.9225.878567.877705@almost.alerce.com> Date: Wed, 14 May 2008 10:40:25 -0700 To: "Kurt J. Lidl" In-Reply-To: <20080514135051.GA21370@pix.net> References: <9FC19AC2-DAD8-418C-8B9C-F129DEC58CEF@gmail.com> <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> <20080513121452.GA70860@eos.sc1.parodius.com> <20080513154137.GA28842@pix.net> <482A02CD.7040308@mansionfamily.plus.com> <20080514135051.GA21370@pix.net> X-Mailer: VM 7.19 under Emacs 22.1.50.1 X-Virus-Scanned: ClamAV using ClamSMTP X-Mailman-Approved-At: Wed, 14 May 2008 18:22:10 +0000 Cc: Garrett Cooper , Jeremy Chadwick , Anthony Pankov , James Mansion , freebsd-hackers@freebsd.org Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: hartzell@alerce.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 18:03:58 -0000 Kurt J. Lidl writes: > On Wed, May 14, 2008 at 12:25:16AM -0700, Garrett Cooper wrote: > > Most of the complaints about other DBs is licensing related, but SQLite's > > complaint was also the fact that the past stability record was a bit rocky. > > One other thing to watch for in SQLite is the lack of atomicity > in updates. It's not ACID, just like BDB 1.8x isn't ACID. > > Without a write-ahead log, you cannot be sure that the data written > actually made it to stable storage, and as such, you cannot be sure > that your database didn't get corrupted when the process stops in a > non-optimal way. In what way is SQLite not atomic? The documentation, Atomic Commit In SQLite, suggests that it is: http://www.sqlite.org/atomiccommit.html I don't know that it supports fully ACID (atomic, consist, isolated, durable) transactions or how it handles the various SQL standard transaction isolation levels (Read Uncommitted, Read Committed, Repeatable Read, Serializable) but I believe that updates are atomic and that it does as well as any db (in the face of lying synch. operations, etc...) to handle "non-optimal" stops. g. From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 19:18:52 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A75B1065674 for ; Wed, 14 May 2008 19:18:52 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from fk-out-0910.google.com (fk-out-0910.google.com [209.85.128.185]) by mx1.freebsd.org (Postfix) with ESMTP id 8EA468FC0C for ; Wed, 14 May 2008 19:18:50 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: by fk-out-0910.google.com with SMTP id k31so32854fkk.11 for ; Wed, 14 May 2008 12:18:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:to:subject:references:organization:from:date:in-reply-to:message-id:user-agent:mime-version:content-type; bh=15lGqO6i93R5LYMNUFAcal6V2DKxrYp8b8LwNBeYDQQ=; b=ZnTHr+8Q/gpAl5YbtIII8nZnstiTaALQ6W8N4yAyuNYmtjEa9LLZ1wU8BypW/zkb6YwrOCe86iE+2X4bWi5mKxApdEGu36YdXjfCVoq1zdZy7jo2oOHu9iUhmiLWoHApoZvK4QWOocVleZ9qut+u1vljz/5IUV86O5OeScxLDA0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=to:subject:references:organization:from:date:in-reply-to:message-id:user-agent:mime-version:content-type; b=qC1UHkp0YfWV6A3M7ByfiLFoVMbxf8u5cUBQCrhXtQc86oD3d8VMgERFm5gM6K1B6RuZBL37Qh4CHWOM6bpvJv8b880Ij6Vuj/h7e5aRdf2iDwufLRGZrlJUY2Tk3cAv2ywjsZnGkLDSkTgtdpM9vsl5/Ik/PWnU/VHenSBB0bw= Received: by 10.125.87.8 with SMTP id p8mr749072mkl.80.1210792729186; Wed, 14 May 2008 12:18:49 -0700 (PDT) Received: from localhost ( [80.85.90.67]) by mx.google.com with ESMTPS id 3sm2010358fge.3.2008.05.14.12.18.47 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 14 May 2008 12:18:48 -0700 (PDT) To: References: <482A2639.7000401@datapipe.com> <81zlqtfazy.fsf@zhuzha.ua1> <482AED3B.1020307@datapipe.com> Organization: TOA Ukraine From: Mikolaj Golub Date: Wed, 14 May 2008 22:18:45 +0300 In-Reply-To: <482AED3B.1020307@datapipe.com> (Mark Saad's message of "Wed\, 14 May 2008 09\:46\:35 -0400") Message-ID: <81y76c7kyy.fsf@zhuzha.ua1> User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: Socket leak X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 19:18:52 -0000 On Wed, 14 May 2008 09:46:35 -0400 Mark Saad wrote: MS> Mikolaj MS> Thanks for the input, did you change any of the options for MS> TimeoutLinger or TimeoutIdle ? No, I didn't MS> The Proftpd I am running is build for 6.3-RELEASE here are the build MS> options MS> Compile-time Settings: MS> Version: 1.3.0a MS> Platform: FREEBSD6 (FREEBSD6_3) MS> Built With: MS> configure CPPFLAGS=-DHAVE_OPENSSL --localstatedir=/var/run MS> --disable-sendfile --disable-ipv6 MS> --with-modules=mod_sql:mod_sql_mysql:mod_check_mysql:mod_check_digest MS> --prefix=/usr/local MS> --with-includes=/usr/local/include/mysql:/usr/include/openssl MS> --with-libraries=/usr/local/lib/mysql It might be that it is not proftpd but other application that cause the leak. Anyway, to check if it is proftpd, look in its logs for entries like these: Entering Passive Mode (192,168,0,213,241,70). FTP session closed. Convert the last two numbers to port (241*256+70) and check by netstat if you still have this connection. If you have, then it is likely this is the same situation as in my case and the proftpd is a problem. Upgrade to 1.3.1 from ports then. If proftpd is ok, look for other applications. Search for connections reported by netstat as ESTABLISHED but not displayed by sockstat utility. You could run something like this: netstat -an | grep ESTABL | while read b l a local remote state; do echo -n "$local $remote: " sockstat | sed -e 's/:/./g' | grep -c "$local *$remote" done Look for sockets with 0 count. These are suspicious ones. Observe these sockets by netstat and try to figure out what application they could belong and dig in that direction. -- Mikolaj Golub From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 19:20:33 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 341D1106566B for ; Wed, 14 May 2008 19:20:33 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail16.syd.optusnet.com.au (mail16.syd.optusnet.com.au [211.29.132.197]) by mx1.freebsd.org (Postfix) with ESMTP id BAFF78FC0C for ; Wed, 14 May 2008 19:20:32 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from server.vk2pj.dyndns.org (c122-106-215-175.belrs3.nsw.optusnet.com.au [122.106.215.175]) by mail16.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m4EJKUZW015740 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 May 2008 05:20:30 +1000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.2/8.14.2) with ESMTP id m4EJKRFc064624; Thu, 15 May 2008 05:20:27 +1000 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.2/8.14.2/Submit) id m4EJKQMa064623; Thu, 15 May 2008 05:20:26 +1000 (EST) (envelope-from peter) Date: Thu, 15 May 2008 05:20:26 +1000 From: Peter Jeremy To: Garrett Cooper , freebsd-hackers@freebsd.org Message-ID: <20080514192026.GY64804@server.vk2pj.dyndns.org> References: <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> <20080513121452.GA70860@eos.sc1.parodius.com> <20080513154137.GA28842@pix.net> <482A02CD.7040308@mansionfamily.plus.com> <20080514135051.GA21370@pix.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="76DTJ5CE0DCVQemd" Content-Disposition: inline In-Reply-To: <20080514135051.GA21370@pix.net> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.17 (2007-11-01) Cc: Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 19:20:33 -0000 --76DTJ5CE0DCVQemd Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2008-May-14 09:50:52 -0400, "Kurt J. Lidl" wrote: >One other thing to watch for in SQLite is the lack of atomicity >in updates. It's not ACID, just like BDB 1.8x isn't ACID. This isn't true. SQLite does provide full ACID. One difference from (eg) Oracle is that you need to explicitly begin a transaction, rather than a transaction implicitly commencing with the first DML statement. (I don't know what the SQL standard requires). >Without a write-ahead log, It does have a log to record incomplete transactions. > it doesn't give you referential integrity. To quote http://www.sqlite.org/omitted.html: "FOREIGN KEY constraints are parsed but are not enforced. However, the equivalent constraint enforcement can be achieved using triggers." >I looked (admitted, a while ago), it didn't even enforce column type >checking on tables. (Put this string in the INT column? No problem!) As noted, this is seen as a feature, not a bug. No-one is claiming SQLite is perfect. It does, however, provide a very usable SQL engine, that provides most of SQL92, with a very small footprint and a friendly license. If FreeBSD does need something with capabilities than BDB1.x, SQLite is an obvious contender. --=20 Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. --76DTJ5CE0DCVQemd Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkgrO3oACgkQ/opHv/APuIcGlACeMy6GjB9JXeti7gmkqFaHNc2W eLUAoKrE5thxmFJnGuejOHRipmk6QPxW =Ycw4 -----END PGP SIGNATURE----- --76DTJ5CE0DCVQemd-- From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 19:45:34 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D655E1065678 for ; Wed, 14 May 2008 19:45:34 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail07.syd.optusnet.com.au (mail07.syd.optusnet.com.au [211.29.132.188]) by mx1.freebsd.org (Postfix) with ESMTP id 699478FC1A for ; Wed, 14 May 2008 19:45:34 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from server.vk2pj.dyndns.org (c122-106-215-175.belrs3.nsw.optusnet.com.au [122.106.215.175]) by mail07.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m4EJjUTu008958 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 May 2008 05:45:31 +1000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.2/8.14.2) with ESMTP id m4EJjU9o064726; Thu, 15 May 2008 05:45:30 +1000 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.2/8.14.2/Submit) id m4EJjTBr064725; Thu, 15 May 2008 05:45:29 +1000 (EST) (envelope-from peter) Date: Thu, 15 May 2008 05:45:29 +1000 From: Peter Jeremy To: Mike Meyer Message-ID: <20080514194529.GB64804@server.vk2pj.dyndns.org> References: <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> <20080513121452.GA70860@eos.sc1.parodius.com> <20080513154137.GA28842@pix.net> <482A02CD.7040308@mansionfamily.plus.com> <20080514071728.GP64804@server.vk2pj.dyndns.org> <20080514102410.639f16ef@mbook.local> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="KlAEzMkarCnErv5Q" Content-Disposition: inline In-Reply-To: <20080514102410.639f16ef@mbook.local> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.17 (2007-11-01) Cc: freebsd-hackers@freebsd.org Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 19:45:34 -0000 --KlAEzMkarCnErv5Q Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2008-May-14 10:24:10 -0400, Mike Meyer wrote: >Just out of curiosity - there seems to be an unspoken assumption that >the ports system can only use tools that are part of the base >system. There have been suggestions that the ports/package infrastructure (pkg_* tools, portsnap etc) be unbundled from the base OS. The difficulty comes when you want to upgrade those components. I know, =66rom experience, that portugrading portupgrade or ruby usually fails as the running portupgrade unexpectedly trips over changed bits of itself. > But this is clearly false - the ports system currently >includes a couple of directories full of software that's not in the >base system. There is a directory full of Makefile includes and another directory full of optional tools but pkg_* sits in the base system. What are you alluding to here. >Adding compiled code to those tools would mean that installing the >ports system gets a bit more complex - you have to run "make install" >after extracting the tarball. Is that so bad it's not going to happen? The problem is not the initial install so much as managing packages and upgrades. I see no problem with having the ports/package infrastructure be part of the ports system as long as: a) A user can install/uninstall/audid (and preferably upgrade) packages without needing to compile anything b) The ports system knows how to upgrade itself without tripping over itself in the process. --=20 Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. --KlAEzMkarCnErv5Q Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkgrQVkACgkQ/opHv/APuIeILACfT2gZ66noHlSlAtFamR+6BW4o vtoAoLzJg88ymeIhmo1yMaQNueiIH/lh =dTjY -----END PGP SIGNATURE----- --KlAEzMkarCnErv5Q-- From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 20:04:03 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 139701065670 for ; Wed, 14 May 2008 20:04:03 +0000 (UTC) (envelope-from reg@openpave.org) Received: from mx1.ucdavis.edu (mx1.ucdavis.edu [128.120.32.31]) by mx1.freebsd.org (Postfix) with ESMTP id D71048FC1E for ; Wed, 14 May 2008 20:04:02 +0000 (UTC) (envelope-from reg@openpave.org) Received: from flint.openpave.org ([169.237.230.40]) by mx1.ucdavis.edu (8.13.7/8.13.1/it-defang-5.4.0) with ESMTP id m4EK3sir026562 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 14 May 2008 13:04:00 -0700 (PDT) Received: from sandy.local (flint.local [192.168.1.5]) by flint.openpave.org (8.14.2/8.14.2) with ESMTP id m4EK3rTv077440; Wed, 14 May 2008 20:03:53 GMT (envelope-from reg@sandy.local) Received: (from reg@localhost) by sandy.local (8.14.2/8.14.2/Submit) id m4EK3rj8077439; Wed, 14 May 2008 13:03:53 -0700 (PDT) (envelope-from reg) Date: Wed, 14 May 2008 13:03:53 -0700 From: Jeremy Lea To: "Daniel O'Connor" Message-ID: <20080514200353.GA77403@flint.openpave.org> References: <20080512204558.GA44114@flint.openpave.org> <48291719.5010801@freebsd.org> <200805141755.35892.doconnor@gsoft.com.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200805141755.35892.doconnor@gsoft.com.au> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on av4 X-Virus-Status: Clean X-Scanned-By: MIMEDefang 2.57 on 128.120.32.8 Cc: freebsd-hackers@FreeBSD.ORG, Tim Kientzle , Anders Nore Subject: Re: Adding .db support to pkg_tools X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 20:04:03 -0000 Hi, On Wed, May 14, 2008 at 05:55:33PM +0930, Daniel O'Connor wrote: > So long as you can tell it is corrupted.. > It's also a drag from a user POV when the tool crashes because the DB is > hosed (seen in portupgrade a number of times) The simplest way to achieve some measure of protection (provided the database itself is not buggy), is to use a single lock file in /var/db/pkg. The ports tools can create and lock this as needed, and if the file exists, but they are able to obtain an exclusive lock on it, then it's a dangling lock. In that case, the tools crashed, and the database caches should be rebuilt. This is not fool proof, but it would reduce the number of FAQs on the mailing lists. This kills two birds with one stone, since it would serialize access to /var/db/pkg, so that various tools don't stomp on one another. An exclusive lock can be held by pkg_create, pkg_add and pkg_delete as needed. pkg_info and pkg_version can downgrade the lock to a shared lock, and just continue if they can get a shared lock, or block if the file is exclusively locked. One problem is the pkg_add calls itself recursively, so it would have to drop the lock around those calls. That could be hard. If I recall correctly, it was not that hard to remove this recursive behaviour. Also, since bsd.port.mk grubs around in /var/db/pkg, it would need to learn to participate in the locking. Regards, -Jeremy -- FreeBSD - Because the best things in life are free... http://www.freebsd.org/ From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 21:02:45 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6AEDE1065670 for ; Wed, 14 May 2008 21:02:45 +0000 (UTC) (envelope-from lidl@rox.fddx.com) Received: from rox.fddx.com (rox.fddx.com [71.241.230.59]) by mx1.freebsd.org (Postfix) with ESMTP id 2D4958FC0C for ; Wed, 14 May 2008 21:02:44 +0000 (UTC) (envelope-from lidl@rox.fddx.com) Received: from rox.fddx.com (localhost.pix.net [127.0.0.1]) by rox.fddx.com (8.13.8+Sun/8.13.8) with ESMTP id m4EL2iCj029094; Wed, 14 May 2008 17:02:44 -0400 (EDT) Received: (from lidl@localhost) by rox.fddx.com (8.13.8+Sun/8.13.8/Submit) id m4EL2fml029093; Wed, 14 May 2008 17:02:41 -0400 (EDT) Date: Wed, 14 May 2008 17:02:41 -0400 From: "Kurt J. Lidl" To: Peter Jeremy Message-ID: <20080514210240.GB28827@pix.net> Mail-Followup-To: Peter Jeremy , Garrett Cooper , freebsd-hackers@freebsd.org References: <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> <20080513121452.GA70860@eos.sc1.parodius.com> <20080513154137.GA28842@pix.net> <482A02CD.7040308@mansionfamily.plus.com> <20080514135051.GA21370@pix.net> <20080514192026.GY64804@server.vk2pj.dyndns.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080514192026.GY64804@server.vk2pj.dyndns.org> User-Agent: Mutt/1.5.16 (2007-06-09) X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on rox.fddx.com X-Virus-Status: Clean Cc: Garrett Cooper , freebsd-hackers@freebsd.org Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 21:02:45 -0000 On Thu, May 15, 2008 at 05:20:26AM +1000, Peter Jeremy wrote: > On 2008-May-14 09:50:52 -0400, "Kurt J. Lidl" wrote: > >One other thing to watch for in SQLite is the lack of atomicity > >in updates. It's not ACID, just like BDB 1.8x isn't ACID. > > This isn't true. SQLite does provide full ACID. One difference from > (eg) Oracle is that you need to explicitly begin a transaction, rather > than a transaction implicitly commencing with the first DML statement. > (I don't know what the SQL standard requires). Generally, you get either implicit transactions, or you need to put your database handle into explicit transaction mode, typically by bracketing your sql with: begin transaction; stuff; stuff; commit; if (error) { rollback; whine() } > >Without a write-ahead log, > > It does have a log to record incomplete transactions. Well, thanks for the various pointers. I see that it grew a transaction log since the last time I bothered to look at it in depth. That's a very good thing. I'll retract my assertion that it doesn't have a commit log. -Kurt From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 21:03:39 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5ABF91065676 for ; Wed, 14 May 2008 21:03:39 +0000 (UTC) (envelope-from msaad@datapipe.com) Received: from exchfe05.datapipe-corp.net (exchfe05.datapipe-corp.net [64.106.130.125]) by mx1.freebsd.org (Postfix) with ESMTP id 290DF8FC2D for ; Wed, 14 May 2008 21:03:38 +0000 (UTC) (envelope-from msaad@datapipe.com) Received: from oceanspray.bad-apples.org (192.168.131.233) by exchfe05.datapipe-corp.net (64.106.130.125) with Microsoft SMTP Server (TLS) id 8.1.278.0; Wed, 14 May 2008 17:03:38 -0400 Message-ID: <482B5364.7080406@datapipe.com> Date: Wed, 14 May 2008 17:02:28 -0400 From: Mark Saad User-Agent: Thunderbird 2.0.0.12 (X11/20080328) MIME-Version: 1.0 To: Mikolaj Golub References: <482A2639.7000401@datapipe.com> <81zlqtfazy.fsf@zhuzha.ua1> <482AED3B.1020307@datapipe.com> <81y76c7kyy.fsf@zhuzha.ua1> In-Reply-To: <81y76c7kyy.fsf@zhuzha.ua1> Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable Cc: "freebsd-hackers@freebsd.org" Subject: Re: Socket leak X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: msaad@datapipe.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 21:03:39 -0000 Mikolaj I looked at netstat and I do not have this many sockets TCP or UNIX. Wed May 14 16:58:37 EDT 2008 ewr# sysctl kern.ipc.numopensockets && netstat -an -p tcp | wc -l && sockstat -u |wc -l kern.ipc.numopensockets: 15903 261 60 ewr# sockstat -46lu | wc -l 82 Running your script I can only find 1 matching 0 count socket . I also shutdown proftpd and left it down for 10 mins and I did not see the number of sockets drop at all. Any ideas ? Mikolaj Golub wrote: > On Wed, 14 May 2008 09:46:35 -0400 Mark Saad wrote: > > MS> Mikolaj > MS> Thanks for the input, did you change any of the options for > MS> TimeoutLinger or TimeoutIdle ? > > No, I didn't > > MS> The Proftpd I am running is build for 6.3-RELEASE here are the buil= d > MS> options > > MS> Compile-time Settings: > MS> Version: 1.3.0a > MS> Platform: FREEBSD6 (FREEBSD6_3) > MS> Built With: > MS> configure CPPFLAGS=3D-DHAVE_OPENSSL --localstatedir=3D/var/run > MS> --disable-sendfile --disable-ipv6 > MS> --with-modules=3Dmod_sql:mod_sql_mysql:mod_check_mysql:mod_check_dig= est > MS> --prefix=3D/usr/local > MS> --with-includes=3D/usr/local/include/mysql:/usr/include/openssl > MS> --with-libraries=3D/usr/local/lib/mysql > > It might be that it is not proftpd but other application that cause the l= eak. > Anyway, to check if it is proftpd, look in its logs for entries like thes= e: > > Entering Passive Mode (192,168,0,213,241,70). > FTP session closed. > > Convert the last two numbers to port (241*256+70) and check by netstat if= you > still have this connection. If you have, then it is likely this is the sa= me > situation as in my case and the proftpd is a problem. Upgrade to 1.3.1 fr= om > ports then. > > If proftpd is ok, look for other applications. Search for connections rep= orted > by netstat as ESTABLISHED but not displayed by sockstat utility. You coul= d run > something like this: > > netstat -an | grep ESTABL | > while read b l a local remote state; do > echo -n "$local $remote: " > sockstat | > sed -e 's/:/./g' | > grep -c "$local *$remote" > done > > Look for sockets with 0 count. These are suspicious ones. Observe these > sockets by netstat and try to figure out what application they could belo= ng > and dig in that direction. > > -- > Mikolaj Golub > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " -- Mark Saad Managed UNIX Support DataPipe Managed Global IT Services msaad@datapipe.com 1.201.792.4847 (international) 1.888.749.5821 (toll free) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments This message may contain confidential or privileged information. If you ar= e not the intended recipient, please advise us immediately and delete this = message. See http://www.datapipe.com/emaildisclaimer.aspx for further info= rmation on confidentiality and the risks of non-secure electronic communica= tion. If you cannot access these links, please notify us by reply message a= nd we will send the contents to you. From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 21:33:09 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8EEF0106564A for ; Wed, 14 May 2008 21:33:09 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (five.mired.org [66.92.153.75]) by mx1.freebsd.org (Postfix) with ESMTP id 4913C8FC1E for ; Wed, 14 May 2008 21:33:09 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 55615 invoked by uid 1001); 14 May 2008 17:33:36 -0400 Received: from bhuda.mired.org (bhuda [192.168.195.1]) by bhuda (tmda-ofmipd) with ESMTP; Wed, 14 May 2008 17:33:35 -0400 Date: Wed, 14 May 2008 17:33:34 -0400 To: Peter Jeremy Message-ID: <20080514173334.11bbae5c@bhuda.mired.org> In-Reply-To: <20080514194529.GB64804@server.vk2pj.dyndns.org> References: <15336578.20080512123806@mail.ru> <200805121153.00809.jonathan+freebsd-hackers@hst.org.za> <1663320218.20080512223531@mail.ru> <20080512152430.3720683e@mbook.local> <2117635718.20080513154406@mail.ru> <20080513121452.GA70860@eos.sc1.parodius.com> <20080513154137.GA28842@pix.net> <482A02CD.7040308@mansionfamily.plus.com> <20080514071728.GP64804@server.vk2pj.dyndns.org> <20080514102410.639f16ef@mbook.local> <20080514194529.GB64804@server.vk2pj.dyndns.org> Organization: Meyer Consulting X-Mailer: Claws Mail 3.4.0 (GTK+ 2.12.9; amd64-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Delivery-Agent: TMDA/1.1.12 (Macallan) From: Mike Meyer Cc: freebsd-hackers@freebsd.org Subject: Re: BDB corrupt X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 21:33:09 -0000 On Thu, 15 May 2008 05:45:29 +1000 Peter Jeremy wrote: > On 2008-May-14 10:24:10 -0400, Mike Meyer wrote: > >Just out of curiosity - there seems to be an unspoken assumption that > >the ports system can only use tools that are part of the base > >system. > There have been suggestions that the ports/package infrastructure > (pkg_* tools, portsnap etc) be unbundled from the base OS. The > difficulty comes when you want to upgrade those components. I know, > from experience, that portugrading portupgrade or ruby usually fails > as the running portupgrade unexpectedly trips over changed bits of > itself. Having the ports system depend on packages that are maintained via the ports system does make things fairly complex. On the other hand, unbundling can be done. OS X has N (for some N > 1) ports systems that run outside the base system. Notably, they don't treat the code for manipulating the ports system as a port, and provide distinct commands for updating an installed port, the ports, and the port system software. However, I don't know that that's necessary for what I asked. > > But this is clearly false - the ports system currently > >includes a couple of directories full of software that's not in the > >base system. > There is a directory full of Makefile includes and another directory > full of optional tools but pkg_* sits in the base system. What are > you alluding to here. I was thinking of those two directories. I wasn't thinking about the pkg_* tools, because I pretty much never use them. > >Adding compiled code to those tools would mean that installing the > >ports system gets a bit more complex - you have to run "make install" > >after extracting the tarball. Is that so bad it's not going to happen? > The problem is not the initial install so much as managing packages > and upgrades. I see no problem with having the ports/package > infrastructure be part of the ports system as long as: > a) A user can install/uninstall/audid (and preferably upgrade) > packages without needing to compile anything > b) The ports system knows how to upgrade itself without tripping over > itself in the process. You could do what I suggested by adding the desired db library to /usr/src/usr.sbin/pkg_install, and linking it statically into the pkg* tools? Yeah, it's not really pretty, but is it any worse than having a BDB in the base system that we recommend be avoided? http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 21:47:54 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE9C51065676 for ; Wed, 14 May 2008 21:47:54 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from hosted.kievnet.com (hosted.kievnet.com [193.138.144.10]) by mx1.freebsd.org (Postfix) with ESMTP id 7938D8FC16 for ; Wed, 14 May 2008 21:47:54 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost ([127.0.0.1] helo=edge.pp.kiev.ua) by hosted.kievnet.com with esmtpa (Exim 4.62) (envelope-from ) id 1JwOpF-000167-BY for freebsd-hackers@freebsd.org; Thu, 15 May 2008 00:47:53 +0300 Message-ID: <482B5E08.9000309@icyb.net.ua> Date: Thu, 15 May 2008 00:47:52 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.12 (X11/20080320) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: using git for freebsd development X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 21:47:54 -0000 I finally found some time and inspiration to write about some things I did about a month ago. I decided to use GIT for local FreeBSD development and I looked for the tools to do initial conversion of FreeBSD CVS src repository to GIT (I wanted to get as much history as possible) and also to do one-way sync-ing from CVS to GIT for subsequent updates. Please note that I just wanted to achieve my goal, I didn't attempt to do objective comparisons, benchmarking, etc. So whatever performance data I give below are very imprecise. The following can be considered as a followup to the excellent FreeBSD/GIT wiki page: http://wiki.freebsd.org/GitConversion So my first task was to do the initial conversion. My research showed that there were the following most popular options: git-cvsimport which is a part of GIT suite parsecvs http://gitweb.freedesktop.org/?p=users/keithp/parsecvs.git fromcvs/togit http://www.selenic.com/mercurial/wiki/index.cgi/fromcvs tailor http://progetti.arstecnica.it/tailor/browser/README.rst All of the tools either required source CVS repository to be available locally or worked much faster in that case, so the first thing to do was to get src-all from my local cvsup mirror. Easy. The first tool I tried was cvsimport because it came with git. It failed. After working for a short while it went into infinite loop on some file, it first complained that version X is before Y on branch Z and then that version Y is before X on the same branch and so on and on. Moving away that file didn't help as there were more troublesome ones. Maybe it had to do with repocopying. Next one was parsecvs. I think that this is the best one for initial import. It worked for about 6 hours on a very old machine: 512MB RAM, 450 MHz Pentium III. Resulting GIT repo took about 8G of space. Subsequent git repack took about 12 hours and reduced the size to ~500M. Quite nice. I should note that during the whole process of conversion parsecvs did not use more than 300M of RAM, this is by far the most conservative of all the tools that I tried (and that worked). There were some warning messages during conversion. Unfortunately parsecvs does not provide any option for keyword handling control and it doesn't expand any keywords. There are reasons to prefer this behavior, but personally I would prefer them to be expanded. I think that this is something that should be very easy to tweak in parsecvs source code. Also, quite unfortunately, parsecvs can only do full repository conversion and doesn't support incremental import. Because of the above, although I already had a converted git FreeBSD repo, I decided to give a try to some other tools - thinking that maybe using the same tool for both tasks would be somehow better. Thus I tried fromcvs/togit. It required me to install couple of ruby packages available via ports and two custom ones - rcsparse and fromcvs. It was quite easy to setup and run. This time I executed conversion on a modern system with Athlon XP 4800+ two core processor and 2GB of RAM. And that was needed - fromcvs worked for about two hours, peak memory usage was around 1.5G. There were some warning messages during conversion. Unfortunately, detailed examination showed that there were some issues with the conversion. Some files that were never changed on some branches in CVS were not to be found on the corresponding GIT branches. What's strange is that when I tried to convert only sys/ subdirectory everything went very well, no issues. Only on the complete src repository this problem did happen. Author of fromcvs (Simon 'corecode' Schubert) is aware of the issue and encountered it himself, so I hope it will be resolved soon. But so far no go. Then I decided to try tailor. I must admit that I had some difficulties understanding its documentation and that's probably the cause of what happened next. I provided what I thought were good options to tailor and it generated its config file. Then I executed it with the config, it worked for about two hours being the most resource hungry of everything I tried - using swap on the mentioned 2GB machine. Then it produced some error that looked like a complaint about configuration problem and then I gave up. Summary: only parsecvs worked good enough for me. Part two, doing incremental updates. I updated my copy of FreeBSD CVS repository with cvsup and proceeded. BTW, csup supported only checkout mode, so it could not be used instead of cvsup. By tradition I tried git-cvsimport first. It went into infinite loop again (maybe not infinite, but too long for me). This time it didn't produce any errors, just consumed 100% CPU, didn't make any system calls at all (ktrace to witness). I waited for about 3 hours (on the modern machine). parsecvs, as I said before, doesn't do incremental imports. tailor, on it I gave up. So I finally tried fromcvs and it worked, and it worked fast and it worked good. At least, so far I do not see any issues in incremental updates that it performs. So my conclusion is that at this time parsecvs is the best tool for initial import and fromcvs is the best tool for incremental imports. One small quirk is that parsecvs imported keywords unexpanded, but fromcvs expands them in incremental updates. Another small quirk: couple of commit messages in CVS contain extended Latin symbols from ISO8859-1. It seems that parsecvs copied them as is to GIT log history, I think they should have been converted to UTF-8. E.g. "Hörnquist Ã…strand" in history of Makefile.inc1 As a concluding word: I decided to clone the converted repository and to create my topic and "integration" and other branches in the cloned repository. Some of the branches are tracking branches, so that it is easy, for example, to synchronize my RELENG_7-specific changes with what is going on in CVS. So I first get CVS updates with cvsup. Then do fromcvs incremental import into "pristine" GIT repository. And then do 'git pull' into the working GIT repo. Hope that this will be interesting and/or useful to the community. -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 22:18:08 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF8721065670 for ; Wed, 14 May 2008 22:18:08 +0000 (UTC) (envelope-from murray@stokely.org) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.170]) by mx1.freebsd.org (Postfix) with ESMTP id 621D38FC16 for ; Wed, 14 May 2008 22:18:08 +0000 (UTC) (envelope-from murray@stokely.org) Received: by ug-out-1314.google.com with SMTP id q2so1261958uge.37 for ; Wed, 14 May 2008 15:18:07 -0700 (PDT) Received: by 10.66.237.14 with SMTP id k14mr9241346ugh.72.1210803486592; Wed, 14 May 2008 15:18:06 -0700 (PDT) Received: by 10.66.254.18 with HTTP; Wed, 14 May 2008 15:18:06 -0700 (PDT) Message-ID: <2a7894eb0805141518k1c50be3yf517afc3a159b11e@mail.gmail.com> Date: Wed, 14 May 2008 15:18:06 -0700 From: "Murray Stokely" Sender: murray@stokely.org To: "John Timony" In-Reply-To: <163189.78306.qm@web46306.mail.sp1.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <163189.78306.qm@web46306.mail.sp1.yahoo.com> X-Google-Sender-Auth: 58771245d61864a1 Cc: freebsd-hackers@freebsd.org Subject: Re: A prob to Installing FreeBSD7.0 in VMWare 5.0 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 22:18:08 -0000 On Wed, May 14, 2008 at 8:21 AM, John Timony wrote: > BlankBODY { MARGIN-TOP: 25px; FONT-SIZE: 10pt; MARGIN-LEFT: 25px; COLOR: #000000; FONT-FAMILY: Arial, Helvetica}P.msoNormal { MARGIN-TOP: 0px; FONT-SIZE: 10pt; MARGIN-LEFT: 0px; COLOR: #ffffcc; FONT-FAMILY: Helvetica, "Times New Roman"}LI.msoNormal { MARGIN-TOP: 0px; FONT-SIZE: 10pt; MARGIN-LEFT: 0px; COLOR: #ffffcc; FONT-FAMILY: Helvetica, "Times New Roman"}Hi,I use the CD-ROM to install FreeBSD7.O in VMWare 5.0,But when i start the Virtual Mathine,I get that "elf32_loadimage failed"... > > By the way,,,I use a FreeBSD7.0.iso mounted in CD-ROM... Are you installing a 32-bit (i386) FreeBSD 7.0 iso on a 32-bit virtual machine? Or an amd64 ISO on a 64-bit virtual machine? I would make sure you are using the right ISO for the virtual machine setup you are using. VMWare Fusion on a Mac at least purports to support both 32bit and 64bit FreeBSD. I had no problems installing Ivan Voras's latest FreeBSD 7.0 + finstall on a 32bit virtual machine last night. - Murray From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 23:39:39 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DDD6E106566B for ; Wed, 14 May 2008 23:39:39 +0000 (UTC) (envelope-from edelkind-freebsd-hackers@episec.com) Received: from episec.com (episec.com [69.55.237.141]) by mx1.freebsd.org (Postfix) with SMTP id AD9708FC14 for ; Wed, 14 May 2008 23:39:39 +0000 (UTC) (envelope-from edelkind-freebsd-hackers@episec.com) Received: (qmail 54832 invoked by uid 1024); 14 May 2008 23:12:58 -0000 Date: Wed, 14 May 2008 19:12:58 -0400 From: ari edelkind To: freebsd-hackers@freebsd.org Message-ID: <20080514231258.GB79355@episec.com> Mail-Followup-To: ari edelkind , freebsd-hackers@freebsd.org References: <482B5E08.9000309@icyb.net.ua> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <482B5E08.9000309@icyb.net.ua> Subject: Re: using git for freebsd development X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 23:39:39 -0000 avg@icyb.net.ua wrote: > The following can be considered as a followup to the excellent > FreeBSD/GIT wiki page: > http://wiki.freebsd.org/GitConversion > [...] > All of the tools either required source CVS repository to be available > locally or worked much faster in that case, so the first thing to do was > to get src-all from my local cvsup mirror. Easy. Everyone who tries converting freebsd's cvs repository to joe-random-RCS attempts to import the entire source tree as a single project. Honestly, it covers too broad a spectrum. Separate these projects into the cvsup collections: src-sys, src-bin, src-lib, etc.. You won't be able to simply tag a single branch into a full system release without a wrapper script to handle your collections, but that's a small price to pay for the added robustness, separation of privilege, and smaller-scale potential for conflict. In fact, even src-bin may be too broad, and it may make sense to have separate projects within the collection hierarchy. This would, at least, make reparenting projects (say, from src-bin to src-usrbin) easier. ari From owner-freebsd-hackers@FreeBSD.ORG Thu May 15 02:21:33 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C109106564A for ; Thu, 15 May 2008 02:21:33 +0000 (UTC) (envelope-from sbruno@miralink.com) Received: from plato.miralink.com (mail.miralink.com [70.103.185.20]) by mx1.freebsd.org (Postfix) with ESMTP id 1CCCB8FC1D for ; Thu, 15 May 2008 02:21:33 +0000 (UTC) (envelope-from sbruno@miralink.com) Received: from localhost (localhost.localdomain [127.0.0.1]) by plato.miralink.com (Postfix) with ESMTP id 8554F1C809B for ; Wed, 14 May 2008 18:50:46 -0700 (PDT) Received: from plato.miralink.com ([127.0.0.1]) by localhost (plato.miralink.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 24429-03 for ; Wed, 14 May 2008 18:50:46 -0700 (PDT) Received: from [10.47.1.126] (vpn.office.miralink.com [10.0.0.5]) by plato.miralink.com (Postfix) with ESMTP id 0EB451C8099 for ; Wed, 14 May 2008 18:50:45 -0700 (PDT) Message-ID: <482B96F5.8030800@miralink.com> Date: Wed, 14 May 2008 18:50:45 -0700 From: Sean Bruno User-Agent: Thunderbird 2.0.0.14 (X11/20080501) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DSPAM-Result: Innocent X-DSPAM-Processed: Wed May 14 18:50:46 2008 X-DSPAM-Confidence: 0.7875 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 482b96f6302181763113579 X-DSPAM-Factors: 27, X-Virus-Scanned: amavisd-new at X-Spam-Status: No, score=-4.493 tagged_above=-10 required=6.6 autolearn=ham tests=[ALL_TRUSTED=-1.8, AWL=0.006, BAYES_00=-2.599, DSPAM_HAM=-0.1] X-Spam-Score: -4.493 X-Spam-Level: Subject: USB Install Method X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 02:21:33 -0000 Does anyone have a pointer to a FreeBSD 7 install method for those of us without CD/DVD Rom drives? The easiest way for me would be to boot off of a USB stick and then install across the network, but I'm open to suggestions. Sean From owner-freebsd-hackers@FreeBSD.ORG Thu May 15 06:07:12 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1616F1065676 for ; Thu, 15 May 2008 06:07:12 +0000 (UTC) (envelope-from jamestoy04@gmail.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.30]) by mx1.freebsd.org (Postfix) with ESMTP id C29A88FC1B for ; Thu, 15 May 2008 06:07:11 +0000 (UTC) (envelope-from jamestoy04@gmail.com) Received: by yw-out-2324.google.com with SMTP id 9so163220ywe.13 for ; Wed, 14 May 2008 23:07:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:references:x-google-sender-auth; bh=Tr+nokVe/zjxe9SbKsC8NriwYdGqgGzgSyswO9YKo90=; b=KpX8epA9EmWKP0332T1WQBJBie5mlwgZexgVpDjgaMA9BnhxKxpUaKkkwipQKfzyYqCi5u6Ok1tl6cAsBxBV+UM8YYkJ5phM09UD+UF5IUJy44iI9wlGL9Lpikmh9dSas8ENe0Rv00kpcCRdiL0gmF29oWYrkRwbpylF1oo+jnk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:references:x-google-sender-auth; b=KAcEw2yKRNJ3C0se7/EcgIjEGTS7YwBPlKXlfyWmIv6wYvuO1vBHIx23o2MhX2mSF2pk/vhisL0l5r+uVZlYXHA5KcwIUx0c7iRnhTukDBlkwf5jMB0ozUBTY9F9qVRJy5QpRG8D49ERxVkCPB9caX4G8eOn6xEqNNRAzRXbTtk= Received: by 10.150.69.41 with SMTP id r41mr1913583yba.51.1210830014981; Wed, 14 May 2008 22:40:14 -0700 (PDT) Received: by 10.151.41.12 with HTTP; Wed, 14 May 2008 22:40:14 -0700 (PDT) Message-ID: <7bb8f98f0805142240x344bad54s7ef671b4d33832c2@mail.gmail.com> Date: Thu, 15 May 2008 01:40:14 -0400 From: "james toy" Sender: jamestoy04@gmail.com To: "Sean Bruno" In-Reply-To: <482B96F5.8030800@miralink.com> MIME-Version: 1.0 References: <482B96F5.8030800@miralink.com> X-Google-Sender-Auth: ad0daeb9b2fa0736 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-hackers@freebsd.org Subject: Re: USB Install Method X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 06:07:12 -0000 Sean, How about a net-boot? That would be my next logical suggestion -- as far as I'm concerned its relatively hard to install via USB key at least i'm not aware of any current ways to do so. respectfully, jt On Wed, May 14, 2008 at 9:50 PM, Sean Bruno wrote: > Does anyone have a pointer to a FreeBSD 7 install method for those of us > without CD/DVD Rom drives? > > The easiest way for me would be to boot off of a USB stick and then install > across the network, but I'm open to suggestions. > > Sean > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Thu May 15 08:52:34 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78E481065673 for ; Thu, 15 May 2008 08:52:34 +0000 (UTC) (envelope-from doconnor@gsoft.com.au) Received: from cain.gsoft.com.au (cain.gsoft.com.au [203.31.81.10]) by mx1.freebsd.org (Postfix) with ESMTP id B499B8FC19 for ; Thu, 15 May 2008 08:52:33 +0000 (UTC) (envelope-from doconnor@gsoft.com.au) Received: from inchoate.gsoft.com.au (ppp121-45-62-238.lns11.adl2.internode.on.net [121.45.62.238]) (authenticated bits=0) by cain.gsoft.com.au (8.13.8/8.13.8) with ESMTP id m4F8qUF6049615 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Thu, 15 May 2008 18:22:30 +0930 (CST) (envelope-from doconnor@gsoft.com.au) From: "Daniel O'Connor" To: freebsd-hackers@freebsd.org Date: Thu, 15 May 2008 18:22:41 +0930 User-Agent: KMail/1.9.7 References: <482B96F5.8030800@miralink.com> In-Reply-To: <482B96F5.8030800@miralink.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart2457858.rKG57SD6G7"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200805151822.43236.doconnor@gsoft.com.au> X-Spam-Score: -2.212 () BAYES_00,RDNS_DYNAMIC X-Scanned-By: MIMEDefang 2.63 on 203.31.81.10 Cc: Sean Bruno Subject: Re: USB Install Method X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 08:52:34 -0000 --nextPart2457858.rKG57SD6G7 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Thu, 15 May 2008, Sean Bruno wrote: > Does anyone have a pointer to a FreeBSD 7 install method for those of > us without CD/DVD Rom drives? > > The easiest way for me would be to boot off of a USB stick and then > install across the network, but I'm open to suggestions. The simplest way would be to get a USB CDROM drive. Note that it is likely you will need a version of FreeBSD released after=20 this commit.. http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/boot/i386/btx/btx/btx.S?rev= =3D1.46 Unfortunately as of today it will only be in a -current snapshot :( eg ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/200805 You might be lucky and it will work but if not you will find the loader=20 crashes (eg spins really fast writing register dumps to the screen..) It is fairly straight forward to create a USB key with FreeBSD on it but=20 it requires a FreeBSD system to create it.. http://typo.submonkey.net/articles/2006/4/13/installing-freebsd-on-usb-stic= k-episode-2 (I haven't actually tried this but it looks OK - I have done it with=20 =46reesBIE though) =2D-=20 Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C --nextPart2457858.rKG57SD6G7 Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.8 (FreeBSD) iD8DBQBIK/nb5ZPcIHs/zowRAjNCAJ9TP0j7Ea5iVkvG6ZW3JsWCMbCalgCfeSuh hdb2HCxYZnQrR3H7mzsQu9A= =JeGl -----END PGP SIGNATURE----- --nextPart2457858.rKG57SD6G7-- From owner-freebsd-hackers@FreeBSD.ORG Thu May 15 10:20:36 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 746131065671 for ; Thu, 15 May 2008 10:20:36 +0000 (UTC) (envelope-from teemu@rinta-aho.org) Received: from smtp.piuha.net (p130.piuha.net [IPv6:2001:14b8:400::130]) by mx1.freebsd.org (Postfix) with ESMTP id 17C478FC30 for ; Thu, 15 May 2008 10:20:35 +0000 (UTC) (envelope-from teemu@rinta-aho.org) Received: from smtp.piuha.net (localhost [127.0.0.1]) by smtp.piuha.net (Postfix) with ESMTP id 2BF2C1987F3 for ; Thu, 15 May 2008 13:20:33 +0300 (EEST) Received: from [127.0.0.1] (unknown [IPv6:2001:14b8:400::130]) by smtp.piuha.net (Postfix) with ESMTP id EC97B198665 for ; Thu, 15 May 2008 13:20:32 +0300 (EEST) Message-ID: <482C0E70.4020305@rinta-aho.org> Date: Thu, 15 May 2008 13:20:32 +0300 From: Teemu Rinta-aho User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org X-Enigmail-Version: 0.95.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP Subject: copy-on-write anonymous memory? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 10:20:36 -0000 Hi all, is it possible to create a memory object that represents anonymous memory pages *and* is copy-on-write? I have this code in a kernel module: object = vm_object_allocate(OBJT_DEFAULT, 1); result = vm_map_find(vmmap_proc, object, 0, &addr, len, TRUE, VM_PROT_ALL, VM_PROT_ALL, MAP_COPY_ON_WRITE); Then I pass the addr to the user space, but when I write to the addr, I see no shadow objects created, i.e. the changes are written to the original memory pages no matter if I have the map entry set as copy-on-write or not... I am assuming a write fault would create a new page and hang it to a shadow object thus leaving the original memory untouched. I'd appreciate any kind of help here. Best regards, Teemu Rinta-aho From owner-freebsd-hackers@FreeBSD.ORG Thu May 15 11:18:56 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A13411065676 for ; Thu, 15 May 2008 11:18:56 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay03.kiev.sovam.com (relay03.kiev.sovam.com [62.64.120.201]) by mx1.freebsd.org (Postfix) with ESMTP id 3CA688FC1D for ; Thu, 15 May 2008 11:18:56 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=skuns.kiev.zoral.com.ua) by relay03.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1JwbU5-000D2s-Rs for freebsd-hackers@freebsd.org; Thu, 15 May 2008 14:18:54 +0300 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by skuns.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4FBIocL070399 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 May 2008 14:18:50 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4FBIlg3099391; Thu, 15 May 2008 14:18:48 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m4FBIlJZ099390; Thu, 15 May 2008 14:18:47 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 15 May 2008 14:18:47 +0300 From: Kostik Belousov To: Teemu Rinta-aho Message-ID: <20080515111847.GX18958@deviant.kiev.zoral.com.ua> References: <482C0E70.4020305@rinta-aho.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Ny9pOG5CW+LNoymi" Content-Disposition: inline In-Reply-To: <482C0E70.4020305@rinta-aho.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.4 X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on skuns.kiev.zoral.com.ua X-Scanner-Signature: 5e2df25549de819b25d1bfd51e9f52eb X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Header: Not Detected X-SpamTest-Info: Profiles 2822 [May 13 2008] X-SpamTest-Info: helo_type=3 X-SpamTest-Method: none X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0278], KAS30/Release Cc: freebsd-hackers@freebsd.org Subject: Re: copy-on-write anonymous memory? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 11:18:56 -0000 --Ny9pOG5CW+LNoymi Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, May 15, 2008 at 01:20:32PM +0300, Teemu Rinta-aho wrote: > Hi all, >=20 > is it possible to create a memory object that represents > anonymous memory pages *and* is copy-on-write? >=20 > I have this code in a kernel module: >=20 > object =3D vm_object_allocate(OBJT_DEFAULT, 1); >=20 > result =3D vm_map_find(vmmap_proc, > object, > 0, > &addr, > len, > TRUE, > VM_PROT_ALL, > VM_PROT_ALL, > MAP_COPY_ON_WRITE); >=20 > Then I pass the addr to the user space, but when > I write to the addr, I see no shadow objects created, > i.e. the changes are written to the original memory > pages no matter if I have the map entry set as > copy-on-write or not... I am assuming a write fault would > create a new page and hang it to a shadow object thus > leaving the original memory untouched. >=20 > I'd appreciate any kind of help here. I cannot get a complete handle on your problem without full code, but I guess that you have only one reference to the backing object through the vm map entry. If this is the case, then the shadow copying actually does not make sense since no other users of the object need it. Look at the vm_object_shadow(), check at the start of the function. --Ny9pOG5CW+LNoymi Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkgsHBcACgkQC3+MBN1Mb4iIgACgoPCEMrYPXQikN3G2neX51+jR txUAn0JJMcoPypfEXwium413Sp7Ratpt =YxN0 -----END PGP SIGNATURE----- --Ny9pOG5CW+LNoymi-- From owner-freebsd-hackers@FreeBSD.ORG Thu May 15 12:33:02 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D37EF1065673 for ; Thu, 15 May 2008 12:33:02 +0000 (UTC) (envelope-from teemu@rinta-aho.org) Received: from smtp.piuha.net (p130.piuha.net [IPv6:2001:14b8:400::130]) by mx1.freebsd.org (Postfix) with ESMTP id 5E4758FC18 for ; Thu, 15 May 2008 12:33:02 +0000 (UTC) (envelope-from teemu@rinta-aho.org) Received: from smtp.piuha.net (localhost [127.0.0.1]) by smtp.piuha.net (Postfix) with ESMTP id B85C21987F0; Thu, 15 May 2008 15:33:00 +0300 (EEST) Received: from [127.0.0.1] (unknown [IPv6:2001:14b8:400::130]) by smtp.piuha.net (Postfix) with ESMTP id 80AF0198665; Thu, 15 May 2008 15:33:00 +0300 (EEST) Message-ID: <482C2D7C.5070703@rinta-aho.org> Date: Thu, 15 May 2008 15:33:00 +0300 From: Teemu Rinta-aho User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Kostik Belousov References: <482C0E70.4020305@rinta-aho.org> <20080515111847.GX18958@deviant.kiev.zoral.com.ua> In-Reply-To: <20080515111847.GX18958@deviant.kiev.zoral.com.ua> X-Enigmail-Version: 0.95.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP Cc: freebsd-hackers@freebsd.org Subject: Re: copy-on-write anonymous memory? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 12:33:02 -0000 Kostik Belousov wrote: > On Thu, May 15, 2008 at 01:20:32PM +0300, Teemu Rinta-aho wrote: >> Hi all, >> >> is it possible to create a memory object that represents >> anonymous memory pages *and* is copy-on-write? >> >> I have this code in a kernel module: >> >> object = vm_object_allocate(OBJT_DEFAULT, 1); >> >> result = vm_map_find(vmmap_proc, >> object, >> 0, >> &addr, >> len, >> TRUE, >> VM_PROT_ALL, >> VM_PROT_ALL, >> MAP_COPY_ON_WRITE); >> >> Then I pass the addr to the user space, but when >> I write to the addr, I see no shadow objects created, >> i.e. the changes are written to the original memory >> pages no matter if I have the map entry set as >> copy-on-write or not... I am assuming a write fault would >> create a new page and hang it to a shadow object thus >> leaving the original memory untouched. >> >> I'd appreciate any kind of help here. > > I cannot get a complete handle on your problem without full code, but > I guess that you have only one reference to the backing object through > the vm map entry. If this is the case, then the shadow copying actually > does not make sense since no other users of the object need it. > > Look at the vm_object_shadow(), check at the start of the function. Hi Kostik, thanks, that was a good point. Adding a reference helps! :-) Thanks for your quick reply, Teemu From owner-freebsd-hackers@FreeBSD.ORG Thu May 15 14:51:32 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B6F401065672 for ; Thu, 15 May 2008 14:51:32 +0000 (UTC) (envelope-from teemu@rinta-aho.org) Received: from smtp.piuha.net (p130.piuha.net [IPv6:2001:14b8:400::130]) by mx1.freebsd.org (Postfix) with ESMTP id 7234E8FC16 for ; Thu, 15 May 2008 14:51:32 +0000 (UTC) (envelope-from teemu@rinta-aho.org) Received: from smtp.piuha.net (localhost [127.0.0.1]) by smtp.piuha.net (Postfix) with ESMTP id E55C41987F5; Thu, 15 May 2008 17:51:30 +0300 (EEST) Received: from [127.0.0.1] (unknown [IPv6:2001:14b8:400::130]) by smtp.piuha.net (Postfix) with ESMTP id B05D5198713; Thu, 15 May 2008 17:51:30 +0300 (EEST) Message-ID: <482C4DF3.8030709@rinta-aho.org> Date: Thu, 15 May 2008 17:51:31 +0300 From: Teemu Rinta-aho User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Kostik Belousov References: <482C0E70.4020305@rinta-aho.org> <20080515111847.GX18958@deviant.kiev.zoral.com.ua> <482C2D7C.5070703@rinta-aho.org> In-Reply-To: <482C2D7C.5070703@rinta-aho.org> X-Enigmail-Version: 0.95.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP Cc: freebsd-hackers@freebsd.org Subject: Re: copy-on-write anonymous memory? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 14:51:32 -0000 Teemu Rinta-aho wrote: > Kostik Belousov wrote: >> On Thu, May 15, 2008 at 01:20:32PM +0300, Teemu Rinta-aho wrote: >>> Hi all, >>> >>> is it possible to create a memory object that represents >>> anonymous memory pages *and* is copy-on-write? >>> >>> I have this code in a kernel module: >>> >>> object = vm_object_allocate(OBJT_DEFAULT, 1); >>> >>> result = vm_map_find(vmmap_proc, >>> object, >>> 0, >>> &addr, >>> len, >>> TRUE, >>> VM_PROT_ALL, >>> VM_PROT_ALL, >>> MAP_COPY_ON_WRITE); >>> >>> Then I pass the addr to the user space, but when >>> I write to the addr, I see no shadow objects created, >>> i.e. the changes are written to the original memory >>> pages no matter if I have the map entry set as >>> copy-on-write or not... I am assuming a write fault would >>> create a new page and hang it to a shadow object thus >>> leaving the original memory untouched. >>> >>> I'd appreciate any kind of help here. >> I cannot get a complete handle on your problem without full code, but >> I guess that you have only one reference to the backing object through >> the vm map entry. If this is the case, then the shadow copying actually >> does not make sense since no other users of the object need it. >> >> Look at the vm_object_shadow(), check at the start of the function. > > Hi Kostik, > > thanks, that was a good point. Adding a reference helps! :-) > > Thanks for your quick reply, > > Teemu New problems... Let me first describe what I try to do: I have created a kernel module that stores references to memory objects. I.e. when a process makes a syscall to the module, it will create a snapshot of the memory area, and after that the writes from the process to that memory area should create a shadow object. The next syscall should again store a pointer to the current topmost shadow object and then the next write creates yet another shadow object. Etc... When the snapshots are removed, the shadow chains may collapse normally. That's what I have assumed. Here's an illustration of what I want (first syscall OK, second one not): * * Legend: U/u = userspace K/k = kernel * * U:vm_map_entry_u -> object * * || * SYSCALL * || * \/ * * U:vm_map_entry_u -> object_shadow -> object * / * K:vm_map_entry_k ---------------- * * || * SYSCALL * || * \/ * * U:vm_map_entry_u -> object_shadow -> object_shadow -> object * / / * K:vm_map_entry_k ---------------- / * K:vm_map_entry_k -------------------------------- Now, the problem is that the first snapshot works as it should. However, the second one doesn't, and the write goes to the one and same shadow object, even if I restore MAP_ENTRY_COW and MAP_ENTRY_NEEDS_COPY manually in my handler function which is storing the snapshot. Any ideas? Teemu From owner-freebsd-hackers@FreeBSD.ORG Thu May 15 18:19:06 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A2815106564A for ; Thu, 15 May 2008 18:19:06 +0000 (UTC) (envelope-from jhs@berklix.org) Received: from flat.berklix.org (flat.berklix.org [83.236.223.115]) by mx1.freebsd.org (Postfix) with ESMTP id 16EA28FC19 for ; Thu, 15 May 2008 18:19:05 +0000 (UTC) (envelope-from jhs@berklix.org) Received: from js.berklix.net (p549A4901.dip.t-dialin.net [84.154.73.1]) (authenticated bits=0) by flat.berklix.org (8.13.8/8.13.8) with ESMTP id m4FIJ36q017895; Thu, 15 May 2008 20:19:04 +0200 (CEST) (envelope-from jhs@berklix.org) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by js.berklix.net (8.13.8/8.13.8) with ESMTP id m4FIOPvj007745; Thu, 15 May 2008 20:24:25 +0200 (CEST) (envelope-from jhs@berklix.org) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.13.8/8.13.8) with ESMTP id m4FIOFBm000937; Thu, 15 May 2008 20:24:25 +0200 (CEST) (envelope-from jhs@fire.js.berklix.net) Message-Id: <200805151824.m4FIOFBm000937@fire.js.berklix.net> To: freebsd-hackers@freebsd.org From: "Julian Stacey" Organization: http://berklix.com BSD Linux Unix Consultancy, Munich Germany. User-agent: EXMH on FreeBSD http://berklix.com/free/ X-URL: http://berklix.com/~jhs/cv/ Date: Thu, 15 May 2008 20:24:15 +0200 Sender: jhs@berklix.org Cc: Subject: Not seeing enough memory though its there. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 18:19:06 -0000 Hi Hackers cc Gary J, On a Compaq Pressario 2254 with 7.0-Release & Generic kernel, I'm not seeing enough RAM: A 128M DIM reports as 56M, a 256M reports as 96M. Any ideas please ? dmesg FreeBSD 7.0-RELEASE #0: Sun Feb 24 19:59:52 UTC 2008 root@logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD-K6tm w/ multimedia extensions (233.03-MHz 586-class CPU) Origin = "AuthenticAMD" Id = 0x562 Stepping = 2 Features=0x8001bf AMD Features=0x400<> real memory = 100663296 (96 MB) avail memory = 84484096 (80 MB) pnpbios: Bad PnP BIOS data checksum # Dont know why bad With a 128M RAM Without hw.physmem= in /boot/loader.conf dmesg real memory = 58720256 (56 MB) avail memory = 43507712 (41 MB) sysctl -a | grep mem hw.physmem: 45572096 hw.usermem: 33267712 hw.realmem: 58720256 With /boot/loader.conf hw.physmem=134217728 same as above except hw.usermem: 33312768 With a 256 RAM power on Bios auto resizes to about 96M ! Without hw.physmem= dmesg real memory = 100663296 (96 MB) avail memory = 84484096 (80 MB) sysctl -a | grep mem hw.physmem: 87515136 hw.usermem: 74280960 hw.realmem: 100663296 With /boot/loader.conf hw.physmem=268435456 same as above except hw.usermem: 74141696 This Compaq has just 1 RAM slot. The RAM DIMs used: 128M RAM (original far as I know from PC in Germany) Module Paper Label: 128MB Sync 133 MHz CL3; 168 pin DIM, 13.3 x 3 cm, Slots: 1 in middle, + 1 @ 2.45 mm from end with pins 1 & 85 2 sides x 8 chips: Ramos 0131, RC28S0801B1-13AC 256M borrowed from another PC bought in UK (with 2 completely different RAM sources, unlikey both might be accidentaly or deliberately mislabeled (as occasionals occurs re CT mag article on cache chips some years back) Module Paper Label: IRL DEM DP133 064323E PC133 333 520 256MB 0137-2328-062[56] 2 sides x 8 chips: Hynix 132A, HY57V2882DAT-H I did check hw.physmem can set (reduce at least) in 7.0 /boot/loader.conf, on another laptop, after a reboot, from 512M to 256M). Julian -- Julian Stacey: BSDUnixLinux C Prog Admin SysEng Consult Munich www.berklix.com Mail just Ascii plain text. HTML & Base64 text are spam. From owner-freebsd-hackers@FreeBSD.ORG Thu May 15 18:40:59 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5A07A106564A for ; Thu, 15 May 2008 18:40:59 +0000 (UTC) (envelope-from jbucht@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.225]) by mx1.freebsd.org (Postfix) with ESMTP id 2E8E38FC15 for ; Thu, 15 May 2008 18:40:59 +0000 (UTC) (envelope-from jbucht@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so681812rvf.43 for ; Thu, 15 May 2008 11:40:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=Dhy3LDo49BvMo+dxTxx5LTpP1YOmtc/k7dIPdHl6FXs=; b=qQi7sJYtt78sso4yvVv0HuTAXhLU5n7l7DcNCyQR8c5BwOY8NebDsJEwbVLUm3Ph28je1ouD7CdtnoYtrP4SWf1sFwrzDWScBKXPTdzX48B1zxvmlz5h+KaKXnIvGaZ77nZGajBwd/WcJ5ustaddd/eHOiHfwCYjlApeRZdcm4A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=Ci8r2MBqrSs1ymPcO9Yw/EWnuV4RY/X44cpXRniof4sWq+UF0kU6G4mYd2Jgr+yu7tbqiC+mQkkpX9nowGVuFARNfyCKAkOG0QQNaqlr5fn5V0k8pLrCybXm6Qfb0W94NhvzhF707MdQiZDixLWBvBClGd+OwN6ktUq/WWtbLdQ= Received: by 10.141.168.7 with SMTP id v7mr1268201rvo.118.1210875245712; Thu, 15 May 2008 11:14:05 -0700 (PDT) Received: by 10.140.166.16 with HTTP; Thu, 15 May 2008 11:14:04 -0700 (PDT) Message-ID: <947010c30805151114w73ecbb95r52f078fd1748ea69@mail.gmail.com> Date: Thu, 15 May 2008 20:14:04 +0200 From: "Johan Bucht" To: "Sean Bruno" In-Reply-To: <482B96F5.8030800@miralink.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <482B96F5.8030800@miralink.com> Cc: freebsd-hackers@freebsd.org Subject: Re: USB Install Method X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 18:40:59 -0000 On Thu, May 15, 2008 at 3:50 AM, Sean Bruno wrote: > Does anyone have a pointer to a FreeBSD 7 install method for those of us > without CD/DVD Rom drives? > > The easiest way for me would be to boot off of a USB stick and then install > across the network, but I'm open to suggestions. > Try Unetbootin, I used it to install FreeBSD on my fit-PC from a USB stick, went pretty smooth. /Johan From owner-freebsd-hackers@FreeBSD.ORG Thu May 15 18:49:52 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C72CE106566B for ; Thu, 15 May 2008 18:49:52 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from mx0.awanti.com (mx0.awanti.com [91.190.112.18]) by mx1.freebsd.org (Postfix) with ESMTP id 832378FC1D for ; Thu, 15 May 2008 18:49:52 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from pstation (unknown [10.28.4.14]) by mx0.awanti.com (Postfix) with ESMTP id AFA3D4C001; Thu, 15 May 2008 22:49:50 +0400 (MSD) Date: Thu, 15 May 2008 22:52:17 +0400 From: Anthony Pankov X-Mailer: The Bat! (v1.51) Personal X-Priority: 3 (Normal) Message-ID: <21866640.20080515225217@mail.ru> To: Sean Bruno In-Reply-To: <482B96F5.8030800@miralink.com> References: <482B96F5.8030800@miralink.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: USB Install Method X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Anthony Pankov List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 18:49:52 -0000 Try the following http://frenzy.org.ua/en/ http://frenzy.org.ua/en/releases/1.0/doc/doc_en-install2flash.html Thursday, May 15, 2008, 5:50:45 AM, you wrote: SB> Does anyone have a pointer to a FreeBSD 7 install method for those of us SB> without CD/DVD Rom drives? SB> The easiest way for me would be to boot off of a USB stick and then SB> install across the network, but I'm open to suggestions. SB> Sean SB> _______________________________________________ SB> freebsd-hackers@freebsd.org mailing list SB> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers SB> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" -- Best regards, Anthony mailto:ap00@mail.ru From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 08:00:28 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE9761065670 for ; Fri, 16 May 2008 08:00:28 +0000 (UTC) (envelope-from stefan@fafoe.narf.at) Received: from viefep15-int.chello.at (viefep18-int.chello.at [213.46.255.22]) by mx1.freebsd.org (Postfix) with ESMTP id 3A3598FC30 for ; Fri, 16 May 2008 08:00:27 +0000 (UTC) (envelope-from stefan@fafoe.narf.at) Received: from lizard.fafoe.narf.at ([213.47.85.26]) by viefep11-int.chello.at (InterMail vM.7.08.02.02 201-2186-121-104-20070414) with ESMTP id <20080516074453.UEMA2384.viefep11-int.chello.at@lizard.fafoe.narf.at>; Fri, 16 May 2008 09:44:53 +0200 Received: by lizard.fafoe.narf.at (Postfix, from userid 1001) id C4513C3DE; Fri, 16 May 2008 09:44:34 +0200 (CEST) Date: Fri, 16 May 2008 09:44:33 +0200 From: Stefan Farfeleder To: Mike Meyer Message-ID: <20080516074433.GA1032@lizard.fafoe.narf.at> Mail-Followup-To: Mike Meyer , hackers@freebsd.org References: <20080309152712.42752293@bhuda.mired.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080309152712.42752293@bhuda.mired.org> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: hackers@freebsd.org Subject: Re: Why doesn't autoconf like our /bin/sh? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 08:00:29 -0000 On Sun, Mar 09, 2008 at 03:27:12PM -0400, Mike Meyer wrote: > I've stumbled on to an obscure problem with autoconf 2.61, and I'm not > sure quite what to do with it. I've already sent mail to the autoconf > folks, but I'd like to understand what's going on. > > The problem is that, on a FreeBSD system with only /bin/sh and the > ports zsh as installed shells, if you have SHELL set to zsh when > invoking the autoconf-generated configure script, the script produces > a broken Makefile. It doesn't generate an error, it just complains > that: Can you please retry? /bin/sh now supports expanding $LINENO which was often the reason for configure not liking it. Stefan From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 13:43:51 2008 Return-Path: Delivered-To: hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07B7B1065671; Fri, 16 May 2008 13:43:51 +0000 (UTC) (envelope-from so14k@valentine.liquidneon.com) Received: from valentine.liquidneon.com (unknown [IPv6:2001:4830:2407:8000:230:48ff:fe71:c2a2]) by mx1.freebsd.org (Postfix) with ESMTP id 21F118FC1E; Fri, 16 May 2008 13:43:49 +0000 (UTC) (envelope-from so14k@valentine.liquidneon.com) Received: by valentine.liquidneon.com (Postfix, from userid 1018) id B8D478FD8B; Fri, 16 May 2008 07:43:46 -0600 (MDT) Date: Fri, 16 May 2008 07:43:46 -0600 From: Brad Davis To: hackers@FreeBSD.org Message-ID: <20080516134346.GB1844@valentine.liquidneon.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: freebsd-current@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: FreeBSD Status Reports for the First Quarter of 2008 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 13:43:51 -0000 Hi Everyone, The FreeBSD Status Reports for the First Quarter of 2008 are now available at: http://www.freebsd.org/news/status/report-2008-01-2008-03.html Regards, Brad Davis From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 14:11:20 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3607A10656F9 for ; Fri, 16 May 2008 14:11:19 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (five.mired.org [66.92.153.75]) by mx1.freebsd.org (Postfix) with ESMTP id 1B4C78FC15 for ; Fri, 16 May 2008 14:11:18 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 15123 invoked by uid 1001); 16 May 2008 10:11:45 -0400 Received: from bhuda.mired.org (bhuda [192.168.195.1]) by bhuda (tmda-ofmipd) with ESMTP; Fri, 16 May 2008 10:11:45 -0400 Date: Fri, 16 May 2008 10:11:43 -0400 To: Stefan Farfeleder Message-ID: <20080516101143.3545937c@bhuda.mired.org> In-Reply-To: <20080516074433.GA1032@lizard.fafoe.narf.at> References: <20080309152712.42752293@bhuda.mired.org> <20080516074433.GA1032@lizard.fafoe.narf.at> Organization: Meyer Consulting X-Mailer: Claws Mail 3.4.0 (GTK+ 2.12.9; amd64-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Delivery-Agent: TMDA/1.1.12 (Macallan) From: Mike Meyer Cc: hackers@freebsd.org, Mike Meyer Subject: Re: Why doesn't autoconf like our /bin/sh? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 14:11:20 -0000 On Fri, 16 May 2008 09:44:33 +0200 Stefan Farfeleder wrote: > On Sun, Mar 09, 2008 at 03:27:12PM -0400, Mike Meyer wrote: > > I've stumbled on to an obscure problem with autoconf 2.61, and I'm not > > sure quite what to do with it. I've already sent mail to the autoconf > > folks, but I'd like to understand what's going on. > > > > The problem is that, on a FreeBSD system with only /bin/sh and the > > ports zsh as installed shells, if you have SHELL set to zsh when > > invoking the autoconf-generated configure script, the script produces > > a broken Makefile. It doesn't generate an error, it just complains > > that: > > Can you please retry? /bin/sh now supports expanding $LINENO which was > often the reason for configure not liking it. Which branch, and how recently? http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 14:13:32 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 133F5106564A for ; Fri, 16 May 2008 14:13:32 +0000 (UTC) (envelope-from stefan@fafoe.narf.at) Received: from viefep14-int.chello.at (viefep18-int.chello.at [213.46.255.22]) by mx1.freebsd.org (Postfix) with ESMTP id 53C1B8FC1E for ; Fri, 16 May 2008 14:13:30 +0000 (UTC) (envelope-from stefan@fafoe.narf.at) Received: from lizard.fafoe.narf.at ([213.47.85.26]) by viefep14-int.chello.at (InterMail vM.7.08.02.02 201-2186-121-104-20070414) with ESMTP id <20080516141329.UHBI6571.viefep14-int.chello.at@lizard.fafoe.narf.at>; Fri, 16 May 2008 16:13:29 +0200 Received: by lizard.fafoe.narf.at (Postfix, from userid 1001) id 03002C3DE; Fri, 16 May 2008 16:13:07 +0200 (CEST) Date: Fri, 16 May 2008 16:13:07 +0200 From: Stefan Farfeleder To: Mike Meyer Message-ID: <20080516141307.GB1032@lizard.fafoe.narf.at> Mail-Followup-To: Mike Meyer , hackers@freebsd.org References: <20080309152712.42752293@bhuda.mired.org> <20080516074433.GA1032@lizard.fafoe.narf.at> <20080516101143.3545937c@bhuda.mired.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080516101143.3545937c@bhuda.mired.org> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: hackers@freebsd.org Subject: Re: Why doesn't autoconf like our /bin/sh? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 14:13:32 -0000 On Fri, May 16, 2008 at 10:11:43AM -0400, Mike Meyer wrote: > On Fri, 16 May 2008 09:44:33 +0200 > Stefan Farfeleder wrote: > > > On Sun, Mar 09, 2008 at 03:27:12PM -0400, Mike Meyer wrote: > > > I've stumbled on to an obscure problem with autoconf 2.61, and I'm not > > > sure quite what to do with it. I've already sent mail to the autoconf > > > folks, but I'd like to understand what's going on. > > > > > > The problem is that, on a FreeBSD system with only /bin/sh and the > > > ports zsh as installed shells, if you have SHELL set to zsh when > > > invoking the autoconf-generated configure script, the script produces > > > a broken Makefile. It doesn't generate an error, it just complains > > > that: > > > > Can you please retry? /bin/sh now supports expanding $LINENO which was > > often the reason for configure not liking it. > > Which branch, and how recently? HEAD and yesterday. From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 14:37:32 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4ECC1106567F for ; Fri, 16 May 2008 14:37:32 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id 0093F8FC1B for ; Fri, 16 May 2008 14:37:31 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id D91C1744003 for ; Fri, 16 May 2008 17:37:29 +0300 (EEST) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id A8suET7ybQg6 for ; Fri, 16 May 2008 17:37:29 +0300 (EEST) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 84517744001 for ; Fri, 16 May 2008 17:37:29 +0300 (EEST) Message-ID: <482D9C28.8080805@icyb.net.ua> Date: Fri, 16 May 2008 17:37:28 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.12 (X11/20080311) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Subject: freebsd i386 and bios X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 14:37:32 -0000 Does i386 FreeBSD depend for anything important on "traditional" BIOS or will it be able to load and work without it? I mean with something like coreboot. -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 15:08:18 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5580E1065672 for ; Fri, 16 May 2008 15:08:18 +0000 (UTC) (envelope-from teemu@rinta-aho.org) Received: from smtp.piuha.net (p130.piuha.net [IPv6:2001:14b8:400::130]) by mx1.freebsd.org (Postfix) with ESMTP id E8E328FC18 for ; Fri, 16 May 2008 15:08:17 +0000 (UTC) (envelope-from teemu@rinta-aho.org) Received: from smtp.piuha.net (localhost [127.0.0.1]) by smtp.piuha.net (Postfix) with ESMTP id BF8041987F3; Fri, 16 May 2008 18:08:15 +0300 (EEST) Received: from [127.0.0.1] (unknown [IPv6:2001:14b8:400::130]) by smtp.piuha.net (Postfix) with ESMTP id 2DF1E1987F1; Fri, 16 May 2008 18:08:15 +0300 (EEST) Message-ID: <482DA364.7000106@rinta-aho.org> Date: Fri, 16 May 2008 18:08:20 +0300 From: Teemu Rinta-aho User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Mark Tinguely References: <200805161434.m4GEY6N3082155@casselton.net> In-Reply-To: <200805161434.m4GEY6N3082155@casselton.net> X-Enigmail-Version: 0.95.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP Cc: freebsd-hackers@freebsd.org Subject: Re: copy-on-write anonymous memory? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 15:08:18 -0000 Mark Tinguely wrote: > Teemu Rinta-aho wrote (some edits): > >> I have created a kernel module that stores references to memory objects. >> I.e. when a process makes a syscall to the module, it will create a >> snapshot of the memory area, and after that the writes from the process to >> that memory area should create a shadow object. The next syscall should >> again store a pointer to the current topmost shadow object and then the >> next write creates yet another shadow object. Etc... When the snapshots >> are removed, the shadow chains may collapse normally. >> >> Here's an illustration of what I want (first syscall OK, second one not): >> >> * Legend: U/u = userspace K/k = kernel >> * >> * U:vm_map_entry_u -> object >> * || >> * SYSCALL >> * || >> * \/ >> * U:vm_map_entry_u -> object_shadow -> object >> * / >> * K:vm_map_entry_k ---------------- >> * || >> * SYSCALL >> * || >> * \/ >> * U:vm_map_entry_u -> object_shadow -> object_shadow -> object >> * / / >> * K:vm_map_entry_k ---------------- / >> * K:vm_map_entry_k -------------------------------- >> >> Now, the problem is that the first snapshot works as it should. However, >> the second one doesn't, and the write goes to the one and same shadow >> object, even if I restore MAP_ENTRY_COW and MAP_ENTRY_NEEDS_COPY manually >> in my handler function which is storing the snapshot. >> >> Any ideas? > > Usually, a fork() creates the inheritance between parent and child COW memory > space. Start: > vm_map_entry -> object_shadow -> object > > fork(): > vm_map_entry -> object_shadow -\ > |-> (object_shadow*) -> object > vm_map_entry -> object_shadow -/ > > This is slightly different from your description/drawing, in the way changes > are inherited; for example: process 1 is created, process 1 writes page 0. > process 2 is created. process 1 writes p1 (or p0 again). Your > description/drawing implies that process 2 see this change from process 1. > > You are not forking over a COW memory area. Sounds like the syscall will have > manually create the inheritance. You can manually link the object_shadows > the way you want to get the desired inheritance. Process removals should > collapse the shadows automatically. > > Matt Dillion wrote a brief VM description > (http://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/). > The book, "The Design and Implementation of the 4.4BSD Operating System" > is another great reference. > > Mark Tinguely Hi Mark, thanks a lot. I was just about to write to the list that I found the solution, to give future Googler's some help :-) The problem was that of course there is no page fault for every write to the same page. However, I can manually set the COW flag in the map entry, and then call pmap_protect() to set the physical page read only. Then the next write after the syscall will trigger the vm_map_lookup() and the creation of a shadow object. So, for now, this problem is solved. I'm about to head towards new problems now :-) Best regards, Teemu Rinta-aho P.S. Yes I have both the D&I of 4.4BSD and D&I of FreeBSD books, they're quite good and become understandable after browsing the code for a while :-) From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 15:41:07 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7C1B31065672 for ; Fri, 16 May 2008 15:41:07 +0000 (UTC) (envelope-from jille@quis.cx) Received: from smtp3.versatel.nl (smtp3.versatel.nl [62.58.50.90]) by mx1.freebsd.org (Postfix) with ESMTP id E41FD8FC1C for ; Fri, 16 May 2008 15:41:06 +0000 (UTC) (envelope-from jille@quis.cx) Received: (qmail 4760 invoked by uid 0); 16 May 2008 15:41:04 -0000 Received: from ip83-113-174-82.adsl2.versatel.nl (HELO istud.quis.cx) ([82.174.113.83]) (envelope-sender ) by smtp3.versatel.nl (qmail-ldap-1.03) with SMTP for < >; 16 May 2008 15:41:04 -0000 Received: by istud.quis.cx (Postfix, from userid 100) id 670173982D; Fri, 16 May 2008 17:41:04 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on istud.quis.cx X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.4 Received: from [192.168.1.10] (unknown [192.168.1.10]) by istud.quis.cx (Postfix) with ESMTP id 9801D3982B for ; Fri, 16 May 2008 17:41:01 +0200 (CEST) Message-ID: <482DAB0E.70600@quis.cx> Date: Fri, 16 May 2008 17:41:02 +0200 From: Jille Timmermans User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Trying (not) to crash with libpthread (6.3-RELEASE) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 15:41:07 -0000 Hello, I'm trying to catch SIGSEGV, to be able to run 'unchecked' (possibly crashing) code. And not letting the entire program die, but just kill that thread. So I wrote some testing code. But, I ran into a problem; the signal-handler does not work when a thread is created (or something like that). The code below crashes on SIGSEGV, but should die in void sigcatcher(int); If you uncomment the crashingthread(NULL); line, it will die in the signal handler. pthread_create seems to take over signal handling. I quickly checked some pthread source, and it seems that it should call my own handler. (There might be an exception for some signals, but I didn't found that) Can anyone explain me what is happening here ? -- Jille cc -lpthread below.c #include #include #include #include #include #include int success=0; void sigcatcher(int sig) { printf("[%p] signal %d\n", pthread_self(), sig); printf("Test (probably) succeeded\n"); fflush(NULL); success=1; exit(0); } void * crashingthread(void *nada) { /* This will likely crash */ char *x=malloc(1); if(signal(SIGSEGV, sigcatcher)==SIG_ERR) err(1, "signal(SIGSEGV, catchz0r)"); x[666]=0; /* HOPEFULLY NOT REACHED (aargh! die harder!) */ int i; for(i=1; 999999>i; i++) x[i]=0; /* NOT REACHED (either killed, or exit()'ed in sigcatcher) */ abort(); } int main(int argc, char **argv) { pthread_t thr; if(signal(SIGSEGV, sigcatcher)==SIG_ERR) err(1, "signal(SIGSEGV, catchz0r)"); //crashingthread(NULL); pthread_create(&thr, NULL, crashingthread, NULL); sleep(3); return 0; } From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 14:48:43 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AF3C81065682 for ; Fri, 16 May 2008 14:48:43 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (casselton.net [63.165.140.2]) by mx1.freebsd.org (Postfix) with ESMTP id 4E1DC8FC15 for ; Fri, 16 May 2008 14:48:43 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (localhost [127.0.0.1]) by casselton.net (8.14.2/8.14.2) with ESMTP id m4GEY7DX082156; Fri, 16 May 2008 09:34:07 -0500 (CDT) (envelope-from tinguely@casselton.net) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=casselton.net; s=ccnMail; t=1210948447; bh=ptgvQ+2vMeOhestrHt8AWrcMc9Q=; h=Date: From:Message-Id:To:Subject:Cc:In-Reply-To; b=NidSMw9NHNCZZPFIkoBeU PkrlpWNp7DkKniVr+YfwmsoKN/O22H+6ltnpvEpe6hex9B51axpKF9Zz899Q5NZj5m7 V2azbL5zbjR7K4zC3nqggyXBu/gHhdnD/hrE7yD5TFuKpPYcofERVJZa1NjX4uaqtAk fyd8KuphUOLMCin4= Received: (from tinguely@localhost) by casselton.net (8.14.2/8.14.2/Submit) id m4GEY6N3082155; Fri, 16 May 2008 09:34:06 -0500 (CDT) (envelope-from tinguely) Date: Fri, 16 May 2008 09:34:06 -0500 (CDT) From: Mark Tinguely Message-Id: <200805161434.m4GEY6N3082155@casselton.net> To: teemu@rinta-aho.org In-Reply-To: <482C4DF3.8030709@rinta-aho.org> X-Mailman-Approved-At: Fri, 16 May 2008 16:09:59 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: copy-on-write anonymous memory? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 14:48:43 -0000 Teemu Rinta-aho wrote (some edits): > I have created a kernel module that stores references to memory objects. > I.e. when a process makes a syscall to the module, it will create a > snapshot of the memory area, and after that the writes from the process to > that memory area should create a shadow object. The next syscall should > again store a pointer to the current topmost shadow object and then the > next write creates yet another shadow object. Etc... When the snapshots > are removed, the shadow chains may collapse normally. > > Here's an illustration of what I want (first syscall OK, second one not): > > * Legend: U/u = userspace K/k = kernel > * > * U:vm_map_entry_u -> object > * || > * SYSCALL > * || > * \/ > * U:vm_map_entry_u -> object_shadow -> object > * / > * K:vm_map_entry_k ---------------- > * || > * SYSCALL > * || > * \/ > * U:vm_map_entry_u -> object_shadow -> object_shadow -> object > * / / > * K:vm_map_entry_k ---------------- / > * K:vm_map_entry_k -------------------------------- > > Now, the problem is that the first snapshot works as it should. However, > the second one doesn't, and the write goes to the one and same shadow > object, even if I restore MAP_ENTRY_COW and MAP_ENTRY_NEEDS_COPY manually > in my handler function which is storing the snapshot. > > Any ideas? Usually, a fork() creates the inheritance between parent and child COW memory space. Start: vm_map_entry -> object_shadow -> object fork(): vm_map_entry -> object_shadow -\ |-> (object_shadow*) -> object vm_map_entry -> object_shadow -/ This is slightly different from your description/drawing, in the way changes are inherited; for example: process 1 is created, process 1 writes page 0. process 2 is created. process 1 writes p1 (or p0 again). Your description/drawing implies that process 2 see this change from process 1. You are not forking over a COW memory area. Sounds like the syscall will have manually create the inheritance. You can manually link the object_shadows the way you want to get the desired inheritance. Process removals should collapse the shadows automatically. Matt Dillion wrote a brief VM description (http://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/). The book, "The Design and Implementation of the 4.4BSD Operating System" is another great reference. Mark Tinguely From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 15:57:05 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E4A6106566B for ; Fri, 16 May 2008 15:57:05 +0000 (UTC) (envelope-from bounces@nabble.com) Received: from kuber.nabble.com (kuber.nabble.com [216.139.236.158]) by mx1.freebsd.org (Postfix) with ESMTP id 7F7CE8FC15 for ; Fri, 16 May 2008 15:57:05 +0000 (UTC) (envelope-from bounces@nabble.com) Received: from isper.nabble.com ([192.168.236.156]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from ) id 1Jx25y-0007Fy-Nm for freebsd-hackers@freebsd.org; Fri, 16 May 2008 08:43:46 -0700 Message-ID: <17277923.post@talk.nabble.com> Date: Fri, 16 May 2008 08:43:46 -0700 (PDT) From: n4ti1us To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: gaspar_i@yahoo.com X-Mailman-Approved-At: Fri, 16 May 2008 16:41:28 +0000 Subject: Spin locks and soft IRQs X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 15:57:05 -0000 Hi All, I was looking at the linux kernel, and in the implementation of spinlocks, one can use different functions. One of them is spin_lock_bh(). What it does is it creates a spin lock and disables the software irqs. In FreeBSD on the other hand, the implementation of spinlocks is done by mtx_lock_spin (which is actually a macro that calls mtx_lock_spin_flags()) but the problem is that is disables all the irq vector (at the bottom of the stack it executes an "cli"). Is there any way one can implement spinlocks that only disable the bottom half (sof irq)? Alternatively, what would be the best way to manualy disable the soft irqs? And all this makes me think... Why has nobody implemented a "_bh" version so far? Is this an architectural decision, or is it just me being blind? -- View this message in context: http://www.nabble.com/Spin-locks-and-soft-IRQs-tp17277923p17277923.html Sent from the freebsd-hackers mailing list archive at Nabble.com. From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 17:23:50 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D3681065672 for ; Fri, 16 May 2008 17:23:50 +0000 (UTC) (envelope-from deathjestr@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.155]) by mx1.freebsd.org (Postfix) with ESMTP id 8E6918FC15 for ; Fri, 16 May 2008 17:23:49 +0000 (UTC) (envelope-from deathjestr@gmail.com) Received: by fg-out-1718.google.com with SMTP id l26so972241fgb.35 for ; Fri, 16 May 2008 10:23:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=mRFSXovDfrP79rP03EGlxavSOlyxuNOm3xQiQRa6ejs=; b=b4rqsmpq6WAicDX/Z/3e2PQNHifHKT1l2gSQ4dLDurd6KUM5oJvN1BRCTbHkv35O8V8IfxkxFomQuiasTuOXyLX3N9p1hFwJvAt6mlsJz5hHbDBdWLbrUP133pmMfsu9PeYf6I1gkF5fMukqTU2cz/pUxR7YZZSvt4LDG9NuOZQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=nqL80tsZzTddCexnWmthW/IlkmJRx7hsPzgLp7OvJWqAUf5F21CmLneRKB88hwFLq3Ot+sNBblsJNoMShw/ms9CeYQkf5swcZQTY9K8GzMSPXQqPPvDcypn9tgpDGDILiiK+IqNAHSDf6og/NVSb8eUi9D2j3CcVNWRGQ386tdE= Received: by 10.78.199.8 with SMTP id w8mr983409huf.106.1210956958923; Fri, 16 May 2008 09:55:58 -0700 (PDT) Received: by 10.78.118.10 with HTTP; Fri, 16 May 2008 09:55:58 -0700 (PDT) Message-ID: <44b564930805160955r76d6861fo1949c7f4d5b71ca5@mail.gmail.com> Date: Fri, 16 May 2008 12:55:58 -0400 From: Mike To: "Jille Timmermans" In-Reply-To: <482DAB0E.70600@quis.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <482DAB0E.70600@quis.cx> Cc: freebsd-hackers@freebsd.org Subject: Re: Trying (not) to crash with libpthread (6.3-RELEASE) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 17:23:50 -0000 > void > sigcatcher(int sig) { > printf("[%p] signal %d\n", pthread_self(), sig); > printf("Test (probably) succeeded\n"); > fflush(NULL); > success=1; > exit(0); > } Take a look at the man page for sigaction(2). Here's an excerpt: "All functions not in the above lists are considered to be unsafe with respect to signals. That is to say, the behaviour of such functions when called from a signal handler is undefined. In general though, signal handlers should do little more than set a flag; most other actions are not safe." > void * > crashingthread(void *nada) { > /* This will likely crash */ > char *x=malloc(1); > > if(signal(SIGSEGV, sigcatcher)==SIG_ERR) > err(1, "signal(SIGSEGV, catchz0r)"); > > x[666]=0; > > /* HOPEFULLY NOT REACHED (aargh! die harder!) */ > > int i; > for(i=1; 999999>i; i++) > x[i]=0; > > /* NOT REACHED (either killed, or exit()'ed in sigcatcher) */ > abort(); > } Something like *(int*)0 = 0 is probably an easier, more reliable way to crash your program. Maybe not quite as much fun though. ;-) From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 18:18:35 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 77B791065676 for ; Fri, 16 May 2008 18:18:35 +0000 (UTC) (envelope-from mark@foster.cc) Received: from mail.bitpusher.com (mail.bitpusher.com [208.75.56.13]) by mx1.freebsd.org (Postfix) with ESMTP id 56E138FC0A for ; Fri, 16 May 2008 18:18:35 +0000 (UTC) (envelope-from mark@foster.cc) Received: from [192.168.1.108] (dsl-242-83.zhonka.net [208.64.242.83]) by mail.bitpusher.com (Postfix) with ESMTP id 6BC244C135 for ; Fri, 16 May 2008 11:01:13 -0700 (PDT) Message-ID: <482DCBEA.3030107@foster.cc> Date: Fri, 16 May 2008 11:01:14 -0700 From: Mark Foster User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: multipart/mixed; boundary="------------010805070901070408070408" Subject: [Fwd: lwresd howto] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 18:18:35 -0000 This is a multi-part message in MIME format. --------------010805070901070408070408 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit No response from -questions, trying here. The gist is: is implementing lwresd feasible as a caching-only stub resolver? I need something I can run locally to intercept and cache DNS responses. BIND is not the answer (too heavyweight) and dnsmasq doesn't appear to cache. nscd is what I'm used to on Linux but it doesn't seem to be in FreeBSD. Something called cached but I don't see it on the systems I'm working with (5.4, 6.1, 6.2 yea I know out-of-date so don't even say it). Can I use lwresd ? The documentation for how to do so are unclear. I've found: http://www.isc.org/sw/bind/arm93/Bv9ARM.ch05.html and reviewed the lwresd(8) man page but there is not a good example of how to setup correctly. Also the ISC doc describes putting lwserver entries into resolv.conf but these are not mentioned in resolver(5). I do not see the hooks around for nsswitch.conf e.g. to put lwres in the hosts: line would have no effect because something like nss_lwres does not exist for FreeBSD. -- Some days it's just not worth chewing through the restraints... Mark D. Foster, CISSP http://mark.foster.cc/ --------------010805070901070408070408 Content-Type: message/rfc822; name="lwresd howto.eml" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="lwresd howto.eml" Message-ID: <482B4FA0.8090800@foster.cc> Date: Wed, 14 May 2008 13:46:24 -0700 From: Mark Foster User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: User Questions Subject: lwresd howto Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit I need something I can run locally to intercept and cache DNS responses. BIND is not the answer (too heavyweight) and dnsmasq doesn't appear to cache. nscd is what I'm used to on Linux but it doesn't seem to be in FreeBSD. Something called cached but I don't see it on the systems I'm working with (5.4, 6.1, 6.2 yea I know out-of-date so don't even say it). So I am considering lwresd(8) Can I use lwresd ? The documentation for how to do so are unclear. I've found: http://www.isc.org/sw/bind/arm93/Bv9ARM.ch05.html and reviewed the lwresd(8) man page but there is not a good example of how to setup correctly. Also the ISC doc mention putting lwserver entries into resolv.conf but these are not mentioned in resolver(5). I have tried putting lwserver 127.0.0.1 and nameserver 127.0.0.1 while running lwresd but not getting response. Does some kind soul have this working and can provide some example or advice? -- Some days it's just not worth chewing through the restraints... Mark D. Foster, CISSP http://mark.foster.cc/ --------------010805070901070408070408-- From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 19:01:50 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1439D1065672 for ; Fri, 16 May 2008 19:01:50 +0000 (UTC) (envelope-from mark@foster.cc) Received: from mail.bitpusher.com (mail.bitpusher.com [208.75.56.13]) by mx1.freebsd.org (Postfix) with ESMTP id F3BA28FC0C for ; Fri, 16 May 2008 19:01:49 +0000 (UTC) (envelope-from mark@foster.cc) Received: from [192.168.1.108] (dsl-242-83.zhonka.net [208.64.242.83]) by mail.bitpusher.com (Postfix) with ESMTP id 6F8A54C12D for ; Fri, 16 May 2008 12:01:49 -0700 (PDT) Message-ID: <482DDA1E.2030208@foster.cc> Date: Fri, 16 May 2008 12:01:50 -0700 From: Mark Foster User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org References: <482DCBEA.3030107@foster.cc> <20080516143917.5246a733@bhuda.mired.org> In-Reply-To: <20080516143917.5246a733@bhuda.mired.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Fwd: lwresd howto] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 19:01:50 -0000 Mike Meyer wrote: > What makes you think dnsmasq doesn't cache? The documentation says > otherwise (second paragraph of the man page): > > Dnsmasq accepts DNS queries and either answers them from a small, > local, cache or forwards them to a real, recursive, DNS server. It > > it has quite a few options to control caching behavior, and certainly > seems to cache for me. > > Thanks for pointing out my error. I have no problem using dnsmasq if that is the best recommendation, in fact I quite like it but am not fond of djbdns. However lwresd *is* in the base install vs. dnsmasq which is a port, so would like to use lwresd if possible (or something else in base like cached). There is nothing I find in the man page, handbook or google that says I shouldn't or can't use lwresd, only hints that I can, yet with no clear working real-world examples. So I'm really hoping for two answers. 1. Can lwresd be used (somehow) or not 2. What are the viable/best alternatives to #1 for a caching-only stub resolver (like nscd on linux) Thanks. -- Some days it's just not worth chewing through the restraints... Mark D. Foster, CISSP http://mark.foster.cc/ From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 19:14:17 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07167106566B for ; Fri, 16 May 2008 19:14:17 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (unknown [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id 703238FC12 for ; Fri, 16 May 2008 19:14:16 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.1/8.14.1) with ESMTP id m4GJEDt0053922; Fri, 16 May 2008 21:14:14 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.1/8.14.1/Submit) id m4GJEBSD053921; Fri, 16 May 2008 21:14:11 +0200 (CEST) (envelope-from olli) Date: Fri, 16 May 2008 21:14:11 +0200 (CEST) Message-Id: <200805161914.m4GJEBSD053921@lurza.secnetix.de> From: Oliver Fromme To: freebsd-hackers@FreeBSD.ORG, deathjestr@gmail.com, jille@quis.cx In-Reply-To: <44b564930805160955r76d6861fo1949c7f4d5b71ca5@mail.gmail.com> X-Newsgroups: list.freebsd-hackers User-Agent: tin/1.8.3-20070201 ("Scotasay") (UNIX) (FreeBSD/6.2-STABLE-20070808 (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Fri, 16 May 2008 21:14:15 +0200 (CEST) Cc: Subject: Re: Trying (not) to crash with libpthread (6.3-RELEASE) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-hackers@FreeBSD.ORG, deathjestr@gmail.com, jille@quis.cx List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 19:14:17 -0000 Mike wrote: > Jille Timmermans wrote: > > void > > sigcatcher(int sig) { > > printf("[%p] signal %d\n", pthread_self(), sig); > > printf("Test (probably) succeeded\n"); > > fflush(NULL); > > success=1; > > exit(0); > > } > > Take a look at the man page for sigaction(2). Here's an excerpt: > > "All functions not in the above lists are considered to be unsafe with > respect to signals. That is to say, the behaviour of such functions when > called from a signal handler is undefined. In general though, signal > handlers should do little more than set a flag; most other actions are > not safe." Additionally, the "success" variable should be declared as "volatile sig_atomic_t". By the way, this code ... > > char *x=malloc(1); > > [...] > > x[666]=0; .. does not necessarily cause a SIGSEGV. It may do so, but it is not guaranteed. It may as well write into a random memory area that happens to belong to your process, too. The result is undefined, i.e. it could be anything. If you need to send a SIGSEGV to yourself reliably, the best way is to kill(getpid(), SIGSEGV). Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "File names are infinite in length, where infinity is set to 255 characters." -- Peter Collinson, "The Unix File System" From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 19:22:00 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D834106564A for ; Fri, 16 May 2008 19:22:00 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (five.mired.org [66.92.153.75]) by mx1.freebsd.org (Postfix) with ESMTP id D5D028FC0A for ; Fri, 16 May 2008 19:21:59 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 22766 invoked by uid 1001); 16 May 2008 15:22:26 -0400 Received: from bhuda.mired.org (bhuda [192.168.195.1]) by bhuda (tmda-ofmipd) with ESMTP; Fri, 16 May 2008 15:22:25 -0400 Date: Fri, 16 May 2008 15:22:25 -0400 To: Mark Foster Message-ID: <20080516152225.75d97deb@bhuda.mired.org> In-Reply-To: <482DDA1E.2030208@foster.cc> References: <482DCBEA.3030107@foster.cc> <20080516143917.5246a733@bhuda.mired.org> <482DDA1E.2030208@foster.cc> Organization: Meyer Consulting X-Mailer: Claws Mail 3.4.0 (GTK+ 2.12.9; amd64-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Delivery-Agent: TMDA/1.1.12 (Macallan) From: Mike Meyer Cc: freebsd-hackers@freebsd.org Subject: Re: [Fwd: lwresd howto] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 19:22:00 -0000 On Fri, 16 May 2008 12:01:50 -0700 Mark Foster wrote: > Mike Meyer wrote: > > What makes you think dnsmasq doesn't cache? The documentation says > > otherwise (second paragraph of the man page): > > > > Dnsmasq accepts DNS queries and either answers them from a small, > > local, cache or forwards them to a real, recursive, DNS server. It > > > > it has quite a few options to control caching behavior, and certainly > > seems to cache for me. > > > > > Thanks for pointing out my error. > I have no problem using dnsmasq if that is the best recommendation, in > fact I quite like it but am not fond of djbdns. I wanted a cooperating dhcp & dns server, and having them in one package is to big a win to ignore. > However lwresd *is* in the base install vs. dnsmasq which is a port, so > would like to use lwresd if possible (or something else in base like > cached). There is nothing I find in the man page, handbook or google > that says I shouldn't or can't use lwresd, only hints that I can, yet > with no clear working real-world examples. > > So I'm really hoping for two answers. > 1. Can lwresd be used (somehow) or not lwresd isn't a dns server. I suspect it's in the base system because it comes bundled with bind, rather than because it's used itself, as there's no hook to start it in /etc/rc or /etc/defaults. > 2. What are the viable/best alternatives to #1 for a caching-only stub > resolver (like nscd on linux) dnsmasq is viable. May not be the best for your needs, but it works great for me. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 20:47:56 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B59D1065677 for ; Fri, 16 May 2008 20:47:56 +0000 (UTC) (envelope-from sr@genyosha.net) Received: from ns.genyosha.net (ns.genyosha.net [216.103.76.250]) by mx1.freebsd.org (Postfix) with ESMTP id 091D68FC12 for ; Fri, 16 May 2008 20:47:55 +0000 (UTC) (envelope-from sr@genyosha.net) Received: from dragon.genyosha.net (dragon.genyosha.net [216.103.76.254]) by ns.genyosha.net (8.12.11.20060614/8.12.11) with ESMTP id m4GKBsYu012620 for ; Fri, 16 May 2008 13:11:54 -0700 (PDT) Received: from dragon.genyosha.net (localhost [127.0.0.1]) by dragon.genyosha.net (8.14.1/8.13.8) with ESMTP id m4GKBs0a082616 for ; Fri, 16 May 2008 13:11:54 -0700 (PDT) (envelope-from sr@dragon.genyosha.net) Received: (from sr@localhost) by dragon.genyosha.net (8.14.1/8.14.2/Submit) id m4GKBsQY082615 for freebsd-hackers@freebsd.org; Fri, 16 May 2008 13:11:54 -0700 (PDT) (envelope-from sr) Date: Fri, 16 May 2008 13:11:54 -0700 From: Steve Rikli To: freebsd-hackers@freebsd.org Message-ID: <20080516201154.GA82576@dragon.genyosha.net> References: <482DCBEA.3030107@foster.cc> <20080516143917.5246a733@bhuda.mired.org> <482DDA1E.2030208@foster.cc> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <482DDA1E.2030208@foster.cc> User-Agent: Mutt/1.4.2.3i Subject: Re: [Fwd: lwresd howto] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 20:47:56 -0000 On Fri, May 16, 2008 at 12:01:50PM -0700, Mark Foster wrote: > Mike Meyer wrote: > ... > I have no problem using dnsmasq if that is the best recommendation, in > fact I quite like it but am not fond of djbdns. > However lwresd *is* in the base install vs. dnsmasq which is a port, so > would like to use lwresd if possible (or something else in base like > cached). There is nothing I find in the man page, handbook or google > that says I shouldn't or can't use lwresd, only hints that I can, yet > with no clear working real-world examples. > > So I'm really hoping for two answers. > 1. Can lwresd be used (somehow) or not > 2. What are the viable/best alternatives to #1 for a caching-only stub > resolver (like nscd on linux) Wrt #2, I was able to compile & install nscd from FreeBSD src, but only on systems running fairly modern releases -- IIRC no older than 6.2 . cheers, sr. From owner-freebsd-hackers@FreeBSD.ORG Fri May 16 22:22:33 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 14785106564A for ; Fri, 16 May 2008 22:22:33 +0000 (UTC) (envelope-from gonzo@pbxpress.com) Received: from bugor.portaone.com (bugor.portaone.com [65.61.203.147]) by mx1.freebsd.org (Postfix) with ESMTP id EE9DC8FC14 for ; Fri, 16 May 2008 22:22:32 +0000 (UTC) (envelope-from gonzo@pbxpress.com) Received: from mail.pbxpress.com ([65.61.203.142] helo=leaf.pbxpress.com) by bugor.portaone.com (8.11.3/8.11.3) with ESMTP (TLSv1:AES256-SHA:256)id 1Jx7vr-000BAI-Ic for freebsd-hackers@freebsd.org; Fri, 16 May 2008 14:57:45 -0700 Received: from jeeves.bluezbox.com (82.193.105.77.ipnet.kiev.ua [82.193.105.77]) (authenticated bits=0) by leaf.pbxpress.com (8.13.3/8.13.3) with ESMTP id m4GM17oR029048 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 16 May 2008 15:01:15 -0700 (PDT) (envelope-from gonzo@pbxpress.com) Message-ID: <482E034C.2080402@pbxpress.com> Date: Sat, 17 May 2008 00:57:32 +0300 From: Oleksandr Tymoshenko User-Agent: Thunderbird 2.0.0.6 (X11/20070907) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, recieved from trusted server Subject: aio_write issues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 22:22:33 -0000 It looks like aio_write is out of sync with its man page(if it ever has been). man 2 aio_write states: If O_APPEND is set for iocb->aio_fildes, aio_write() operations append to the file in the same order as the calls were made. Though simple test[1] reveals that it's not true. Order is not preserved. In addition the rest of the paragraph contradicts to SUSv3 specification of aio_write (or I got that part of man page wrong, because function behaves just as should be conforming to SUSv3). man page: If O_APPEND is not set for the file descriptor, the write operation will occur at the absolute position from the beginning of the file plus iocb->aio_offset. excerpt from IEEE 1003.1: If O_APPEND is not set for the file descriptor aio_fildes, then the requested operation shall take place at the absolute position in the file as given by aio_offset, as if lseek() were called immediately prior to the operation with an offset equal to aio_offset and a whence equal to SEEK_SET. I'd like to know if I got things right in order to avoid fixing things that are not broken. [1] http://people.freebsd.org/~gonzo/aio.c -- gonzo From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 01:34:18 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ADCDF1065674 for ; Sat, 17 May 2008 01:34:18 +0000 (UTC) (envelope-from bilouro@bilouro.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.248]) by mx1.freebsd.org (Postfix) with ESMTP id 725348FC0A for ; Sat, 17 May 2008 01:34:18 +0000 (UTC) (envelope-from bilouro@bilouro.com) Received: by an-out-0708.google.com with SMTP id b33so299991ana.13 for ; Fri, 16 May 2008 18:34:17 -0700 (PDT) Received: by 10.100.96.9 with SMTP id t9mr5598712anb.78.1210986440141; Fri, 16 May 2008 18:07:20 -0700 (PDT) Received: by 10.100.210.19 with HTTP; Fri, 16 May 2008 18:07:20 -0700 (PDT) Message-ID: Date: Fri, 16 May 2008 22:07:20 -0300 From: "Victor Hugo Bilouro" To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: GSoC Student X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 01:34:18 -0000 Hi, I'm Victor Hugo *Bilouro*, GSoC participant. I'll be working on tcptest. It's name came from TCP/IP Regression test suite. As a testing tool it can: *be a conformance test *test a bug already fixed (regression test - one test for each fixed bug) *test a new feature (help protocol developer) *simulate certain cases It will be build on top of pcs.sf.net (Package Construction Set) "PCS is a set of Python modules and objects that make building network protocol code easier for the protocol developer. The core of the system is the pcs module itself which provides the necessary functionality to create classes that implement packets. " My mentor is George V. Neville-Neil, no comments. :-) GSoC started a month ago, by this time, I had almost all environment done, and I'm spending several times reading and often playing with pcs. (cool) Books i bought: (*)TCP/IP Illustrated Volume 1 (far the best) Stevens (*)TCP/IP Guide Charles M.Kazierok (www.tcpipguide.com) (*)Internetworking with tcp/ip I will post on __link__ links, books, status report, futures releases information, news, everything about the project. __link__: http://wiki.freebsd.org/VictorBilouro/TCP-IP_regression_test_suite Regards -- Victor Hugo Bilouro FreeBSD From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 02:16:06 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D73A1065673 for ; Sat, 17 May 2008 02:16:06 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outD.internet-mail-service.net (outd.internet-mail-service.net [216.240.47.227]) by mx1.freebsd.org (Postfix) with ESMTP id 53C5A8FC1B for ; Sat, 17 May 2008 02:16:06 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Sat, 17 May 2008 11:36:19 -0700 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id 7755D2D6017; Fri, 16 May 2008 19:16:05 -0700 (PDT) Message-ID: <482E3FE8.2000509@elischer.org> Date: Fri, 16 May 2008 22:16:08 -0400 From: Julian Elischer User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Jille Timmermans References: <482DAB0E.70600@quis.cx> In-Reply-To: <482DAB0E.70600@quis.cx> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: Trying (not) to crash with libpthread (6.3-RELEASE) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 02:16:06 -0000 Jille Timmermans wrote: > Hello, > > I'm trying to catch SIGSEGV, to be able to run 'unchecked' (possibly > crashing) code. > And not letting the entire program die, but just kill that thread. > So I wrote some testing code. > But, I ran into a problem; the signal-handler does not work when a > thread is created (or something like that). > The code below crashes on SIGSEGV, but should die in void sigcatcher(int); > If you uncomment the crashingthread(NULL); line, it will die in the > signal handler. > > pthread_create seems to take over signal handling. > I quickly checked some pthread source, and it seems that it should call > my own handler. > (There might be an exception for some signals, but I didn't found that) > > Can anyone explain me what is happening here ? yes, you are correct. I a vain attempt to give something close to the semanticcs defined by the pthread spec, the pthreads app sets a special thread to do nothing but catch signals. Originally when we started the immediate signals such as SIGSEGV were exempt from this and were delivered to the thread that caused it immediatly, however it was impossible to do this effectively and still provide correct sematics for the rest of the signals. Eventually we just gave up on SIGSEGV because we just couldn't get it to work. We could catch the signal in the signal thread, but we couldn't get it to deliver it to the errant thread reliably, (because the errant thread couldn't run any more at that point). using top -H you should be able to see the extgra thread in the kernel waiting for the signals. This was one of the reasons that 1:1 threading eventually won out in our threading evaluation period. In M:N threading the aim is to switch threads without kernel involvement, however iti turns out that to do SIGSEGV and friends properly, you need to re-do the thread signal mask for every thread with a syscall for every thread swap which defeats teh whole point. With 1:1 threads this all "just works" try libthr. it should work for you. > > -- Jille > > > cc -lpthread below.c > #include > #include > #include > #include > #include > #include > > int success=0; > > void > sigcatcher(int sig) { > printf("[%p] signal %d\n", pthread_self(), sig); > printf("Test (probably) succeeded\n"); > fflush(NULL); > success=1; > exit(0); > } > > void * > crashingthread(void *nada) { > /* This will likely crash */ > char *x=malloc(1); > > if(signal(SIGSEGV, sigcatcher)==SIG_ERR) > err(1, "signal(SIGSEGV, catchz0r)"); > > x[666]=0; > > /* HOPEFULLY NOT REACHED (aargh! die harder!) */ > > int i; > for(i=1; 999999>i; i++) > x[i]=0; > > /* NOT REACHED (either killed, or exit()'ed in sigcatcher) */ > abort(); > } > > int > main(int argc, char **argv) { > pthread_t thr; > > if(signal(SIGSEGV, sigcatcher)==SIG_ERR) > err(1, "signal(SIGSEGV, catchz0r)"); > > //crashingthread(NULL); > pthread_create(&thr, NULL, crashingthread, NULL); > > sleep(3); > return 0; > } > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 08:13:54 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01ABF1065670 for ; Sat, 17 May 2008 08:13:54 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from hosted.kievnet.com (hosted.kievnet.com [193.138.144.10]) by mx1.freebsd.org (Postfix) with ESMTP id B2F238FC15 for ; Sat, 17 May 2008 08:13:53 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost ([127.0.0.1] helo=edge.pp.kiev.ua) by hosted.kievnet.com with esmtpa (Exim 4.62) (envelope-from ) id 1JxHY8-0009G5-Lh for freebsd-hackers@freebsd.org; Sat, 17 May 2008 11:13:52 +0300 Message-ID: <482E93C0.4070802@icyb.net.ua> Date: Sat, 17 May 2008 11:13:52 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.12 (X11/20080320) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: rdmsr from userspace X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 08:13:54 -0000 It seems that rdmsr instruction can be executed only at the highest privilege level and thus is not permitted from userland. Maybe we should provide something like Linux /dev/cpu/msr? I don't like interface of that device, I think that ioctl approach would be preferable in this case. Something like create /dev/cpuN and allow some ioctls on it: ioctl(cpu_fd, CPU_RDMSR, arg). What do you think? -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 09:20:35 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4C5E81065674 for ; Sat, 17 May 2008 09:20:35 +0000 (UTC) (envelope-from tataz@tataz.chchile.org) Received: from smtp5-g19.free.fr (smtp5-g19.free.fr [212.27.42.35]) by mx1.freebsd.org (Postfix) with ESMTP id DAB7E8FC23 for ; Sat, 17 May 2008 09:20:34 +0000 (UTC) (envelope-from tataz@tataz.chchile.org) Received: from smtp5-g19.free.fr (localhost.localdomain [127.0.0.1]) by smtp5-g19.free.fr (Postfix) with ESMTP id A333D3F62AD for ; Sat, 17 May 2008 11:20:33 +0200 (CEST) Received: from tatooine.tataz.chchile.org (tataz.chchile.org [82.233.239.98]) by smtp5-g19.free.fr (Postfix) with ESMTP id 8FABA3F6286 for ; Sat, 17 May 2008 11:20:33 +0200 (CEST) Received: from obiwan.tataz.chchile.org (unknown [192.168.1.25]) by tatooine.tataz.chchile.org (Postfix) with ESMTP id AECCD9F2E6 for ; Sat, 17 May 2008 09:17:40 +0000 (UTC) Received: by obiwan.tataz.chchile.org (Postfix, from userid 1000) id A6137408A; Sat, 17 May 2008 11:17:40 +0200 (CEST) Date: Sat, 17 May 2008 11:17:40 +0200 From: Jeremie Le Hen To: freebsd-hackers@FreeBSD.org Message-ID: <20080517091740.GI70896@obiwan.tataz.chchile.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.15 (2007-04-06) Cc: Subject: Debugging rtld X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 09:20:35 -0000 Hi there, I tried to compile my source tree with -fstack-protector-all, and it happens that rtld breaks with this: once the new rtld is installed every single problem coredumps. I tried to compile rtld-elf without SSP, but it didn't solve the problem. Then I had to compile libc_pic.a without SSP and it worked, but I don't understand the root of the problem. So I want to use the generated coredump for post-mortem analysis with gdb. I compiled world with DEBUG_FLAGS=-g. But GDB gives me a backtrace so long that it can't be real. Moreoever it doesn't seem to bring in the required symbols. I'm a GDB novice, so I'd like some help. chroot> ===> libexec/rtld-elf (install) chroot> chflags noschg /usr/libexec/ld-elf.so.1 chroot> install -s -o root -g wheel -m 555 -C -b -fschg -S ld-elf.so.1 /libexec chroot> install -o root -g wheel -m 444 rtld.1.gz /usr/share/man/man1 chroot> *** Signal 11 chroot> chroot> jarjarbinks# cd /root; ls chroot> Segmentation fault host> jarjarbinks:145# ls -l /space/chroot/root/ls.core host> -rw------- 1 root wheel 184320 May 17 10:19 /space/chroot/root/ls.core host> jarjarbinks:149# gdb -c /space/chroot/root/ls.core -e /space/chroot/bin/ls host> GNU gdb 6.1.1 [FreeBSD] host> [...] host> This GDB was configured as "i386-marcel-freebsd". host> Core was generated by `ls'. host> Program terminated with signal 11, Segmentation fault. host> #0 0x280583e4 in ?? () host> (gdb) bt host> #0 0x280583e4 in ?? () host> #1 0x00000000 in ?? () host> #2 0x00000000 in ?? () host> #3 0x00000000 in ?? () host> #4 0x00000000 in ?? () host> #5 0x00000000 in ?? () host> #6 0x00000000 in ?? () host> #7 0x00000000 in ?? () host> #8 0x00000000 in ?? () host> #9 0x00000000 in ?? () host> #10 0x00000000 in ?? () host> #11 0xffffffff in ?? () host> #12 0x00001000 in ?? () host> [...] host> #359 0x73763a68 in ?? () host> #360 0x5b455c3d in ?? () host> [...] host> #855 0x00000000 in ?? () host> [...] Any hint on how to proceed would be welcome. Thank you. Regards, -- Jeremie Le Hen < jeremie at le-hen dot org >< ttz at chchile dot org > From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 10:13:09 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7FFA2106567E for ; Sat, 17 May 2008 10:13:09 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from anti-4.kiev.sovam.com (anti-4.kiev.sovam.com [62.64.120.202]) by mx1.freebsd.org (Postfix) with ESMTP id D97D08FC15 for ; Sat, 17 May 2008 10:13:08 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=skuns.kiev.zoral.com.ua) by anti-4.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1JxJPV-000ALp-PS for freebsd-hackers@freebsd.org; Sat, 17 May 2008 13:13:06 +0300 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by skuns.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4HACl0K069896 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 17 May 2008 13:12:47 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4HACjUg029350; Sat, 17 May 2008 13:12:45 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m4HAChwB029349; Sat, 17 May 2008 13:12:43 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 17 May 2008 13:12:43 +0300 From: Kostik Belousov To: Andriy Gapon Message-ID: <20080517101243.GH18958@deviant.kiev.zoral.com.ua> References: <482E93C0.4070802@icyb.net.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ejhj5mF8juGI/iew" Content-Disposition: inline In-Reply-To: <482E93C0.4070802@icyb.net.ua> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.4 X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on skuns.kiev.zoral.com.ua X-Scanner-Signature: 7b1a4900bea4ee3727a65c989d5962bc X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Header: Not Detected X-SpamTest-Info: Profiles 2835 [May 13 2008] X-SpamTest-Info: helo_type=3 X-SpamTest-Method: none X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0278], KAS30/Release Cc: freebsd-hackers@freebsd.org Subject: Re: rdmsr from userspace X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 10:13:09 -0000 --ejhj5mF8juGI/iew Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, May 17, 2008 at 11:13:52AM +0300, Andriy Gapon wrote: >=20 > It seems that rdmsr instruction can be executed only at the highest=20 > privilege level and thus is not permitted from userland. Maybe we should= =20 > provide something like Linux /dev/cpu/msr? > I don't like interface of that device, I think that ioctl approach would= =20 > be preferable in this case. > Something like create /dev/cpuN and allow some ioctls on it:=20 > ioctl(cpu_fd, CPU_RDMSR, arg). > What do you think? I believe ports/sysutils/devcpu is what you need. In fact, I think this ought to be committed to the base. I do not know why Stas did not asked for this, at least at public. --ejhj5mF8juGI/iew Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkgur5sACgkQC3+MBN1Mb4hURQCfcnilZbpQqqMmy1xQsPSoZoqA 8qEAoK1OxSCTxAGZ53Jebag6x1Im1S7q =Lcx4 -----END PGP SIGNATURE----- --ejhj5mF8juGI/iew-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 10:14:54 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3EFD61065674 for ; Sat, 17 May 2008 10:14:54 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from hosted.kievnet.com (hosted.kievnet.com [193.138.144.10]) by mx1.freebsd.org (Postfix) with ESMTP id EEBD98FC1F for ; Sat, 17 May 2008 10:14:53 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost ([127.0.0.1] helo=edge.pp.kiev.ua) by hosted.kievnet.com with esmtpa (Exim 4.62) (envelope-from ) id 1JxJRC-000H41-Af; Sat, 17 May 2008 13:14:50 +0300 Message-ID: <482EB019.3060503@icyb.net.ua> Date: Sat, 17 May 2008 13:14:49 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.12 (X11/20080320) MIME-Version: 1.0 To: Kostik Belousov References: <482E93C0.4070802@icyb.net.ua> <20080517101243.GH18958@deviant.kiev.zoral.com.ua> In-Reply-To: <20080517101243.GH18958@deviant.kiev.zoral.com.ua> Content-Type: text/plain; charset=KOI8-U; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: rdmsr from userspace X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 10:14:54 -0000 on 17/05/2008 13:12 Kostik Belousov said the following: > On Sat, May 17, 2008 at 11:13:52AM +0300, Andriy Gapon wrote: >> It seems that rdmsr instruction can be executed only at the highest >> privilege level and thus is not permitted from userland. Maybe we should >> provide something like Linux /dev/cpu/msr? >> I don't like interface of that device, I think that ioctl approach would >> be preferable in this case. >> Something like create /dev/cpuN and allow some ioctls on it: >> ioctl(cpu_fd, CPU_RDMSR, arg). >> What do you think? > > I believe ports/sysutils/devcpu is what you need. In fact, I think this > ought to be committed to the base. I do not know why Stas did not asked > for this, at least at public. Thanks a lot! This is exactly it. I agree, this would be quite useful in the OS. -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 10:27:08 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 929BF1065671 for ; Sat, 17 May 2008 10:27:08 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay01.kiev.sovam.com (relay01.kiev.sovam.com [62.64.120.200]) by mx1.freebsd.org (Postfix) with ESMTP id AF87B8FC1E for ; Sat, 17 May 2008 10:27:07 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=skuns.kiev.zoral.com.ua) by relay01.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1JxJd2-000OkF-P4 for freebsd-hackers@freebsd.org; Sat, 17 May 2008 13:27:05 +0300 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by skuns.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4HAQuSc070242 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 17 May 2008 13:26:56 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4HAQsGQ029677; Sat, 17 May 2008 13:26:54 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m4HAQr3C029676; Sat, 17 May 2008 13:26:53 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 17 May 2008 13:26:53 +0300 From: Kostik Belousov To: Jeremie Le Hen Message-ID: <20080517102653.GI18958@deviant.kiev.zoral.com.ua> References: <20080517091740.GI70896@obiwan.tataz.chchile.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="1Gc66+8b3xhjZJxW" Content-Disposition: inline In-Reply-To: <20080517091740.GI70896@obiwan.tataz.chchile.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.4 X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on skuns.kiev.zoral.com.ua X-Scanner-Signature: eee30a1c0c3b578efc84796d9b7c2682 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Header: Not Detected X-SpamTest-Info: Profiles 2835 [May 13 2008] X-SpamTest-Info: helo_type=3 X-SpamTest-Method: none X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0278], KAS30/Release Cc: freebsd-hackers@freebsd.org Subject: Re: Debugging rtld X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 10:27:08 -0000 --1Gc66+8b3xhjZJxW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, May 17, 2008 at 11:17:40AM +0200, Jeremie Le Hen wrote: > Hi there, >=20 > I tried to compile my source tree with -fstack-protector-all, and it > happens that rtld breaks with this: once the new rtld is installed every > single problem coredumps. I tried to compile rtld-elf without SSP, but > it didn't solve the problem. Then I had to compile libc_pic.a without > SSP and it worked, but I don't understand the root of the problem. > So I want to use the generated coredump for post-mortem analysis with > gdb. >=20 > I compiled world with DEBUG_FLAGS=3D-g. But GDB gives me a backtrace so > long that it can't be real. Moreoever it doesn't seem to bring in the > required symbols. I'm a GDB novice, so I'd like some help. >=20 > chroot> =3D=3D=3D> libexec/rtld-elf (install) > chroot> chflags noschg /usr/libexec/ld-elf.so.1 > chroot> install -s -o root -g wheel -m 555 -C -b -fschg -S ld-elf.so.1 /= libexec > chroot> install -o root -g wheel -m 444 rtld.1.gz /usr/share/man/man1 > chroot> *** Signal 11 > chroot> > chroot> jarjarbinks# cd /root; ls > chroot> Segmentation fault >=20 > host> jarjarbinks:145# ls -l /space/chroot/root/ls.core=20 > host> -rw------- 1 root wheel 184320 May 17 10:19 /space/chroot/root/l= s.core > host> jarjarbinks:149# gdb -c /space/chroot/root/ls.core -e /space/chroot= /bin/ls > host> GNU gdb 6.1.1 [FreeBSD] > host> [...] > host> This GDB was configured as "i386-marcel-freebsd". > host> Core was generated by `ls'. > host> Program terminated with signal 11, Segmentation fault. > host> #0 0x280583e4 in ?? () > host> (gdb) bt > host> #0 0x280583e4 in ?? () > host> #1 0x00000000 in ?? () > host> #2 0x00000000 in ?? () > host> #3 0x00000000 in ?? () > host> #4 0x00000000 in ?? () > host> #5 0x00000000 in ?? () > host> #6 0x00000000 in ?? () > host> #7 0x00000000 in ?? () > host> #8 0x00000000 in ?? () > host> #9 0x00000000 in ?? () > host> #10 0x00000000 in ?? () > host> #11 0xffffffff in ?? () > host> #12 0x00001000 in ?? () > host> [...] > host> #359 0x73763a68 in ?? () > host> #360 0x5b455c3d in ?? () > host> [...] > host> #855 0x00000000 in ?? () > host> [...] >=20 > Any hint on how to proceed would be welcome. I usually add the CFLAGS+=3D-g to the rtld-elf/Makefile. Also, you do not need to bring down the whole host by the broken ld.so.1. Do not install it at all, and specify the path to the rtld by the --dynamic-linker switch, see into ld. BTW, Solaris seems to support $ORIGIN in the interpreter path, but I feel that this is an overkill. --1Gc66+8b3xhjZJxW Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkgusu0ACgkQC3+MBN1Mb4jaqACgz6k2u1rOr9x6q16hFZ5DqH0h CDUAn1xbcoskey+CApBlFCS93/T95NiM =7+4r -----END PGP SIGNATURE----- --1Gc66+8b3xhjZJxW-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 14:40:42 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8C9801065671 for ; Sat, 17 May 2008 14:40:42 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from kientzle.com (h-66-166-149-50.snvacaid.covad.net [66.166.149.50]) by mx1.freebsd.org (Postfix) with ESMTP id 687188FC1D for ; Sat, 17 May 2008 14:40:42 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from [10.0.0.128] (p54.kientzle.com [66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id m4HEebtv043201; Sat, 17 May 2008 07:40:37 -0700 (PDT) (envelope-from kientzle@freebsd.org) Message-ID: <482EEE65.8070009@freebsd.org> Date: Sat, 17 May 2008 07:40:37 -0700 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060422 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Kostik Belousov References: <20080517091740.GI70896@obiwan.tataz.chchile.org> <20080517102653.GI18958@deviant.kiev.zoral.com.ua> In-Reply-To: <20080517102653.GI18958@deviant.kiev.zoral.com.ua> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Jeremie Le Hen Subject: Re: Debugging rtld X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 14:40:42 -0000 >>I tried to compile my source tree with -fstack-protector-all, and it >>happens that rtld breaks with this ... > > I usually add the CFLAGS+=-g to the rtld-elf/Makefile. Also, you do not > need to bring down the whole host by the broken ld.so.1. Do not install > it at all, and specify the path to the rtld by the --dynamic-linker switch, > see into ld. If you do (accidentally?) break ld.so.1, remember that /rescue is all statically linked and should still work. Single-user boot to /rescue/sh is your friend. Cheers, Tim Kientzle From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 14:41:24 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3DE9D106566B for ; Sat, 17 May 2008 14:41:24 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from kientzle.com (h-66-166-149-50.snvacaid.covad.net [66.166.149.50]) by mx1.freebsd.org (Postfix) with ESMTP id 15B498FC13 for ; Sat, 17 May 2008 14:41:24 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from [10.0.0.128] (p54.kientzle.com [66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id m4HEfHtv043213; Sat, 17 May 2008 07:41:21 -0700 (PDT) (envelope-from kientzle@freebsd.org) Message-ID: <482EEE8D.5000208@freebsd.org> Date: Sat, 17 May 2008 07:41:17 -0700 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060422 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Victor Hugo Bilouro References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: GSoC Student X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 14:41:24 -0000 Sounds interesting. I'm looking forward to hearing more about your work. Cheers, Tim Kientzle Victor Hugo Bilouro wrote: > Hi, > > I'm Victor Hugo *Bilouro*, GSoC participant. > > I'll be working on tcptest. It's name came from TCP/IP Regression test suite. > > As a testing tool it can: > *be a conformance test > *test a bug already fixed (regression test - one test for each fixed bug) > *test a new feature (help protocol developer) > *simulate certain cases > > It will be build on top of pcs.sf.net (Package Construction Set) > "PCS is a set of Python modules and objects that make building network > protocol code easier for the protocol developer. The core of the > system is the pcs module itself which provides the necessary > functionality to create classes that implement packets. " > > My mentor is George V. Neville-Neil, no comments. :-) > > GSoC started a month ago, by this time, I had almost all environment > done, and I'm spending several times reading and often playing with > pcs. (cool) > > Books i bought: > (*)TCP/IP Illustrated Volume 1 (far the best) Stevens > (*)TCP/IP Guide Charles M.Kazierok (www.tcpipguide.com) > (*)Internetworking with tcp/ip > > I will post on __link__ links, books, status report, futures releases > information, news, everything about the project. > > __link__: http://wiki.freebsd.org/VictorBilouro/TCP-IP_regression_test_suite > > > Regards From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 16:05:26 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 712741065672 for ; Sat, 17 May 2008 16:05:26 +0000 (UTC) (envelope-from rpaulo@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.156]) by mx1.freebsd.org (Postfix) with ESMTP id ECD388FC1D for ; Sat, 17 May 2008 16:05:25 +0000 (UTC) (envelope-from rpaulo@gmail.com) Received: by fg-out-1718.google.com with SMTP id l26so1382484fgb.35 for ; Sat, 17 May 2008 09:05:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; bh=MdTouOI2vdupALD+kwPYXv2+QeHN50T/DRR+kgu0VDU=; b=n4ZZS93EDKwB/PG60rUgEhmrLe3tSNQ26DnmJh+eevLQwnFM3jv3mKbyptKy0CDse3amLpTxUqWIuCPJOqfedqNuVYF3c9y1VqcSCnZGcDNAP07UgNrvWyn+0MX1TKSwsZOeHUwW8RpvqfeMWjMys+YP06jS204s4H8OksMfPxg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=At2mtHlTlPXv/sgCbB/ny3rV7pY2UvwSXrZXFsMjkgrl7b9QS+havMlLweXSZACa7jYlaxfug6CKmkqtnuawF6xD+Ti17J1sP0e130eVwJtvbRkqafR3o9odMxY7DJ3Qkzzm3GKP1ZM0zG/DY+JCr8m/gFOw71dJLPhauLgidGs= Received: by 10.210.113.16 with SMTP id l16mr4867906ebc.195.1211038630152; Sat, 17 May 2008 08:37:10 -0700 (PDT) Received: from epsilon.local ( [89.214.240.107]) by mx.google.com with ESMTPS id g17sm7461515nfd.3.2008.05.17.08.37.07 (version=SSLv3 cipher=RC4-MD5); Sat, 17 May 2008 08:37:08 -0700 (PDT) Message-ID: <482EFBA0.30107@FreeBSD.org> Date: Sat, 17 May 2008 16:37:04 +0100 From: Rui Paulo User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Andriy Gapon References: <482E93C0.4070802@icyb.net.ua> In-Reply-To: <482E93C0.4070802@icyb.net.ua> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: Rui Paulo Cc: freebsd-hackers@freebsd.org Subject: Re: rdmsr from userspace X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 16:05:26 -0000 Andriy Gapon wrote: > > It seems that rdmsr instruction can be executed only at the highest > privilege level and thus is not permitted from userland. Maybe we should > provide something like Linux /dev/cpu/msr? > I don't like interface of that device, I think that ioctl approach would > be preferable in this case. > Something like create /dev/cpuN and allow some ioctls on it: > ioctl(cpu_fd, CPU_RDMSR, arg). > What do you think? > While I think this (devcpu) is good for testing and development, I prefer having a device driver to handle that specific MSR than a generic /dev/cpuN where you can issue MSRs. Both for security and reliability reasons. Regards, -- Rui Paulo From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 17:10:48 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ECBCF106564A for ; Sat, 17 May 2008 17:10:48 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from hosted.kievnet.com (hosted.kievnet.com [193.138.144.10]) by mx1.freebsd.org (Postfix) with ESMTP id A5AE28FC16 for ; Sat, 17 May 2008 17:10:48 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost ([127.0.0.1] helo=edge.pp.kiev.ua) by hosted.kievnet.com with esmtpa (Exim 4.62) (envelope-from ) id 1JxPvi-000P6h-Ss; Sat, 17 May 2008 20:10:46 +0300 Message-ID: <482F1191.70709@icyb.net.ua> Date: Sat, 17 May 2008 20:10:41 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.12 (X11/20080320) MIME-Version: 1.0 To: Rui Paulo References: <482E93C0.4070802@icyb.net.ua> <482EFBA0.30107@FreeBSD.org> In-Reply-To: <482EFBA0.30107@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@FreeBSD.org Subject: Re: rdmsr from userspace X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 17:10:49 -0000 on 17/05/2008 18:37 Rui Paulo said the following: > Andriy Gapon wrote: >> >> It seems that rdmsr instruction can be executed only at the highest >> privilege level and thus is not permitted from userland. Maybe we >> should provide something like Linux /dev/cpu/msr? >> I don't like interface of that device, I think that ioctl approach >> would be preferable in this case. >> Something like create /dev/cpuN and allow some ioctls on it: >> ioctl(cpu_fd, CPU_RDMSR, arg). >> What do you think? >> > > While I think this (devcpu) is good for testing and development, I > prefer having a device driver to handle that specific MSR than a generic > /dev/cpuN where you can issue MSRs. > Both for security and reliability reasons. What about /dev/pci, /dev/io? Aren't they a precedent? -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 17:26:11 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3F7C7106566C for ; Sat, 17 May 2008 17:26:11 +0000 (UTC) (envelope-from rpaulo@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.186]) by mx1.freebsd.org (Postfix) with ESMTP id BEBCB8FC15 for ; Sat, 17 May 2008 17:26:10 +0000 (UTC) (envelope-from rpaulo@gmail.com) Received: by nf-out-0910.google.com with SMTP id h3so592850nfh.33 for ; Sat, 17 May 2008 10:26:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; bh=wAoZGNr2zL8JGNb7j6PdHkVKYx98BSeWUNsFut68hgc=; b=v8/UA6iJ3d8i2X9jOI+aAvk+WPGxMtVcw9x0l6s0Sfm/OWh/bkuUXI2NUOt+JBrwhPiyh9FQJyFuRXdkB9U/eYeK3WpWXfAs4P7JhBUvz4jOylQzVslf1qvZPJxTMU697KzqwaKrfdGudx62xzaMFE/xIERx8ZkcKP3yrneXhX8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=gL4d/zNgT0beALKHs7dPJ8VtrrDL83OPVrDH0SVdQTAKG6nD6qff/3pKA3DVOImuj+2ehdKByBDjtxXHskwwupKjVPJMdCxAOG7Z48gw+3WuwlsiAmtDwhqTsvUw02e+CE0JA70pz6fOzXre/S0e2zWPjRuVUowdx2tfPVFv+34= Received: by 10.210.45.17 with SMTP id s17mr5043617ebs.9.1211045169544; Sat, 17 May 2008 10:26:09 -0700 (PDT) Received: from epsilon.local ( [89.214.240.107]) by mx.google.com with ESMTPS id 2sm8332286nfv.5.2008.05.17.10.26.04 (version=SSLv3 cipher=RC4-MD5); Sat, 17 May 2008 10:26:05 -0700 (PDT) Message-ID: <482F1529.5080409@FreeBSD.org> Date: Sat, 17 May 2008 18:26:01 +0100 From: Rui Paulo User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Andriy Gapon References: <482E93C0.4070802@icyb.net.ua> <482EFBA0.30107@FreeBSD.org> <482F1191.70709@icyb.net.ua> In-Reply-To: <482F1191.70709@icyb.net.ua> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: Rui Paulo Cc: freebsd-hackers@FreeBSD.org Subject: Re: rdmsr from userspace X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 17:26:11 -0000 Andriy Gapon wrote: > on 17/05/2008 18:37 Rui Paulo said the following: >> Andriy Gapon wrote: >>> >>> It seems that rdmsr instruction can be executed only at the highest >>> privilege level and thus is not permitted from userland. Maybe we >>> should provide something like Linux /dev/cpu/msr? >>> I don't like interface of that device, I think that ioctl approach >>> would be preferable in this case. >>> Something like create /dev/cpuN and allow some ioctls on it: >>> ioctl(cpu_fd, CPU_RDMSR, arg). >>> What do you think? >>> >> >> While I think this (devcpu) is good for testing and development, I >> prefer having a device driver to handle that specific MSR than a >> generic /dev/cpuN where you can issue MSRs. >> Both for security and reliability reasons. > > What about /dev/pci, /dev/io? Aren't they a precedent? They are, but, IMHO, we should no longer continue to create this type of interfaces. Regards, -- Rui Paulo From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 17:30:13 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 052C9106564A for ; Sat, 17 May 2008 17:30:13 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (five.mired.org [66.92.153.75]) by mx1.freebsd.org (Postfix) with ESMTP id 83D8F8FC17 for ; Sat, 17 May 2008 17:30:12 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 47811 invoked by uid 1001); 17 May 2008 13:30:38 -0400 Received: from bhuda.mired.org (bhuda [192.168.195.1]) by bhuda (tmda-ofmipd) with ESMTP; Sat, 17 May 2008 13:30:38 -0400 Date: Sat, 17 May 2008 13:30:37 -0400 Cc: freebsd-hackers@freebsd.org Message-ID: <20080517133037.3a3935db@bhuda.mired.org> In-Reply-To: <482E93C0.4070802@icyb.net.ua> References: <482E93C0.4070802@icyb.net.ua> Organization: Meyer Consulting X-Mailer: Claws Mail 3.4.0 (GTK+ 2.12.9; amd64-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Delivery-Agent: TMDA/1.1.12 (Macallan) From: Mike Meyer Subject: Re: rdmsr from userspace X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 17:30:13 -0000 On Sat, 17 May 2008 11:13:52 +0300 Andriy Gapon wrote: > It seems that rdmsr instruction can be executed only at the highest > privilege level and thus is not permitted from userland. Maybe we should > provide something like Linux /dev/cpu/msr? > I don't like interface of that device, I think that ioctl approach would > be preferable in this case. > Something like create /dev/cpuN and allow some ioctls on it: > ioctl(cpu_fd, CPU_RDMSR, arg). > What do you think? Ok, this points directly at a question I've been wondering about, but haven't been able to find an answer in the google. I've been mucking about with general access to sysctl's (a sysctl plugin for gkrellm, and a python module for accessing sysctls), and with that hammer in my hand, the nail for this problem is obviously a dev.cpu.#.msr sysctl. Except that this would be harder to use from languages that don't provide direct access to libc functions than a file, with or without an ioctl (ioctl's are at at least POSIX, which is a functionality level a lot of languages aspire to), and also easier to manipulate with standard Unix tools in general. However, I thought I'd sense some hostility towards /dev/proc-like things from the freebsd community. Which is where my questions come in: Am I imagining that? If not, is there a real basis for it - and what would that be? Thanks, http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 17:32:03 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4163E106564A for ; Sat, 17 May 2008 17:32:03 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (five.mired.org [66.92.153.75]) by mx1.freebsd.org (Postfix) with ESMTP id BEBEF8FC12 for ; Sat, 17 May 2008 17:32:02 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 47893 invoked by uid 1001); 17 May 2008 13:32:29 -0400 Received: from bhuda.mired.org (bhuda [192.168.195.1]) by bhuda (tmda-ofmipd) with ESMTP; Sat, 17 May 2008 13:32:29 -0400 Date: Sat, 17 May 2008 13:32:28 -0400 To: Rui Paulo Message-ID: <20080517133228.02c9ea5c@bhuda.mired.org> In-Reply-To: <482F1529.5080409@FreeBSD.org> References: <482E93C0.4070802@icyb.net.ua> <482EFBA0.30107@FreeBSD.org> <482F1191.70709@icyb.net.ua> <482F1529.5080409@FreeBSD.org> Organization: Meyer Consulting X-Mailer: Claws Mail 3.4.0 (GTK+ 2.12.9; amd64-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Delivery-Agent: TMDA/1.1.12 (Macallan) From: Mike Meyer Cc: freebsd-hackers@FreeBSD.org, Andriy Gapon Subject: Re: rdmsr from userspace X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 17:32:03 -0000 On Sat, 17 May 2008 18:26:01 +0100 Rui Paulo wrote: > Andriy Gapon wrote: > > on 17/05/2008 18:37 Rui Paulo said the following: > >> Andriy Gapon wrote: > >>> > >>> It seems that rdmsr instruction can be executed only at the highest > >>> privilege level and thus is not permitted from userland. Maybe we > >>> should provide something like Linux /dev/cpu/msr? > >>> I don't like interface of that device, I think that ioctl approach > >>> would be preferable in this case. > >>> Something like create /dev/cpuN and allow some ioctls on it: > >>> ioctl(cpu_fd, CPU_RDMSR, arg). > >>> What do you think? > >>> > >> > >> While I think this (devcpu) is good for testing and development, I > >> prefer having a device driver to handle that specific MSR than a > >> generic /dev/cpuN where you can issue MSRs. > >> Both for security and reliability reasons. > > > > What about /dev/pci, /dev/io? Aren't they a precedent? > > They are, but, IMHO, we should no longer continue to create this type of > interfaces. Ok, in relation to the question I asked about sysctl's vs. /dev/* - why not? http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 17:39:04 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A0D2D1065686 for ; Sat, 17 May 2008 17:39:04 +0000 (UTC) (envelope-from tataz@tataz.chchile.org) Received: from smtp5-g19.free.fr (smtp5-g19.free.fr [212.27.42.35]) by mx1.freebsd.org (Postfix) with ESMTP id 37E6A8FC23 for ; Sat, 17 May 2008 17:39:04 +0000 (UTC) (envelope-from tataz@tataz.chchile.org) Received: from smtp5-g19.free.fr (localhost.localdomain [127.0.0.1]) by smtp5-g19.free.fr (Postfix) with ESMTP id 65B013F6157; Sat, 17 May 2008 19:39:02 +0200 (CEST) Received: from tatooine.tataz.chchile.org (tataz.chchile.org [82.233.239.98]) by smtp5-g19.free.fr (Postfix) with ESMTP id CCDAB3F634F; Sat, 17 May 2008 19:38:19 +0200 (CEST) Received: from obiwan.tataz.chchile.org (unknown [192.168.1.25]) by tatooine.tataz.chchile.org (Postfix) with ESMTP id E88C49F2EA; Sat, 17 May 2008 17:35:25 +0000 (UTC) Received: by obiwan.tataz.chchile.org (Postfix, from userid 1000) id DAE6F4089; Sat, 17 May 2008 19:35:25 +0200 (CEST) Date: Sat, 17 May 2008 19:35:25 +0200 From: Jeremie Le Hen To: Kostik Belousov Message-ID: <20080517173525.GJ70896@obiwan.tataz.chchile.org> References: <20080517091740.GI70896@obiwan.tataz.chchile.org> <20080517102653.GI18958@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080517102653.GI18958@deviant.kiev.zoral.com.ua> User-Agent: Mutt/1.5.15 (2007-04-06) Cc: freebsd-hackers@freebsd.org Subject: Re: Debugging rtld X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 17:39:04 -0000 Hi, On Sat, May 17, 2008 at 01:26:53PM +0300, Kostik Belousov wrote: > On Sat, May 17, 2008 at 11:17:40AM +0200, Jeremie Le Hen wrote: > > I tried to compile my source tree with -fstack-protector-all, and it > > happens that rtld breaks with this: once the new rtld is installed every > > single problem coredumps. I tried to compile rtld-elf without SSP, but > > it didn't solve the problem. Then I had to compile libc_pic.a without > > SSP and it worked, but I don't understand the root of the problem. > > So I want to use the generated coredump for post-mortem analysis with > > gdb. > > > > I compiled world with DEBUG_FLAGS=-g. But GDB gives me a backtrace so > > long that it can't be real. Moreoever it doesn't seem to bring in the > > required symbols. I'm a GDB novice, so I'd like some help. > > > > chroot> ===> libexec/rtld-elf (install) > > chroot> chflags noschg /usr/libexec/ld-elf.so.1 > > chroot> install -s -o root -g wheel -m 555 -C -b -fschg -S ld-elf.so.1 /libexec > > chroot> install -o root -g wheel -m 444 rtld.1.gz /usr/share/man/man1 > > chroot> *** Signal 11 > > chroot> > > chroot> jarjarbinks# cd /root; ls > > chroot> Segmentation fault > > > > host> jarjarbinks:145# ls -l /space/chroot/root/ls.core > > host> -rw------- 1 root wheel 184320 May 17 10:19 /space/chroot/root/ls.core > > host> jarjarbinks:149# gdb -c /space/chroot/root/ls.core -e /space/chroot/bin/ls > > host> GNU gdb 6.1.1 [FreeBSD] > > host> [...] > > host> This GDB was configured as "i386-marcel-freebsd". > > host> Core was generated by `ls'. > > host> Program terminated with signal 11, Segmentation fault. > > host> #0 0x280583e4 in ?? () > > host> (gdb) bt > > host> #0 0x280583e4 in ?? () > > host> #1 0x00000000 in ?? () > > host> #2 0x00000000 in ?? () > > host> #3 0x00000000 in ?? () > > host> #4 0x00000000 in ?? () > > host> #5 0x00000000 in ?? () > > host> #6 0x00000000 in ?? () > > host> #7 0x00000000 in ?? () > > host> #8 0x00000000 in ?? () > > host> #9 0x00000000 in ?? () > > host> #10 0x00000000 in ?? () > > host> #11 0xffffffff in ?? () > > host> #12 0x00001000 in ?? () > > host> [...] > > host> #359 0x73763a68 in ?? () > > host> #360 0x5b455c3d in ?? () > > host> [...] > > host> #855 0x00000000 in ?? () > > host> [...] > > > > Any hint on how to proceed would be welcome. > > I usually add the CFLAGS+=-g to the rtld-elf/Makefile. Also, you do not > need to bring down the whole host by the broken ld.so.1. Do not install > it at all, and specify the path to the rtld by the --dynamic-linker switch, > see into ld. Thank you for this tip. However the backtrace is still unusable. I've recompiled libc_pic.a with -g, then rtld-elf with -g and finally /bin/ls with you tip and -g. I'm really brought to a standstill here. Thank you. Best regards, -- Jeremie Le Hen < jeremie at le-hen dot org >< ttz at chchile dot org > From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 17:46:47 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 13AC6106564A for ; Sat, 17 May 2008 17:46:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay03.kiev.sovam.com (relay03.kiev.sovam.com [62.64.120.201]) by mx1.freebsd.org (Postfix) with ESMTP id 98F548FC1B for ; Sat, 17 May 2008 17:46:46 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=skuns.kiev.zoral.com.ua) by relay03.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1JxQUW-0002qg-TF for freebsd-hackers@freebsd.org; Sat, 17 May 2008 20:46:45 +0300 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by skuns.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4HHkdZD083207 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 17 May 2008 20:46:39 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4HHkbSM088931; Sat, 17 May 2008 20:46:37 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m4HHkbaV088930; Sat, 17 May 2008 20:46:37 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 17 May 2008 20:46:37 +0300 From: Kostik Belousov To: Jeremie Le Hen Message-ID: <20080517174637.GL18958@deviant.kiev.zoral.com.ua> References: <20080517091740.GI70896@obiwan.tataz.chchile.org> <20080517102653.GI18958@deviant.kiev.zoral.com.ua> <20080517173525.GJ70896@obiwan.tataz.chchile.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="hYef4tn27tWYDrFX" Content-Disposition: inline In-Reply-To: <20080517173525.GJ70896@obiwan.tataz.chchile.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.4 X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on skuns.kiev.zoral.com.ua X-Scanner-Signature: 6b869f9e50b0a12d7314c849f37d9376 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Header: Not Detected X-SpamTest-Info: Profiles 2835 [May 13 2008] X-SpamTest-Info: helo_type=3 X-SpamTest-Method: none X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0278], KAS30/Release Cc: freebsd-hackers@freebsd.org Subject: Re: Debugging rtld X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 17:46:47 -0000 --hYef4tn27tWYDrFX Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, May 17, 2008 at 07:35:25PM +0200, Jeremie Le Hen wrote: > Hi, >=20 > On Sat, May 17, 2008 at 01:26:53PM +0300, Kostik Belousov wrote: > > On Sat, May 17, 2008 at 11:17:40AM +0200, Jeremie Le Hen wrote: > > > I tried to compile my source tree with -fstack-protector-all, and it > > > happens that rtld breaks with this: once the new rtld is installed ev= ery > > > single problem coredumps. I tried to compile rtld-elf without SSP, b= ut > > > it didn't solve the problem. Then I had to compile libc_pic.a without > > > SSP and it worked, but I don't understand the root of the problem. > > > So I want to use the generated coredump for post-mortem analysis with > > > gdb. > > >=20 > > > I compiled world with DEBUG_FLAGS=3D-g. But GDB gives me a backtrace= so > > > long that it can't be real. Moreoever it doesn't seem to bring in the > > > required symbols. I'm a GDB novice, so I'd like some help. > > >=20 > > > chroot> =3D=3D=3D> libexec/rtld-elf (install) > > > chroot> chflags noschg /usr/libexec/ld-elf.so.1 > > > chroot> install -s -o root -g wheel -m 555 -C -b -fschg -S ld-elf.so= .1 /libexec > > > chroot> install -o root -g wheel -m 444 rtld.1.gz /usr/share/man/man1 > > > chroot> *** Signal 11 > > > chroot> > > > chroot> jarjarbinks# cd /root; ls > > > chroot> Segmentation fault > > >=20 > > > host> jarjarbinks:145# ls -l /space/chroot/root/ls.core=20 > > > host> -rw------- 1 root wheel 184320 May 17 10:19 /space/chroot/ro= ot/ls.core > > > host> jarjarbinks:149# gdb -c /space/chroot/root/ls.core -e /space/ch= root/bin/ls > > > host> GNU gdb 6.1.1 [FreeBSD] > > > host> [...] > > > host> This GDB was configured as "i386-marcel-freebsd". > > > host> Core was generated by `ls'. > > > host> Program terminated with signal 11, Segmentation fault. > > > host> #0 0x280583e4 in ?? () > > > host> (gdb) bt > > > host> #0 0x280583e4 in ?? () > > > host> #1 0x00000000 in ?? () > > > host> #2 0x00000000 in ?? () > > > host> #3 0x00000000 in ?? () > > > host> #4 0x00000000 in ?? () > > > host> #5 0x00000000 in ?? () > > > host> #6 0x00000000 in ?? () > > > host> #7 0x00000000 in ?? () > > > host> #8 0x00000000 in ?? () > > > host> #9 0x00000000 in ?? () > > > host> #10 0x00000000 in ?? () > > > host> #11 0xffffffff in ?? () > > > host> #12 0x00001000 in ?? () > > > host> [...] > > > host> #359 0x73763a68 in ?? () > > > host> #360 0x5b455c3d in ?? () > > > host> [...] > > > host> #855 0x00000000 in ?? () > > > host> [...] > > >=20 > > > Any hint on how to proceed would be welcome. > >=20 > > I usually add the CFLAGS+=3D-g to the rtld-elf/Makefile. Also, you do n= ot > > need to bring down the whole host by the broken ld.so.1. Do not install > > it at all, and specify the path to the rtld by the --dynamic-linker swi= tch, > > see into ld. Hmm, ^^^^ info >=20 > Thank you for this tip. However the backtrace is still unusable. > I've recompiled libc_pic.a with -g, then rtld-elf with -g and finally > /bin/ls with you tip and -g. >=20 > I'm really brought to a standstill here. Looks like you have a stack corruption, that is reasonable given the matters you touching. The easiest, althought somewhat time-consuming way of searching the point where the things break is to insert some break into the code of the rtld, "int3" may be good, and moving it forward until you start hitting the breakage instead of the breakpoint. --hYef4tn27tWYDrFX Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkgvGfwACgkQC3+MBN1Mb4hcnwCaAu3itaIGZJtLSR0HmmfZIfy6 TVYAn1+rlIS+nJ4ivHjN8rameBJNiMbj =72Jk -----END PGP SIGNATURE----- --hYef4tn27tWYDrFX-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 17:53:23 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F926106564A for ; Sat, 17 May 2008 17:53:23 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay03.kiev.sovam.com (relay03.kiev.sovam.com [62.64.120.201]) by mx1.freebsd.org (Postfix) with ESMTP id 026918FC15 for ; Sat, 17 May 2008 17:53:22 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=skuns.kiev.zoral.com.ua) by relay03.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1JxQav-00043Z-Kf; Sat, 17 May 2008 20:53:21 +0300 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by skuns.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4HHrEId083294 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 17 May 2008 20:53:14 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m4HHrCni089181; Sat, 17 May 2008 20:53:12 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m4HHrCXA089180; Sat, 17 May 2008 20:53:12 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 17 May 2008 20:53:12 +0300 From: Kostik Belousov To: Rui Paulo Message-ID: <20080517175312.GM18958@deviant.kiev.zoral.com.ua> References: <482E93C0.4070802@icyb.net.ua> <482EFBA0.30107@FreeBSD.org> <482F1191.70709@icyb.net.ua> <482F1529.5080409@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="jva9z9iU4vGix/sz" Content-Disposition: inline In-Reply-To: <482F1529.5080409@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.4 X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on skuns.kiev.zoral.com.ua X-Scanner-Signature: fa718887f43e9e3967a9d10f43843ba4 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Header: Not Detected X-SpamTest-Info: Profiles 2835 [May 13 2008] X-SpamTest-Info: helo_type=3 X-SpamTest-Method: none X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0278], KAS30/Release Cc: freebsd-hackers@freebsd.org, Andriy Gapon Subject: Re: rdmsr from userspace X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 17:53:23 -0000 --jva9z9iU4vGix/sz Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, May 17, 2008 at 06:26:01PM +0100, Rui Paulo wrote: > Andriy Gapon wrote: > >on 17/05/2008 18:37 Rui Paulo said the following: > >>Andriy Gapon wrote: > >>> > >>>It seems that rdmsr instruction can be executed only at the highest=20 > >>>privilege level and thus is not permitted from userland. Maybe we=20 > >>>should provide something like Linux /dev/cpu/msr? > >>>I don't like interface of that device, I think that ioctl approach=20 > >>>would be preferable in this case. > >>>Something like create /dev/cpuN and allow some ioctls on it:=20 > >>>ioctl(cpu_fd, CPU_RDMSR, arg). > >>>What do you think? > >>> > >> > >>While I think this (devcpu) is good for testing and development, I=20 > >>prefer having a device driver to handle that specific MSR than a=20 > >>generic /dev/cpuN where you can issue MSRs. > >>Both for security and reliability reasons. > > > >What about /dev/pci, /dev/io? Aren't they a precedent? >=20 > They are, but, IMHO, we should no longer continue to create this type of= =20 > interfaces. Why ? Are developers some kind of the second-class users ? I would have no opinion on providing /dev/cpu by the loadable module, not compiled into GENERIC. But the interface itself is useful at least for three things: - CPU identification (see x86info or whatever it is called); - CPU tweaking for bugs workaround without patching the kernel; - updating the CPU microcode. None of these is limited to the developers only. I am interested why Stanislav still did not submitted it for inclusion into the base still. Maybe, some other reasons exist. --jva9z9iU4vGix/sz Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkgvG4gACgkQC3+MBN1Mb4iodgCcCKViPJjwM7qHV/sFSDZOK/zS OQkAoN6vNYhb9CltjRgoBAU3fkNyP4Rt =1Z2a -----END PGP SIGNATURE----- --jva9z9iU4vGix/sz-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 17 21:26:14 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 355B4106564A for ; Sat, 17 May 2008 21:26:14 +0000 (UTC) (envelope-from tataz@tataz.chchile.org) Received: from smtp5-g19.free.fr (smtp5-g19.free.fr [212.27.42.35]) by mx1.freebsd.org (Postfix) with ESMTP id ECA3A8FC17 for ; Sat, 17 May 2008 21:26:13 +0000 (UTC) (envelope-from tataz@tataz.chchile.org) Received: from smtp5-g19.free.fr (localhost.localdomain [127.0.0.1]) by smtp5-g19.free.fr (Postfix) with ESMTP id 68CE63F6314; Sat, 17 May 2008 23:26:12 +0200 (CEST) Received: from tatooine.tataz.chchile.org (tataz.chchile.org [82.233.239.98]) by smtp5-g19.free.fr (Postfix) with ESMTP id 525833F6310; Sat, 17 May 2008 23:26:12 +0200 (CEST) Received: from obiwan.tataz.chchile.org (unknown [192.168.1.25]) by tatooine.tataz.chchile.org (Postfix) with ESMTP id 37E649F288; Sat, 17 May 2008 21:23:18 +0000 (UTC) Received: by obiwan.tataz.chchile.org (Postfix, from userid 1000) id 2D5194089; Sat, 17 May 2008 23:23:18 +0200 (CEST) Date: Sat, 17 May 2008 23:23:18 +0200 From: Jeremie Le Hen To: "Igor A. Valcov" Message-ID: <20080517212318.GK70896@obiwan.tataz.chchile.org> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.15 (2007-04-06) Cc: freebsd-hackers@freebsd.org Subject: Re: do not work nested unnamed anchor X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 May 2008 21:26:14 -0000 Hi Igor, On Fri, May 09, 2008 at 04:55:23PM +0400, Igor A. Valcov wrote: > Hello. > > For example: > > ==== pf.conf ==== > > ext_if="xl0" > ip_world="nn.nn.nn.nn" > > # Filter rules > block log all > > anchor in on $ext_if { > pass quick proto tcp to $ip_world port 22 keep state > # SSH > pass quick proto tcp to $ip_world port 25 keep state > # SMTP > pass quick proto tcp to $ip_world port 110 keep state > # POP3 > anchor { > pass quick proto tcp to $ip_world port 995 keep state > # POP3S > } > } > > ============ > > nmap results: > > PORT STATE SERVICE VERSION > 22/tcp open ssh OpenSSH 4.5p1 (FreeBSD 20061110; protocol 2.0) > 25/tcp open smtp? > 110/tcp open pop3 Openwall popa3d > > > I can not understand what the problem... > > FreeBSD-7.0-RELEASE-p1 > i386 You should ask this on pf mailing-list [1]. freebsd-hackers@ is not the right place for this, freebsd-net@ or freebsd-pf@ would have been far more better. [1] http://www.benzedrine.cx/mailinglist.html Thank you. Best regards, -- Jeremie Le Hen < jeremie at le-hen dot org >< ttz at chchile dot org >