From owner-freebsd-announce@FreeBSD.ORG Mon Dec 20 11:31:47 2004 Return-Path: Delivered-To: freebsd-announce@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 79EB316A4CE for ; Mon, 20 Dec 2004 11:31:47 +0000 (GMT) Received: from critter.freebsd.dk (f170.freebsd.dk [212.242.86.170]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4251B43D31 for ; Mon, 20 Dec 2004 11:31:46 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.1/8.13.1) with ESMTP id iBKBViCW053827 for ; Mon, 20 Dec 2004 12:31:44 +0100 (CET) (envelope-from phk@critter.freebsd.dk) To: announce@freebsd.org From: Poul-Henning Kamp Date: Mon, 20 Dec 2004 12:31:44 +0100 Message-ID: <53826.1103542304@critter.freebsd.dk> Sender: phk@critter.freebsd.dk X-Mailman-Approved-At: Mon, 20 Dec 2004 12:49:24 +0000 Subject: [FreeBSD-Announce] Final report on funded FreeBSD development project X-BeenThere: freebsd-announce@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Project Announcements [moderated] List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Dec 2004 11:31:47 -0000 Final report on funded FreeBSD development project [1] ------------------------------------------------------ As promised, here follows the status report on how my six months of user-community funded FreeBSD development went. The goal of this effort was to get as far in the "buf-junta" plans (http://people.freebsd.org/~phk/plan.html) as possible, and as I have outlined below, a significant dent was made in that list over the last six months. I would like to thank 187 persons and companies for, though their donations small and large, making this work possible. The future of FreeBSD is very much in the hands of the developers, and by helping this particular developer get his hands free for half a year certainly moved a lot of stuff from the future into the present. If you, my donors, are not happy with what you got out of your donations, the fault is entirely mine. (and I would like you to tell me about it as fast as possible.) Please do not fault the FreeBSD project as such or the FreeBSD Foundation for that matter, if this was not a success, it is all my personal fault. If on the other hand you think you got your moneys worth and you would be willing to help support FreeBSD development financially also in the future, then, and only then, will I declare my project to have been a complete success. Feedback either way is most welcome! The future ---------- A lot of people have asked me "what about the future ?" I promised them to make up my mind and let them know in this report: I have decided to not do another fundraising drive at this time. There are a number of reasons for this. The primary reason is that I do not want to be stuck in an ivory tower doing only the FreeBSD work I can see from my lab in the basement. I need to get out amongst people and get inspiration and input from real-world usage of FreeBSD. To that end I hope to land a couple of contracts currently under discussion and occupy half of my time this way. The other reason is that I hope the FreeBSD Foundation will be the place where this kind of fundraising happens. Channeling the funds through the FreeBSD Foundation means that this will not grow into a personality cult around my person and that money can be spent where it is most needed rather than on where I happen to fancy spending it. The minor reason is that the paperwork associated with fundraising is not a favourite hobby of mine and I'm still not done resolving various sticky details [4] But since there are still a lot of work to be done in this area of the kernel and since that is what I think I am best at, I have sent a proposal to the FreeBSD Foundation for funding me part time for a period to continue working on this stuff. I was somewhat late in sending my proposal to the Foundation, and have not yet received an answer from them, so the outcome of this is not yet known. If it falls through, I will find a couple of contracts for now, and then reconsider the situation later in the 2005. If you want to support FreeBSD development financially, the way to do it right now is to donate money to the FreeBSD Foundation. If you do so, some of the money may or may not end up funding my work, either way, it will be money well spent. High-lights from my personal score-card: ---------------------------------------- June ... October -- TTY code/driver cleanup In order to not clash with the RELENG_5 release work, I attacked the tty code/drivers in order to try to drive Giant out of the pty driver. I found that a massive amount of code which should have lived in the generic tty code was copy&pasted into almost every and all tty device drivers. In the end I did not manage to do any of the actual locking work, but I did get a lot of preparation for it done: I eliminated the 3100 lines of copy&paste'd code so that there will be many fewer places where locking will have to be handled (and bringing a lot of consistency to our tty drivers as a side effect). I also managed to get struct tty properly reference counted, but some sticky issues about unloading tty drivers with dangling sessions still need to be handled. June ... November -- Device and Fifo vnode bypass In terms of architecture, this is one of the most significant changes I have made in the last six months. Despite being a relatively small change in terms of number of lines of code changed, this project was spread out over a long time in order to be sure to catch all the little details in the corners. One of my test machines have been more or less dedicated to to this subproject for 5 months. The end result is that access to devices and fifos ('named pipes"), from programs now does not go through the vnode layer for the "traffic" calls, read, write & ioctl. (For device drivers we may still need Giant if the driver is marked as needing this.) August -- Rewrite Floppy driver. The internal state engine was rewritten and the driver made INTR_FAST, Giant safe and GEOM aware. September ... October -- DEVFS/SPECFS consolidation Devices are now only supported in the devfs filesystem, and supporting code was removed from ufs, ext2fs, cd9660 and a few minor filesystems. In practice this have had very little impact when seen from outside the kernel, but the simplifications in the kernel are extensive and very significant. The end result is that specfs is retired and only devfs handles devices now. A significant part of this work has been to properly reference count access into device drivers in order to make it safe(r) to unload device drivers. There are still many hurdles here before it is 100% safe to unload a driver, but at least now the cdevsw interface should not be one of them. October ... [ongoing] -- struct buf/vnode -> buf/bufobj/vnode This is also one of the architecturally very significant changes which had its fingers all over filesystems and buffer cache code. The work is not carried all the way through, but has progressed sufficiently far to reap the first major benefits (see below). It is hard to fairly summarize this stuff without it turning into a long architectural monologue, but the gist of it is that the aspect of vnods which serve as the attachment point for stuff in the buffer cache has been given an indenpendent data structure (bufobj) which in the future will not be tied to vnodes only. The untangling has reached and stopped right in front of the syncer code, and will continue from there as time permits. October -- Move copyonwrite and prewrite into FFS where they belong. These two functions are private to snapshots and softupdates, and with a private strategy implementation in FFS they can be isolated there entirely. This simplifies the buffer cache code a bit. October -- Proper waiting for pending geom events A process which caused events to be created in geom will now wait for them to be completed before returning to userland. This removed an insufficient lot of explicit waits. October ... --- Put local filesystems directly on GEOM Another piece of significant architecture [2] Instead of taking a tour through the vnode layer, DEVFS and geom_dev, go directly to the new Geom class geom_vfs to access disk devices for local filesystems. Amongst many other benefits, this gives us correct read/write tracking on devices, and it is now possible to mount the same filesystem r/o many places. Together with the device vnode bypass, this entirely takes I/O traffic from devices out of the vnode layer (and more significantly: vnode locking). November ... December -- FILEDESC locking In order to make the locking of file descriptors work better with devices and fifos bypassing vnodes, the locking was reworked and struct filedesc got a reference count more for protecting the memory allocation. November ... December -- omount(2) -> nmount(2) kernel transition. All filesystems converted to implement the nmount(2) API now. Calls to the old omount(2) API are converted in a special compatibility function in the filesystems. Userland conversion to nmount(2) should happen in the spring so that RELENG_6 will be entirely nmount(2) but still support omount(2) calls. Support for omount(2) will be dropped in RELENG_7. December -- Make VOP_* implementations typesafe. By using sparse struct initialization, it was possible to avoid a lot of the magic pointer gymnastics in creation of the VOP method vectors for filesystems. The result is typesafe and faster VOP_ method calling. December -- Root filesystem mount rework Since all filesystems now accept nmount(2) calls, the magic code mounting the root filesystem was cleaned up, removing the need for magic per-filesystem code to handle the root file system. This eliminated the main source of "bogo-vnodes" in the system, leaving only the syncer-vnodes now. June ... December -- Bugs introduced, code broken. Throughout a number of mistakes, minunderstandings, keyboard misfires and just plain stupidity on my part have introduced bugs and broken code. This will be fixed as soon and as fast as I can. Pointless statistics: --------------------- In the mock sprit of sports pages everywhere, and general journalistic "that was the year that was" reporting approaching the end of a year: Total donations: 33336 USD Average donation (sans outliers): 72 USD Total number of commits to CVS: 577 Cost per commit: 58 USD Average time between commits to CVS: 7h36m Number of submits to p4: 996 (minium ?) Average time between submits to p4: 4h25m Netto lines of code added, not counting retired device drivers: -7291 Pointy hats awarded: 3 Cost per codeline: -.22 USD Disks crashed: 3 Super strong fridge magnets retrieved: 6 Shoulder injuries: 1 Electricity used by computers 5275 kWh Successful attempts to communicate non-criminal status to Paypal: 0 Futile attempts to communicate non-criminal status to Paypal: 10 Random acts of kindness received: 3 Random acts of kindness not delivered due to botch by amazon.com: 3 [3] Merry X-mas & happy new-year, Poul-Henning [1] For those of you who don't know what I'm talking about in this email, a quick visit to http://people.freebsd.org/~phk/funding.html would be a good idea. [2] I rather fear I sound as boring as the tourist guides on the ruins of Akropolis, my apologies. [3] Amazon.com had a wrong shipping address on my "wish-list", so at least three packages of gifts from kind people have travelled all the way from Amazon to my old address and back again. I'm told people will get a full refund from Amazon once the package makes it back to them. My apologies for this mess. Much appreciated anyway. [4] Paypal is very well and nice, but trying to get through their CMS system to explain to them that just because you are located in the EU and received money doesn't make you a member of the mafia is still not an art I have managed. If you happen to know an email address of a senior executive let me know. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-announce@FreeBSD.ORG Tue Dec 21 12:07:44 2004 Return-Path: Delivered-To: freebsd-announce@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B47C116A4CE for ; Tue, 21 Dec 2004 12:07:44 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 25B9D43D46 for ; Tue, 21 Dec 2004 12:07:44 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.13.1/8.13.1) with ESMTP id iBLC4gtr084393 for ; Tue, 21 Dec 2004 07:04:43 -0500 (EST) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)iBLC4gG1084390 for ; Tue, 21 Dec 2004 12:04:42 GMT (envelope-from robert@fledge.watson.org) Date: Tue, 21 Dec 2004 12:04:42 +0000 (GMT) From: Robert Watson X-Sender: robert@fledge.watson.org To: freebsd-announce@FreeBSD.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Mailman-Approved-At: Tue, 21 Dec 2004 12:55:05 +0000 Subject: [FreeBSD-Announce] FreeBSD Foundation Quarterly Newsletter, December 21, 2004 X-BeenThere: freebsd-announce@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Project Announcements [moderated] List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Dec 2004 12:07:44 -0000 The FreeBSD Foundation Quarterly Newsletter December 21, 2004 In this Edition: o Letter from the Secretary/Treasurer o An update on the FreeBSD Foundation's 501(c)3 Status o A Roundup of Recent Foundation Activities - Event Sponsorship - PAE/BusDMA/PCIe Hardware grant - The FreeBSD Performance Cluster - FreeBSD Foundation Web Site o The FreeBSD Foundation acquires the FreeBSD Trademark o An Update on the Netperf Project o The State of Java Runtime Licensing o New Proposal Submission Process o Future Projects o Foundation Financial Data Letter From the Secretary/Treasurer =================================== FreeBSD Community, For almost five years now, the FreeBSD Foundation has been working quietly to support the FreeBSD project. Many of you may be aware of the FreeBSD Foundation, but not know much about our activities. This quarterly newsletter, along with a revamped web site debuting on January 1st, marks a renewed effort to educate the FreeBSD community about our efforts. This first newsletter includes important information about the FreeBSD Foundation's current status as a public-charity and the steps you can take to help us keep that status. Even if you don't have time to read the whole newsletter, please take the time to read and understand this section. The remainder of the newsletter is devoted to answering the question we get most often, "What does the FreeBSD Foundation do?" From financial data to status updates on recent and planned projects, we hope you'll find the answer to all of your questions in the newsletter. If you don't find what you're looking here or on our web site, please don't hesitate to contact us via email. Thanks for your support! Justin T. Gibbs Secretary/Treasurer gibbs@FreeBSDFoundation.org FreeBSD Foundation Board of Directors bod@FreeBSDFoundation.org The FreeBSD Foundation's 501(c)3 Status ======================================= December 31st of this year will mark the end of the FreeBSD Foundation's probationary period as a 501(c)3. The IRS requires all non-profit public charities to prove, at the five year mark, that they have met the "public support test" or be classified as a private foundation. For several reasons that are too technical to go into, it is in the Foundation's interest to remain a public charity. The easiest way for the Foundation to demonstrate "public support" is to pass the "1/3rd test": Limited Donations 1 ================== = === All Donations 3 For the purposes of this test, no single donor may contribute more than 2% of the total donation pool to the numerator. Through good fortune and the extreme charity of a handful of donors, the FreeBSD Foundation finds itself at risk of failing the "1/3rd test". As of today, the ratio of limited donations to total donations stands at 27.9%. While we have every confidence that our support history will substantiate that the FreeBSD Foundation is a public charity, going through the appeals process can be a lengthy and expensive ordeal. For this reason we are directly appealing to the community to help us meet the "1/3rd test". To do so, the Foundation will need to raise $30,400 in donations not affected by the 2% contribution limit. Given the Foundation's expected donation balance, the 2% limit for any single person or entity is $8000 given during the Foundation's 5 year probationary period. If you use FreeBSD and would like the FreeBSD Foundation to continue its support of FreeBSD, please consider making a donation. No amount is too small to make a difference. The FreeBSD Foundation accepts donations via check: The FreeBSD Foundation 7321 Brockway Dr. Boulder, CO 80303 or PayPal: http://www.freebsdfoundation.org/donating.shtml Checks must be postmarked by December 31st. to count toward the advanced ruling period. Thanks again for your support! Recent Foundation Activities ============================ o Event Sponsorship The FreeBSD Foundation has taken an active role in sponsoring BSD-related and FreeBSD-related events, including the highly successful AsiaBSDCon conference held in Taipei during March of 2004, and FreeBSD developer summits co-scheduled with major conferences such as the USENIX Annual Technical Conference (ATC) in Boston, MA during June 2004, and the EuroBSDCon conference in Karlsruhe, Germany during November 2004. In addition, the FreeBSD Foundation has provided limited travel grants to a number of FreeBSD developers to attend conferences and workshops on behalf of the FreeBSD Project, including assisting in travel costs to attend the above events. Both BSD-centered conferences and FreeBSD-specific developer events play critical roles in expanding the FreeBSD user community and supporting collaborative development. The FreeBSD Foundation would like, in particular, to recognize the contributions of Michael Wu and the AsiaBSDCon organizers, Scott Long, the EuroBSDCon program committee, the USENIX ATC BSD sig program committee, the USENIX board, and the BSDCan organizers for their outstanding contributions in organizing these and other important events. The FreeBSD Foundation is committed to supporting similar future events. o PAE/BusDMA/PCIe Hardware grant Typical system configurations continue to gain more memory -- it is not uncommon to see 2GB or more in a desktop. Recently Scott Long has taken on the task of polishing up the system components that handle these large memory configurations. To aid in his development and testing of the PAE, BusDMA, and new PCI-Express extended configuration space framework, the Foundation has purchased 4x2GB DIMMS of DDR3200 ECC memory and a case to house a donated, large memory capable, system. The fruits of Scott's efforts are already being seen in the FreeBSD CVS tree. Thanks Scott! o The FreeBSD Performance Cluster Originally created in support of the Netperf project, our performance cluster continues to grow. We expect to add 64bit systems to the cluster early next year as well as simplify the mechanism for having developers "check-out" the cluster for test runs. For more details on the FreeBSD Performance Cluster, see the "Network Performance Work Update" section later in the newsletter. o The FreeBSD Foundation Web site Gets a New Look The members of the Board of the FreeBSD Foundation may be many things, but certainly not web site designers! After a year of neglecting the current web site, we finally enlisted some help in the form of Dag-Erling Smorgrav. Dag-Erling is putting the finishing touches on our revamped site just in time to ring in the the new year. Look for the site to debut on January 1. Thanks Dag-Erling for all your hard work! The FreeBSD Foundation acquires the FreeBSD Trademark ===================================================== The FreeBSD Foundation is pleased to announce the acquisition of the FreeBSD trademark. In October of last year, Wind River Inc. agreed to assign the FreeBSD trademark to the FreeBSD Foundation. As with most things involving paperwork and government entities, progress has been slow, but transfer is now complete in both the U.S. and Germany. Transfer requests for the United Kingdom and Japan are expected to complete shortly. The Foundation would like to thank Wind River for their assistance with the Trademark transfer, and Murray Stokely for his diligence and hard work in shepherding the process. As originally registered by Walnut Creek CDROM, the FreeBSD trademark applies to "CD ROMs featuring an archive of computer programs which may be accessed for use archived on a CDROM." With the trademark transfer complete in two jurisdictions, the Foundation is now turning its attention to updating the trademark to reflect its current usage by the FreeBSD project. The new trademark filing should be submitted to all four jurisdictions in January. Network Performance Work Update =============================== The defining architectural drive behind FreeBSD 5.x is "SMPng", the adoption of fine-grained synchronization primitives and additional threading to improve concurrency and scalability. During this massive undertaking, kernel components not yet converted to the new architecture are supported, but suffer a significant performance penalty. Removing this penalty from the networking stack was a requirement for FreeBSD 5.x to become the new -STABLE branch and replace FreeBSD 4.x in Internet server installations. Through the sponsorship of the FreeBSD Foundation and the support of many volunteer developers, the initial lock down of the FreeBSD network stack was completed for the FreeBSD 5.3 release. The Foundation's initial contribution to the network stack lock down project ("Netperf") was to award a six-month development contract to Sam Leffler. Sam, a FreeBSD developer and one of the original developers of the BSD networking stack, completed locking of the firewall, Ethernet bridging, DUMMYNET, and socket subsystems. He also cleaned up existing synchronization work in the routing code and several areas of the TCP and UDP implementations. To support the volunteers continuing this work in testing the robustness and performance of the networking stack, the FreeBSD Foundation created the "FreeBSD Performance Cluster". Located in donated space at Sentex Data Systems' Ontario, Canada colocation center, the cluster has proven an invaluable asset for determining network stack bottlenecks and performing regression testing. Through a matching hardware donation from FreeBSD Systems, Inc. the cluster was christened with three 3.06GHz Dual-Xeon machines. Recently a Dell 2650, donated by IronPort Systems, was added the cluster. Permitting full remote management, including network booting, remote power, and remote console, the cluster provides an important experimentation resource available for developers throughout the world. Even with the great success of the Netperf project, there is still much to be done to complete the SMPng initiative. The FreeBSD Foundation plans to continue its work supporting this and other architecture and performance initiatives through targeted development grants and improvements to the FreeBSD Performance Cluster. Please see the "New Proposal Submission Process" section of this newsletter for details on the Foundation's funding process. Java Update =========== In August of 2003, the FreeBSD Foundation released its first binary distribution of the Java runtime environment for the FreeBSD platform. Since that time, the Foundation has funded additional development to port the Java 1.4.x to FreeBSD, and worked diligently to simplify the licensing process for "OEM" distribution of our binaries. Work on the Java 5 port has also recently begun. While we are pleased to report tremendous success in supporting Java development initiatives, our negotiations with Sun Microsystems have yielded few results. The seemingly constant restructuring at Sun has made it difficult to find and retain consistent contacts in their licensing program. The latest blow to our efforts was the recent notification of Sun's desire to revoke and renegotiate the FreeBSD Foundation's SCSL license. From what we can determine, Sun is re-negotiating all SCSL licenses to standardize their Java revenue model. Even after receiving notice of the termination of our license attempts to contact Sun to renegotiate the license have gone unanswered. For now, it is safe to assume that the Foundation will engage in another lengthy, and potentially costly, licensing negotiation before our binary distributions can continue. In the mean time, the FreeBSD Foundation is continuing its support for Java development. With the recent introduction of Java 5.0, and FreeBSD 5/6's new KSE thread library, there is still much work to do. This should ensure that FreeBSD continues to enjoy excellent Java support while we await resolution of the Java licensing issue. New Proposal Submission Process =============================== As the Foundation looks on to 2005, the board of directors has started to develop a more formal proposal review process for vetting responses to its proposal requests. The new scheme has many of the same features as a paper review committee for a technical conference. The Foundation will solicit reviewers from both inside and outside the BSD community with technical expertise in fields applicable to proposals. Each member will review a fixed, small, number of proposals and submit comments on those proposals for group discussion and ranking. The committee's recommendations will then be used by the board of directors to choose the funded proposals. Our hope is that by limiting the workload and commitment for volunteers we can attract high caliber reviewers. We also feel that bringing input from outside the BSD community will increase the visibility of FreeBSD while also broadening the community's knowledge of other technologies. Future Projects =============== The FreeBSD Foundation is always looking for new ways to enhance and promote the FreeBSD operating system. Below are a few of our new initiatives that are just getting started: o Legal Council on the DMCA, copyright auditing, and code licenses The SCO lawsuit has heightened the awareness of the Open Source community to intellectual property issues. In early December, the FreeBSD Foundation tasked its legal council with exploring different ways the FreeBSD Foundation, the FreeBSD Project, and its developers can limit liability for IP infringements without hobbling the project in paper work. Working closely with the FreeBSD Core team, the Foundation hopes to provide FreeBSD developers with accurate and useful information about copyright law, the DMCA, and software licensing. o Enterprise Storage Initiative The first test of the FreeBSD Foundation's new proposal review process will concentrate on FreeBSD's storage subsystems. From VFS and file systems to the CAM SCSI framework, this area of the FreeBSD kernel is one of the largest still lacking the fine-grained locking of the SMPng architecture. FreeBSD also lacks many industry standard storage features including a journaling file system, software RAID support for many vendor's meta-data, and iSCSI. While the Foundation lacks the resources to fund all of this work, we hope to address some of these issues via development grants in 2005. Look for our request for proposals early in the new year. Monetary donations to the Foundation will help to make funding for these projects possible! FreeBSD Foundation Financial Data ================================= Profit/Loss Jul 2004 - Sep 2004 Ordinary Income/Expense Income Grants 1,485.95 Total Income 1,485.95 Expense Computer Equipment 8,022.00 Event Sponsorship Conference Fee & Travel Grants 1,668.90 Event Sponsorship - Other 3,622.47 Total Event Sponsorship 5,291.37 PayPal Fees 33.85 Postage and Delivery 39.53 Professional Fees Accounting 450.00 Legal Fees 72.30 Total Professional Fees 522.30 Total Expense 13,909.05 Net Ordinary Income -12,423.10 Net Income -12,423.10 Balance Sheet as of Dec 21, 2004 ASSETS Current Assets Checking/Savings 194,605.36 Total Current Assets 194,605.36 TOTAL ASSETS 194,605.36 LIABILITIES & EQUITY Equity Retained Earnings 182,803.98 Net Income 11,801.38 Total Equity 194,605.36 TOTAL LIABILITIES & EQUITY 194,605.36