From owner-freebsd-current Sun Feb 2 9:41:28 2003 Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 86B6F37B401 for ; Sun, 2 Feb 2003 09:41:26 -0800 (PST) Received: from shell.dragondata.com (shell.dragondata.com [66.17.132.249]) by mx1.FreeBSD.org (Postfix) with ESMTP id A889D43FA3 for ; Sun, 2 Feb 2003 09:41:25 -0800 (PST) (envelope-from toasty@dragondata.com) Received: (from root@localhost) by shell.dragondata.com (8.12.6/8.12.6) id h12HfMSR062609 for freebsd-current@freebsd.org; Sun, 2 Feb 2003 11:41:22 -0600 (CST) (envelope-from toasty@dragondata.com) Received: from KEVIN-AW.dragondata.com (toasty@localhost.dragondata.com [127.0.0.1]) by shell.dragondata.com (8.12.6/8.12.6av) with ESMTP id h12HfHeA062599 for ; Sun, 2 Feb 2003 11:41:18 -0600 (CST) (envelope-from toasty@dragondata.com) Message-Id: <5.1.1.5.2.20030202112759.0461fcc8@127.0.0.1> X-Sender: toasty@127.0.0.1 X-Mailer: QUALCOMM Windows Eudora Version 5.1.1 Date: Sun, 02 Feb 2003 11:41:32 -0600 To: freebsd-current@freebsd.org From: Kevin Day Subject: Using 4.3-RELEASE's libc on 5.0 causes hard lockups Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Virus-Scanned: by Dragondata mail server Sender: owner-freebsd-current@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG We had a system running 4.3-RELEASE that I used the sysinstall upgrade mechanism to upgrade to 5.0-RELEASE. I installed "compat4x" to use our existing 4.x binaries. Immediately after rebooting, I noticed most old 4.x binaries were complaining about "_stdoutp" being an undefined symbol. However, the scary part was that when I started apache/mod_php4 the server crashed (hard lockup) within 10 seconds under load. This was easily reproducible, at least a dozen times while trying to debug this I started httpd, and the server locked up within 10 seconds. I recompiled all of apache, mod_php4 and all of its libraries, started up httpd and had no problems with that. Things were fine that night until an "analog" cron job ran, every time THAT ran, I also got a hard lockup of the server, OR between 100 and 500 of my httpd processes would suddenly SEGV. After a little more poking around, I saw in /usr/lib: lrwxr-xr-x 1 root wheel 9 Feb 1 00:18 libc.so -> libc.so.5 lrwxr-xr-x 1 root wheel 16 Jul 5 2002 libc.so.3 -> /usr/lib/libc.so -r--r--r-- 1 root wheel 571480 Aug 5 13:45 libc.so.4 -r--r--r-- 1 root wheel 836892 Feb 1 00:18 libc.so.5 Shouldn't libc.so.4 have been a symlink to libc.so after a compat4x install? In any case, doing that myself seemed to fix everything. My questions: 1) Shouldn't something along the way of doing a sysinstall upgrade or installing compat4x have fixed /usr/lib/libc.so.4 into a symlink? (That is the correct situation, right?) 2) Is it possible that some kernel interface has changed, and something isn't being validated in the kernel side? Non-root userland applications being able to lockup the server, and/or affect other processes simply by using a different libc would seem to indicate this. I know this is a pretty vague bug report, but this is a production server, so I wasn't able to play around too much with it. I do have a backup of the entire server before it was upgraded to 5.0 if you'd like me to check anything there. I did compile with INVARIANTS and WITNESS and got no debugging output when things did lock up. The keyboard and serial console were totally dead when this happened, so DDB isn't an option either. (originally emailed security-officer about this because of the possibility for a security issue, who told me to forward this here) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-current" in the body of the message