From owner-freebsd-amd64@FreeBSD.ORG Wed Feb 9 03:56:30 2005 Return-Path: Delivered-To: freebsd-amd64@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A367316A4CE for ; Wed, 9 Feb 2005 03:56:30 +0000 (GMT) Received: from buxton.digitalspy.co.uk (buxton.digitalspy.co.uk [212.42.1.208]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9F75043D39 for ; Wed, 9 Feb 2005 03:56:29 +0000 (GMT) (envelope-from jwelsh@digitalspy.co.uk) Received: from MEEPMEEP (rdu163-32-205.nc.rr.com [24.163.32.205]) by buxton.digitalspy.co.uk (Postfix) with ESMTP id C1E32548AE; Wed, 9 Feb 2005 03:56:27 +0000 (GMT) From: "James Welsh" To: Date: Tue, 8 Feb 2005 22:56:28 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook, Build 11.0.6353 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 thread-index: AcUOW1TljD5VjK3sRYay+lNzJudiWg== Message-Id: <20050209035627.C1E32548AE@buxton.digitalspy.co.uk> Subject: Hard lockup on 5.3-STABLE X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2005 03:56:30 -0000 Hi, Our AMD Opteron server is locking up every few hours - no core dumps being made, and by the look of the output in /var/log/messages, whatever is occurring is fairly severe: Feb 9 03:29:26 pollux kernel: Feb 9 03:29:26 pollux kernel: Feb 9 03:29:26 pollux kernel: Fatal trap 12: page fault while in kernel mode Feb 9 03:29:26 pollux kernel: cpuid = 1; apic id = 01 Feb 9 03:29:26 pollux kernel: fault virtual address = 0x28 Feb 9 03:29:26 pollux kernel: fault code = supervis Feb 9 03:29:26 pollux kernel: or Feb 9 03:29:26 pollux kernel: wr Feb 9 03:29:26 pollux kernel: it Feb 9 03:29:26 pollux kernel: e Feb 9 03:29:26 pollux kernel: , page Feb 9 03:29:26 pollux kernel: n Feb 9 03:29:26 pollux kernel: ot pr Feb 9 03:29:26 pollux kernel: esent Feb 9 03:29:26 pollux kernel: Feb 9 03:29:26 pollux kernel: in Feb 9 03:29:26 pollux kernel: st The machine then reboots. In case it helps narrow the problem down, lots of the following messages are also being logged to /var/log/messages: Feb 8 16:47:27 pollux kernel: Limiting closed port RST response from 313 to 200 packets/sec Feb 8 16:47:28 pollux kernel: Limiting closed port RST response from 281 to 200 packets/sec Feb 8 16:47:30 pollux kernel: Limiting closed port RST response from 279 to 200 packets/sec Feb 8 16:47:31 pollux kernel: Limiting closed port RST response from 265 to 200 packets/sec The server is running MySQL Server 4.1.9 compiled from ports - this happens regardless of whether it's compiled with process scope threads or not. It also doesn't seem to be related to load - I was monitoring things in top at the time of the most recent crash - the output looked like this: last pid: 3980; load averages: 0.00, 0.00, 0.00 up 0+10:42:49 03:29:25 97 processes: 1 running, 96 sleeping CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 88M Active, 147M Inact, 150M Wired, 1272K Cache, 214M Buf, 7438M Free Swap: 4096M Total, 4096M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMAND 520 mysql 96 0 138M 83540K select 1 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K sigwai 1 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K ksesig 1 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 1 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld 520 mysql 20 0 138M 83540K kserel 0 20:10 0.00% 0.00% mysqld Server: Dual AMD Opteron 248 2.2GHz 8GB RAM Tyan S2882 Thunder K86 motherboard Running FreeBSD 5.3-STABLE Any help very gratefully received. Best regards, James Welsh