From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 2 15:09:07 2005 Return-Path: X-Original-To: freebsd-hackers@FreeBSD.org Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9F63816A41F; Sun, 2 Oct 2005 15:09:07 +0000 (GMT) (envelope-from SRS0=/H3MBfoq=YC=metro.cx=fbsd@sonologic.nl) Received: from mx1.sonologic.nl (mx1.sonologic.nl [82.94.245.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9F78443D45; Sun, 2 Oct 2005 15:09:06 +0000 (GMT) (envelope-from SRS0=/H3MBfoq=YC=metro.cx=fbsd@sonologic.nl) Received: from [10.1.4.2] (sonolo.xs4all.nl [80.126.206.91]) (authenticated bits=0) by mx1.sonologic.nl (8.13.3/8.13.3) with ESMTP id j92F90GU073325; Sun, 2 Oct 2005 15:09:01 GMT Message-ID: <433FF87C.3090101@metro.cx> Date: Sun, 02 Oct 2005 17:10:52 +0200 From: Koen Martens Organization: Sonologic User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.6) Gecko/20050317 Thunderbird/1.0.2 Mnenhy/0.7.2.0 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Robert Watson References: <2B3B2AA816369A4E87D7BE63EC9D2F269B7B4D@SDCEXCHANGE01.ad.amcc.com> <432F1310.80007@metro.cx> <20050920153806.F34322@fledge.watson.org> In-Reply-To: <20050920153806.F34322@fledge.watson.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Helo-Milter-Authen: gmc@sonologic.nl, fbsd@metro.cx, mx1 Received-SPF: pass (mx1.sonologic.nl: 80.126.206.91 is authenticated by a trusted mechanism) Cc: freebsd-hackers@FreeBSD.org, Dimitry Andric , Vinod Kashyap , jhb@FreeBSD.org Subject: Re: panic in propagate_priority w/ postgresql under heavy load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Oct 2005 15:09:07 -0000 Robert Watson wrote: > I can't speak to the problem with the core dumps, as it sounds like that > is device/firmware related. However, I probably can lend a hand in > debugging the problems you're seeing. I don't think the dump problem is device/firmware related, as a reboot -d gives me a dump just fine. > Often, this means the actual panic or failure has not > occurred in the thread that prints out the panic you see, but another > panic. So the first task on hitting a propagate_priority() panic is to > identify the thread that actually had the problem. Hmmm, so we have a very elusive problem here, one that is not easily pinpointed.. Somehow, this does not come as a surprise :) > If you want to do this by e-mail so we can lend a hand, you probably > want to hook up a serial console so you can copy and paste the debugging > session. Compile DDB into the kernel (this should have no performance > overhead), and when the system panics, you'll (ideally) get a db> > prompt. [excellent help in debugging deleted for brevity] Right, so perhaps i am missing something here, but this in my kernel config should do it (full config included below for completeness sake, as well as dmesg output): # debug options KDB options DDB # options KDB_TRACE # makeoptions DEBUG=-g # debug Furthermore, since reboot -d does dump to swap now (by limiting physical memory to just below the swap partition size in the bootloader config), i would expect to get a dump also when a panic occurs, and i would expect a ddb> prompt. Alas, this is what i get: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 06 fault virtual address = 0x24 fault code = supervisor read, page not present instruction pointer = 0x8:0xc051c253 stack pointer = 0x10:0xe93efb3c frame pointer = 0x10:0xe93efb50 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 6092 (postgres) And that, that is all.. No ddb> no 'dumping xxxxMB', just that. So basically, i fear this is a non-debugable problem, since putting in witness and such slows the kernel to a point where the panic does not occur anymore (at least, not in the 4 weeks i've been running the box with witness & invariants). Clueless :) Best, Koen ================[ full kernel config: # # GENERIC -- Generic kernel configuration file for FreeBSD/i386 # # For more information on this file, please read the handbook section on # Kernel Configuration Files: # # http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html # # The handbook is also available locally in /usr/share/doc/handbook # if you've installed the doc distribution, otherwise always see the # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the # latest information. # # An exhaustive list of options and more detailed explanations of the # device lines is also present in the ../../conf/NOTES and NOTES files. # If you are in doubt as to the purpose or necessity of a line, check first # in NOTES. # # $FreeBSD: src/sys/i386/conf/GENERIC,v 1.413.2.13 2005/04/02 16:37:58 scottl Exp $ machine i386 cpu I486_CPU cpu I586_CPU cpu I686_CPU ident YIN-YANG # debug #options WITNESS #options INVARIANTS #options INVARIANT_SUPPORT options KDB options DDB # options KDB_TRACE # makeoptions DEBUG=-g # debug # To statically compile in device wiring instead of /boot/device.hints #hints "GENERIC.hints" # Default places to look for devices. options SCHED_4BSD # 4BSD scheduler options INET # InterNETworking options INET6 # IPv6 communications protocols options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options MD_ROOT # MD is a potential root device options NFSCLIENT # Network Filesystem Client options NFSSERVER # Network Filesystem Server options NFS_ROOT # NFS usable as /, requires NFSCLIENT options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_GPT # GUID Partition Tables. options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!] options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options SCSI_DELAY=15000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options KBD_INSTALL_CDEV # install a CDEV entry in /dev options AHC_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~128k to driver. options AHD_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~215k to driver. options ADAPTIVE_GIANT # Giant mutex is adaptive. options IPFILTER options IPFILTER_LOG options IPFILTER_DEFAULT_BLOCK options IPFILTER options IPFILTER_LOG options IPFIREWALL #firewall options IPFIREWALL_VERBOSE #enable logging to syslogd(8) options IPFIREWALL_FORWARD #enable transparent proxy support options IPFIREWALL_VERBOSE_LIMIT=10 #limit verbosity options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by default options IPV6FIREWALL #firewall for IPv6 options IPV6FIREWALL_VERBOSE options IPV6FIREWALL_VERBOSE_LIMIT=10 options IPV6FIREWALL_DEFAULT_TO_ACCEPT options SHMMAXPGS=262144 options SHMSEG=512 options SHMMNI=512 options SEMMNI=512 options SEMMNS=1024 options SEMMNU=512 options SEMMAP=512 options NMBCLUSTERS=32768 device apic # I/O APIC options SMP # Symmetric MultiProcessor Kernel # Bus support. Do not remove isa, even if you have no isa slots device isa device eisa device pci # Floppy drives device fdc # ATA and ATAPI devices device ata device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives #device atapifd # ATAPI floppy drives #device atapist # ATAPI tape drives options ATA_STATIC_ID # Static device numbering # SCSI Controllers #device ahb # EISA AHA1742 family #device ahc # AHA2940 and onboard AIC7xxx devices #device ahd # AHA39320/29320 and onboard AIC79xx devices #device amd # AMD 53C974 (Tekram DC-390(T)) #device isp # Qlogic family #device mpt # LSI-Logic MPT-Fusion ##device ncr # NCR/Symbios Logic #device sym # NCR/Symbios Logic (newer chipsets + those of `ncr') #device trm # Tekram DC395U/UW/F DC315U adapters #device adv # Advansys SCSI adapters #device adw # Advansys wide SCSI adapters #device aha # Adaptec 154x SCSI adapters #device aic # Adaptec 15[012]x SCSI adapters, AIC-6[23]60. #device bt # Buslogic/Mylex MultiMaster SCSI adapters #device ncv # NCR 53C500 #device nsp # Workbit Ninja SCSI-3 #device stg # TMC 18C30/18C50 # SCSI peripherals device scbus # SCSI bus (required for SCSI) device ch # SCSI media changers device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct SCSI access) device ses # SCSI Environmental Services (and SAF-TE) # RAID controllers interfaced to the SCSI subsystem #device amr # AMI MegaRAID #device arcmsr # Areca SATA II RAID #device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID #device ciss # Compaq Smart RAID 5* #device dpt # DPT Smartcache III, IV - See NOTES for options #device hptmv # Highpoint RocketRAID 182x #device iir # Intel Integrated RAID #device ips # IBM (Adaptec) ServeRAID #device mly # Mylex AcceleRAID/eXtremeRAID device twa # 3ware 9000 series PATA/SATA RAID # RAID controllers #device aac # Adaptec FSA RAID #device aacp # SCSI passthrough for aac (requires CAM) #device ida # Compaq Smart RAID #device mlx # Mylex DAC960 family #device pst # Promise Supertrak SX6000 device twe # 3ware ATA RAID # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device vga # VGA video card driver #device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc # Enable this for the pcvt (VT220 compatible) console driver #device vt #options XSERVER # support for X server on a vt console #options FAT_CURSOR # start with block cursor device agp # support several AGP chipsets # Floating point support - do not disable. device npx # Power management support (see NOTES for more options) #device apm # Add suspend/resume support for the i8254. device pmtimer # PCCARD (PCMCIA) support # PCMCIA and cardbus bridge support #device cbb # cardbus (yenta) bridge #device pccard # PC Card (16-bit) bus #device cardbus # CardBus (32-bit) bus # Serial (COM) ports device sio # 8250, 16[45]50 based serial ports # Parallel port device ppc device ppbus # Parallel port bus (required) device lpt # Printer device plip # TCP/IP over parallel device ppi # Parallel port interface device #device vpo # Requires scbus and da # If you've got a "dumb" serial or parallel PCI card that is # supported by the puc(4) glue driver, uncomment the following # line to enable it (connects to the sio and/or ppc drivers): #device puc # PCI Ethernet NICs. #device de # DEC/Intel DC21x4x (``Tulip'') device em # Intel PRO/1000 adapter Gigabit Ethernet Card #device ixgb # Intel PRO/10GbE Ethernet Card #device txp # 3Com 3cR990 (``Typhoon'') #device vx # 3Com 3c590, 3c595 (``Vortex'') # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support #device bfe # Broadcom BCM440x 10/100 Ethernet #device bge # Broadcom BCM570xx Gigabit Ethernet #device dc # DEC/Intel 21143 and various workalikes device fxp # Intel EtherExpress PRO/100B (82557, 82558) #device lge # Level 1 LXT1001 gigabit ethernet #device nge # NatSemi DP83820 gigabit ethernet #evice pcn # AMD Am79C97x PCI 10/100 (precedence over 'lnc') #device re # RealTek 8139C+/8169/8169S/8110S #device rl # RealTek 8129/8139 #device sf # Adaptec AIC-6915 (``Starfire'') #device sis # Silicon Integrated Systems SiS 900/SiS 7016 #device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet #device ste # Sundance ST201 (D-Link DFE-550TX) #device ti # Alteon Networks Tigon I/II gigabit Ethernet #device tl # Texas Instruments ThunderLAN #device tx # SMC EtherPower II (83c170 ``EPIC'') #device vge # VIA VT612x gigabit ethernet #device vr # VIA Rhine, Rhine II #device wb # Winbond W89C840F #device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'') # ISA Ethernet NICs. pccard NICs included. #device cs # Crystal Semiconductor CS89x0 NIC # 'device ed' requires 'device miibus' #device ed # NE[12]000, SMC Ultra, 3c503, DS8390 cards #device ex # Intel EtherExpress Pro/10 and Pro/10+ #device ep # Etherlink III based cards #device fe # Fujitsu MB8696x based cards #device ie # EtherExpress 8/16, 3C507, StarLAN 10 etc. #device lnc # NE2100, NE32-VL Lance Ethernet cards #device sn # SMC's 9000 series of Ethernet chips #device xe # Xircom pccard Ethernet # ISA devices that use the old ISA shims #device le # Wireless NIC cards #device wlan # 802.11 support #device an # Aironet 4500/4800 802.11 wireless NICs. #device awi # BayStack 660 and others #device wi # WaveLAN/Intersil/Symbol 802.11 wireless NICs. #device wl # Older non 802.11 Wavelan wireless NIC. # Pseudo devices. device loop # Network loopback device mem # Memory and kernel memory devices device io # I/O device device random # Entropy device device ether # Ethernet support device sl # Kernel SLIP device ppp # Kernel PPP device tun # Packet tunnel. device pty # Pseudo-ttys (telnet etc) device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter # USB support #device uhci # UHCI PCI->USB interface #device ohci # OHCI PCI->USB interface #device ehci # EHCI PCI->USB interface (USB 2.0) #device usb # USB Bus (required) #device udbp # USB Double Bulk Pipe devices #device ugen # Generic #device uhid # "Human Interface Devices" #device ukbd # Keyboard #device ulpt # Printer #device umass # Disks/Mass storage - Requires scbus and da #device ums # Mouse #device urio # Diamond Rio 500 MP3 player #device uscanner # Scanners # USB Ethernet, requires mii #device aue # ADMtek USB Ethernet #device axe # ASIX Electronics USB Ethernet #device cdce # Generic USB over Ethernet #device cue # CATC USB Ethernet #device kue # Kawasaki LSI USB Ethernet #device rue # RealTek RTL8150 USB Ethernet # FireWire support #device firewire # FireWire bus code #device sbp # SCSI over FireWire (Requires scbus and da) #device fwe # Ethernet over FireWire (non-standard!) ==============================[ full boot-up console output: KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2005 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.4-RELEASE-p6 #3: Mon Sep 19 21:47:58 CEST 2005 root@yin.sonologic.nl:/usr/obj/usr/src/sys/yin-yang-5.4 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3056.50-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf29 Stepping = 9 Features=0xbfebfbff Hyperthreading: 2 logical CPUs real memory = 1887436800 (1800 MB) avail memory = 1840525312 (1755 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 6 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard npx0: on motherboard npx0: INT 16 interface acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: on acpi0 cpu1: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pci0: at device 0.1 (no driver attached) pcib1: at device 2.0 on pci0 pci1: on pcib1 pci1: at device 28.0 (no driver attached) pcib2: at device 29.0 on pci1 pci2: on pcib2 pci1: at device 30.0 (no driver attached) pcib3: at device 31.0 on pci1 pci3: on pcib3 3ware device driver for 9000 series storage controllers, version: 2.50.02.012 twa0: <3ware 9000 series Storage Controller> port 0x7000-0x70ff mem 0xfd800000-0xfdffffff,0xfb200000-0xfb2000ff irq 48 at de vice 1.0 on pci3 twa0: 4 ports, Firmware FE9X 2.02.00.008, BIOS BE9X 2.02.01.037 pci0: at device 2.1 (no driver attached) pcib4: at device 30.0 on pci0 pci4: on pcib4 pci4: at device 3.0 (no driver attached) fxp0: port 0x8400-0x843f mem 0xfb300000-0xfb31ffff,0xfb341000-0xfb341fff irq 20 at device 4.0 on pci4 miibus0: on fxp0 inphy0: on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp0: Ethernet address: 00:02:b3:d8:8a:b5 em0: port 0x8440-0x847f mem 0xfb320000-0xfb33ffff irq 23 at device 5.0 on pci4 em0: Ethernet address: 00:02:b3:d8:8b:05 em0: Speed:N/A Duplex:N/A isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x6c60-0x6c6f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 at device 31.1 on pci0 ata0: channel #0 on atapci0 ata1: channel #1 on atapci0 pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 atkbdc0: port 0x64,0x60 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 fdc0: port 0x3f7,0x3f4-0x3f5,0x3f0-0x3f3 irq 6 drq 2 on acpi0 fd0: <1440-KB 3.5" drive> on fdc0 drive 0 sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A, console sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A ppc0: port 0x778-0x77b,0x378-0x37f irq 7 drq 3 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/9 bytes threshold ppbus0: on ppc0 plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 pmtimer0 on isa0 orm0: at iomem 0xe3000-0xe3fff,0xc8000-0xc97ff,0xc0000-0xc7fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x100> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 10.000 msec IPv6 packet filtering initialized, default to accept, logging limited to 10 packets/entry IP Filter: v3.4.35 initialized. Default = block all, Logging = enabled ipfw2 initialized, divert disabled, rule-based forwarding enabled, default to accept, logging limited to 10 packets/entry by default acd0: CDROM at ata0-master PIO4 SMP: AP CPU #1 Launched! da0 at twa0 bus 0 target 0 lun 0 da0: <3ware Logical Disk 00 1.00> Fixed Direct Access SCSI-0 device da0: 100.000MB/s transfers da0: 114430MB (234352640 512 byte sectors: 255H 63S/T 14587C) Mounting root from ufs:/dev/da0s1a WARNING: / was not properly dismounted Pre-seeding PRNG: kickstart. Loading configuration files. Entropy harvesting: interrupts ethernet point_to_point kickstart. kernel dumps on /dev/da0s1b swapon: adding /dev/da0s1b as swap device Starting file system checks: /dev/da0s1a: 3713 files, 91129 used, 424942 free (1918 frags, 52878 blocks, 0.4% fragmentation) /dev/da0s1e: DEFER FOR BACKGROUND CHECKING /dev/da0s1f: DEFER FOR BACKGROUND CHECKING WARNING: /usr was not properly dismounted WARNING: /var was not properly dismounted /var: mount pending error: blocks 268 files 3 Enabling ipfilter. Setting hostname: yin.sonologic.nl. fxp0: flags=8843 mtu 1500 options=b inet 82.94.245.12 netmask 0xffffffc0 broadcast 82.94.245.63 inet6 fe80::202:b3ff:fed8:8ab5%fxp0 prefixlen 64 tentative scopeid 0x1 ether 00:02:b3:d8:8a:b5 media: Ethernet autoselect (100baseTX ) status: active em0: flags=8843 mtu 1500 options=b inet 192.168.0.1 netmask 0xffff0000 broadcast 192.168.255.255 inet6 fe80::202:b3ff:fed8:8b05%em0 prefixlen 64 tentative scopeid 0x2 ether 00:02:b3:d8:8b:05 media: Ethernet autoselect status: no carrier lo0: flags=8049 mtu 16384 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 filter sync'd add net default: gateway 82.94.245.1 Additional routing options:. Starting devd. Mounting NFS file systems:. Starting syslogd. Sep 28 13:37:41 yin syslogd: kernel boot file is /boot/kernel/kernel Checking for core dump on /dev/da0s1b ... savecore: no dumps found Setting date via ntp. Looking for host ntp.xs4all.nl and service ntp host found : auth1.xs4all.nl Looking for host 82.94.245.2 and service ntp host found : enterprise.sonologic.nl 28 Sep 13:37:43 ntpdate[336]: step time server 194.109.22.18 offset 1.514066 sec ELF ldconfig path: /lib /usr/leib /usr/lib/compmat /usr/local/li0b :a.out ldconfig p ath: /usr/lib/aoLut /usr/lib/compiat/aout Recoverning vi editor sekssions: is up 100 Mbps Full Duplex . Starting usbd. No USB host controllers found Starting local daemons:rm: /var/run/bwmon.dat: No such file or directory Loading configuration files. Setting hostname: www.issuecrawler.net. ln: /dev/log: Operation not permitted Starting syslogd. Sep 28 13:37:46 www syslogd: kernel boot file is /boot/kernel/kernel ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout Starting local daemons:/usr/local/www/bin/apachectl start: httpd started . Updating motd. Starting sshd. Starting mysqld daemon with databases from /usr/local/mysql-standard-4.0.21-unknown-freebsd4.7-i386/data Starting cron. Local package initialization:. Wed Sep 28 13:37:48 CEST 2005 Loading configuration files. Setting hostname: wiki.issuecrawler.net. ln: /dev/log: Operation not permitted Starting syslogd. Sep 28 13:37:49 wiki syslogd: kernel boot file is /boot/kernel/kernel ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Starting local daemons:Starting mysqld daemon with databases from /usr/local/mysql-standard-4.0.21-unknown-freebsd4.7-i386/d ata Syntax error on line 205 of /usr/local/www/conf/httpd.conf: Cannot load /usr/local/www/libexec/libphp5.so into server: /usr/local/lib/libiconv.so.3: Undefined symbol "mbrtowc" /usr/local/www/bin/apachectl start: httpd could not be started . Updating motd. Starting sshd. Starting sendmail. Sep 28 13:37:51 wiki sm-mta[734]: NOQUEUE: SYSERR(root): opendaemonsocket: daemon IPv6: can't create server SMTP socket: Pro tocol not supported Sep 28 13:37:51 wiki sm-mta[734]: NOQUEUE: SYSERR(root): opendaemonsocket: daemon IPv6: optional socket disabled Starting cron. Local package initialization:. Wed Sep 28 13:37:51 CEST 2005 Loading configuration files. Setting hostname: devel.issuecrawler.net. ln: /dev/log: Operation not permitted Starting syslogd. Sep 28 13:37:52 devel syslogd: kernel boot file is /boot/kernel/kernel ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout Recovering vi editor sessions:. Starting local daemons:/usr/local/www/bin/apachectl start: httpd started . Updating motd. Starting sshd. Starting cron. Starting mysqld daemon with databases from /usr/local/mysql-standard-4.0.21-unknown-freebsd4.7-i386/data Local package initialization:. Wed Sep 28 13:37:59 CEST 2005 . Updating motd. Starting ntpd. Configuring syscons: blanktime. Starting sshd. Sep 28 13:38:01 devel syslogd: Logging subprocess 889 (/usr/local/libexec/authlog.pl) exited with status 255. Starting sendmail. Initial i386 initialization:. Additional ABI support:. Starting cron. Local package initialization:. Additional TCP options:. Starting inetd. Starting background file system checks in 60 seconds. Wed Sep 28 13:38:02 CEST 2005 FreeBSD/i386 (yin.sonologic.nl) (ttyd0) login: -- K.F.J. Martens, Sonologic, http://www.sonologic.nl/ Networking, hosting, embedded systems, unix, artificial intelligence. Public PGP key: http://www.metro.cx/pubkey-gmc.asc Wondering about the funny attachment your mail program can't read? Visit http://www.openpgp.org/