Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 8 Jan 2005 21:34:56 +0000 (GMT)
From:      Robert Watson <rwatson@freebsd.org>
To:        matrix@itlegion.ru
Cc:        freebsd-stable@freebsd.org
Subject:   Re: Server locks up if more than one jail in running
Message-ID:  <Pine.NEB.3.96L.1050108213144.43829I-100000@fledge.watson.org>
In-Reply-To: <20050108211749.77950.qmail@omni.ipform.ru>

next in thread | previous in thread | raw e-mail | index | archive | help

On 8 Jan 2005 matrix@itlegion.ru wrote:

> I have a very bad problem. I need to seetup large number of jails (about
> 100) to do virtual hosting and i am doing it on 5.3-STABLE (not
> 5.3-RELEASE, i mean -STABLE which is cvsupped). So, when i setup one
> jail - everything runs fine for many days. If i setup one more jail or 3
> or more - server locks up (freezes, hangs, becomes totally
> unsresponsive) within 48 hours. Sometime it runs only a couple hours,
> sometimes almost a couple of days. So, to diagnose the problem i setup a
> cron process which saves a lot of system statistics every minutes. The
> last working minute stats are included below as well as basic system
> configuration.  Jails are complete distributions, but very basic
> services are running:  cron, sshd. adjtimezone is disabled. All other
> records on /etc/crontab are left just like in distro.  I really need
> help with this problem because i canno migrate to 5.3 because of it. And
> i really need 5.3 because of some new features. Maybe somone could help. 

If you haven't already: could you compile the kernel with "options DDB",
"options KDB", and "options BREAK_TO_DEBUGGER" and see if you can get into
the debugger using a console or (preferably) a serial break?  That way we
can attempt to diagnose the state of the system once it is wedged.  It
would also be helpful if you could set the system up to run with "options
INVARIANTS", which will have some performance impact but turn on
additional self-checking in the OS.  Finally, depending on whether you can
tolerate the performance hit, running with "options WITNESS" (and "options
WITNESS_SKIPSPIN" to help a bit with performance) will allow us to
generate additional debugging information.  If you are able to get into
the debugger, it would be helpful if you could run the following, and
ideal send the output copied using a serial console:

- "show pcpu" "show pcpu 0" "show pcpu 1" "show pcpu 2" "show pcpu 3"
- "ps"
- "show threads"
- "show alllocks"			# will only work with witness
- "show lockedvnods"

This is a substantial volume of output, so if you don't have access to a
serial console so you can copy and paste on another box, it may be
somewhat inconvenient. 

Robert N M Watson


> 
> 
> SYSTEM:
> 
> Multiprocessor. Dual XEON 2.667Ghz with HTT. 4GB DDR RAM.
> RAID 5 (4 SATA disks x 120gb) on 3Ware SATA controller in
> PCI 64 slot on Supermicro MB.
> 
> ------------ dmesg --------------------------------------
> 
> Copyright (c) 1992-2004 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>  The Regents of the University of California. All rights reserved.
> FreeBSD 5.3-STABLE #4: Wed Dec 22 10:25:48 MSK 2004
>     root@new.itlegion.ru:/usr/obj/usr/src/sys/OMNI
> ACPI APIC Table: <PTLTD    APIC  >
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: Intel(R) Xeon(TM) CPU 2.66GHz (2665.92-MHz 686-class CPU)
>   Origin = "GenuineIntel"  Id = 0xf25  Stepping = 5
>   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>   Hyperthreading: 2 logical CPUs
> real memory  = 4160225280 (3967 MB)
> avail memory = 4077481984 (3888 MB)
> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
>  cpu0 (BSP): APIC ID:  0
>  cpu1 (AP): APIC ID:  1
>  cpu2 (AP): APIC ID:  6
>  cpu3 (AP): APIC ID:  7
> ioapic0 <Version 2.0> irqs 0-23 on motherboard
> ioapic1 <Version 2.0> irqs 24-47 on motherboard
> ioapic2 <Version 2.0> irqs 48-71 on motherboard
> ioapic3 <Version 2.0> irqs 72-95 on motherboard
> ioapic4 <Version 2.0> irqs 96-119 on motherboard
> npx0: [FAST]
> npx0: <math processor> on motherboard
> npx0: INT 16 interface
> acpi0: <PTLTD   RSDT> on motherboard
> acpi0: Power Button (fixed)
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
> cpu0: <ACPI CPU (2 Cx states)> on acpi0
> cpu1: <ACPI CPU (2 Cx states)> on acpi0
> cpu2: <ACPI CPU (2 Cx states)> on acpi0
> cpu3: <ACPI CPU (2 Cx states)> on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> pci0: <unknown> at device 0.1 (no driver attached)
> pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
> pci1: <ACPI PCI bus> on pcib1
> pci1: <base peripheral, interrupt controller> at device 28.0 (no driver attached)
> pcib2: <ACPI PCI-PCI bridge> at device 29.0 on pci1
> pci2: <ACPI PCI bus> on pcib2
> pci1: <base peripheral, interrupt controller> at device 30.0 (no driver attached)
> pcib3: <ACPI PCI-PCI bridge> at device 31.0 on pci1
> pci3: <ACPI PCI bus> on pcib3
> em0: <Intel(R) PRO/1000 Network Connection, Version - 1.7.35> port 0x3000-0x303f mem 0xfc200000-0xfc21ffff irq 28 at device 2.0 on 
> pci3
> em0: Ethernet address: 00:30:48:2a:2d:bc
> em0:  Speed:N/A  Duplex:N/A
> em1: <Intel(R) PRO/1000 Network Connection, Version - 1.7.35> port 0x3040-0x307f mem 0xfc220000-0xfc23ffff irq 29 at device 2.1 on 
> pci3
> em1: Ethernet address: 00:30:48:2a:2d:bd
> em1:  Speed:N/A  Duplex:N/A
> pcib4: <ACPI PCI-PCI bridge> at device 3.0 on pci0
> pci4: <ACPI PCI bus> on pcib4
> pci4: <base peripheral, interrupt controller> at device 28.0 (no driver attached)
> pcib5: <ACPI PCI-PCI bridge> at device 29.0 on pci4
> pci5: <ACPI PCI bus> on pcib5
> pci4: <base peripheral, interrupt controller> at device 30.0 (no driver attached)
> pcib6: <ACPI PCI-PCI bridge> at device 31.0 on pci4
> pci6: <ACPI PCI bus> on pcib6
> twe0: <3ware Storage Controller. Driver version 1.50.01.002> port 0x4000-0x400f mem 0xfc800000-0xfcffffff irq 72 at device 1.0 on 
> pci6
> twe0: [GIANT-LOCKED]
> twe0: 4 ports, Firmware FE7S 1.05.00.063, BIOS BE7X 1.08.00.048
> uhci0: <Intel 82801CA/CAM (ICH3) USB controller USB-A> port 0x2000-0x201f irq 16 at device 29.0 on pci0
> uhci0: [GIANT-LOCKED]
> usb0: <Intel 82801CA/CAM (ICH3) USB controller USB-A> on uhci0
> usb0: USB revision 1.0
> uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
> uhub0: 2 ports with 2 removable, self powered
> uhci1: <Intel 82801CA/CAM (ICH3) USB controller USB-B> port 0x2020-0x203f irq 19 at device 29.1 on pci0
> uhci1: [GIANT-LOCKED]
> usb1: <Intel 82801CA/CAM (ICH3) USB controller USB-B> on uhci1
> usb1: USB revision 1.0
> uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
> uhub1: 2 ports with 2 removable, self powered
> uhci2: <Intel 82801CA/CAM (ICH3) USB controller USB-C> port 0x2040-0x205f irq 18 at device 29.2 on pci0
> uhci2: [GIANT-LOCKED]
> usb2: <Intel 82801CA/CAM (ICH3) USB controller USB-C> on uhci2
> usb2: USB revision 1.0
> uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
> uhub2: 2 ports with 2 removable, self powered
> pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
> pci7: <ACPI PCI bus> on pcib7
> pci7: <display, VGA> at device 1.0 (no driver attached)
> isab0: <PCI-ISA bridge> at device 31.0 on pci0
> isa0: <ISA bus> on isab0
> atapci0: <Intel ICH3 UDMA100 controller> port 0x2060-0x206f,0x3f6,0x1f0-0x1f7 at device 31.1 on pci0
> ata0: channel #0 on atapci0
> ata2: channel #1 on atapci0
> pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
> acpi_button0: <Power Button> on acpi0
> atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
> sio0: type 16550A
> sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
> sio1: type 16550A
> fdc0: <floppy drive controller> port 0x3f7,0x3f0-0x3f5 irq 6 drq 2 on acpi0
> fdc0: [FAST]
> fd0: <1440-KB 3.5" drive> on fdc0 drive 0
> orm0: <ISA Option ROMs> at iomem 0xe0000-0xe3fff,0xc9000-0xc9fff,0xc8000-0xc8fff,0xc0000-0xc7fff on isa0
> sc0: <System console> at flags 0x100 on isa0
> sc0: VGA <16 virtual consoles, flags=0x300>
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
> Timecounters tick every 10.000 msec
> acd0: CDRW <TOSHIBA DVD-ROM SD-R2412/1015> at ata0-slave UDMA33
> twed0: <Unit 0, RAID5, Normal> on twe0
> twed0: 343417MB (703318656 sectors)
> SMP: AP CPU #2 Launched!
> SMP: AP CPU #1 Launched!
> SMP: AP CPU #3 Launched!
> Mounting root from ufs:/dev/twed0s1a
> em0: Link is up 100 Mbps Full Duplex
> 
> ------------ rc.conf --------------------------------------
> 
> font8x14="cp866-8x14"
> font8x16="cp866b-8x16"
> font8x8="cp866-8x8"
> keymap="ru.koi8-r"
> keyrate="fast"
> linux_enable="YES"
> mousechar_start="3"
> saver="no"
> scrnmap="koi8-r2cp866"
> 
> gateway_enable="YES"
> hostname="omni2.itlegion.ru"
> defaultrouter="213.252.68.1"
> ifconfig_em0="inet 213.252.68.2 netmask 255.255.255.0"
> 
> sshd_enable="YES"
> named_enable="YES"
> enable_quotas="YES"
> usbd_enable="NO"
> 
> healthd_enable="NO"
> healthd_flags="-6 -l"
> 
> syslogd_flags="-ss"
> 
> background_fsck="NO"
> 
> rpcbind="NO"
> sendmail_enable="NO"
> sendmail_submit_enable="NO"
> sendmail_outbound_enable="NO"
> sendmail_msp_queue_enable="NO"
> inetd_enable="NO"
> 
> jail_enable="YES"
> jail_list="213_252_68_6 213_252_68_5"
> 
> jail_213_252_68_5_rootdir="/hosts/213.252.68.5"
> jail_213_252_68_5_hostname="jail1.itlegion.ru"
> jail_213_252_68_5_ip="213.252.68.5"
> jail_213_252_68_5_exec="/bin/sh /etc/rc"
> jail_213_252_68_5_devfs_enable="YES"
> jail_213_252_68_5_fdescfs_enable="NO"
> jail_213_252_68_5_procfs_enable="NO"
> jail_213_252_68_5_devfs_ruleset="devfsrules_jail"
> 
> jail_213_252_68_6_rootdir="/hosts/213.252.68.6"
> jail_213_252_68_6_hostname="jail2.itlegion.ru"
> jail_213_252_68_6_ip="213.252.68.6"
> jail_213_252_68_6_exec="/bin/sh /etc/rc"
> jail_213_252_68_6_devfs_enable="YES"
> jail_213_252_68_6_fdescfs_enable="NO"
> jail_213_252_68_6_procfs_enable="NO"
> jail_213_252_68_6_devfs_ruleset="devfsrules_jail"
> 
> -------- KERNEL CONFIGURATION FILE -------------
> machine  i386
> cpu  I686_CPU
> ident  OMNI2
> 
> options  SMP
> 
> options  SCHED_4BSD  # 4BSD scheduler
> options  INET   # InterNETworking
> options  INET6   # IPv6 communications protocols
> options  FFS   # Berkeley Fast Filesystem
> options  SOFTUPDATES  # Enable FFS soft updates support
> options  UFS_ACL   # Support for access control lists
> options  UFS_DIRHASH  # Improve performance on big directories
> options  MD_ROOT   # MD is a potential root device
> options  NFSCLIENT  # Network Filesystem Client
> options  NFSSERVER  # Network Filesystem Server
> options  NFS_ROOT  # NFS usable as /, requires NFSCLIENT
> options  MSDOSFS   # MSDOS Filesystem
> options  CD9660   # ISO 9660 Filesystem
> options  PROCFS   # Process filesystem (requires PSEUDOFS)
> options  PSEUDOFS  # Pseudo-filesystem framework
> options  GEOM_GPT  # GUID Partition Tables.
> options  COMPAT_43  # Compatible with BSD 4.3 [KEEP THIS!]
> options  COMPAT_FREEBSD4  # Compatible with FreeBSD4
> options  SCSI_DELAY=15000 # Delay (in ms) before probing SCSI
> #options  KTRACE   # ktrace(1) support
> options  SYSVSHM   # SYSV-style shared memory
> options  SYSVMSG   # SYSV-style message queues
> options  SYSVSEM   # SYSV-style semaphores
> options  _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
> options  KBD_INSTALL_CDEV # install a CDEV entry in /dev
> options  ADAPTIVE_GIANT  # Giant mutex is adaptive.
> 
> device  apic  # I/O APIC
> 
> # Bus support.  Do not remove isa, even if you have no isa slots
> device  isa
> device  pci
> 
> # Floppy drives
> device  fdc
> 
> # ATA and ATAPI devices
> device  ata
> device  atadisk  # ATA disk drives
> device  ataraid  # ATA RAID drives
> device  atapicd  # ATAPI CDROM drives
> device  atapifd  # ATAPI floppy drives
> device  atapist  # ATAPI tape drives
> options  ATA_STATIC_ID # Static device numbering
> 
> # SCSI peripherals
> device  scbus  # SCSI bus (required for SCSI)
> device  ch  # SCSI media changers
> device  da  # Direct Access (disks)
> device  sa  # Sequential Access (tape etc)
> device  cd  # CD
> device  pass  # Passthrough device (direct SCSI access)
> device  ses  # SCSI Environmental Services (and SAF-TE)
> 
> device  twe  # 3ware ATA RAID
> 
> # atkbdc0 controls both the keyboard and the PS/2 mouse
> device  atkbdc  # AT keyboard controller
> device  atkbd  # AT keyboard
> device  psm  # PS/2 mouse
> 
> device  vga  # VGA video card driver
> 
> device  splash  # Splash screen and screen saver support
> 
> # syscons is the default console driver, resembling an SCO console
> device  sc
> 
> device  agp  # support several AGP chipsets
> 
> # Floating point support - do not disable.
> device  npx
> 
> # Power management support (see NOTES for more options)
> #device  apm
> # Add suspend/resume support for the i8254.
> #device  pmtimer
> 
> # Serial (COM) ports
> device  sio  # 8250, 16[45]50 based serial ports
> 
> # Parallel port
> device  ppc
> device  ppbus  # Parallel port bus (required)
> device  lpt  # Printer
> device  ppi  # Parallel port interface device
> #device  vpo  # Requires scbus and da
> 
> 
> device  miibus  # MII bus support
> device  fxp  # Intel EtherExpress PRO/100B (82557, 82558)
> device  em
> 
> 
> device  loop  # Network loopback
> device  mem  # Memory and kernel memory devices
> device  io  # I/O device
> device  random  # Entropy device
> device  ether  # Ethernet support
> #device  sl  # Kernel SLIP
> #device  ppp  # Kernel PPP
> device  tun  # Packet tunnel.
> device  pty  # Pseudo-ttys (telnet etc)
> device  md  # Memory "disks"
> #device  gif  # IPv6 and IPv4 tunneling
> #device  faith  # IPv6-to-IPv4 relaying (translation)
> 
> device  bpf  # Berkeley packet filter
> 
> # USB support
> device  uhci  # UHCI PCI->USB interface
> device  ohci  # OHCI PCI->USB interface
> device  usb  # USB Bus (required)
> #device  udbp  # USB Double Bulk Pipe devices
> device  ugen  # Generic
> device  uhid  # "Human Interface Devices"
> device  ukbd  # Keyboard
> device  ulpt  # Printer
> device  umass  # Disks/Mass storage - Requires scbus and da
> device  ums  # Mouse
> 
> 
> # FireWire support
> device  firewire # FireWire bus code
> device  sbp  # SCSI over FireWire (Requires scbus and da)
> device  fwe  # Ethernet over FireWire (non-standard!)
> 
> 
> ------- SCRIPT TO SAVE SYSTEM STATUS ------------------
> 
> echo '---------------------------------------------------' >> /var/log/_stat
> date >> /var/log/_stat
> echo '---------------------------------------------------' >> /var/log/_stat
> uptime >> /var/log/_stat
> echo '---------------------------------------------------' >> /var/log/_stat
> ps -axwlH >> /var/log/_stat
> echo '---------------------------------------------------' >> /var/log/_stat
> iostat -c 2 -w 10 >> /var/log/_stat
> echo '---------------------------------------------------' >> /var/log/_stat
> jls >> /var/log/_stat
> echo '---------------------------------------------------' >> /var/log/_stat
> df >> /var/log/_stat
> echo '---------------------------------------------------' >> /var/log/_stat
> vmstat -f >> /var/log/_stat
> echo '---------------------------------------------------' >> /var/log/_stat
> vmstat -a  >> /var/log/_stat
> echo '---------------------------------------------------' >> /var/log/_stat
> vmstat -i  >> /var/log/_stat
> echo '---------------------------------------------------' >> /var/log/_stat
> netstat -an   >> /var/log/_stat
> echo '---------------------------------------------------' >> /var/log/_stat
> top -b -d1 >> /var/log/_stat
> echo '---------------------------------------------------' >> /var/log/_stat
> 
> 
> ----------- LAST WORKING MINUTE STATS ---------------------------------------
> 
> 
> ---------------------------------------------------
> Thu Jan  6 19:56:00 MSK 2005
> ---------------------------------------------------
>  7:56PM  up  4:28, 0 users, load averages: 0.06, 0.03, 0.01
> ---------------------------------------------------
>   UID   PID  PPID CPU PRI NI   VSZ  RSS MWCHAN STAT  TT       TIME COMMAND
>     0     0     0   0 -16  0     0    4 sched  DLs   ??    0:00.04 [swapper]
>     0     1     0   0   8  0   740  376 wait   ILs   ??    0:00.01 /sbin/init --
>     0     2     0   0  -8  0     0   12 -      DL    ??    0:01.67 [g_event]
>     0     3     0   0  -8  0     0   12 -      DL    ??    0:01.79 [g_up]
>     0     4     0   0  -8  0     0   12 -      DL    ??    0:01.78 [g_down]
>     0     5     0   0   8  0     0   12 -      DL    ??    0:00.00 [thread taskq]
>     0     6     0   0   8  0     0   12 -      DL    ??    0:00.00 [kqueue taskq]
>     0     7     0   0 -84  0     0   12 actask IL    ??    0:00.00 [acpi_task0]
>     0     8     0   0 -84  0     0   12 actask IL    ??    0:00.00 [acpi_task1]
>     0     9     0   0 -84  0     0   12 actask IL    ??    0:00.00 [acpi_task2]
>     0    10     0  45  97  0     0   12 -      RL    ??  267:38.74 [idle: cpu3]
>     0    11     0  44  97  0     0   12 -      RL    ??  267:24.91 [idle: cpu2]
>     0    12     0  45  97  0     0   12 -      RL    ??  267:22.85 [idle: cpu1]
>     0    13     0  44  97  0     0   12 -      RL    ??  267:23.30 [idle: cpu0]
>     0    14     0   0 -60  0     0   12 -      WL    ??    0:00.00 [irq1: atkbd0]
>     0    15     0   0 -60  0     0   12 -      WL    ??    0:00.00 [irq3: sio1]
>     0    16     0   0 -60  0     0   12 -      WL    ??    0:00.00 [irq4: sio0]
>     0    17     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq5:]
>     0    18     0   0 -64  0     0   12 -      WL    ??    0:00.00 [irq6: fdc0]
>     0    19     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq7:]
>     0    20     0   0 -84  0     0   12 -      WL    ??    0:00.00 [irq8: rtc]
>     0    21     0   0 -52  0     0   12 -      WL    ??    0:00.00 [irq9: acpi0]
>     0    22     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq10:]
>     0    23     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq11:]
>     0    24     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq12:]
>     0    25     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq13:]
>     0    26     0   0 -64  0     0   12 -      WL    ??    0:00.00 [irq14: ata0]
>     0    27     0   0 -64  0     0   12 -      WL    ??    0:00.00 [irq15: ata2]
>     0    28     0   0 -64  0     0   12 -      WL    ??    0:00.00 [irq16: uhci0]
>     0    29     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq17:]
>     0    30     0   0 -64  0     0   12 -      WL    ??    0:00.00 [irq18: uhci2]
>     0    31     0   0 -64  0     0   12 -      WL    ??    0:00.00 [irq19: uhci1]
>     0    32     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq20:]
>     0    33     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq21:]
>     0    34     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq22:]
>     0    35     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq23:]
>     0    36     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq24:]
>     0    37     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq25:]
>     0    38     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq26:]
>     0    39     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq27:]
>     0    40     0   0 -68  0     0   12 -      WL    ??    0:00.40 [irq28: em0]
>     0    41     0   0 -68  0     0   12 -      WL    ??    0:00.00 [irq29: em1]
>     0    42     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq30:]
>     0    43     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq31:]
>     0    44     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq32:]
>     0    45     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq33:]
>     0    46     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq34:]
>     0    47     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq35:]
>     0    48     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq36:]
>     0    49     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq37:]
>     0    50     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq38:]
>     0    51     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq39:]
>     0    52     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq40:]
>     0    53     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq41:]
>     0    54     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq42:]
>     0    55     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq43:]
>     0    56     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq44:]
>     0    57     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq45:]
>     0    58     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq46:]
>     0    59     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq47:]
>     0    60     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq48:]
>     0    61     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq49:]
>     0    62     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq50:]
>     0    63     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq51:]
>     0    64     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq52:]
>     0    65     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq53:]
>     0    66     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq54:]
>     0    67     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq55:]
>     0    68     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq56:]
>     0    69     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq57:]
>     0    70     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq58:]
>     0    71     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq59:]
>     0    72     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq60:]
>     0    73     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq61:]
>     0    74     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq62:]
>     0    75     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq63:]
>     0    76     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq64:]
>     0    77     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq65:]
>     0    78     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq66:]
>     0    79     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq67:]
>     0    80     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq68:]
>     0    81     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq69:]
>     0    82     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq70:]
>     0    83     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq71:]
>     0    84     0   0 -64  0     0   12 -      WL    ??    0:00.50 [irq72: twe0]
>     0    85     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq73:]
>     0    86     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq74:]
>     0    87     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq75:]
>     0    88     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq76:]
>     0    89     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq77:]
>     0    90     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq78:]
>     0    91     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq79:]
>     0    92     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq80:]
>     0    93     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq81:]
>     0    94     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq82:]
>     0    95     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq83:]
>     0    96     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq84:]
>     0    97     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq85:]
>     0    98     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq86:]
>     0    99     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq87:]
>     0   100     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq88:]
>     0   101     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq89:]
>     0   102     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq90:]
>     0   103     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq91:]
>     0   104     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq92:]
>     0   105     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq93:]
>     0   106     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq94:]
>     0   107     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq95:]
>     0   108     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq96:]
>     0   109     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq97:]
>     0   110     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq98:]
>     0   111     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq99:]
>     0   112     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq100:]
>     0   113     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq101:]
>     0   114     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq102:]
>     0   115     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq103:]
>     0   116     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq104:]
>     0   117     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq105:]
>     0   118     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq106:]
>     0   119     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq107:]
>     0   120     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq108:]
>     0   121     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq109:]
>     0   122     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq110:]
>     0   123     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq111:]
>     0   124     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq112:]
>     0   125     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq113:]
>     0   126     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq114:]
>     0   127     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq115:]
>     0   128     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq116:]
>     0   129     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq117:]
>     0   130     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq118:]
>     0   131     0   0 -21  0     0   12 -      WL    ??    0:00.00 [irq119:]
>     0   132     0   0 -84  0     0   12 -      WL    ??    0:00.00 [irq0: clk]
>     0   133     0   0 -44  0     0   12 -      WL    ??    0:00.06 [swi1: net]
>     0   134     0   0 -28  0     0   12 -      WL    ??    0:22.54 [swi5: clock sio]
>     0   135     0   0 -32  0     0   12 -      WL    ??    0:00.00 [swi4: vm]
>     0   136     0   0 -16  0     0   12 -      DL    ??    0:01.62 [yarrow]
>     0   137     0   0 -24  0     0   12 -      WL    ??    0:00.00 [swi6:+]
>     0   138     0   0 -40  0     0   12 -      WL    ??    0:00.00 [swi2: camnet]
>     0   139     0   0 -36  0     0   12 -      WL    ??    0:00.00 [swi3: cambio]
>     0   140     0   0 -24  0     0   12 -      WL    ??    0:00.00 [swi6: acpitaskq]
>     0   141     0   0 -24  0     0   12 -      WL    ??    0:00.00 [swi6: task queue]
>     0   142     0   0 -24  0     0   12 -      WL    ??    0:00.00 [swi6:+]
>     0   143     0   0   8  0     0   12 usbevt DL    ??    0:00.00 [usb0]
>     0   144     0   0   8  0     0   12 usbtsk DL    ??    0:00.00 [usbtask]
>     0   145     0   0   8  0     0   12 usbevt DL    ??    0:00.00 [usb1]
>     0   146     0   0   8  0     0   12 usbevt DL    ??    0:00.00 [usb2]
>     0   147     0   0 -48  0     0   12 -      WL    ??    0:00.00 [swi0: sio]
>     0   148     0   0  -8  0     0   12 -      DL    ??    0:00.17 [fdc0]
>     0   149     0   0 -16  0     0   12 psleep DL    ??    0:00.04 [pagedaemon]
>     0   150     0   0  20  0     0   12 psleep DL    ??    0:00.00 [vmdaemon]
>     0   151     0   0 171  0     0   12 pgzero DL    ??    0:02.31 [pagezero]
>     0   152     0   0 -16  0     0   12 psleep DL    ??    0:00.60 [bufdaemon]
>     0   153     0   0  -4  0     0   12 vlruwt DL    ??    0:00.58 [vnlru]
>     0   154     0   0  20  0     0   12 syncer DL    ??    0:00.66 [syncer]
>     0   155     0   0   8  0     0   12 -      IL    ??    0:00.00 [nfsiod 0]
>     0   156     0   0   8  0     0   12 -      IL    ??    0:00.00 [nfsiod 1]
>     0   157     0   0   8  0     0   12 -      IL    ??    0:00.00 [nfsiod 2]
>     0   158     0   0   8  0     0   12 -      IL    ??    0:00.00 [nfsiod 3]
>     0   159     0   0 -16  0     0   12 -      DL    ??    0:01.52 [schedcpu]
>     0   278     1 611  20  0  1188  640 pause  ILs   ??    0:00.00 adjkerntz -i
>     0   339     1 611 115  0   516  360 select ILs   ??    0:00.00 /sbin/devd
>     0   363     1   0  96  0  1320  864 select SLs   ??    0:00.10 /usr/sbin/syslogd -l /var/run/log -l /var/named/var/run/log -ss
>    53   377     1   0  20  0  5552 4288 kserel ILs   ??    0:00.50 /usr/sbin/named -u bind -t /var/named
>    53   377     1   0  20  0  5552 4288 kserel ILs   ??    0:00.50 /usr/sbin/named -u bind -t /var/named
>    53   377     1   0  20  0  5552 4288 ksesig SLs   ??    0:00.50 /usr/sbin/named -u bind -t /var/named
>    53   377     1   0  20  0  5552 4288 kserel ILs   ??    0:00.50 /usr/sbin/named -u bind -t /var/named
>    53   377     1   0  96  0  5552 4288 select ILs   ??    0:00.50 /usr/sbin/named -u bind -t /var/named
>    53   377     1   0  20  0  5552 4288 kserel ILs   ??    0:00.50 /usr/sbin/named -u bind -t /var/named
>     0   488     1   0  96  0  3364 2516 select ILs   ??    0:00.01 /usr/sbin/sshd
>     0   505     1   1   8  0  1360 1052 nanslp SLs   ??    0:00.17 /usr/sbin/cron -s
>     0   619     1   0  96  0  1320  872 select ILsJ  ??    0:00.03 /usr/sbin/syslogd -ss
>     0   671     1   0  96  0  3364 2492 select ILsJ  ??    0:00.00 /usr/sbin/sshd
>     0   678     1   0   8  0  1384 1060 nanslp SLsJ  ??    0:00.04 /usr/sbin/cron -s
>     0   737     1   0  96  0  1320  888 select ILsJ  ??    0:00.05 /usr/sbin/syslogd -s
>     0   789     1   0  96  0  3364 2508 select ILsJ  ??    0:00.00 /usr/sbin/sshd
>     0   796     1  69   8  0  1384 1060 nanslp ILsJ  ??    0:00.06 /usr/sbin/cron -s
>     0   801     1   0  96  0  1668 1216 select ILsJ  ??    0:00.00 /usr/libexec/ftpd -D -S
>     0  4756     1   0   4  0  4880 2856 sbwait ILJ   ??    0:00.00 sshd: rooter [pam] (sshd)
>     0  5869   505   1  -8  0  1360 1136 piperd SL    ??    0:00.00 cron: running job (cron)
>     0  5870  5869   1   8  0  1640 1160 wait   SLs   ??    0:00.00 /bin/sh -c /root/logstatus.sh
>     0  5871  5870   5   8  0  1644 1164 wait   SL    ??    0:00.00 /bin/sh -c /root/logstatus.sh
>     0  5874  5871   5  96  0  1496  988 -      RL    ??    0:00.00 ps -axwlH
>     0   835     1 567   5  0  1284  924 ttyin  ILs+  v0    0:00.01 /usr/libexec/getty Pc ttyv0
>     0   836     1 567   5  0  1284  928 ttyin  ILs+  v1    0:00.01 /usr/libexec/getty Pc ttyv1
>     0   837     1 567   5  0  1284  928 ttyin  ILs+  v2    0:00.01 /usr/libexec/getty Pc ttyv2
>     0   838     1 567   5  0  1284  928 ttyin  ILs+  v3    0:00.01 /usr/libexec/getty Pc ttyv3
>     0   839     1 567   5  0  1284  928 ttyin  ILs+  v4    0:00.01 /usr/libexec/getty Pc ttyv4
>     0   840     1 567   5  0  1284  928 ttyin  ILs+  v5    0:00.01 /usr/libexec/getty Pc ttyv5
>     0   841     1 567   5  0  1284  928 ttyin  ILs+  v6    0:00.01 /usr/libexec/getty Pc ttyv6
>     0   842     1 567   5  0  1284  928 ttyin  ILs+  v7    0:00.01 /usr/libexec/getty Pc ttyv7
> ---------------------------------------------------
>       tty           twed0             cpu
>  tin tout  KB/t tps  MB/s  us ni sy in id
>    0    2 14.46   1  0.01   0  0  0  0 100
>    0    0 16.00   0  0.01   0  0  0  0 100
> ---------------------------------------------------
>    JID  IP Address      Hostname                      Path
>      2  213.252.68.6    jail2.itlegion.ru             /hosts/213.252.68.6
>      1  213.252.68.5    jail1.itlegion.ru             /hosts/213.252.68.5
> ---------------------------------------------------
> Filesystem    512-blocks    Used     Avail Capacity  Mounted on
> /dev/twed0s1a    2025948  112836   1751040     6%    /
> devfs                  2       2         0   100%    /dev
> /dev/twed0s1f  650705836 1939120 596710252     0%    /hosts
> /dev/twed0s1e   20308316 5340096  13343556    29%    /usr
> /dev/twed0s1d    6090012   98196   5504616     2%    /var
> devfs                  2       2         0   100%    /var/named/dev
> devfs                  2       2         0   100%    /hosts/213.252.68.5/dev
> devfs                  2       2         0   100%    /hosts/213.252.68.6/dev
> ---------------------------------------------------
> 5182 forks, 247213 pages, average 47.71
> 537 vforks, 32666 pages, average 60.83
> 0 rforks, 0 pages, average 0.00
> ---------------------------------------------------
>  procs      memory      page                   disk   faults      cpu
>  r b w     avm    fre  flt  re  pi  po  fr  sr tw0   in   sy  cs us sy id
>  0 1 0   40420 3920948   30   0   0   0  28   0   0  330   50 621  0  0 100
> ---------------------------------------------------
> interrupt                          total       rate
> irq6: fdc0                             9          0
> irq8: rtc                        2057141        127
> irq13: npx0                            1          0
> irq14: ata0                           47          0
> irq28: em0                          8184          0
> irq72: twe0                         9739          0
> irq0: clk                        1607249         99
> Total                            3682370        229
> ---------------------------------------------------
> Active Internet connections (including servers)
> Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
> tcp4       0      0  213.252.68.5.22        195.90.153.20.3119     CLOSE_WAIT
> tcp4       0      0  213.252.68.6.21        *.*                    LISTEN
> tcp4       0      0  213.252.68.6.22        *.*                    LISTEN
> tcp4       0      0  213.252.68.5.22        *.*                    LISTEN
> tcp4       0      0  213.252.68.2.22        *.*                    LISTEN
> tcp6       0      0  ::1.953                *.*                    LISTEN
> tcp4       0      0  127.0.0.1.953          *.*                    LISTEN
> tcp4       0      0  127.0.0.1.53           *.*                    LISTEN
> tcp4       0      0  213.252.68.2.53        *.*                    LISTEN
> udp4       0      0  213.252.68.6.514       *.*
> udp6       0      0  *.61254                *.*
> udp4       0      0  *.61253                *.*
> udp4       0      0  127.0.0.1.53           *.*
> udp4       0      0  213.252.68.2.53        *.*
> Active UNIX domain sockets
> Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
> c5854d20 stream      0      0        0 c585dd20        0        0
> c585dd20 stream      0      0        0 c5854d20        0        0
> c5854b7c stream      0      0        0        0        0        0
> c5855000 stream      0      0 c5850108        0        0        0 /var/run/devd.pipe
> c5854e38 dgram       0      0        0 c596c118        0        0
> c596c118 dgram       0      0 c59be948        0 c5854e38        0 /var/run/logpriv
> c596c1a4 dgram       0      0 c5962948        0        0        0 /var/run/log
> c596c230 dgram       0      0 c5930c60        0        0        0 /var/run/logpriv
> c596c2bc dgram       0      0 c5930d68        0        0        0 /var/run/log
> c5854ec4 dgram       0      0        0 c585d08c        0        0
> c5855ec4 dgram       0      0 c584f948        0        0        0 /var/named/var/run/log
> c585d000 dgram       0      0 c5782a50        0        0        0 /var/run/log
> c585d08c dgram       0      0 c5782b58        0 c5854ec4        0 /var/run/logpriv
> c585d118 dgram       0      0 c5782c60        0        0        0 /var/run/log
> ---------------------------------------------------
> last pid:  5882;  load averages:  0.05,  0.02,  0.00  up 0+04:28:24    19:56:10
> 31 processes:  1 running, 30 sleeping
> 
> Mem: 9280K Active, 15M Inact, 40M Wired, 1508K Cache, 28M Buf, 3827M Free
> Swap: 1024M Total, 1024M Free
> 
> 
>   PID USERNAME PRI NICE   SIZE    RES STATE  C   TIME   WCPU    CPU COMMAND
>   377 bind      20    0  5552K  4288K kserel 2   0:01  0.00%  0.00% named
>   505 root       8    0  1360K  1052K nanslp 0   0:00  0.00%  0.00% cron
>   363 root      96    0  1320K   864K select 0   0:00  0.00%  0.00% syslogd
>   796 root       8    0  1384K  1060K nanslp 0   0:00  0.00%  0.00% cron
>   737 root      96    0  1320K   888K select 0   0:00  0.00%  0.00% syslogd
>   678 root       8    0  1384K  1060K nanslp 0   0:00  0.00%  0.00% cron
>   619 root      96    0  1320K   872K select 0   0:00  0.00%  0.00% syslogd
>   837 root       5    0  1284K   928K ttyin  1   0:00  0.00%  0.00% getty
>   835 root       5    0  1284K   924K ttyin  0   0:00  0.00%  0.00% getty
>   836 root       5    0  1284K   928K ttyin  0   0:00  0.00%  0.00% getty
>   841 root       5    0  1284K   928K ttyin  0   0:00  0.00%  0.00% getty
>   838 root       5    0  1284K   928K ttyin  1   0:00  0.00%  0.00% getty
>   839 root       5    0  1284K   928K ttyin  3   0:00  0.00%  0.00% getty
>   842 root       5    0  1284K   928K ttyin  2   0:00  0.00%  0.00% getty
>  5871 root       8    0  1644K  1164K wait   1   0:00  0.00%  0.00% sh
>   840 root       5    0  1284K   928K ttyin  2   0:00  0.00%  0.00% getty
>   488 root      96    0  3364K  2516K select 0   0:00  0.00%  0.00% sshd
>  5870 root       8    0  1640K  1160K wait   2   0:00  0.00%  0.00% sh
> 
> ---------------------------------------------------
> 
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
> 



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.NEB.3.96L.1050108213144.43829I-100000>