From owner-freebsd-stable@FreeBSD.ORG Sun Dec 11 15:18:42 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 567A1106564A for ; Sun, 11 Dec 2011 15:18:42 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id CBFC18FC12 for ; Sun, 11 Dec 2011 15:18:41 +0000 (UTC) Received: by eaaf13 with SMTP id f13so391243eaa.13 for ; Sun, 11 Dec 2011 07:18:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=/ZG9mA+N068k+hbmJVl5x8KIpnhBSHcl7s9pbpS1Zxs=; b=NruznPvYYY30OoVhJwkQvhJga0e2TNufs2nYI+6CF+1jockTdsMsw774GVvos8xbFi HRL06GTLbrUThvkD+T2CybFzieSYoSVw1pkOWyuGD5hh2JfMoGHNYlkmTb/nXtV0T7hv WGo0T0P0TwgKwRtqdU05UCqFhmKhimnpDKQZQ= Received: by 10.213.8.199 with SMTP id i7mr1308073ebi.129.1323616720709; Sun, 11 Dec 2011 07:18:40 -0800 (PST) Received: from [192.168.1.13] (5ED0E470.cm-7-1d.dynamic.ziggo.nl. [94.208.228.112]) by mx.google.com with ESMTPS id 53sm62269588eef.2.2011.12.11.07.18.39 (version=SSLv3 cipher=OTHER); Sun, 11 Dec 2011 07:18:40 -0800 (PST) Message-ID: <4EE4C9CD.3000906@gmail.com> Date: Sun, 11 Dec 2011 16:18:37 +0100 From: Johan Hendriks User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable References: <201112090913.CAA03333@lariat.net> <4EE1DEAD.2040108@gmail.com> <201112091731.KAA08794@lariat.net> <20111209224610.GA13369@icarus.home.lan> In-Reply-To: <20111209224610.GA13369@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Two problems still present in RC3 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Dec 2011 15:18:42 -0000 Jeremy Chadwick schreef: > On Fri, Dec 09, 2011 at 10:30:43AM -0700, Brett Glass wrote: >> At 03:10 AM 12/9/2011, Johan Hendriks wrote: >> >>> After a /etc/netstart, i get the following: >>> ifconfig: create: bad value. >> I get the "create: bad value" messages as well. What's more, if I >> change rc.conf to assign variables of the form ifconfig_*, such as >> >> ifconfig_re0_1="inet 192.168.0.1" >> ifconfig_re0_1_alias0="inet 192.168.0.2" >> ... >> >> instead of using an ip_addrs_* variable, I still get those messages. >> >> I have confirmed that if I run /etc/netstart after booting, I lose >> connectivity just as you do. >> >> I recognize the challenge of specifying the network configuration in >> a declaratory rather than a procedural environment, because there >> are so many contingencies and possible combinations of parameters. >> But there doesn't seem to be any combination of variables I can >> assign in rc.conf that doesn't cause errors when I try to create >> VLANs. > I would advise everyone experiencing ""those messages"" or any anomalies > of this sort to please set rc_debug="yes" in /etc/rc.conf and make sure > they have serial or firewire console set up. If you don't have some > form of remote console, then this is going to be difficult to debug. > See rc.conf(5) for what the option does and therefore why serial console > will be needed. > > The information that's needed is what ifconfig commands are being issued > on your systems. This cannot be easily determined from rc.conf variables > like ifconfig_snakes_in_the_grass="rocks moss", hence the above request. > > "Your" means Brett Glass and Johan Hendriks. > > Finally, please keep in mind the possibility that both of you are > experiencing different problems. I have no evidence at this point to > support or refute that claim, but it's something that needs to be > considered. > Ok did some test on the backup machine, i must do it remotely, so no serial access. Lagg0 consist of two interfaces em0 and em1. The machine booted with the following in /etc/rc.conf, and all is fine this way. ifconfig_em0="up" ifconfig_em1="up" cloned_interfaces="lagg0" ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 192.168.100.1/24" I changed the ip adress from 192.168.100.1 to 192.168.100.2 in /etc/rc.conf, and added rc_debug="yes" ifconfig_em0="up" ifconfig_em1="up" cloned_interfaces="lagg0" ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 192.168.100.2/24" I did a /etc/netstart nas01-bw ~ # /etc/netstart /etc/rc.d/devd: DEBUG: checkyesno: devd_enable is set to YES. devd already running? (pid=1767). /etc/rc.d/hostid: DEBUG: checkyesno: hostid_enable is set to YES. /etc/rc.d/hostid: DEBUG: run_rc_command: doit: hostid_start /etc/rc.d/hostid: DEBUG: checkyesno: rc_startmsgs is set to YES. Setting hostuuid: 9abf4e13-9a77-b31d-9a77-b31d9fbfee13. /etc/rc.d/hostid: DEBUG: checkyesno: rc_startmsgs is set to YES. Setting hostid: 0xf505f6a8. /etc/rc.d/hostname: DEBUG: run_rc_command: doit: hostname_start /etc/rc.d/ipmon: DEBUG: checkyesno: ipmon_enable is set to NO. /etc/rc.d/ipfilter: DEBUG: checkyesno: ipfilter_enable is set to NO. /etc/rc.d/ipnat: DEBUG: checkyesno: ipnat_enable is set to NO. /etc/rc.d/ipfs: DEBUG: checkyesno: ipfs_enable is set to NO. /etc/rc.d/sppp: DEBUG: run_rc_command: doit: sppp_start /etc/rc.d/netif: DEBUG: run_rc_command: doit: network_start ifconfig: create: bad value /etc/rc.d/netif: DEBUG: Cloned: /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. ifconfig: SIOCSLAGGPORT: Device busy /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. Starting Network: lo0 em0 em1 bge0 pflog0 pfsync0 lagg0. /etc/rc.d/netif: DEBUG: checkyesno: rc_startmsgs is set to YES. lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xe inet 127.0.0.1 netmask 0xff000000 nd6 options=21 em0: flags=8843 metric 0 mtu 1500 options=219b ether 00:1b:21:d4:77:fb inet6 fe80::21b:21ff:fed4:77fb%em0 prefixlen 64 scopeid 0x5 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8843 metric 0 mtu 1500 options=219b ether 00:1b:21:d4:77:fb inet6 fe80::21b:21ff:fed4:749b%em1 prefixlen 64 scopeid 0x6 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active bge0: flags=8843 metric 0 mtu 1500 options=c01db ether 00:23:7d:f4:4a:55 inet6 fe80::223:7dff:fef4:4a55%bge0 prefixlen 64 scopeid 0x7 inet 192.168.1.17 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=29 media: Ethernet autoselect (none) status: no carrier pflog0: flags=0<> metric 0 mtu 33152 nd6 options=29 pfsync0: flags=0<> metric 0 mtu 1500 nd6 options=29 syncpeer: 0.0.0.0 maxupd: 128 lagg0: flags=8843 metric 0 mtu 1500 options=219b ether 00:1b:21:d4:77:fb inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255 inet6 fe80::21b:21ff:fed4:77fb%lagg0 prefixlen 64 scopeid 0xf nd6 options=29 media: Ethernet autoselect status: active laggproto lacp laggport: em1 flags=0<> laggport: em0 flags=0<> /etc/rc.d/netif: DEBUG: The following interfaces were not configured: /etc/rc.d/ipfilter: DEBUG: checkyesno: ipfilter_enable is set to NO. /etc/rc.d/ipsec: DEBUG: checkyesno: ipsec_enable is set to NO. /etc/rc.d/ppp: DEBUG: checkyesno: ppp_enable is set to NO. /etc/rc.d/ipfw: DEBUG: checkyesno: firewall_enable is set to NO. /etc/rc.d/routing: DEBUG: run_rc_command: doit: routing_start doall add net default: gateway 192.168.1.254 /etc/rc.d/routing: DEBUG: checkyesno: icmp_bmcastecho is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: icmp_drop_redirect is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: icmp_log_redirect is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: gateway_enable is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: forward_sourceroute is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: accept_sourceroute is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: arpproxy_all is set to NO. route: writing to routing socket: File exists add net ::ffff:0.0.0.0: gateway ::1: route already in table route: writing to routing socket: File exists add net ::0.0.0.0: gateway ::1: route already in table /etc/rc.d/routing: DEBUG: checkyesno: ipv6_gateway_enable is set to NO. route: writing to routing socket: File exists add net fe80::: gateway ::1: route already in table route: writing to routing socket: File exists add net ff02::: gateway ::1: route already in table /etc/rc.d/routing: DEBUG: checkyesno: ipv6_gatew ay_enable is set to NO. /etc/rc.d/mroute6d: DEBUG: checkyesno: mroute6d_enable is set to NO. /etc/rc.d/route6d: DEBUG: checkyesno: route6d_enable is set to NO. /etc/rc.d/mrouted: DEBUG: pid file (/var/run/mrouted.pid): not readable. /etc/rc.d/mrouted: DEBUG: checkyesno: mrouted_enable is set to NO. /etc/rc.d/routed: DEBUG: checkyesno: routed_enable is set to NO. /etc/rc.d/nisdomain: DEBUG: run_rc_command: doit: nisdomain_start As you can see i get the bad value, and secondly, my ipadress of lagg0 has not changed, it is still 192.168.100.1 nas01-bw ~ # ifconfig lagg0 lagg0: flags=8843 metric 0 mtu 1500 options=219b ether 00:1b:21:d4:77:fb inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255 inet6 fe80::21b:21ff:fed4:77fb%lagg0 prefixlen 64 scopeid 0xf nd6 options=29 media: Ethernet autoselect status: active laggproto lacp laggport: em1 flags=1c laggport: em0 flags=1c The strange thing is i can not get rid off the bad value error anymore. i changed the line in /etc/rc.conf to the following as suggested and add the inet statement ifconfig_em0="up" ifconfig_em1="up" cloned_interfaces="lagg0" ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 inet 192.168.100.2/24" still the same after a /etc/netstart nas01-bw ~ # /etc/netstart /etc/rc.d/devd: DEBUG: checkyesno: devd_enable is set to YES. devd already running? (pid=1767). /etc/rc.d/hostid: DEBUG: checkyesno: hostid_enable is set to YES. /etc/rc.d/hostid: DEBUG: run_rc_command: doit: hostid_start /etc/rc.d/hostid: DEBUG: checkyesno: rc_startmsgs is set to YES. Setting hostuuid: 9abf4e13-9a77-b31d-9a77-b31d9fbfee13. /etc/rc.d/hostid: DEBUG: checkyesno: rc_startmsgs is set to YES. Setting hostid: 0xf505f6a8. /etc/rc.d/hostname: DEBUG: run_rc_command: doit: hostname_start /etc/rc.d/ipmon: DEBUG: checkyesno: ipmon_enable is set to NO. /etc/rc.d/ipfilter: DEBUG: checkyesno: ipfilter_enable is set to NO. /etc/rc.d/ipnat: DEBUG: checkyesno: ipnat_enable is set to NO. /etc/rc.d/ipfs: DEBUG: checkyesno: ipfs_enable is set to NO. /etc/rc.d/sppp: DEBUG: run_rc_command: doit: sppp_start /etc/rc.d/netif: DEBUG: run_rc_command: doit: network_start ifconfig: create: bad value /etc/rc.d/netif: DEBUG: Cloned: /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. ifconfig: SIOCSLAGGPORT: Device busy /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. Starting Network: lo0 em0 em1 bge0 pflog0 pfsync0 lagg0. /etc/rc.d/netif: DEBUG: checkyesno: rc_startmsgs is set to YES. lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xe inet 127.0.0.1 netmask 0xff000000 nd6 options=21 em0: flags=8843 metric 0 mtu 1500 options=219b ether 00:1b:21:d4:77:fb inet6 fe80::21b:21ff:fed4:77fb%em0 prefixlen 64 scopeid 0x5 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8843 metric 0 mtu 1500 options=219b ether 00:1b:21:d4:77:fb inet6 fe80::21b:21ff:fed4:749b%em1 prefixlen 64 scopeid 0x6 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active bge0: flags=8843 metric 0 mtu 1500 options=c01db ether 00:23:7d:f4:4a:55 inet6 fe80::223:7dff:fef4:4a55%bge0 prefixlen 64 scopeid 0x7 inet 192.168.1.17 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=29 media: Ethernet autoselect (none) status: no carrier pflog0: flags=0<> metric 0 mtu 33152 nd6 options=29 pfsync0: flags=0<> metric 0 mtu 1500 nd6 options=29 syncpeer: 0.0.0.0 maxupd: 128 lagg0: flags=8843 metric 0 mtu 1500 options=219b ether 00:1b:21:d4:77:fb inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255 inet6 fe80::21b:21ff:fed4:77fb%lagg0 prefixlen 64 scopeid 0xf nd6 options=29 media: Ethernet autoselect status: active laggproto lacp laggport: em1 flags=0<> laggport: em0 flags=0<> /etc/rc.d/netif: DEBUG: The following interfaces were not configured: /etc/rc.d/ipfilter: DEBUG: checkyesno: ipfilter_enable is set to NO. /etc/rc.d/ipsec: DEBUG: checkyesno: ipsec_enable is set to NO. /etc/rc.d/ppp: DEBUG: checkyesno: ppp_enable is set to NO. /etc/rc.d/ipfw: DEBUG: checkyesno: firewall_enable is set to NO. /etc/rc.d/routing: DEBUG: run_rc_command: doit: routing_start doall add net default: gateway 192.168.1.254 /etc/rc.d/routing: DEBUG: checkyesno: icmp_bmcastecho is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: icmp_drop_redirect is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: icmp_log_redirect is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: gateway_enable is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: forward_sourceroute is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: accept_sourceroute is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: arpproxy_all is set to NO. route: writing to routing socket: File exists add net ::ffff:0.0.0.0: gateway ::1: route already in table route: writing to routing socket: File exists add net ::0.0.0.0: gateway ::1: route already in table /etc/rc.d/routing: DEBUG: checkyesno: ipv6_gateway_enable is set to NO. route: writing to routing socket: File exists add net fe80::: gateway ::1: route already in table route: writing to routing socket: File exists add net ff02::: gateway ::1: route already in table /etc/rc.d/routing: DEBUG: checkyesno: ipv6_gateway_enable is set to NO. /etc/rc.d/mroute6d: DEBUG: checkyesno: mroute6d_enable is set to NO. /etc/rc.d/route6d: DEBUG: checkyesno: route6d_enable is set to NO. /etc/rc.d/mrouted: DEBUG: pid file (/var/run/mrouted.pid): not readable. /etc/rc.d/mrouted: DEBUG: checkyesno: mrouted_enable is set to NO. /etc/rc.d/routed: DEBUG: checkyesno: routed_enable is set to NO. /etc/rc.d/nisdomain: DEBUG: run_rc_command: doit: nisdomain_start The lagg interface still stays at 192.168.100.1 if i comment out cloned_interfaces="lagg0" i get the following after a /etc/netstart No bad value, but the following ifconfig: SIOCSLAGGPORT: Device busy ifconfig_em0="up" ifconfig_em1="up" #cloned_interfaces="lagg0" ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 inet 192.168.100.2/24" nas01-bw ~ # /etc/netstart /etc/rc.d/devd: DEBUG: checkyesno: devd_enable is set to YES. devd already running? (pid=1767). /etc/rc.d/hostid: DEBUG: checkyesno: hostid_enable is set to YES. /etc/rc.d/hostid: DEBUG: run_rc_command: doit: hostid_start /etc/rc.d/hostid: DEBUG: checkyesno: rc_startmsgs is set to YES. Setting hostuuid: 9abf4e13-9a77-b31d-9a77-b31d9fbfee13. /etc/rc.d/hostid: DEBUG: checkyesno: rc_startmsgs is set to YES. Setting hostid: 0xf505f6a8. /etc/rc.d/hostname: DEBUG: run_rc_command: doit: hostname_start /etc/rc.d/ipmon: DEBUG: checkyesno: ipmon_enable is set to NO. /etc/rc.d/ipfilter: DEBUG: checkyesno: ipfilter_enable is set to NO. /etc/rc.d/ipnat: DEBUG: checkyesno: ipnat_enable is set to NO. /etc/rc.d/ipfs: DEBUG: checkyesno: ipfs_enable is set to NO. /etc/rc.d/sppp: DEBUG: run_rc_command: doit: sppp_start /etc/rc.d/netif: DEBUG: run_rc_command: doit: network_start /etc/rc.d/netif: DEBUG: Cloned: /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. ifconfig: SIOCSLAGGPORT: Device busy /etc/rc.d/netif: DEBUG: checkyesno: ipv6_activate_all_interfaces is set to NO. Starting Network: lo0 em0 em1 bge0 pflog0 pfsync0 lagg0. /etc/rc.d/netif: DEBUG: checkyesno: rc_startmsgs is set to YES. lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xe inet 127.0.0.1 netmask 0xff000000 nd6 options=21 em0: flags=8843 metric 0 mtu 1500 options=219b ether 00:1b:21:d4:77:fb inet6 fe80::21b:21ff:fed4:77fb%em0 prefixlen 64 scopeid 0x5 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8843 metric 0 mtu 1500 options=219b ether 00:1b:21:d4:77:fb inet6 fe80::21b:21ff:fed4:749b%em1 prefixlen 64 scopeid 0x6 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active bge0: flags=8843 metric 0 mtu 1500 options=c01db ether 00:23:7d:f4:4a:55 inet6 fe80::223:7dff:fef4:4a55%bge0 prefixlen 64 scopeid 0x7 inet 192.168.1.17 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=29 media: Ethernet autoselect (none) status: no carrier pflog0: flags=0<> metric 0 mtu 33152 nd6 options=29 pfsync0: flags=0<> metric 0 mtu 1500 nd6 options=29 syncpeer: 0.0.0.0 maxupd: 128 lagg0: flags=8843 metric 0 mtu 1500 options=219b ether 00:1b:21:d4:77:fb inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255 inet6 fe80::21b:21ff:fed4:77fb%lagg0 prefixlen 64 scopeid 0xf nd6 options=29 media: Ethernet autoselect status: active laggproto lacp laggport: em1 flags=0<> laggport: em0 flags=0<> /etc/rc.d/netif: DEBUG: The following interfaces were not configured: /etc/rc.d/ipfilter: DEBUG: checkyesno: ipfilter_enable is set to NO. /etc/rc.d/ipsec: DEBUG: checkyesno: ipsec_enable is set to NO. /etc/rc.d/ppp: DEBUG: checkyesno: ppp_enable is set to NO. /etc/rc.d/ipfw: DEBUG: checkyesno: firewall_enable is set to NO. /etc/rc.d/routing: DEBUG: run_rc_command: doit: routing_start doall add net default: gateway 192.168.1.254 /etc/rc.d/routing: DEBUG: checkyesno: icmp_bmcastecho is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: icmp_drop_redirect is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: icmp_log_redirect is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: gateway_enable is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: forward_sourceroute is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: accept_sourceroute is set to NO. /etc/rc.d/routing: DEBUG: checkyesno: arpproxy_all is set to NO. route: writing to routing socket: File exists add net ::ffff:0.0.0.0: gateway ::1: route already in table route: writing to routing socket: File exists add net ::0.0.0.0: gateway ::1: route already in table /etc/rc.d/routing: DEBUG: checkyesno: ipv6_gateway_enable is set to NO. route: writing to routing socket: File exists add net fe80::: gateway ::1: route already in table route: writing to routing socket: File exists add net ff02::: gateway ::1: route already in table /etc/rc.d/routing: DEBUG: checkyesno: ipv6_gateway_enable is set to NO. /etc/rc.d/mroute6d: DEBUG: checkyesno: mroute6d_enable is set to NO. /etc/rc.d/route6d: DEBUG: checkyesno: route6d_enable is set to NO. /etc/rc.d/mrouted: DEBUG: pid file (/var/run/mrouted.pid): not readable. /etc/rc.d/mrouted: DEBUG: checkyesno: mrouted_enable is set to NO. /etc/rc.d/routed: DEBUG: checkyesno: routed_enable is set to NO. /etc/rc.d/nisdomain: DEBUG: run_rc_command: doit: nisdomain_start I hope this helps. The night i was struggling with the other machine i was able to get rid of the bad value, but it did not set the ipaddres, the same thing i see now. So, the ipadress is not changeable by doing a /etc/netstart on the machine, and only a reboot sets the new address. And it works with no problems what so ever. regards Johan Hendriks From owner-freebsd-stable@FreeBSD.ORG Sun Dec 11 19:04:18 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F4E51065672 for ; Sun, 11 Dec 2011 19:04:18 +0000 (UTC) (envelope-from minimarmot@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 330DA8FC0C for ; Sun, 11 Dec 2011 19:04:17 +0000 (UTC) Received: by yenl9 with SMTP id l9so4286739yen.13 for ; Sun, 11 Dec 2011 11:04:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Qsam2hBWGx0W35ikVZozfp1zefcDIzi7+QJeYxFlHAw=; b=OIpvzN8eMbpwKCKH+UqO/uI67kvGrk+k+NAodf+nsRmSO2PjxvOU4rkacrNOJOdaa1 PeG7z3pYX1qaJM+JiKK2DoZY0+9+cYDyUofY7qnEX7Hge7RQbvOyqNhCkTC6TXLfqeg1 7sSpAxQaLVYAYJ083uyM8i+NiJCmjsAwopMRM= MIME-Version: 1.0 Received: by 10.236.77.72 with SMTP id c48mr21773208yhe.55.1323628583067; Sun, 11 Dec 2011 10:36:23 -0800 (PST) Received: by 10.236.186.1 with HTTP; Sun, 11 Dec 2011 10:36:23 -0800 (PST) In-Reply-To: <201112090913.CAA03333@lariat.net> References: <201112090913.CAA03333@lariat.net> Date: Sun, 11 Dec 2011 13:36:23 -0500 Message-ID: From: Ben Kaduk To: Brett Glass Content-Type: text/plain; charset=ISO-8859-1 Cc: stable@freebsd.org Subject: Re: Two problems still present in RC3 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Dec 2011 19:04:18 -0000 On Fri, Dec 9, 2011 at 4:13 AM, Brett Glass wrote: > Secondly, there's still some strangeness in the sc terminal emulation. When > I run jove, the status line at the bottom of the screen isn't entirely in > reverse video as it should be. Only parts of it are, and the highlighting > changes -- seemingly at random -- as I work. Did you take the change to /etc/ttys going from cons25 to xterm 'type'? -Ben Kaduk From owner-freebsd-stable@FreeBSD.ORG Sun Dec 11 22:33:29 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B2881065676 for ; Sun, 11 Dec 2011 22:33:29 +0000 (UTC) (envelope-from uqs@spoerlein.net) Received: from acme.spoerlein.net (acme.spoerlein.net [IPv6:2a01:4f8:131:23c2::1]) by mx1.freebsd.org (Postfix) with ESMTP id 9CCB08FC16 for ; Sun, 11 Dec 2011 22:33:28 +0000 (UTC) Received: from localhost (acme.spoerlein.net [IPv6:2a01:4f8:131:23c2::1]) by acme.spoerlein.net (8.14.4/8.14.4) with ESMTP id pBBMXRAO033206 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Sun, 11 Dec 2011 23:33:27 +0100 (CET) (envelope-from uqs@spoerlein.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=spoerlein.net; s=dkim200908; t=1323642807; bh=JVLknuTQwD9tveZx/vtyG0P36kXp8+UkJMY4upOVGyE=; h=Date:From:To:Subject:Message-ID:MIME-Version:Content-Type; b=a4Fx/KnJVIMh1C2ePsiPo5GPFAO5iWeGEMojGtmg+NfgmlTJJMoEhG8oCEF4AR0Dw Y22mQVmv0h0whOtpQo8QvfkV6iIlL//xGQF5EXOhRlg2moq87m6UAyB8kiRB07r40N ob0u/KOP6tVMBEvCvKLG9On1eHZoqb/sPyPKYK6Y= Date: Sun, 11 Dec 2011 23:33:27 +0100 From: Ulrich =?utf-8?B?U3DDtnJsZWlu?= To: stable@freebsd.org Message-ID: <20111211223327.GJ83814@acme.spoerlein.net> Mail-Followup-To: stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Subject: stable/9 preferring IPv4 over IPv6, what changed? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Dec 2011 22:33:29 -0000 Hello long story short: telnet foo on stable/8 will first try connecting via IPv6, then via IPv4 (foo has A and AAAA records). On stable/9 it's the other way round. This trips up my setup, where a bunch of hosts (some behind NAT) can all talk to each other over their IPv6 addresses (some are tunneled), but cannot do so via IPv4. Is this due to changes in bind or the resolver? Cheers, Uli From owner-freebsd-stable@FreeBSD.ORG Sun Dec 11 22:52:41 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F881106564A for ; Sun, 11 Dec 2011 22:52:41 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from tensor.andric.com (cl-327.ede-01.nl.sixxs.net [IPv6:2001:7b8:2ff:146::2]) by mx1.freebsd.org (Postfix) with ESMTP id E6B178FC13 for ; Sun, 11 Dec 2011 22:52:40 +0000 (UTC) Received: from [IPv6:2001:7b8:3a7:0:619e:d461:e78e:1ac9] (unknown [IPv6:2001:7b8:3a7:0:619e:d461:e78e:1ac9]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 1DED15C37 for ; Sun, 11 Dec 2011 23:52:38 +0100 (CET) Message-ID: <4EE53431.1040609@FreeBSD.org> Date: Sun, 11 Dec 2011 23:52:33 +0100 From: Dimitry Andric Organization: The FreeBSD Project User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0) Gecko/20111130 Thunderbird/9.0 MIME-Version: 1.0 To: stable@freebsd.org References: <20111211223327.GJ83814@acme.spoerlein.net> In-Reply-To: <20111211223327.GJ83814@acme.spoerlein.net> X-Enigmail-Version: 1.3.4 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: stable/9 preferring IPv4 over IPv6, what changed? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Dec 2011 22:52:41 -0000 On 2011-12-11 23:33, Ulrich Sp=F6rlein wrote: > long story short: telnet foo on stable/8 will first try connecting via > IPv6, then via IPv4 (foo has A and AAAA records). On stable/9 it's the > other way round. >=20 > This trips up my setup, where a bunch of hosts (some behind NAT) can al= l > talk to each other over their IPv6 addresses (some are tunneled), but > cannot do so via IPv4. >=20 > Is this due to changes in bind or the resolver? Most likely due to changes in the IPv6 startup scripts and rc.conf settings. The behaviour seems to be determined by multiple settings in rc.conf, first of all: ip6addrctl_policy=3D{ipv4_prefer|ipv6_prefer|AUTO} where the default value is AUTO. Values of ipv4_prefer and ipv6_prefer do what you expect them to. In case of AUTO, and if you don't have /etc/ip6addrctl.conf with explicit settings, /etc/rc.d/ip6addrctl checks the value of ipv6_activate_all_interfaces. If it is YES, IPv6 is preferred, if it is NO or unset, IPv4 is preferred. What are your IPv6-related settings in rc.conf? From owner-freebsd-stable@FreeBSD.ORG Sun Dec 11 22:53:56 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 72BA01065677 for ; Sun, 11 Dec 2011 22:53:56 +0000 (UTC) (envelope-from listen@heringa.de) Received: from eka.heringa.de (eka.heringa.de [IPv6:2001:1af8:4100:a04a:1::b00b]) by mx1.freebsd.org (Postfix) with ESMTP id 052118FC13 for ; Sun, 11 Dec 2011 22:53:56 +0000 (UTC) Received: from eka.heringa.de (localhost.heringa.de [127.0.0.1]) by eka.heringa.de (Postfix) with ESMTP id E10F517044 for ; Sun, 11 Dec 2011 23:53:54 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=heringa.de; h= user-agent:message-id:references:in-reply-to:subject:subject :from:from:date:date:content-transfer-encoding:content-type :content-type:mime-version:received:received; s=abc; t= 1323644033; x=1325458434; bh=dwITtBFpWXaZbQrnpOJPmYQlGMpxAkrbuSu RaEnbg2Q=; b=tSgB+jp3cwtaOh92WT9NDYxZR0C87i+VtqZu9ZeR433cEUscP9M 9yf36NWr3BEO3W7x/k2XGj7+ygRZxyP948IUpWSrX0hfTuaORcaSZdfuIYxIBO6r nZbnY9YwX9np497nJzerG08vBgtUUHdOSs3UNA2NNAdykJJ8PrmXqj+c= X-Virus-Scanned: amavisd-new at eka.heringa.de Received: from eka.heringa.de ([127.0.0.1]) by eka.heringa.de (eka.heringa.de [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id IxfuEBdKDOt7 for ; Sun, 11 Dec 2011 23:53:53 +0100 (CET) Received: from eka.heringa.de (eka.heringa.de [IPv6:2001:1af8:4100:a04a:1::b00b]) (Authenticated sender: dp) by eka.heringa.de (Postfix) with ESMTPSA for ; Sun, 11 Dec 2011 23:53:53 +0100 (CET) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Sun, 11 Dec 2011 23:53:53 +0100 From: Detlef Peeters To: In-Reply-To: <20111211223327.GJ83814@acme.spoerlein.net> References: <20111211223327.GJ83814@acme.spoerlein.net> Message-ID: X-Sender: listen@heringa.de User-Agent: Roundcube Webmail/0.6 Cc: Subject: Re: stable/9 preferring IPv4 over IPv6, what changed? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Dec 2011 22:53:56 -0000 Am 11.12.2011 23:33, schrieb Ulrich Spörlein: > long story short: telnet foo on stable/8 will first try connecting > via > IPv6, then via IPv4 (foo has A and AAAA records). On stable/9 it's > the > other way round. > > This trips up my setup, where a bunch of hosts (some behind NAT) can > all > talk to each other over their IPv6 addresses (some are tunneled), but > cannot do so via IPv4. > > Is this due to changes in bind or the resolver? You can change this via ip6addrctl_policy="ipv6_prefer" in /etc/rc.conf regards, Detlef From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 00:22:41 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE28B1065672 for ; Mon, 12 Dec 2011 00:22:41 +0000 (UTC) (envelope-from janm@transactionware.com) Received: from midgard.transactionware.com (mail2.transactionware.com [203.14.245.36]) by mx1.freebsd.org (Postfix) with SMTP id 3CA348FC14 for ; Mon, 12 Dec 2011 00:22:40 +0000 (UTC) Received: (qmail 19448 invoked by uid 907); 12 Dec 2011 00:22:38 -0000 Received: from jmmacpro.transactionware.com (HELO jmmacpro.transactionware.com) (192.168.1.33) by midgard.transactionware.com (qpsmtpd/0.82) with ESMTP; Mon, 12 Dec 2011 11:22:38 +1100 Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=iso-8859-1 From: Jan Mikkelsen In-Reply-To: <201112091103.51345.jhb@freebsd.org> Date: Mon, 12 Dec 2011 11:22:38 +1100 Content-Transfer-Encoding: quoted-printable Message-Id: <51DA2621-F65E-40AD-87EE-F43BE4AEBA5B@transactionware.com> References: <20E452A2-9180-4FC9-87CB-6F16D1D4D01A@transactionware.com> <201112091103.51345.jhb@freebsd.org> To: John Baldwin X-Mailer: Apple Mail (2.1251.1) Cc: freebsd-stable@freebsd.org Subject: Re: mfi(4) issues in 9.0-RC3 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 00:22:41 -0000 On 10/12/2011, at 3:03 AM, John Baldwin wrote: > On Friday, December 09, 2011 8:38:51 am Jan Mikkelsen wrote: >> Hi, >>=20 >> Can rev 227562 be merged into 9.0? >> [...] >=20 > It is probably too late to make 9.0 at this point as they've already = cut the=20 > final release candidate. OK; I'll maintain the patch myself until it gets merged. The MegaRAID 9261-8i requires hw.mfi.msi=3D1 to be set by hand for a = boot to successfully complete. It it worth adding a check for that = device? Thanks, Jan. From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 04:55:58 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5950D106566B for ; Mon, 12 Dec 2011 04:55:58 +0000 (UTC) (envelope-from brett@lariat.net) Received: from lariat.net (lariat.net [66.62.230.51]) by mx1.freebsd.org (Postfix) with ESMTP id EF6CE8FC08 for ; Mon, 12 Dec 2011 04:55:57 +0000 (UTC) Received: from WildRover.lariat.net (IDENT:ppp1000.lariat.net@lariat.net [66.119.58.2] (may be forged)) by lariat.net (8.9.3/8.9.3) with ESMTP id VAA03028; Sun, 11 Dec 2011 21:55:51 -0700 (MST) Message-Id: <201112120455.VAA03028@lariat.net> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Sun, 11 Dec 2011 21:54:23 -0700 To: Ben Kaduk From: Brett Glass In-Reply-To: References: <201112090913.CAA03333@lariat.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Cc: stable@freebsd.org Subject: Re: Two problems still present in RC3 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 04:55:58 -0000 At 11:36 AM 12/11/2011, Ben Kaduk wrote: >Did you take the change to /etc/ttys going from cons25 to xterm 'type'? I didn't have to change it; it was that way when the OS was installed. Problem seems to be that the behavior (specifically, reverse video on the 25th line) doesn't quite match the xterm termcap entry. --Brett From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 10:32:33 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 332F9106564A for ; Mon, 12 Dec 2011 10:32:33 +0000 (UTC) (envelope-from freebsd_gb@yahoo.com.hk) Received: from nm27-vm2.bullet.mail.sg3.yahoo.com (nm27-vm2.bullet.mail.sg3.yahoo.com [106.10.151.129]) by mx1.freebsd.org (Postfix) with SMTP id 780158FC13 for ; Mon, 12 Dec 2011 10:32:32 +0000 (UTC) Received: from [106.10.166.113] by nm27.bullet.mail.sg3.yahoo.com with NNFMP; 12 Dec 2011 10:32:31 -0000 Received: from [106.10.151.15] by tm2.bullet.mail.sg3.yahoo.com with NNFMP; 12 Dec 2011 10:32:31 -0000 Received: from [127.0.0.1] by omp1020.mail.sg3.yahoo.com with NNFMP; 12 Dec 2011 10:32:31 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 151388.17002.bm@omp1020.mail.sg3.yahoo.com Received: (qmail 42476 invoked by uid 60001); 12 Dec 2011 10:32:31 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.hk; s=s1024; t=1323685951; bh=JW4r70kA0bYyqER+0chFHOkwEffqaTFJN1lHuDcB730=; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=jj0n8IB2WiYM4+AoMZD3a4mVbcP9TQ7g6MGt9JMghBAoaCCD19mNUR4rItygrV+3XuMEl03yKDSAJi6viS6ZxjTg6Rgi3utaG7bLm0U9dF1vWq+k2jm4udHquJsP7wnvP2raESiy2n/i7Fwob5ynhWnKudkeWZHNi41wcDqBvO4= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com.hk; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=UiD43x1OD9P70jV04oozBI06K/LsUk+3Nr5IxIlKQiF08L2vkArgaWc0sa0Y/otcDDU0nQINZTSK3mbNVyxAWk7h3/N/pypFpHp8vlI1JqlUDeeKcTwo9xHlsDFXg/3FCTCLKFj9dl+kWrX5bk/pn9xrYTDzm3t/xjw6ecU2mjY=; X-YMail-OSG: 6zEDnrsVM1mGYr8kfF6RZJcKpcmgVWzMNe9VBDZXUIEeoSc u.qGVUZDzN4dCqyDfYMkRh9RrxSEVud9TsI2mlUDTnpBdHUmZDO0xMBfLok2 bo234nkmbqfX4e2WamA7z.tBQvVEOWARJ2g2MVDjTzHeEtinIpoIwCuwWc9G YrHjmDflqH2yxFp5BvInD.J.uxE5DOX3wfIzHbCgOLhXz3UuKFQp.H49FWON UDQBNiFkBguLr.f21xTTvery6wyA.Obsuod5wZNxyvX9Eb1e.9plkBkrD2CJ RHoaV.V94x8HfyILI8rDFBzWlnjyVQ1P6sX4094qaAOu9fPTD6VePksOTEhu 12As2gPyQABRBqyuWLa9hy6Q- Received: from [203.218.76.208] by web190802.mail.sg3.yahoo.com via HTTP; Mon, 12 Dec 2011 18:32:30 SGT X-Mailer: YahooMailWebService/0.8.115.331698 Message-ID: <1323685950.42466.YahooMailNeo@web190802.mail.sg3.yahoo.com> Date: Mon, 12 Dec 2011 18:32:30 +0800 (SGT) From: Man Chan To: "freebsd-stable@freebsd.org" MIME-Version: 1.0 X-Mailman-Approved-At: Mon, 12 Dec 2011 11:31:12 +0000 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: how to get the source tree X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Man Chan List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 10:32:33 -0000 Hi,=0A=0AI had tried to obtain the source tree of my system (8.2 release) t= his afternoon and feel miss about the command cvs.=0AI use the following.= =0Acvs get -rRELENG_8_2 ports=A0 after several minutes only the CVS is in t= he /usr/ports and nothing else. I search the handbook and the internet for = the answer but no useful answer.=A0 Can anyone tell me what is the correct = command (cvs)=A0 to get the source and keep it up to date.=A0 Thanks.=0A=0A= Man=0A From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 13:04:59 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8B5D0106566B for ; Mon, 12 Dec 2011 13:04:59 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (smtp6.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) by mx1.freebsd.org (Postfix) with ESMTP id 13C328FC08 for ; Mon, 12 Dec 2011 13:04:58 +0000 (UTC) Received: from seedling.black-earth.co.uk (seedling.black-earth.co.uk [IPv6:2001:8b0:151:1:fa1e:dfff:feda:c0bb]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.14.5/8.14.5) with ESMTP id pBCD4pnG083955 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Mon, 12 Dec 2011 13:04:52 GMT (envelope-from m.seaman@infracaninophile.co.uk) X-DKIM: OpenDKIM Filter v2.4.1 smtp.infracaninophile.co.uk pBCD4pnG083955 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=infracaninophile.co.uk; s=201001-infracaninophile; t=1323695092; bh=cccXKE3bZ6+AXXW90cn93+P3jER1TWUukDFsQYPoJFs=; h=Message-ID:Date:From:MIME-Version:To:Subject:References: In-Reply-To:Content-Type:Cc; b=BhS0mXdffNe0E2X8DIVIAsYTge0sD/rUJ8LTivxxie+/UbGQT32sq/2mWVlC/jbjm YXsO4kUaAkHnoVhvFspcb51EwdmicHetyocMJmeIpAxDr8U2Y5RZIrPks2Cn0OUKO0 aksVc1N/wqThvDyF0qDPITXtT9OwMpojH1FQDt0M= Message-ID: <4EE5FBEA.3040203@infracaninophile.co.uk> Date: Mon, 12 Dec 2011 13:04:42 +0000 From: Matthew Seaman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <1323685950.42466.YahooMailNeo@web190802.mail.sg3.yahoo.com> In-Reply-To: <1323685950.42466.YahooMailNeo@web190802.mail.sg3.yahoo.com> X-Enigmail-Version: 1.3.3 OpenPGP: id=60AE908C Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig4D7F50DF9EED6AA5016ED106" X-Virus-Scanned: clamav-milter 0.97.3 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,SPF_FAIL autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lucid-nonsense.infracaninophile.co.uk Subject: Re: how to get the source tree X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 13:04:59 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig4D7F50DF9EED6AA5016ED106 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 12/12/2011 10:32, Man Chan wrote: > I had tried to obtain the source tree of my system (8.2 release) this a= fternoon and feel miss about the command cvs. > I use the following. > cvs get -rRELENG_8_2 ports after several minutes only the CVS is in th= e /usr/ports and nothing else. I search the handbook and the internet for= the answer but no useful answer. Can anyone tell me what is the correct= command (cvs) to get the source and keep it up to date. Thanks. RELENG_8_2 is not a tag used on the ports. It only applies to the system sources. Also, there isn't a version of the ports specifically for 8.2-RELEASE. All there is, is a tag that marked the state of the ports tree at the time the 8.2-RELEASE media were created. That's mostly for bookkeeping purposes and not a coded hint that these are the ports you are looking fo= r. Seriously, just check out the HEAD revision of the ports. It has all the updates and fixes for problems that have been discovered since 8.2 came out. Usually for values of 'check out' that translate to 'use csup or portsnap to download' as that's a lot more efficient (not to say faster) than using plain cvs(1). Cheers, Matthew --=20 Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate JID: matthew@infracaninophile.co.uk Kent, CT11 9PW --------------enig4D7F50DF9EED6AA5016ED106 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk7l+/IACgkQ8Mjk52CukIwtdACfQoqUgtBDzUjEvubtg/9Juknl 0f0AnAzbr/ryXdLOEUqMIFZiU2Rze5oP =n+4M -----END PGP SIGNATURE----- --------------enig4D7F50DF9EED6AA5016ED106-- From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 14:09:32 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 344E0106564A; Mon, 12 Dec 2011 14:09:32 +0000 (UTC) (envelope-from ohartman@mail.zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id E0AC78FC08; Mon, 12 Dec 2011 14:09:31 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1Ra6Ed-00011K-IT>; Mon, 12 Dec 2011 14:48:03 +0100 Received: from telesto.geoinf.fu-berlin.de ([130.133.86.198]) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1Ra6Ed-0004V9-FJ>; Mon, 12 Dec 2011 14:48:03 +0100 Message-ID: <4EE6060D.5060201@mail.zedat.fu-berlin.de> Date: Mon, 12 Dec 2011 14:47:57 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Current FreeBSD , freebsd-stable@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> In-Reply-To: <4EE22421.9060707@gmail.com> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig3C34C0DC945A3DBF809616AD" X-Originating-IP: 130.133.86.198 Cc: freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 14:09:32 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig3C34C0DC945A3DBF809616AD Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable > Not fully right, boinc defaults to run on idprio 31 so this isn't an > issue. And yes, there are cases where SCHED_ULE shows much better > performance then SCHED_4BSD. [...] Do we have any proof at hand for such cases where SCHED_ULE performs much better than SCHED_4BSD? Whenever the subject comes up, it is mentioned, that SCHED_ULE has better performance on boxes with a ncpu > 2. But in the end I see here contradictionary statements. People complain about poor performance (especially in scientific environments), and other give contra not being the case. Within our department, we developed a highly scalable code for planetary science purposes on imagery. It utilizes present GPUs via OpenCL if present. Otherwise it grabs as many cores as it can. By the end of this year I'll get a new desktop box based on Intels new Sandy Bridge-E architecture with plenty of memory. If the colleague who developed the code is willing performing some benchmarks on the same hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most recent Suse. For FreeBSD I intent also to look for performance with both different schedulers available. O. --------------enig3C34C0DC945A3DBF809616AD Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iF4EAREIAAYFAk7mBhMACgkQU6Ni+wtCKv9mzQD9EQm3oYTJl1UY826v0xiOzMeF nrbUBR3bAjsxSp3A2hYA/0TL0ltenxGrjE6h8DEiQg6ozCymbh7vkFCTBnHkZlaP =8zGJ -----END PGP SIGNATURE----- --------------enig3C34C0DC945A3DBF809616AD-- From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 15:13:03 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 043ED1065670; Mon, 12 Dec 2011 15:13:03 +0000 (UTC) (envelope-from vince@unsane.co.uk) Received: from unsane.co.uk (unsane-pt.tunnel.tserv5.lon1.ipv6.he.net [IPv6:2001:470:1f08:110::2]) by mx1.freebsd.org (Postfix) with ESMTP id 616528FC1B; Mon, 12 Dec 2011 15:13:02 +0000 (UTC) Received: from vhoffman-macbooklocal.local ([10.10.10.20]) (authenticated bits=0) by unsane.co.uk (8.14.4/8.14.4) with ESMTP id pBCFD0EV014133 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 12 Dec 2011 15:13:00 GMT (envelope-from vince@unsane.co.uk) Message-ID: <4EE619FC.4000601@unsane.co.uk> Date: Mon, 12 Dec 2011 15:13:00 +0000 From: Vincent Hoffman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: "O. Hartmann" References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> In-Reply-To: <4EE6060D.5060201@mail.zedat.fu-berlin.de> X-Enigmail-Version: 1.3.3 X-Enigmail-Draft-Status: 513 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 15:13:03 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/12/2011 13:47, O. Hartmann wrote: > >> Not fully right, boinc defaults to run on idprio 31 so this isn't an >> issue. And yes, there are cases where SCHED_ULE shows much better >> performance then SCHED_4BSD. [...] > > Do we have any proof at hand for such cases where SCHED_ULE performs > much better than SCHED_4BSD? Whenever the subject comes up, it is > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > 2. But in the end I see here contradictionary statements. People > complain about poor performance (especially in scientific environments), > and other give contra not being the case. It all a little old now but some if the stuff in http://people.freebsd.org/~kris/scaling/ covers improvements that were seen. http://jeffr-tech.livejournal.com/5705.html shows a little too, reading though Jeffs blog is worth it as it has some interesting stuff on SHED_ULE. I thought there were some more benchmarks floating round but cant find any with a quick google. Vince > > Within our department, we developed a highly scalable code for planetary > science purposes on imagery. It utilizes present GPUs via OpenCL if > present. Otherwise it grabs as many cores as it can. > By the end of this year I'll get a new desktop box based on Intels new > Sandy Bridge-E architecture with plenty of memory. If the colleague who > developed the code is willing performing some benchmarks on the same > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > recent Suse. For FreeBSD I intent also to look for performance with both > different schedulers available. > > O. > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJO5hn7AAoJEF4mgOY1fXowOLAP/2EjhAFPb88NgKM0ieBb4X7R NSw/9HTiwcshkfEdvYjAzYZ0cUWetEuRfnPVnh+abwfJEmMzZkwA0KIz8UYGHHik 22Z2SWSVDiwZAluz0ca7Xc931ojbzrK/zVMbivqW3cvnz8P4oEnASiENnsoa89Jy Oskjd4QpAyIpB/AsYgc9FLT3kPX13fXC5bzw/zAPDsaupOYssRRlZu8nnqsEc1i1 IanLIPKLnIbpZTx75ehWxxRW8IjiQRvIe+7eBaDMhXO/Kvftotf0JzknrBnJezDQ ZdhiOTq7F1Pm3dxra+DNKD+Dw+xUCYPFq/kuyqrZNz44H3qwT60vDhvw0yDz6422 nNP11z2+G4M85sahBak5AmSHuyek7HWb6uIHHnfvwNKSX4ZsdS8MVBViNJjmCYtL PwuHDU3WdCes/vvKRNDopSp/s6RSLK9w3RT7jlMkaTu2Mmtw0BwGziDJ2pGaCQ14 68R5eO/SfNxoVp0g4lIzObyQR+//0OmALzElVK3VmHM9NoL3qZGCwBRLqjN5re82 dX6nsBr/DFJOpaFfdFLwPNyCNdNpg/WVegRkq2BEL/BaMISNiKzoVbM0Psh9gnb3 LW1j3LP2fOHhuN1bW3S31JmbNzvAnlRNynoNMldrwj5PWJY2HPk+mMFRjmRwdDTJ 9mhscz8++WRPvDZQXefl =XqaR -----END PGP SIGNATURE----- From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 15:52:00 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 102E61065670; Mon, 12 Dec 2011 15:52:00 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) by mx1.freebsd.org (Postfix) with ESMTP id C3CAA8FC12; Mon, 12 Dec 2011 15:51:59 +0000 (UTC) Received: from troutmask.apl.washington.edu (localhost.apl.washington.edu [127.0.0.1]) by troutmask.apl.washington.edu (8.14.5/8.14.5) with ESMTP id pBCFpxf6073711; Mon, 12 Dec 2011 07:51:59 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.14.5/8.14.5/Submit) id pBCFpxO9073710; Mon, 12 Dec 2011 07:51:59 -0800 (PST) (envelope-from sgk) Date: Mon, 12 Dec 2011 07:51:59 -0800 From: Steve Kargl To: "O. Hartmann" Message-ID: <20111212155159.GB73597@troutmask.apl.washington.edu> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE6060D.5060201@mail.zedat.fu-berlin.de> User-Agent: Mutt/1.4.2.3i Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 15:52:00 -0000 On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: > > > Not fully right, boinc defaults to run on idprio 31 so this isn't an > > issue. And yes, there are cases where SCHED_ULE shows much better > > performance then SCHED_4BSD. [...] > > Do we have any proof at hand for such cases where SCHED_ULE performs > much better than SCHED_4BSD? Whenever the subject comes up, it is > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > 2. But in the end I see here contradictionary statements. People > complain about poor performance (especially in scientific environments), > and other give contra not being the case. > > Within our department, we developed a highly scalable code for planetary > science purposes on imagery. It utilizes present GPUs via OpenCL if > present. Otherwise it grabs as many cores as it can. > By the end of this year I'll get a new desktop box based on Intels new > Sandy Bridge-E architecture with plenty of memory. If the colleague who > developed the code is willing performing some benchmarks on the same > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > recent Suse. For FreeBSD I intent also to look for performance with both > different schedulers available. > This comes up every 9 months or so, and must be approaching FAQ status. In a HPC environment, I recommend 4BSD. Depending on the workload, ULE can cause a severe increase in turn around time when doing already long computations. If you have an MPI application, simply launching greater than ncpu+1 jobs can show the problem. PS: search the list archives for "kargl and ULE". -- Steve From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 16:01:07 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07EF8106564A; Mon, 12 Dec 2011 16:01:07 +0000 (UTC) (envelope-from gljennjohn@googlemail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 5EC3B8FC0C; Mon, 12 Dec 2011 16:01:05 +0000 (UTC) Received: by eekc50 with SMTP id c50so3330964eek.13 for ; Mon, 12 Dec 2011 08:01:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :x-mailer:mime-version:content-type:content-transfer-encoding; bh=SHPI/Eh4GHYyLbnl/U4ZuP6rptb+8Q9ldKyvRbK7Oi4=; b=mX7wboXam0RjDri+pOgLL53QpJqlOn0UsUBsVJViLlkWG+UPG7h4kq9Co+mHQgnVnO D3n9kN135RvCgJ2ZMMFBK9WR1GFN25fzHY9OPtojQrdAzMlgrBL7iVT8YiO1sqkLWzkC Vjy170H5PWDvbzTMx/skrjZ3zMDrTnpKVzh7U= Received: by 10.14.14.77 with SMTP id c53mr268271eec.85.1323703945006; Mon, 12 Dec 2011 07:32:25 -0800 (PST) Received: from ernst.jennejohn.org (p578E15DC.dip.t-dialin.net. [87.142.21.220]) by mx.google.com with ESMTPS id a60sm76476635eeb.4.2011.12.12.07.32.23 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 12 Dec 2011 07:32:24 -0800 (PST) Date: Mon, 12 Dec 2011 16:32:21 +0100 From: Gary Jennejohn To: Vincent Hoffman Message-ID: <20111212163221.33d0b8a2@ernst.jennejohn.org> In-Reply-To: <4EE619FC.4000601@unsane.co.uk> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE619FC.4000601@unsane.co.uk> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gljennjohn@googlemail.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:01:07 -0000 On Mon, 12 Dec 2011 15:13:00 +0000 Vincent Hoffman wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 12/12/2011 13:47, O. Hartmann wrote: > > > >> Not fully right, boinc defaults to run on idprio 31 so this isn't an > >> issue. And yes, there are cases where SCHED_ULE shows much better > >> performance then SCHED_4BSD. [...] > > > > Do we have any proof at hand for such cases where SCHED_ULE performs > > much better than SCHED_4BSD? Whenever the subject comes up, it is > > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > > 2. But in the end I see here contradictionary statements. People > > complain about poor performance (especially in scientific environments), > > and other give contra not being the case. > It all a little old now but some if the stuff in > http://people.freebsd.org/~kris/scaling/ > covers improvements that were seen. > > http://jeffr-tech.livejournal.com/5705.html > shows a little too, reading though Jeffs blog is worth it as it has some > interesting stuff on SHED_ULE. > > I thought there were some more benchmarks floating round but cant find > any with a quick google. > > > Vince > > > > > Within our department, we developed a highly scalable code for planetary > > science purposes on imagery. It utilizes present GPUs via OpenCL if > > present. Otherwise it grabs as many cores as it can. > > By the end of this year I'll get a new desktop box based on Intels new > > Sandy Bridge-E architecture with plenty of memory. If the colleague who > > developed the code is willing performing some benchmarks on the same > > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > > recent Suse. For FreeBSD I intent also to look for performance with both > > different schedulers available. > > These observations are not scientific, but I have a CPU from AMD with 6 cores (AMD Phenom(tm) II X6 1090T Processor). My simple test was ``make buildkernel'' while watching the core usage with gkrellm. With SCHED_4BSD all 6 cores are loaded to 97% during the build phase. I've never seen any value above 97% with gkrellm. With SCHED_ULE I never saw all 6 cores loaded this heavily. Usually 2 or more cores were at or below 90%. Not really that significant, but still a noticeable difference in apparent scheduling behavior. Whether the observed difference is due to some change in data from the kernel to gkrellm is beyond me. -- Gary Jennejohn From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 16:11:27 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C8A0F106567B; Mon, 12 Dec 2011 16:11:27 +0000 (UTC) (envelope-from lars.engels@0x20.net) Received: from mail.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) by mx1.freebsd.org (Postfix) with ESMTP id 1E2838FC21; Mon, 12 Dec 2011 16:11:27 +0000 (UTC) Received: from mail.0x20.net (mail.0x20.net [217.69.76.211]) by mail.0x20.net (Postfix) with ESMTP id 38ED66A661D; Mon, 12 Dec 2011 17:11:26 +0100 (CET) X-Virus-Scanned: amavisd-new at mail.0x20.net Received: from mail.0x20.net ([217.69.76.211]) by mail.0x20.net (mail.0x20.net [217.69.76.211]) (amavisd-new, port 10024) with ESMTP id NOQH_WXjvpPe; Mon, 12 Dec 2011 17:11:26 +0100 (CET) Received: from [10.26.32.239] (ip-109-84-0-111.web.vodafone.de [109.84.0.111]) (Authenticated sender: lala) by mail.0x20.net (Postfix) with ESMTPA id DBC096A6619; Mon, 12 Dec 2011 17:11:22 +0100 (CET) References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE619FC.4000601@unsane.co.uk> <20111212163221.33d0b8a2@ernst.jennejohn.org> User-Agent: K-9 Mail for Android In-Reply-To: <20111212163221.33d0b8a2@ernst.jennejohn.org> MIME-Version: 1.0 From: Lars Engels Date: Mon, 12 Dec 2011 17:10:46 +0100 To: gljennjohn@googlemail.com,Vincent Hoffman Message-ID: <1a8a6d6f-6756-4cda-b4d6-b39d335678c1@email.android.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:11:27 -0000 Did you use -jX to build the world? _____________________________________________ Von: Gary Jennejohn Versendet am: Mon Dec 12 16:32:21 MEZ 2011 An: Vincent Hoffman CC: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Betreff: Re: SCHED_ULE should not be the default On Mon, 12 Dec 2011 15:13:00 +0000 Vincent Hoffman wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 12/12/2011 13:47, O. Hartmann wrote: > > > >> Not fully right, boinc defaults to run on idprio 31 so this isn't an > >> issue. And yes, there are cases where SCHED_ULE shows much better > >> performance then SCHED_4BSD. [...] > > > > Do we have any proof at hand for such cases where SCHED_ULE performs > > much better than SCHED_4BSD? Whenever the subject comes up, it is > > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > > 2. But in the end I see here contradictionary statements. People > > complain about poor performance (especially in scientific environments), > > and other give contra not being the case. > It all a little old now but some if the stuff in > http://people.freebsd.org/~kris/scaling/ > covers improvements that were seen. > > http://jeffr-tech.livejournal.com/5705.html > shows a little too, reading though Jeffs blog is worth it as it has some > interesting stuff on SHED_ULE. > > I thought there were some more benchmarks floating round but cant find > any with a quick google. > > > Vince > > > > > Within our department, we developed a highly scalable code for planetary > > science purposes on imagery. It utilizes present GPUs via OpenCL if > > present. Otherwise it grabs as many cores as it can. > > By the end of this year I'll get a new desktop box based on Intels new > > Sandy Bridge-E architecture with plenty of memory. If the colleague who > > developed the code is willing performing some benchmarks on the same > > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > > recent Suse. For FreeBSD I intent also to look for performance with both > > different schedulers available. > > These observations are not scientific, but I have a CPU from AMD with 6 cores (AMD Phenom(tm) II X6 1090T Processor). My simple test was ``make buildkernel'' while watching the core usage with gkrellm. With SCHED_4BSD all 6 cores are loaded to 97% during the build phase. I've never seen any value above 97% with gkrellm. With SCHED_ULE I never saw all 6 cores loaded this heavily. Usually 2 or more cores were at or below 90%. Not really that significant, but still a noticeable difference in apparent scheduling behavior. Whether the observed difference is due to some change in data from the kernel to gkrellm is beyond me. -- Gary Jennejohn _____________________________________________ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 16:14:03 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 75D9C106564A; Mon, 12 Dec 2011 16:14:03 +0000 (UTC) (envelope-from lars.engels@0x20.net) Received: from mail.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) by mx1.freebsd.org (Postfix) with ESMTP id EA3D88FC13; Mon, 12 Dec 2011 16:14:02 +0000 (UTC) Received: from mail.0x20.net (mail.0x20.net [217.69.76.211]) by mail.0x20.net (Postfix) with ESMTP id 439316A61CD; Mon, 12 Dec 2011 17:14:02 +0100 (CET) X-Virus-Scanned: amavisd-new at mail.0x20.net Received: from mail.0x20.net ([217.69.76.211]) by mail.0x20.net (mail.0x20.net [217.69.76.211]) (amavisd-new, port 10024) with ESMTP id lhy24cgq8Rc2; Mon, 12 Dec 2011 17:14:02 +0100 (CET) Received: from [10.26.32.239] (ip-109-84-0-111.web.vodafone.de [109.84.0.111]) (Authenticated sender: lala) by mail.0x20.net (Postfix) with ESMTPA id 77B4F6A6016; Mon, 12 Dec 2011 17:13:59 +0100 (CET) References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> User-Agent: K-9 Mail for Android In-Reply-To: <20111212155159.GB73597@troutmask.apl.washington.edu> MIME-Version: 1.0 From: Lars Engels Date: Mon, 12 Dec 2011 17:13:08 +0100 To: Steve Kargl , "O. Hartmann" Message-ID: <158db475-0d35-4ff2-a101-cf056bdfcab8@email.android.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:14:03 -0000 Would it be possible to implement a mechanism that lets one change the scheduler on the fly? Afaik Solaris can do that. _____________________________________________ Von: Steve Kargl Versendet am: Mon Dec 12 16:51:59 MEZ 2011 An: "O. Hartmann" CC: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Betreff: Re: SCHED_ULE should not be the default On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: > > > Not fully right, boinc defaults to run on idprio 31 so this isn't an > > issue. And yes, there are cases where SCHED_ULE shows much better > > performance then SCHED_4BSD. [...] > > Do we have any proof at hand for such cases where SCHED_ULE performs > much better than SCHED_4BSD? Whenever the subject comes up, it is > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > 2. But in the end I see here contradictionary statements. People > complain about poor performance (especially in scientific environments), > and other give contra not being the case. > > Within our department, we developed a highly scalable code for planetary > science purposes on imagery. It utilizes present GPUs via OpenCL if > present. Otherwise it grabs as many cores as it can. > By the end of this year I'll get a new desktop box based on Intels new > Sandy Bridge-E architecture with plenty of memory. If the colleague who > developed the code is willing performing some benchmarks on the same > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > recent Suse. For FreeBSD I intent also to look for performance with both > different schedulers available. > This comes up every 9 months or so, and must be approaching FAQ status. In a HPC environment, I recommend 4BSD. Depending on the workload, ULE can cause a severe increase in turn around time when doing already long computations. If you have an MPI application, simply launching greater than ncpu+1 jobs can show the problem. PS: search the list archives for "kargl and ULE". -- Steve _____________________________________________ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 16:18:39 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 54E8A1065673; Mon, 12 Dec 2011 16:18:39 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [IPv6:2a01:348:0:15:5d59:5c40:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id AA9AB8FC18; Mon, 12 Dec 2011 16:18:38 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id C9CE3E6208; Mon, 12 Dec 2011 16:18:36 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=mail; bh=Q9z1bk1qmKZA XQ8nRStBeMc3/Jk=; b=SxXMh7kUdQYpi15Kt2J0Rs6fYC1olhC17WIs0zDjIT5D A1Iq96vlmLVpcO1A6qoh99oVO5W12+/fCMZtP1D+efVwW3aBMFT4sZTaoXUOlFDh rGWM1gOuR1TCFVUlHquestEdOYF2nPQGwZnksQNhdP74MsDVCh/kWz9Ns5moGS4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=mail; b=RcWMeC 1y8njc525uJc8U1Tmzb2xH95iBdBPjvx9fu+qY8W+IegM0A5JKq62k7OQUsNg5df YhaIreE02yvjKlB4ycT5fpwQ1N+QZPLK56mfkiGjFav7JnbvhQh6Fy5NrFdv/ULi t+KDjFXsZ1d3YNfJatjra7CllK0LWd/TQoKCk= Received: from [192.168.1.120] (188-220-36-32.zone11.bethere.co.uk [188.220.36.32]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id 50421E6200; Mon, 12 Dec 2011 16:18:36 +0000 (GMT) Message-ID: <4EE6295B.3020308@cran.org.uk> Date: Mon, 12 Dec 2011 16:18:35 +0000 From: Bruce Cran User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Steve Kargl References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> In-Reply-To: <20111212155159.GB73597@troutmask.apl.washington.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:18:39 -0000 On 12/12/2011 15:51, Steve Kargl wrote: > This comes up every 9 months or so, and must be approaching FAQ > status. In a HPC environment, I recommend 4BSD. Depending on the > workload, ULE can cause a severe increase in turn around time when > doing already long computations. If you have an MPI application, > simply launching greater than ncpu+1 jobs can show the problem. PS: > search the list archives for "kargl and ULE". This isn't something that can be fixed by tuning ULE? For example for desktop applications kern.sched.preempt_thresh should be set to 224 from its default. I'm wondering if the installer should ask people what the typical use will be, and tune the scheduler appropriately. -- Bruce Cran From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 16:30:39 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E510106564A; Mon, 12 Dec 2011 16:30:38 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 2CB8A8FC12; Mon, 12 Dec 2011 16:30:37 +0000 (UTC) Received: by dakp5 with SMTP id p5so7144112dak.13 for ; Mon, 12 Dec 2011 08:30:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=+dryFrlmia5wNaD6PzjKbJ//sZuyakBBXiNiNL3Oq4Q=; b=lgfORXOm2V59Cfjmi+1ior8IoNtO4RvD68YAQJubbjcUkRg9uRh2blmihRprEjzK3F XXz0muY87H88yNPU+7SBGMsBVodsCdguO/hc8nhrugIpy77NclKf9Djthf3lsHR+fZeG F1lAK9kLQMEnkqQMVVhnGCN4P2Ag0+rg+6BRM= MIME-Version: 1.0 Received: by 10.68.211.5 with SMTP id my5mr36432945pbc.17.1323705877044; Mon, 12 Dec 2011 08:04:37 -0800 (PST) Sender: mdf356@gmail.com Received: by 10.68.197.198 with HTTP; Mon, 12 Dec 2011 08:04:37 -0800 (PST) In-Reply-To: <20111212163221.33d0b8a2@ernst.jennejohn.org> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE619FC.4000601@unsane.co.uk> <20111212163221.33d0b8a2@ernst.jennejohn.org> Date: Mon, 12 Dec 2011 08:04:37 -0800 X-Google-Sender-Auth: Z-9Mf-awLdKOtIu2Ho6RiSTBpfc Message-ID: From: mdf@FreeBSD.org To: gljennjohn@googlemail.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Vincent Hoffman Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:30:39 -0000 On Mon, Dec 12, 2011 at 7:32 AM, Gary Jennejohn wrote: > On Mon, 12 Dec 2011 15:13:00 +0000 > Vincent Hoffman wrote: > >> >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> On 12/12/2011 13:47, O. Hartmann wrote: >> > >> >> Not fully right, boinc defaults to run on idprio 31 so this isn't an >> >> issue. And yes, there are cases where SCHED_ULE shows much better >> >> performance then SCHED_4BSD. [...] >> > >> > Do we have any proof at hand for such cases where SCHED_ULE performs >> > much better than SCHED_4BSD? Whenever the subject comes up, it is >> > mentioned, that SCHED_ULE has better performance on boxes with a ncpu = > >> > 2. But in the end I see here contradictionary statements. People >> > complain about poor performance (especially in scientific environments= ), >> > and other give contra not being the case. >> It all a little old now but some if the stuff in >> http://people.freebsd.org/~kris/scaling/ >> covers improvements that were seen. >> >> http://jeffr-tech.livejournal.com/5705.html >> shows a little too, reading though Jeffs blog is worth it as it has some >> interesting stuff on SHED_ULE. >> >> I thought there were some more benchmarks floating round but cant find >> any with a quick google. >> >> >> Vince >> >> > >> > Within our department, we developed a highly scalable code for planeta= ry >> > science purposes on imagery. It utilizes present GPUs via OpenCL if >> > present. Otherwise it grabs as many cores as it can. >> > By the end of this year I'll get a new desktop box based on Intels new >> > Sandy Bridge-E architecture with plenty of memory. If the colleague wh= o >> > developed the code is willing performing some benchmarks on the same >> > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most >> > recent Suse. For FreeBSD I intent also to look for performance with bo= th >> > different schedulers available. >> > > > These observations are not scientific, but I have a CPU from AMD with > 6 cores (AMD Phenom(tm) II X6 1090T Processor). > > My simple test was ``make buildkernel'' while watching the core usage wit= h > gkrellm. > > With SCHED_4BSD all 6 cores are loaded to 97% during the build phase. > I've never seen any value above 97% with gkrellm. > > With SCHED_ULE I never saw all 6 cores loaded this heavily. =A0Usually > 2 or more cores were at or below 90%. =A0Not really that significant, but > still a noticeable difference in apparent scheduling behavior. =A0Whether > the observed difference is due to some change in data from the kernel to > gkrellm is beyond me. SCHED_ULE is much sloppier about calculating which thread used a timeslice -- unless the timeslice went 100% to a thread, the fraction it used may get attributed elsewhere. So top's reporting of thread usage is not a useful metric. Total buildworld time is, potentially. Thanks, matthew From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 16:31:01 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11AE010656E7; Mon, 12 Dec 2011 16:31:01 +0000 (UTC) (envelope-from pieter@degoeje.nl) Received: from mx.utwente.nl (mx2.utsp.utwente.nl [130.89.2.13]) by mx1.freebsd.org (Postfix) with ESMTP id 703058FC1E; Mon, 12 Dec 2011 16:31:00 +0000 (UTC) Received: from nox-laptop.student.utwente.nl (nox-laptop.student.utwente.nl [130.89.160.140]) by mx.utwente.nl (8.12.10/SuSE Linux 0.7) with ESMTP id pBCFr4ms020823; Mon, 12 Dec 2011 16:53:04 +0100 From: Pieter de Goeje To: freebsd-current@freebsd.org Date: Mon, 12 Dec 2011 16:53:04 +0100 User-Agent: KMail/1.9.10 References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> In-Reply-To: <4EE6060D.5060201@mail.zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <201112121653.04522.pieter@degoeje.nl> X-UTwente-MailScanner-Information: Scanned by MailScanner. Contact icts.servicedesk@utwente.nl for more information. X-UTwente-MailScanner: Found to be clean X-UTwente-MailScanner-From: pieter@degoeje.nl X-Spam-Status: No Cc: "O. Hartmann" , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:31:01 -0000 On Monday 12 December 2011 14:47:57 O. Hartmann wrote: > > Not fully right, boinc defaults to run on idprio 31 so this isn't an > > issue. And yes, there are cases where SCHED_ULE shows much better > > performance then SCHED_4BSD. [...] > > Do we have any proof at hand for such cases where SCHED_ULE performs > much better than SCHED_4BSD? Whenever the subject comes up, it is > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > 2. But in the end I see here contradictionary statements. People > complain about poor performance (especially in scientific environments), > and other give contra not being the case. > > Within our department, we developed a highly scalable code for planetary > science purposes on imagery. It utilizes present GPUs via OpenCL if > present. Otherwise it grabs as many cores as it can. > By the end of this year I'll get a new desktop box based on Intels new > Sandy Bridge-E architecture with plenty of memory. If the colleague who > developed the code is willing performing some benchmarks on the same > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > recent Suse. For FreeBSD I intent also to look for performance with both > different schedulers available. > > O. In my spare time I do some stuff which can be considered "HPC". If I recall correctly the most loud supporters of the notion that SCHED_BSD is faster than SCHED_ULE are using more threads than there are cores, causing CPU core contention and more importantly unevenly distributed runtimes among threads, resulting in suboptimal execution times for their programs. Since I've never actually seen that code in question it's hard to say whether or not this "unfair" distribution actually results in lower throughput or that it simply violates an assumption in the code that each thread takes about as long to finish its task. Although I haven't actually benchmarked the two schedulers directly, I have no reason to suspect SCHED_ULE of suboptimal performance because: 1) A program model where there are N threads on N cores which take work items from a shared queue until it is empty has almost perfect scaling on SCHED_ULE (I get 398% CPU usage on a quadcore) 2) The same program on Linux (dual boot) compiled with exactly the same compiler and flags runs slightly slower. I think this has to do with VM differences. What I'm trying to say is that until someone actually shows some code which has demonstrably lower performance on SCHED_ULE and this is not caused by IMHO improper timing dependencies between threads I'd say that there is no cause for concern here. I actually expect performance differences between the two schedulers to show in problems which cause a lot more contention on the CPU cores and use lots of locks internally so threads are frequently waiting on each other, for instance the MySQL benchmarks done a couple of years ago by Kris Kennaway. Aside from algorithmic limitations (SCHED_BSD doesn't really scale all that well), there will always exist some problems in which SCHED_BSD is faster because it by chance has a better execution order for these problems... The good thing is people have a choice :-). I'm looking forward to the results of your benchmark. -- Pieter de Goeje From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 16:43:52 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F158106564A; Mon, 12 Dec 2011 16:43:52 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm1.ukr.net (fsm1.ukr.net [195.214.192.120]) by mx1.freebsd.org (Postfix) with ESMTP id D1BC88FC0A; Mon, 12 Dec 2011 16:43:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=jD/DGWevIEiPAWIQLQuAD22uufZYP5CNuRFCmUv51UA=; b=Dl0Gj5PbcN+1zz7mVWHE8/hPnfQxmsQdacvdqYFZvJtXvtn6Vuo9nFSEGw5UGmTAbc3eMexcRz+2Fi96kma36t1L9updJ005PSmNRxyWYNRFpgXcQd2MzgGq0oSNK7BFhPkdH7Bk1m/Yibe7yklOeunEW8URctf1hqGoINM0gRY=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm1.ukr.net with esmtpsa ID 1Ra8k5-0000sP-U4 ; Mon, 12 Dec 2011 18:28:41 +0200 Date: Mon, 12 Dec 2011 18:28:41 +0200 From: Ivan Klymenko To: Bruce Cran Message-ID: <20111212182841.31f669ca@nonamehost.> In-Reply-To: <4EE6295B.3020308@cran.org.uk> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE6295B.3020308@cran.org.uk> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Steve Kargl Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:43:52 -0000 =D0=92 Mon, 12 Dec 2011 16:18:35 +0000 Bruce Cran =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > On 12/12/2011 15:51, Steve Kargl wrote: > > This comes up every 9 months or so, and must be approaching FAQ=20 > > status. In a HPC environment, I recommend 4BSD. Depending on the=20 > > workload, ULE can cause a severe increase in turn around time when=20 > > doing already long computations. If you have an MPI application,=20 > > simply launching greater than ncpu+1 jobs can show the problem. PS:=20 > > search the list archives for "kargl and ULE".=20 >=20 > This isn't something that can be fixed by tuning ULE? For example for=20 > desktop applications kern.sched.preempt_thresh should be set to 224 > from its default. I'm wondering if the installer should ask people > what the typical use will be, and tune the scheduler appropriately. >=20 This is by and large does not help in certain situations ... From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 16:47:37 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5CC591065672; Mon, 12 Dec 2011 16:47:37 +0000 (UTC) (envelope-from gljennjohn@googlemail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 920F18FC17; Mon, 12 Dec 2011 16:47:36 +0000 (UTC) Received: by eekc50 with SMTP id c50so3382840eek.13 for ; Mon, 12 Dec 2011 08:47:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :x-mailer:mime-version:content-type:content-transfer-encoding; bh=Ne7LltZXZK7mD1SEiZFcIDyP8Mb797kkP51TT0gaTgc=; b=HpKS7+i6ORdEKlB0ZZOMNdQLfVhCj5U1R4RQ9U66CjKYvQBgsHLWhqZBxIOfpqu2Or NigZkoIhrXioa8tlSTgW2OEImo8bAE24IoqSjT7grHtNK0gDPGwVnRH7+1Dca0+C++UP /Ed0tVkrMrHxhwfOOE4LoQdxyW0c4ptsO0cm4= Received: by 10.14.4.134 with SMTP id 6mr2809056eej.183.1323708454160; Mon, 12 Dec 2011 08:47:34 -0800 (PST) Received: from ernst.jennejohn.org (p578E15DC.dip.t-dialin.net. [87.142.21.220]) by mx.google.com with ESMTPS id 53sm77232850eef.2.2011.12.12.08.47.31 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 12 Dec 2011 08:47:32 -0800 (PST) Date: Mon, 12 Dec 2011 17:47:30 +0100 From: Gary Jennejohn To: Lars Engels Message-ID: <20111212174730.0aee3e2c@ernst.jennejohn.org> In-Reply-To: <1a8a6d6f-6756-4cda-b4d6-b39d335678c1@email.android.com> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE619FC.4000601@unsane.co.uk> <20111212163221.33d0b8a2@ernst.jennejohn.org> <1a8a6d6f-6756-4cda-b4d6-b39d335678c1@email.android.com> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Vincent Hoffman Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gljennjohn@googlemail.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:47:37 -0000 On Mon, 12 Dec 2011 17:10:46 +0100 Lars Engels wrote: > Did you use -jX to build the world? > I'm top posting since Lars did. It was buildkernel, not buildworld. Yes, -j6. > _____________________________________________ > Von: Gary Jennejohn > Versendet am: Mon Dec 12 16:32:21 MEZ 2011 > An: Vincent Hoffman > CC: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org > Betreff: Re: SCHED_ULE should not be the default > > > On Mon, 12 Dec 2011 15:13:00 +0000 > Vincent Hoffman wrote: > > > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > On 12/12/2011 13:47, O. Hartmann wrote: > > > > > >> Not fully right, boinc defaults to run on idprio 31 so this isn't an > > >> issue. And yes, there are cases where SCHED_ULE shows much better > > >> performance then SCHED_4BSD. [...] > > > > > > Do we have any proof at hand for such cases where SCHED_ULE performs > > > much better than SCHED_4BSD? Whenever the subject comes up, it is > > > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > > > 2. But in the end I see here contradictionary statements. People > > > complain about poor performance (especially in scientific environments), > > > and other give contra not being the case. > > It all a little old now but some if the stuff in > > http://people.freebsd.org/~kris/scaling/ > > covers improvements that were seen. > > > > http://jeffr-tech.livejournal.com/5705.html > > shows a little too, reading though Jeffs blog is worth it as it has some > > interesting stuff on SHED_ULE. > > > > I thought there were some more benchmarks floating round but cant find > > any with a quick google. > > > > > > Vince > > > > > > > > Within our department, we developed a highly scalable code for planetary > > > science purposes on imagery. It utilizes present GPUs via OpenCL if > > > present. Otherwise it grabs as many cores as it can. > > > By the end of this year I'll get a new desktop box based on Intels new > > > Sandy Bridge-E architecture with plenty of memory. If the colleague who > > > developed the code is willing performing some benchmarks on the same > > > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > > > recent Suse. For FreeBSD I intent also to look for performance with both > > > different schedulers available. > > > > > These observations are not scientific, but I have a CPU from AMD with > 6 cores (AMD Phenom(tm) II X6 1090T Processor). > > My simple test was ``make buildkernel'' while watching the core usage with > gkrellm. > > With SCHED_4BSD all 6 cores are loaded to 97% during the build phase. > I've never seen any value above 97% with gkrellm. > > With SCHED_ULE I never saw all 6 cores loaded this heavily. Usually > 2 or more cores were at or below 90%. Not really that significant, but > still a noticeable difference in apparent scheduling behavior. Whether > the observed difference is due to some change in data from the kernel to > gkrellm is beyond me. > > -- > Gary Jennejohn > _____________________________________________ > > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > -- Gary Jennejohn From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 16:48:59 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D3F4106567A; Mon, 12 Dec 2011 16:48:59 +0000 (UTC) (envelope-from gljennjohn@googlemail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id A03D38FC14; Mon, 12 Dec 2011 16:48:58 +0000 (UTC) Received: by eekc50 with SMTP id c50so3384185eek.13 for ; Mon, 12 Dec 2011 08:48:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :x-mailer:mime-version:content-type:content-transfer-encoding; bh=QFQyElN+lsakH/hXyvqde4RifLlcVgBZTDRMOHlFA6E=; b=L49yFzALu8AFiSjmnq5Zoxakto8S8RZJ5uihrtPrGtPlKoedDQdm0FXoIuWxq91kbJ x1fEYqXOLpCrAM4S8Kku5agZBB0aRgPTOrDIo0G9jd3liNL+dgZMrdNVsf66vaXA6D1P 2/uSipRpqUkwxpwUmlgeRdrYxltzRBNO1Yj+g= Received: by 10.14.4.154 with SMTP id 26mr2818946eej.36.1323708537575; Mon, 12 Dec 2011 08:48:57 -0800 (PST) Received: from ernst.jennejohn.org (p578E15DC.dip.t-dialin.net. [87.142.21.220]) by mx.google.com with ESMTPS id 39sm12128785eei.1.2011.12.12.08.48.55 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 12 Dec 2011 08:48:56 -0800 (PST) Date: Mon, 12 Dec 2011 17:48:54 +0100 From: Gary Jennejohn To: mdf@FreeBSD.org Message-ID: <20111212174854.4f6ea14c@ernst.jennejohn.org> In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE619FC.4000601@unsane.co.uk> <20111212163221.33d0b8a2@ernst.jennejohn.org> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Vincent Hoffman Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gljennjohn@googlemail.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:48:59 -0000 On Mon, 12 Dec 2011 08:04:37 -0800 mdf@FreeBSD.org wrote: > On Mon, Dec 12, 2011 at 7:32 AM, Gary Jennejohn > wrote: > > On Mon, 12 Dec 2011 15:13:00 +0000 > > Vincent Hoffman wrote: > > > >> > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA1 > >> > >> On 12/12/2011 13:47, O. Hartmann wrote: > >> > > >> >> Not fully right, boinc defaults to run on idprio 31 so this isn't an > >> >> issue. And yes, there are cases where SCHED_ULE shows much better > >> >> performance then SCHED_4BSD. [...] > >> > > >> > Do we have any proof at hand for such cases where SCHED_ULE performs > >> > much better than SCHED_4BSD? Whenever the subject comes up, it is > >> > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > >> > 2. But in the end I see here contradictionary statements. People > >> > complain about poor performance (especially in scientific environments), > >> > and other give contra not being the case. > >> It all a little old now but some if the stuff in > >> http://people.freebsd.org/~kris/scaling/ > >> covers improvements that were seen. > >> > >> http://jeffr-tech.livejournal.com/5705.html > >> shows a little too, reading though Jeffs blog is worth it as it has some > >> interesting stuff on SHED_ULE. > >> > >> I thought there were some more benchmarks floating round but cant find > >> any with a quick google. > >> > >> > >> Vince > >> > >> > > >> > Within our department, we developed a highly scalable code for planetary > >> > science purposes on imagery. It utilizes present GPUs via OpenCL if > >> > present. Otherwise it grabs as many cores as it can. > >> > By the end of this year I'll get a new desktop box based on Intels new > >> > Sandy Bridge-E architecture with plenty of memory. If the colleague who > >> > developed the code is willing performing some benchmarks on the same > >> > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > >> > recent Suse. For FreeBSD I intent also to look for performance with both > >> > different schedulers available. > >> > > > > > These observations are not scientific, but I have a CPU from AMD with > > 6 cores (AMD Phenom(tm) II X6 1090T Processor). > > > > My simple test was ``make buildkernel'' while watching the core usage with > > gkrellm. > > > > With SCHED_4BSD all 6 cores are loaded to 97% during the build phase. > > I've never seen any value above 97% with gkrellm. > > > > With SCHED_ULE I never saw all 6 cores loaded this heavily.  Usually > > 2 or more cores were at or below 90%.  Not really that significant, but > > still a noticeable difference in apparent scheduling behavior.  Whether > > the observed difference is due to some change in data from the kernel to > > gkrellm is beyond me. > > SCHED_ULE is much sloppier about calculating which thread used a > timeslice -- unless the timeslice went 100% to a thread, the fraction > it used may get attributed elsewhere. So top's reporting of thread > usage is not a useful metric. Total buildworld time is, potentially. > I suspect you're right since the buildworld time, a much better test, was pretty much the same with 4BSD and ULE. -- Gary Jennejohn From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 17:06:07 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0371106564A; Mon, 12 Dec 2011 17:06:07 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) by mx1.freebsd.org (Postfix) with ESMTP id 8DE578FC12; Mon, 12 Dec 2011 17:06:07 +0000 (UTC) Received: from troutmask.apl.washington.edu (localhost.apl.washington.edu [127.0.0.1]) by troutmask.apl.washington.edu (8.14.5/8.14.5) with ESMTP id pBCH64v5074234; Mon, 12 Dec 2011 09:06:04 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.14.5/8.14.5/Submit) id pBCH64Fe074233; Mon, 12 Dec 2011 09:06:04 -0800 (PST) (envelope-from sgk) Date: Mon, 12 Dec 2011 09:06:04 -0800 From: Steve Kargl To: Bruce Cran Message-ID: <20111212170604.GA74044@troutmask.apl.washington.edu> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE6295B.3020308@cran.org.uk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE6295B.3020308@cran.org.uk> User-Agent: Mutt/1.4.2.3i Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 17:06:07 -0000 On Mon, Dec 12, 2011 at 04:18:35PM +0000, Bruce Cran wrote: > On 12/12/2011 15:51, Steve Kargl wrote: > >This comes up every 9 months or so, and must be approaching FAQ > >status. In a HPC environment, I recommend 4BSD. Depending on the > >workload, ULE can cause a severe increase in turn around time when > >doing already long computations. If you have an MPI application, > >simply launching greater than ncpu+1 jobs can show the problem. PS: > >search the list archives for "kargl and ULE". > > This isn't something that can be fixed by tuning ULE? For example for > desktop applications kern.sched.preempt_thresh should be set to 224 from > its default. I'm wondering if the installer should ask people what the > typical use will be, and tune the scheduler appropriately. > Tuning kern.sched.preempt_thresh did not seem to help for my workload. My code is a classic master-slave OpenMPI application where the master runs on one node and all cpu-bound slaves are sent to a second node. If I send send ncpu+1 jobs to the 2nd node with ncpu's, then ncpu-1 jobs are assigned to the 1st ncpu-1 cpus. The last two jobs are assigned to the ncpu'th cpu, and these ping-pong on the this cpu. AFAICT, it is a cpu affinity issue, where ULE is trying to keep each job associated with its initially assigned cpu. While one might suggest that starting ncpu+1 jobs is not prudent, my example is just that. It is an example showing that ULE has performance issues. So, I now can start only ncpu jobs on each node in the cluster and send emails to all other users to not use those node, or use 4BSD and not worry about loading issues. -- Steve From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 18:50:14 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C6341065675; Mon, 12 Dec 2011 18:50:14 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 3066C8FC16; Mon, 12 Dec 2011 18:50:14 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id D06F946B3C; Mon, 12 Dec 2011 13:50:13 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 42D2DB961; Mon, 12 Dec 2011 13:50:13 -0500 (EST) From: John Baldwin To: freebsd-current@freebsd.org Date: Mon, 12 Dec 2011 13:50:11 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <4EE1EAFE.3070408@m5p.com> <4EE6295B.3020308@cran.org.uk> <20111212170604.GA74044@troutmask.apl.washington.edu> In-Reply-To: <20111212170604.GA74044@troutmask.apl.washington.edu> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201112121350.11784.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 12 Dec 2011 13:50:13 -0500 (EST) Cc: Bruce Cran , "O. Hartmann" , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Steve Kargl Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 18:50:14 -0000 On Monday, December 12, 2011 12:06:04 pm Steve Kargl wrote: > On Mon, Dec 12, 2011 at 04:18:35PM +0000, Bruce Cran wrote: > > On 12/12/2011 15:51, Steve Kargl wrote: > > >This comes up every 9 months or so, and must be approaching FAQ > > >status. In a HPC environment, I recommend 4BSD. Depending on the > > >workload, ULE can cause a severe increase in turn around time when > > >doing already long computations. If you have an MPI application, > > >simply launching greater than ncpu+1 jobs can show the problem. PS: > > >search the list archives for "kargl and ULE". > > > > This isn't something that can be fixed by tuning ULE? For example for > > desktop applications kern.sched.preempt_thresh should be set to 224 from > > its default. I'm wondering if the installer should ask people what the > > typical use will be, and tune the scheduler appropriately. > > > > Tuning kern.sched.preempt_thresh did not seem to help for > my workload. My code is a classic master-slave OpenMPI > application where the master runs on one node and all > cpu-bound slaves are sent to a second node. If I send > send ncpu+1 jobs to the 2nd node with ncpu's, then > ncpu-1 jobs are assigned to the 1st ncpu-1 cpus. The > last two jobs are assigned to the ncpu'th cpu, and > these ping-pong on the this cpu. AFAICT, it is a cpu > affinity issue, where ULE is trying to keep each job > associated with its initially assigned cpu. > > While one might suggest that starting ncpu+1 jobs > is not prudent, my example is just that. It is an > example showing that ULE has performance issues. > So, I now can start only ncpu jobs on each node > in the cluster and send emails to all other users > to not use those node, or use 4BSD and not worry > about loading issues. This is a case where 4BSD's naive algorithm will spread out the load more evenly because all the threads are on a single, shared queue and each CPU just grabs the head of the queue when it finishes a timeslice. ULE always assigns threads to a single CPU (even if they aren't pinned to a single CPU using cpuset, etc.) and then tries to balance the load across cores later, but I believe in this case it's rebalancer won't have anything to really do as no matter what it does with the N+1 job it's going to be sharing a CPU with another job. -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 18:58:32 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC06A1065672; Mon, 12 Dec 2011 18:58:32 +0000 (UTC) (envelope-from uqs@spoerlein.net) Received: from acme.spoerlein.net (acme.spoerlein.net [IPv6:2a01:4f8:131:23c2::1]) by mx1.freebsd.org (Postfix) with ESMTP id 449998FC13; Mon, 12 Dec 2011 18:58:32 +0000 (UTC) Received: from localhost (acme.spoerlein.net [IPv6:2a01:4f8:131:23c2::1]) by acme.spoerlein.net (8.14.4/8.14.4) with ESMTP id pBCIwUgW066143 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Mon, 12 Dec 2011 19:58:31 +0100 (CET) (envelope-from uqs@spoerlein.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=spoerlein.net; s=dkim200908; t=1323716311; bh=qLZDzTRa//EkXZVdrYBPjXXggU3M3PM21doSSE2Dvs8=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Transfer-Encoding:In-Reply-To; b=o+UDh4JkTd8vOvoA9ZUP7xXJVZqVw/Qmy1VHsyrYpgyPa/Hm8BFwuN9wtL/LM8a62 YTt52olCmHoBzQIVgeRwwDDfhX75DeRH+0JfNWYJgcwQELgcD2CX6xZ2iNrGb1+sdF ccYyviuP5I0NPGpEFaj38AQeI0rqUfxwb1WZsNR8= Date: Mon, 12 Dec 2011 19:58:30 +0100 From: Ulrich =?utf-8?B?U3DDtnJsZWlu?= To: Dimitry Andric Message-ID: <20111212185830.GL83814@acme.spoerlein.net> Mail-Followup-To: Dimitry Andric , stable@freebsd.org References: <20111211223327.GJ83814@acme.spoerlein.net> <4EE53431.1040609@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <4EE53431.1040609@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: stable@freebsd.org Subject: Re: stable/9 preferring IPv4 over IPv6, what changed? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 18:58:32 -0000 On Sun, 2011-12-11 at 23:52:33 +0100, Dimitry Andric wrote: > On 2011-12-11 23:33, Ulrich Spörlein wrote: > > long story short: telnet foo on stable/8 will first try connecting via > > IPv6, then via IPv4 (foo has A and AAAA records). On stable/9 it's the > > other way round. > > > > This trips up my setup, where a bunch of hosts (some behind NAT) can all > > talk to each other over their IPv6 addresses (some are tunneled), but > > cannot do so via IPv4. > > > > Is this due to changes in bind or the resolver? > > Most likely due to changes in the IPv6 startup scripts and rc.conf > settings. The behaviour seems to be determined by multiple settings in > rc.conf, first of all: > > ip6addrctl_policy={ipv4_prefer|ipv6_prefer|AUTO} > > where the default value is AUTO. Values of ipv4_prefer and ipv6_prefer > do what you expect them to. > > In case of AUTO, and if you don't have /etc/ip6addrctl.conf with > explicit settings, /etc/rc.d/ip6addrctl checks the value of > ipv6_activate_all_interfaces. If it is YES, IPv6 is preferred, if it is > NO or unset, IPv4 is preferred. > > What are your IPv6-related settings in rc.conf? Well, I had ipv6_enable set from the stable/8 days. The warnings and the code make me believe it should behave as if ipv6_activate_all_interfaces was set, somehow that's not the case, though. I've now set ip6addrctl_policy and everything is back working again. Thanks! Uli From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 19:16:07 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3CBC11065677; Mon, 12 Dec 2011 19:16:07 +0000 (UTC) (envelope-from lambert@lambertfam.org) Received: from sysmon.tcworks.net (sysmon.tcworks.net [65.66.76.4]) by mx1.freebsd.org (Postfix) with ESMTP id B16A08FC1E; Mon, 12 Dec 2011 19:16:06 +0000 (UTC) Received: from sysmon.tcworks.net (localhost [127.0.0.1]) by sysmon.tcworks.net (8.13.1/8.13.1) with ESMTP id pBCJ3Ua5079662; Mon, 12 Dec 2011 13:03:30 -0600 (CST) (envelope-from lambert@lambertfam.org) Received: (from lambert@localhost) by sysmon.tcworks.net (8.13.1/8.13.1/Submit) id pBCJ3UQl079661; Mon, 12 Dec 2011 13:03:30 -0600 (CST) (envelope-from lambert@lambertfam.org) X-Authentication-Warning: sysmon.tcworks.net: lambert set sender to lambert@lambertfam.org using -f Date: Mon, 12 Dec 2011 13:03:30 -0600 From: Scott Lambert To: Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Message-ID: <20111212190330.GA69380@sysmon.tcworks.net> Mail-Followup-To: Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE6295B.3020308@cran.org.uk> <20111212170604.GA74044@troutmask.apl.washington.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111212170604.GA74044@troutmask.apl.washington.edu> User-Agent: Mutt/1.4.2.2i Cc: Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 19:16:07 -0000 On Mon, Dec 12, 2011 at 09:06:04AM -0800, Steve Kargl wrote: > Tuning kern.sched.preempt_thresh did not seem to help for > my workload. My code is a classic master-slave OpenMPI > application where the master runs on one node and all > cpu-bound slaves are sent to a second node. If I send > send ncpu+1 jobs to the 2nd node with ncpu's, then > ncpu-1 jobs are assigned to the 1st ncpu-1 cpus. The > last two jobs are assigned to the ncpu'th cpu, and > these ping-pong on the this cpu. AFAICT, it is a cpu > affinity issue, where ULE is trying to keep each job > associated with its initially assigned cpu. > > While one might suggest that starting ncpu+1 jobs > is not prudent, my example is just that. It is an > example showing that ULE has performance issues. > So, I now can start only ncpu jobs on each node > in the cluster and send emails to all other users > to not use those node, or use 4BSD and not worry > about loading issues. Does it meet your expectations if you start (j modulo ncpu) = 0 jobs on a node? -- Scott Lambert KC5MLE Unix SysAdmin lambert@lambertfam.org From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 19:23:07 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D6761065680 for ; Mon, 12 Dec 2011 19:23:07 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) by mx1.freebsd.org (Postfix) with ESMTP id 40EAC8FC17 for ; Mon, 12 Dec 2011 19:23:07 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.4/8.14.4) with ESMTP id pBCJN2x8043916; Mon, 12 Dec 2011 14:23:02 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.4/8.14.4/Submit) id pBCJN2Jk043913; Mon, 12 Dec 2011 14:23:02 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <20198.21654.915449.536365@hergotha.csail.mit.edu> Date: Mon, 12 Dec 2011 14:23:02 -0500 From: Garrett Wollman To: bruce@cran.org.uk In-Reply-To: <4EE6295B.3020308@cran.org.uk> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> X-Newsgroups: mit.lcs.mail.freebsd-stable X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (hergotha.csail.mit.edu [127.0.0.1]); Mon, 12 Dec 2011 14:23:03 -0500 (EST) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (hergotha.csail.mit.edu [127.0.0.1]); Mon, 12 Dec 2011 14:16:36 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hergotha.csail.mit.edu Cc: stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 19:23:07 -0000 In article <4EE6295B.3020308@cran.org.uk>, brucecran.org.uk writes: >This isn't something that can be fixed by tuning ULE? For example for >desktop applications kern.sched.preempt_thresh should be set to 224 from >its default. Where do you get that idea? I've never seen any evidence for this proposition (although the claim is repeated often enough). What are the specific circumstances that make this useful? Where did the number come from? -GAWollman From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 19:26:38 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 327C0106566B; Mon, 12 Dec 2011 19:26:38 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) by mx1.freebsd.org (Postfix) with ESMTP id E4EEC8FC17; Mon, 12 Dec 2011 19:26:37 +0000 (UTC) Received: from troutmask.apl.washington.edu (localhost.apl.washington.edu [127.0.0.1]) by troutmask.apl.washington.edu (8.14.5/8.14.5) with ESMTP id pBCJQbTv087779; Mon, 12 Dec 2011 11:26:37 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.14.5/8.14.5/Submit) id pBCJQbD3087778; Mon, 12 Dec 2011 11:26:37 -0800 (PST) (envelope-from sgk) Date: Mon, 12 Dec 2011 11:26:37 -0800 From: Steve Kargl To: Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Message-ID: <20111212192637.GA87729@troutmask.apl.washington.edu> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE6295B.3020308@cran.org.uk> <20111212170604.GA74044@troutmask.apl.washington.edu> <20111212190330.GA69380@sysmon.tcworks.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111212190330.GA69380@sysmon.tcworks.net> User-Agent: Mutt/1.4.2.3i Cc: Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 19:26:38 -0000 On Mon, Dec 12, 2011 at 01:03:30PM -0600, Scott Lambert wrote: > On Mon, Dec 12, 2011 at 09:06:04AM -0800, Steve Kargl wrote: > > Tuning kern.sched.preempt_thresh did not seem to help for > > my workload. My code is a classic master-slave OpenMPI > > application where the master runs on one node and all > > cpu-bound slaves are sent to a second node. If I send > > send ncpu+1 jobs to the 2nd node with ncpu's, then > > ncpu-1 jobs are assigned to the 1st ncpu-1 cpus. The > > last two jobs are assigned to the ncpu'th cpu, and > > these ping-pong on the this cpu. AFAICT, it is a cpu > > affinity issue, where ULE is trying to keep each job > > associated with its initially assigned cpu. > > > > While one might suggest that starting ncpu+1 jobs > > is not prudent, my example is just that. It is an > > example showing that ULE has performance issues. > > So, I now can start only ncpu jobs on each node > > in the cluster and send emails to all other users > > to not use those node, or use 4BSD and not worry > > about loading issues. > > Does it meet your expectations if you start (j modulo ncpu) = 0 > jobs on a node? > I've never tried to launch more than ncpu + 1 (or + 2) jobs. I suppose at the time I was investigating the issue, it was determined that 4BSD allowed me to get my work done in a more timely manner. So, I took the path of least resistance. -- Steve From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 19:43:27 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE03E106564A for ; Mon, 12 Dec 2011 19:43:27 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [IPv6:2a01:348:0:15:5d59:5c40:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id 506198FC14 for ; Mon, 12 Dec 2011 19:43:27 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id AD1CEE6208; Mon, 12 Dec 2011 19:43:25 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=mail; bh=KBofS3P/8735 YfGbLXBay3DOQ+w=; b=Rvk1DiqI5UkQIJwWCALJ4e/oPCrfXF6QBJmjbSxYTR2y 4slF4su6ItFEQvPmt5zsMIV/vtnk77+JcjRH2Jx9gYA51lEwGOT8TVKcql/fRujm 7g2pXydMfIM8Be0sCTTYfz4+iGSyg83UftquVutmIGX2VXvCNi2YfKW/cKBUCLk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=mail; b=PBclhI T5FM9Ykjv+FdHcDtw6hggo59P0Tsv0SsaB7+/pURV3MMPKSva5Yvmm7hhad3ouie 8SANjss0+7+2T18zC4FAO0LBGLeMcHvANwdVMieCZDKj4oPojIgf3SXNt/PBX3by PWYl70LmHuCArs2rhZK022u0FnoxwJ+s7i2TU= Received: from [192.168.1.120] (188-220-36-32.zone11.bethere.co.uk [188.220.36.32]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id 51243E6200; Mon, 12 Dec 2011 19:43:25 +0000 (GMT) Message-ID: <4EE6595C.3080608@cran.org.uk> Date: Mon, 12 Dec 2011 19:43:24 +0000 From: Bruce Cran User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Garrett Wollman References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <20198.21654.915449.536365@hergotha.csail.mit.edu> In-Reply-To: <20198.21654.915449.536365@hergotha.csail.mit.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 19:43:28 -0000 On 12/12/2011 19:23, Garrett Wollman wrote: > Where do you get that idea? I've never seen any evidence for this > proposition (although the claim is repeated often enough). What are > the specific circumstances that make this useful? Where did the > number come from? It's just something I've heard repeated, and people claiming that setting it improves performance. This explains how the value 224 was obtained: http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058686.html -- Bruce Cran From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 20:25:48 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 08D85106564A for ; Mon, 12 Dec 2011 20:25:48 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) by mx1.freebsd.org (Postfix) with ESMTP id B26208FC0C for ; Mon, 12 Dec 2011 20:25:47 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.4/8.14.4) with ESMTP id pBCKPgkO044580; Mon, 12 Dec 2011 15:25:43 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.4/8.14.4/Submit) id pBCKPgtB044579; Mon, 12 Dec 2011 15:25:42 -0500 (EST) (envelope-from wollman) Date: Mon, 12 Dec 2011 15:25:42 -0500 (EST) From: Garrett Wollman Message-Id: <201112122025.pBCKPgtB044579@hergotha.csail.mit.edu> To: bruce@cran.org.uk X-Newsgroups: mit.lcs.mail.freebsd-stable In-Reply-To: <4EE6595C.3080608@cran.org.uk> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <20198.21654.915449.536365@hergotha.csail.mit.edu> Organization: none X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (hergotha.csail.mit.edu [127.0.0.1]); Mon, 12 Dec 2011 15:25:43 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hergotha.csail.mit.edu Cc: stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 20:25:48 -0000 In article <4EE6595C.3080608@cran.org.uk>, bruce@cran.org.uk writes: >On 12/12/2011 19:23, Garrett Wollman wrote: >> Where do you get that idea? I've never seen any evidence for this >> proposition (although the claim is repeated often enough). What are >> the specific circumstances that make this useful? Where did the >> number come from? > >It's just something I've heard repeated, and people claiming that >setting it improves performance. > >This explains how the value 224 was obtained: >http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058686.html Not so far as I can see. The message does suggest that it helps if you are running a CPU-hog GUI, which seems plausible to me, but doesn't justify making it the default -- particularly when the setting is undocumented. (It appears to control how CPU-bound a process can be and still preempt another even more CPU-bound process, so using this as a "desktop performance" "fix" looks doubly wrong.) -GAWollman From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 20:31:53 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49D331065672 for ; Mon, 12 Dec 2011 20:31:53 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 204F28FC0C for ; Mon, 12 Dec 2011 20:31:53 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id B31A446B1A; Mon, 12 Dec 2011 15:31:52 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 4362FB91E; Mon, 12 Dec 2011 15:31:52 -0500 (EST) From: John Baldwin To: Jan Mikkelsen Date: Mon, 12 Dec 2011 14:08:45 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <20E452A2-9180-4FC9-87CB-6F16D1D4D01A@transactionware.com> <201112091103.51345.jhb@freebsd.org> <51DA2621-F65E-40AD-87EE-F43BE4AEBA5B@transactionware.com> In-Reply-To: <51DA2621-F65E-40AD-87EE-F43BE4AEBA5B@transactionware.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201112121408.45396.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 12 Dec 2011 15:31:52 -0500 (EST) Cc: freebsd-stable@freebsd.org Subject: Re: mfi(4) issues in 9.0-RC3 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 20:31:53 -0000 On Sunday, December 11, 2011 7:22:38 pm Jan Mikkelsen wrote: > On 10/12/2011, at 3:03 AM, John Baldwin wrote: > > > On Friday, December 09, 2011 8:38:51 am Jan Mikkelsen wrote: > >> Hi, > >> > >> Can rev 227562 be merged into 9.0? > >> [...] > > > > It is probably too late to make 9.0 at this point as they've already cut the > > final release candidate. > > OK; I'll maintain the patch myself until it gets merged. > > The MegaRAID 9261-8i requires hw.mfi.msi=1 to be set by hand for a boot to successfully complete. It it worth adding a check for that device? I think what I might ask is for folks to try enabling that to see if we can just enable MSI for all mfi(4) devices by default. MSI-X did not work on several devices it seems, but I think MSI has worked on all the folks where I've asked them to try it. -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 21:15:33 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7E570106566B; Mon, 12 Dec 2011 21:15:33 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-stable.sentex.ca (freebsd-stable.sentex.ca [IPv6:2607:f3e0:0:3::6502:9b]) by mx1.freebsd.org (Postfix) with ESMTP id 376FA8FC0C; Mon, 12 Dec 2011 21:15:33 +0000 (UTC) Received: from freebsd-stable.sentex.ca (localhost [127.0.0.1]) by freebsd-stable.sentex.ca (8.14.5/8.14.5) with ESMTP id pBCLFWwE087567; Mon, 12 Dec 2011 21:15:32 GMT (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-stable.sentex.ca (8.14.5/8.14.5/Submit) id pBCLFWVb087554; Mon, 12 Dec 2011 21:15:32 GMT (envelope-from tinderbox@freebsd.org) Date: Mon, 12 Dec 2011 21:15:32 GMT Message-Id: <201112122115.pBCLFWVb087554@freebsd-stable.sentex.ca> X-Authentication-Warning: freebsd-stable.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Precedence: bulk Cc: Subject: [releng_9 tinderbox] failure on powerpc64/powerpc X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 21:15:33 -0000 TB --- 2011-12-12 20:58:39 - tinderbox 2.8 running on freebsd-stable.sentex.ca TB --- 2011-12-12 20:58:39 - starting RELENG_9 tinderbox run for powerpc64/powerpc TB --- 2011-12-12 20:58:39 - cleaning the object tree TB --- 2011-12-12 20:58:55 - cvsupping the source tree TB --- 2011-12-12 20:58:55 - /usr/bin/csup -z -r 3 -g -L 1 -h cvsup.sentex.ca /tinderbox/RELENG_9/powerpc64/powerpc/supfile TB --- 2011-12-12 21:15:32 - WARNING: /usr/bin/csup returned exit code 1 TB --- 2011-12-12 21:15:32 - ERROR: unable to cvsup the source tree TB --- 2011-12-12 21:15:32 - 1.97 user 4.13 system 1013.05 real http://tinderbox.freebsd.org/tinderbox-releng_9-RELENG_9-powerpc64-powerpc.full From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 00:04:06 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 629391065676; Tue, 13 Dec 2011 00:04:06 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 150A08FC1C; Tue, 13 Dec 2011 00:04:05 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RaFbr-000792-2w>; Tue, 13 Dec 2011 00:48:39 +0100 Received: from e178008161.adsl.alicedsl.de ([85.178.8.161] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RaFbq-0005wx-U2>; Tue, 13 Dec 2011 00:48:39 +0100 Message-ID: <4EE692D6.5010208@zedat.fu-berlin.de> Date: Tue, 13 Dec 2011 00:48:38 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Steve Kargl References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE6295B.3020308@cran.org.uk> <20111212170604.GA74044@troutmask.apl.washington.edu> In-Reply-To: <20111212170604.GA74044@troutmask.apl.washington.edu> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigBAD169A868B0E5A48D5A4634" X-Originating-IP: 85.178.8.161 Cc: Bruce Cran , "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 00:04:06 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigBAD169A868B0E5A48D5A4634 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 12/12/11 18:06, Steve Kargl wrote: > On Mon, Dec 12, 2011 at 04:18:35PM +0000, Bruce Cran wrote: >> On 12/12/2011 15:51, Steve Kargl wrote: >>> This comes up every 9 months or so, and must be approaching FAQ=20 >>> status. In a HPC environment, I recommend 4BSD. Depending on the=20 >>> workload, ULE can cause a severe increase in turn around time when=20 >>> doing already long computations. If you have an MPI application,=20 >>> simply launching greater than ncpu+1 jobs can show the problem. PS:=20 >>> search the list archives for "kargl and ULE".=20 >> >> This isn't something that can be fixed by tuning ULE? For example for = >> desktop applications kern.sched.preempt_thresh should be set to 224 fr= om=20 >> its default. I'm wondering if the installer should ask people what the= =20 >> typical use will be, and tune the scheduler appropriately. >> Is the tuning of kern.sched.preempt_thresh and a proper method of estimating its correct value for the intended to use workload documented in the manpages, maybe tuning()? I find it hard to crawl a lot of pros and cons of mailing lists for evaluating a correct value of this, seemingly, important tunable. >=20 > Tuning kern.sched.preempt_thresh did not seem to help for > my workload. My code is a classic master-slave OpenMPI > application where the master runs on one node and all > cpu-bound slaves are sent to a second node. If I send > send ncpu+1 jobs to the 2nd node with ncpu's, then=20 > ncpu-1 jobs are assigned to the 1st ncpu-1 cpus. The > last two jobs are assigned to the ncpu'th cpu, and=20 > these ping-pong on the this cpu. AFAICT, it is a cpu > affinity issue, where ULE is trying to keep each job > associated with its initially assigned cpu. >=20 > While one might suggest that starting ncpu+1 jobs > is not prudent, my example is just that. It is an > example showing that ULE has performance issues.=20 > So, I now can start only ncpu jobs on each node > in the cluster and send emails to all other users > to not use those node, or use 4BSD and not worry > about loading issues. >=20 --------------enigBAD169A868B0E5A48D5A4634 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQEcBAEBAgAGBQJO5pLWAAoJEOgBcD7A/5N86FIIAMlp2MmSfYGAw+Gqn5MuN/s1 VxWt+47R+tii3x2I5rvjigs2+c5BbMhQ5B/+LS1qU8OspeAwWcvqYnXCXwKs7kUo FG+8mmdyVaqt9s1hoh/W4tHgDgL/DCMxwkIfS3yVubjqOltDo7npcre7sMoUaEjL lv0ySiLArwHbnD4mdrC3gJz/fW0enmNOl9wGYWWcUPcDdJ5XdYMSfSGk0W6bpSgA ewDaoPtz1jh/CkLAVH59/cxcHowtsM9YcrdTOPKOIAI9amNChlvtuv8Sv8g2LC9e RhgNHCE6RKVqAIpyIZLTFZ6pUfTtQeI6CtqWHDDAvhYAUEZxZmBDErazPkkirWQ= =prJ+ -----END PGP SIGNATURE----- --------------enigBAD169A868B0E5A48D5A4634-- From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 00:15:17 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D9921065672; Tue, 13 Dec 2011 00:15:17 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [IPv6:2a01:348:0:15:5d59:5c40:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id E32588FC16; Tue, 13 Dec 2011 00:15:16 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id 64AEAE6209; Tue, 13 Dec 2011 00:15:15 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=mail; bh=gJC95oEqPFyd PxKhOGrYzUyGTDw=; b=lxgw94TUBeTsBgWu2YSPBvEpRbjQcuqTJny+hvnZIiFU tHpeqewHNhRiny1zu6kxuMq9MtveVpYMV4ARIwScFH5LK/wbqs3do4spXmz+aKkU 53to5Y0Rb5TCj5DjChXujp47CaMhTdshe6DkZOlXc6EOfhSTZZzvq5AgNt8lf64= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=mail; b=Mq2NRg aaRj5CQxemqhNyn9XyA4O4PRCSn1UIY2zSw8JCM6SkjRIxcya2ZbuKOMw6ShJ5Ce WDDQnMskQEeJF3RIQ+eyWgORWCB3/3TW1oi305p5LAIVWxNAVOcnAbBwhaa+6XMP KmyZZwAc3ftWGIVyez2HIur75sxN5zkzmvenI= Received: from [192.168.1.120] (188-220-36-32.zone11.bethere.co.uk [188.220.36.32]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id D898DE6200; Tue, 13 Dec 2011 00:15:14 +0000 (GMT) Message-ID: <4EE69911.9040201@cran.org.uk> Date: Tue, 13 Dec 2011 00:15:13 +0000 From: Bruce Cran User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: "O. Hartmann" References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE6295B.3020308@cran.org.uk> <20111212170604.GA74044@troutmask.apl.washington.edu> <4EE692D6.5010208@zedat.fu-berlin.de> In-Reply-To: <4EE692D6.5010208@zedat.fu-berlin.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Steve Kargl Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 00:15:17 -0000 On 12/12/2011 23:48, O. Hartmann wrote: > Is the tuning of kern.sched.preempt_thresh and a proper method of > estimating its correct value for the intended to use workload > documented in the manpages, maybe tuning()? I find it hard to crawl a > lot of pros and cons of mailing lists for evaluating a correct value > of this, seemingly, important tunable. Note that I said "for example" :) I was suggesting that there may be sysctl's that can be tweaked to improve performance. -- Bruce Cran From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 00:29:15 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id B754E1065672; Tue, 13 Dec 2011 00:29:15 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 0F52614FAE3; Tue, 13 Dec 2011 00:29:15 +0000 (UTC) Message-ID: <4EE69C5A.3090005@FreeBSD.org> Date: Mon, 12 Dec 2011 16:29:14 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: "O. Hartmann" References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> In-Reply-To: <4EE6060D.5060201@mail.zedat.fu-berlin.de> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 00:29:15 -0000 On 12/12/2011 05:47, O. Hartmann wrote: > Do we have any proof at hand for such cases where SCHED_ULE performs > much better than SCHED_4BSD? I complained about poor interactive performance of ULE in a desktop environment for years. I had numerous people try to help, including Jeff, with various tunables, dtrace'ing, etc. The cause of the problem was never found. I switched to 4BSD, problem gone. This is on 2 separate systems with core 2 duos. hth, Doug -- [^L] Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 01:29:21 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7D56A1065670 for ; Tue, 13 Dec 2011 01:29:21 +0000 (UTC) (envelope-from george@m5p.com) Received: from mailhost.m5p.com (ip-3-2-0-2.r20.asbnva02.us.ce.gin.ntt.net [IPv6:2001:418:0:5000::16]) by mx1.freebsd.org (Postfix) with ESMTP id 359ED8FC0C for ; Tue, 13 Dec 2011 01:29:21 +0000 (UTC) Received: from wonderland.m5p.com (wonderland.m5p.com [IPv6:2001:418:3fd::19]) by mailhost.m5p.com (8.14.4/8.14.4) with ESMTP id pBD1TFRh090307 for ; Mon, 12 Dec 2011 20:29:20 -0500 (EST) (envelope-from george@m5p.com) Message-ID: <4EE6AA6B.6020803@m5p.com> Date: Mon, 12 Dec 2011 20:29:15 -0500 From: George Mitchell User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111127 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> In-Reply-To: <4EE69C5A.3090005@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (mailhost.m5p.com [IPv6:2001:418:3fd::f7]); Mon, 12 Dec 2011 20:29:20 -0500 (EST) X-Scanned-By: MIMEDefang 2.72 on IPv6:2001:418:3fd::f7 Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 01:29:21 -0000 On 12/12/11 19:29, Doug Barton wrote: > [...] > > I switched to 4BSD, problem gone. > > [...] Ditto. If there's some common situation where the average user would have a perceptibly better experience with ULE, let's go for it. But when there's a plausible usage scenario in which ULE gives OVER AN ORDER OF MAGNITUDE worse performance[1], making ULE the default seems like a bad choice. -- George Mitchell [1] http://lists.freebsd.org/pipermail/freebsd-stable/2011-December/064773.html From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 07:36:18 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F9551065673 for ; Tue, 13 Dec 2011 07:36:18 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta06.westchester.pa.mail.comcast.net (qmta06.westchester.pa.mail.comcast.net [76.96.62.56]) by mx1.freebsd.org (Postfix) with ESMTP id D34348FC1B for ; Tue, 13 Dec 2011 07:36:17 +0000 (UTC) Received: from omta24.westchester.pa.mail.comcast.net ([76.96.62.76]) by qmta06.westchester.pa.mail.comcast.net with comcast id 8Xc01i0011ei1Bg56XcJee; Tue, 13 Dec 2011 07:36:18 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta24.westchester.pa.mail.comcast.net with comcast id 8XcG1i00E1t3BNj3kXcHDk; Tue, 13 Dec 2011 07:36:18 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 6FAD3102C1A; Mon, 12 Dec 2011 23:36:15 -0800 (PST) Date: Mon, 12 Dec 2011 23:36:15 -0800 From: Jeremy Chadwick To: "O. Hartmann" Message-ID: <20111213073615.GA69641@icarus.home.lan> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE6060D.5060201@mail.zedat.fu-berlin.de> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 07:36:18 -0000 On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: > > Not fully right, boinc defaults to run on idprio 31 so this isn't an > > issue. And yes, there are cases where SCHED_ULE shows much better > > performance then SCHED_4BSD. [...] > > Do we have any proof at hand for such cases where SCHED_ULE performs > much better than SCHED_4BSD? Whenever the subject comes up, it is > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > 2. But in the end I see here contradictionary statements. People > complain about poor performance (especially in scientific environments), > and other give contra not being the case. > > Within our department, we developed a highly scalable code for planetary > science purposes on imagery. It utilizes present GPUs via OpenCL if > present. Otherwise it grabs as many cores as it can. > By the end of this year I'll get a new desktop box based on Intels new > Sandy Bridge-E architecture with plenty of memory. If the colleague who > developed the code is willing performing some benchmarks on the same > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > recent Suse. For FreeBSD I intent also to look for performance with both > different schedulers available. This is in no way shape or form the same kind of benchmark as what you're planning to do, but I thought I'd throw it out there for folks to take in as they see fit. I know folks were focused mainly on buildworld. I personally would find it interesting if someone with a higher-end system (e.g. 2 physical CPUs, with 6 or 8 cores per CPU) was to do the same test (changing -jX to -j{numofcores} of course). -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | sched_ule =========== - time make -j2 buildworld 1689.831u 229.328s 18:46.20 170.4% 6566+2051k 432+4264io 4565pf+0w - time make -j2 buildkernel 640.542u 87.737s 9:01.38 134.5% 6490+1920k 134+5968io 0pf+0w sched_4bsd ============ - time make -j2 buildworld 1662.793u 206.908s 17:12.02 181.1% 6578+2054k 23750+4271io 6451pf+0w - time make -j2 buildkernel 638.717u 76.146s 8:34.90 138.8% 6530+1927k 6415+5903io 0pf+0w software ========== * sched_ule test: FreeBSD 8.2-STABLE, Thu Dec 1 04:37:29 PST 2011 * sched_4bsd test: FreeBSD 8.2-STABLE, Mon Dec 12 22:42:54 PST 2011 hardware ========== * Intel Core 2 Duo E8400, 3GHz * Supermicro X7SBA * 8GB ECC RAM (4x2GB), DDR2-800 * Intel 320-series SSD, 80GB: /, swap, /var, /tmp, /usr tuning adjustments / etc. =========================== * Before each scheduler test, system was rebooted to ensure I/O cache and other whatnots were empty * All filesystems stock UFS2 + SU (root is non-SU) * All filesystems had tunefs -t enable applied to them * powerd(8) in use, with two rc.conf variables (per CPU spec): performance_cx_lowest="C2" economy_cx_lowest="C2" * loader.conf kern.maxdsiz="2560M" kern.dfldsiz="2560M" kern.maxssiz="256M" ahci_load="yes" hint.p4tcc.0.disabled="1" hint.acpi_throttle.0.disabled="1" vfs.zfs.arc_max="5120M" * make.conf CPUTYPE?=core2 * src.conf WITHOUT_INET6=true WITHOUT_IPFILTER=true WITHOUT_LIB32=true WITHOUT_KERBEROS=true WITHOUT_PAM_SUPPORT=true WITHOUT_PROFILE=true WITHOUT_SENDMAIL=true * kernel configuration - note: between kernel builds, config was changed to either use SCHED_4BSD or SCHED_ULE respectively. cpu HAMMER ident GENERIC makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols options SCHED_4BSD # Classic BSD scheduler #options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options MD_ROOT # MD is a potential root device options NFSCLIENT # Network Filesystem Client options NFSSERVER # Network Filesystem Server options NFSLOCKD # Network Lock Manager options NFS_ROOT # NFS usable as /, requires NFSCLIENT options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options COMPAT_43TTY # BSD 4.3 TTY compat (sgtty) options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options P1003_1B_SEMAPHORES # POSIX-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework options FLOWTABLE # per-cpu routing cache #options KDTRACE_FRAME # Ensure frames are compiled in #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcessor Kernel # Debugging options options BREAK_TO_DEBUGGER # Sending a serial BREAK drops to DDB options ALT_BREAK_TO_DEBUGGER # Permit ~ to drop to DDB options KDB # Enable kernel debugger support options KDB_TRACE # Print stack trace automatically on panic options DDB # Support DDB options DDB_NUMSYM # Print numeric value of symbols options GDB # Support remote GDB # CPU frequency control device cpufreq # Bus support. device acpi device pci # Floppy drives device fdc # ATA and ATAPI devices # NOTE: "device ata" is missing because we use the Modular ATA core # to only include the ATA-related drivers we need (e.g. AHCI). device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives options ATA_STATIC_ID # Static device numbering # Modular ATA device atacore # Core ATA functionality device ataisa # ISA bus support device atapci # PCI bus support; only generic chipset support device ataahci # AHCI SATA device ataintel # Intel # SCSI peripherals device scbus # SCSI bus (required for SCSI) device da # Direct Access (disks) device cd # CD device pass # Passthrough device (direct SCSI access) device ses # SCSI Environmental Services (and SAF-TE) options CAMDEBUG # CAM debugging (camcontrol debug) # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc device agp # support several AGP chipsets # Serial (COM) ports device uart # Generic UART driver # PCI Ethernet NICs. device em # Intel PRO/1000 Gigabit Ethernet Family # Wireless NIC cards device wlan # 802.11 support options IEEE80211_DEBUG # enable debug msgs options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's device wlan_wep # 802.11 WEP support device wlan_ccmp # 802.11 CCMP support device wlan_tkip # 802.11 TKIP support device wlan_amrr # AMRR transmit rate control algorithm device wlan_acl # MAC Access Control List support # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device pty # BSD-style compatibility pseudo ttys device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter # USB support device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device usb # USB Bus (required) #device udbp # USB Double Bulk Pipe devices device uhid # "Human Interface Devices" device ukbd # Keyboard device umass # Disks/Mass storage - Requires scbus and da device ums # Mouse # Intel Core/Core2Duo CPU temperature monitoring driver device coretemp # SMBus support, needed for bsdhwmon device smbus device smb device ichsmb # Intel ICH hardware watchdog support device ichwd # pf ALTQ support options ALTQ options ALTQ_CBQ # Class Bases Queueing options ALTQ_RED # Random Early Detection options ALTQ_RIO # RED In/Out options ALTQ_HFSC # Hierarchical Packet Scheduler options ALTQ_CDNR # Traffic conditioner options ALTQ_PRIQ # Priority Queueing options ALTQ_NOPCC # Required for SMP build From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 08:14:04 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 09005106564A for ; Tue, 13 Dec 2011 08:14:04 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 8306A8FC12 for ; Tue, 13 Dec 2011 08:14:03 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id pBD8DpHH086299 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Tue, 13 Dec 2011 10:13:56 +0200 (EET) (envelope-from daniel@digsys.bg) Message-ID: <4EE7093E.4050006@digsys.bg> Date: Tue, 13 Dec 2011 10:13:50 +0200 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111213073615.GA69641@icarus.home.lan> In-Reply-To: <20111213073615.GA69641@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 08:14:04 -0000 On 13.12.11 09:36, Jeremy Chadwick wrote: > I personally would find it interesting if someone with a higher-end > system (e.g. 2 physical CPUs, with 6 or 8 cores per CPU) was to do the > same test (changing -jX to -j{numofcores} of course). Is 4 way 8 core Opteron ok? That is 32 cores, 64GB RAM. Testing with buildworld in my opinion is not adequate, as it involves way too much I/O. Any advice on proper testing methodology? These systems run ZFS, but could be booted diskless for the tests. Daniel From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 08:40:54 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 60E04106564A; Tue, 13 Dec 2011 08:40:54 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm2.ukr.net (fsm2.ukr.net [195.214.192.121]) by mx1.freebsd.org (Postfix) with ESMTP id 0F9278FC08; Tue, 13 Dec 2011 08:40:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=02qllFexF+QvnW1q5ymUZZxjyYi7Er87q5EPTv3rxFA=; b=kRx4fPERvEuj3vMPAbWT5xSfgRyH2ufNdtJmGOjw62bi+Al4rw6lEJxT0y8LkWCq0Rzzk4AH93ZQ2c609v2pQ3bg9iFtYEZ/9k2GKJOhJIkY4iz6XeLHmyFIZ2fuPILkBi2udCx4NZfLQedqEtdLvWNpaDxsm73tgGZCCH0wei4=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm2.ukr.net with esmtpsa ID 1RaNut-000G89-UB ; Tue, 13 Dec 2011 10:40:52 +0200 Date: Tue, 13 Dec 2011 10:40:48 +0200 From: Ivan Klymenko To: Doug Barton Message-ID: <20111213104048.40f3e3de@nonamehost.> In-Reply-To: <4EE69C5A.3090005@FreeBSD.org> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 08:40:54 -0000 > On 12/12/2011 05:47, O. Hartmann wrote: > > Do we have any proof at hand for such cases where SCHED_ULE performs > > much better than SCHED_4BSD? > > I complained about poor interactive performance of ULE in a desktop > environment for years. I had numerous people try to help, including > Jeff, with various tunables, dtrace'ing, etc. The cause of the problem > was never found. > > I switched to 4BSD, problem gone. > > This is on 2 separate systems with core 2 duos. > > > hth, > > Doug > If the algorithm ULE does not contain problems - it means the problem has Core2Duo, or in a piece of code that uses the ULE scheduler. I already wrote in a mailing list that specifically in my case (Core2Duo) partially helps the following patch: --- sched_ule.c.orig 2011-11-24 18:11:48.000000000 +0200 +++ sched_ule.c 2011-12-10 22:47:08.000000000 +0200 @@ -794,7 +794,8 @@ * 1.5 * balance_interval. */ balance_ticks = max(balance_interval / 2, 1); - balance_ticks += random() % balance_interval; +// balance_ticks += random() % balance_interval; + balance_ticks += ((int)random()) % balance_interval; if (smp_started == 0 || rebalance == 0) return; tdq = TDQ_SELF(); @@ -2118,13 +2119,21 @@ struct td_sched *ts; THREAD_LOCK_ASSERT(td, MA_OWNED); + if (td->td_pri_class & PRI_FIFO_BIT) + return; + ts = td->td_sched; + /* + * We used up one time slice. + */ + if (--ts->ts_slice > 0) + return; tdq = TDQ_SELF(); #ifdef SMP /* * We run the long term load balancer infrequently on the first cpu. */ - if (balance_tdq == tdq) { - if (balance_ticks && --balance_ticks == 0) + if (balance_ticks && --balance_ticks == 0) { + if (balance_tdq == tdq) sched_balance(); } #endif @@ -2144,9 +2153,6 @@ if (TAILQ_EMPTY(&tdq->tdq_timeshare.rq_queues[tdq->tdq_ridx])) tdq->tdq_ridx = tdq->tdq_idx; } - ts = td->td_sched; - if (td->td_pri_class & PRI_FIFO_BIT) - return; if (PRI_BASE(td->td_pri_class) == PRI_TIMESHARE) { /* * We used a tick; charge it to the thread so @@ -2157,11 +2163,6 @@ sched_priority(td); } /* - * We used up one time slice. - */ - if (--ts->ts_slice > 0) - return; - /* * We're out of time, force a requeue at userret(). */ ts->ts_slice = sched_slice; and refusal to use options FULL_PREEMPTION But no one has unsubscribed to my letter, my patch helps or not in the case of Core2Duo... There is a suspicion that the problems stem from the sections of code associated with the SMP... Maybe I'm in something wrong, but I want to help in solving this problem ... From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 09:13:26 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C93B1065673; Tue, 13 Dec 2011 09:13:26 +0000 (UTC) (envelope-from ache@vniz.net) Received: from vniz.net (vniz.net [194.87.13.69]) by mx1.freebsd.org (Postfix) with ESMTP id 957758FC14; Tue, 13 Dec 2011 09:13:25 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by vniz.net (8.14.5/8.14.5) with ESMTP id pBD90rxo003476; Tue, 13 Dec 2011 13:00:53 +0400 (MSK) (envelope-from ache@vniz.net) Received: (from ache@localhost) by localhost (8.14.5/8.14.5/Submit) id pBD90qqf003475; Tue, 13 Dec 2011 13:00:52 +0400 (MSK) (envelope-from ache) Date: Tue, 13 Dec 2011 13:00:51 +0400 From: Andrey Chernov To: Ivan Klymenko Message-ID: <20111213090051.GA3339@vniz.net> Mail-Followup-To: Andrey Chernov , Ivan Klymenko , Doug Barton , "O. Hartmann" , Current FreeBSD , freebsd-stable@FreeBSD.ORG, freebsd-performance@FreeBSD.ORG References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111213104048.40f3e3de@nonamehost.> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@FreeBSD.ORG, "O. Hartmann" , Doug Barton , Current FreeBSD , freebsd-performance@FreeBSD.ORG Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 09:13:26 -0000 On Tue, Dec 13, 2011 at 10:40:48AM +0200, Ivan Klymenko wrote: > > On 12/12/2011 05:47, O. Hartmann wrote: > > > Do we have any proof at hand for such cases where SCHED_ULE performs > > > much better than SCHED_4BSD? > > > > I complained about poor interactive performance of ULE in a desktop > > environment for years. I had numerous people try to help, including > > Jeff, with various tunables, dtrace'ing, etc. The cause of the problem > > was never found. > > > > I switched to 4BSD, problem gone. > > > > This is on 2 separate systems with core 2 duos. > > > > > > hth, > > > > Doug > > > > If the algorithm ULE does not contain problems - it means the problem > has Core2Duo, or in a piece of code that uses the ULE scheduler. I observe ULE interactivity slowness even on single core machine (Pentium 4) in very visible places, like 'ps ax' output stucks in the middle by ~1 second. When I switch back to SHED_4BSD, all slowness is gone. -- http://ache.vniz.net/ From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 10:22:49 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A121F1065672; Tue, 13 Dec 2011 10:22:49 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0A0828FC17; Tue, 13 Dec 2011 10:22:48 +0000 (UTC) Received: by vcbfk1 with SMTP id fk1so7670449vcb.13 for ; Tue, 13 Dec 2011 02:22:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=ZX6KqOeJ20evUcl6Cqd6lqgE4dxILwyIbQ+sbvlpRng=; b=b+if3omSbxdzMRidkCt2TljuSzNNZXoCMYHDOhRUqjArbxZChUgwSBPgUDyAXayK3l vm2mZh0iQVBnmIV7x3GBxWJIcbRmuvH4N/a12Bcy+nSeYAIL2Rf56pEYgNlSX71d7p8E rjngzQjC5WO5c3vdJWVU8Hlj4onsk3Kb7JUOE= MIME-Version: 1.0 Received: by 10.220.230.67 with SMTP id jl3mr1004355vcb.60.1323771768334; Tue, 13 Dec 2011 02:22:48 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.52.26.50 with HTTP; Tue, 13 Dec 2011 02:22:48 -0800 (PST) In-Reply-To: <20111213090051.GA3339@vniz.net> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213090051.GA3339@vniz.net> Date: Tue, 13 Dec 2011 02:22:48 -0800 X-Google-Sender-Auth: Z0I2tEWuW7JTpBYJ2eEndqbOV5U Message-ID: From: Adrian Chadd To: Andrey Chernov , Ivan Klymenko , Doug Barton , "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 10:22:49 -0000 On 13 December 2011 01:00, Andrey Chernov wrote: >> If the algorithm ULE does not contain problems - it means the problem >> has Core2Duo, or in a piece of code that uses the ULE scheduler. > > I observe ULE interactivity slowness even on single core machine (Pentium > 4) in very visible places, like 'ps ax' output stucks in the middle by ~1 > second. When I switch back to SHED_4BSD, all slowness is gone. Are you able to provide KTR traces of the scheduler results? Something that can be fed to schedgraph? Adrian From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 11:04:42 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B5641065672 for ; Tue, 13 Dec 2011 11:04:42 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (ip-3-2-0-2.r20.asbnva02.us.ce.gin.ntt.net [IPv6:2001:418:0:5000::16]) by mx1.freebsd.org (Postfix) with ESMTP id 11AE18FC19 for ; Tue, 13 Dec 2011 11:04:41 +0000 (UTC) Received: from wonderland.m5p.com (wonderland.m5p.com [IPv6:2001:418:3fd::19]) by mailhost.m5p.com (8.14.4/8.14.4) with ESMTP id pBDB4ZAC095106 for ; Tue, 13 Dec 2011 06:04:41 -0500 (EST) (envelope-from george+freebsd@m5p.com) Message-ID: <4EE73143.5090100@m5p.com> Date: Tue, 13 Dec 2011 06:04:35 -0500 From: George Mitchell User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111127 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE6295B.3020308@cran.org.uk> In-Reply-To: <4EE6295B.3020308@cran.org.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (mailhost.m5p.com [IPv6:2001:418:3fd::f7]); Tue, 13 Dec 2011 06:04:41 -0500 (EST) X-Scanned-By: MIMEDefang 2.72 on IPv6:2001:418:3fd::f7 Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 11:04:42 -0000 On 12/12/11 11:18, Bruce Cran wrote: > On 12/12/2011 15:51, Steve Kargl wrote: >> This comes up every 9 months or so, and must be approaching FAQ >> status. In a HPC environment, I recommend 4BSD. Depending on the >> workload, ULE can cause a severe increase in turn around time when >> doing already long computations. If you have an MPI application, >> simply launching greater than ncpu+1 jobs can show the problem. PS: >> search the list archives for "kargl and ULE". > > This isn't something that can be fixed by tuning ULE? For example for > desktop applications kern.sched.preempt_thresh should be set to 224 from > its default. I'm wondering if the installer should ask people what the > typical use will be, and tune the scheduler appropriately. > I tried my "make buildkernel" test with "dnetc" running after setting kern.sched.preempt_thresh set to 224. It did far worse than before, getting only as far as compiling bxe overnight (compared to getting to netgragh with the default kern.sched.preempt_thresh setting). -- George Mitchell From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 11:13:44 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 505F9106564A; Tue, 13 Dec 2011 11:13:44 +0000 (UTC) (envelope-from ohartman@mail.zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 057EE8FC12; Tue, 13 Dec 2011 11:13:43 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RaQIo-0001nB-Vq>; Tue, 13 Dec 2011 12:13:43 +0100 Received: from telesto.geoinf.fu-berlin.de ([130.133.86.198]) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RaQIo-0006uN-Sb>; Tue, 13 Dec 2011 12:13:42 +0100 Message-ID: <4EE73366.7080304@mail.zedat.fu-berlin.de> Date: Tue, 13 Dec 2011 12:13:42 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Vincent Hoffman References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE619FC.4000601@unsane.co.uk> In-Reply-To: <4EE619FC.4000601@unsane.co.uk> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig801D5BA68A4D4252A6C8F2B4" X-Originating-IP: 130.133.86.198 Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 11:13:44 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig801D5BA68A4D4252A6C8F2B4 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 12/12/11 16:13, Vincent Hoffman wrote: >=20 > On 12/12/2011 13:47, O. Hartmann wrote: >=20 >>> Not fully right, boinc defaults to run on idprio 31 so this isn't an >>> issue. And yes, there are cases where SCHED_ULE shows much better >>> performance then SCHED_4BSD. [...] >=20 >> Do we have any proof at hand for such cases where SCHED_ULE performs >> much better than SCHED_4BSD? Whenever the subject comes up, it is >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu = > >> 2. But in the end I see here contradictionary statements. People >> complain about poor performance (especially in scientific environments= ), >> and other give contra not being the case. > It all a little old now but some if the stuff in > http://people.freebsd.org/~kris/scaling/ > covers improvements that were seen. >=20 > http://jeffr-tech.livejournal.com/5705.html > shows a little too, reading though Jeffs blog is worth it as it has som= e > interesting stuff on SHED_ULE. >=20 > I thought there were some more benchmarks floating round but cant find > any with a quick google. >=20 >=20 > Vince >=20 >=20 Interesting, there seems to be a much more performant scheduler in 7.0, called SCHED_SMP. I have some faint recalls on that ... where is this beast gone? Oliver --------------enig801D5BA68A4D4252A6C8F2B4 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iF4EAREIAAYFAk7nM2YACgkQU6Ni+wtCKv+SoQD9E1daXYU8i3DtYikG3KoKXf3b J+ujUpCBkPNh4fs1RHUA/RkDAdKThLx4xcV7WgblHwEkkZgyLAaAEbfOz2S/s94I =TMYp -----END PGP SIGNATURE----- --------------enig801D5BA68A4D4252A6C8F2B4-- From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 12:47:10 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 76FED106564A for ; Tue, 13 Dec 2011 12:47:10 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta01.westchester.pa.mail.comcast.net (qmta01.westchester.pa.mail.comcast.net [76.96.62.16]) by mx1.freebsd.org (Postfix) with ESMTP id 1FEF78FC13 for ; Tue, 13 Dec 2011 12:47:09 +0000 (UTC) Received: from omta05.westchester.pa.mail.comcast.net ([76.96.62.43]) by qmta01.westchester.pa.mail.comcast.net with comcast id 8cn81i0010vyq2s51cnADx; Tue, 13 Dec 2011 12:47:10 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta05.westchester.pa.mail.comcast.net with comcast id 8cn81i0191t3BNj3Rcn9GG; Tue, 13 Dec 2011 12:47:10 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 56D23102C19; Tue, 13 Dec 2011 04:47:07 -0800 (PST) Date: Tue, 13 Dec 2011 04:47:07 -0800 From: Jeremy Chadwick To: "O. Hartmann" Message-ID: <20111213124707.GA75440@icarus.home.lan> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE619FC.4000601@unsane.co.uk> <4EE73366.7080304@mail.zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE73366.7080304@mail.zedat.fu-berlin.de> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org, Vincent Hoffman Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 12:47:10 -0000 On Tue, Dec 13, 2011 at 12:13:42PM +0100, O. Hartmann wrote: > On 12/12/11 16:13, Vincent Hoffman wrote: > > > > On 12/12/2011 13:47, O. Hartmann wrote: > > > >>> Not fully right, boinc defaults to run on idprio 31 so this isn't an > >>> issue. And yes, there are cases where SCHED_ULE shows much better > >>> performance then SCHED_4BSD. [...] > > > >> Do we have any proof at hand for such cases where SCHED_ULE performs > >> much better than SCHED_4BSD? Whenever the subject comes up, it is > >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > >> 2. But in the end I see here contradictionary statements. People > >> complain about poor performance (especially in scientific environments), > >> and other give contra not being the case. > > It all a little old now but some if the stuff in > > http://people.freebsd.org/~kris/scaling/ > > covers improvements that were seen. > > > > http://jeffr-tech.livejournal.com/5705.html > > shows a little too, reading though Jeffs blog is worth it as it has some > > interesting stuff on SHED_ULE. > > > > I thought there were some more benchmarks floating round but cant find > > any with a quick google. > > > > > > Vince > > > > > > Interesting, there seems to be a much more performant scheduler in 7.0, > called SCHED_SMP. I have some faint recalls on that ... where is this > beast gone? Boy I sure hope I remember this right. I strongly urge others to correct me where I'm wrong; thanks in advance! The classic scheduler, SCHED_4BSD, was implemented back before there was oxygen. sched_4bsd(4) mentions this. No need to discuss it. Jeff Robertson began working on the "first-generation ULE scheduler" during the days of FreeBSD 5.x (I believe 5.1), and a paper on it was presented at USENIX circa 2003: http://www.usenix.org/event/bsdcon03/tech/full_papers/roberson/roberson.pdf Over the following years, Jeff (and others I assume -- maybe folks like George Neville-Neil and/or Kirk McKusick?) adjusted and tinkered with some of the semantics and models/methods. If I remember right, some of these quirks/fixes were committed. All of this was happening under the scheduler that was then called SCHED_ULE, but it was "ULE 1.0" for lack of better terminology. This scheduler did not perform well, if I remember right, and Jeff was quite honest about that. From this point forward, Jeff began idealising and working on a scheduler which he called SCHED_SMP -- think of it as "ULE 2.0", again, for lack of better terminology. It was different than the existing SCHED_ULE scheduler, hence a different name. Jeff blogged about this in early 2007, using exactly that term ("ULE 2.0"): http://jeffr-tech.livejournal.com/3729.html In mid-2007, prior to FreeBSD 7.0-RELEASE, Jeff announced that effectively he wanted to make SCHED_ULE do what SCHED_SMP did, and provided a patch to SCHED_ULE to accomplish just that: http://unix.derkeiler.com/Mailing-Lists/FreeBSD/current/2007-07/msg00755.html Full thread is here (beware -- many replies): http://unix.derkeiler.com/Mailing-Lists/FreeBSD/current/2007-07/threads.html#00755 The patch mentioned above was merged into HEAD on 2007/07/19. http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/sched_ule.c#rev1.202 So in effect, as of 2007/07/19, SCHED_ULE became SCHED_SMP. FreeBSD 7.0-RELEASE was released on 2008/02/27, and the above commit/changes were available at that time as well (meaning: RELENG_7 and RELENG_7_0 at that moment in time should have included the patch from the above paragraph). The document released by Kris Kenneway hinted at those changes and performance improvements: http://people.freebsd.org/~kris/scaling/7.0%20Preview.pdf Keep in mind, however, that at that time kernel configuration files (GENERIC, etc.) still defaulted to SCHED_4BSD. The default scheduler in kernel config files (GENERIC, etc.) for i386 and amd64 (not sure about others) was changed in 2007/10/19: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/i386/conf/GENERIC#rev1.475 http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/amd64/conf/GENERIC#rev1.485 This was done *prior* to FreeBSD 7.1-RELEASE. So, it first became available as the default scheduler "for the masses" when 7.1-RELEASE came out on 2009/01/05. "All of the answers", in a roundabout and non-user-friendly way, are available by examining the commit history for src/sys/kern/sched_ule.c. It's hard to follow especially given that you have to consider all the releases/branchpoints that took place over time, but: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/sched_ule.c Are we having fun yet? :-) -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 13:23:53 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BFC161065673; Tue, 13 Dec 2011 13:23:53 +0000 (UTC) (envelope-from ohartman@mail.zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 725158FC1F; Tue, 13 Dec 2011 13:23:53 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RaSKm-0006OK-Av>; Tue, 13 Dec 2011 14:23:52 +0100 Received: from telesto.geoinf.fu-berlin.de ([130.133.86.198]) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RaSKm-0007tJ-7i>; Tue, 13 Dec 2011 14:23:52 +0100 Message-ID: <4EE751E2.60204@mail.zedat.fu-berlin.de> Date: Tue, 13 Dec 2011 14:23:46 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Steve Kargl References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> In-Reply-To: <20111212155159.GB73597@troutmask.apl.washington.edu> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig1B23B528A575FB784F3E4BE3" X-Originating-IP: 130.133.86.198 Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 13:23:54 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig1B23B528A575FB784F3E4BE3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 12/12/11 16:51, Steve Kargl wrote: > On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: >> >>> Not fully right, boinc defaults to run on idprio 31 so this isn't an >>> issue. And yes, there are cases where SCHED_ULE shows much better >>> performance then SCHED_4BSD. [...] >> >> Do we have any proof at hand for such cases where SCHED_ULE performs >> much better than SCHED_4BSD? Whenever the subject comes up, it is >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu = > >> 2. But in the end I see here contradictionary statements. People >> complain about poor performance (especially in scientific environments= ), >> and other give contra not being the case. >> >> Within our department, we developed a highly scalable code for planeta= ry >> science purposes on imagery. It utilizes present GPUs via OpenCL if >> present. Otherwise it grabs as many cores as it can. >> By the end of this year I'll get a new desktop box based on Intels new= >> Sandy Bridge-E architecture with plenty of memory. If the colleague wh= o >> developed the code is willing performing some benchmarks on the same >> hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most >> recent Suse. For FreeBSD I intent also to look for performance with bo= th >> different schedulers available. >> >=20 > This comes up every 9 months or so, and must be approaching > FAQ status. >=20 > In a HPC environment, I recommend 4BSD. Depending on > the workload, ULE can cause a severe increase in turn > around time when doing already long computations. If > you have an MPI application, simply launching greater > than ncpu+1 jobs can show the problem. Well, those recommendations should based on "WHY". As the mostly negative experiences with SCHED_ULE in highly computative workloads get allways contradicted by "...but there are workloads that show the opposite ..." this should be shown by more recent benchmarks and explanations than legacy benchmarks from years ago. And, indeed, I highly would recommend having a FAQ or a short note in "tuning" or the handbook in which it is mentioned to use SCHED_4BSD in HPC environments and SCHED_ULE for other workloads (which has to be more specific). It is not an easy task setting up a certain kind of OS for a specific purpose and tuning by crawling the mailing lists. Some notes and hints in the documentation is always a valuable hint and highly appreciated by folks not deep into development. And by the way, I have the deep impression that most of these discussions about the poor performance of SCHED_ULE tend to always end up in a covering up that flaw and the conclusive waste of development. But this is only my personal impression. --------------enig1B23B528A575FB784F3E4BE3 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iF4EAREIAAYFAk7nUegACgkQU6Ni+wtCKv9RzAD9GhKwCryupmWNA64Wp5N7rMzk +WcXFTQXR19FqiE3OkMA/3PRpLaK7SJcAh3PiyujWMGBt1smTuui80KUWTatd5+5 =SLRk -----END PGP SIGNATURE----- --------------enig1B23B528A575FB784F3E4BE3-- From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 15:54:57 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B0C9106564A; Tue, 13 Dec 2011 15:54:57 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) by mx1.freebsd.org (Postfix) with ESMTP id BD1178FC12; Tue, 13 Dec 2011 15:54:56 +0000 (UTC) Received: from troutmask.apl.washington.edu (localhost.apl.washington.edu [127.0.0.1]) by troutmask.apl.washington.edu (8.14.5/8.14.5) with ESMTP id pBDFsuaL093141; Tue, 13 Dec 2011 07:54:56 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.14.5/8.14.5/Submit) id pBDFsuJ2093140; Tue, 13 Dec 2011 07:54:56 -0800 (PST) (envelope-from sgk) Date: Tue, 13 Dec 2011 07:54:56 -0800 From: Steve Kargl To: "O. Hartmann" Message-ID: <20111213155456.GA93017@troutmask.apl.washington.edu> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE751E2.60204@mail.zedat.fu-berlin.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE751E2.60204@mail.zedat.fu-berlin.de> User-Agent: Mutt/1.4.2.3i Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 15:54:57 -0000 On Tue, Dec 13, 2011 at 02:23:46PM +0100, O. Hartmann wrote: > On 12/12/11 16:51, Steve Kargl wrote: > > On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: > >> > >>> Not fully right, boinc defaults to run on idprio 31 so this isn't an > >>> issue. And yes, there are cases where SCHED_ULE shows much better > >>> performance then SCHED_4BSD. [...] > >> > >> Do we have any proof at hand for such cases where SCHED_ULE performs > >> much better than SCHED_4BSD? Whenever the subject comes up, it is > >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > >> 2. But in the end I see here contradictionary statements. People > >> complain about poor performance (especially in scientific environments), > >> and other give contra not being the case. > >> > >> Within our department, we developed a highly scalable code for planetary > >> science purposes on imagery. It utilizes present GPUs via OpenCL if > >> present. Otherwise it grabs as many cores as it can. > >> By the end of this year I'll get a new desktop box based on Intels new > >> Sandy Bridge-E architecture with plenty of memory. If the colleague who > >> developed the code is willing performing some benchmarks on the same > >> hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > >> recent Suse. For FreeBSD I intent also to look for performance with both > >> different schedulers available. > >> > > > > This comes up every 9 months or so, and must be approaching > > FAQ status. > > > > In a HPC environment, I recommend 4BSD. Depending on > > the workload, ULE can cause a severe increase in turn > > around time when doing already long computations. If > > you have an MPI application, simply launching greater > > than ncpu+1 jobs can show the problem. > > Well, those recommendations should based on "WHY". As the mostly > negative experiences with SCHED_ULE in highly computative workloads get > allways contradicted by "...but there are workloads that show the > opposite ..." this should be shown by more recent benchmarks and > explanations than legacy benchmarks from years ago. > I have given the WHY in previous discussions of ULE, based on what you call legacy benchmarks. I have not seen any commit to sched_ule.c that would lead me to believe that the performance issues with ULE and cpu-bound numerical codes have been addressed. Repeating the benchmark would be a waste of time. -- Steve From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 19:34:49 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC10F106566B for ; Tue, 13 Dec 2011 19:34:49 +0000 (UTC) (envelope-from minimarmot@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6C93D8FC1A for ; Tue, 13 Dec 2011 19:34:49 +0000 (UTC) Received: by ggnp1 with SMTP id p1so14208ggn.13 for ; Tue, 13 Dec 2011 11:34:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=qv2hMCnHLv4OWT/1XEHEvXIxXn6D5sQoOqoLL6X952Y=; b=g8L2khxMlP9yGo62O+YjPd1uUIETeq0xjFugXhyBOjbq0mtDKShQKxONR8nxLze8O8 9fCe0ER9VsKuCnHtThGz9KnYPO8E+/O8ka482UtzFkkHPR1f1ph0AZ+z6lyS+XSHoll1 h4qFdyUx6ZmgpEp06BP6qm+oAwkNvN8ki37IY= MIME-Version: 1.0 Received: by 10.182.222.106 with SMTP id ql10mr4579433obc.53.1323804888676; Tue, 13 Dec 2011 11:34:48 -0800 (PST) Received: by 10.182.50.169 with HTTP; Tue, 13 Dec 2011 11:34:48 -0800 (PST) In-Reply-To: <201112120455.VAA03028@lariat.net> References: <201112090913.CAA03333@lariat.net> <201112120455.VAA03028@lariat.net> Date: Tue, 13 Dec 2011 14:34:48 -0500 Message-ID: From: Ben Kaduk To: Brett Glass , Ed Schouten Content-Type: text/plain; charset=ISO-8859-1 Cc: stable@freebsd.org Subject: Re: Two problems still present in RC3 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 19:34:49 -0000 On 12/11/11, Brett Glass wrote: > At 11:36 AM 12/11/2011, Ben Kaduk wrote: > >>Did you take the change to /etc/ttys going from cons25 to xterm 'type'? > > I didn't have to change it; it was that way when the OS was installed. > You were sending to -stable, so I wasn't sure. Thanks for confirming. > Problem seems to be that the behavior (specifically, reverse video on the > 25th line) doesn't quite match the xterm termcap entry. If I remember correctly, your original message mentioned seeing this issue in emacs; have you tried reproducing it in a simpler test case? Maybe something like: (for i in $(jot 25); do echo; tput mr; echo -n "reverse"; tput me; echo -n " regular"; done) && sleep 10000 That would presumably help narrow down the issue. -Ben Kaduk From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 20:23:34 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 220641065754 for ; Tue, 13 Dec 2011 20:23:34 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: from mx0.hoeg.nl (mx0.hoeg.nl [IPv6:2a01:4f8:101:5343::aa]) by mx1.freebsd.org (Postfix) with ESMTP id DB5508FC08 for ; Tue, 13 Dec 2011 20:23:33 +0000 (UTC) Received: by mx0.hoeg.nl (Postfix, from userid 1000) id 4C2662A28CD6; Tue, 13 Dec 2011 21:23:33 +0100 (CET) Date: Tue, 13 Dec 2011 21:23:33 +0100 From: Ed Schouten To: Ben Kaduk Message-ID: <20111213202333.GH1771@hoeg.nl> References: <201112090913.CAA03333@lariat.net> <201112120455.VAA03028@lariat.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="J/zg8ciPNcraoWb6" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Brett Glass , stable@freebsd.org Subject: Re: Two problems still present in RC3 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 20:23:34 -0000 --J/zg8ciPNcraoWb6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable * Ben Kaduk , 20111213 20:34: > > Problem seems to be that the behavior (specifically, reverse video on t= he > > 25th line) doesn't quite match the xterm termcap entry. >=20 > If I remember correctly, your original message mentioned seeing this > issue in emacs; have you tried reproducing it in a simpler test case? Well, I don't care whether the test case is small or large, as long as I know how to reproduce it. I have never used jove before, so what kind of buttons should I press to get the artifacts? Does it only happen on the console, or also when using a regular xterm? Thanks, --=20 Ed Schouten WWW: http://80386.nl/ --J/zg8ciPNcraoWb6 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQIcBAEBAgAGBQJO57RFAAoJEG5e2P40kaK7I0oP/Rl52Tg26RD34PS3hdNatUjM Xk0ogmyE5UxMQ2lDDY5vpgake23f/8ZGbwgaOLZ9kzS+DnWTxCAgKBIY2l/jMyCy vQ6UkCdBMUA0kgV1zFB2v2NZsrIoCSFeJvJBMRsSEDAjenf01hXR8BiHXsq5tsZM +JywkHLs+JXU8Q3X5YGfE3j/UVJPdup1lYPDwOuK2bdhxaW3YtAjRbLPrRiqpdkD EgN3mSFBKwA4iCFvrMW5X+SgjEc5WVzRzgRTDQeJztHn3bwSJJQ5sF6RrxoznT8V pWitzkkzSkz73DuIo1BfMjnf+Abd7hyyCf4CtwSIZVnLw5gH66TktVmpkfjQXkzB kwMQUb0l7N+2JZ0fPgeNj4ANugAD/9z9R974Y9fDi0DUydnBCsLvTVsaARqklDyI x4cWELk349EmvE6ftsgZYq4xxK1Sfm/B5urMZguBQmHh5PgDz6CM9morPuBCuXSW BTEC6oLHG3e1L0Vr5lL3YINPXOtTIkJOK30TYn1A5r0sYPEYkQSvt2fu1D59i9JB 5qgYdTfuya3N2L6ZiJBe67vvcecBU8MYiATRdcVkBvoEM7NSt/YwaSDWbZaYKYEC L9c1psvGK9lCdb1uH84wmI3bY9cnMnBnAYL5mTzVvW+KtEG4swHyDePm+r8PhQd6 sVxFjzrXLUXnWu73P6No =57YA -----END PGP SIGNATURE----- --J/zg8ciPNcraoWb6-- From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 20:39:36 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C2802106566B for ; Tue, 13 Dec 2011 20:39:36 +0000 (UTC) (envelope-from mandrews@bit0.com) Received: from magnum.bit0.com (magnum.bit0.com [IPv6:2604:e700:b0:1::200]) by mx1.freebsd.org (Postfix) with ESMTP id 89B1C8FC0C for ; Tue, 13 Dec 2011 20:39:36 +0000 (UTC) Received: from magnum.int.bit0.com (localhost [127.0.0.1]) by magnum.bit0.com (Postfix) with ESMTP id 1826F189D4 for ; Tue, 13 Dec 2011 15:39:36 -0500 (EST) X-Virus-Scanned: amavisd-new at bit0.com Received: from magnum.bit0.com ([127.0.0.1]) by magnum.int.bit0.com (magnum.int.bit0.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NkK9KKZb7xnf for ; Tue, 13 Dec 2011 15:39:33 -0500 (EST) Received: from beast.int.bit0.com (beast.int.bit0.com [172.27.0.2]) by magnum.bit0.com (Postfix) with ESMTP for ; Tue, 13 Dec 2011 15:39:33 -0500 (EST) Date: Tue, 13 Dec 2011 15:39:33 -0500 (EST) From: Mike Andrews X-X-Sender: mandrews@beast.int.bit0.com To: freebsd-stable@freebsd.org In-Reply-To: <4EE02643.8050502@bit0.com> Message-ID: References: <4ED807D9.7080708@bit0.com> <4EDD8047.6030606@bit0.com> <4EE02643.8050502@bit0.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: Re: Sporadic 9.0-RC2 boot-time panic X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 20:39:36 -0000 On Wed, 7 Dec 2011, Mike Andrews wrote: > On 12/5/11 9:39 PM, Mike Andrews wrote: >> On 12/1/2011 6:03 PM, Mike Andrews wrote: >>> On 11/28/11 5:48 PM, Ronald Klop wrote: >>>> On Mon, 28 Nov 2011 23:37:27 +0100, Mike Andrews >>>> wrote: >>>> >>>>> *Sometimes* when booting 9.0-RC2 on *some* of my machines, I'll get >>>>> one of the following two panics during multiuser startup, usually >>>>> while running the /usr/local/etc/rc.d scripts. (The instruction >>>>> pointer is always exactly one of these two, and they look fairly >>>>> related.) If after two or three reboots it manages to not panic, the >>>>> system will run perfectly stable. >>> >> FYI this is still happening on 9.0-RC3 -- r228247 to be precise. >> >> It only seems to be happening on one particular model of motherboard >> (Supermicro X8STi-F) but it is happening on several identical machines with >> them -- running on several other (mostly Supermicro) boards is just fine, >> including at least one with the exact same 82574L NICs. >> >> Whoever's wanting to work on this, contact me off-list to get some more up >> to date console logs and the kernel config. > > ...or just look at the newly opened kern/163117 PR. ... *crickets* ...or wait and hope r228386 and r228387 fix it, though I doubt they'll get merged before release. I will see about finding test hardware that I can run -current on. From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 21:25:31 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F38F106564A; Tue, 13 Dec 2011 21:25:31 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id 2E5C58FC08; Tue, 13 Dec 2011 21:25:31 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.5/8.14.4) with ESMTP id pBDLPTtq038329; Tue, 13 Dec 2011 16:25:29 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4EE7C2BE.9020509@sentex.net> Date: Tue, 13 Dec 2011 16:25:18 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE751E2.60204@mail.zedat.fu-berlin.de> <20111213155456.GA93017@troutmask.apl.washington.edu> In-Reply-To: <20111213155456.GA93017@troutmask.apl.washington.edu> X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.71 on IPv6:2607:f3e0:0:1::12 Cc: Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 21:25:31 -0000 On 12/13/2011 10:54 AM, Steve Kargl wrote: > > I have given the WHY in previous discussions of ULE, based > on what you call legacy benchmarks. I have not seen any > commit to sched_ule.c that would lead me to believe that > the performance issues with ULE and cpu-bound numerical > codes have been addressed. Repeating the benchmark would > be a waste of time. Trying a simple pbzip2 on a large file, the results are pretty consistent through iterations. pbzip2 with 4BSD is barely faster on a file thats 322MB in size. after a reboot, I did a strings bigfile > /dev/null then ran pbzip2 -v xaa -c > /dev/null 7 times If I do a burnP6 in the background, they perform about the same. (from sysutils/cpuburn) eg pbzip2 -v xaa -c > /dev/null Parallel BZIP2 v1.1.6 - by: Jeff Gilchrist [http://compression.ca] [Oct. 30, 2011] (uses libbzip2 by Julian Seward) Major contributions: Yavor Nikolov # CPUs: 4 BWT Block Size: 900 KB File Block Size: 900 KB Maximum Memory: 100 MB ------------------------------------------- File #: 1 of 1 Input Name: xaa Output Name: Input Size: 352404831 bytes Compressing data... Output Size: 50630745 bytes ------------------------------------------- Wall Clock: 18.139342 seconds ULE 18.113204 18.116896 18.123400 18.105894 18.163332 18.139342 18.082888 ULE with burnP6 23.076085 22.003666 21.162987 21.682445 21.935568 23.595781 21.601277 4BSD 17.983395 17.986218 18.009254 18.004312 18.001494 17.997032 4BSD with burnP6 22.215508 21.886459 21.595179 21.361830 21.325351 21.244793 # ministat uleP6 bsdP6 x uleP6 + bsdP6 +------------------------------------------------------------------------------------------------------------------------------------------+ |x + + + + x + x x + x x| | |____|______________MA____________________|M_____________A__________________________________________________| | +------------------------------------------------------------------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 6 21.162987 23.595781 22.003666 22.242755 0.91175566 + 6 21.244793 22.215508 21.595179 21.604853 0.3792413 No difference proven at 95.0% confidence x ule + bsd +------------------------------------------------------------------------------------------------------------------------------------------+ |+ + + + + + x x x x x x x| | |______A___M___| |________________M__A__________________| | +------------------------------------------------------------------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 7 18.082888 18.163332 18.116896 18.120708 0.025468695 + 6 17.983395 18.009254 18.001494 17.996951 0.010248473 Difference at 95.0% confidence -0.123757 +/- 0.024538 -0.68296% +/- 0.135414% (Student's t, pooled s = 0.0200388) hardware is X3450 with 8G of memory. RELENG8 ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 21:33:20 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10A0E1065670 for ; Tue, 13 Dec 2011 21:33:20 +0000 (UTC) (envelope-from andrey@zonov.org) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id A3DF98FC0C for ; Tue, 13 Dec 2011 21:33:19 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so204309wgb.31 for ; Tue, 13 Dec 2011 13:33:18 -0800 (PST) Received: by 10.227.198.142 with SMTP id eo14mr80010wbb.28.1323810681943; Tue, 13 Dec 2011 13:11:21 -0800 (PST) Received: from [10.254.254.77] ([95.165.122.0]) by mx.google.com with ESMTPS id fw16sm435458wbb.13.2011.12.13.13.11.20 (version=SSLv3 cipher=OTHER); Tue, 13 Dec 2011 13:11:21 -0800 (PST) Message-ID: <4EE7BF77.5000504@zonov.org> Date: Wed, 14 Dec 2011 01:11:19 +0400 From: Andrey Zonov User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.24) Gecko/20100228 Thunderbird/2.0.0.24 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: directory listing hangs in "ufs" state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 21:33:20 -0000 Hi, I've got STABLE-8 (r221983) with mongodb-1.8.1 installed on it. A couple days ago I observed that listing of mongodb directory stuck in a few minutes in "ufs" state. I've run it again with ktrace and got following (kdump -R): 91324 ls 0.000003 CALL lstat(0x32c199c8,0x32c19950) 91324 ls 0.000003 NAMI "base.1" 91324 ls 21.357255 STRU struct stat {dev=116, ino=45125633, mode=-rw------- , nlink=1, uid=922, gid=922, rdev=180226648, atime=1323709877, stime=1323776461, ctime=1323776461, birthtime=1314798592, size=134217728, blksize=16384, blocks=262304, flags=0x0 } 91324 ls 0.000014 RET lstat 0 kgdb backtrace of this process was looked like this: Thread 297 (Thread 100372): #0 sched_switch (td=0xffffff0095c008c0, newtd=0xffffff000357b8c0, flags=) at /usr/src/sys/kern/sched_ule.c:1866 #1 0xffffffff80406696 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:449 #2 0xffffffff8043c072 in sleepq_wait (wchan=0xffffff0103aaf7f8, pri=80) at /usr/src/sys/kern/subr_sleepqueue.c:609 #3 0xffffffff803e4a5a in __lockmgr_args (lk=0xffffff0103aaf7f8, flags=2097408, ilk=0xffffff0103aaf820, wmesg=) at /usr/src/sys/kern/kern_lock.c:220 #5 0xffffffff8061239c in ffs_lock (ap=0xffffff84867fc550) at lockmgr.h:94 #5 0xffffffff806d2462 in VOP_LOCK1_APV (vop=0xffffffff80921fe0, a=0xffffff84867fc550) at vnode_if.c:1988 #6 0xffffffff804a58b7 in _vn_lock (vp=0xffffff0103aaf760, flags=2097152, file=0xffffffff80736e70 "/usr/src/sys/kern/vfs_subr.c", line=2137) at vnode_if.h:859 #7 0xffffffff80498bc0 in vget (vp=0xffffff0103aaf760, flags=2097408, td=0xffffff0095c008c0) at /usr/src/sys/kern/vfs_subr.c:2137 #8 0xffffffff804845f4 in cache_lookup (dvp=0xffffff0095675b10, vpp=0xffffff84867fc910, cnp=0xffffff84867fc938) at /usr/src/sys/kern/vfs_cache.c:587 #9 0xffffffff80484a30 in vfs_cache_lookup (ap=) at /usr/src/sys/kern/vfs_cache.c:905 #10 0xffffffff806d2e7c in VOP_LOOKUP_APV (vop=0xffffffff80922820, a=0xffffff84867fc790) at vnode_if.c:123 #11 0xffffffff8048bc80 in lookup (ndp=0xffffff84867fc8e0) at vnode_if.h:54 #12 0xffffffff8048cf0e in namei (ndp=0xffffff84867fc8e0) at /usr/src/sys/kern/vfs_lookup.c:269 #13 0xffffffff8049c972 in kern_statat_vnhook (td=0xffffff0095c008c0, flag=) at /usr/src/sys/kern/vfs_syscalls.c:2346 #14 0xffffffff8049cbb5 in kern_statat (td=) at /usr/src/sys/kern/vfs_syscalls.c:2327 #15 0xffffffff8049cc7a in lstat (td=) at /usr/src/sys/kern/vfs_syscalls.c:2390 #16 0xffffffff8043e7dd in syscallenter (td=0xffffff0095c008c0, sa=0xffffff84867fcbb0) at /usr/src/sys/kern/subr_trap.c:326 #17 0xffffffff8066a5eb in syscall (frame=0xffffff84867fcc50) at /usr/src/sys/amd64/amd64/trap.c:916 #18 0xffffffff806517f2 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:384 #19 0x000000003298f75c in ?? () The very first idea was to turn off name caching (set debug.vfscache to 0), but it didn't help. The second idea was to reboot, but it didn't help too. This directory locks fine. It has 10 files and 1 empty directory. Have you any ideas what is going on? or how to catch the problem? -- Andrey Zonov From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 21:52:35 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 2675C1065670 for ; Tue, 13 Dec 2011 21:52:35 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 207FA14D93A; Tue, 13 Dec 2011 21:52:33 +0000 (UTC) Message-ID: <4EE7C920.3010109@FreeBSD.org> Date: Tue, 13 Dec 2011 13:52:32 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@FreeBSD.org References: <4EE29D3F.2030009@gridfury.com> In-Reply-To: <4EE29D3F.2030009@gridfury.com> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 X-Forwarded-Message-Id: <4EE29D3F.2030009@gridfury.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: kernel: negative sbsize for uid = 0 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 21:52:35 -0000 I'm running 8.2-RELEASE-p4 i386 on some web servers that are generally lightly-moderately loaded, but occasionally see some heavy spikes where load average goes way up. When that is happening, but sometimes even when it's not, I get hundreds of this message spewing into the logs: kernel: negative sbsize for uid = 0 I haven't found anything particularly useful by searching for that message, the one reference was to mbufs, but that seems not to be the problem. Here is the output of 'netstat -m' during one of the load spikes: 598/1712/2310 mbufs in use (current/cache/total) 559/1533/2092/32768 mbuf clusters in use (current/cache/total/max) 559/1105 mbuf+clusters out of packet secondary zone in use (current/cache) 0/528/528/16384 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/8192 9k jumbo clusters in use (current/cache/total/max) 0/0/0/4096 16k jumbo clusters in use (current/cache/total/max) 1267K/5606K/6873K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/2239/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 809790 requests for I/O initiated by sendfile 0 calls to protocol drain routines So is this message something to worry about? If so, how can I diagnose what's happening, and how do I fix it? Doug From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 21:58:23 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA1D9106566B; Tue, 13 Dec 2011 21:58:23 +0000 (UTC) (envelope-from malin.randstrom@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5CF768FC12; Tue, 13 Dec 2011 21:58:23 +0000 (UTC) Received: by ggnp1 with SMTP id p1so178082ggn.13 for ; Tue, 13 Dec 2011 13:58:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=sNUT/fjw6EchSejWXjJP78ubljFlvCF9/iUH/FOJoMw=; b=PfIJj08Hg3htPuqeML1U2v1eJXBM77qQyqpdPPq+VNEjSULs4zPKRI2Bu18wWL237o qEd1GrX4vdWLE4fWlX5YDR87QWLA8rFXDsoHM0bOJHPTeLEVpYw+5yK3zG3cQrDSn+OU AfZ94IGdq58983Kbm6A1Y3ZAN9v/1UPA4MGOw= MIME-Version: 1.0 Received: by 10.182.218.100 with SMTP id pf4mr4801602obc.12.1323811881013; Tue, 13 Dec 2011 13:31:21 -0800 (PST) Received: by 10.182.6.133 with HTTP; Tue, 13 Dec 2011 13:31:20 -0800 (PST) Received: by 10.182.6.133 with HTTP; Tue, 13 Dec 2011 13:31:20 -0800 (PST) In-Reply-To: <20111213155456.GA93017@troutmask.apl.washington.edu> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE751E2.60204@mail.zedat.fu-berlin.de> <20111213155456.GA93017@troutmask.apl.washington.edu> Date: Tue, 13 Dec 2011 22:31:20 +0100 Message-ID: From: Malin Randstrom To: Steve Kargl Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org, "O. Hartmann" Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: malin@randstrom.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 21:58:23 -0000 stop sending me spam mail ... you never stop despite me having unsubscribeb several times. stop this! On Dec 13, 2011 8:12 PM, "Steve Kargl" wrote: > On Tue, Dec 13, 2011 at 02:23:46PM +0100, O. Hartmann wrote: > > On 12/12/11 16:51, Steve Kargl wrote: > > > On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: > > >> > > >>> Not fully right, boinc defaults to run on idprio 31 so this isn't an > > >>> issue. And yes, there are cases where SCHED_ULE shows much better > > >>> performance then SCHED_4BSD. [...] > > >> > > >> Do we have any proof at hand for such cases where SCHED_ULE performs > > >> much better than SCHED_4BSD? Whenever the subject comes up, it is > > >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > > > >> 2. But in the end I see here contradictionary statements. People > > >> complain about poor performance (especially in scientific > environments), > > >> and other give contra not being the case. > > >> > > >> Within our department, we developed a highly scalable code for > planetary > > >> science purposes on imagery. It utilizes present GPUs via OpenCL if > > >> present. Otherwise it grabs as many cores as it can. > > >> By the end of this year I'll get a new desktop box based on Intels new > > >> Sandy Bridge-E architecture with plenty of memory. If the colleague > who > > >> developed the code is willing performing some benchmarks on the same > > >> hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > > >> recent Suse. For FreeBSD I intent also to look for performance with > both > > >> different schedulers available. > > >> > > > > > > This comes up every 9 months or so, and must be approaching > > > FAQ status. > > > > > > In a HPC environment, I recommend 4BSD. Depending on > > > the workload, ULE can cause a severe increase in turn > > > around time when doing already long computations. If > > > you have an MPI application, simply launching greater > > > than ncpu+1 jobs can show the problem. > > > > Well, those recommendations should based on "WHY". As the mostly > > negative experiences with SCHED_ULE in highly computative workloads get > > allways contradicted by "...but there are workloads that show the > > opposite ..." this should be shown by more recent benchmarks and > > explanations than legacy benchmarks from years ago. > > > > I have given the WHY in previous discussions of ULE, based > on what you call legacy benchmarks. I have not seen any > commit to sched_ule.c that would lead me to believe that > the performance issues with ULE and cpu-bound numerical > codes have been addressed. Repeating the benchmark would > be a waste of time. > > -- > Steve > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 22:02:45 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 3BFDF1065670; Tue, 13 Dec 2011 22:02:45 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id DFB4D14E18D; Tue, 13 Dec 2011 22:02:44 +0000 (UTC) Message-ID: <4EE7CB84.5090801@FreeBSD.org> Date: Tue, 13 Dec 2011 14:02:44 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Edwin Groothuis References: <4EDABADD.9000009@FreeBSD.org> <4EDBDD50.4000303@FreeBSD.org> In-Reply-To: <4EDBDD50.4000303@FreeBSD.org> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: edwin@FreeBSD.org, freebsd-stable@freebsd.org, Christian Weisgerber , Max Khon Subject: Re: 7-STABLE: mergemaster tzsetup question X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 22:02:45 -0000 On 12/04/2011 12:51, Doug Barton wrote: > On 12/3/2011 6:00 PM, Edwin Groothuis wrote: >> On 04/12/2011, at 11:12 , Doug Barton wrote: >>> On 12/3/2011 8:14 AM, Max Khon wrote: >>>> Christian, >>>> >>>> On Sat, Dec 3, 2011 at 10:24 PM, Christian Weisgerber >>>> wrote: >>>> >>>>> Every time I run mergemaster(8) on 7.4-STABLE, I'm now presented >>>>> with >>>>> >>>>> *** There is no /var/db/zoneinfo file to update /etc/localtime. >>>>> You should run tzsetup >>>>> >>>>> Running tzsetup(8) does however not create /var/db/zoneinfo, so >>>>> mergemaster will prompt the next time, too. I guess I can just >>>>> ignore it, but it seems weird that mergemaster would keep nagging >>>>> about this. >>>>> >>>>> Where is /var/db/zoneinfo supposed to come from? >>>>> >>>>> I also notice that mergemaster can issue tzsetup arguments -C and >>>>> -r, but tzsetup doesn't support those. >>>> >>>> tzsetup in FreeBSD 8 and later creates /var/db/zoneinfo. It seems that >>>> mergemaster was merged to RELENG_7 but appropiate version of tzsetup >>>> was not. >>> >>> Well that's embarrassing. :) >>> >>> Edwin, what are the chances that you could MFC your changes to tzsetup? >> >> If you still have a machine running 7.x (which I don't have anymore), then go for it and do your damage :-) > > Well I was kind of hoping you'd be responsible for doing your own work. > :) (You see, I'm kind of tired of doing other people's MFCs.) You can > use ref7.freebsd.org if you need a test platform. Edwin, I haven't seen a response from you on this, apologies if you sent one and I missed it. Can you let me know your plans? Doug -- [^L] Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 22:03:51 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 95AED1065679; Tue, 13 Dec 2011 22:03:51 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 391AD162704; Tue, 13 Dec 2011 22:03:51 +0000 (UTC) Message-ID: <4EE7CBC6.5010001@FreeBSD.org> Date: Tue, 13 Dec 2011 14:03:50 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: malin@randstrom.com References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE751E2.60204@mail.zedat.fu-berlin.de> <20111213155456.GA93017@troutmask.apl.washington.edu> In-Reply-To: X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org, Malin Randstrom , "O. Hartmann" , Current FreeBSD , Steve Kargl , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 22:03:51 -0000 On 12/13/2011 13:31, Malin Randstrom wrote: > stop sending me spam mail ... you never stop despite me having unsubscribeb > several times. stop this! If you had actually unsubscribed, the mail would have stopped. :) You can see the instructions you need to follow below. > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > -- [^L] Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 22:15:04 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B3FD2106566B for ; Tue, 13 Dec 2011 22:15:04 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta12.westchester.pa.mail.comcast.net (qmta12.westchester.pa.mail.comcast.net [76.96.59.227]) by mx1.freebsd.org (Postfix) with ESMTP id 58C278FC13 for ; Tue, 13 Dec 2011 22:15:03 +0000 (UTC) Received: from omta10.westchester.pa.mail.comcast.net ([76.96.62.28]) by qmta12.westchester.pa.mail.comcast.net with comcast id 8m2H1i0040cZkys5CmF3AB; Tue, 13 Dec 2011 22:15:03 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta10.westchester.pa.mail.comcast.net with comcast id 8mF21i01a1t3BNj3WmF3EF; Tue, 13 Dec 2011 22:15:03 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 6E6D2102C19; Tue, 13 Dec 2011 14:15:01 -0800 (PST) Date: Tue, 13 Dec 2011 14:15:01 -0800 From: Jeremy Chadwick To: Andrey Zonov Message-ID: <20111213221501.GA85563@icarus.home.lan> References: <4EE7BF77.5000504@zonov.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE7BF77.5000504@zonov.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: directory listing hangs in "ufs" state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 22:15:04 -0000 On Wed, Dec 14, 2011 at 01:11:19AM +0400, Andrey Zonov wrote: > Hi, > > I've got STABLE-8 (r221983) with mongodb-1.8.1 installed on it. A > couple days ago I observed that listing of mongodb directory stuck > in a few minutes in "ufs" state. I've run it again with ktrace and > got following (kdump -R): > > 91324 ls 0.000003 CALL lstat(0x32c199c8,0x32c19950) > 91324 ls 0.000003 NAMI "base.1" > 91324 ls 21.357255 STRU struct stat {dev=116, ino=45125633, > mode=-rw------- , nlink=1, uid=922, gid=922, rdev=180226648, > atime=1323709877, stime=1323776461, ctime=1323776461, > birthtime=1314798592, size=134217728, blksize=16384, blocks=262304, > flags=0x0 } > 91324 ls 0.000014 RET lstat 0 > > kgdb backtrace of this process was looked like this: > > Thread 297 (Thread 100372): > #0 sched_switch (td=0xffffff0095c008c0, newtd=0xffffff000357b8c0, > flags=) at /usr/src/sys/kern/sched_ule.c:1866 > #1 0xffffffff80406696 in mi_switch (flags=260, newtd=0x0) at > /usr/src/sys/kern/kern_synch.c:449 > #2 0xffffffff8043c072 in sleepq_wait (wchan=0xffffff0103aaf7f8, > pri=80) at /usr/src/sys/kern/subr_sleepqueue.c:609 > #3 0xffffffff803e4a5a in __lockmgr_args (lk=0xffffff0103aaf7f8, > flags=2097408, ilk=0xffffff0103aaf820, wmesg=) at > /usr/src/sys/kern/kern_lock.c:220 > #5 0xffffffff8061239c in ffs_lock (ap=0xffffff84867fc550) at lockmgr.h:94 > #5 0xffffffff806d2462 in VOP_LOCK1_APV (vop=0xffffffff80921fe0, > a=0xffffff84867fc550) at vnode_if.c:1988 > #6 0xffffffff804a58b7 in _vn_lock (vp=0xffffff0103aaf760, > flags=2097152, file=0xffffffff80736e70 > "/usr/src/sys/kern/vfs_subr.c", line=2137) at vnode_if.h:859 > #7 0xffffffff80498bc0 in vget (vp=0xffffff0103aaf760, > flags=2097408, td=0xffffff0095c008c0) at > /usr/src/sys/kern/vfs_subr.c:2137 > #8 0xffffffff804845f4 in cache_lookup (dvp=0xffffff0095675b10, > vpp=0xffffff84867fc910, cnp=0xffffff84867fc938) at > /usr/src/sys/kern/vfs_cache.c:587 > #9 0xffffffff80484a30 in vfs_cache_lookup (ap=) at > /usr/src/sys/kern/vfs_cache.c:905 > #10 0xffffffff806d2e7c in VOP_LOOKUP_APV (vop=0xffffffff80922820, > a=0xffffff84867fc790) at vnode_if.c:123 > #11 0xffffffff8048bc80 in lookup (ndp=0xffffff84867fc8e0) at vnode_if.h:54 > #12 0xffffffff8048cf0e in namei (ndp=0xffffff84867fc8e0) at > /usr/src/sys/kern/vfs_lookup.c:269 > #13 0xffffffff8049c972 in kern_statat_vnhook (td=0xffffff0095c008c0, > flag=) at /usr/src/sys/kern/vfs_syscalls.c:2346 > #14 0xffffffff8049cbb5 in kern_statat (td=) at > /usr/src/sys/kern/vfs_syscalls.c:2327 > #15 0xffffffff8049cc7a in lstat (td=) at > /usr/src/sys/kern/vfs_syscalls.c:2390 > #16 0xffffffff8043e7dd in syscallenter (td=0xffffff0095c008c0, > sa=0xffffff84867fcbb0) at /usr/src/sys/kern/subr_trap.c:326 > #17 0xffffffff8066a5eb in syscall (frame=0xffffff84867fcc50) at > /usr/src/sys/amd64/amd64/trap.c:916 > #18 0xffffffff806517f2 in Xfast_syscall () at > /usr/src/sys/amd64/amd64/exception.S:384 > #19 0x000000003298f75c in ?? () > > The very first idea was to turn off name caching (set debug.vfscache > to 0), but it didn't help. The second idea was to reboot, but it > didn't help too. > > This directory locks fine. It has 10 files and 1 empty directory. > > Have you any ideas what is going on? or how to catch the problem? Assuming this isn't a file on the root filesystem, try booting the machine in single-user mode and using "fsck -f" on the filesystem in question. Can you verify there's no problems with the disk this file lives on as well (smartctl -a /dev/disk)? I'm doubting this is the problem, but thought I'd mention it. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 23:04:44 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 83997106564A; Tue, 13 Dec 2011 23:04:43 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay04.stack.nl [IPv6:2001:610:1108:5010::107]) by mx1.freebsd.org (Postfix) with ESMTP id 19D0D8FC17; Tue, 13 Dec 2011 23:04:43 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 4714C1DD4FB; Wed, 14 Dec 2011 00:04:42 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id 1F56528468; Wed, 14 Dec 2011 00:04:42 +0100 (CET) Date: Wed, 14 Dec 2011 00:04:42 +0100 From: Jilles Tjoelker To: Ivan Klymenko Message-ID: <20111213230441.GB42285@stack.nl> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111213104048.40f3e3de@nonamehost.> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "O. Hartmann" , Doug Barton , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Current FreeBSD Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 23:04:44 -0000 On Tue, Dec 13, 2011 at 10:40:48AM +0200, Ivan Klymenko wrote: > If the algorithm ULE does not contain problems - it means the problem > has Core2Duo, or in a piece of code that uses the ULE scheduler. > I already wrote in a mailing list that specifically in my case (Core2Duo) > partially helps the following patch: > --- sched_ule.c.orig 2011-11-24 18:11:48.000000000 +0200 > +++ sched_ule.c 2011-12-10 22:47:08.000000000 +0200 > @@ -794,7 +794,8 @@ > * 1.5 * balance_interval. > */ > balance_ticks = max(balance_interval / 2, 1); > - balance_ticks += random() % balance_interval; > +// balance_ticks += random() % balance_interval; > + balance_ticks += ((int)random()) % balance_interval; > if (smp_started == 0 || rebalance == 0) > return; > tdq = TDQ_SELF(); This avoids a 64-bit division on 64-bit platforms but seems to have no effect otherwise. Because this function is not called very often, the change seems unlikely to help. > @@ -2118,13 +2119,21 @@ > struct td_sched *ts; > > THREAD_LOCK_ASSERT(td, MA_OWNED); > + if (td->td_pri_class & PRI_FIFO_BIT) > + return; > + ts = td->td_sched; > + /* > + * We used up one time slice. > + */ > + if (--ts->ts_slice > 0) > + return; This skips most of the periodic functionality (long term load balancer, saving switch count (?), insert index (?), interactivity score update for long running thread) if the thread is not going to be rescheduled right now. It looks wrong but it is a data point if it helps your workload. > tdq = TDQ_SELF(); > #ifdef SMP > /* > * We run the long term load balancer infrequently on the first cpu. > */ > - if (balance_tdq == tdq) { > - if (balance_ticks && --balance_ticks == 0) > + if (balance_ticks && --balance_ticks == 0) { > + if (balance_tdq == tdq) > sched_balance(); > } > #endif The main effect of this appears to be to disable the long term load balancer completely after some time. At some point, a CPU other than the first CPU (which uses balance_tdq) will set balance_ticks = 0, and sched_balance() will never be called again. It also introduces a hypothetical race condition because the access to balance_ticks is no longer restricted to one CPU under a spinlock. If the long term load balancer may be causing trouble, try setting kern.sched.balance_interval to a higher value with unpatched code. > @@ -2144,9 +2153,6 @@ > if (TAILQ_EMPTY(&tdq->tdq_timeshare.rq_queues[tdq->tdq_ridx])) > tdq->tdq_ridx = tdq->tdq_idx; > } > - ts = td->td_sched; > - if (td->td_pri_class & PRI_FIFO_BIT) > - return; > if (PRI_BASE(td->td_pri_class) == PRI_TIMESHARE) { > /* > * We used a tick; charge it to the thread so > @@ -2157,11 +2163,6 @@ > sched_priority(td); > } > /* > - * We used up one time slice. > - */ > - if (--ts->ts_slice > 0) > - return; > - /* > * We're out of time, force a requeue at userret(). > */ > ts->ts_slice = sched_slice; > and refusal to use options FULL_PREEMPTION > But no one has unsubscribed to my letter, my patch helps or not in the > case of Core2Duo... > There is a suspicion that the problems stem from the sections of code > associated with the SMP... > Maybe I'm in something wrong, but I want to help in solving this > problem ... -- Jilles Tjoelker From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 23:19:40 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34672106567A; Tue, 13 Dec 2011 23:19:40 +0000 (UTC) (envelope-from marcus@odin.blazingdot.com) Received: from odin.blazingdot.com (odin.blazingdot.com [199.48.133.254]) by mx1.freebsd.org (Postfix) with ESMTP id 1347F8FC16; Tue, 13 Dec 2011 23:19:39 +0000 (UTC) Received: by odin.blazingdot.com (Postfix, from userid 1001) id 01B5D114241; Tue, 13 Dec 2011 23:02:15 +0000 (UTC) Date: Tue, 13 Dec 2011 23:02:15 +0000 From: Marcus Reid To: Doug Barton Message-ID: <20111213230215.GA83159@blazingdot.com> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE69C5A.3090005@FreeBSD.org> X-Coffee-Level: nearly-fatal User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 23:19:40 -0000 On Mon, Dec 12, 2011 at 04:29:14PM -0800, Doug Barton wrote: > On 12/12/2011 05:47, O. Hartmann wrote: > > Do we have any proof at hand for such cases where SCHED_ULE performs > > much better than SCHED_4BSD? > > I complained about poor interactive performance of ULE in a desktop > environment for years. I had numerous people try to help, including > Jeff, with various tunables, dtrace'ing, etc. The cause of the problem > was never found. The issues that I've seen with ULE on the desktop seem to be caused by X taking up a steady amount of CPU, and being demoted from being an "interactive" process. X then becomes the bottleneck for other processes that would otherwise be "interactive". Try 'renice -20 ' and see if that makes your problems go away. Marcus From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 23:39:14 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 09B051065670; Tue, 13 Dec 2011 23:39:14 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm2.ukr.net (fsm2.ukr.net [195.214.192.121]) by mx1.freebsd.org (Postfix) with ESMTP id 8A7C18FC08; Tue, 13 Dec 2011 23:39:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=5F70nZqRcYgm7bijrHgHyKB7UoBju6OC5/bO8dW/88o=; b=UCCsjgOajR4VEgZZUmFT8w1hirtKquOO5Imc+Y+MxOaCwFvWJHpSRzawb4RG8UtOwO39O8ln9yMHmfUQFZfk4sHuyDdTgiD+e5YJGQ9o6LPDgvP/+XA1/4abE7Fxon5FSIdselgvFv3ZKh+9DCi7ybFm7aRNXgx+H0jDZkIHJv0=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm2.ukr.net with esmtpsa ID 1RabwD-0003M3-1I ; Wed, 14 Dec 2011 01:39:09 +0200 Date: Wed, 14 Dec 2011 01:39:06 +0200 From: Ivan Klymenko To: Jilles Tjoelker Message-ID: <20111214013906.068f69df@nonamehost.> In-Reply-To: <20111213230441.GB42285@stack.nl> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "O. Hartmann" , Doug Barton , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Current FreeBSD Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 23:39:14 -0000 =D0=92 Wed, 14 Dec 2011 00:04:42 +0100 Jilles Tjoelker =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > On Tue, Dec 13, 2011 at 10:40:48AM +0200, Ivan Klymenko wrote: > > If the algorithm ULE does not contain problems - it means the > > problem has Core2Duo, or in a piece of code that uses the ULE > > scheduler. I already wrote in a mailing list that specifically in > > my case (Core2Duo) partially helps the following patch: > > --- sched_ule.c.orig 2011-11-24 18:11:48.000000000 +0200 > > +++ sched_ule.c 2011-12-10 22:47:08.000000000 +0200 > > @@ -794,7 +794,8 @@ > > * 1.5 * balance_interval. > > */ > > balance_ticks =3D max(balance_interval / 2, 1); > > - balance_ticks +=3D random() % balance_interval; > > +// balance_ticks +=3D random() % balance_interval; > > + balance_ticks +=3D ((int)random()) % balance_interval; > > if (smp_started =3D=3D 0 || rebalance =3D=3D 0) > > return; > > tdq =3D TDQ_SELF(); >=20 > This avoids a 64-bit division on 64-bit platforms but seems to have no > effect otherwise. Because this function is not called very often, the > change seems unlikely to help. Yes, this section does not apply to this problem :) Just I posted the latest patch which i using now... >=20 > > @@ -2118,13 +2119,21 @@ > > struct td_sched *ts; > > =20 > > THREAD_LOCK_ASSERT(td, MA_OWNED); > > + if (td->td_pri_class & PRI_FIFO_BIT) > > + return; > > + ts =3D td->td_sched; > > + /* > > + * We used up one time slice. > > + */ > > + if (--ts->ts_slice > 0) > > + return; >=20 > This skips most of the periodic functionality (long term load > balancer, saving switch count (?), insert index (?), interactivity > score update for long running thread) if the thread is not going to > be rescheduled right now. >=20 > It looks wrong but it is a data point if it helps your workload. Yes, I did it for as long as possible to delay the execution of the code in= section: ... #ifdef SMP /* * We run the long term load balancer infrequently on the first cpu. */ if (balance_tdq =3D=3D tdq) { if (balance_ticks && --balance_ticks =3D=3D 0) sched_balance(); } #endif ... >=20 > > tdq =3D TDQ_SELF(); > > #ifdef SMP > > /* > > * We run the long term load balancer infrequently on the > > first cpu. */ > > - if (balance_tdq =3D=3D tdq) { > > - if (balance_ticks && --balance_ticks =3D=3D 0) > > + if (balance_ticks && --balance_ticks =3D=3D 0) { > > + if (balance_tdq =3D=3D tdq) > > sched_balance(); > > } > > #endif >=20 > The main effect of this appears to be to disable the long term load > balancer completely after some time. At some point, a CPU other than > the first CPU (which uses balance_tdq) will set balance_ticks =3D 0, and > sched_balance() will never be called again. >=20 That is, for the same reason as above in the text... > It also introduces a hypothetical race condition because the access to > balance_ticks is no longer restricted to one CPU under a spinlock. >=20 > If the long term load balancer may be causing trouble, try setting > kern.sched.balance_interval to a higher value with unpatched code. I checked it in the first place - but it did not help fix the situation... The impression of malfunction rebalancing... It seems that the thread is passed on to the same core that is loaded and s= o... Perhaps this is a consequence of an incorrect definition of the topology CP= U? >=20 > > @@ -2144,9 +2153,6 @@ > > if > > (TAILQ_EMPTY(&tdq->tdq_timeshare.rq_queues[tdq->tdq_ridx])) > > tdq->tdq_ridx =3D tdq->tdq_idx; } > > - ts =3D td->td_sched; > > - if (td->td_pri_class & PRI_FIFO_BIT) > > - return; > > if (PRI_BASE(td->td_pri_class) =3D=3D PRI_TIMESHARE) { > > /* > > * We used a tick; charge it to the thread so > > @@ -2157,11 +2163,6 @@ > > sched_priority(td); > > } > > /* > > - * We used up one time slice. > > - */ > > - if (--ts->ts_slice > 0) > > - return; > > - /* > > * We're out of time, force a requeue at userret(). > > */ > > ts->ts_slice =3D sched_slice; >=20 > > and refusal to use options FULL_PREEMPTION > > But no one has unsubscribed to my letter, my patch helps or not in > > the case of Core2Duo... > > There is a suspicion that the problems stem from the sections of > > code associated with the SMP... > > Maybe I'm in something wrong, but I want to help in solving this > > problem ... >=20 From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 23:42:15 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F12F81065703; Tue, 13 Dec 2011 23:42:14 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm1.ukr.net (fsm1.ukr.net [195.214.192.120]) by mx1.freebsd.org (Postfix) with ESMTP id A00CD8FC1A; Tue, 13 Dec 2011 23:42:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=sw3zgxPAR3QAfGKCe6TuQNxbPW4sLLWcsOBc2vusc8o=; b=M+tGRDGYgsoljjuix/VjsnDBpi6i74Dlzd8dfI6e/B9ADT1z/g07FroZ3i3PrwHMZSKrk5ZMvuPz+hGrXrKXdv4F48onYzXCsj6A6osK4DwY+C1igtpK4mUPbE5VlNiD4pun5cGDHng615clnM6+XbGUMwaZdZXCz/o0vpbLGzo=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm1.ukr.net with esmtpsa ID 1RabzA-000Ofi-Px ; Wed, 14 Dec 2011 01:42:12 +0200 Date: Wed, 14 Dec 2011 01:42:11 +0200 From: Ivan Klymenko To: Marcus Reid Message-ID: <20111214014211.3e108b53@nonamehost.> In-Reply-To: <20111213230215.GA83159@blazingdot.com> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213230215.GA83159@blazingdot.com> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "O. Hartmann" , Doug Barton , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Current FreeBSD Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 23:42:15 -0000 =D0=92 Tue, 13 Dec 2011 23:02:15 +0000 Marcus Reid =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > On Mon, Dec 12, 2011 at 04:29:14PM -0800, Doug Barton wrote: > > On 12/12/2011 05:47, O. Hartmann wrote: > > > Do we have any proof at hand for such cases where SCHED_ULE > > > performs much better than SCHED_4BSD? > >=20 > > I complained about poor interactive performance of ULE in a desktop > > environment for years. I had numerous people try to help, including > > Jeff, with various tunables, dtrace'ing, etc. The cause of the > > problem was never found. >=20 > The issues that I've seen with ULE on the desktop seem to be caused > by X taking up a steady amount of CPU, and being demoted from being an > "interactive" process. X then becomes the bottleneck for other > processes that would otherwise be "interactive". Try 'renice -20 > ' and see if that makes your problems go away. Why, then X is not a bottleneck when using 4BSD? > Marcus From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 00:01:57 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 96C2C1065670; Wed, 14 Dec 2011 00:01:57 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 54AFE8FC0C; Wed, 14 Dec 2011 00:01:57 +0000 (UTC) Received: by dakp5 with SMTP id p5so253302dak.13 for ; Tue, 13 Dec 2011 16:01:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=/d9tPhNRKuZlrGasaA/On3wANIhZMN6DaAHZeY1WxuE=; b=AQ1f2p6HGVtPmZlrafZzUATT3m0rwxjUkIK6Ma1mcTjP7CD6fH+TWTFgIVK0IuVIHI 96gnQ2Nl+aVo8/aYuPhbl4qlyVDtp4aoostkuzuCIn905eq7/lfh/FQyQoKaiCl5UAQZ zya9nkvb9uUfsueC4TJ/LzQLC+sOLfMOixItU= MIME-Version: 1.0 Received: by 10.68.201.193 with SMTP id kc1mr18801pbc.51.1323820916837; Tue, 13 Dec 2011 16:01:56 -0800 (PST) Sender: mdf356@gmail.com Received: by 10.68.197.198 with HTTP; Tue, 13 Dec 2011 16:01:56 -0800 (PST) In-Reply-To: <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> Date: Tue, 13 Dec 2011 16:01:56 -0800 X-Google-Sender-Auth: cXWpb1OQ43-qtKIw90RlKMbBfDs Message-ID: From: mdf@FreeBSD.org To: Ivan Klymenko Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 00:01:57 -0000 On Tue, Dec 13, 2011 at 3:39 PM, Ivan Klymenko wrote: > =D0=92 Wed, 14 Dec 2011 00:04:42 +0100 > Jilles Tjoelker =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > >> On Tue, Dec 13, 2011 at 10:40:48AM +0200, Ivan Klymenko wrote: >> > If the algorithm ULE does not contain problems - it means the >> > problem has Core2Duo, or in a piece of code that uses the ULE >> > scheduler. I already wrote in a mailing list that specifically in >> > my case (Core2Duo) partially helps the following patch: >> > --- sched_ule.c.orig =C2=A0 =C2=A0 =C2=A0 =C2=A02011-11-24 18:11:48.00= 0000000 +0200 >> > +++ sched_ule.c =C2=A0 =C2=A0 2011-12-10 22:47:08.000000000 +0200 >> > @@ -794,7 +794,8 @@ >> > =C2=A0 =C2=A0 =C2=A0* 1.5 * balance_interval. >> > =C2=A0 =C2=A0 =C2=A0*/ >> > =C2=A0 =C2=A0 balance_ticks =3D max(balance_interval / 2, 1); >> > - =C2=A0 balance_ticks +=3D random() % balance_interval; >> > +// balance_ticks +=3D random() % balance_interval; >> > + =C2=A0 balance_ticks +=3D ((int)random()) % balance_interval; >> > =C2=A0 =C2=A0 if (smp_started =3D=3D 0 || rebalance =3D=3D 0) >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; >> > =C2=A0 =C2=A0 tdq =3D TDQ_SELF(); >> >> This avoids a 64-bit division on 64-bit platforms but seems to have no >> effect otherwise. Because this function is not called very often, the >> change seems unlikely to help. > > Yes, this section does not apply to this problem :) > Just I posted the latest patch which i using now... > >> >> > @@ -2118,13 +2119,21 @@ >> > =C2=A0 =C2=A0 struct td_sched *ts; >> > >> > =C2=A0 =C2=A0 THREAD_LOCK_ASSERT(td, MA_OWNED); >> > + =C2=A0 if (td->td_pri_class & PRI_FIFO_BIT) >> > + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; >> > + =C2=A0 ts =3D td->td_sched; >> > + =C2=A0 /* >> > + =C2=A0 =C2=A0* We used up one time slice. >> > + =C2=A0 =C2=A0*/ >> > + =C2=A0 if (--ts->ts_slice > 0) >> > + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; >> >> This skips most of the periodic functionality (long term load >> balancer, saving switch count (?), insert index (?), interactivity >> score update for long running thread) if the thread is not going to >> be rescheduled right now. >> >> It looks wrong but it is a data point if it helps your workload. > > Yes, I did it for as long as possible to delay the execution of the code = in section: > ... > #ifdef SMP > =C2=A0 =C2=A0 =C2=A0 =C2=A0/* > =C2=A0 =C2=A0 =C2=A0 =C2=A0 * We run the long term load balancer infreque= ntly on the first cpu. > =C2=A0 =C2=A0 =C2=A0 =C2=A0 */ > =C2=A0 =C2=A0 =C2=A0 =C2=A0if (balance_tdq =3D=3D tdq) { > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (balance_ticks = && --balance_ticks =3D=3D 0) > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0sched_balance(); > =C2=A0 =C2=A0 =C2=A0 =C2=A0} > #endif > ... > >> >> > =C2=A0 =C2=A0 tdq =3D TDQ_SELF(); >> > =C2=A0#ifdef SMP >> > =C2=A0 =C2=A0 /* >> > =C2=A0 =C2=A0 =C2=A0* We run the long term load balancer infrequently = on the >> > first cpu. */ >> > - =C2=A0 if (balance_tdq =3D=3D tdq) { >> > - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (balance_ticks && --balance_ti= cks =3D=3D 0) >> > + =C2=A0 if (balance_ticks && --balance_ticks =3D=3D 0) { >> > + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (balance_tdq =3D=3D tdq) >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = sched_balance(); >> > =C2=A0 =C2=A0 } >> > =C2=A0#endif >> >> The main effect of this appears to be to disable the long term load >> balancer completely after some time. At some point, a CPU other than >> the first CPU (which uses balance_tdq) will set balance_ticks =3D 0, and >> sched_balance() will never be called again. >> > > That is, for the same reason as above in the text... > >> It also introduces a hypothetical race condition because the access to >> balance_ticks is no longer restricted to one CPU under a spinlock. >> >> If the long term load balancer may be causing trouble, try setting >> kern.sched.balance_interval to a higher value with unpatched code. > > I checked it in the first place - but it did not help fix the situation..= . > > The impression of malfunction rebalancing... > It seems that the thread is passed on to the same core that is loaded and= so... > Perhaps this is a consequence of an incorrect definition of the topology = CPU? > >> >> > @@ -2144,9 +2153,6 @@ >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if >> > (TAILQ_EMPTY(&tdq->tdq_timeshare.rq_queues[tdq->tdq_ridx])) >> > tdq->tdq_ridx =3D tdq->tdq_idx; } >> > - =C2=A0 ts =3D td->td_sched; >> > - =C2=A0 if (td->td_pri_class & PRI_FIFO_BIT) >> > - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; >> > =C2=A0 =C2=A0 if (PRI_BASE(td->td_pri_class) =3D=3D PRI_TIMESHARE) { >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* We used a tick; char= ge it to the thread so >> > @@ -2157,11 +2163,6 @@ >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sched_priority(td); >> > =C2=A0 =C2=A0 } >> > =C2=A0 =C2=A0 /* >> > - =C2=A0 =C2=A0* We used up one time slice. >> > - =C2=A0 =C2=A0*/ >> > - =C2=A0 if (--ts->ts_slice > 0) >> > - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; >> > - =C2=A0 /* >> > =C2=A0 =C2=A0 =C2=A0* We're out of time, force a requeue at userret(). >> > =C2=A0 =C2=A0 =C2=A0*/ >> > =C2=A0 =C2=A0 ts->ts_slice =3D sched_slice; >> >> > and refusal to use options FULL_PREEMPTION >> > But no one has unsubscribed to my letter, my patch helps or not in >> > the case of Core2Duo... >> > There is a suspicion that the problems stem from the sections of >> > code associated with the SMP... >> > Maybe I'm in something wrong, but I want to help in solving this >> > problem ... Has anyone experiencing problems tried to set sysctl kern.sched.steal_thres= h=3D1 ? I don't remember what our specific problem at $WORK was, perhaps it was just interrupt threads not getting serviced fast enough, but we've hard-coded this to 1 and removed the code that sets it in sched_initticks(). The same effect should be had by setting the sysctl after a box is up. Thanks, matthew From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 00:36:32 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17C73106564A; Wed, 14 Dec 2011 00:36:32 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm2.ukr.net (fsm2.ukr.net [195.214.192.121]) by mx1.freebsd.org (Postfix) with ESMTP id 9EFAE8FC1A; Wed, 14 Dec 2011 00:36:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=18YzJbAWI+aRJeAk77qSqDa/FDJYKyD0tpp/v1biG78=; b=hU8yO3T7w8C35eMlOLcyxigofcXvWzFwT0yKaGwGu0+CIHOUQ65gVLbMS9S0uYZHPuuCF0dr1ReSYBXAbNGWg4ek9lS9Uf1kCB87IsQAYiE4xYutmpjDX20QLU1txXgmYO/xq6gVxXyGrg+LbeHfSbx/TXN1rF8RP0YafM1EnTg=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm2.ukr.net with esmtpsa ID 1Racpi-000EAS-H6 ; Wed, 14 Dec 2011 02:36:30 +0200 Date: Wed, 14 Dec 2011 02:36:29 +0200 From: Ivan Klymenko To: mdf@FreeBSD.org Message-ID: <20111214023629.3ae8c928@nonamehost.> In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Doug Barton , freebsd-stable@freebsd.org, Tjoelker , "O. Hartmann" , Current FreeBSD , Jilles, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 00:36:32 -0000 =D0=92 Tue, 13 Dec 2011 16:01:56 -0800 mdf@FreeBSD.org =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > On Tue, Dec 13, 2011 at 3:39 PM, Ivan Klymenko wrote: > > =D0=92 Wed, 14 Dec 2011 00:04:42 +0100 > > Jilles Tjoelker =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > > > >> On Tue, Dec 13, 2011 at 10:40:48AM +0200, Ivan Klymenko wrote: > >> > If the algorithm ULE does not contain problems - it means the > >> > problem has Core2Duo, or in a piece of code that uses the ULE > >> > scheduler. I already wrote in a mailing list that specifically in > >> > my case (Core2Duo) partially helps the following patch: > >> > --- sched_ule.c.orig =C2=A0 =C2=A0 =C2=A0 =C2=A02011-11-24 18:11:48.= 000000000 +0200 > >> > +++ sched_ule.c =C2=A0 =C2=A0 2011-12-10 22:47:08.000000000 +0200 > >> > @@ -794,7 +794,8 @@ > >> > =C2=A0 =C2=A0 =C2=A0* 1.5 * balance_interval. > >> > =C2=A0 =C2=A0 =C2=A0*/ > >> > =C2=A0 =C2=A0 balance_ticks =3D max(balance_interval / 2, 1); > >> > - =C2=A0 balance_ticks +=3D random() % balance_interval; > >> > +// balance_ticks +=3D random() % balance_interval; > >> > + =C2=A0 balance_ticks +=3D ((int)random()) % balance_interval; > >> > =C2=A0 =C2=A0 if (smp_started =3D=3D 0 || rebalance =3D=3D 0) > >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; > >> > =C2=A0 =C2=A0 tdq =3D TDQ_SELF(); > >> > >> This avoids a 64-bit division on 64-bit platforms but seems to > >> have no effect otherwise. Because this function is not called very > >> often, the change seems unlikely to help. > > > > Yes, this section does not apply to this problem :) > > Just I posted the latest patch which i using now... > > > >> > >> > @@ -2118,13 +2119,21 @@ > >> > =C2=A0 =C2=A0 struct td_sched *ts; > >> > > >> > =C2=A0 =C2=A0 THREAD_LOCK_ASSERT(td, MA_OWNED); > >> > + =C2=A0 if (td->td_pri_class & PRI_FIFO_BIT) > >> > + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; > >> > + =C2=A0 ts =3D td->td_sched; > >> > + =C2=A0 /* > >> > + =C2=A0 =C2=A0* We used up one time slice. > >> > + =C2=A0 =C2=A0*/ > >> > + =C2=A0 if (--ts->ts_slice > 0) > >> > + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; > >> > >> This skips most of the periodic functionality (long term load > >> balancer, saving switch count (?), insert index (?), interactivity > >> score update for long running thread) if the thread is not going to > >> be rescheduled right now. > >> > >> It looks wrong but it is a data point if it helps your workload. > > > > Yes, I did it for as long as possible to delay the execution of the > > code in section: ... > > #ifdef SMP > > =C2=A0 =C2=A0 =C2=A0 =C2=A0/* > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 * We run the long term load balancer infreq= uently on the > > first cpu. */ > > =C2=A0 =C2=A0 =C2=A0 =C2=A0if (balance_tdq =3D=3D tdq) { > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (balance_tick= s && --balance_ticks =3D=3D 0) > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0sched_balance(); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0} > > #endif > > ... > > > >> > >> > =C2=A0 =C2=A0 tdq =3D TDQ_SELF(); > >> > =C2=A0#ifdef SMP > >> > =C2=A0 =C2=A0 /* > >> > =C2=A0 =C2=A0 =C2=A0* We run the long term load balancer infrequentl= y on the > >> > first cpu. */ > >> > - =C2=A0 if (balance_tdq =3D=3D tdq) { > >> > - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (balance_ticks && --balance_= ticks =3D=3D 0) > >> > + =C2=A0 if (balance_ticks && --balance_ticks =3D=3D 0) { > >> > + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (balance_tdq =3D=3D tdq) > >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 sched_balance(); > >> > =C2=A0 =C2=A0 } > >> > =C2=A0#endif > >> > >> The main effect of this appears to be to disable the long term load > >> balancer completely after some time. At some point, a CPU other > >> than the first CPU (which uses balance_tdq) will set balance_ticks > >> =3D 0, and sched_balance() will never be called again. > >> > > > > That is, for the same reason as above in the text... > > > >> It also introduces a hypothetical race condition because the > >> access to balance_ticks is no longer restricted to one CPU under a > >> spinlock. > >> > >> If the long term load balancer may be causing trouble, try setting > >> kern.sched.balance_interval to a higher value with unpatched code. > > > > I checked it in the first place - but it did not help fix the > > situation... > > > > The impression of malfunction rebalancing... > > It seems that the thread is passed on to the same core that is > > loaded and so... Perhaps this is a consequence of an incorrect > > definition of the topology CPU? > > > >> > >> > @@ -2144,9 +2153,6 @@ > >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if > >> > (TAILQ_EMPTY(&tdq->tdq_timeshare.rq_queues[tdq->tdq_ridx])) > >> > tdq->tdq_ridx =3D tdq->tdq_idx; } > >> > - =C2=A0 ts =3D td->td_sched; > >> > - =C2=A0 if (td->td_pri_class & PRI_FIFO_BIT) > >> > - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; > >> > =C2=A0 =C2=A0 if (PRI_BASE(td->td_pri_class) =3D=3D PRI_TIMESHARE) { > >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* > >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* We used a tick; ch= arge it to the thread so > >> > @@ -2157,11 +2163,6 @@ > >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sched_priority(td); > >> > =C2=A0 =C2=A0 } > >> > =C2=A0 =C2=A0 /* > >> > - =C2=A0 =C2=A0* We used up one time slice. > >> > - =C2=A0 =C2=A0*/ > >> > - =C2=A0 if (--ts->ts_slice > 0) > >> > - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; > >> > - =C2=A0 /* > >> > =C2=A0 =C2=A0 =C2=A0* We're out of time, force a requeue at userret(= ). > >> > =C2=A0 =C2=A0 =C2=A0*/ > >> > =C2=A0 =C2=A0 ts->ts_slice =3D sched_slice; > >> > >> > and refusal to use options FULL_PREEMPTION > >> > But no one has unsubscribed to my letter, my patch helps or not > >> > in the case of Core2Duo... > >> > There is a suspicion that the problems stem from the sections of > >> > code associated with the SMP... > >> > Maybe I'm in something wrong, but I want to help in solving this > >> > problem ... >=20 >=20 > Has anyone experiencing problems tried to set sysctl > kern.sched.steal_thresh=3D1 ? >=20 In my case, the variable kern.sched.steal_thresh and so has the value 1. > I don't remember what our specific problem at $WORK was, perhaps it > was just interrupt threads not getting serviced fast enough, but we've > hard-coded this to 1 and removed the code that sets it in > sched_initticks(). The same effect should be had by setting the > sysctl after a box is up. >=20 > Thanks, > matthew From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 01:37:48 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A8A8A1065670 for ; Wed, 14 Dec 2011 01:37:48 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (ip-3-2-0-2.r20.asbnva02.us.ce.gin.ntt.net [IPv6:2001:418:0:5000::16]) by mx1.freebsd.org (Postfix) with ESMTP id 5C2B98FC16 for ; Wed, 14 Dec 2011 01:37:48 +0000 (UTC) Received: from wonderland.m5p.com (wonderland.m5p.com [IPv6:2001:418:3fd::19]) by mailhost.m5p.com (8.14.4/8.14.4) with ESMTP id pBE1bgfA003516 for ; Tue, 13 Dec 2011 20:37:47 -0500 (EST) (envelope-from george+freebsd@m5p.com) Message-ID: <4EE7FDE6.2010006@m5p.com> Date: Tue, 13 Dec 2011 20:37:42 -0500 From: George Mitchell User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111127 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213230215.GA83159@blazingdot.com> In-Reply-To: <20111213230215.GA83159@blazingdot.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (mailhost.m5p.com [IPv6:2001:418:3fd::f7]); Tue, 13 Dec 2011 20:37:47 -0500 (EST) X-Scanned-By: MIMEDefang 2.72 on IPv6:2001:418:3fd::f7 Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 01:37:48 -0000 On 12/13/11 18:02, Marcus Reid wrote: > [...] > The issues that I've seen with ULE on the desktop seem to be caused by X > taking up a steady amount of CPU, and being demoted from being an > "interactive" process. X then becomes the bottleneck for other > processes that would otherwise be "interactive". Try 'renice -20 > ' and see if that makes your problems go away. > > Marcus > [...] renice on X has no effect. Stopping my compute-bound dnetc process immediately speeds everything up; restarting it slows it back down. On 12/13/11 19:01, mdf@freebsd.org wrote: > [...] > Has anyone experiencing problems tried to set sysctl kern.sched.steal_thresh=1 ? > [...] 1 appears to be the default value for kern.sched.steal_thresh. -- George Mitchell From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 02:28:22 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 44A63106564A for ; Wed, 14 Dec 2011 02:28:22 +0000 (UTC) (envelope-from brett@lariat.net) Received: from lariat.net (lariat.net [66.62.230.51]) by mx1.freebsd.org (Postfix) with ESMTP id C58B98FC0C for ; Wed, 14 Dec 2011 02:28:21 +0000 (UTC) Received: from WildRover.lariat.net (IDENT:ppp1000.lariat.net@lariat.net [66.119.58.2] (may be forged)) by lariat.net (8.9.3/8.9.3) with ESMTP id TAA26206; Tue, 13 Dec 2011 19:28:16 -0700 (MST) Message-Id: <201112140228.TAA26206@lariat.net> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 13 Dec 2011 19:28:05 -0700 To: Ben Kaduk , Ed Schouten From: Brett Glass In-Reply-To: References: <201112090913.CAA03333@lariat.net> <201112120455.VAA03028@lariat.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Cc: stable@freebsd.org Subject: Re: Two problems still present in RC3 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 02:28:22 -0000 At 12:34 PM 12/13/2011, Ben Kaduk wrote: >If I remember correctly, your original message mentioned seeing this >issue in emacs; have you tried reproducing it in a simpler test case? No; when we hit the bug, we moved to SSH with a VT100 emulator so that we could configure the system. But the system console really should work as well. I expect to be configuring another system this evening and will try the script fragment you sent. --Brett From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 02:51:38 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26517106564A for ; Wed, 14 Dec 2011 02:51:38 +0000 (UTC) (envelope-from brett@lariat.net) Received: from lariat.net (lariat.net [66.62.230.51]) by mx1.freebsd.org (Postfix) with ESMTP id B8A668FC1A for ; Wed, 14 Dec 2011 02:51:37 +0000 (UTC) Received: from WildRover.lariat.net (IDENT:ppp1000.lariat.net@lariat.net [66.119.58.2] (may be forged)) by lariat.net (8.9.3/8.9.3) with ESMTP id TAA26350; Tue, 13 Dec 2011 19:51:25 -0700 (MST) Message-Id: <201112140251.TAA26350@lariat.net> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 13 Dec 2011 19:51:15 -0700 To: Ed Schouten , Ben Kaduk From: Brett Glass In-Reply-To: <20111213202333.GH1771@hoeg.nl> References: <201112090913.CAA03333@lariat.net> <201112120455.VAA03028@lariat.net> <20111213202333.GH1771@hoeg.nl> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Cc: stable@freebsd.org Subject: Re: Two problems still present in RC3 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 02:51:38 -0000 At 01:23 PM 12/13/2011, Ed Schouten wrote: >Well, I don't care whether the test case is small or large, as long as I >know how to reproduce it. I have never used jove before, so what kind of >buttons should I press to get the artifacts? Just start inserting text, deleting characters and lines (Ctrl-D and Ctrl-K), etc. It's very obvious. >Does it only happen on the console, or also when using a regular xterm? I do not use regular xterms, so I can't answer that, alas. --Brett Glass From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 03:25:18 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 373EB106566B; Wed, 14 Dec 2011 03:25:18 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx06.syd.optusnet.com.au (fallbackmx06.syd.optusnet.com.au [211.29.132.8]) by mx1.freebsd.org (Postfix) with ESMTP id A20CD8FC08; Wed, 14 Dec 2011 03:25:16 +0000 (UTC) Received: from mail15.syd.optusnet.com.au (mail15.syd.optusnet.com.au [211.29.132.196]) by fallbackmx06.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id pBE1PJRp009709; Wed, 14 Dec 2011 12:25:19 +1100 Received: from c211-28-227-231.carlnfd1.nsw.optusnet.com.au (c211-28-227-231.carlnfd1.nsw.optusnet.com.au [211.28.227.231]) by mail15.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id pBE1PEwA000923 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 14 Dec 2011 12:25:14 +1100 Date: Wed, 14 Dec 2011 12:25:14 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Ivan Klymenko In-Reply-To: <20111214013906.068f69df@nonamehost.> Message-ID: <20111214115809.T1563@besplex.bde.org> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <20111214013906.068f69df@nonamehost.> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="0-1963381402-1323825914=:1563" Cc: Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 03:25:18 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --0-1963381402-1323825914=:1563 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Wed, 14 Dec 2011, Ivan Klymenko wrote: > =D0=92 Wed, 14 Dec 2011 00:04:42 +0100 > Jilles Tjoelker =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > >> On Tue, Dec 13, 2011 at 10:40:48AM +0200, Ivan Klymenko wrote: >>> If the algorithm ULE does not contain problems - it means the >>> problem has Core2Duo, or in a piece of code that uses the ULE >>> scheduler. I already wrote in a mailing list that specifically in >>> my case (Core2Duo) partially helps the following patch: >>> --- sched_ule.c.orig=092011-11-24 18:11:48.000000000 +0200 >>> +++ sched_ule.c=092011-12-10 22:47:08.000000000 +0200 >>> ... >>> @@ -2118,13 +2119,21 @@ >>> =09struct td_sched *ts; >>> >>> =09THREAD_LOCK_ASSERT(td, MA_OWNED); >>> +=09if (td->td_pri_class & PRI_FIFO_BIT) >>> +=09=09return; >>> +=09ts =3D td->td_sched; >>> +=09/* >>> +=09 * We used up one time slice. >>> +=09 */ >>> +=09if (--ts->ts_slice > 0) >>> +=09=09return; >> >> This skips most of the periodic functionality (long term load >> balancer, saving switch count (?), insert index (?), interactivity >> score update for long running thread) if the thread is not going to >> be rescheduled right now. >> >> It looks wrong but it is a data point if it helps your workload. > > Yes, I did it for as long as possible to delay the execution of the code = in section: I don't understand what you are doing here, but recently noticed that the timeslicing in SCHED_4BSD is completely broken. This bug may be a feature. SCHED_4BSD doesn't have its own timeslice counter like ts_slice above. It uses `switchticks' instead. But switchticks hasn't been usable for this purpose since long before SCHED_4BSD started using it for this purpose. switchticks is reset on every context switch, so it is useless for almost all purposes -- any interrupt activity on a non-fast interrupt clobbers it. Removing the check of ts_slice in the above and always returning might give a similar bug to the SCHED_4BSD one. I noticed this while looking for bugs in realtime scheduling. In the above, returning early for PRI_FIFO_BIT also skips most of the periodic functionality. In SCHED_4BSD, returning early is the usual case, so the PRI_FIFO_BIT might as well not be checked, and it is the unusual fifo scheduling case (which is supposed to only apply to realtime priority threads) which has a chance of working as intended, while the usual roundrobin case degenerates to an impure form of fifo scheduling (iit is impure since priority decay still works so it is only fifo among threads of the same priority). >>... >>> @@ -2144,9 +2153,6 @@ >>> =09=09if >>> (TAILQ_EMPTY(&tdq->tdq_timeshare.rq_queues[tdq->tdq_ridx])) >>> tdq->tdq_ridx =3D tdq->tdq_idx; } >>> -=09ts =3D td->td_sched; >>> -=09if (td->td_pri_class & PRI_FIFO_BIT) >>> -=09=09return; >>> =09if (PRI_BASE(td->td_pri_class) =3D=3D PRI_TIMESHARE) { >>> =09=09/* >>> =09=09 * We used a tick; charge it to the thread so >>> @@ -2157,11 +2163,6 @@ >>> =09=09sched_priority(td); >>> =09} >>> =09/* >>> -=09 * We used up one time slice. >>> -=09 */ >>> -=09if (--ts->ts_slice > 0) >>> -=09=09return; >>> -=09/* >>> =09 * We're out of time, force a requeue at userret(). >>> =09 */ >>> =09ts->ts_slice =3D sched_slice; With the ts_slice check here before you moved it, removing it might give buggy behaviour closer to SCHED_4BSD. >>> and refusal to use options FULL_PREEMPTION 4-5 years ago, I found that any form of PREMPTION was a pessimization for at least makeworld (since it caused too many context switches). PREEMPTION was needed for the !SMP case, at least partly because of the broken switchticks (switchticks, when it works, gives voluntary yielding by some CPU hogs in the kernel. PREEMPTION, if it works, should do this better). So I used PREEMPTION in the !SMP case and not for the SMP case. I didn't worry about the CPU hogs in the SMP case since it is rare to have more than 1 of them and 1 will use at most 1/2 of a multi-CPU system. >>> But no one has unsubscribed to my letter, my patch helps or not in >>> the case of Core2Duo... >>> There is a suspicion that the problems stem from the sections of >>> code associated with the SMP... >>> Maybe I'm in something wrong, but I want to help in solving this >>> problem ... The main point of SCHED_ULE is to give better affinity for multi-CPU systems. But the `multi' apparently needs to be strictly more than 2 for it to brak even. Bruce --0-1963381402-1323825914=:1563-- From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 06:10:12 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A9571065675; Wed, 14 Dec 2011 06:10:12 +0000 (UTC) (envelope-from kaduk@mit.edu) Received: from dmz-mailsec-scanner-7.mit.edu (DMZ-MAILSEC-SCANNER-7.MIT.EDU [18.7.68.36]) by mx1.freebsd.org (Postfix) with ESMTP id 445EF8FC13; Wed, 14 Dec 2011 06:10:07 +0000 (UTC) X-AuditID: 12074424-b7fae6d000000906-fc-4ee83a3aeeed Received: from mailhub-auth-4.mit.edu ( [18.7.62.39]) by dmz-mailsec-scanner-7.mit.edu (Symantec Messaging Gateway) with SMTP id F8.66.02310.A3A38EE4; Wed, 14 Dec 2011 00:55:06 -0500 (EST) Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103]) by mailhub-auth-4.mit.edu (8.13.8/8.9.2) with ESMTP id pBE5t5Hh021598; Wed, 14 Dec 2011 00:55:06 -0500 Received: from multics.mit.edu (MULTICS.MIT.EDU [18.187.1.73]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id pBE5t3Jf027540 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 14 Dec 2011 00:55:04 -0500 (EST) Received: (from kaduk@localhost) by multics.mit.edu (8.12.9.20060308) id pBE5t3Ku023742; Wed, 14 Dec 2011 00:55:03 -0500 (EST) Date: Wed, 14 Dec 2011 00:55:02 -0500 (EST) From: Benjamin Kaduk To: Brett Glass In-Reply-To: <201112140228.TAA26206@lariat.net> Message-ID: References: <201112090913.CAA03333@lariat.net> <201112120455.VAA03028@lariat.net> <201112140228.TAA26206@lariat.net> User-Agent: Alpine 1.10 (GSO 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrNIsWRmVeSWpSXmKPExsUixG6nrmtl9cLP4NB/AYs5X/6xWjQ3u1q8 afnBYrFg6TM2BxaPGZ/ms3jsnHWX3ePKK5MA5igum5TUnMyy1CJ9uwSujJXbdzIW3GGrOPHr JnMD43zWLkZODgkBE4mVRzezQNhiEhfurWfrYuTiEBLYxyjRtXsJE4SzgVFiwu3DjBDOASaJ W4sus0I4DYwS/862MIL0swhoS8y7tI4NxGYTUJGY+WYjmC0ioCRxe9Fb5i5GDg5mgXiJVcfS QcLCAnoSnXd2gbVyCuhLfD94hx3E5hWwl9gy+TvUsn+MEgu2nQa7T1RAR2L1/iksEEWCEidn PgGzmQUsJc79uc42gVFwFpLULCSpBYxMqxhlU3KrdHMTM3OKU5N1i5MT8/JSi3TN9XIzS/RS U0o3MYLD2EVlB2PzIaVDjAIcjEo8vBNbnvsJsSaWFVfmHmKU5GBSEuVlt3jhJ8SXlJ9SmZFY nBFfVJqTWnyIUYKDWUmEd+1qoHLelMTKqtSifJiUNAeLkjhvw66HfkIC6YklqdmpqQWpRTBZ GQ4OJQleY0ugoYJFqempFWmZOSUIaSYOTpDhPEDDc0BqeIsLEnOLM9Mh8qcYFaXEee1AEgIg iYzSPLheWJp5xSgO9IowbxZIFQ8wRcF1vwIazAQ0OC7lCcjgkkSElFQDo9H1M+a6qT96jLNW eXVu3MtxcE5UnovKjIlVaw7LtmbuOXB//pv2v7seXD2fvNpz4y1mnup/Dqkn5JMyz8458ZMn XKnKd23bvkynvrm8TgWtH053FUp33drByjv/pixXtPqaGWzHZm3P2qofn3WlcdWygs+/xWyu Sl7KTmcxN36nXfMtr+D/ESWW4oxEQy3mouJEAIK+++IOAwAA Cc: stable@freebsd.org, Ed Schouten , Ben Kaduk Subject: Re: Two problems still present in RC3 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 06:10:12 -0000 On Tue, 13 Dec 2011, Brett Glass wrote: > At 12:34 PM 12/13/2011, Ben Kaduk wrote: > >> If I remember correctly, your original message mentioned seeing this >> issue in emacs; have you tried reproducing it in a simpler test case? > > No; when we hit the bug, we moved to SSH with a VT100 emulator so that > we could configure the system. But the system console really should work > as well. I expect to be configuring another system this evening and will Definitely! I use the system console pretty regularly, but I just haven't gotten a chance to pull my laptop up to an RC or -current, recently, so I can't test it myself. > try the script fragment you sent. Thanks. (It turns out one has to consult terminfo(5) to figure out what attributes to pass in to tput; it might be worth adding an EXAMPLES section to tput(1).) -Ben From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 06:37:40 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0BC68106564A for ; Wed, 14 Dec 2011 06:37:40 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: from mx0.hoeg.nl (mx0.hoeg.nl [IPv6:2a01:4f8:101:5343::aa]) by mx1.freebsd.org (Postfix) with ESMTP id C45E48FC15 for ; Wed, 14 Dec 2011 06:37:39 +0000 (UTC) Received: by mx0.hoeg.nl (Postfix, from userid 1000) id 387ED2A28CF1; Wed, 14 Dec 2011 07:37:39 +0100 (CET) Date: Wed, 14 Dec 2011 07:37:39 +0100 From: Ed Schouten To: Brett Glass Message-ID: <20111214063739.GL1771@hoeg.nl> References: <201112090913.CAA03333@lariat.net> <201112120455.VAA03028@lariat.net> <20111213202333.GH1771@hoeg.nl> <201112140251.TAA26350@lariat.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="KIbT1ud6duwZIwNL" Content-Disposition: inline In-Reply-To: <201112140251.TAA26350@lariat.net> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: stable@freebsd.org, Ben Kaduk Subject: Re: Two problems still present in RC3 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 06:37:40 -0000 --KIbT1ud6duwZIwNL Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi Brett, * Brett Glass , 20111214 03:51: > Just start inserting text, deleting characters and lines (Ctrl-D and > Ctrl-K), etc. It's very obvious. That's odd. I've just spent some time deleting huge portions of /etc/services and on my system it doesn't seem to do this. Maybe you can do the following: - run script -t 0 ~/log jove ... - Delete characters - As soon as possible as jove starts to misbehave and the screen contents are wrong, run `killall -9 script' - Send me the log Thanks, --=20 Ed Schouten WWW: http://80386.nl/ --KIbT1ud6duwZIwNL Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQIcBAEBAgAGBQJO6EQzAAoJEG5e2P40kaK7e4EP/RmUTK1L5U70e3XELP2C6U++ DJNNaMogI9hPMk/dyxsHqfjMDfS0fwMAB6ve59ES37Er9graPwCOL5cbZW8QpsQn zKY246d9MCBY3CWLVoxnkmQwDdRbuo7wBzH4j+dwv5QLFzXQ5pfIuF4d8cRJ8Acs Pw+XI/d0oipZvEvXeaQRE1u3mn+labWpDO4MGFRSdyFZ27/jeLeZk19/laUy8ySz uQ4O4PzH6jTQRVtLjkZ1CdIv1PPBHKVRyslS1o93quDSKkENlESIba02dcRBXTK1 T/QnlSjZBDtTOnv/0EKROhIdplFOP5uE81lfsMmE2sOiwhqapwN/t+06OBmVvwGz uem84pRMInIIVYSizeBtOvQ/Zk/qnRaShKbEFogInNnj3Maw8AWwgJooOU6V3JrG YnTIgoLkNWEPrZOSEcoRRhcjSbYZDsIhc101qI6CMP0UkkVSLIRDs3PKJpqg63WN At9+SBBA28ONMF31yGrQ7UENwHYcl+GfTqSFPJoGTad3GX65AT687h1L1J0ciYXM ay/wbtQDI7Olz8GPQpkPs0EUGilAku9+ylw84wxJHRAvUe8lXdEbvVJXnjFeeCMM rmQg7eY4+T01yNoSCulV7P4hNOxEqD7FlH4LkcMG2Cq4nDOziDuHHPfmMs2kIA+E z9z3w7xKyWdO82t3wPCj =w+mU -----END PGP SIGNATURE----- --KIbT1ud6duwZIwNL-- From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 08:41:08 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34E78106566B for ; Wed, 14 Dec 2011 08:41:08 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) by mx1.freebsd.org (Postfix) with ESMTP id B58D98FC08 for ; Wed, 14 Dec 2011 08:41:07 +0000 (UTC) Received: from hugo10.ka.punkt.de ([217.29.45.10]) by gate1.intern.punkt.de with ESMTP id pBE8TrpY064701 for ; Wed, 14 Dec 2011 09:29:53 +0100 (CET) Received: from [217.29.45.158] ([217.29.45.158]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id pBE8Tr38036655 for ; Wed, 14 Dec 2011 09:29:53 +0100 (CET) (envelope-from hausen@punkt.de) From: "Patrick M. Hausen" Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Date: Wed, 14 Dec 2011 09:29:52 +0100 Message-Id: To: FreeBSD Stable Mime-Version: 1.0 (Apple Message framework v1251.1) X-Mailer: Apple Mail (2.1251.1) Subject: Hot-changing a failed HDD with ahci.ko X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 08:41:08 -0000 Hi, all, while most cheap servers with SATA disks are not really hot-plug capable, changing a failed disk (either gmirror or zfs) was possible without a reboot by executing e.g. if ad4 failed: atacontrol detach ata2 atacontrol attach ata2 What is the proper equivalent for ahci, ada0 and camcontrol? Stop unit commands seem not to work with SATA disks, so I tried: -> system logs about lost device, so far so good camcontrol reset 1 camcontrol devlist -> disk still not there camcontrol rescan 1 -> command hangs shutdown -r now -> system panics, eventually reboots I can provide details about the panic if someone is interested, but maybe there is a proper procedure already, which I simply missed. System is RELENG_8_2 amd64. ahci0: port = 0xf090-0xf097,0xf080-0xf083,0xf070-0xf077,0xf060-0xf063,0xf020-0xf03f = mem 0xfb921000-0xfb9217ff irq 19 at device 31.2 on pci0 ada0 at ahcich0 bus 0 scbus1 target 0 lun 0 ada0: ATA-8 SATA 1.x device ada0: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada1 at ahcich1 bus 0 scbus2 target 0 lun 0 ada1: ATA-8 SATA 1.x device ada1: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) Thanks, Patrick --=20 punkt.de GmbH * Kaiserallee 13a * 76133 Karlsruhe Tel. 0721 9109 0 * Fax 0721 9109 100 info@punkt.de http://www.punkt.de Gf: J=FCrgen Egeling AG Mannheim 108285 From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 09:26:27 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E4BB1065676 for ; Wed, 14 Dec 2011 09:26:27 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta11.emeryville.ca.mail.comcast.net (qmta11.emeryville.ca.mail.comcast.net [76.96.27.211]) by mx1.freebsd.org (Postfix) with ESMTP id 44E838FC0C for ; Wed, 14 Dec 2011 09:26:26 +0000 (UTC) Received: from omta23.emeryville.ca.mail.comcast.net ([76.96.30.90]) by qmta11.emeryville.ca.mail.comcast.net with comcast id 8xQt1i0021wfjNsABxSKl7; Wed, 14 Dec 2011 09:26:19 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta23.emeryville.ca.mail.comcast.net with comcast id 8xJa1i0061t3BNj8jxJbY5; Wed, 14 Dec 2011 09:18:35 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 9059B102C19; Wed, 14 Dec 2011 01:26:24 -0800 (PST) Date: Wed, 14 Dec 2011 01:26:24 -0800 From: Jeremy Chadwick To: "Patrick M. Hausen" Message-ID: <20111214092624.GA96153@icarus.home.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD Stable Subject: Re: Hot-changing a failed HDD with ahci.ko X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 09:26:27 -0000 On Wed, Dec 14, 2011 at 09:29:52AM +0100, Patrick M. Hausen wrote: > Hi, all, > > while most cheap servers with SATA disks are not really hot-plug > capable, changing a failed disk (either gmirror or zfs) was possible > without a reboot by executing e.g. if ad4 failed: > > atacontrol detach ata2 > > atacontrol attach ata2 > > What is the proper equivalent for ahci, ada0 and camcontrol? None is needed: yank the disk, reinsert, wait a few seconds, done. Validation, with full output, hardware, etc: http://koitsu.wordpress.com/2010/07/22/freebsd-and-zfs-hot-swapping-sata-disks-with-ahci/ I've made videos to demonstrate this as well, but need to edit them and upload them. > Stop unit commands seem not to work with SATA disks, so I > tried: > > > -> system logs about lost device, so far so good > > camcontrol reset 1 > camcontrol devlist > -> disk still not there > camcontrol rescan 1 > -> command hangs > > shutdown -r now > -> system panics, eventually reboots Before you yanked the disk, were any non-ZFS filesystems mounted? This sounds similar to what happens if you were to yank a classic SATA disk from a non-AHCI system, or under ata(4), without detaching first. Or, on some systems, when SATA disks are yanked without use of a hot-swap backplane. > I can provide details about the panic if someone is interested, > but maybe there is a proper procedure already, which I simply missed. > > System is RELENG_8_2 amd64. > ahci0: port 0xf090-0xf097,0xf080-0xf083,0xf070-0xf077,0xf060-0xf063,0xf020-0xf03f mem 0xfb921000-0xfb9217ff irq 19 at device 31.2 on pci0 > ada0 at ahcich0 bus 0 scbus1 target 0 lun 0 > ada0: ATA-8 SATA 1.x device > ada0: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes) > ada0: Command Queueing enabled > ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) > ada1 at ahcich1 bus 0 scbus2 target 0 lun 0 > ada1: ATA-8 SATA 1.x device > ada1: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes) > ada1: Command Queueing enabled > ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) You might try booting RELENG_9 (which has ahci.ko as the default, so no need to mess about) on a LiveCD or equivalent and attempt the same thing. I'm left wondering if there's some stuff in RELENG_8 (not a typo compared to the above RELENG_9 reference) that you do not have in RELENG_8_2. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 09:52:16 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4C7CD106564A for ; Wed, 14 Dec 2011 09:52:16 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) by mx1.freebsd.org (Postfix) with ESMTP id D2BD88FC12 for ; Wed, 14 Dec 2011 09:52:15 +0000 (UTC) Received: from hugo10.ka.punkt.de ([217.29.45.10]) by gate1.intern.punkt.de with ESMTP id pBE9qEpZ066479; Wed, 14 Dec 2011 10:52:14 +0100 (CET) Received: from [217.29.45.158] ([217.29.45.158]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id pBE9qEh6040163; Wed, 14 Dec 2011 10:52:14 +0100 (CET) (envelope-from hausen@punkt.de) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=iso-8859-1 From: "Patrick M. Hausen" In-Reply-To: <20111214092624.GA96153@icarus.home.lan> Date: Wed, 14 Dec 2011 10:52:13 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20111214092624.GA96153@icarus.home.lan> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1251.1) Cc: FreeBSD Stable Subject: Re: Hot-changing a failed HDD with ahci.ko X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 09:52:16 -0000 Hi! Am 14.12.2011 um 10:26 schrieb Jeremy Chadwick: >> What is the proper equivalent for ahci, ada0 and camcontrol? >=20 > None is needed: yank the disk, reinsert, wait a few seconds, done. > Validation, with full output, hardware, etc: >=20 > = http://koitsu.wordpress.com/2010/07/22/freebsd-and-zfs-hot-swapping-sata-d= isks-with-ahci/ Yank the disk: (ada0:ahcich0:0:0:0): lost device Reinsert - nothing happens. >> shutdown -r now >> -> system panics, eventually reboots >=20 > Before you yanked the disk, were any non-ZFS filesystems mounted? Yes - my fault. I had an active swap partition on the disk which = perfectly explains the panic. > You might try booting RELENG_9 (which has ahci.ko as the default, so = no > need to mess about) on a LiveCD or equivalent and attempt the same > thing. I'm left wondering if there's some stuff in RELENG_8 (not a = typo > compared to the above RELENG_9 reference) that you do not have in > RELENG_8_2. I'll try upgrading to RELENG_8 first and report the results. Side note: ada0: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes) I noticed that while preparing the last mail and removed those little = jumpers limiting my hard drives ;-) Thanks, Patrick --=20 punkt.de GmbH * Kaiserallee 13a * 76133 Karlsruhe Tel. 0721 9109 0 * Fax 0721 9109 100 info@punkt.de http://www.punkt.de Gf: J=FCrgen Egeling AG Mannheim 108285 From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 11:06:50 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F1D3106564A; Wed, 14 Dec 2011 11:06:50 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (ip-3-2-0-2.r20.asbnva02.us.ce.gin.ntt.net [IPv6:2001:418:0:5000::16]) by mx1.freebsd.org (Postfix) with ESMTP id CB96E8FC15; Wed, 14 Dec 2011 11:06:49 +0000 (UTC) Received: from wonderland.m5p.com (wonderland.m5p.com [IPv6:2001:418:3fd::19]) by mailhost.m5p.com (8.14.4/8.14.4) with ESMTP id pBEB6hRV008765; Wed, 14 Dec 2011 06:06:48 -0500 (EST) (envelope-from george+freebsd@m5p.com) Message-ID: <4EE88343.2050302@m5p.com> Date: Wed, 14 Dec 2011 06:06:43 -0500 From: George Mitchell User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111127 Thunderbird/8.0 MIME-Version: 1.0 To: Attilio Rao , freebsd-stable@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> In-Reply-To: <4EE2AE64.9060802@m5p.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (mailhost.m5p.com [IPv6:2001:418:3fd::f7]); Wed, 14 Dec 2011 06:06:48 -0500 (EST) X-Scanned-By: MIMEDefang 2.72 on IPv6:2001:418:3fd::f7 Cc: Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 11:06:50 -0000 On 12/09/11 19:57, George Mitchell wrote: > On 12/09/11 10:17, Attilio Rao wrote: >> [...] >> More precisely I'd be interested in KTR traces. >> To be even more precise: >> With a completely stable GENERIC configuration (or otherwise please >> post your kernel config) please add the following: >> options KTR >> options KTR_ENTRIES=262144 >> options KTR_COMPILE=(KTR_SCHED) >> options KTR_MASK=(KTR_SCHED) >> >> While you are in the middle of the slow-down (so once it is well >> established) please do: >> # sysclt debug.ktr.cpumask="" > > wonderland# sysctl debug.ktr.cpumask="" > debug.ktr.cpumask: ffffffffffffffff > sysctl: debug.ktr.cpumask: Invalid argument > >> >> In the end go with: >> # ktrdump -ctf> ktr-ule-problem.out > > It's 44MB, so it's at http://www.m5p.com/~george/ktr-ule-problem.out There have been 22 downloads of this file so far; does anyone who looked at it have any results to report? Dear Secret Masters of FreeBSD: Can we have a decision on whether to change back to SCHED_4BSD while SCHED_ULE gets properly fixed? -- George Mitchell > >> >> and send the file to this mailing list. >> >> Thanks, >> Attilio >> >> > > I hope this helps. -- George Mitchell From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 12:49:31 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9483F106566B for ; Wed, 14 Dec 2011 12:49:31 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) by mx1.freebsd.org (Postfix) with ESMTP id 208118FC18 for ; Wed, 14 Dec 2011 12:49:30 +0000 (UTC) Received: from hugo10.ka.punkt.de ([217.29.45.10]) by gate1.intern.punkt.de with ESMTP id pBECnTdd070146; Wed, 14 Dec 2011 13:49:29 +0100 (CET) Received: from [217.29.45.158] ([217.29.45.158]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id pBECnT6x047140; Wed, 14 Dec 2011 13:49:29 +0100 (CET) (envelope-from hausen@punkt.de) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=iso-8859-1 From: "Patrick M. Hausen" In-Reply-To: Date: Wed, 14 Dec 2011 13:49:29 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20111214092624.GA96153@icarus.home.lan> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1251.1) Cc: FreeBSD Stable Subject: Re: Hot-changing a failed HDD with ahci.ko X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 12:49:31 -0000 Hi! Am 14.12.2011 um 10:52 schrieb Patrick M. Hausen: > Yes - my fault. I had an active swap partition on the disk which = perfectly > explains the panic. I replaced that one with a gmirror device, now. >> You might try booting RELENG_9 (which has ahci.ko as the default, so = no >> need to mess about) on a LiveCD or equivalent and attempt the same >> thing. I'm left wondering if there's some stuff in RELENG_8 (not a = typo >> compared to the above RELENG_9 reference) that you do not have in >> RELENG_8_2. >=20 > I'll try upgrading to RELENG_8 first and report the results. datatomb2# uname -a FreeBSD datatomb2.pluspunkthosting.de 8.2-STABLE FreeBSD 8.2-STABLE #0: = Wed Dec 14 12:44:07 CET 2011 = root@datatomb2.pluspunkthosting.de:/usr/obj/usr/src/sys/GENERIC amd64 Yank disk: ada1:GEOM_MIRROR: Device swap: provider ada1p2 disconnected. (ahcich1:0:0:0): lost device (ada1:ahcich1:0:0:0): removing device entry Put it back in: ada1 at ahcich1 bus 0 scbus2 target 0 lun 0 ada1: ATA-8 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) And I get ZFS v28, too :-)) Thanks and kind regards, Patrick --=20 punkt.de GmbH * Kaiserallee 13a * 76133 Karlsruhe Tel. 0721 9109 0 * Fax 0721 9109 100 info@punkt.de http://www.punkt.de Gf: J=FCrgen Egeling AG Mannheim 108285 From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 16:59:53 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D732A1065676; Wed, 14 Dec 2011 16:59:53 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id 919838FC19; Wed, 14 Dec 2011 16:59:53 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.5/8.14.4) with ESMTP id pBEGxlPE000375; Wed, 14 Dec 2011 11:59:47 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4EE8D607.1000504@sentex.net> Date: Wed, 14 Dec 2011 11:59:51 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: mdf@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.71 on IPv6:2607:f3e0:0:1::12 Cc: Ivan Klymenko , Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 16:59:54 -0000 On 12/13/2011 7:01 PM, mdf@freebsd.org wrote: > > Has anyone experiencing problems tried to set sysctl kern.sched.steal_thresh=1 ? > > I don't remember what our specific problem at $WORK was, perhaps it > was just interrupt threads not getting serviced fast enough, but we've > hard-coded this to 1 and removed the code that sets it in > sched_initticks(). The same effect should be had by setting the > sysctl after a box is up. FWIW, this does impact the performance of pbzip2 on an i7. Using a 1.1G file pbzip2 -v -c big > /dev/null with burnP6 running in the background, sysctl kern.sched.steal_thresh=1 vs sysctl kern.sched.steal_thresh=3 N Min Max Median Avg Stddev x 10 38.005022 38.42238 38.194648 38.165052 0.15546188 + 9 38.695417 40.595544 39.392127 39.435384 0.59814114 Difference at 95.0% confidence 1.27033 +/- 0.412636 3.32852% +/- 1.08119% (Student's t, pooled s = 0.425627) a value of 1 is *slightly* faster. -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 17:34:38 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8475E106564A; Wed, 14 Dec 2011 17:34:38 +0000 (UTC) (envelope-from ache@vniz.net) Received: from vniz.net (vniz.net [194.87.13.69]) by mx1.freebsd.org (Postfix) with ESMTP id EC28B8FC0C; Wed, 14 Dec 2011 17:34:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by vniz.net (8.14.5/8.14.5) with ESMTP id pBEHYaPU041992; Wed, 14 Dec 2011 21:34:36 +0400 (MSK) (envelope-from ache@vniz.net) Received: (from ache@localhost) by localhost (8.14.5/8.14.5/Submit) id pBEHYZP4041991; Wed, 14 Dec 2011 21:34:35 +0400 (MSK) (envelope-from ache) Date: Wed, 14 Dec 2011 21:34:35 +0400 From: Andrey Chernov To: Adrian Chadd Message-ID: <20111214173435.GA41893@vniz.net> Mail-Followup-To: Andrey Chernov , Adrian Chadd , Ivan Klymenko , Doug Barton , "O. Hartmann" , Current FreeBSD , freebsd-stable@FreeBSD.ORG, freebsd-performance@FreeBSD.ORG References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213090051.GA3339@vniz.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Ivan Klymenko , Doug Barton , freebsd-stable@FreeBSD.ORG, "O. Hartmann" , Current FreeBSD , freebsd-performance@FreeBSD.ORG Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 17:34:38 -0000 On Tue, Dec 13, 2011 at 02:22:48AM -0800, Adrian Chadd wrote: > On 13 December 2011 01:00, Andrey Chernov wrote: > > >> If the algorithm ULE does not contain problems - it means the problem > >> has Core2Duo, or in a piece of code that uses the ULE scheduler. > > > > I observe ULE interactivity slowness even on single core machine (Pentium > > 4) in very visible places, like 'ps ax' output stucks in the middle by ~1 > > second. When I switch back to SHED_4BSD, all slowness is gone. > > Are you able to provide KTR traces of the scheduler results? Something > that can be fed to schedgraph? Sorry, this machine is not mine anymore. I try SCHED_ULE on Core 2 Duo instead and don't notice this effect, but it is overall pretty fast comparing to that Pentium 4. -- http://ache.vniz.net/ From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 17:54:16 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E97E106566B for ; Wed, 14 Dec 2011 17:54:16 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 0C7858FC14 for ; Wed, 14 Dec 2011 17:54:15 +0000 (UTC) Received: by vbbfr13 with SMTP id fr13so1354394vbb.13 for ; Wed, 14 Dec 2011 09:54:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ZuWRfIJTDcJjLLxqXi8O0dmeMp6HDI/8ce/pL4l/RO8=; b=c+mc/0c9xybr7L7/5yTOZ/OZF1XZ7INBnbndAFDEn9KjlM8kxsdBbZ1LhNNVkZhXta jCVMiF3o8iAYLg9jUWgpNCxk1HuzShPvJEuVY1daZqCOTslTaD2kG63dzafEkbas9vmO 1h2IsOunrItSow+8+XqftZFzSPhlNPtqGn+Nw= MIME-Version: 1.0 Received: by 10.52.26.110 with SMTP id k14mr5686343vdg.75.1323885255247; Wed, 14 Dec 2011 09:54:15 -0800 (PST) Received: by 10.52.162.202 with HTTP; Wed, 14 Dec 2011 09:54:15 -0800 (PST) In-Reply-To: <4EE88343.2050302@m5p.com> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> Date: Wed, 14 Dec 2011 17:54:15 +0000 Message-ID: From: Tom Evans To: George Mitchell Content-Type: text/plain; charset=UTF-8 Cc: Attilio Rao , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 17:54:16 -0000 On Wed, Dec 14, 2011 at 11:06 AM, George Mitchell wrote: > > Dear Secret Masters of FreeBSD: Can we have a decision on whether to > change back to SCHED_4BSD while SCHED_ULE gets properly fixed? > Please do not do this. This thread has shown that ULE performs poorly in very specific scenarios where the server is loaded with NCPU+1 CPU bound processes, and brought forward more complaints about interactivity in X (I've never noticed this, and use a FreeBSD desktop daily). On the other hand, we have very many benchmarks showing how poorly 4BSD scales on things like postgresql. We get much more load out of our 8.1 ULE DB and web servers than we do out of our 7.0 ones. It's easy to look at what you do and say "well, what suits my environment is clearly the best default", but I think there are probably more users typically running IO bound processes than CPU bound processes. I believe the correct thing to do is to put some extra documentation into the handbook about scheduler choice, noting the potential issues with loading NCPU+1 CPU bound processes. Perhaps making it easier to switch scheduler would also help? Cheers Tom References: http://people.freebsd.org/~kris/scaling/mysql-freebsd.png http://suckit.blog.hu/2009/10/05/freebsd_8_is_it_worth_to_upgrade From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 17:55:53 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34EE01065679; Wed, 14 Dec 2011 17:55:53 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm2.ukr.net (fsm2.ukr.net [195.214.192.121]) by mx1.freebsd.org (Postfix) with ESMTP id D3CA08FC17; Wed, 14 Dec 2011 17:55:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=H5HuC6f1zLZn/iZE9rSzVBa6GimuIdr5WV8rZoyf9bw=; b=P0a5+C+QspdhOkFWZHN5MJx+rooqxJQlNCwltxzVAO88ay7bFsjjzeY2MqGt/+9YIJ/mC0vkLkMoFTbNcYNuJ3bKdvQo7vwgj/Xbngp+M2usczXPalskGQ2vzlbYNGVUVyptOud0DluvzMXe3sV+ttAaRd1sR8dQCWPYAdOAQCA=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm2.ukr.net with esmtpsa ID 1Rat3X-000O37-NU ; Wed, 14 Dec 2011 19:55:51 +0200 Date: Wed, 14 Dec 2011 19:55:49 +0200 From: Ivan Klymenko To: Andrey Chernov Message-ID: <20111214195549.415196f0@nonamehost.> In-Reply-To: <20111214173435.GA41893@vniz.net> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213090051.GA3339@vniz.net> <20111214173435.GA41893@vniz.net> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Adrian Chadd , Doug Barton , freebsd-stable@FreeBSD.ORG, "O. Hartmann" , Current FreeBSD , freebsd-performance@FreeBSD.ORG Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 17:55:53 -0000 =D0=92 Wed, 14 Dec 2011 21:34:35 +0400 Andrey Chernov =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > On Tue, Dec 13, 2011 at 02:22:48AM -0800, Adrian Chadd wrote: > > On 13 December 2011 01:00, Andrey Chernov wrote: > >=20 > > >> If the algorithm ULE does not contain problems - it means the > > >> problem has Core2Duo, or in a piece of code that uses the ULE > > >> scheduler. > > > > > > I observe ULE interactivity slowness even on single core machine > > > (Pentium 4) in very visible places, like 'ps ax' output stucks in > > > the middle by ~1 second. When I switch back to SHED_4BSD, all > > > slowness is gone. > >=20 > > Are you able to provide KTR traces of the scheduler results? > > Something that can be fed to schedgraph? >=20 > Sorry, this machine is not mine anymore. I try SCHED_ULE on Core 2 > Duo instead and don't notice this effect, but it is overall pretty > fast comparing to that Pentium 4. >=20 Give me, please, detailed instructions on how to do it - I'll do it ... Be a shame if this the theme is will end again just only the discussions ... :( From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 18:11:51 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6AF031065675 for ; Wed, 14 Dec 2011 18:11:51 +0000 (UTC) (envelope-from andrey@zonov.org) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id F3B7D8FC15 for ; Wed, 14 Dec 2011 18:11:50 +0000 (UTC) Received: by eekc50 with SMTP id c50so1329663eek.13 for ; Wed, 14 Dec 2011 10:11:50 -0800 (PST) Received: by 10.180.103.170 with SMTP id fx10mr6834682wib.56.1323886309802; Wed, 14 Dec 2011 10:11:49 -0800 (PST) Received: from [10.254.254.77] (ppp94-29-56-7.pppoe.spdop.ru. [94.29.56.7]) by mx.google.com with ESMTPS id fi11sm5127714wbb.9.2011.12.14.10.11.48 (version=SSLv3 cipher=OTHER); Wed, 14 Dec 2011 10:11:49 -0800 (PST) Message-ID: <4EE8E6E3.7050202@zonov.org> Date: Wed, 14 Dec 2011 22:11:47 +0400 From: Andrey Zonov User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.24) Gecko/20100228 Thunderbird/2.0.0.24 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Jeremy Chadwick References: <4EE7BF77.5000504@zonov.org> <20111213221501.GA85563@icarus.home.lan> In-Reply-To: <20111213221501.GA85563@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: directory listing hangs in "ufs" state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 18:11:51 -0000 Hi Jeremy, This is not hardware problem, I've already checked that. I also ran fsck today and got no errors. After some more exploration of how mongodb works, I found that then listing hangs, one of mongodb thread is in "biowr" state for a long time. It periodically calls msync(MS_SYNC) accordingly to ktrace out. If I'll remove msync() calls from mongodb, how often data will be sync by OS? -- Andrey Zonov On 14.12.2011 2:15, Jeremy Chadwick wrote: > On Wed, Dec 14, 2011 at 01:11:19AM +0400, Andrey Zonov wrote: >> >> Have you any ideas what is going on? or how to catch the problem? > > Assuming this isn't a file on the root filesystem, try booting the > machine in single-user mode and using "fsck -f" on the filesystem in > question. > > Can you verify there's no problems with the disk this file lives on as > well (smartctl -a /dev/disk)? I'm doubting this is the problem, but > thought I'd mention it. > From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 18:22:57 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 164791065678 for ; Wed, 14 Dec 2011 18:22:57 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta13.westchester.pa.mail.comcast.net (qmta13.westchester.pa.mail.comcast.net [76.96.59.243]) by mx1.freebsd.org (Postfix) with ESMTP id B76D28FC0C for ; Wed, 14 Dec 2011 18:22:56 +0000 (UTC) Received: from omta17.westchester.pa.mail.comcast.net ([76.96.62.89]) by qmta13.westchester.pa.mail.comcast.net with comcast id 96CB1i0021vXlb85D6NwG3; Wed, 14 Dec 2011 18:22:56 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta17.westchester.pa.mail.comcast.net with comcast id 96Nu1i00d1t3BNj3d6NuTV; Wed, 14 Dec 2011 18:22:56 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id B9510102C19; Wed, 14 Dec 2011 10:22:52 -0800 (PST) Date: Wed, 14 Dec 2011 10:22:52 -0800 From: Jeremy Chadwick To: Andrey Zonov Message-ID: <20111214182252.GA5176@icarus.home.lan> References: <4EE7BF77.5000504@zonov.org> <20111213221501.GA85563@icarus.home.lan> <4EE8E6E3.7050202@zonov.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE8E6E3.7050202@zonov.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: directory listing hangs in "ufs" state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 18:22:57 -0000 On Wed, Dec 14, 2011 at 10:11:47PM +0400, Andrey Zonov wrote: > Hi Jeremy, > > This is not hardware problem, I've already checked that. I also ran > fsck today and got no errors. > > After some more exploration of how mongodb works, I found that then > listing hangs, one of mongodb thread is in "biowr" state for a long > time. It periodically calls msync(MS_SYNC) accordingly to ktrace > out. > > If I'll remove msync() calls from mongodb, how often data will be > sync by OS? > > -- > Andrey Zonov > > On 14.12.2011 2:15, Jeremy Chadwick wrote: > >On Wed, Dec 14, 2011 at 01:11:19AM +0400, Andrey Zonov wrote: > >> > >>Have you any ideas what is going on? or how to catch the problem? > > > >Assuming this isn't a file on the root filesystem, try booting the > >machine in single-user mode and using "fsck -f" on the filesystem in > >question. > > > >Can you verify there's no problems with the disk this file lives on as > >well (smartctl -a /dev/disk)? I'm doubting this is the problem, but > >thought I'd mention it. I have no real answer, I'm sorry. msync(2) indicates it's effectively deprecated (see BUGS). It looks like this is effectively a mmap-version of fsync(2). I'm extremely confused by this problem. What you're describing above is that the process is "stuck in biowr state for a long time", but what you stated originally was that the process was "stuck in ufs state for a few minutes": > I've got STABLE-8 (r221983) with mongodb-1.8.1 installed on it. A > couple days ago I observed that listing of mongodb directory stuck in > a few minutes in "ufs" state. Can we narrow down what we're talking about here? Does the process actually deadlock? Or are you concerned about performance implications? I know nothing about this "mongodb" software, but the reason it's calling msync() is because it wants to try and ensure that the data it changed in an mmap()-mapped page to be reflected (fully written) on the disk. This behaviour is fairly common within database software, but "how often" the software chooses to do this is entirely a design implementation choice by the authors. Meaning: if mongodb is either 1) continually calling msync(), or 2) waiting for too long a period of time before calling msync(), performance within the process will suffer. #1 could result in overall bad performance, while #2 could result in a process that's spending a lot of time doing I/O (flushing to disk) and therefore appears "deadlocked" when in fact the kernel/subsystems are doing exactly what they were told to do. Removing the msync() call could result in inconsistent data (possibly non-recoverable) if the mongodb software crashes or if some other piece (thread or child? Not sure) expects to open a new fd on that file which has mmap()'d data. This is about all I know. I would love to be able to tell you "consider a different database" but that seems like an excuse rather than an actual solution. I guess if all you're seeing is the process "stall" for long periods of time, but recover normally, then I would open up a support ticket with the mongodb folks to discuss performance. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 18:53:26 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40FFB106566B for ; Wed, 14 Dec 2011 18:53:26 +0000 (UTC) (envelope-from alan.l.cox@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 167D58FC12 for ; Wed, 14 Dec 2011 18:53:25 +0000 (UTC) Received: by dakp5 with SMTP id p5so1217512dak.13 for ; Wed, 14 Dec 2011 10:53:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=AdBUXgDs4PJRE/UNIV7mFtxta4pmkCEUELZ/hh8/cTk=; b=lmjrWfpRZqZIpH6gtnCZupsfpPYp9opa/VQa6CwQhvG+dIfwYHe5je+EwvyKb2dEV7 t53rpEkNuNjb8fjbc6JnD8E12TduQIPMeRdpvadw65hO9jixQaHtyGhCN1oy+pAhm1fg RbdQKO6V9f2Xsh8/pYT9Q1/N+XN6dF22LjGyQ= MIME-Version: 1.0 Received: by 10.68.73.228 with SMTP id o4mr4891120pbv.2.1323888805524; Wed, 14 Dec 2011 10:53:25 -0800 (PST) Received: by 10.142.154.16 with HTTP; Wed, 14 Dec 2011 10:53:25 -0800 (PST) In-Reply-To: <20111214182252.GA5176@icarus.home.lan> References: <4EE7BF77.5000504@zonov.org> <20111213221501.GA85563@icarus.home.lan> <4EE8E6E3.7050202@zonov.org> <20111214182252.GA5176@icarus.home.lan> Date: Wed, 14 Dec 2011 12:53:25 -0600 Message-ID: From: Alan Cox To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Andrey Zonov , freebsd-stable@freebsd.org Subject: Re: directory listing hangs in "ufs" state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: alc@freebsd.org List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 18:53:26 -0000 On Wed, Dec 14, 2011 at 12:22 PM, Jeremy Chadwick wrote: > On Wed, Dec 14, 2011 at 10:11:47PM +0400, Andrey Zonov wrote: > > Hi Jeremy, > > > > This is not hardware problem, I've already checked that. I also ran > > fsck today and got no errors. > > > > After some more exploration of how mongodb works, I found that then > > listing hangs, one of mongodb thread is in "biowr" state for a long > > time. It periodically calls msync(MS_SYNC) accordingly to ktrace > > out. > > > > If I'll remove msync() calls from mongodb, how often data will be > > sync by OS? > > > > -- > > Andrey Zonov > > > > On 14.12.2011 2:15, Jeremy Chadwick wrote: > > >On Wed, Dec 14, 2011 at 01:11:19AM +0400, Andrey Zonov wrote: > > >> > > >>Have you any ideas what is going on? or how to catch the problem? > > > > > >Assuming this isn't a file on the root filesystem, try booting the > > >machine in single-user mode and using "fsck -f" on the filesystem in > > >question. > > > > > >Can you verify there's no problems with the disk this file lives on as > > >well (smartctl -a /dev/disk)? I'm doubting this is the problem, but > > >thought I'd mention it. > > I have no real answer, I'm sorry. msync(2) indicates it's effectively > deprecated (see BUGS). It looks like this is effectively a mmap-version > of fsync(2). > > Yikes, I just looked at this man page. I'm afraid that the text in the BUGS section is highly misleading. The MS_INVALIDATE option should be obsolete for the reason given there. Under a strict reading of the applicable standard, FreeBSD could implement this option as a NOP. However, we treat it something like madvise(MADV_DONTNEED|FREE). In contrast, MS_SYNC is definitely not obsolete. Alan P.S. If someone wants to take a crack at fixing this man page, contact me off list. From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 19:00:13 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E93A1065676 for ; Wed, 14 Dec 2011 19:00:13 +0000 (UTC) (envelope-from andrey@zonov.org) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id DF6288FC15 for ; Wed, 14 Dec 2011 19:00:12 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so2244966wgb.31 for ; Wed, 14 Dec 2011 11:00:11 -0800 (PST) Received: by 10.216.209.228 with SMTP id s78mr1829355weo.26.1323889211469; Wed, 14 Dec 2011 11:00:11 -0800 (PST) Received: from [10.254.254.77] (ppp94-29-56-7.pppoe.spdop.ru. [94.29.56.7]) by mx.google.com with ESMTPS id hn15sm4829051wib.22.2011.12.14.11.00.10 (version=SSLv3 cipher=OTHER); Wed, 14 Dec 2011 11:00:11 -0800 (PST) Message-ID: <4EE8F239.4070000@zonov.org> Date: Wed, 14 Dec 2011 23:00:09 +0400 From: Andrey Zonov User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.24) Gecko/20100228 Thunderbird/2.0.0.24 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: alc@freebsd.org References: <4EE7BF77.5000504@zonov.org> <20111213221501.GA85563@icarus.home.lan> <4EE8E6E3.7050202@zonov.org> <20111214182252.GA5176@icarus.home.lan> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Alan Cox , freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: directory listing hangs in "ufs" state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 19:00:13 -0000 On 14.12.2011 22:53, Alan Cox wrote: > On Wed, Dec 14, 2011 at 12:22 PM, Jeremy Chadwick > > wrote: > > On Wed, Dec 14, 2011 at 10:11:47PM +0400, Andrey Zonov wrote: > > Hi Jeremy, > > > > This is not hardware problem, I've already checked that. I also ran > > fsck today and got no errors. > > > > After some more exploration of how mongodb works, I found that then > > listing hangs, one of mongodb thread is in "biowr" state for a long > > time. It periodically calls msync(MS_SYNC) accordingly to ktrace > > out. > > > > If I'll remove msync() calls from mongodb, how often data will be > > sync by OS? > > > > -- > > Andrey Zonov > > > > On 14.12.2011 2:15, Jeremy Chadwick wrote: > > >On Wed, Dec 14, 2011 at 01:11:19AM +0400, Andrey Zonov wrote: > > >> > > >>Have you any ideas what is going on? or how to catch the problem? > > > > > >Assuming this isn't a file on the root filesystem, try booting the > > >machine in single-user mode and using "fsck -f" on the filesystem in > > >question. > > > > > >Can you verify there's no problems with the disk this file lives > on as > > >well (smartctl -a /dev/disk)? I'm doubting this is the problem, but > > >thought I'd mention it. > > I have no real answer, I'm sorry. msync(2) indicates it's effectively > deprecated (see BUGS). It looks like this is effectively a mmap-version > of fsync(2). > > > Yikes, I just looked at this man page. I'm afraid that the text in the > BUGS section is highly misleading. The MS_INVALIDATE option should be > obsolete for the reason given there. Under a strict reading of the > applicable standard, FreeBSD could implement this option as a NOP. > However, we treat it something like madvise(MADV_DONTNEED|FREE). In > contrast, MS_SYNC is definitely not obsolete. > > Alan > > P.S. If someone wants to take a crack at fixing this man page, contact > me off list. > Please don't remove support for MS_INVALIDATE, this is only one way to purge disk cache. MADV_DONTNEED does nothing here in my experience. -- Andrey Zonov From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 19:29:12 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9946A106564A; Wed, 14 Dec 2011 19:29:12 +0000 (UTC) (envelope-from linimon@lonesome.com) Received: from mail.soaustin.net (pancho.soaustin.net [76.74.250.40]) by mx1.freebsd.org (Postfix) with ESMTP id 79A2A8FC12; Wed, 14 Dec 2011 19:29:12 +0000 (UTC) Received: by mail.soaustin.net (Postfix, from userid 502) id 4F2125619E; Wed, 14 Dec 2011 13:11:46 -0600 (CST) Date: Wed, 14 Dec 2011 13:11:46 -0600 From: Mark Linimon To: George Mitchell Message-ID: <20111214191146.GA14062@lonesome.com> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE88343.2050302@m5p.com> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: Attilio Rao , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 19:29:12 -0000 I'm not on the Release Engineering Team, and in fact don't have a src commit bit ... but this close to a major release, no, it's too late to change the default. mcl From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 19:46:35 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC677106564A for ; Wed, 14 Dec 2011 19:46:35 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from email2.allantgroup.com (email2.emsphone.com [199.67.51.116]) by mx1.freebsd.org (Postfix) with ESMTP id 5711E8FC0A for ; Wed, 14 Dec 2011 19:46:35 +0000 (UTC) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by email2.allantgroup.com (8.14.4/8.14.4) with ESMTP id pBEJkVho094246 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 14 Dec 2011 13:46:32 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.5/8.14.5) with ESMTP id pBEJkVXG073800 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 14 Dec 2011 13:46:31 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.5/8.14.5/Submit) id pBEJkVZA073799; Wed, 14 Dec 2011 13:46:31 -0600 (CST) (envelope-from dan) Date: Wed, 14 Dec 2011 13:46:30 -0600 From: Dan Nelson To: Doug Barton Message-ID: <20111214194630.GI53453@dan.emsphone.com> References: <4EE29D3F.2030009@gridfury.com> <4EE7C920.3010109@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE7C920.3010109@FreeBSD.org> X-OS: FreeBSD 8.2-STABLE User-Agent: Mutt/1.5.21 (2010-09-15) X-Virus-Scanned: clamav-milter 0.97.2 at email2.allantgroup.com X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.6 (email2.allantgroup.com [199.67.51.78]); Wed, 14 Dec 2011 13:46:32 -0600 (CST) X-Scanned-By: MIMEDefang 2.68 on 199.67.51.78 Cc: freebsd-stable@freebsd.org Subject: Re: kernel: negative sbsize for uid = 0 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 19:46:35 -0000 In the last episode (Dec 13), Doug Barton said: > I'm running 8.2-RELEASE-p4 i386 on some web servers that are generally > lightly-moderately loaded, but occasionally see some heavy spikes where > load average goes way up. When that is happening, but sometimes even when > it's not, I get hundreds of this message spewing into the logs: > > kernel: negative sbsize for uid = 0 > > I haven't found anything particularly useful by searching for that > message, the one reference was to mbufs, but that seems not to be the > problem. Here is the output of 'netstat -m' during one of the load > spikes: [...] > So is this message something to worry about? If so, how can I diagnose > what's happening, and how do I fix it? I've seen it ocassionally too. The error message is printed in /sys/kern/kern_resource.c when the ui_sbsize resource counter goes negative. There's probably insufficient locking somewhere in the functions that call chgsbsize. The increment/decrement is done atomically, but the data pointed to by the "hiwat" argument is read then updated later without an explicit lock, so if that value changes while the function is executing, it could cause problems. ui_sbsize is only used by the resource limiting code, though, so unless you're enforcing an sbsize rlimit, it should be harmless. -- Dan Nelson dnelson@allantgroup.com From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 19:47:14 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E908106566B for ; Wed, 14 Dec 2011 19:47:14 +0000 (UTC) (envelope-from andrey@zonov.org) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 398818FC15 for ; Wed, 14 Dec 2011 19:47:13 +0000 (UTC) Received: by eekc50 with SMTP id c50so1458462eek.13 for ; Wed, 14 Dec 2011 11:47:13 -0800 (PST) Received: by 10.180.7.195 with SMTP id l3mr72177wia.30.1323892032967; Wed, 14 Dec 2011 11:47:12 -0800 (PST) Received: from [10.254.254.77] (ppp94-29-56-7.pppoe.spdop.ru. [94.29.56.7]) by mx.google.com with ESMTPS id k5sm5042380wiz.9.2011.12.14.11.47.12 (version=SSLv3 cipher=OTHER); Wed, 14 Dec 2011 11:47:12 -0800 (PST) Message-ID: <4EE8FD3E.8030902@zonov.org> Date: Wed, 14 Dec 2011 23:47:10 +0400 From: Andrey Zonov User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.24) Gecko/20100228 Thunderbird/2.0.0.24 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Jeremy Chadwick References: <4EE7BF77.5000504@zonov.org> <20111213221501.GA85563@icarus.home.lan> <4EE8E6E3.7050202@zonov.org> <20111214182252.GA5176@icarus.home.lan> In-Reply-To: <20111214182252.GA5176@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: directory listing hangs in "ufs" state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 19:47:14 -0000 On 14.12.2011 22:22, Jeremy Chadwick wrote: > On Wed, Dec 14, 2011 at 10:11:47PM +0400, Andrey Zonov wrote: >> Hi Jeremy, >> >> This is not hardware problem, I've already checked that. I also ran >> fsck today and got no errors. >> >> After some more exploration of how mongodb works, I found that then >> listing hangs, one of mongodb thread is in "biowr" state for a long >> time. It periodically calls msync(MS_SYNC) accordingly to ktrace >> out. >> >> If I'll remove msync() calls from mongodb, how often data will be >> sync by OS? >> >> -- >> Andrey Zonov >> >> On 14.12.2011 2:15, Jeremy Chadwick wrote: >>> On Wed, Dec 14, 2011 at 01:11:19AM +0400, Andrey Zonov wrote: >>>> >>>> Have you any ideas what is going on? or how to catch the problem? >>> >>> Assuming this isn't a file on the root filesystem, try booting the >>> machine in single-user mode and using "fsck -f" on the filesystem in >>> question. >>> >>> Can you verify there's no problems with the disk this file lives on as >>> well (smartctl -a /dev/disk)? I'm doubting this is the problem, but >>> thought I'd mention it. > > I have no real answer, I'm sorry. msync(2) indicates it's effectively > deprecated (see BUGS). It looks like this is effectively a mmap-version > of fsync(2). I replaced msync(2) with fsync(2). Unfortunately, from man pages it is not obvious that I can do this. Anyway, thanks. > > I'm extremely confused by this problem. What you're describing above is > that the process is "stuck in biowr state for a long time", but what you > stated originally was that the process was "stuck in ufs state for a > few minutes": Listing of the directory with mongodb files by ls(1) stuck in "ufs" state when one of mongodb's thread in "biowr" state. It looks like system holds global lock of the file which is msync(2)-ed and can't immediately return from lstat(2) call. > >> I've got STABLE-8 (r221983) with mongodb-1.8.1 installed on it. A >> couple days ago I observed that listing of mongodb directory stuck in >> a few minutes in "ufs" state. > > Can we narrow down what we're talking about here? Does the process > actually deadlock? Or are you concerned about performance implications? > > I know nothing about this "mongodb" software, but the reason it's > calling msync() is because it wants to try and ensure that the data it > changed in an mmap()-mapped page to be reflected (fully written) on the > disk. This behaviour is fairly common within database software, but > "how often" the software chooses to do this is entirely a design > implementation choice by the authors. > > Meaning: if mongodb is either 1) continually calling msync(), or 2) > waiting for too long a period of time before calling msync(), > performance within the process will suffer. #1 could result in overall > bad performance, while #2 could result in a process that's spending a > lot of time doing I/O (flushing to disk) and therefore appears > "deadlocked" when in fact the kernel/subsystems are doing exactly what > they were told to do. > > Removing the msync() call could result in inconsistent data (possibly > non-recoverable) if the mongodb software crashes or if some other piece > (thread or child? Not sure) expects to open a new fd on that file which > has mmap()'d data. Yes, I clearly understand this. I think of any system tuning instead, but nothing arose in my head. > > This is about all I know. I would love to be able to tell you "consider > a different database" but that seems like an excuse rather than an > actual solution. I guess if all you're seeing is the process "stall" > for long periods of time, but recover normally, then I would open up a > support ticket with the mongodb folks to discuss performance. > -- Andrey Zonov From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 19:53:38 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E4F34106564A for ; Wed, 14 Dec 2011 19:53:38 +0000 (UTC) (envelope-from marcus@odin.blazingdot.com) Received: from odin.blazingdot.com (odin.blazingdot.com [199.48.133.254]) by mx1.freebsd.org (Postfix) with ESMTP id C0B558FC0A for ; Wed, 14 Dec 2011 19:53:38 +0000 (UTC) Received: by odin.blazingdot.com (Postfix, from userid 1001) id 0B504114248; Wed, 14 Dec 2011 19:53:38 +0000 (UTC) Date: Wed, 14 Dec 2011 19:53:38 +0000 From: Marcus Reid To: Tom Evans Message-ID: <20111214195337.GA90758@blazingdot.com> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Coffee-Level: nearly-fatal User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Attilio Rao , George Mitchell , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 19:53:39 -0000 On Wed, Dec 14, 2011 at 05:54:15PM +0000, Tom Evans wrote: > brought forward more complaints about interactivity in X (I've never > noticed this, and use a FreeBSD desktop daily). .. that was me, but I forgot to add that it almost never happens, and it can only be triggered when there are processes that want to take up 100% of the CPU running on the system along with X and friends. Don't want to spread FUD, I've been happily using FreeBSD on the desktop for a decade and ULE seems to work great. Marcus From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 20:42:03 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E4196106566B for ; Wed, 14 Dec 2011 20:42:03 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta06.westchester.pa.mail.comcast.net (qmta06.westchester.pa.mail.comcast.net [76.96.62.56]) by mx1.freebsd.org (Postfix) with ESMTP id 8F9C18FC0C for ; Wed, 14 Dec 2011 20:42:03 +0000 (UTC) Received: from omta23.westchester.pa.mail.comcast.net ([76.96.62.74]) by qmta06.westchester.pa.mail.comcast.net with comcast id 98L91i0021c6gX8568i3dC; Wed, 14 Dec 2011 20:42:03 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta23.westchester.pa.mail.comcast.net with comcast id 98i21i02D1t3BNj3j8i3bd; Wed, 14 Dec 2011 20:42:03 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 261F2102C19; Wed, 14 Dec 2011 12:42:01 -0800 (PST) Date: Wed, 14 Dec 2011 12:42:01 -0800 From: Jeremy Chadwick To: Andrey Zonov Message-ID: <20111214204201.GA7372@icarus.home.lan> References: <4EE7BF77.5000504@zonov.org> <20111213221501.GA85563@icarus.home.lan> <4EE8E6E3.7050202@zonov.org> <20111214182252.GA5176@icarus.home.lan> <4EE8FD3E.8030902@zonov.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE8FD3E.8030902@zonov.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: directory listing hangs in "ufs" state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 20:42:04 -0000 On Wed, Dec 14, 2011 at 11:47:10PM +0400, Andrey Zonov wrote: > On 14.12.2011 22:22, Jeremy Chadwick wrote: > >On Wed, Dec 14, 2011 at 10:11:47PM +0400, Andrey Zonov wrote: > >>Hi Jeremy, > >> > >>This is not hardware problem, I've already checked that. I also ran > >>fsck today and got no errors. > >> > >>After some more exploration of how mongodb works, I found that then > >>listing hangs, one of mongodb thread is in "biowr" state for a long > >>time. It periodically calls msync(MS_SYNC) accordingly to ktrace > >>out. > >> > >>If I'll remove msync() calls from mongodb, how often data will be > >>sync by OS? > >> > >>-- > >>Andrey Zonov > >> > >>On 14.12.2011 2:15, Jeremy Chadwick wrote: > >>>On Wed, Dec 14, 2011 at 01:11:19AM +0400, Andrey Zonov wrote: > >>>> > >>>>Have you any ideas what is going on? or how to catch the problem? > >>> > >>>Assuming this isn't a file on the root filesystem, try booting the > >>>machine in single-user mode and using "fsck -f" on the filesystem in > >>>question. > >>> > >>>Can you verify there's no problems with the disk this file lives on as > >>>well (smartctl -a /dev/disk)? I'm doubting this is the problem, but > >>>thought I'd mention it. > > > >I have no real answer, I'm sorry. msync(2) indicates it's effectively > >deprecated (see BUGS). It looks like this is effectively a mmap-version > >of fsync(2). > > I replaced msync(2) with fsync(2). Unfortunately, from man pages it > is not obvious that I can do this. Anyway, thanks. Sorry, that wasn't what I was implying. Let me try to explain differently. msync(2) looks, to me, like an mmap-specific version of fsync(2). Based on the man page, it seems that the with msync() you can effectively guaranteed flushing of certain pages within an mmap()'d region to disk. fsync() would flush **all** buffers/internal pages to be flushed to disk. One would need to look at the code to mongodb to find out what it's actually doing with msync(). That is to say, if it's doing something like this (I probably have the semantics wrong -- I've never spent much time with mmap()): fd = open("/some/file", O_RDWR); ptr = mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0); ret = msync(ptr, 65536, MS_SYNC); /* or alternatively, this: ret = msync(ptr, NULL, MS_SYNC); */ Then this, to me, would be mostly the equivalent to: fd = fopen("/some/file", "r+"); ret = fsync(fd); Otherwise, if it's calling msync() only on an address/location within the region ptr points to, then that may be more efficient (less pages to flush). The mmap() arguments -- specifically flags (see man page) -- also play a role here. The one that catches my attention is MAP_NOSYNC. So you may need to look at the mongodb code to figure out what it's mmap() call is. One might wonder why they don't just use open() with the O_SYNC. I imagine that has to do with, again, performance; possibly the don't want all I/O synchronous, and would rather flush certain pages in the mmap'd region to disk as needed. I see the legitimacy in that approach (vs. just using O_SYNC). There's really no easy way for me to tell you which is more efficient, better, blah blah without spending a lot of time with a benchmarking program that tests all of this, *plus* an entire system (world) built with profiling. All of this would really fall into the hands of the mongodb people to figure out, if you ask me. But I should note that mmap() on BSD behaves and performs very differently than on, say, Linux; so if the authors wrote what they did intended for Linux systems, I wouldn't be too surprised. :-) > >I'm extremely confused by this problem. What you're describing above is > >that the process is "stuck in biowr state for a long time", but what you > >stated originally was that the process was "stuck in ufs state for a > >few minutes": > > Listing of the directory with mongodb files by ls(1) stuck in "ufs" > state when one of mongodb's thread in "biowr" state. It looks like > system holds global lock of the file which is msync(2)-ed and can't > immediately return from lstat(2) call. Thanks for the clarification -- yes this helps. To some degree it makes sense, some piece of the filesystem or VFS layer is blocking intentionally. How to figure out what layer I do not know. Kernel folks familiar with this aspect would need to chime in here. > >>I've got STABLE-8 (r221983) with mongodb-1.8.1 installed on it. A > >>couple days ago I observed that listing of mongodb directory stuck in > >>a few minutes in "ufs" state. > > > >Can we narrow down what we're talking about here? Does the process > >actually deadlock? Or are you concerned about performance implications? > > > >I know nothing about this "mongodb" software, but the reason it's > >calling msync() is because it wants to try and ensure that the data it > >changed in an mmap()-mapped page to be reflected (fully written) on the > >disk. This behaviour is fairly common within database software, but > >"how often" the software chooses to do this is entirely a design > >implementation choice by the authors. > > > >Meaning: if mongodb is either 1) continually calling msync(), or 2) > >waiting for too long a period of time before calling msync(), > >performance within the process will suffer. #1 could result in overall > >bad performance, while #2 could result in a process that's spending a > >lot of time doing I/O (flushing to disk) and therefore appears > >"deadlocked" when in fact the kernel/subsystems are doing exactly what > >they were told to do. > > > >Removing the msync() call could result in inconsistent data (possibly > >non-recoverable) if the mongodb software crashes or if some other piece > >(thread or child? Not sure) expects to open a new fd on that file which > >has mmap()'d data. > > Yes, I clearly understand this. I think of any system tuning > instead, but nothing arose in my head. Nor I. I think this is more of a userland/application thing than a kernel thing, but there is a love-and-hate relationship between userland and kernel when it comes to the above syscalls and framework. Wish I could be of more help -- sorry. :-( -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 22:51:44 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 0F95F1065672 for ; Wed, 14 Dec 2011 22:51:44 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id B722F16329A; Wed, 14 Dec 2011 22:51:43 +0000 (UTC) Message-ID: <4EE9287F.9060704@FreeBSD.org> Date: Wed, 14 Dec 2011 14:51:43 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Dan Nelson References: <4EE29D3F.2030009@gridfury.com> <4EE7C920.3010109@FreeBSD.org> <20111214194630.GI53453@dan.emsphone.com> In-Reply-To: <20111214194630.GI53453@dan.emsphone.com> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: kernel: negative sbsize for uid = 0 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 22:51:44 -0000 On 12/14/2011 11:46, Dan Nelson wrote: > In the last episode (Dec 13), Doug Barton said: >> I'm running 8.2-RELEASE-p4 i386 on some web servers that are generally >> lightly-moderately loaded, but occasionally see some heavy spikes where >> load average goes way up. When that is happening, but sometimes even when >> it's not, I get hundreds of this message spewing into the logs: >> >> kernel: negative sbsize for uid = 0 >> >> I haven't found anything particularly useful by searching for that >> message, the one reference was to mbufs, but that seems not to be the >> problem. Here is the output of 'netstat -m' during one of the load >> spikes: > [...] >> So is this message something to worry about? If so, how can I diagnose >> what's happening, and how do I fix it? > > I've seen it ocassionally too. The error message is printed in > /sys/kern/kern_resource.c when the ui_sbsize resource counter goes negative. > There's probably insufficient locking somewhere in the functions that call > chgsbsize. The increment/decrement is done atomically, but the data pointed > to by the "hiwat" argument is read then updated later without an explicit > lock, so if that value changes while the function is executing, it could > cause problems. ui_sbsize is only used by the resource limiting code, > though, so unless you're enforcing an sbsize rlimit, it should be harmless. Thanks for the response ... I'll double-check were we might be setting that kind of limit. Doug -- [^L] Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Wed Dec 14 23:39:53 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5626A1065670 for ; Wed, 14 Dec 2011 23:39:53 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 0B0348FC12 for ; Wed, 14 Dec 2011 23:39:52 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RayQR-0007zN-2d>; Thu, 15 Dec 2011 00:39:51 +0100 Received: from e178035148.adsl.alicedsl.de ([85.178.35.148] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RayQQ-0005EX-Tf>; Thu, 15 Dec 2011 00:39:51 +0100 Message-ID: <4EE933C6.4020209@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 00:39:50 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Tom Evans References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> In-Reply-To: X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig9DC4960F627852564DB45495" X-Originating-IP: 85.178.35.148 Cc: Attilio Rao , George Mitchell , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 23:39:53 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig9DC4960F627852564DB45495 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 12/14/11 18:54, Tom Evans wrote: > On Wed, Dec 14, 2011 at 11:06 AM, George Mitchell > wrote: >> >> Dear Secret Masters of FreeBSD: Can we have a decision on whether to >> change back to SCHED_4BSD while SCHED_ULE gets properly fixed? >> >=20 > Please do not do this. This thread has shown that ULE performs poorly > in very specific scenarios where the server is loaded with NCPU+1 CPU > bound processes, and brought forward more complaints about > interactivity in X (I've never noticed this, and use a FreeBSD desktop > daily). I would highly appreciate a decission against SCHED_ULE as the default scheduler! SCHED_4BSD is considered a more mature entity and obviously it seems that SCHED_ULE needs some refinements to achieve a better level of quality. >=20 > On the other hand, we have very many benchmarks showing how poorly > 4BSD scales on things like postgresql. We get much more load out of > our 8.1 ULE DB and web servers than we do out of our 7.0 ones. It's > easy to look at what you do and say "well, what suits my environment > is clearly the best default", but I think there are probably more > users typically running IO bound processes than CPU bound processes. You compare SCHED_ULE on FBSD 8.1 with SCHED_4BSD on FBSD 7.0? Shouldn't you compare SCHED_ULE and SCHED_4BSD on the very same platform? Development of SCHED_ULE has been focused very much on DB like PostgreSQL, no wonder the performance benefit. But this is also a very specific scneario where SCHED_ULE shows a real benefit compared to SCHED_4BSD. >=20 > I believe the correct thing to do is to put some extra documentation > into the handbook about scheduler choice, noting the potential issues > with loading NCPU+1 CPU bound processes. Perhaps making it easier to > switch scheduler would also help? Many people more experst in the issue than myself revealed some issues in the code of both SCHED_ULE and even SCHED_4BSD. It would be a pitty if all the discussions get flushed away like a "toilette-busisness" as it has been done all the way in the past. Well, I'd like to see a kind of "standardized" benchmark. Like on openbenchmark.org or at phoronix.com. I know that Phoronix' way of performing benchmarks is questionable and do not reveal much of the issues, but it is better than nothing. I'm always surprised by the worse performance of FreeBSD when it comes to threaded I/O. The differences between Linux and FreeBSD of the same development maturity are tremendous and scaring! It is a long time since I saw a SPEC benchmark on a FreeBSD driven HPC box. Most benchmark around for testing hardware are performed with Linux and Linux seems to make the race in nearly every scenario. It would be highly appreciable and interesting to see how Linux and FreeBSD would perform in SPEC on the same hardware platform. This is only an idea. Without a suitable benchmark with a codebase understood the discussion is in many aspects pointless -both ways. >=20 > Cheers >=20 > Tom >=20 > References: >=20 > http://people.freebsd.org/~kris/scaling/mysql-freebsd.png > http://suckit.blog.hu/2009/10/05/freebsd_8_is_it_worth_to_upgrade > _______________________________________________ --------------enig9DC4960F627852564DB45495 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQEcBAEBAgAGBQJO6TPGAAoJEOgBcD7A/5N8kNcH/RweDwiplqNdrO032qPjM7cK l0+FIgDpDUCbF7RlsDl0ojta0QwxCnbWwr6qibyoDUndryy+1op/YOpObe+oXckI 5+I/5+1bTPAIOlxZSUvtTWiUbM9qriEPCMWW+oLOf27YNRR9s5/o0Pu9+K1GDE4I 0X5hujliZQqo83Q50jmwGj6gOz0NBv8g3ZN5M6MRR16PCv0Qzg6U5nNKWOEPqHJ0 /duHrj3nRBtHz9csj5oKOn6U3tuDtW6XZU1a9nTL6Z6qINpsrdDDD5WeWbWEkMrU 4BPfMomutBNvB9+dNgES0G0Mi1orcijy0h7GHz8KAPrKMt8eSMsNFc45YVJj50I= =qUFw -----END PGP SIGNATURE----- --------------enig9DC4960F627852564DB45495-- From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 00:11:51 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2A77D1065693 for ; Thu, 15 Dec 2011 00:11:51 +0000 (UTC) (envelope-from george@m5p.com) Received: from mailhost.m5p.com (ip-3-2-0-2.r20.asbnva02.us.ce.gin.ntt.net [IPv6:2001:418:0:5000::16]) by mx1.freebsd.org (Postfix) with ESMTP id D5BC28FC0A for ; Thu, 15 Dec 2011 00:11:50 +0000 (UTC) Received: from wonderland.m5p.com (wonderland.m5p.com [IPv6:2001:418:3fd::19]) by mailhost.m5p.com (8.14.4/8.14.4) with ESMTP id pBF0BiHO017623; Wed, 14 Dec 2011 19:11:49 -0500 (EST) (envelope-from george@m5p.com) Message-ID: <4EE93B40.3090101@m5p.com> Date: Wed, 14 Dec 2011 19:11:44 -0500 From: George Mitchell User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111127 Thunderbird/8.0 MIME-Version: 1.0 To: Tom Evans References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (mailhost.m5p.com [IPv6:2001:418:3fd::f7]); Wed, 14 Dec 2011 19:11:49 -0500 (EST) X-Scanned-By: MIMEDefang 2.72 on IPv6:2001:418:3fd::f7 Cc: freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 00:11:51 -0000 On 12/14/11 12:54, Tom Evans wrote: > [...] This thread has shown that ULE performs poorly > in very specific scenarios where the server is loaded with NCPU+1 CPU > bound processes, [...] Minor correction: Problem occurs when there are nCPU compute-bound processes, not nCPU + 1. -- George Mitchell From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 00:42:07 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F3D11065670 for ; Thu, 15 Dec 2011 00:42:07 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta13.emeryville.ca.mail.comcast.net (qmta13.emeryville.ca.mail.comcast.net [76.96.27.243]) by mx1.freebsd.org (Postfix) with ESMTP id 627688FC16 for ; Thu, 15 Dec 2011 00:42:07 +0000 (UTC) Received: from omta08.emeryville.ca.mail.comcast.net ([76.96.30.12]) by qmta13.emeryville.ca.mail.comcast.net with comcast id 9CfW1i0070FhH24ADCi0J4; Thu, 15 Dec 2011 00:42:00 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta08.emeryville.ca.mail.comcast.net with comcast id 9Cgy1i01b1t3BNj8UCgzoK; Thu, 15 Dec 2011 00:40:59 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id DB595102C19; Wed, 14 Dec 2011 16:42:05 -0800 (PST) Date: Wed, 14 Dec 2011 16:42:05 -0800 From: Jeremy Chadwick To: "O. Hartmann" Message-ID: <20111215004205.GA11556@icarus.home.lan> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE933C6.4020209@zedat.fu-berlin.de> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Tom Evans , Attilio Rao , George Mitchell , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 00:42:07 -0000 On Thu, Dec 15, 2011 at 12:39:50AM +0100, O. Hartmann wrote: > On 12/14/11 18:54, Tom Evans wrote: > > On the other hand, we have very many benchmarks showing how poorly > > 4BSD scales on things like postgresql. We get much more load out of > > our 8.1 ULE DB and web servers than we do out of our 7.0 ones. It's > > easy to look at what you do and say "well, what suits my environment > > is clearly the best default", but I think there are probably more > > users typically running IO bound processes than CPU bound processes. > > You compare SCHED_ULE on FBSD 8.1 with SCHED_4BSD on FBSD 7.0? Shouldn't > you compare SCHED_ULE and SCHED_4BSD on the very same platform? Agreed -- this is a bad comparison. Again, I'm going to tell people to do the one thing that's painful and nobody likes to do: *look at commits* and pay close attention to the branches and any commits that involve "tagging" for a release (so you can determine what "version" of the code you might be running). http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/sched_ule.c http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/sched_4bsd.c I'm a bit busy today, otherwise I would offer to go over the SCHED_4BSD changes between 7.0-RELEASE and 8.1-RELEASE (I would need Tom to confirm those are the exact versions being used; I wish people would stop saying things like "FreeBSD x.y" because it's inaccurate). But the data is there at the above URLs, including the committers/those involves. > > I believe the correct thing to do is to put some extra documentation > > into the handbook about scheduler choice, noting the potential issues > > with loading NCPU+1 CPU bound processes. Perhaps making it easier to > > switch scheduler would also help? Replying to Tom's comment here: It is already easy to switch schedulers. You change the option in your kernel config, rebuild kernel (world isn't necessary as long as you haven't csup'd between your last rebuild and now), make installkernel, shutdown -r now, done. If what you're proposing is to make the scheduler changeable in real-time? I think that would require a **lot** of work for something that very few people would benefit from (please stop for a moment and think about the majority of the userbase, not just niche environments; I say this politely, not with any condescension BTW). Sure, it'd be "nice to have", but should be extremely low on the priority list (IMO). > Many people more experst in the issue than myself revealed some issues > in the code of both SCHED_ULE and even SCHED_4BSD. It would be a pitty > if all the discussions get flushed away like a "toilette-busisness" as > it has been done all the way in the past. Gut feeling says this is what will happen, and that's because the people who are (and have in the past) touching the scheduler bits are not involved in this conversation. We're not going to get anywhere unless those people are involved and are available to make adjustments/etc. I would love to start CC'ing them all, but I don't think that's necessarily effective. I will take the time to point out/remind folks that the number of people who *truly understand* the schedulers are few and far between. We're talking single-digit numbers, folks. And those people are already busy enough as-is. This makes solving this problem difficult. So, what I think WOULD be effective would be for someone to catalogue a list of their systems/specifications/benchmarks/software/etc. that show exactly where the problems are in their workspace when using ULE vs. 4BSD, or vice-versa. That may give the developers some leads as to how to progress. Let's also not forget about the compiler ordeal; gcc versions greatly differ (some folks overwrite the default base gcc with ones in ports), and then there's the clang stuff... Sigh. > Well, I'd like to see a kind of "standardized" benchmark. Like on > openbenchmark.org or at phoronix.com. I know that Phoronix' way of > performing benchmarks is questionable and do not reveal much of the > issues, but it is better than nothing. I would love to run such benchmarks on all of our systems, but I have no idea what kind of benchmark suites/etc. would be beneficial for the developers who maintain/touch the schedulers. You understand what I'm saying? For example, some folks earlier in the thread said the best thing to do for this would be buildworld, but then further follow-ups from others said buildworld is not effective given the I/O demands. Furthermore, I want whatever benchmark/app suite thing to be minimal as hell. It should be standalone, no dependencies (or only 1 or 2). Regarding threadsing: a colleague of mine, ex-co-worker who now works at Apple as a developer, wrote a C program while he was at my current workplace which -- pardon my French -- "beat the shit out of our Solaris boxes, thread-wise". It was customisable via command-line. The thing got some of our Solaris machines up to load averages of nearly 42000 (yes you read that right!), and spit out some benchmark-esque results when finished. I'll mention this thread to him, let him read it, and see if he has anything to say. He is *extremely* busy (even more so with the holiday coming up), so I have little faith he can/will help here, but he may give the code for it if he still has it. I believe he did have me run it on FreeBSD, but it was a long time ago. > I'm always surprised by the worse > performance of FreeBSD when it comes to threaded I/O. The differences > between Linux and FreeBSD of the same development maturity are > tremendous and scaring! Agreed. Linux has the upper hand in many areas, and this is one of them. Please do not think this means "Linux is better, FreeBSD sucks". It simply means that there are more people active/working on these things in Linux. People need to use what works best for them! -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 02:33:35 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2D87D106564A; Thu, 15 Dec 2011 02:33:35 +0000 (UTC) (envelope-from oliver.pntr@gmail.com) Received: from mail-tul01m020-f182.google.com (mail-tul01m020-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id A20388FC12; Thu, 15 Dec 2011 02:33:34 +0000 (UTC) Received: by obbwd18 with SMTP id wd18so178613obb.13 for ; Wed, 14 Dec 2011 18:33:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=BUcaVYL8TDB5cwfDYbQ15rstKcoOzhFEep3FF72bZv4=; b=ATgHywL1IVi7nqY6JNEFYw+J3Zt4HT4pseVDn87YFKsvp4V+3c0UAW8WHtXdSklURQ U+fZY0bCPJOmhqsYdE4Son46fZNoeYSlcYaShKmzpUANqRtq/2Yd3N+8HEhXqjAuu7XB qQI5uwbTVSR6XuFeDkTBsIThYP20FrZIM54X4= MIME-Version: 1.0 Received: by 10.182.180.5 with SMTP id dk5mr607948obc.7.1323914713026; Wed, 14 Dec 2011 18:05:13 -0800 (PST) Received: by 10.182.74.168 with HTTP; Wed, 14 Dec 2011 18:05:12 -0800 (PST) In-Reply-To: <4EE933C6.4020209@zedat.fu-berlin.de> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 03:05:12 +0100 Message-ID: From: Oliver Pinter To: "O. Hartmann" Content-Type: text/plain; charset=ISO-8859-1 Cc: Tom Evans , Attilio Rao , George Mitchell , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 02:33:35 -0000 On 12/15/11, O. Hartmann wrote: > On 12/14/11 18:54, Tom Evans wrote: >> On Wed, Dec 14, 2011 at 11:06 AM, George Mitchell >> wrote: >>> >>> Dear Secret Masters of FreeBSD: Can we have a decision on whether to >>> change back to SCHED_4BSD while SCHED_ULE gets properly fixed? >>> >> >> Please do not do this. This thread has shown that ULE performs poorly >> in very specific scenarios where the server is loaded with NCPU+1 CPU >> bound processes, and brought forward more complaints about >> interactivity in X (I've never noticed this, and use a FreeBSD desktop >> daily). > > I would highly appreciate a decission against SCHED_ULE as the default > scheduler! SCHED_4BSD is considered a more mature entity and obviously > it seems that SCHED_ULE needs some refinements to achieve a better level > of quality. > >> >> On the other hand, we have very many benchmarks showing how poorly >> 4BSD scales on things like postgresql. We get much more load out of >> our 8.1 ULE DB and web servers than we do out of our 7.0 ones. It's >> easy to look at what you do and say "well, what suits my environment >> is clearly the best default", but I think there are probably more >> users typically running IO bound processes than CPU bound processes. > > You compare SCHED_ULE on FBSD 8.1 with SCHED_4BSD on FBSD 7.0? Shouldn't > you compare SCHED_ULE and SCHED_4BSD on the very same platform? > > Development of SCHED_ULE has been focused very much on DB like > PostgreSQL, no wonder the performance benefit. But this is also a very > specific scneario where SCHED_ULE shows a real benefit compared to > SCHED_4BSD. > >> >> I believe the correct thing to do is to put some extra documentation >> into the handbook about scheduler choice, noting the potential issues >> with loading NCPU+1 CPU bound processes. Perhaps making it easier to >> switch scheduler would also help? > > Many people more experst in the issue than myself revealed some issues > in the code of both SCHED_ULE and even SCHED_4BSD. It would be a pitty > if all the discussions get flushed away like a "toilette-busisness" as > it has been done all the way in the past. > > > Well, I'd like to see a kind of "standardized" benchmark. Like on > openbenchmark.org or at phoronix.com. I know that Phoronix' way of > performing benchmarks is questionable and do not reveal much of the > issues, but it is better than nothing. I'm always surprised by the worse > performance of FreeBSD when it comes to threaded I/O. The differences > between Linux and FreeBSD of the same development maturity are > tremendous and scaring! > > It is a long time since I saw a SPEC benchmark on a FreeBSD driven HPC > box. Most benchmark around for testing hardware are performed with Linux > and Linux seems to make the race in nearly every scenario. It would be > highly appreciable and interesting to see how Linux and FreeBSD would > perform in SPEC on the same hardware platform. This is only an idea. > Without a suitable benchmark with a codebase understood the discussion > is in many aspects pointless -both ways. > > >> >> Cheers >> >> Tom >> >> References: >> >> http://people.freebsd.org/~kris/scaling/mysql-freebsd.png >> http://suckit.blog.hu/2009/10/05/freebsd_8_is_it_worth_to_upgrade >> _______________________________________________ > > Hi! Can you try with this settings: op@opn ~> sysctl kern.sched. kern.sched.cpusetsize: 8 kern.sched.preemption: 0 kern.sched.name: ULE kern.sched.slice: 13 kern.sched.interact: 30 kern.sched.preempt_thresh: 224 kern.sched.static_boost: 152 kern.sched.idlespins: 10000 kern.sched.idlespinthresh: 16 kern.sched.affinity: 1 kern.sched.balance: 1 kern.sched.balance_interval: 133 kern.sched.steal_htt: 1 kern.sched.steal_idle: 1 kern.sched.steal_thresh: 1 kern.sched.topology_spec: 0, 1 0, 1 Most of them from 7-STABLE settings, and with this, "works for me". This an laptop with core2 duo cpu (with enabled powerd), and my kernel config is here: http://oliverp.teteny.bme.hu/freebsd/kernel_conf From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 02:42:53 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A2779106564A for ; Thu, 15 Dec 2011 02:42:53 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta10.westchester.pa.mail.comcast.net (qmta10.westchester.pa.mail.comcast.net [76.96.62.17]) by mx1.freebsd.org (Postfix) with ESMTP id 5C9068FC0C for ; Thu, 15 Dec 2011 02:42:53 +0000 (UTC) Received: from omta03.westchester.pa.mail.comcast.net ([76.96.62.27]) by qmta10.westchester.pa.mail.comcast.net with comcast id 9Ei91i0020bG4ec5AEisZt; Thu, 15 Dec 2011 02:42:52 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta03.westchester.pa.mail.comcast.net with comcast id 9Eir1i00b1t3BNj3PEirrR; Thu, 15 Dec 2011 02:42:52 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id D7F8A102C19; Wed, 14 Dec 2011 18:42:49 -0800 (PST) Date: Wed, 14 Dec 2011 18:42:49 -0800 From: Jeremy Chadwick To: Oliver Pinter Message-ID: <20111215024249.GA13557@icarus.home.lan> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Tom Evans , Attilio Rao , George Mitchell , "O. Hartmann" , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 02:42:53 -0000 On Thu, Dec 15, 2011 at 03:05:12AM +0100, Oliver Pinter wrote: > On 12/15/11, O. Hartmann wrote: > > On 12/14/11 18:54, Tom Evans wrote: > >> On Wed, Dec 14, 2011 at 11:06 AM, George Mitchell > >> wrote: > >>> > >>> Dear Secret Masters of FreeBSD: Can we have a decision on whether to > >>> change back to SCHED_4BSD while SCHED_ULE gets properly fixed? > >>> > >> > >> Please do not do this. This thread has shown that ULE performs poorly > >> in very specific scenarios where the server is loaded with NCPU+1 CPU > >> bound processes, and brought forward more complaints about > >> interactivity in X (I've never noticed this, and use a FreeBSD desktop > >> daily). > > > > I would highly appreciate a decission against SCHED_ULE as the default > > scheduler! SCHED_4BSD is considered a more mature entity and obviously > > it seems that SCHED_ULE needs some refinements to achieve a better level > > of quality. > > > >> > >> On the other hand, we have very many benchmarks showing how poorly > >> 4BSD scales on things like postgresql. We get much more load out of > >> our 8.1 ULE DB and web servers than we do out of our 7.0 ones. It's > >> easy to look at what you do and say "well, what suits my environment > >> is clearly the best default", but I think there are probably more > >> users typically running IO bound processes than CPU bound processes. > > > > You compare SCHED_ULE on FBSD 8.1 with SCHED_4BSD on FBSD 7.0? Shouldn't > > you compare SCHED_ULE and SCHED_4BSD on the very same platform? > > > > Development of SCHED_ULE has been focused very much on DB like > > PostgreSQL, no wonder the performance benefit. But this is also a very > > specific scneario where SCHED_ULE shows a real benefit compared to > > SCHED_4BSD. > > > >> > >> I believe the correct thing to do is to put some extra documentation > >> into the handbook about scheduler choice, noting the potential issues > >> with loading NCPU+1 CPU bound processes. Perhaps making it easier to > >> switch scheduler would also help? > > > > Many people more experst in the issue than myself revealed some issues > > in the code of both SCHED_ULE and even SCHED_4BSD. It would be a pitty > > if all the discussions get flushed away like a "toilette-busisness" as > > it has been done all the way in the past. > > > > > > Well, I'd like to see a kind of "standardized" benchmark. Like on > > openbenchmark.org or at phoronix.com. I know that Phoronix' way of > > performing benchmarks is questionable and do not reveal much of the > > issues, but it is better than nothing. I'm always surprised by the worse > > performance of FreeBSD when it comes to threaded I/O. The differences > > between Linux and FreeBSD of the same development maturity are > > tremendous and scaring! > > > > It is a long time since I saw a SPEC benchmark on a FreeBSD driven HPC > > box. Most benchmark around for testing hardware are performed with Linux > > and Linux seems to make the race in nearly every scenario. It would be > > highly appreciable and interesting to see how Linux and FreeBSD would > > perform in SPEC on the same hardware platform. This is only an idea. > > Without a suitable benchmark with a codebase understood the discussion > > is in many aspects pointless -both ways. > > > > > >> > >> Cheers > >> > >> Tom > >> > >> References: > >> > >> http://people.freebsd.org/~kris/scaling/mysql-freebsd.png > >> http://suckit.blog.hu/2009/10/05/freebsd_8_is_it_worth_to_upgrade > >> _______________________________________________ > > Hi! > > Can you try with this settings: > op@opn ~> sysctl kern.sched. I'm replying with a list of each setting which differs compared to RELENG_8 stock on our ULE systems. Note that our ULE systems are 1 physical CPU with 4 cores. > kern.sched.cpusetsize: 8 I see no such tunable/sysctl on any of our RELENG_8 and RELENG_7 systems. Nor do I find any references to it in /usr/src (on any system). Is this a RELENG_9 setting? Please explain where it comes from. I hope it's not a custom kernel patch... > kern.sched.preemption: 0 This differs; default value is 1. > kern.sched.name: ULE > kern.sched.slice: 13 > kern.sched.interact: 30 > kern.sched.preempt_thresh: 224 This differs; default value is 64. The "magic value" of 224 has been discussed in the past, in this thread even. > kern.sched.static_boost: 152 This differs; on our systems it's 160. > kern.sched.idlespins: 10000 > kern.sched.idlespinthresh: 16 This differs; on our systems it's 4. > Most of them from 7-STABLE settings, and with this, "works for me". > This an laptop with core2 duo cpu (with enabled powerd), and my kernel > config is here: > http://oliverp.teteny.bme.hu/freebsd/kernel_conf -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 07:32:57 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4FF881065672; Thu, 15 Dec 2011 07:32:57 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id F31158FC16; Thu, 15 Dec 2011 07:32:56 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1Rb5oE-0005nB-UC>; Thu, 15 Dec 2011 08:32:55 +0100 Received: from e178002216.adsl.alicedsl.de ([85.178.2.216] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1Rb5oE-0007kp-PT>; Thu, 15 Dec 2011 08:32:54 +0100 Message-ID: <4EE9A2A0.80607@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 08:32:48 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Jeremy Chadwick , freebsd-performance@freebsd.org, Current FreeBSD References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> In-Reply-To: <20111215024249.GA13557@icarus.home.lan> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig3AC99262226F5AAF3DFA6856" X-Originating-IP: 85.178.2.216 Cc: FreeBSD Stable Mailing List Subject: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 07:32:57 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig3AC99262226F5AAF3DFA6856 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Just saw this shot benchmark on Phoronix dot com today: http://www.phoronix.com/scan.php?page=3Dnews_item&px=3DMTAyNzA It may be worth to discuss the sad performance of FBSD in some parts of the benchmark. A difference of a factor 10 or 100 is simply far beyond disapointing, it is more than inacceptable and by just reading those benchmarks, I'd like to drop thinking of using FreeBSD even as a backend server in scientific and business environments. In detail, some of the SciMark benches look disappointing. The overall image can't help over the fact that in C-Ray FreeBSD is better performing. =46rom the compiler, I'd like say there couldn't be a drop of more than 1= 0 - 15% in performance - but not 10 or 100 times. I'm just thinking about the discussion of SCHED_ULE and all the saur spots we discussed when I stumbled over the test. Regards, Oliver --------------enig3AC99262226F5AAF3DFA6856 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQEcBAEBAgAGBQJO6aKmAAoJEOgBcD7A/5N8dvYH/1pfvvuy5BJWEEf5LTNhAaIv awHwt5jJ3WJA7zmwtnbGSw2qkRFC8E9D5+jOQ0rissGrSYH4qBakSpfnnJiRTtOm iYzAwnQYXt2STTKuNaz4rcG3bnX8i1SpbHre6Kj1p4cij/sQJXty9CMdVIR3dwYD pLfxSk9yFYrWi2Xpy9zqxdMKC1g/FITIuScwQeXtD3tfQlrh+LPvDK21c+OhukeZ cgVuzNw2274pTPlLNaJpAGkcMw1kPJ3U1cEGaI4nwGLFKvduQp2z13mRHLXATh/a lmVvV/0AIJ6UVLGpwOaBcaCXFxWJ+ez9aDlYM18z2dlfOvLnYcxND/u5GoHHyJg= =3PDg -----END PGP SIGNATURE----- --------------enig3AC99262226F5AAF3DFA6856-- From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 07:40:34 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 188321065672; Thu, 15 Dec 2011 07:40:34 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 89E108FC16; Thu, 15 Dec 2011 07:40:33 +0000 (UTC) Received: by vbbfr13 with SMTP id fr13so2103260vbb.13 for ; Wed, 14 Dec 2011 23:40:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=z/7NpF07+xvfN4qI0sYfDhw2Q31cy25YPhN7EvKe9tI=; b=xYzKxuzfXTM9YnmvW9fwiVbRDobNVzSTGbhecrQTSS+SdcqwJI9Aq97K6nmz45H3Ya Hja4UYIFjpodGDLXcoHbuDdt9JFShzlS1R7W7AG0Z6ln17JzHfIVNKKR38b+BgQToMoe uZqsruqX6c8W9+6gDbRzq7kLJEA/k6bnmECbE= MIME-Version: 1.0 Received: by 10.52.67.111 with SMTP id m15mr1625754vdt.96.1323934832733; Wed, 14 Dec 2011 23:40:32 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.52.26.50 with HTTP; Wed, 14 Dec 2011 23:40:32 -0800 (PST) In-Reply-To: <4EE9A2A0.80607@zedat.fu-berlin.de> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> Date: Wed, 14 Dec 2011 23:40:32 -0800 X-Google-Sender-Auth: URhgjaXgcqV2vgLQyrCg_OMuz-M Message-ID: From: Adrian Chadd To: "O. Hartmann" Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-performance@freebsd.org, Current FreeBSD , FreeBSD Stable Mailing List , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 07:40:34 -0000 On 14 December 2011 23:32, O. Hartmann wrote: > Just saw this shot benchmark on Phoronix dot com today: > > http://www.phoronix.com/scan.php?page=news_item&px=MTAyNzA > > It may be worth to discuss the sad performance of FBSD in some parts of > the benchmark. A difference of a factor 10 or 100 is simply far beyond Well, the only way it's going to get fixed is if someone sits down, replicates it, and starts to document exactly what it is that these benchmarks are/aren't doing. Sometimes it's because the benchmark is very much tickling things incorrectly. In a lot of cases though, the benchmark is testing something synthetic that Linux just happens to have micro-optimised. So if you care about this a lot, someone needs to stand up, work with Phronix to get some actual feedback about what's going on, and see if it can be fixed. Maybe you'll find ULE is broken in some instances; I bet you'll find something like "the disk driver is suboptimal." For example, I remember seeing someone mess up a test because they split their filesystems across raid5 boundaries, and this was hidden by the choice of raid controller and stripe size. This made FreeBSD look worse; when this was corrected for, it sped up far past Linux. Adrian From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 07:41:12 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F09211065672; Thu, 15 Dec 2011 07:41:12 +0000 (UTC) (envelope-from oliver.pntr@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id 99DEA8FC15; Thu, 15 Dec 2011 07:41:12 +0000 (UTC) Received: by yhfq46 with SMTP id q46so2480176yhf.13 for ; Wed, 14 Dec 2011 23:41:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=+ufjbOZU6kj6jiwYgv69DnFVeh/C3ZeoMS8k6LbWRGA=; b=OzVqsG0IZ6nQz9ocefl/mZpoUEx/XvdRnPSTZXNuLQVDS8OoAQiIg/MjlTAPmtCKVZ X4m4VzD5Rt0j88UsOQAVXjPoOmd29hwCX59V5r2b8Cc6sk9wGdRn9MimzRz6V7K3Zx8f rsSzTKEIqH/P8GJ8zyIZk/ghEc73dtNkU3FDA= MIME-Version: 1.0 Received: by 10.236.128.138 with SMTP id f10mr4171425yhi.2.1323934871530; Wed, 14 Dec 2011 23:41:11 -0800 (PST) Received: by 10.236.174.66 with HTTP; Wed, 14 Dec 2011 23:41:10 -0800 (PST) In-Reply-To: <20111215024249.GA13557@icarus.home.lan> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> Date: Thu, 15 Dec 2011 08:41:10 +0100 Message-ID: From: Oliver Pinter To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 Cc: Tom Evans , Attilio Rao , George Mitchell , "O. Hartmann" , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 07:41:13 -0000 On 12/15/11, Jeremy Chadwick wrote: > On Thu, Dec 15, 2011 at 03:05:12AM +0100, Oliver Pinter wrote: >> On 12/15/11, O. Hartmann wrote: >> > On 12/14/11 18:54, Tom Evans wrote: >> >> On Wed, Dec 14, 2011 at 11:06 AM, George Mitchell >> >> wrote: >> >>> >> >>> Dear Secret Masters of FreeBSD: Can we have a decision on whether to >> >>> change back to SCHED_4BSD while SCHED_ULE gets properly fixed? >> >>> >> >> >> >> Please do not do this. This thread has shown that ULE performs poorly >> >> in very specific scenarios where the server is loaded with NCPU+1 CPU >> >> bound processes, and brought forward more complaints about >> >> interactivity in X (I've never noticed this, and use a FreeBSD desktop >> >> daily). >> > >> > I would highly appreciate a decission against SCHED_ULE as the default >> > scheduler! SCHED_4BSD is considered a more mature entity and obviously >> > it seems that SCHED_ULE needs some refinements to achieve a better level >> > of quality. >> > >> >> >> >> On the other hand, we have very many benchmarks showing how poorly >> >> 4BSD scales on things like postgresql. We get much more load out of >> >> our 8.1 ULE DB and web servers than we do out of our 7.0 ones. It's >> >> easy to look at what you do and say "well, what suits my environment >> >> is clearly the best default", but I think there are probably more >> >> users typically running IO bound processes than CPU bound processes. >> > >> > You compare SCHED_ULE on FBSD 8.1 with SCHED_4BSD on FBSD 7.0? Shouldn't >> > you compare SCHED_ULE and SCHED_4BSD on the very same platform? >> > >> > Development of SCHED_ULE has been focused very much on DB like >> > PostgreSQL, no wonder the performance benefit. But this is also a very >> > specific scneario where SCHED_ULE shows a real benefit compared to >> > SCHED_4BSD. >> > >> >> >> >> I believe the correct thing to do is to put some extra documentation >> >> into the handbook about scheduler choice, noting the potential issues >> >> with loading NCPU+1 CPU bound processes. Perhaps making it easier to >> >> switch scheduler would also help? >> > >> > Many people more experst in the issue than myself revealed some issues >> > in the code of both SCHED_ULE and even SCHED_4BSD. It would be a pitty >> > if all the discussions get flushed away like a "toilette-busisness" as >> > it has been done all the way in the past. >> > >> > >> > Well, I'd like to see a kind of "standardized" benchmark. Like on >> > openbenchmark.org or at phoronix.com. I know that Phoronix' way of >> > performing benchmarks is questionable and do not reveal much of the >> > issues, but it is better than nothing. I'm always surprised by the worse >> > performance of FreeBSD when it comes to threaded I/O. The differences >> > between Linux and FreeBSD of the same development maturity are >> > tremendous and scaring! >> > >> > It is a long time since I saw a SPEC benchmark on a FreeBSD driven HPC >> > box. Most benchmark around for testing hardware are performed with Linux >> > and Linux seems to make the race in nearly every scenario. It would be >> > highly appreciable and interesting to see how Linux and FreeBSD would >> > perform in SPEC on the same hardware platform. This is only an idea. >> > Without a suitable benchmark with a codebase understood the discussion >> > is in many aspects pointless -both ways. >> > >> > >> >> >> >> Cheers >> >> >> >> Tom >> >> >> >> References: >> >> >> >> http://people.freebsd.org/~kris/scaling/mysql-freebsd.png >> >> http://suckit.blog.hu/2009/10/05/freebsd_8_is_it_worth_to_upgrade >> >> _______________________________________________ >> >> Hi! >> >> Can you try with this settings: >> op@opn ~> sysctl kern.sched. > > I'm replying with a list of each setting which differs compared to > RELENG_8 stock on our ULE systems. Note that our ULE systems are 1 > physical CPU with 4 cores. On other system that has 4 core I use 7-STABLE, because I have not enough time for upgraded it, and the system has some custom patches. The values what I send in previous mail mostly based on this 4 cores system. > >> kern.sched.cpusetsize: 8 > > I see no such tunable/sysctl on any of our RELENG_8 and RELENG_7 > systems. Nor do I find any references to it in /usr/src (on any > system). Is this a RELENG_9 setting? Please explain where it comes > from. I hope it's not a custom kernel patch... Yes, this is 9-STABLE. > >> kern.sched.preemption: 0 > > This differs; default value is 1. PREEMPTION is disabled via kernel config. > >> kern.sched.name: ULE >> kern.sched.slice: 13 >> kern.sched.interact: 30 > >> kern.sched.preempt_thresh: 224 > > This differs; default value is 64. The "magic value" of 224 has been > discussed in the past, in this thread even. This magic value has discussed before 1 or 1.5 year here, first for 8-STABLE. > >> kern.sched.static_boost: 152 > > This differs; on our systems it's 160. > >> kern.sched.idlespins: 10000 > >> kern.sched.idlespinthresh: 16 > > This differs; on our systems it's 4. > >> Most of them from 7-STABLE settings, and with this, "works for me". >> This an laptop with core2 duo cpu (with enabled powerd), and my kernel >> config is here: >> http://oliverp.teteny.bme.hu/freebsd/kernel_conf > > -- > | Jeremy Chadwick jdc at parodius.com | > | Parodius Networking http://www.parodius.com/ | > | UNIX Systems Administrator Mountain View, CA, US | > | Making life hard for others since 1977. PGP 4BD6C0CB | > > From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 07:46:25 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9BC8A106566C for ; Thu, 15 Dec 2011 07:46:25 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 208A28FC0A for ; Thu, 15 Dec 2011 07:46:24 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id pBF7kEMR000181 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Thu, 15 Dec 2011 09:46:20 +0200 (EET) (envelope-from daniel@digsys.bg) Message-ID: <4EE9A5C5.2070001@digsys.bg> Date: Thu, 15 Dec 2011 09:46:13 +0200 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> In-Reply-To: <4EE933C6.4020209@zedat.fu-berlin.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 07:46:25 -0000 On 15.12.11 01:39, O. Hartmann wrote: > On 12/14/11 18:54, Tom Evans wrote: >> On Wed, Dec 14, 2011 at 11:06 AM, George Mitchell >> wrote: >>> Dear Secret Masters of FreeBSD: Can we have a decision on whether to >>> change back to SCHED_4BSD while SCHED_ULE gets properly fixed? >>> >> Please do not do this. This thread has shown that ULE performs poorly >> in very specific scenarios where the server is loaded with NCPU+1 CPU >> bound processes, and brought forward more complaints about >> interactivity in X (I've never noticed this, and use a FreeBSD desktop >> daily). > I would highly appreciate a decission against SCHED_ULE as the default > scheduler! SCHED_4BSD is considered a more mature entity and obviously > it seems that SCHED_ULE needs some refinements to achieve a better level > of quality. > My logic would be, if SCHED_ULE works better on multi-CPU systems, or if SCHED_4BSD works poor on multi-CPU systems, then by all means keep SCHED_ULE as default scheduler. We are at the end of 2011 and the number of single or dual core CPU systems is decreasing. Most people would just try the newest FreeBSD version on their newest hardware and on that base make an "informed" decision if it is worth it. If on newer hardware SCHED_ULE gives better performance, then again it should be the default. Then, FreeBSD is used in an extremely wide set fo different environments. What scheduler might benefit an one CPU, simple architecture X workstation may be damaging for the performance of multiple CPU, NUMA based server with a large number of non-interactive processes running. Perhaps an knob should be provided with sufficient documentation for those that will not go forward to recompile the kernel (the majority of users, I would guess). I tried switching my RELENG8 desktop from SCHED_ULE to SCHED_4BSD yesterday and cannot see any measurable difference in responsiveness. My 'stress test' is typically an FLASH game, that get's firefox in an almost unresponsive state, eats one of the CPU cores -- but no difference. Well, FLASH has it's own set of problems on FreeBSD, but these are typical "desktop" uses. Running 100% compute intensive processes in background is not. Daniel PS: As to why Linux is "better" in these usages: they do not care much to do things "right", but rather to achieve performance. In my opinion, most of us are with FreeBSD for the "do it right" attitude. From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 08:24:57 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7A8AC106564A for ; Thu, 15 Dec 2011 08:24:57 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm1.ukr.net (fsm1.ukr.net [195.214.192.120]) by mx1.freebsd.org (Postfix) with ESMTP id 0DEAF8FC17 for ; Thu, 15 Dec 2011 08:24:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=Qs2w3PTUYrB6Ur+uTD2AdAThjDDFrthAXvc+QwvxfoU=; b=qIRNTcE3aAEAwIdEWujjNgJ7Vw27OU1Wze+TP19K/oBWs+ljrSw9zWYvg6NYmaCWYPE76z8j5BnGhcaSYrkfVZU8gHDKNRzi1X5o73J9myoBJJrVqeeaDvJZuqeaY4IjyNJgkasF4nWA2DOPFkJ/b//ap5cvWnv/0jdVXg6qQT4=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm1.ukr.net with esmtpsa ID 1Rb6cY-0002A4-7j ; Thu, 15 Dec 2011 10:24:54 +0200 Date: Thu, 15 Dec 2011 10:24:53 +0200 From: Ivan Klymenko To: Oliver Pinter Message-ID: <20111215102453.38580cf8@nonamehost.> In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Tom Evans , freebsd-stable@freebsd.org, Attilio Rao , Mitchell , "O. Hartmann" , George Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 08:24:57 -0000 =D0=92 Thu, 15 Dec 2011 03:05:12 +0100 Oliver Pinter =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > On 12/15/11, O. Hartmann wrote: > > On 12/14/11 18:54, Tom Evans wrote: > >> On Wed, Dec 14, 2011 at 11:06 AM, George Mitchell > >> wrote: > >>> > >>> Dear Secret Masters of FreeBSD: Can we have a decision on whether > >>> to change back to SCHED_4BSD while SCHED_ULE gets properly fixed? > >>> > >> > >> Please do not do this. This thread has shown that ULE performs > >> poorly in very specific scenarios where the server is loaded with > >> NCPU+1 CPU bound processes, and brought forward more complaints > >> about interactivity in X (I've never noticed this, and use a > >> FreeBSD desktop daily). > > > > I would highly appreciate a decission against SCHED_ULE as the > > default scheduler! SCHED_4BSD is considered a more mature entity > > and obviously it seems that SCHED_ULE needs some refinements to > > achieve a better level of quality. > > > >> > >> On the other hand, we have very many benchmarks showing how poorly > >> 4BSD scales on things like postgresql. We get much more load out of > >> our 8.1 ULE DB and web servers than we do out of our 7.0 ones. It's > >> easy to look at what you do and say "well, what suits my > >> environment is clearly the best default", but I think there are > >> probably more users typically running IO bound processes than CPU > >> bound processes. > > > > You compare SCHED_ULE on FBSD 8.1 with SCHED_4BSD on FBSD 7.0? > > Shouldn't you compare SCHED_ULE and SCHED_4BSD on the very same > > platform? > > > > Development of SCHED_ULE has been focused very much on DB like > > PostgreSQL, no wonder the performance benefit. But this is also a > > very specific scneario where SCHED_ULE shows a real benefit > > compared to SCHED_4BSD. > > > >> > >> I believe the correct thing to do is to put some extra > >> documentation into the handbook about scheduler choice, noting the > >> potential issues with loading NCPU+1 CPU bound processes. Perhaps > >> making it easier to switch scheduler would also help? > > > > Many people more experst in the issue than myself revealed some > > issues in the code of both SCHED_ULE and even SCHED_4BSD. It would > > be a pitty if all the discussions get flushed away like a > > "toilette-busisness" as it has been done all the way in the past. > > > > > > Well, I'd like to see a kind of "standardized" benchmark. Like on > > openbenchmark.org or at phoronix.com. I know that Phoronix' way of > > performing benchmarks is questionable and do not reveal much of the > > issues, but it is better than nothing. I'm always surprised by the > > worse performance of FreeBSD when it comes to threaded I/O. The > > differences between Linux and FreeBSD of the same development > > maturity are tremendous and scaring! > > > > It is a long time since I saw a SPEC benchmark on a FreeBSD driven > > HPC box. Most benchmark around for testing hardware are performed > > with Linux and Linux seems to make the race in nearly every > > scenario. It would be highly appreciable and interesting to see how > > Linux and FreeBSD would perform in SPEC on the same hardware > > platform. This is only an idea. Without a suitable benchmark with a > > codebase understood the discussion is in many aspects pointless > > -both ways. > > > > > >> > >> Cheers > >> > >> Tom > >> > >> References: > >> > >> http://people.freebsd.org/~kris/scaling/mysql-freebsd.png > >> http://suckit.blog.hu/2009/10/05/freebsd_8_is_it_worth_to_upgrade > >> _______________________________________________ > > > > >=20 > Hi! >=20 > Can you try with this settings: >=20 > op@opn ~> sysctl kern.sched. > kern.sched.cpusetsize: 8 > kern.sched.preemption: 0 > kern.sched.name: ULE > kern.sched.slice: 13 > kern.sched.interact: 30 > kern.sched.preempt_thresh: 224 > kern.sched.static_boost: 152 > kern.sched.idlespins: 10000 > kern.sched.idlespinthresh: 16 > kern.sched.affinity: 1 > kern.sched.balance: 1 > kern.sched.balance_interval: 133 > kern.sched.steal_htt: 1 > kern.sched.steal_idle: 1 > kern.sched.steal_thresh: 1 > kern.sched.topology_spec: > > 0, 1 > > > 0, 1 > > > > >=20 > Most of them from 7-STABLE settings, and with this, "works for me". > This an laptop with core2 duo cpu (with enabled powerd), and my kernel > config is here: > http://oliverp.teteny.bme.hu/freebsd/kernel_conf And you try to do like there http://www.youtube.com/watch?v=3D1CLCp-dqWu0 what would your the cursor mouse and Xorg NOT froze for a split second or more... And I'll see how really good your ULE ;) From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 09:06:28 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 13EC9106566C; Thu, 15 Dec 2011 09:06:28 +0000 (UTC) (envelope-from gmx@ross.cx) Received: from www81.your-server.de (www81.your-server.de [213.133.104.81]) by mx1.freebsd.org (Postfix) with ESMTP id BE3298FC13; Thu, 15 Dec 2011 09:06:27 +0000 (UTC) Received: from [188.108.237.120] (helo=michael-think) by www81.your-server.de with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1Rb6zt-0001bL-6P; Thu, 15 Dec 2011 09:49:01 +0100 Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes To: "Jeremy Chadwick" , freebsd-performance@freebsd.org, "Current FreeBSD" , "O. Hartmann" References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 09:48:52 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Michael Ross" Message-ID: In-Reply-To: <4EE9A2A0.80607@zedat.fu-berlin.de> User-Agent: Opera Mail/11.60 (Win32) X-Authenticated-Sender: gmx@ross.cx X-Virus-Scanned: Clear (ClamAV 0.97.3/14124/Wed Dec 14 16:10:02 2011) Cc: FreeBSD Stable Mailing List Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 09:06:28 -0000 Am 15.12.2011, 08:32 Uhr, schrieb O. Hartmann : > Just saw this shot benchmark on Phoronix dot com today: > > http://www.phoronix.com/scan.php?page=news_item&px=MTAyNzA > > It may be worth to discuss the sad performance of FBSD in some parts of > the benchmark. A difference of a factor 10 or 100 is simply far beyond > disapointing, it is more than inacceptable and by just reading those > benchmarks, I'd like to drop thinking of using FreeBSD even as a backend > server in scientific and business environments. In detail, some of the > SciMark benches look disappointing. Why SciMark? SciMark FreeBSD : Oracle, Mflops Composite 884.79 : 844.03 (Faster: FreeBSD) FFT 236.17 : 213.65 (Faster: FreeBSD) Jacobi 970.76 : 974.84 (Faster: Oracle) Monte Carlo 443.00 : 246.27 (Faster: FreeBSD) Sparse Matrix 1213.64 : 1228.22 (Faster: Oracle) Dense LU 1560.39 : 1557.18 (Faster: FreeBSD) The threaded I/O results (Oracle outperforms FreeBSD by x10 on one, by x100 on another test) or the disc TPS ( 486 : 3526 ) sure look worse and are worth looking into. Anyway these tests were performed on different hardware, FWIW. And with different filesystems, different compilers, different GUIs... Regards, Michael From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 09:11:22 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 59D6E1065670; Thu, 15 Dec 2011 09:11:22 +0000 (UTC) (envelope-from perryh@pluto.rain.com) Received: from agora.rdrop.com (agora.rdrop.com [IPv6:2607:f678:1010::34]) by mx1.freebsd.org (Postfix) with ESMTP id 33C958FC13; Thu, 15 Dec 2011 09:11:20 +0000 (UTC) Received: from agora.rdrop.com (66@localhost [127.0.0.1]) by agora.rdrop.com (8.13.1/8.12.7) with ESMTP id pBF9BJKL014878 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Thu, 15 Dec 2011 01:11:20 -0800 (PST) (envelope-from perryh@pluto.rain.com) Received: (from uucp@localhost) by agora.rdrop.com (8.13.1/8.12.9/Submit) with UUCP id pBF9BJd3014877; Thu, 15 Dec 2011 01:11:19 -0800 (PST) Received: from fbsd81 ([192.168.200.81]) by pluto.rain.com (4.1/SMI-4.1-pluto-M2060407) id AA00785; Thu, 15 Dec 11 01:04:17 PST Date: Thu, 15 Dec 2011 08:03:22 -0800 From: perryh@pluto.rain.com To: freebsd@jdc.parodius.com Message-Id: <4eea1a4a.nJRbEc1jgKpVnVk4%perryh@pluto.rain.com> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215004205.GA11556@icarus.home.lan> In-Reply-To: <20111215004205.GA11556@icarus.home.lan> User-Agent: nail 11.25 7/29/05 Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: tevans.uk@googlemail.com, attilio@freebsd.org, george+freebsd@m5p.com, freebsd-stable@freebsd.org, ohartman@zedat.fu-berlin.de Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 09:11:22 -0000 Jeremy Chadwick wrote: > It is already easy to switch schedulers. You change the > option in your kernel config, rebuild kernel (world isn't > necessary as long as you haven't csup'd between your last > rebuild and now), make installkernel, shutdown -r now, > done. and you have thereby shot freebsd-update in the foot, because you are no longer using a generic kernel. > If what you're proposing is to make the scheduler changeable > in real-time? I think that would require a **lot** of work > for something that very few people would benefit from ... Switching on the fly sounds frightfully difficult, as long as 4BSD and ULE are separate code bases. (It might not be so bad if a tunable or 3 could be added to ULE, so that it could be configured to behave like 4BSD.) However, the freebsd-update complication could in principle be relieved by building both schedulers into the generic kernel, with the choice being configurable in loader.conf. It would still take a reboot to switch, but not a kernel rebuild. Of course there may be practical issues, e.g. name collisions. From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 09:14:00 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 192FC106566B for ; Thu, 15 Dec 2011 09:14:00 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (smtp6.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) by mx1.freebsd.org (Postfix) with ESMTP id 9B6468FC12 for ; Thu, 15 Dec 2011 09:13:59 +0000 (UTC) Received: from seedling.black-earth.co.uk (seedling.black-earth.co.uk [IPv6:2001:8b0:151:1:fa1e:dfff:feda:c0bb]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.14.5/8.14.5) with ESMTP id pBF9DoK6015967 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Thu, 15 Dec 2011 09:13:55 GMT (envelope-from m.seaman@infracaninophile.co.uk) X-DKIM: OpenDKIM Filter v2.4.1 smtp.infracaninophile.co.uk pBF9DoK6015967 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=infracaninophile.co.uk; s=201001-infracaninophile; t=1323940435; bh=sxnGtHdnZClxO7kCkepOgP63ibzzPNRTLg5rebd63bE=; h=Message-ID:Date:From:MIME-Version:To:Subject:References: In-Reply-To:Content-Type:Cc; b=nWA0zML3WmhU8c5eKvaaP3sAzBkxu9ndRwV2JZ/JjcG4XG1w470kCZXGvl1crAMqA Rnp8oaP2iIe7jTp7jdTLKNiNNTtm/SOWtMyW3KEkMxS+Z44Pnu70lVZaXzUgYdNDIm nk/+T+lvbWsF1GzFQEfs7JW5SHNfp/eTPsw7Czvw= Message-ID: <4EE9BA47.7080005@infracaninophile.co.uk> Date: Thu, 15 Dec 2011 09:13:43 +0000 From: Matthew Seaman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215004205.GA11556@icarus.home.lan> In-Reply-To: <20111215004205.GA11556@icarus.home.lan> X-Enigmail-Version: 1.3.4 OpenPGP: id=60AE908C Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigED13262F1122E774546949F3" X-Virus-Scanned: clamav-milter 0.97.3 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-3.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU autolearn=ham version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lucid-nonsense.infracaninophile.co.uk Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 09:14:00 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigED13262F1122E774546949F3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 15/12/2011 00:42, Jeremy Chadwick wrote: > It is already easy to switch schedulers. You change the option in your= > kernel config, rebuild kernel (world isn't necessary as long as you > haven't csup'd between your last rebuild and now), make installkernel, > shutdown -r now, done. >=20 > If what you're proposing is to make the scheduler changeable in > real-time? I think that would require a **lot** of work for something > that very few people would benefit from (please stop for a moment and > think about the majority of the userbase, not just niche environments; = I > say this politely, not with any condescension BTW). Sure, it'd be > "nice to have", but should be extremely low on the priority list (IMO).= Somewhere in between might be a good idea it seems to me: viz, change a setting in loader.conf and reboot to switch to a new scheduler. Having to juggle different kernels is no big deal for the likes of you and me, but it is quite a barrier in many environments. Cheers, Matthew --=20 Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate JID: matthew@infracaninophile.co.uk Kent, CT11 9PW --------------enigED13262F1122E774546949F3 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk7puk0ACgkQ8Mjk52CukIzNsACffjb/WRQ7+Oh1BhjBSlLC+0K9 uDgAnivS0QRfbkjrqQ4tC9K38uzv5gZj =fqQM -----END PGP SIGNATURE----- --------------enigED13262F1122E774546949F3-- From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 09:44:25 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0BD63106566B for ; Thu, 15 Dec 2011 09:44:25 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id B4AAB8FC19 for ; Thu, 15 Dec 2011 09:44:24 +0000 (UTC) Received: by vbbfr13 with SMTP id fr13so2220200vbb.13 for ; Thu, 15 Dec 2011 01:44:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=mmPHE2NiEq+cRe+tfaqPX0Z8/KCa95i+Hmpas6bJ8R0=; b=iHzD8x8/iArXv4G7kdangK6W3FaWgtPRuu42cmfSnVDHzy6pcbhlEodFKrThc8NUjy 1R0d6WsJzXrRfQHtGq4TDebU44w75X70O8LJHCVBfbhHnjYjhHHGjveTLgk3lgapII9p u3Q+Lg0hJoLQIE3D43fs4Hae7DP9+b55Ylt7U= MIME-Version: 1.0 Received: by 10.52.23.2 with SMTP id i2mr1854149vdf.126.1323942264058; Thu, 15 Dec 2011 01:44:24 -0800 (PST) Received: by 10.52.162.202 with HTTP; Thu, 15 Dec 2011 01:44:23 -0800 (PST) In-Reply-To: <20111215004205.GA11556@icarus.home.lan> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215004205.GA11556@icarus.home.lan> Date: Thu, 15 Dec 2011 09:44:23 +0000 Message-ID: From: Tom Evans To: Jeremy Chadwick Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 09:44:25 -0000 On Thu, Dec 15, 2011 at 12:42 AM, Jeremy Chadwick wrote: > On Thu, Dec 15, 2011 at 12:39:50AM +0100, O. Hartmann wrote: >> On 12/14/11 18:54, Tom Evans wrote: >> > I believe the correct thing to do is to put some extra documentation >> > into the handbook about scheduler choice, noting the potential issues >> > with loading NCPU+1 CPU bound processes. Perhaps making it easier to >> > switch scheduler would also help? > > Replying to Tom's comment here: > > It is already easy to switch schedulers. =C2=A0You change the option in y= our > kernel config, rebuild kernel (world isn't necessary as long as you > haven't csup'd between your last rebuild and now), make installkernel, > shutdown -r now, done. Your definition of 'easy' differs wildly from mine. How is that in any way 'easy' to do across 200 servers? > > If what you're proposing is to make the scheduler changeable in > real-time? =C2=A0I think that would require a **lot** of work for somethi= ng > that very few people would benefit from (please stop for a moment and > think about the majority of the userbase, not just niche environments; I > say this politely, not with any condescension BTW). =C2=A0Sure, it'd be > "nice to have", but should be extremely low on the priority list (IMO). Real time scheduler changing would be insane! I was thinking that both/any/all schedulers could be compiled into the kernel, and the choice of which one to use becomes a boot time configuration. You don't have to recompile the kernel to change timecounter. Cheers Tom From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 10:41:16 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3EFDF106564A; Thu, 15 Dec 2011 10:41:16 +0000 (UTC) (envelope-from gmx@ross.cx) Received: from www81.your-server.de (www81.your-server.de [213.133.104.81]) by mx1.freebsd.org (Postfix) with ESMTP id E745C8FC08; Thu, 15 Dec 2011 10:41:15 +0000 (UTC) Received: from [188.108.237.120] (helo=michael-think) by www81.your-server.de with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1Rb8kU-0008KX-Lp; Thu, 15 Dec 2011 11:41:14 +0100 Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes To: "Michael Larabel" References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> Date: Thu, 15 Dec 2011 11:41:05 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Michael Ross" Message-ID: In-Reply-To: <4EE9C79B.7080607@phoronix.com> User-Agent: Opera Mail/11.60 (Win32) X-Authenticated-Sender: gmx@ross.cx X-Virus-Scanned: Clear (ClamAV 0.97.3/14124/Wed Dec 14 16:10:02 2011) Cc: FreeBSD Stable Mailing List , freebsd-performance@freebsd.org, Current FreeBSD , "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 10:41:16 -0000 Am 15.12.2011, 11:10 Uhr, schrieb Michael Larabel : > On 12/15/2011 02:48 AM, Michael Ross wrote: >> Anyway these tests were performed on different hardware, FWIW. >> And with different filesystems, different compilers, different GUIs... >> >> > > No, the same hardware was used for each OS. > The picture under the heading "System Hardware / Software" does not reflect that. Motherboard description differs, Chipset description for FreeBSD is empty. Regards, Michael > In terms of the software, the stock software stack for each OS was used. > > -- Michael From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 10:55:21 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0498106566C; Thu, 15 Dec 2011 10:55:21 +0000 (UTC) (envelope-from michael.larabel@phoronix.com) Received: from phx1.phoronix.com (173.192.77.202-static.reverse.softlayer.com [173.192.77.202]) by mx1.freebsd.org (Postfix) with ESMTP id 9892D8FC16; Thu, 15 Dec 2011 10:55:21 +0000 (UTC) Received: from c-98-193-96-120.hsd1.il.comcast.net ([98.193.96.120] helo=[172.16.93.133]) by phx1.phoronix.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69) (envelope-from ) id 1Rb8y8-0001HY-46; Thu, 15 Dec 2011 04:55:20 -0600 Message-ID: <4EE9D214.3070906@phoronix.com> Date: Thu, 15 Dec 2011 04:55:16 -0600 From: Michael Larabel User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Michael Ross References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - phx1.phoronix.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - phoronix.com Cc: "O. Hartmann" , freebsd-performance@freebsd.org, Current FreeBSD , FreeBSD Stable Mailing List , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 10:55:21 -0000 On 12/15/2011 04:41 AM, Michael Ross wrote: > Am 15.12.2011, 11:10 Uhr, schrieb Michael Larabel > : > >> On 12/15/2011 02:48 AM, Michael Ross wrote: > >>> Anyway these tests were performed on different hardware, FWIW. >>> And with different filesystems, different compilers, different GUIs... >>> >>> >> >> No, the same hardware was used for each OS. >> > > The picture under the heading "System Hardware / Software" does not > reflect that. > > Motherboard description differs, Chipset description for FreeBSD is > empty. > I was the on that carried out the testing and know that it was on the same system. All of the testing, including the system tables, is fully automated. Under FreeBSD sometimes the parsing of some component strings isn't as nice as Linux and other supported operating systems by the Phoronix Test Suite. For the BSD motherboard string parsing it's grabbing hw.vendor/hw.product from sysctl. Is there a better place to read the motherboard DMI information from? -- Michael > > Regards, > > Michael > > >> In terms of the software, the stock software stack for each OS was used. >> >> -- Michael > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 11:02:10 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 36934106564A; Thu, 15 Dec 2011 11:02:10 +0000 (UTC) (envelope-from michael.larabel@phoronix.com) Received: from phx1.phoronix.com (173.192.77.202-static.reverse.softlayer.com [173.192.77.202]) by mx1.freebsd.org (Postfix) with ESMTP id 002F28FC19; Thu, 15 Dec 2011 11:02:09 +0000 (UTC) Received: from c-98-193-96-120.hsd1.il.comcast.net ([98.193.96.120] helo=[172.16.93.133]) by phx1.phoronix.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69) (envelope-from ) id 1Rb8Gs-0007DG-OQ; Thu, 15 Dec 2011 04:10:38 -0600 Message-ID: <4EE9C79B.7080607@phoronix.com> Date: Thu, 15 Dec 2011 04:10:35 -0600 From: Michael Larabel User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Michael Ross References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - phx1.phoronix.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - phoronix.com Cc: FreeBSD Stable Mailing List , freebsd-performance@freebsd.org, Current FreeBSD , "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 11:02:10 -0000 On 12/15/2011 02:48 AM, Michael Ross wrote: > Am 15.12.2011, 08:32 Uhr, schrieb O. Hartmann > : > >> Just saw this shot benchmark on Phoronix dot com today: >> >> http://www.phoronix.com/scan.php?page=news_item&px=MTAyNzA >> >> It may be worth to discuss the sad performance of FBSD in some parts of >> the benchmark. A difference of a factor 10 or 100 is simply far beyond >> disapointing, it is more than inacceptable and by just reading those >> benchmarks, I'd like to drop thinking of using FreeBSD even as a backend >> server in scientific and business environments. In detail, some of the >> SciMark benches look disappointing. > > Why SciMark? > > SciMark FreeBSD : Oracle, Mflops > > Composite 884.79 : 844.03 (Faster: FreeBSD) > FFT 236.17 : 213.65 (Faster: FreeBSD) > Jacobi 970.76 : 974.84 (Faster: Oracle) > Monte Carlo 443.00 : 246.27 (Faster: FreeBSD) > Sparse Matrix 1213.64 : 1228.22 (Faster: Oracle) > Dense LU 1560.39 : 1557.18 (Faster: FreeBSD) > > > The threaded I/O results (Oracle outperforms FreeBSD by x10 on one, by > x100 on another test) > or the disc TPS ( 486 : 3526 ) sure look worse and are worth looking > into. > > > Anyway these tests were performed on different hardware, FWIW. > And with different filesystems, different compilers, different GUIs... > > No, the same hardware was used for each OS. In terms of the software, the stock software stack for each OS was used. -- Michael > > Regards, > > Michael > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 11:14:23 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DA6801065672; Thu, 15 Dec 2011 11:14:23 +0000 (UTC) (envelope-from prvs=1330762df1=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id E89548FC13; Thu, 15 Dec 2011 11:14:22 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 15 Dec 2011 11:02:29 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 15 Dec 2011 11:02:24 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50017140276.msg; Thu, 15 Dec 2011 11:02:23 +0000 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1330762df1=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <95A7C3EFE7B4406EAFF0259CD8B61D1C@multiplay.co.uk> From: "Steven Hartland" To: "Michael Larabel" , "Michael Ross" References: <4EE1EAFE.3070408@m5p.com><4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com><4EE933C6.4020209@zedat.fu-berlin.de><20111215024249.GA13557@icarus.home.lan><4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9D214.3070906@phoronix.com> Date: Thu, 15 Dec 2011 11:02:47 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-15"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: FreeBSD Stable Mailing List , freebsd-performance@freebsd.org, Current FreeBSD , "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 11:14:24 -0000 ----- Original Message ----- From: "Michael Larabel" > > I was the on that carried out the testing and know that it was on the > same system. > > All of the testing, including the system tables, is fully automated. > Under FreeBSD sometimes the parsing of some component strings isn't as > nice as Linux and other supported operating systems by the Phoronix Test > Suite. For the BSD motherboard string parsing it's grabbing > hw.vendor/hw.product from sysctl. Is there a better place to read the > motherboard DMI information from? dmidecode may provide better info? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 11:19:10 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9D734106564A; Thu, 15 Dec 2011 11:19:10 +0000 (UTC) (envelope-from gmx@ross.cx) Received: from www81.your-server.de (www81.your-server.de [213.133.104.81]) by mx1.freebsd.org (Postfix) with ESMTP id 50C488FC12; Thu, 15 Dec 2011 11:19:09 +0000 (UTC) Received: from [188.108.237.120] (helo=michael-think) by www81.your-server.de with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1Rb9L7-0005Np-8H; Thu, 15 Dec 2011 12:19:07 +0100 Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes To: "Michael Larabel" References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9D214.3070906@phoronix.com> Date: Thu, 15 Dec 2011 12:18:55 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Michael Ross" Message-ID: In-Reply-To: <4EE9D214.3070906@phoronix.com> User-Agent: Opera Mail/11.60 (Win32) X-Authenticated-Sender: gmx@ross.cx X-Virus-Scanned: Clear (ClamAV 0.97.3/14124/Wed Dec 14 16:10:02 2011) Cc: "O. Hartmann" , freebsd-performance@freebsd.org, Current FreeBSD , FreeBSD Stable Mailing List , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 11:19:10 -0000 Am 15.12.2011, 11:55 Uhr, schrieb Michael Larabel : > On 12/15/2011 04:41 AM, Michael Ross wrote: >> Am 15.12.2011, 11:10 Uhr, schrieb Michael Larabel >> : >> >>> On 12/15/2011 02:48 AM, Michael Ross wrote: >> >>>> Anyway these tests were performed on different hardware, FWIW. >>>> And with different filesystems, different compilers, different GUIs... >>>> >>>> >>> >>> No, the same hardware was used for each OS. >>> >> >> The picture under the heading "System Hardware / Software" does not >> reflect that. >> >> Motherboard description differs, Chipset description for FreeBSD is >> empty. >> > > I was the on that carried out the testing and know that it was on the > same system. No offense. I'm not doubting you. But I didn't know this: > All of the testing, including the system tables, is fully automated. > Under FreeBSD sometimes the parsing of some component strings isn't as > nice as Linux and other supported operating systems by the Phoronix Test > Suite. For the BSD motherboard string parsing it's grabbing > hw.vendor/hw.product from sysctl. so maybe you can understand how I got my impression. NVidia Audio and Realtek Audio. Looks different to me :-) > Is there a better place to read the motherboard DMI information from? > Following Steven Hartlands' suggestion, from one of my machines: /usr/ports/sysutils/dmidecode/#sysctl -a | egrep "hw.vendor|hw.product" /usr/ports/sysutils/dmidecode/#dmidecode -t 2 # dmidecode 2.11 SMBIOS 2.6 present. Handle 0x0002, DMI type 2, 15 bytes Base Board Information Manufacturer: FUJITSU Product Name: D2759 Version: S26361-D2759-A13 WGS04 GS02 Serial Number: 35838599 Asset Tag: - Features: Board is a hosting board Board is removable Location In Chassis: - Chassis Handle: 0x0003 Type: Motherboard Contained Object Handles: 0 Nice. Didn't know about that. Regards, Michael From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 11:19:32 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8748410656DA; Thu, 15 Dec 2011 11:19:32 +0000 (UTC) (envelope-from michael.larabel@phoronix.com) Received: from phx1.phoronix.com (173.192.77.202-static.reverse.softlayer.com [173.192.77.202]) by mx1.freebsd.org (Postfix) with ESMTP id 4D5138FC1D; Thu, 15 Dec 2011 11:19:32 +0000 (UTC) Received: from c-98-193-96-120.hsd1.il.comcast.net ([98.193.96.120] helo=[172.16.93.133]) by phx1.phoronix.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69) (envelope-from ) id 1Rb9LT-0002Vs-4s; Thu, 15 Dec 2011 05:19:27 -0600 Message-ID: <4EE9D7B8.50209@phoronix.com> Date: Thu, 15 Dec 2011 05:19:20 -0600 From: Michael Larabel User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Steven Hartland References: <4EE1EAFE.3070408@m5p.com><4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com><4EE933C6.4020209@zedat.fu-berlin.de><20111215024249.GA13557@icarus.home.lan><4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9D214.3070906@phoronix.com> <95A7C3EFE7B4406EAFF0259CD8B61D1C@multiplay.co.uk> In-Reply-To: <95A7C3EFE7B4406EAFF0259CD8B61D1C@multiplay.co.uk> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - phx1.phoronix.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - phoronix.com Cc: FreeBSD Stable Mailing List , Current FreeBSD , Michael Ross , freebsd-performance@freebsd.org, "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 11:19:32 -0000 On 12/15/2011 05:02 AM, Steven Hartland wrote: > ----- Original Message ----- From: "Michael Larabel" > > >> >> I was the on that carried out the testing and know that it was on the >> same system. >> >> All of the testing, including the system tables, is fully automated. >> Under FreeBSD sometimes the parsing of some component strings isn't >> as nice as Linux and other supported operating systems by the >> Phoronix Test Suite. For the BSD motherboard string parsing it's >> grabbing hw.vendor/hw.product from sysctl. Is there a better place to >> read the motherboard DMI information from? > > dmidecode may provide better info? > > Regards > Steve dmidecode is used on Linux for parsing some of the hardware information. I think I looked at using it for BSD too, but offhand I don't recall what the problem was. I'll check into it again with the latest release when time allows. Michael > > ================================================ > This e.mail is private and confidential between Multiplay (UK) Ltd. > and the person or entity to whom it is addressed. In the event of > misdirection, the recipient is prohibited from using, copying, > printing or otherwise disseminating it or any information contained in > it. > In the event of misdirection, illegible or incomplete transmission > please telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. > > From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 11:28:07 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 61C9C106566B for ; Thu, 15 Dec 2011 11:28:07 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) by mx1.freebsd.org (Postfix) with ESMTP id B0C5C8FC1D for ; Thu, 15 Dec 2011 11:28:06 +0000 (UTC) Received: from hugo10.ka.punkt.de ([217.29.45.10]) by gate1.intern.punkt.de with ESMTP id pBFBS2Se088641; Thu, 15 Dec 2011 12:28:02 +0100 (CET) Received: from [217.29.45.158] ([217.29.45.158]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id pBFBS1ub088347; Thu, 15 Dec 2011 12:28:01 +0100 (CET) (envelope-from hausen@punkt.de) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=windows-1252 From: "Patrick M. Hausen" In-Reply-To: Date: Thu, 15 Dec 2011 12:28:01 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <0F38630E-D884-4B33-B64F-2631218C60E4@punkt.de> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9D214.3070906@phoronix.com> To: "Michael Ross" X-Mailer: Apple Mail (2.1251.1) Cc: Michael Larabel , FreeBSD Stable Mailing List , Current FreeBSD , freebsd-performance@freebsd.org, "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 11:28:07 -0000 Hi, all, Am 15.12.2011 um 12:18 schrieb Michael Ross: > Following Steven Hartlands' suggestion, > from one of my machines: >=20 > /usr/ports/sysutils/dmidecode/#sysctl -a | egrep = "hw.vendor|hw.product" >=20 > /usr/ports/sysutils/dmidecode/#dmidecode -t 2 > # dmidecode 2.11 > SMBIOS 2.6 present. >=20 > Handle 0x0002, DMI type 2, 15 bytes > Base Board Information > Manufacturer: FUJITSU > Product Name: D2759 > Version: S26361-D2759-A13 WGS04 GS02 > Serial Number: 35838599 > Asset Tag: - > Features: > Board is a hosting board > Board is removable > Location In Chassis: - > Chassis Handle: 0x0003 > Type: Motherboard > Contained Object Handles: 0 Without the need to install an additional port: datatomb2# kenv =85 smbios.bios.reldate=3D"11/03/2011" smbios.bios.vendor=3D"FUJITSU // American Megatrends Inc." smbios.bios.version=3D"V4.6.4.1 R1.18.0 for D3034-A1x" smbios.chassis.maker=3D"FUJITSU" smbios.chassis.serial=3D"YLAP004857" smbios.chassis.tag=3D"System Asset Tag " smbios.chassis.version=3D"RX100S7R2" smbios.memory.enabled=3D"8388608" smbios.planar.maker=3D"FUJITSU" smbios.planar.product=3D"D3034-A1" smbios.planar.serial=3D"LJ1B-P00996" smbios.planar.version=3D"S26361-D3034-A100 WGS01 GS02" smbios.socket.enabled=3D"1" smbios.socket.populated=3D"1" smbios.system.maker=3D"FUJITSU" smbios.system.product=3D"PRIMERGY RX100 S7" smbios.system.serial=3D"YLAP004857" smbios.system.uuid=3D"f0493081-f5ca-e011-b8a5-a1c4d143da5f" smbios.system.version=3D"GS02" smbios.version=3D"2.7" =85 Kind regards, Patrick --=20 punkt.de GmbH * Kaiserallee 13a * 76133 Karlsruhe Tel. 0721 9109 0 * Fax 0721 9109 100 info@punkt.de http://www.punkt.de Gf: J=FCrgen Egeling AG Mannheim 108285 From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 11:51:05 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 681C9106564A; Thu, 15 Dec 2011 11:51:05 +0000 (UTC) (envelope-from andrey@zonov.org) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 959378FC0A; Thu, 15 Dec 2011 11:51:04 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so3789389wgb.31 for ; Thu, 15 Dec 2011 03:51:03 -0800 (PST) MIME-Version: 1.0 Received: by 10.180.94.195 with SMTP id de3mr4291716wib.14.1323949862696; Thu, 15 Dec 2011 03:51:02 -0800 (PST) Received: by 10.180.19.67 with HTTP; Thu, 15 Dec 2011 03:51:02 -0800 (PST) X-Originating-IP: [95.108.170.17] In-Reply-To: <20111214204201.GA7372@icarus.home.lan> References: <4EE7BF77.5000504@zonov.org> <20111213221501.GA85563@icarus.home.lan> <4EE8E6E3.7050202@zonov.org> <20111214182252.GA5176@icarus.home.lan> <4EE8FD3E.8030902@zonov.org> <20111214204201.GA7372@icarus.home.lan> Date: Thu, 15 Dec 2011 15:51:02 +0400 Message-ID: From: Andrey Zonov To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: alc@freebsd.org, kib@freebsd.org, freebsd-stable@freebsd.org Subject: Re: directory listing hangs in "ufs" state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 11:51:05 -0000 On Thu, Dec 15, 2011 at 12:42 AM, Jeremy Chadwick wrote: > On Wed, Dec 14, 2011 at 11:47:10PM +0400, Andrey Zonov wrote: > > On 14.12.2011 22:22, Jeremy Chadwick wrote: > > >On Wed, Dec 14, 2011 at 10:11:47PM +0400, Andrey Zonov wrote: > > >>Hi Jeremy, > > >> > > >>This is not hardware problem, I've already checked that. I also ran > > >>fsck today and got no errors. > > >> > > >>After some more exploration of how mongodb works, I found that then > > >>listing hangs, one of mongodb thread is in "biowr" state for a long > > >>time. It periodically calls msync(MS_SYNC) accordingly to ktrace > > >>out. > > >> > > >>If I'll remove msync() calls from mongodb, how often data will be > > >>sync by OS? > > >> > > >>-- > > >>Andrey Zonov > > >> > > >>On 14.12.2011 2:15, Jeremy Chadwick wrote: > > >>>On Wed, Dec 14, 2011 at 01:11:19AM +0400, Andrey Zonov wrote: > > >>>> > > >>>>Have you any ideas what is going on? or how to catch the problem? > > >>> > > >>>Assuming this isn't a file on the root filesystem, try booting the > > >>>machine in single-user mode and using "fsck -f" on the filesystem in > > >>>question. > > >>> > > >>>Can you verify there's no problems with the disk this file lives on as > > >>>well (smartctl -a /dev/disk)? I'm doubting this is the problem, but > > >>>thought I'd mention it. > > > > > >I have no real answer, I'm sorry. msync(2) indicates it's effectively > > >deprecated (see BUGS). It looks like this is effectively a mmap-version > > >of fsync(2). > > > > I replaced msync(2) with fsync(2). Unfortunately, from man pages it > > is not obvious that I can do this. Anyway, thanks. > > Sorry, that wasn't what I was implying. Let me try to explain > differently. > > msync(2) looks, to me, like an mmap-specific version of fsync(2). Based > on the man page, it seems that the with msync() you can effectively > guaranteed flushing of certain pages within an mmap()'d region to disk. > fsync() would flush **all** buffers/internal pages to be flushed to > disk. > > One would need to look at the code to mongodb to find out what it's > actually doing with msync(). That is to say, if it's doing something > like this (I probably have the semantics wrong -- I've never spent much > time with mmap()): > > fd = open("/some/file", O_RDWR); > ptr = mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0); > ret = msync(ptr, 65536, MS_SYNC); > /* or alternatively, this: > ret = msync(ptr, NULL, MS_SYNC); > */ > > Then this, to me, would be mostly the equivalent to: > > fd = fopen("/some/file", "r+"); > ret = fsync(fd); > > Otherwise, if it's calling msync() only on an address/location within > the region ptr points to, then that may be more efficient (less pages to > flush). > They call msync() for the whole file. So, there will not be any difference. > The mmap() arguments -- specifically flags (see man page) -- also play > a role here. The one that catches my attention is MAP_NOSYNC. So you > may need to look at the mongodb code to figure out what it's mmap() > call is. > > One might wonder why they don't just use open() with the O_SYNC. I > imagine that has to do with, again, performance; possibly the don't want > all I/O synchronous, and would rather flush certain pages in the mmap'd > region to disk as needed. I see the legitimacy in that approach (vs. > just using O_SYNC). > > There's really no easy way for me to tell you which is more efficient, > better, blah blah without spending a lot of time with a benchmarking > program that tests all of this, *plus* an entire system (world) built > with profiling. > I ran for two hours mongodb with fsync() and got the following: STARTED INBLK OUBLK MAJFLT MINFLT Thu Dec 15 10:34:52 2011 3 192744 314 3080182 This is output of `ps -o lstart,inblock,oublock,majflt,minflt -U mongodb'. Then I ran it with default msync(): STARTED INBLK OUBLK MAJFLT MINFLT Thu Dec 15 12:34:53 2011 0 7241555 79 5401945 There are also two graphics of disk business [1] [2]. The difference is significant, in 37 times! That what I expected to get. In commentaries for vm_object_page_clean() I found this: * When stuffing pages asynchronously, allow clustering. XXX we need a * synchronous clustering mode implementation. It means for me that msync(MS_SYNC) flush every page on disk in single IO transaction. If we multiply 4K and 37 we get 150K. This number is size of the single transaction in my experience. +alc@, kib@ Am I right? Is there any plan to implement this? > All of this would really fall into the hands of the mongodb people to > figure out, if you ask me. But I should note that mmap() on BSD behaves > and performs very differently than on, say, Linux; so if the authors > wrote what they did intended for Linux systems, I wouldn't be too > surprised. :-) > https://jira.mongodb.org/browse/SERVER-663 > > >I'm extremely confused by this problem. What you're describing above > is > > >that the process is "stuck in biowr state for a long time", but what you > > >stated originally was that the process was "stuck in ufs state for a > > >few minutes": > > > > Listing of the directory with mongodb files by ls(1) stuck in "ufs" > > state when one of mongodb's thread in "biowr" state. It looks like > > system holds global lock of the file which is msync(2)-ed and can't > > immediately return from lstat(2) call. > > Thanks for the clarification -- yes this helps. To some degree it makes > sense, some piece of the filesystem or VFS layer is blocking > intentionally. How to figure out what layer I do not know. Kernel > folks familiar with this aspect would need to chime in here. > > > >>I've got STABLE-8 (r221983) with mongodb-1.8.1 installed on it. A > > >>couple days ago I observed that listing of mongodb directory stuck in > > >>a few minutes in "ufs" state. > > > > > >Can we narrow down what we're talking about here? Does the process > > >actually deadlock? Or are you concerned about performance implications? > > > > > >I know nothing about this "mongodb" software, but the reason it's > > >calling msync() is because it wants to try and ensure that the data it > > >changed in an mmap()-mapped page to be reflected (fully written) on the > > >disk. This behaviour is fairly common within database software, but > > >"how often" the software chooses to do this is entirely a design > > >implementation choice by the authors. > > > > > >Meaning: if mongodb is either 1) continually calling msync(), or 2) > > >waiting for too long a period of time before calling msync(), > > >performance within the process will suffer. #1 could result in overall > > >bad performance, while #2 could result in a process that's spending a > > >lot of time doing I/O (flushing to disk) and therefore appears > > >"deadlocked" when in fact the kernel/subsystems are doing exactly what > > >they were told to do. > > > > > >Removing the msync() call could result in inconsistent data (possibly > > >non-recoverable) if the mongodb software crashes or if some other piece > > >(thread or child? Not sure) expects to open a new fd on that file which > > >has mmap()'d data. > > > > Yes, I clearly understand this. I think of any system tuning > > instead, but nothing arose in my head. > > Nor I. I think this is more of a userland/application thing than a > kernel thing, but there is a love-and-hate relationship between userland > and kernel when it comes to the above syscalls and framework. > > Wish I could be of more help -- sorry. :-( > Your help was very important. Thanks! > > -- > | Jeremy Chadwick jdc at parodius.com | > | Parodius Networking http://www.parodius.com/ | > | UNIX Systems Administrator Mountain View, CA, US | > | Making life hard for others since 1977. PGP 4BD6C0CB | > > [1] http://zonov.org/mongodb/155647.png [2] http://zonov.org/mongodb/155657.png -- Andrey Zonov From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 11:52:57 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 845B0106567C for ; Thu, 15 Dec 2011 11:52:57 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta03.emeryville.ca.mail.comcast.net (qmta03.emeryville.ca.mail.comcast.net [76.96.30.32]) by mx1.freebsd.org (Postfix) with ESMTP id 623F88FC1A for ; Thu, 15 Dec 2011 11:52:57 +0000 (UTC) Received: from omta22.emeryville.ca.mail.comcast.net ([76.96.30.89]) by qmta03.emeryville.ca.mail.comcast.net with comcast id 9Pny1i0011vN32cA3Psp4l; Thu, 15 Dec 2011 11:52:49 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta22.emeryville.ca.mail.comcast.net with comcast id 9Q9m1i0071t3BNj8iQ9mXM; Thu, 15 Dec 2011 12:09:47 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 95117102C19; Thu, 15 Dec 2011 03:52:54 -0800 (PST) Date: Thu, 15 Dec 2011 03:52:54 -0800 From: Jeremy Chadwick To: Michael Larabel Message-ID: <20111215115254.GA21530@icarus.home.lan> References: <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9D214.3070906@phoronix.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE9D214.3070906@phoronix.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "O. Hartmann" , Michael Ross , freebsd-performance@freebsd.org, Current FreeBSD , FreeBSD Stable Mailing List Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 11:52:57 -0000 On Thu, Dec 15, 2011 at 04:55:16AM -0600, Michael Larabel wrote: > On 12/15/2011 04:41 AM, Michael Ross wrote: > >Am 15.12.2011, 11:10 Uhr, schrieb Michael Larabel > >: > > > >>On 12/15/2011 02:48 AM, Michael Ross wrote: > > > >>>Anyway these tests were performed on different hardware, FWIW. > >>>And with different filesystems, different compilers, different GUIs... > >>> > >>> > >> > >>No, the same hardware was used for each OS. > >> > > > >The picture under the heading "System Hardware / Software" does > >not reflect that. > > > >Motherboard description differs, Chipset description for FreeBSD > >is empty. > > > > I was the on that carried out the testing and know that it was on > the same system. > > All of the testing, including the system tables, is fully automated. > Under FreeBSD sometimes the parsing of some component strings isn't > as nice as Linux and other supported operating systems by the > Phoronix Test Suite. For the BSD motherboard string parsing it's > grabbing hw.vendor/hw.product from sysctl. > > Is there a better place to read the motherboard DMI information from? I *think* what you're referring to is SMBIOS strings -- and these are available from kenv(1) / kenv(2), not sysctl. But keep reading for why SMBIOS data is not 100% reliable (greatly depends on the hardware). For actual device strings/etc. for all devices on busses (PCI, AGP, etc.) you can use pciconf -lvcb. That's about as good as it's going to get via software. SMBIOS data (e.g. smbios.{bios,chassis,planar,system}) is never going to give you fully-identifiable data; I can point you to tons of systems where the data inserted there is nonsense, sometimes even just ASCII spaces (and that is the fault of the system vendor/BIOS manufacturer, not FreeBSD). Sometimes identical strings are used across completely different systems/boards (sometimes even server-class boards like ones from Supermicro). And PCI vendor strings don't give you things like speeds, frequency/voltages, etc.. Sometimes this matters. For example (just making something up): "the video benchmark was horrible on FreeBSD", when in fact it turned out that a run of "pciconf -lvcb" showed your PCIe card was running at x4 link speed instead of x16. The best place to get your specifications from are: * The box * The physical hardware (by physically inspecting it) * The user manual / product documentation/ * Purchase orders from whoever bought the hardware * And, of course, operational speed (if possible) from the OS/userland utilities When I read a benchmark/review, I have to assume the person is doing them on a system they have 100% control over, all the way down to the hardware. Thus, they should know what exact hardware they have. Also, when publishing results online, you should take the time to proofread everything (with a 2nd set of eyes if possible) and be patient and thorough. People like accuracy, especially when there's hard data/evidence to back it up that can be made available for download. Try to understand: so many review-esque sites consist of individuals who do not understand even remotely what they're doing. I'm going to give you two examples -- one personal, one word-of-mouth but from someone I trust dearly. I have a "reverse analysis" of Anantech's Intel 510 SSD review that has been sitting in my "draft" folder on my blog for a month now because I'm downright afraid to publish how their data seems completely and totally wrong (with evidence to prove it). I'm afraid/stalling because I want to make absolutely damn sure I'm not missing some key piece of evidence that explains it, and I've had multiple people read it and go "...wow, I didn't notice that, that benchmark data makes no sense", but I'm STILL reluctant. The last thing I want to do is "publish" something that sparks a controversy where it turns out I'm wrong (and I AM wrong, quite often!). As for the other: http://www.overclockers.com/bulldozer-architecture-explained/ The author of this "review" talks about CPU arch and is praised for writing a "wonderful article that speaks the truth". But sadly that doesn't appear to be the case. A colleague of mine is long-time friends with another individual who is getting his Ph.D in computer architecture and recently submit a paper to a journal (and was published/accepted) which has published papers on things like RAID (when it was first introduced as a concept/method), and hardware watchpoints. Said individual read the above "review" and described it as, quote, "the worst article on computer architecture on the entire Internet". One of the amusing quotes (that got me laughing since I did understand it; my understanding of CPUs on a silicon level is limited, I'm just an old 65xxx assembly programmer...) was how the article states "this is the first time AMD has implemented branch prediction". Sigh. Here's the kicker: said individual immediately recognised that the article was a near dry cut-and-paste from one of two commonly-used computer architecture books in college/universities; the first book is basically a "beginner's guide to CPU architecture". The book is also a bit old at that. Individual proceeded to look up where the article author went to school, and noted that said school's CPU architecture course **ends** with that book. The user/viewer demographic of overclockers.com is going to be significantly different from that of phoronix.com -- you know that I'm sure. The point is that you should be aware that there is going to be significant discussions that come from publishing such benchmark comparisons with such a demographic. Things that indicate severe performance differential (e.g. "10x to 100x worse") are going to be focused on and criticised -- and hopefully in a socially-agreeable manner[1] -- and in a much different way than, say, a 3D video card review site ("lol ur pc sux if u spend onl $4000 on it lol"). The first step is to try and figure out what exactly you're seeing and why it's so significantly different when compared to other OSes. [1]: I'm sure by now you know that the BSDs in general tend to harbour a community of folks who are more argumentative/aggressive than, say, Linux (generally speaking). In this thread though, I think all of us really want to assist in some way to figure out what exactly is going on here, scheduler-wise, and see if we can put something together to hand developers who are "responsible" for said code and see what comes of it. Remember, we're all here to try and make things better... I hope. :-) Footnote: It's nice meeting you (indirectly), I was always curious who did the phoronix.com reviews/"stuff" when it came to FreeBSD. Greetings! -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 12:07:00 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 338711065670 for ; Thu, 15 Dec 2011 12:07:00 +0000 (UTC) (envelope-from andrej@antiszoc.hu) Received: from mail.deployis.eu (mail.deployis.eu [217.20.135.253]) by mx1.freebsd.org (Postfix) with ESMTP id ACFF98FC0C for ; Thu, 15 Dec 2011 12:06:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=antiszoc.hu; s=default; h=Message-ID:References:In-Reply-To:Subject:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=36myQMJ01DVeGJqQjSH48A/mRaO4ZIRzKwmCePdVcH0=; b=fAJqejFvKUhYQh/zPGS4I6Y5ci5klskif0I0kQJpbSTV02plpWFcnZnxcmZK0iA8K+N5tLQ3OI+W17gYmnetUYui5tBWBs8AaVxGh+tMpQyZSl+fkZAA4il4ymFGSj1L; Authentication-Results: mail.deployis.eu dkim=none Received: from localhost ([127.0.0.1]:52054 helo=mail.deployis.eu) by mail.deployis.eu with esmtp (Exim 4.71 #1 (Debian)) id 1Rb9Wq-00037X-SK from for ; Thu, 15 Dec 2011 12:31:13 +0100 Received: from 131-75.95.80.dunakanyar.net ([80.95.75.131]) by mail.deployis.eu with HTTP (HTTP/1.1 POST); Thu, 15 Dec 2011 12:31:12 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Thu, 15 Dec 2011 12:31:12 +0100 From: =?UTF-8?Q?G=C3=B3t_Andr=C3=A1s?= To: In-Reply-To: <4EE9D7B8.50209@phoronix.com> References: <4EE1EAFE.3070408@m5p.com><4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com><4EE933C6.4020209@zedat.fu-berlin.de><20111215024249.GA13557@icarus.home.lan><4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9D214.3070906@phoronix.com> <95A7C3EFE7B4406EAFF0259CD8B61D1C@multiplay.co.uk> <4EE9D7B8.50209@phoronix.com> Message-ID: <575ccb9c8c7915f04d526abaa194d9f5@antiszoc.hu> X-Sender: andrej@antiszoc.hu User-Agent: RoundCube Webmail X-Mail-Status-postahivatal: trustedmail (from 127.0.0.1) X-Spam-Score-postahivatal: -37 Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 12:07:00 -0000 It would be also nice to see whether compiling the kernel and the world for the specific machine counts. I think it's an advantage of FreeBSD, but never could do a benchmark comparing this. Andras 15.12.2011 12:19 napján Michael Larabel ezt írta: > On 12/15/2011 05:02 AM, Steven Hartland wrote: >> ----- Original Message ----- From: "Michael Larabel" >> >> >>> >>> I was the on that carried out the testing and know that it was on >>> the same system. >>> >>> All of the testing, including the system tables, is fully >>> automated. Under FreeBSD sometimes the parsing of some component >>> strings isn't as nice as Linux and other supported operating systems >>> by the Phoronix Test Suite. For the BSD motherboard string parsing >>> it's grabbing hw.vendor/hw.product from sysctl. Is there a better >>> place to read the motherboard DMI information from? >> >> dmidecode may provide better info? >> >> Regards >> Steve > > dmidecode is used on Linux for parsing some of the hardware > information. I think I looked at using it for BSD too, but offhand I > don't recall what the problem was. I'll check into it again with the > latest release when time allows. > > Michael > >> >> ================================================ >> This e.mail is private and confidential between Multiplay (UK) Ltd. >> and the person or entity to whom it is addressed. In the event of >> misdirection, the recipient is prohibited from using, copying, >> printing or otherwise disseminating it or any information contained in >> it. >> In the event of misdirection, illegible or incomplete transmission >> please telephone +44 845 868 1337 >> or return the E.mail to postmaster@multiplay.co.uk. >> >> > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 12:29:26 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 343EE106566C for ; Thu, 15 Dec 2011 12:29:26 +0000 (UTC) (envelope-from prvs=1330762df1=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id A7FFD8FC15 for ; Thu, 15 Dec 2011 12:29:25 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 15 Dec 2011 12:29:06 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 15 Dec 2011 12:29:01 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50017141405.msg for ; Thu, 15 Dec 2011 12:28:59 +0000 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1330762df1=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-stable@freebsd.org Message-ID: From: "Steven Hartland" To: References: <4EE1EAFE.3070408@m5p.com><4EE2AE64.9060802@m5p.com><4EE88343.2050302@m5p.com><4EE933C6.4020209@zedat.fu-berlin.de><20111215024249.GA13557@icarus.home.lan><4EE9A2A0.80607@zedat.fu-berlin.de><4EE9C79B.7080607@phoronix.com><4EE9D214.3070906@phoronix.com><95A7C3EFE7B4406EAFF0259CD8B61D1C@multiplay.co.uk><4EE9D7B8.50209@phoronix.com> <575ccb9c8c7915f04d526abaa194d9f5@antiszoc.hu> Date: Thu, 15 Dec 2011 12:29:19 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="utf-8"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 12:29:26 -0000 Having a quick look at those results aren't there a few annomolies e.g. THREADED I/O TESTER for Oracle reports 10255.75MB/s Which is clearly impossible for a single HD system meaning its basically caching the entire data set? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 12:56:03 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 04486106564A; Thu, 15 Dec 2011 12:56:03 +0000 (UTC) (envelope-from sjg@evilcode.net) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9B0658FC12; Thu, 15 Dec 2011 12:56:02 +0000 (UTC) Received: by iakl21 with SMTP id l21so5157427iak.13 for ; Thu, 15 Dec 2011 04:56:02 -0800 (PST) MIME-Version: 1.0 Received: by 10.42.161.10 with SMTP id r10mr2375060icx.22.1323952367276; Thu, 15 Dec 2011 04:32:47 -0800 (PST) Received: by 10.50.163.106 with HTTP; Thu, 15 Dec 2011 04:32:47 -0800 (PST) In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 05:32:47 -0700 Message-ID: From: "Samuel J. Greear" To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 Cc: FreeBSD Stable Mailing List , freebsd-performance@freebsd.org, Current FreeBSD , "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 12:56:03 -0000 > Well, the only way it's going to get fixed is if someone sits down, > replicates it, and starts to document exactly what it is that these > benchmarks are/aren't doing. > I think you will find that investigation is largely a waste of time, because not only are some of these benchmarks just downright silly, there are huge differences in the environments (compiler versions), etc., etc. leading to a largely apples/oranges comparison. But also the the analysis and reporting of the results by Phoronix is simply moronic to the point of being worse than useful, they are spreading misinformation. Take the first test as an example, Blogbench read. This doesn't raise any red flags, right? At least not until you realize that Blogbench isn't a read test, it's a read/write test. So what they have done here is run a read/write test and then thrown away the write results for both platforms and reported only the read results. If you dig down into the actual results, http://openbenchmarking.org/result/1112113-AR-ORACLELIN37 -- you will see two Blogbench numbers, one for read and another for write. These were both taken from the same Blogbench run, so FreeBSD optimizes writes over reads, that's probably a good thing for your data but a bad thing when someone totally misrepresents benchmark results. Other benchmarks in the Phoronix suite and their representations are similarly flawed, _ALL_ of these results should be ignored and no time should be wasted by any FreeBSD committer further evaluating this garbage. (Yes, I have been down this rabbit hole). Best, Sam From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 13:01:21 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 84332106566C; Thu, 15 Dec 2011 13:01:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id E3CEB8FC1A; Thu, 15 Dec 2011 13:01:20 +0000 (UTC) Received: from alf.home (alf.kiev.zoral.com.ua [10.1.1.177]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id pBFD1CKT028125 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 Dec 2011 15:01:12 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from alf.home (kostik@localhost [127.0.0.1]) by alf.home (8.14.5/8.14.5) with ESMTP id pBFD1CS4048896; Thu, 15 Dec 2011 15:01:12 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by alf.home (8.14.5/8.14.5/Submit) id pBFD1Btm048895; Thu, 15 Dec 2011 15:01:11 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: alf.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 15 Dec 2011 15:01:11 +0200 From: Kostik Belousov To: Andrey Zonov Message-ID: <20111215130111.GN50300@deviant.kiev.zoral.com.ua> References: <4EE7BF77.5000504@zonov.org> <20111213221501.GA85563@icarus.home.lan> <4EE8E6E3.7050202@zonov.org> <20111214182252.GA5176@icarus.home.lan> <4EE8FD3E.8030902@zonov.org> <20111214204201.GA7372@icarus.home.lan> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="GC54EUgUnLpxzrwX" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: alc@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: directory listing hangs in "ufs" state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 13:01:21 -0000 --GC54EUgUnLpxzrwX Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Dec 15, 2011 at 03:51:02PM +0400, Andrey Zonov wrote: > On Thu, Dec 15, 2011 at 12:42 AM, Jeremy Chadwick > wrote: >=20 > > On Wed, Dec 14, 2011 at 11:47:10PM +0400, Andrey Zonov wrote: > > > On 14.12.2011 22:22, Jeremy Chadwick wrote: > > > >On Wed, Dec 14, 2011 at 10:11:47PM +0400, Andrey Zonov wrote: > > > >>Hi Jeremy, > > > >> > > > >>This is not hardware problem, I've already checked that. I also ran > > > >>fsck today and got no errors. > > > >> > > > >>After some more exploration of how mongodb works, I found that then > > > >>listing hangs, one of mongodb thread is in "biowr" state for a long > > > >>time. It periodically calls msync(MS_SYNC) accordingly to ktrace > > > >>out. > > > >> > > > >>If I'll remove msync() calls from mongodb, how often data will be > > > >>sync by OS? > > > >> > > > >>-- > > > >>Andrey Zonov > > > >> > > > >>On 14.12.2011 2:15, Jeremy Chadwick wrote: > > > >>>On Wed, Dec 14, 2011 at 01:11:19AM +0400, Andrey Zonov wrote: > > > >>>> > > > >>>>Have you any ideas what is going on? or how to catch the problem? > > > >>> > > > >>>Assuming this isn't a file on the root filesystem, try booting the > > > >>>machine in single-user mode and using "fsck -f" on the filesystem = in > > > >>>question. > > > >>> > > > >>>Can you verify there's no problems with the disk this file lives o= n as > > > >>>well (smartctl -a /dev/disk)? I'm doubting this is the problem, b= ut > > > >>>thought I'd mention it. > > > > > > > >I have no real answer, I'm sorry. msync(2) indicates it's effective= ly > > > >deprecated (see BUGS). It looks like this is effectively a mmap-ver= sion > > > >of fsync(2). > > > > > > I replaced msync(2) with fsync(2). Unfortunately, from man pages it > > > is not obvious that I can do this. Anyway, thanks. > > > > Sorry, that wasn't what I was implying. Let me try to explain > > differently. > > > > msync(2) looks, to me, like an mmap-specific version of fsync(2). Based > > on the man page, it seems that the with msync() you can effectively > > guaranteed flushing of certain pages within an mmap()'d region to disk. > > fsync() would flush **all** buffers/internal pages to be flushed to > > disk. > > > > One would need to look at the code to mongodb to find out what it's > > actually doing with msync(). That is to say, if it's doing something > > like this (I probably have the semantics wrong -- I've never spent much > > time with mmap()): > > > > fd =3D open("/some/file", O_RDWR); > > ptr =3D mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0); > > ret =3D msync(ptr, 65536, MS_SYNC); > > /* or alternatively, this: > > ret =3D msync(ptr, NULL, MS_SYNC); > > */ > > > > Then this, to me, would be mostly the equivalent to: > > > > fd =3D fopen("/some/file", "r+"); > > ret =3D fsync(fd); > > > > Otherwise, if it's calling msync() only on an address/location within > > the region ptr points to, then that may be more efficient (less pages to > > flush). > > >=20 > They call msync() for the whole file. So, there will not be any differen= ce. >=20 >=20 > > The mmap() arguments -- specifically flags (see man page) -- also play > > a role here. The one that catches my attention is MAP_NOSYNC. So you > > may need to look at the mongodb code to figure out what it's mmap() > > call is. > > > > One might wonder why they don't just use open() with the O_SYNC. I > > imagine that has to do with, again, performance; possibly the don't want > > all I/O synchronous, and would rather flush certain pages in the mmap'd > > region to disk as needed. I see the legitimacy in that approach (vs. > > just using O_SYNC). > > > > There's really no easy way for me to tell you which is more efficient, > > better, blah blah without spending a lot of time with a benchmarking > > program that tests all of this, *plus* an entire system (world) built > > with profiling. > > >=20 > I ran for two hours mongodb with fsync() and got the following: > STARTED INBLK OUBLK MAJFLT MINFLT > Thu Dec 15 10:34:52 2011 3 192744 314 3080182 >=20 > This is output of `ps -o lstart,inblock,oublock,majflt,minflt -U mongodb'. >=20 > Then I ran it with default msync(): > STARTED INBLK OUBLK MAJFLT MINFLT > Thu Dec 15 12:34:53 2011 0 7241555 79 5401945 >=20 > There are also two graphics of disk business [1] [2]. >=20 > The difference is significant, in 37 times! That what I expected to get. >=20 > In commentaries for vm_object_page_clean() I found this: >=20 > * When stuffing pages asynchronously, allow clustering. XXX we nee= d a > * synchronous clustering mode implementation. >=20 > It means for me that msync(MS_SYNC) flush every page on disk in single IO > transaction. If we multiply 4K and 37 we get 150K. This number is size = of > the single transaction in my experience. >=20 > +alc@, kib@ >=20 > Am I right? Is there any plan to implement this? Current buffer clustering code can only do only async writes. In fact, I am not quite sure what would consitute the sync clustering, because the ability to delay the write is important to be able to cluster at all. Also, I am not sure that lack of clustering is the biggest problem. IMO, the fact that each write is sync is the first problem there. It would be quite a work to add the tracking of the issued writes to the vm_object_page_clean() and down the stack. Esp. due to custom page write vops in several fses. The only guarantee that POSIX requires from msync(MS_SYNC) is that the writes are finished when the syscall returned, and not that the writes are done synchronously. Below is the hack which should help if the msync()ed region contains the mapping of the whole file, since it is possible to fsync() the file after all writes are scheduled asynchronous then. It will causes unneeded metadata update, but I think it would be much faster still. diff --git a/sys/vm/vm_object.c b/sys/vm/vm_object.c index 250b769..a9de554 100644 --- a/sys/vm/vm_object.c +++ b/sys/vm/vm_object.c @@ -938,7 +938,7 @@ vm_object_sync(vm_object_t object, vm_ooffset_t offset,= vm_size_t size, vm_object_t backing_object; struct vnode *vp; struct mount *mp; - int flags; + int flags, fsync_after; =20 if (object =3D=3D NULL) return; @@ -971,11 +971,26 @@ vm_object_sync(vm_object_t object, vm_ooffset_t offse= t, vm_size_t size, (void) vn_start_write(vp, &mp, V_WAIT); vfslocked =3D VFS_LOCK_GIANT(vp->v_mount); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); - flags =3D (syncio || invalidate) ? OBJPC_SYNC : 0; - flags |=3D invalidate ? OBJPC_INVAL : 0; + if (syncio && !invalidate && offset =3D=3D 0 && + OFF_TO_IDX(size) =3D=3D object->size) { + /* + * If syncing the whole mapping of the file, + * it is faster to schedule all the writes in + * async mode, also allowing the clustering, + * and then wait for i/o to complete. + */ + flags =3D 0; + fsync_after =3D TRUE; + } else { + flags =3D (syncio || invalidate) ? OBJPC_SYNC : 0; + flags |=3D invalidate ? (OBJPC_SYNC | OBJPC_INVAL) : 0; + fsync_after =3D FALSE; + } VM_OBJECT_LOCK(object); vm_object_page_clean(object, offset, offset + size, flags); VM_OBJECT_UNLOCK(object); + if (fsync_after) + (void) VOP_FSYNC(vp, MNT_WAIT, curthread); VOP_UNLOCK(vp, 0); VFS_UNLOCK_GIANT(vfslocked); vn_finished_write(mp); --GC54EUgUnLpxzrwX Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7p75cACgkQC3+MBN1Mb4inKQCfQH7Ln2h+5ERC3KqmowDwefh8 fUcAmwcKFtD68MKzUvmmhARdkVP5G41x =P/8l -----END PGP SIGNATURE----- --GC54EUgUnLpxzrwX-- From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 13:36:23 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34DC31065670; Thu, 15 Dec 2011 13:36:23 +0000 (UTC) (envelope-from michael.larabel@phoronix.com) Received: from phx1.phoronix.com (173.192.77.202-static.reverse.softlayer.com [173.192.77.202]) by mx1.freebsd.org (Postfix) with ESMTP id C962E8FC0C; Thu, 15 Dec 2011 13:36:22 +0000 (UTC) Received: from c-98-193-96-120.hsd1.il.comcast.net ([98.193.96.120] helo=[172.16.93.133]) by phx1.phoronix.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69) (envelope-from ) id 1RbBTx-0005eR-HR; Thu, 15 Dec 2011 07:36:21 -0600 Message-ID: <4EE9F7D2.4050607@phoronix.com> Date: Thu, 15 Dec 2011 07:36:18 -0600 From: Michael Larabel User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Stefan Esser References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9F546.6060503@freebsd.org> In-Reply-To: <4EE9F546.6060503@freebsd.org> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - phx1.phoronix.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - phoronix.com Cc: FreeBSD Stable Mailing List , Current FreeBSD , Michael Ross , freebsd-performance@freebsd.org, "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 13:36:23 -0000 On 12/15/2011 07:25 AM, Stefan Esser wrote: > Am 15.12.2011 11:10, schrieb Michael Larabel: >> No, the same hardware was used for each OS. >> >> In terms of the software, the stock software stack for each OS was used. > Just curious: Why did you choose ZFS on FreeBSD, while UFS2 (with > journaling enabled) should be an obvious choice since it is more similar > in concept to ext4 and since that is what most FreeBSD users will use > with FreeBSD? I was running some ZFS vs. UFS tests as well and this happened to have ZFS on when I was running some other tests. > > Did you tune the ZFS ARC (e.g. vfs.zfs.arc_max="6G") for the tests? The OS was left in its stock configuration. > > And BTW: Did your measured run times account for the effect, that Linux > keeps much more dirty data in the buffer cache (FreeBSD has a low limit > on dirty buffers since under realistic load the already cached data is > much more likely to be reused and thus more valuable than freshly > written data; aggressively caching dirty data would significantly reduce > throughput and responsiveness under high load). Given the hardware specs > of the test system, I guess that Linux accepts at least 100 times the > dirty data in the buffer cache, compared to FreeBSD (where this number > is at most in the tens of megabyte range). > > If you did not, then your results do not represent a server load (which > I'd expect relevant, if you are testing against Oracle Linux 6.1 > server), where continuous performance is required. Tests that run on an > idle system starting in a clean state and ignoring background flushing > of the buffer cache after the timed program has stopped are perhaps > useful for a very lowly loaded PC, but not for a system with high load > average as the default. > > I bet that if you compared the systems under higher load (which > admittedly makes it much harder to get sensible numbers for the program > under test) or with reduced buffer cache size (or raise the dirty buffer > limit in FreeBSD accordingly, which ought to be possible with sysctl > and/or boot time tuneables, e.g. "vfs.hidirtybuffers"). > > And a last remark: Single benchmark runs do not provide reliable data. > FreeBSD comes with "ministat" to check the significance of benchmark > results. Each test should be repeated at least 5 times for meaningful > averages with acceptable confidence level. The Phoronix Test Suite runs most tests a minimum of three times and if the standard deviation exceeds 3.5% the run count is dynamically increased, among other safeguards. -- Michael > > Regards, STefan > From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 13:38:24 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 516D6106566C for ; Thu, 15 Dec 2011 13:38:24 +0000 (UTC) (envelope-from se@freebsd.org) Received: from nm14-vm0.bullet.mail.sp2.yahoo.com (nm14-vm0.bullet.mail.sp2.yahoo.com [98.139.91.246]) by mx1.freebsd.org (Postfix) with SMTP id 27D598FC1C for ; Thu, 15 Dec 2011 13:38:24 +0000 (UTC) Received: from [98.139.91.66] by nm14.bullet.mail.sp2.yahoo.com with NNFMP; 15 Dec 2011 13:25:31 -0000 Received: from [208.71.42.199] by tm6.bullet.mail.sp2.yahoo.com with NNFMP; 15 Dec 2011 13:25:31 -0000 Received: from [127.0.0.1] by smtp210.mail.gq1.yahoo.com with NNFMP; 15 Dec 2011 13:25:31 -0000 X-Yahoo-Newman-Id: 255449.13797.bm@smtp210.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: D.d22ugVM1kW77kJo_KuiJ4ECL6o_d1T5Ot8tz7TqMkZlin sfpHDsvh31_RkqrqtPHcYzl8LRB6hLub297OjeYgJ6cNnZmnUV.0T9SBElQl BaCAkt41gK1QdPUwcV.sql4kxUReZNPG32fY9Mlrs4JnWXzN9h9TDrmXgrj5 IMI3VW2cpFUgvE8PDFu4fU2kKbUna5Y9ugkVhaZ_bsRSJeagYGMGmH0fav7v UEShzLmPeMsl9t6nAfB0yy3WVcLdZNdvNzE7JrxK20S4vZMgdYOzHiS1oMyN yNQ2eKgihkl1mHUBwm4q0tc_ts350EU7VsFSY_sDst7Ohv9LfDxu72pGN42b klqFVdeAwvmf72wdUZXzfhBbSZbXHDYKgoAvjhnyObXUHwiByw7gZ5YDELJf Jly.jDQ.d7PoTBg-- X-Yahoo-SMTP: iDf2N9.swBDAhYEh7VHfpgq0lnq. Received: from [192.168.119.20] (se@81.173.151.71 with plain) by smtp210.mail.gq1.yahoo.com with SMTP; 15 Dec 2011 05:25:30 -0800 PST Message-ID: <4EE9F546.6060503@freebsd.org> Date: Thu, 15 Dec 2011 14:25:26 +0100 From: Stefan Esser User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Michael Larabel References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> In-Reply-To: <4EE9C79B.7080607@phoronix.com> Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit Cc: FreeBSD Stable Mailing List , Current FreeBSD , Michael Ross , freebsd-performance@freebsd.org, "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 13:38:24 -0000 Am 15.12.2011 11:10, schrieb Michael Larabel: > No, the same hardware was used for each OS. > > In terms of the software, the stock software stack for each OS was used. Just curious: Why did you choose ZFS on FreeBSD, while UFS2 (with journaling enabled) should be an obvious choice since it is more similar in concept to ext4 and since that is what most FreeBSD users will use with FreeBSD? Did you tune the ZFS ARC (e.g. vfs.zfs.arc_max="6G") for the tests? And BTW: Did your measured run times account for the effect, that Linux keeps much more dirty data in the buffer cache (FreeBSD has a low limit on dirty buffers since under realistic load the already cached data is much more likely to be reused and thus more valuable than freshly written data; aggressively caching dirty data would significantly reduce throughput and responsiveness under high load). Given the hardware specs of the test system, I guess that Linux accepts at least 100 times the dirty data in the buffer cache, compared to FreeBSD (where this number is at most in the tens of megabyte range). If you did not, then your results do not represent a server load (which I'd expect relevant, if you are testing against Oracle Linux 6.1 server), where continuous performance is required. Tests that run on an idle system starting in a clean state and ignoring background flushing of the buffer cache after the timed program has stopped are perhaps useful for a very lowly loaded PC, but not for a system with high load average as the default. I bet that if you compared the systems under higher load (which admittedly makes it much harder to get sensible numbers for the program under test) or with reduced buffer cache size (or raise the dirty buffer limit in FreeBSD accordingly, which ought to be possible with sysctl and/or boot time tuneables, e.g. "vfs.hidirtybuffers"). And a last remark: Single benchmark runs do not provide reliable data. FreeBSD comes with "ministat" to check the significance of benchmark results. Each test should be repeated at least 5 times for meaningful averages with acceptable confidence level. Regards, STefan From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 13:48:57 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 72041106566C for ; Thu, 15 Dec 2011 13:48:57 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta03.westchester.pa.mail.comcast.net (qmta03.westchester.pa.mail.comcast.net [76.96.62.32]) by mx1.freebsd.org (Postfix) with ESMTP id 12DF78FC1B for ; Thu, 15 Dec 2011 13:48:56 +0000 (UTC) Received: from omta19.westchester.pa.mail.comcast.net ([76.96.62.98]) by qmta03.westchester.pa.mail.comcast.net with comcast id 9RTv1i00127AodY53Roxlm; Thu, 15 Dec 2011 13:48:57 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta19.westchester.pa.mail.comcast.net with comcast id 9Rov1i00Y1t3BNj3fRov2n; Thu, 15 Dec 2011 13:48:57 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id F0D36102C19; Thu, 15 Dec 2011 05:48:53 -0800 (PST) Date: Thu, 15 Dec 2011 05:48:53 -0800 From: Jeremy Chadwick To: "Samuel J. Greear" Message-ID: <20111215134853.GA24753@icarus.home.lan> References: <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "O. Hartmann" , Adrian Chadd , Current FreeBSD , FreeBSD Stable Mailing List , freebsd-performance@freebsd.org Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 13:48:57 -0000 On Thu, Dec 15, 2011 at 05:32:47AM -0700, Samuel J. Greear wrote: > > Well, the only way it's going to get fixed is if someone sits down, > > replicates it, and starts to document exactly what it is that these > > benchmarks are/aren't doing. > > > > I think you will find that investigation is largely a waste of time, > because not only are some of these benchmarks just downright silly, > there are huge differences in the environments (compiler versions), > etc., etc. leading to a largely apples/oranges comparison. But also > the the analysis and reporting of the results by Phoronix is simply > moronic to the point of being worse than useful, they are spreading > misinformation. > > Take the first test as an example, Blogbench read. This doesn't raise > any red flags, right? At least not until you realize that Blogbench > isn't a read test, it's a read/write test. So what they have done here > is run a read/write test and then thrown away the write results for > both platforms and reported only the read results. If you dig down > into the actual results, > http://openbenchmarking.org/result/1112113-AR-ORACLELIN37 -- you will > see two Blogbench numbers, one for read and another for write. These > were both taken from the same Blogbench run, so FreeBSD optimizes > writes over reads, that's probably a good thing for your data but a > bad thing when someone totally misrepresents benchmark results. > > Other benchmarks in the Phoronix suite and their representations are > similarly flawed, _ALL_ of these results should be ignored and no time > should be wasted by any FreeBSD committer further evaluating this > garbage. (Yes, I have been down this rabbit hole). For sake of argument, let's say we throw out the Phoronix benchmarks as a data source (I don't think the benchmark specifically implied or stated "this is all because of SCHED_ULE" though; remember, that's what we're supposed to be focusing on. There may not be a direct correlation between the Phoronix benchmarks and the ULE issue reported here...). That said: thrown out, data ignored, done. Now what? Where are we? We're right back where we were a day or two ago; meaning no closer to solving the dilemma reported by users and SCHED_ULE. Heck, we're not even sure if there is an issue, other than some folks confirming that SCHED_4BSD performs better for them (that's what started this whole thread), and there are at least a couple which have stated this. So given the above semi-devil's-advocate response -- Sam, do you have something positive or progressive to offer so we can move forward on the ULE vs. 4BSD debacle? :-) The smiley is meant to be sincere, not sarcastic. I'm getting to the point where I'm considering formulating a private mail to Jeff Roberson, requesting that he be aware of the discussion that's happening (not that he necessarily follow or read it), and that based on what I can tell we're at a roadblock -- nobody so far is absolutely certain how to "benchmark" and compare ULE vs. 4BSD in multiple ways, so that those of us involved here can run such utilities and provide the data somewhere central for devs to review. I only mention this because so far I haven't seen anyone really say "okay, this is what we should be using for these kinds of tests". Yay nature of the beast. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 13:51:54 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8BF4E106566B for ; Thu, 15 Dec 2011 13:51:54 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 0C60B8FC17 for ; Thu, 15 Dec 2011 13:51:53 +0000 (UTC) Received: from [192.168.100.59] (varna2.digsys.bg [193.68.0.70]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id pBFDpWq6002143 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 15 Dec 2011 15:51:41 +0200 (EET) (envelope-from daniel@digsys.bg) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=us-ascii From: Daniel Kalchev In-Reply-To: <4EE9F546.6060503@freebsd.org> Date: Thu, 15 Dec 2011 15:51:33 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9F546.6060503@freebsd.org> To: Stefan Esser X-Mailer: Apple Mail (2.1251.1) Cc: Michael Larabel , FreeBSD Stable Mailing List , Current FreeBSD , Michael Ross , freebsd-performance@freebsd.org, "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 13:51:54 -0000 On Dec 15, 2011, at 3:25 PM, Stefan Esser wrote: > Am 15.12.2011 11:10, schrieb Michael Larabel: >> No, the same hardware was used for each OS. >>=20 >> In terms of the software, the stock software stack for each OS was = used. >=20 > Just curious: Why did you choose ZFS on FreeBSD, while UFS2 (with > journaling enabled) should be an obvious choice since it is more = similar > in concept to ext4 and since that is what most FreeBSD users will use > with FreeBSD? Or perhaps, since it is "server" Linux distribution, use ZFS on Linux as = well. With identical tuning on both Linux and FreeBSD. Having the same = FS used by both OS will help make the comparison more sensible for FS = I/O. Daniel= From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 13:58:56 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C59F0106566C for ; Thu, 15 Dec 2011 13:58:56 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 2B90D8FC0C for ; Thu, 15 Dec 2011 13:58:55 +0000 (UTC) Received: from [192.168.100.59] (varna2.digsys.bg [193.68.0.70]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id pBFDwgXO002173 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 15 Dec 2011 15:58:48 +0200 (EET) (envelope-from daniel@digsys.bg) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=windows-1252 From: Daniel Kalchev In-Reply-To: <20111215134853.GA24753@icarus.home.lan> Date: Thu, 15 Dec 2011 15:58:44 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <0C72D682-CF5E-42D6-91F3-FEF1AB02F5D6@digsys.bg> References: <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <20111215134853.GA24753@icarus.home.lan> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1251.1) Cc: Adrian Chadd , "Samuel J. Greear" , Current FreeBSD , FreeBSD Stable Mailing List , freebsd-performance@freebsd.org, "O. Hartmann" Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 13:58:56 -0000 On Dec 15, 2011, at 3:48 PM, Jeremy Chadwick wrote: [=85] > That said: thrown out, data ignored, done. >=20 > Now what? Where are we? We're right back where we were a day or two > ago; meaning no closer to solving the dilemma reported by users and > SCHED_ULE. Heck, we're not even sure if there is an issue, other than > some folks confirming that SCHED_4BSD performs better for them (that's > what started this whole thread), and there are at least a couple which > have stated this. But, are any of these benchmarks really engaging the 4BSD/ULE scheduler = differences? Most such benchmarks are run on a system with no other load = whatsoever and in no way represent real world experience. What is more, I believe in such benchmarks "the system feels sluggish" = is not measured at all. Even if it is measured, if in such case the = benchmark finishes "better" - that is, faster, or say, makes the system = freeze for the user for the duration of the test -- it will be = considered "win", because the benchmark suite ran faster on that = particular system -- whereas a system which ran the benchmark fast, = provided good interactive response etc would be considered "loser". I think it is not good idea to hijack this thread, but instead focusing = on the other SCHED_ULE bashing thread to define an reasonable benchmark = or a set of benchmarks rather -- so that many would run it and provide = feedback. Daniel= From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 14:19:48 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 48617106566C for ; Thu, 15 Dec 2011 14:19:48 +0000 (UTC) (envelope-from prvs=1330762df1=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id C31A98FC15 for ; Thu, 15 Dec 2011 14:19:47 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 15 Dec 2011 14:19:41 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 15 Dec 2011 14:19:41 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50017142104.msg; Thu, 15 Dec 2011 14:19:38 +0000 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1330762df1=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <5350EDAC93604B6E93EDBD65A3038F88@multiplay.co.uk> From: "Steven Hartland" To: , References: <4EE1EAFE.3070408@m5p.com><4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com><4EE933C6.4020209@zedat.fu-berlin.de><20111215004205.GA11556@icarus.home.lan> <4eea1a4a.nJRbEc1jgKpVnVk4%perryh@pluto.rain.com> Date: Thu, 15 Dec 2011 14:20:04 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: tevans.uk@googlemail.com, attilio@freebsd.org, george+freebsd@m5p.com, freebsd-stable@freebsd.org, ohartman@zedat.fu-berlin.de Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 14:19:48 -0000 With all the discussion I thought I'd give a buildworld benchmark a go here on a spare 24 core machine. ULE tested fine but with 4BSD it wont even boot panicing with the following:- http://screensnapr.com/v/hwysGV.png This is on a clean 8.2-RELEASE-p4 Upgrading to RELENG_9 fixed this but its a bit concerning that just changing the scheduler would cause the machine to panic on boot. Its only a single run so varience could be high but here's the result of a buildworld on this machine running the two different schedulers:- 4BSD: 24m54.10s real 2h43m12.42s user 56m20.07s sys ULE: 23m54.68s real 2h34m59.04s user 50m59.91s sys What really sticks out is that this is over double that of an 8.2 buildworld on the same machine with the same kernel ULE: 11m12.76s real 1h27m59.39s user 28m59.57s sys This was run 9.0-PRERELEASE kernel due to 4BSD panicing on boot under 8.2. So for this use ULE vs 4BSD is neither here-nor-there but 9.0 buildworld is very slow (x2 slower) compared with 8.2 so whats a bigger question in my mind. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 14:28:26 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0D4B106564A; Thu, 15 Dec 2011 14:28:26 +0000 (UTC) (envelope-from michael.larabel@phoronix.com) Received: from phx1.phoronix.com (173.192.77.202-static.reverse.softlayer.com [173.192.77.202]) by mx1.freebsd.org (Postfix) with ESMTP id 9EBED8FC13; Thu, 15 Dec 2011 14:28:26 +0000 (UTC) Received: from c-98-193-96-120.hsd1.il.comcast.net ([98.193.96.120] helo=[172.16.93.133]) by phx1.phoronix.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69) (envelope-from ) id 1RbCIL-00079g-BN; Thu, 15 Dec 2011 08:28:25 -0600 Message-ID: <4EEA0406.2010002@phoronix.com> Date: Thu, 15 Dec 2011 08:28:22 -0600 From: Michael Larabel User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Sergey Matveychuk References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9F546.6060503@freebsd.org> <4EE9F7D2.4050607@phoronix.com> <4EEA0388.7010605@FreeBSD.org> In-Reply-To: <4EEA0388.7010605@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - phx1.phoronix.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - phoronix.com Cc: FreeBSD Stable Mailing List , Current FreeBSD , Michael Ross , freebsd-performance@freebsd.org, "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 14:28:26 -0000 On 12/15/2011 08:26 AM, Sergey Matveychuk wrote: > 15.12.2011 17:36, Michael Larabel пишет: >> On 12/15/2011 07:25 AM, Stefan Esser wrote: >>> Am 15.12.2011 11:10, schrieb Michael Larabel: >>>> No, the same hardware was used for each OS. >>>> >>>> In terms of the software, the stock software stack for each OS was >>>> used. >>> Just curious: Why did you choose ZFS on FreeBSD, while UFS2 (with >>> journaling enabled) should be an obvious choice since it is more >>> similar >>> in concept to ext4 and since that is what most FreeBSD users will use >>> with FreeBSD? >> >> I was running some ZFS vs. UFS tests as well and this happened to have >> ZFS on when I was running some other tests. >> > > Can we look at the tests? > My opinion is ZFS without tuning is much slower than UFS2. > http://www.phoronix.com/scan.php?page=news_item&px=MTAyNjg From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 14:37:34 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 29BE71065672; Thu, 15 Dec 2011 14:37:34 +0000 (UTC) (envelope-from c.kworr@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 277108FC15; Thu, 15 Dec 2011 14:37:32 +0000 (UTC) Received: by eaaf13 with SMTP id f13so2808734eaa.13 for ; Thu, 15 Dec 2011 06:37:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=MwE4PWmmZsIrl3Fv2tOOByGa4iu4zNYVZ+pfk7K/CbQ=; b=WEPBi2oUCm0WMb3/rGnGlDFgZ6oMJAoS6jfGmZDV2uJWn+fiP+AeUG7zYHXRWMkv+a fThzvtxH7VuuhFW1S6pihiYXbW8GoAd8h4yMJaSEwk5LVceVG+DSHOMXSuYpCaJPFzF4 EijjxPSKUZ49lumVSGr7w4tAHs62WzyZHpTVc= Received: by 10.205.122.133 with SMTP id gg5mr1158106bkc.65.1323959852122; Thu, 15 Dec 2011 06:37:32 -0800 (PST) Received: from green.tandem.local (utwig.xim.bz. [91.216.237.46]) by mx.google.com with ESMTPS id j9sm14858864bkd.2.2011.12.15.06.37.29 (version=SSLv3 cipher=OTHER); Thu, 15 Dec 2011 06:37:30 -0800 (PST) Message-ID: <4EEA0627.3000409@gmail.com> Date: Thu, 15 Dec 2011 16:37:27 +0200 From: Volodymyr Kostyrko User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111111 Thunderbird/8.0 MIME-Version: 1.0 To: Jeremy Chadwick References: <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <20111215134853.GA24753@icarus.home.lan> In-Reply-To: <20111215134853.GA24753@icarus.home.lan> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Adrian Chadd , "Samuel J. Greear" , Current FreeBSD , FreeBSD Stable Mailing List , freebsd-performance@freebsd.org, "O. Hartmann" Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 14:37:34 -0000 15.12.2011 15:48, Jeremy Chadwick wrote: > I'm getting to the point where I'm considering formulating a private > mail to Jeff Roberson, requesting that he be aware of the discussion > that's happening (not that he necessarily follow or read it), and that > based on what I can tell we're at a roadblock -- nobody so far is > absolutely certain how to "benchmark" and compare ULE vs. 4BSD in > multiple ways, so that those of us involved here can run such utilities > and provide the data somewhere central for devs to review. I only > mention this because so far I haven't seen anyone really say "okay, this > is what we should be using for these kinds of tests". Yay nature of the > beast. I'll try to summarize and propose a test scenario. I don't know whether this helps or not. We should have two different task types for this one. The first would be Super Affine tasks. They should use few to none syscalls, use medium math, have low memory footprint. No syscalls means this tasks will never stop for memory/disk or other activity so each time the queue is looked upon this task will be ready to run. Medium math means this shouldn't be just a simple big loop so that processor will really compute something with this data. Low memory footprint means this task can reside with data on CPU L1 cache for eons. I'm not sure about branch prediction, should it be distorted or not... The other task type would be Worker. It doesn't matter what it does but it agressively uses syscalls like working with files/directories. There should be at least one SA-task per core and at least 10 (?) W-tasks per core. -- Sphinx of black quartz judge my vow. From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 14:44:40 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 87A0C1065677; Thu, 15 Dec 2011 14:44:40 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from mail.ciam.ru (mail.ciam.ru [91.209.218.18]) by mx1.freebsd.org (Postfix) with ESMTP id 315948FC0C; Thu, 15 Dec 2011 14:44:40 +0000 (UTC) Received: from dhcp170-160-red.yandex.net ([95.108.170.160] helo=dhcp170-205-red.yandex.net) by mail.ciam.ru with esmtpa (Exim 4.x) id 1RbCGG-0004u8-IB; Thu, 15 Dec 2011 17:26:16 +0300 Message-ID: <4EEA0388.7010605@FreeBSD.org> Date: Thu, 15 Dec 2011 18:26:16 +0400 From: Sergey Matveychuk User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Michael Larabel References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9F546.6060503@freebsd.org> <4EE9F7D2.4050607@phoronix.com> In-Reply-To: <4EE9F7D2.4050607@phoronix.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: FreeBSD Stable Mailing List , Current FreeBSD , Michael Ross , freebsd-performance@freebsd.org, "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 14:44:40 -0000 15.12.2011 17:36, Michael Larabel пишет: > On 12/15/2011 07:25 AM, Stefan Esser wrote: >> Am 15.12.2011 11:10, schrieb Michael Larabel: >>> No, the same hardware was used for each OS. >>> >>> In terms of the software, the stock software stack for each OS was used. >> Just curious: Why did you choose ZFS on FreeBSD, while UFS2 (with >> journaling enabled) should be an obvious choice since it is more similar >> in concept to ext4 and since that is what most FreeBSD users will use >> with FreeBSD? > > I was running some ZFS vs. UFS tests as well and this happened to have > ZFS on when I was running some other tests. > Can we look at the tests? My opinion is ZFS without tuning is much slower than UFS2. From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 15:04:28 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D356106564A; Thu, 15 Dec 2011 15:04:28 +0000 (UTC) (envelope-from lars@e-new.0x20.net) Received: from mail.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) by mx1.freebsd.org (Postfix) with ESMTP id E837E8FC1C; Thu, 15 Dec 2011 15:04:27 +0000 (UTC) Received: from mail.0x20.net (mail.0x20.net [217.69.76.211]) by mail.0x20.net (Postfix) with ESMTP id B44916A6619; Thu, 15 Dec 2011 16:04:26 +0100 (CET) X-Virus-Scanned: amavisd-new at mail.0x20.net Received: from mail.0x20.net ([217.69.76.211]) by mail.0x20.net (mail.0x20.net [217.69.76.211]) (amavisd-new, port 10024) with ESMTP id UFt6g4reHwxF; Thu, 15 Dec 2011 16:04:26 +0100 (CET) Received: from e-new.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.0x20.net (Postfix) with ESMTPS id 6601D6A61CD; Thu, 15 Dec 2011 16:04:26 +0100 (CET) Received: from e-new.0x20.net (localhost [127.0.0.1]) by e-new.0x20.net (8.14.4/8.14.4) with ESMTP id pBFF4QFp060435; Thu, 15 Dec 2011 16:04:26 +0100 (CET) (envelope-from lars@e-new.0x20.net) Received: (from lars@localhost) by e-new.0x20.net (8.14.4/8.14.4/Submit) id pBFF4OUM059271; Thu, 15 Dec 2011 16:04:24 +0100 (CET) (envelope-from lars) Date: Thu, 15 Dec 2011 16:04:24 +0100 From: Lars Engels To: Steven Hartland Message-ID: <20111215150424.GO54469@e-new.0x20.net> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215004205.GA11556@icarus.home.lan> <4eea1a4a.nJRbEc1jgKpVnVk4%perryh@pluto.rain.com> <5350EDAC93604B6E93EDBD65A3038F88@multiplay.co.uk> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="PWWBnvahXisRTnWJ" Content-Disposition: inline In-Reply-To: <5350EDAC93604B6E93EDBD65A3038F88@multiplay.co.uk> X-Editor: VIM - Vi IMproved 7.3 X-Operation-System: FreeBSD 8.2-RELEASE-p3 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: tevans.uk@googlemail.com, perryh@pluto.rain.com, attilio@freebsd.org, george+freebsd@m5p.com, freebsd-stable@freebsd.org, ohartman@zedat.fu-berlin.de, freebsd@jdc.parodius.com Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 15:04:28 -0000 --PWWBnvahXisRTnWJ Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Dec 15, 2011 at 02:20:04PM -0000, Steven Hartland wrote: > With all the discussion I thought I'd give a buildworld > benchmark a go here on a spare 24 core machine. ULE > tested fine but with 4BSD it wont even boot panicing > with the following:- > http://screensnapr.com/v/hwysGV.png >=20 > This is on a clean 8.2-RELEASE-p4 >=20 > Upgrading to RELENG_9 fixed this but its a bit concerning > that just changing the scheduler would cause the machine > to panic on boot. >=20 > Its only a single run so varience could be high but here's > the result of a buildworld on this machine running the > two different schedulers:- > 4BSD: 24m54.10s real 2h43m12.42s user 56m20.07s sys > ULE: 23m54.68s real 2h34m59.04s user 50m59.91s sys >=20 > What really sticks out is that this is over double that > of an 8.2 buildworld on the same machine with the same > kernel > ULE: 11m12.76s real 1h27m59.39s user 28m59.57s sys 9.0 ships with gcc and clang which both need to be compiled, 8.2 only has gcc. >=20 > This was run 9.0-PRERELEASE kernel due to 4BSD panicing > on boot under 8.2. >=20 > So for this use ULE vs 4BSD is neither here-nor-there > but 9.0 buildworld is very slow (x2 slower) compared > with 8.2 so whats a bigger question in my mind. >=20 > Regards > Steve --PWWBnvahXisRTnWJ Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7qDHgACgkQKc512sD3afjIYwCgn3jnoJIKN8DZCxYJTP+SXF/7 wWUAn1a4d2gUG9DvPx/qscebVHUAXg2y =lqnD -----END PGP SIGNATURE----- --PWWBnvahXisRTnWJ-- From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 15:31:50 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 418EF106564A; Thu, 15 Dec 2011 15:31:50 +0000 (UTC) (envelope-from prvs=1330762df1=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 882AC8FC13; Thu, 15 Dec 2011 15:31:49 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 15 Dec 2011 15:31:45 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 15 Dec 2011 15:31:44 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50017142661.msg; Thu, 15 Dec 2011 15:31:42 +0000 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1330762df1=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <02B71680F10B43DEBB5F8B05912025D9@multiplay.co.uk> From: "Steven Hartland" To: "Lars Engels" References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215004205.GA11556@icarus.home.lan> <4eea1a4a.nJRbEc1jgKpVnVk4%perryh@pluto.rain.com> <5350EDAC93604B6E93EDBD65A3038F88@multiplay.co.uk> <20111215150424.GO54469@e-new.0x20.net> Date: Thu, 15 Dec 2011 15:32:05 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="utf-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: tevans.uk@googlemail.com, perryh@pluto.rain.com, attilio@freebsd.org, george+freebsd@m5p.com, freebsd-stable@freebsd.org, ohartman@zedat.fu-berlin.de, freebsd@jdc.parodius.com Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 15:31:50 -0000 Lars Engels wrote: > 9.0 ships with gcc and clang which both need to be compiled, 8.2 only > has gcc. Ahh, any reason we need both, and is it possible to disable clang? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 15:43:16 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A42B51065675; Thu, 15 Dec 2011 15:43:16 +0000 (UTC) (envelope-from lists@eitanadler.com) Received: from mail-lpp01m010-f54.google.com (mail-lpp01m010-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id E36E98FC14; Thu, 15 Dec 2011 15:43:15 +0000 (UTC) Received: by lahl5 with SMTP id l5so1425494lah.13 for ; Thu, 15 Dec 2011 07:43:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eitanadler.com; s=0xdeadbeef; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=7GViZ3zduRgL804gBEIXKXvIkrgln+mNuzqiFwxOZM4=; b=fmqUeffuWIfneWgaTWz9HGy+dfTY6D7lGBUip+meMQQaI58QXJlyS9k8lUAJsghB3j v+bKqFjFgsgszn471WYh2m8/5cvNkgQURSBPFtk4BAbB/xndz5UbjK59DdIQ3PYWDlFP vRSMFBkJawmcOZ117lV5deUij1QMWXgijqWv8= Received: by 10.152.134.10 with SMTP id pg10mr3716619lab.3.1323963794538; Thu, 15 Dec 2011 07:43:14 -0800 (PST) MIME-Version: 1.0 Received: by 10.152.23.68 with HTTP; Thu, 15 Dec 2011 07:42:43 -0800 (PST) In-Reply-To: <02B71680F10B43DEBB5F8B05912025D9@multiplay.co.uk> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215004205.GA11556@icarus.home.lan> <4eea1a4a.nJRbEc1jgKpVnVk4%perryh@pluto.rain.com> <5350EDAC93604B6E93EDBD65A3038F88@multiplay.co.uk> <20111215150424.GO54469@e-new.0x20.net> <02B71680F10B43DEBB5F8B05912025D9@multiplay.co.uk> From: Eitan Adler Date: Thu, 15 Dec 2011 10:42:43 -0500 Message-ID: To: Steven Hartland Content-Type: text/plain; charset=UTF-8 Cc: tevans.uk@googlemail.com, perryh@pluto.rain.com, Lars Engels , attilio@freebsd.org, george+freebsd@m5p.com, freebsd-stable@freebsd.org, ohartman@zedat.fu-berlin.de, freebsd@jdc.parodius.com Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 15:43:16 -0000 On Thu, Dec 15, 2011 at 10:32 AM, Steven Hartland wrote: > Lars Engels wrote: >> >> 9.0 ships with gcc and clang which both need to be compiled, 8.2 only >> has gcc. > > > Ahh, any reason we need both, and is it possible to disable clang? man src.conf add WITHOUT_CLANG=yes to /etc/src.conf -- Eitan Adler From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 16:04:29 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9EE4C1065670 for ; Thu, 15 Dec 2011 16:04:29 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id DAED58FC08 for ; Thu, 15 Dec 2011 16:04:28 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id RAA00711; Thu, 15 Dec 2011 17:49:51 +0200 (EET) (envelope-from avg@FreeBSD.org) Message-ID: <4EEA171F.8040408@FreeBSD.org> Date: Thu, 15 Dec 2011 17:49:51 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Steven Hartland References: <4EE1EAFE.3070408@m5p.com><4EE2AE64.9060802@m5p.com><4EE88343.2050302@m5p.com><4EE933C6.4020209@zedat.fu-berlin.de><20111215024249.GA13557@icarus.home.lan><4EE9A2A0.80607@zedat.fu-berlin.de><4EE9C79B.7080607@phoronix.com><4EE9D214.3070906@phoronix.com><95A7C3EFE7B4406EAFF0259CD8B61D1C@multiplay.co.uk><4EE9D7B8.50209@phoronix.com> <575ccb9c8c7915f04d526abaa194d9f5@antiszoc.hu> In-Reply-To: X-Enigmail-Version: undefined Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:04:29 -0000 on 15/12/2011 14:29 Steven Hartland said the following: > Having a quick look at those results aren't there a few annomolies e.g. THREADED > I/O TESTER for Oracle reports 10255.75MB/s > > Which is clearly impossible for a single HD system meaning > its basically caching the entire data set? I think that the Stefan Esser's post in this thread has likely nailed this one. -- Andriy Gapon From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 16:25:55 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8CBB5106566B for ; Thu, 15 Dec 2011 16:25:55 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 195088FC0A for ; Thu, 15 Dec 2011 16:25:54 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so4232165wgb.31 for ; Thu, 15 Dec 2011 08:25:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=e18RyzCm5bGkuu7/104/bsa7n8pIGCgc5evEUK/H9bQ=; b=ILZHqqg87Dn2V/EH0m4F6fmvObcZX/J3Ez5Qoi6vSWEIGhqIgkjG4U4V/aD0mwZEC0 3u6wlYkLdr3WXZ88Mvyr6tL8JXCHE2PnvDdaUS1RIqPwppNO2Nntd9ah0gFS96nSDTcb lZsGJs+gbOjB8vB0xxvSifnP9l7IdfLmB4DWo= MIME-Version: 1.0 Received: by 10.227.60.4 with SMTP id n4mr2895748wbh.9.1323966352515; Thu, 15 Dec 2011 08:25:52 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Thu, 15 Dec 2011 08:25:51 -0800 (PST) In-Reply-To: <4EE1EAFE.3070408@m5p.com> References: <4EE1EAFE.3070408@m5p.com> Date: Thu, 15 Dec 2011 17:25:51 +0100 X-Google-Sender-Auth: Ve5xDZvMqomh5W63iycsnn-5km4 Message-ID: From: Attilio Rao To: George Mitchell Content-Type: multipart/mixed; boundary=00151747c69622e1e404b423f00d X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Andrey Chernov , Doug Barton , freebsd-stable@freebsd.org, Steve Kargl Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:25:55 -0000 --00151747c69622e1e404b423f00d Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable 2011/12/9 George Mitchell : > dnetc is an open-source program from http://www.distributed.net/. =C2=A0I= t > tries a brute-force approach to cracking RC4 puzzles and also computes > optimal Golomb rulers. =C2=A0It starts up one process per CPU and runs at > nice 20 and is, for all intents and purposes, 100% compute bound. [Posting on the first message of the thread] I basically went through all the e-mail you just sent and identified 4 real report on which we could work on and summarizied in the attached Excel file. I'd like that George, Steve, Doug, Andrey and Mike possibly review the few datas there and add more, if they want, or make more important clarifications in particular about the Xorg presence (or rather not) in their workload. I've readed a couple of message in the thread pointing the finger to Xorg to be excessively CPU-intensive and I think they are right, we might try to find a solution for that at some point, but it is really a very edge case. Geroge's and Steve's case, instead, look very different from this and I want to analyze them in detail. George already provided schedgraph traces and for others, if they cannot provide them directly, I'd really appreciate they would at least describe in detail the workload so that I get a chance to reproduce it. If someone else thinks he has a specific problem that is not characterized by one of the cases above please let me know and I will put this in the chart. Thanks for the hard work you guys put in pointing out ULE's problem, I think we will get at the bottom of this if we keep up sharing thoughts and reports. Attilio --=20 Peace can only be achieved by understanding - A. Einstein --00151747c69622e1e404b423f00d-- From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 16:26:06 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E9FD10657A5; Thu, 15 Dec 2011 16:26:06 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id EC5318FC15; Thu, 15 Dec 2011 16:26:05 +0000 (UTC) Received: by qcse13 with SMTP id e13so1932910qcs.13 for ; Thu, 15 Dec 2011 08:26:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=qn2SX6Icz3xoH1/LZzTteZ2wkKvJHUss5vkQbFTGw88=; b=fgLJMqgdAh/wrr7kqbGLvkkEYs/vl6PwMjS1eUHzw+Eqwmeba6B1V4Bqvz3ZzFiXbA uBH1/bfrSGDC5L+hehANmNWfGcPqfSl4swHJjcmGBNxNONeZcJEIg1zVYOm3Hw0io+gZ U+OFVnzlUpdDILUKmKplz5cJrlJthd9/6/aY8= MIME-Version: 1.0 Received: by 10.180.74.211 with SMTP id w19mr6552377wiv.7.1323966364698; Thu, 15 Dec 2011 08:26:04 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Thu, 15 Dec 2011 08:26:04 -0800 (PST) In-Reply-To: <4EE8D607.1000504@sentex.net> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> <4EE8D607.1000504@sentex.net> Date: Thu, 15 Dec 2011 17:26:04 +0100 X-Google-Sender-Auth: RQHg7a4l9VdsnVFWmvoxYxji0G8 Message-ID: From: Attilio Rao To: Mike Tancsa Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Ivan Klymenko , mdf@freebsd.org, Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:26:06 -0000 2011/12/14 Mike Tancsa : > On 12/13/2011 7:01 PM, mdf@freebsd.org wrote: >> >> Has anyone experiencing problems tried to set sysctl kern.sched.steal_th= resh=3D1 ? >> >> I don't remember what our specific problem at $WORK was, perhaps it >> was just interrupt threads not getting serviced fast enough, but we've >> hard-coded this to 1 and removed the code that sets it in >> sched_initticks(). =C2=A0The same effect should be had by setting the >> sysctl after a box is up. > > FWIW, this does impact the performance of pbzip2 on an i7. Using a 1.1G f= ile > > pbzip2 -v -c big > /dev/null > > with burnP6 running in the background, > > sysctl kern.sched.steal_thresh=3D1 > vs > sysctl kern.sched.steal_thresh=3D3 > > > > =C2=A0 =C2=A0N =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Min =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 Max =C2=A0 =C2=A0 =C2=A0 =C2=A0Median =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 Avg =C2=A0 =C2=A0 =C2=A0 =C2=A0Stddev > x =C2=A010 =C2=A0 =C2=A0 38.005022 =C2=A0 =C2=A0 =C2=A038.42238 =C2=A0 = =C2=A0 38.194648 =C2=A0 =C2=A0 38.165052 =C2=A0 =C2=A00.15546188 > + =C2=A0 9 =C2=A0 =C2=A0 38.695417 =C2=A0 =C2=A0 40.595544 =C2=A0 =C2=A0 = 39.392127 =C2=A0 =C2=A0 39.435384 =C2=A0 =C2=A00.59814114 > Difference at 95.0% confidence > =C2=A0 =C2=A0 =C2=A0 =C2=A01.27033 +/- 0.412636 > =C2=A0 =C2=A0 =C2=A0 =C2=A03.32852% +/- 1.08119% > =C2=A0 =C2=A0 =C2=A0 =C2=A0(Student's t, pooled s =3D 0.425627) > > a value of 1 is *slightly* faster. Hi Mike, was that just the same codebase with the switch SCHED_4BSD/SCHED_ULE? Also, the results here should be in the 3% interval for the avg case, which is not yet at the 'alarm level' but could still be an indication. I still suspect I/O plays a big role here, however, thus it could be detemined by other factors. Could you retry the bench checking CPU usage and possible thread migration around for both cases? Thanks, Attilio --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 16:26:21 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4A6A6106579D for ; Thu, 15 Dec 2011 16:26:21 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-ww0-f42.google.com (mail-ww0-f42.google.com [74.125.82.42]) by mx1.freebsd.org (Postfix) with ESMTP id D31A88FC0C for ; Thu, 15 Dec 2011 16:26:20 +0000 (UTC) Received: by wgbds13 with SMTP id ds13so1377914wgb.1 for ; Thu, 15 Dec 2011 08:26:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=DO7ACBjoPPpDM3ygvSwYB5UwKbUXgxIPUYjuPY0gHQc=; b=ADERguF3ny0KDvSHkZPFlIDTyjeyDLZ4qHhXnOz76TDP5uajzHCm9IT4A1yACP6wvb Uc8DIaIx+BZWyZHUDBH1xfHmETaSJRsfGnUULx3OMXujnVWLAbofioeRczvgJ53mb3Xr XgPvkU7Nz3Y1Ckyaj8QMF/i27gWWouNz/rxvY= MIME-Version: 1.0 Received: by 10.216.137.230 with SMTP id y80mr1352142wei.56.1323966379781; Thu, 15 Dec 2011 08:26:19 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Thu, 15 Dec 2011 08:26:19 -0800 (PST) In-Reply-To: <4EE7093E.4050006@digsys.bg> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111213073615.GA69641@icarus.home.lan> <4EE7093E.4050006@digsys.bg> Date: Thu, 15 Dec 2011 17:26:19 +0100 X-Google-Sender-Auth: lZg6Px7ci9GZs18OqscJL78v0z8 Message-ID: From: Attilio Rao To: Daniel Kalchev Content-Type: text/plain; charset=UTF-8 Cc: freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:26:21 -0000 2011/12/13 Daniel Kalchev : > > > On 13.12.11 09:36, Jeremy Chadwick wrote: >> >> I personally would find it interesting if someone with a higher-end system >> (e.g. 2 physical CPUs, with 6 or 8 cores per CPU) was to do the same test >> (changing -jX to -j{numofcores} of course). > > > Is 4 way 8 core Opteron ok? That is 32 cores, 64GB RAM. > > Testing with buildworld in my opinion is not adequate, as it involves way > too much I/O. Any advice on proper testing methodology? I'm sure that I/O and pmap subsystem contention (because of buildworld) and TLB shootdown overhead (because of 32 CPUs) will be so overwhelming that you are not really going to benchmark the scheduler activity at all. However I still don't get what you want to verify exactly? Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 16:26:30 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 08DAC1065781; Thu, 15 Dec 2011 16:26:30 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 3BA088FC16; Thu, 15 Dec 2011 16:26:28 +0000 (UTC) Received: by eekc50 with SMTP id c50so2679678eek.13 for ; Thu, 15 Dec 2011 08:26:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=zm71dKbrHwDITPaDjF+CgkHwSvzGuXVS65Im4Up4UGA=; b=vy84MwsN3OjrK5jTpfX3KeozyU74p0n/QCFtpJkdxapqgaMyzMHrBEefLcpsxinmig qu6vaTOscLfWDyPT5APaxY5MNL5zcLqS7DN06TLqhWZAW9VtbR3AaVUpSL1+mloigV9E FRpDp64k/fatCR6282rafWlJbRDiY2eBPOvGI= MIME-Version: 1.0 Received: by 10.180.96.103 with SMTP id dr7mr6603082wib.16.1323966388055; Thu, 15 Dec 2011 08:26:28 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Thu, 15 Dec 2011 08:26:27 -0800 (PST) In-Reply-To: <20111213073615.GA69641@icarus.home.lan> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111213073615.GA69641@icarus.home.lan> Date: Thu, 15 Dec 2011 17:26:27 +0100 X-Google-Sender-Auth: 0ndvCp4x9Kuw_LOxQPoNVB24Q2Q Message-ID: From: Attilio Rao To: Jeremy Chadwick Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:26:30 -0000 2011/12/13 Jeremy Chadwick : > On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: >> > Not fully right, boinc defaults to run on idprio 31 so this isn't an >> > issue. And yes, there are cases where SCHED_ULE shows much better >> > performance then SCHED_4BSD. =C2=A0[...] >> >> Do we have any proof at hand for such cases where SCHED_ULE performs >> much better than SCHED_4BSD? Whenever the subject comes up, it is >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu > >> 2. But in the end I see here contradictionary statements. People >> complain about poor performance (especially in scientific environments), >> and other give contra not being the case. >> >> Within our department, we developed a highly scalable code for planetary >> science purposes on imagery. It utilizes present GPUs via OpenCL if >> present. Otherwise it grabs as many cores as it can. >> By the end of this year I'll get a new desktop box based on Intels new >> Sandy Bridge-E architecture with plenty of memory. If the colleague who >> developed the code is willing performing some benchmarks on the same >> hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most >> recent Suse. For FreeBSD I intent also to look for performance with both >> different schedulers available. > > This is in no way shape or form the same kind of benchmark as what > you're planning to do, but I thought I'd throw it out there for folks to > take in as they see fit. > > I know folks were focused mainly on buildworld. > > I personally would find it interesting if someone with a higher-end > system (e.g. 2 physical CPUs, with 6 or 8 cores per CPU) was to do the > same test (changing -jX to -j{numofcores} of course). > > -- > | Jeremy Chadwick =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0jdc at parodius.com= | > | Parodius Networking =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 http://www.parodius.com/ | > | UNIX Systems Administrator =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 Mountain View, CA, US | > | Making life hard for others since 1977. =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 PGP 4BD6C0CB | > > > sched_ule > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > - time make -j2 buildworld > =C2=A01689.831u 229.328s 18:46.20 170.4% 6566+2051k 432+4264io 4565pf+0w > - time make -j2 buildkernel > =C2=A0640.542u 87.737s 9:01.38 134.5% 6490+1920k 134+5968io 0pf+0w > > > sched_4bsd > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > - time make -j2 buildworld > =C2=A01662.793u 206.908s 17:12.02 181.1% 6578+2054k 23750+4271io 6451pf+0= w > - time make -j2 buildkernel > =C2=A0638.717u 76.146s 8:34.90 138.8% 6530+1927k 6415+5903io 0pf+0w > > > software > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > * sched_ule test: =C2=A0FreeBSD 8.2-STABLE, Thu Dec =C2=A01 04:37:29 PST = 2011 > * sched_4bsd test: FreeBSD 8.2-STABLE, Mon Dec 12 22:42:54 PST 2011 Hi Jeremy, thanks for the time you spent on this. However, I wanted to ask/let you note 3 things: 1) Did you use 2 different code base for the test? (one updated on December 1 and another one on December 12) 2) Please note that you should have repeated this test several times (basically until you don't get a standard deviation which is acceptable with ministat) and report the ministat output 3) The difference is less than 2% which I suspect is really statistically unuseful/the same I'm not really even surprised ULE is not faster than 4BSD in this case because usually buildworld/buildkernel tests are driven for the vast majority by I/O overhead rather than scheduler capacity. It would be more interesting to analyze how buildworld does while another type of workload is going on. Thanks, Attilio --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 16:38:28 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2FD39106567C; Thu, 15 Dec 2011 16:38:28 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id DB9AC8FC15; Thu, 15 Dec 2011 16:38:27 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.5/8.14.4) with ESMTP id pBFGcLBQ067563; Thu, 15 Dec 2011 11:38:21 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4EEA227E.7080704@sentex.net> Date: Thu, 15 Dec 2011 11:38:22 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Attilio Rao References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> <4EE8D607.1000504@sentex.net> In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.71 on IPv6:2607:f3e0:0:1::12 Cc: Ivan Klymenko , mdf@freebsd.org, Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:38:28 -0000 On 12/15/2011 11:26 AM, Attilio Rao wrote: > > Hi Mike, > was that just the same codebase with the switch SCHED_4BSD/SCHED_ULE? Hi Attilio, It was the same codebase. > Could you retry the bench checking CPU usage and possible thread > migration around for both cases? I can, but how do I do that ? ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 16:42:12 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3E911106566C; Thu, 15 Dec 2011 16:42:12 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 1CB118FC0A; Thu, 15 Dec 2011 16:42:10 +0000 (UTC) Received: by faaf16 with SMTP id f16so3582013faa.13 for ; Thu, 15 Dec 2011 08:42:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=8kUH6rJcxKtm25qBqXuAeBlmVWtVD5+Q5ma5HlD4/UY=; b=Ya4r1MsNOoyFFjTbRWnuqq4Ra5KhzIU8HozC5H8ltzEm+SBP4X8PZACbHOrS4yHl4Y 25bymJ4J8rp0Fqrsc9h4fzC4QOSmVi2D9lzjE9cfBRDhF+ZDj0bjcU078v8Clp34msjX 4gk07oPtDNL7ZAREgzlbPvlzDV+8w1/Ty9u+o= MIME-Version: 1.0 Received: by 10.180.96.103 with SMTP id dr7mr6732293wib.16.1323967329953; Thu, 15 Dec 2011 08:42:09 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Thu, 15 Dec 2011 08:42:09 -0800 (PST) In-Reply-To: <4EEA227E.7080704@sentex.net> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> <4EE8D607.1000504@sentex.net> <4EEA227E.7080704@sentex.net> Date: Thu, 15 Dec 2011 17:42:09 +0100 X-Google-Sender-Auth: rWvfkJJt4u5T0-KL-sef3vLPTr0 Message-ID: From: Attilio Rao To: Mike Tancsa Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Ivan Klymenko , mdf@freebsd.org, Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:42:12 -0000 2011/12/15 Mike Tancsa : > On 12/15/2011 11:26 AM, Attilio Rao wrote: >> >> Hi Mike, >> was that just the same codebase with the switch SCHED_4BSD/SCHED_ULE? > > Hi Attilio, > =C2=A0 =C2=A0 =C2=A0 =C2=A0It was the same codebase. > > >> Could you retry the bench checking CPU usage and possible thread >> migration around for both cases? > > I can, but how do I do that ? I'm thinking now to a better test-case for this: can you try that on a tmpfs volume? Also what filesystem you were using? How many CPUs were in place? Did you reboot before to move the steal_thresh value? Attilio --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 16:52:12 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 925D81065678; Thu, 15 Dec 2011 16:52:12 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id 492BF8FC21; Thu, 15 Dec 2011 16:52:12 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.5/8.14.4) with ESMTP id pBFGq9M5070372; Thu, 15 Dec 2011 11:52:09 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4EEA25BB.7040706@sentex.net> Date: Thu, 15 Dec 2011 11:52:11 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Attilio Rao References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> <4EE8D607.1000504@sentex.net> <4EEA227E.7080704@sentex.net> In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.71 on IPv6:2607:f3e0:0:1::12 Cc: Ivan Klymenko , mdf@freebsd.org, Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:52:12 -0000 On 12/15/2011 11:42 AM, Attilio Rao wrote: > > I'm thinking now to a better test-case for this: can you try that on a > tmpfs volume? There is enough RAM in the box so that it should not touch the disk, and I was sending the output to /dev/null, so it was not writing to the disk. > > Also what filesystem you were using? UFS > How many CPUs were in place? 4 > Did you reboot before to move the steal_thresh value? No. ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 16:56:40 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BDDF31065689; Thu, 15 Dec 2011 16:56:40 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id A393F8FC13; Thu, 15 Dec 2011 16:56:39 +0000 (UTC) Received: by eaaf13 with SMTP id f13so3011475eaa.13 for ; Thu, 15 Dec 2011 08:56:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=Kek1F5J+opXNanowC4R0970JGtNQGCoNTh4rCJ7uQUs=; b=vwcrBfVBxoHdDZ0X9irxmOTnwQG2xiAdkINUqIRLpowzvqixyUK/v7XoJLnIAfhsfg nuFVtMU9qs+Gk0AZ49qunFN6W5nCAy3GG2qFgHS0grwgBAQDI7Z8tgIL/CKsWQ9V8a9B lUkWrQmoGkWfels9rs4vFz+4uz8LtlVhXZtFA= MIME-Version: 1.0 Received: by 10.216.131.152 with SMTP id m24mr1934372wei.56.1323968198451; Thu, 15 Dec 2011 08:56:38 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Thu, 15 Dec 2011 08:56:38 -0800 (PST) In-Reply-To: <4EEA25BB.7040706@sentex.net> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> <4EE8D607.1000504@sentex.net> <4EEA227E.7080704@sentex.net> <4EEA25BB.7040706@sentex.net> Date: Thu, 15 Dec 2011 17:56:38 +0100 X-Google-Sender-Auth: eN811JKam6VzmdGL06meH8StiGo Message-ID: From: Attilio Rao To: Mike Tancsa Content-Type: text/plain; charset=UTF-8 Cc: Ivan Klymenko , mdf@freebsd.org, Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:56:40 -0000 2011/12/15 Mike Tancsa : > On 12/15/2011 11:42 AM, Attilio Rao wrote: >> >> I'm thinking now to a better test-case for this: can you try that on a >> tmpfs volume? > > There is enough RAM in the box so that it should not touch the disk, and > I was sending the output to /dev/null, so it was not writing to the disk. > >> >> Also what filesystem you were using? > > UFS > >> How many CPUs were in place? > > 4 > >> Did you reboot before to move the steal_thresh value? > > No. So, as very first thing, can you try the following: - Same codebase, etc. etc. - Make the test 4 times, discard the first and ministat for the other 3 - Reboot - Change the steal_thresh value - Make the test 4 times, discard the first and ministat for the other 3 Then report discarded values and the ministated one and we will have more informations I guess (also, I don't think devfs contention should play a role here, thus nevermind about it for now). Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 17:37:40 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A52B3106566C; Thu, 15 Dec 2011 17:37:40 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [IPv6:2a01:348:0:15:5d59:5c40:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id 03D268FC19; Thu, 15 Dec 2011 17:37:40 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id 0C711E6210; Thu, 15 Dec 2011 17:37:38 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=mail; bh=s2d05cZbY9DU CUzzFBeRf4FDdYg=; b=ab5i0x5z9hc3rM4fFszQhV/de3x22R1O8ckoFKw6ewIX ikGhYQhut0jJ28NzOLhZRDMKN0xvQCfLdzoBQuwCtlaCRPgrukyKarPvA6/4qEAJ BebuvB6rUakNZV7ejK2S8QsLoqK2GqgdO5wRkP+Rk3eVZHOlIiZB8bego2h7yYo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=mail; b=QQrs+y Jn/CM30GJHbVeuyPswp352BR7jPfmzVOfMogmlBXPnrqruqRefDw/OFpRR7kdSdE BpiX3FQ7WWMUPSxF9kbhbSmZhBoXaBN/5Z2MSyFPm/fC4bWEbx79mCUHaqkFd96U PmVMIcGRIxon4Bz2n117LoKVX9ddtLnMMbN2A= Received: from [192.168.1.68] (188-220-36-32.zone11.bethere.co.uk [188.220.36.32]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id 2CDEDE6209; Thu, 15 Dec 2011 17:37:37 +0000 (GMT) Message-ID: <4EEA3055.1030605@cran.org.uk> Date: Thu, 15 Dec 2011 17:37:25 +0000 From: Bruce Cran User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Steven Hartland References: <4EE1EAFE.3070408@m5p.com><4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com><4EE933C6.4020209@zedat.fu-berlin.de><20111215004205.GA11556@icarus.home.lan> <4eea1a4a.nJRbEc1jgKpVnVk4%perryh@pluto.rain.com> <5350EDAC93604B6E93EDBD65A3038F88@multiplay.co.uk> In-Reply-To: <5350EDAC93604B6E93EDBD65A3038F88@multiplay.co.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: tevans.uk@googlemail.com, perryh@pluto.rain.com, attilio@freebsd.org, george+freebsd@m5p.com, freebsd-stable@freebsd.org, ohartman@zedat.fu-berlin.de, freebsd@jdc.parodius.com Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 17:37:40 -0000 On 15/12/2011 14:20, Steven Hartland wrote: > So for this use ULE vs 4BSD is neither here-nor-there > but 9.0 buildworld is very slow (x2 slower) compared > with 8.2 so whats a bigger question in my mind. clang is new in 9.0 and takes a long time to build. -- Bruce Cran From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 17:49:00 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 988861065673 for ; Thu, 15 Dec 2011 17:49:00 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta03.westchester.pa.mail.comcast.net (qmta03.westchester.pa.mail.comcast.net [76.96.62.32]) by mx1.freebsd.org (Postfix) with ESMTP id 54B248FC16 for ; Thu, 15 Dec 2011 17:49:00 +0000 (UTC) Received: from omta22.westchester.pa.mail.comcast.net ([76.96.62.73]) by qmta03.westchester.pa.mail.comcast.net with comcast id 9SJC1i0071ap0As53VozoX; Thu, 15 Dec 2011 17:48:59 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta22.westchester.pa.mail.comcast.net with comcast id 9Voy1i01K1t3BNj3iVoydW; Thu, 15 Dec 2011 17:48:59 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 2B4DD102C19; Thu, 15 Dec 2011 09:48:57 -0800 (PST) Date: Thu, 15 Dec 2011 09:48:57 -0800 From: Jeremy Chadwick To: Attilio Rao Message-ID: <20111215174857.GA28551@icarus.home.lan> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111213073615.GA69641@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 17:49:00 -0000 On Thu, Dec 15, 2011 at 05:26:27PM +0100, Attilio Rao wrote: > 2011/12/13 Jeremy Chadwick : > > On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: > >> > Not fully right, boinc defaults to run on idprio 31 so this isn't an > >> > issue. And yes, there are cases where SCHED_ULE shows much better > >> > performance then SCHED_4BSD. ??[...] > >> > >> Do we have any proof at hand for such cases where SCHED_ULE performs > >> much better than SCHED_4BSD? Whenever the subject comes up, it is > >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > >> 2. But in the end I see here contradictionary statements. People > >> complain about poor performance (especially in scientific environments), > >> and other give contra not being the case. > >> > >> Within our department, we developed a highly scalable code for planetary > >> science purposes on imagery. It utilizes present GPUs via OpenCL if > >> present. Otherwise it grabs as many cores as it can. > >> By the end of this year I'll get a new desktop box based on Intels new > >> Sandy Bridge-E architecture with plenty of memory. If the colleague who > >> developed the code is willing performing some benchmarks on the same > >> hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > >> recent Suse. For FreeBSD I intent also to look for performance with both > >> different schedulers available. > > > > This is in no way shape or form the same kind of benchmark as what > > you're planning to do, but I thought I'd throw it out there for folks to > > take in as they see fit. > > > > I know folks were focused mainly on buildworld. > > > > I personally would find it interesting if someone with a higher-end > > system (e.g. 2 physical CPUs, with 6 or 8 cores per CPU) was to do the > > same test (changing -jX to -j{numofcores} of course). > > > > -- > > | Jeremy Chadwick ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??jdc at parodius.com | > > | Parodius Networking ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? http://www.parodius.com/ | > > | UNIX Systems Administrator ?? ?? ?? ?? ?? ?? ?? ?? ?? Mountain View, CA, US | > > | Making life hard for others since 1977. ?? ?? ?? ?? ?? ?? ?? PGP 4BD6C0CB | > > > > > > sched_ule > > =========== > > - time make -j2 buildworld > > ??1689.831u 229.328s 18:46.20 170.4% 6566+2051k 432+4264io 4565pf+0w > > - time make -j2 buildkernel > > ??640.542u 87.737s 9:01.38 134.5% 6490+1920k 134+5968io 0pf+0w > > > > > > sched_4bsd > > ============ > > - time make -j2 buildworld > > ??1662.793u 206.908s 17:12.02 181.1% 6578+2054k 23750+4271io 6451pf+0w > > - time make -j2 buildkernel > > ??638.717u 76.146s 8:34.90 138.8% 6530+1927k 6415+5903io 0pf+0w > > > > > > software > > ========== > > * sched_ule test: ??FreeBSD 8.2-STABLE, Thu Dec ??1 04:37:29 PST 2011 > > * sched_4bsd test: FreeBSD 8.2-STABLE, Mon Dec 12 22:42:54 PST 2011 > > Hi Jeremy, > thanks for the time you spent on this. > > However, I wanted to ask/let you note 3 things: > 1) Did you use 2 different code base for the test? (one updated on > December 1 and another one on December 12) No; src-all (/usr/src on this system) was not updated between December 1st and December 12th PST. I do believe I updated it today (15th PST). I can/will obviously hold off so that we have a consistent code base for comparing numbers between schedulers during buildworld and/or buildkernel. > 2) Please note that you should have repeated this test several times > (basically until you don't get a standard deviation which is > acceptable with ministat) and report the ministat output This is the first time I have heard of ministat(1). I'm pretty sure I see what it's for and how it applies to this situation, but boy that man page could use some clarification (I have 3 people looking at this thing right now trying to figure out what means what in the graph :-) ). Anyway, graph or not, I see the point. Regarding multiple tests: yup, you're absolutely right, the only way to do it would be to run a sequence of tests repeatedly (probably 10 per scheduler). Reboots and rm -fr /usr/obj/* would be required after each test too, to guarantee empty kernel caches (of all types) consistently every time. What I posted was supposed to give people just a "general idea" if there was any gigantic difference between the two, and there really isn't. But, as others have stated (and you below), buildworld may not be an effective way to "benchmark" what we're trying to test. Hence me wondering exactly what would make for a good test. Example: 1. Run + background some program that "beats on things" (I really don't know what; creation/deletion of threads? CPU benchmark? bonnie++?), with output going to /dev/null. 2. Run + background "time make -j2 buildworld" with output going to /dev/null 3. Record/save output from "time". 4. rm -fr /usr/obj && shutdown -r now 5. Repeat all steps ~10 times 6. Adjust kernel configuration file to use other scheduler 7. Repeat steps 1-5. What I'm trying to figure out is what #1 and #2 should be in the above example. > 3) The difference is less than 2% which I suspect is really > statistically unuseful/the same Understood. > I'm not really even surprised ULE is not faster than 4BSD in this case > because usually buildworld/buildkernel tests are driven for the vast > majority by I/O overhead rather than scheduler capacity. It would be > more interesting to analyze how buildworld does while another type of > workload is going on. Yup, agreed/understood, hence me trying to find out what would classify as a good stress test for all of this. I have a testbed system in my garage which I could set up to literally do all of this in a loop, meaning automate the entire above process and just let it go, writing stderr from time to a file (which wouldn't skew the results at all). Let me know what #1 and #2 above, re: "the workloads", should be and I'll be happy to set it up. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 17:58:56 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8061C106566B; Thu, 15 Dec 2011 17:58:56 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id EA6368FC13; Thu, 15 Dec 2011 17:58:55 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RbFa2-0002Al-Qy>; Thu, 15 Dec 2011 18:58:54 +0100 Received: from e178037243.adsl.alicedsl.de ([85.178.37.243] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RbFa2-00087M-Ls>; Thu, 15 Dec 2011 18:58:54 +0100 Message-ID: <4EEA3556.7030105@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 18:58:46 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Daniel Kalchev References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9F546.6060503@freebsd.org> In-Reply-To: X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig90C2E873E806961B03779663" X-Originating-IP: 85.178.37.243 Cc: Michael Larabel , FreeBSD Stable Mailing List , Current FreeBSD , Michael Ross , freebsd-performance@freebsd.org, Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 17:58:56 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig90C2E873E806961B03779663 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Am 12/15/11 14:51, schrieb Daniel Kalchev: >=20 > On Dec 15, 2011, at 3:25 PM, Stefan Esser wrote: >=20 >> Am 15.12.2011 11:10, schrieb Michael Larabel: >>> No, the same hardware was used for each OS. >>> >>> In terms of the software, the stock software stack for each OS was us= ed. >> >> Just curious: Why did you choose ZFS on FreeBSD, while UFS2 (with >> journaling enabled) should be an obvious choice since it is more simil= ar >> in concept to ext4 and since that is what most FreeBSD users will use >> with FreeBSD? >=20 >=20 > Or perhaps, since it is "server" Linux distribution, use ZFS on Linux a= s well. With identical tuning on both Linux and FreeBSD. Having the same = FS used by both OS will help make the comparison more sensible for FS I/O= =2E >=20 > Daniel_______________________________________________ Since ZFS in Linux can only be achieved via FUSE (ad far as I know), it is legitimate to compare ZFS and ext4. It would be much more competetive to compare Linux BTRFS and FreeBSD ZFS. Each OS does optimize on different filesystems and a user/manager can assume that the vendor offers the best performance available by turning on the default FS by a standard stock installation. Using ZFS on Linux would be a great disadvantage and the benchmark would turn out the same bullsh... as comparing Linux-domain only with FreeBSD weknesses only ... Linux distributions offer setups for desktop and server. The FreeBSD folks have the choice to do it themselfes. And maybe I'm one of those puritain people appreciating this. "Out of the box" OS is Windooze, with all its consequences. Oliver Post scriptum: It seems to be hard to follow the benchmark environment on Phoronix since the URL refers to a setup of different systems. --------------enig90C2E873E806961B03779663 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQEcBAEBAgAGBQJO6jVeAAoJEOgBcD7A/5N8ALYH/0un2B7HHTHdeoxEzN9UJ8x+ WhlqiupymlpJR2UJDkWlDRETa9JABFE6Iuc84iAPbcHExzyd6BbYMhr9pvX0OlCM p1IWXUHrpXzr3fs3qoWtQIJi4yr6/Wb2dvJJHBK8tuwyOd2XR9GC4/sIwFXpw7Up 7343FJKIhZXpacEkJP/rQz9PxjcmifzuuDQhwjvrbTJDoJn5CQvya9gcVFExlTBS 2qHd7UwCjRf6xiu9lhTEtYy4O5uZoqSLeprxTEowP4DRwSbBJ33Ix0eAGmEc4vfB OnijWNZVnR4J8VrSj1DXltb8t/wTKe9bWzT8lVf98cVK018vQ2h3juiASY/y++A= =6R/p -----END PGP SIGNATURE----- --------------enig90C2E873E806961B03779663-- From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 17:59:16 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0725310656EA for ; Thu, 15 Dec 2011 17:59:16 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 8423C8FC15 for ; Thu, 15 Dec 2011 17:59:15 +0000 (UTC) Received: from [192.168.100.122] (varna2.digsys.bg [193.68.0.70]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id pBFHx2bM003300 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 15 Dec 2011 19:59:07 +0200 (EET) (envelope-from daniel@digsys.bg) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=us-ascii From: Daniel Kalchev In-Reply-To: Date: Thu, 15 Dec 2011 19:59:01 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111213073615.GA69641@icarus.home.lan> <4EE7093E.4050006@digsys.bg> To: Attilio Rao X-Mailer: Apple Mail (2.1251.1) Cc: freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 17:59:16 -0000 On Dec 15, 2011, at 6:26 PM, Attilio Rao wrote: > 2011/12/13 Daniel Kalchev : >>=20 >>=20 >> On 13.12.11 09:36, Jeremy Chadwick wrote: >>>=20 >>> I personally would find it interesting if someone with a higher-end = system >>> (e.g. 2 physical CPUs, with 6 or 8 cores per CPU) was to do the same = test >>> (changing -jX to -j{numofcores} of course). >>=20 >>=20 >> Is 4 way 8 core Opteron ok? That is 32 cores, 64GB RAM. >>=20 >> Testing with buildworld in my opinion is not adequate, as it involves = way >> too much I/O. Any advice on proper testing methodology? >=20 > I'm sure that I/O and pmap subsystem contention (because of > buildworld) and TLB shootdown overhead (because of 32 CPUs) will be so > overwhelming that you are not really going to benchmark the scheduler > activity at all. Can't pmap / TLB be tuned for 32 CPUs and 64GB of RAM? >=20 > However I still don't get what you want to verify exactly? The obvious: is SCHED_ULE better or worse than SCHED_4BSD on such = platform.=20 Problem is how to test "interactivity" -- that is a blade server and = doesn't really have a display and keyboard, nor does it have X etc. I have spare pair of those, that might be put to crunch tests to see how = things compare for different scenarios - but I need ideas what to test, = really. Daniel= From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 18:00:47 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5810D106566C; Thu, 15 Dec 2011 18:00:46 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 02E8C8FC1A; Thu, 15 Dec 2011 18:00:45 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RbFbo-0002Sq-J3>; Thu, 15 Dec 2011 19:00:44 +0100 Received: from e178037243.adsl.alicedsl.de ([85.178.37.243] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RbFbo-0008Dz-DT>; Thu, 15 Dec 2011 19:00:44 +0100 Message-ID: <4EEA35CB.7030203@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 19:00:43 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Daniel Kalchev References: <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <20111215134853.GA24753@icarus.home.lan> <0C72D682-CF5E-42D6-91F3-FEF1AB02F5D6@digsys.bg> In-Reply-To: <0C72D682-CF5E-42D6-91F3-FEF1AB02F5D6@digsys.bg> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigB97FEA6B9A0C0983989EA20A" X-Originating-IP: 85.178.37.243 Cc: Adrian Chadd , FreeBSD Stable Mailing List , Current FreeBSD , "Samuel J. Greear" , freebsd-performance@freebsd.org, Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 18:00:47 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigB97FEA6B9A0C0983989EA20A Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Am 12/15/11 14:58, schrieb Daniel Kalchev: >=20 > On Dec 15, 2011, at 3:48 PM, Jeremy Chadwick wrote: >=20 > [=85] >> That said: thrown out, data ignored, done. >> >> Now what? Where are we? We're right back where we were a day or two >> ago; meaning no closer to solving the dilemma reported by users and >> SCHED_ULE. Heck, we're not even sure if there is an issue, other than= >> some folks confirming that SCHED_4BSD performs better for them (that's= >> what started this whole thread), and there are at least a couple which= >> have stated this. >=20 > But, are any of these benchmarks really engaging the 4BSD/ULE scheduler= differences? Most such benchmarks are run on a system with no other load= whatsoever and in no way represent real world experience. >=20 > What is more, I believe in such benchmarks "the system feels sluggish" = is not measured at all. Even if it is measured, if in such case the bench= mark finishes "better" - that is, faster, or say, makes the system freeze= for the user for the duration of the test -- it will be considered "win"= , because the benchmark suite ran faster on that particular system -- whe= reas a system which ran the benchmark fast, provided good interactive res= ponse etc would be considered "loser". I guess you have some proofs on that "feeling"? >=20 > I think it is not good idea to hijack this thread, but instead focusing= on the other SCHED_ULE bashing thread to define an reasonable benchmark = or a set of benchmarks rather -- so that many would run it and provide fe= edback. >=20 >=20 > Daniel_______________________________________________ --------------enigB97FEA6B9A0C0983989EA20A Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQEcBAEBAgAGBQJO6jXLAAoJEOgBcD7A/5N8eHMH+gIP8qH2klzQvMwrpp40QhU1 E1Bd4Q13P5RAc69oJJWdBzz4jV9Oz9aJZzpc4uHnFI9FyxBVY9LL3QVuX3cErK7u NmxS6Hl3AkrfAZ2I0O/XGq6LF6Kmcw83LCKWubexRAaIIr4YjZd/AiTd5TlU1nyy Nml9b8yyJlt9aggS22TO6UTnqRxcvqFQhP8hAZnPjYsoN6sDd3TRynAJqNc7LWeW P8jBxo2+gqEnNDl4LYrr+RDM6Gsbr3k2+YYK98miX/DUHBLEBx0liVCpy+lPNWhl XBqGGGPjveUqBEVvUiOixU7aO8rxQDnL3PBSdreL7xeOTvP9bRdZ1lnaxpEq+4M= =YwYJ -----END PGP SIGNATURE----- --------------enigB97FEA6B9A0C0983989EA20A-- From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 18:09:27 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC2B3106566C; Thu, 15 Dec 2011 18:09:27 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 6A2EF8FC16; Thu, 15 Dec 2011 18:09:27 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RbFkC-0003th-3N>; Thu, 15 Dec 2011 19:09:24 +0100 Received: from e178037243.adsl.alicedsl.de ([85.178.37.243] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RbFkB-0000HR-Ub>; Thu, 15 Dec 2011 19:09:24 +0100 Message-ID: <4EEA37D3.7000803@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 19:09:23 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Steven Hartland References: <4EE1EAFE.3070408@m5p.com><4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com><4EE933C6.4020209@zedat.fu-berlin.de><20111215004205.GA11556@icarus.home.lan> <4eea1a4a.nJRbEc1jgKpVnVk4%perryh@pluto.rain.com> <5350EDAC93604B6E93EDBD65A3038F88@multiplay.co.uk> In-Reply-To: <5350EDAC93604B6E93EDBD65A3038F88@multiplay.co.uk> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigCCAE13C8844593D3BFB4F812" X-Originating-IP: 85.178.37.243 Cc: tevans.uk@googlemail.com, perryh@pluto.rain.com, attilio@freebsd.org, george+freebsd@m5p.com, "freebsd-current >> Current FreeBSD" , freebsd-stable@freebsd.org, freebsd@jdc.parodius.com Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 18:09:27 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigCCAE13C8844593D3BFB4F812 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Am 12/15/11 15:20, schrieb Steven Hartland: > With all the discussion I thought I'd give a buildworld > benchmark a go here on a spare 24 core machine. ULE > tested fine but with 4BSD it wont even boot panicing > with the following:- > http://screensnapr.com/v/hwysGV.png >=20 > This is on a clean 8.2-RELEASE-p4 >=20 > Upgrading to RELENG_9 fixed this but its a bit concerning > that just changing the scheduler would cause the machine > to panic on boot. >=20 > Its only a single run so varience could be high but here's > the result of a buildworld on this machine running the > two different schedulers:- > 4BSD: 24m54.10s real 2h43m12.42s user 56m20.07s sys > ULE: 23m54.68s real 2h34m59.04s user 50m59.91s sys >=20 > What really sticks out is that this is over double that > of an 8.2 buildworld on the same machine with the same > kernel > ULE: 11m12.76s real 1h27m59.39s user 28m59.57s sys >=20 > This was run 9.0-PRERELEASE kernel due to 4BSD panicing > on boot under 8.2. >=20 > So for this use ULE vs 4BSD is neither here-nor-there > but 9.0 buildworld is very slow (x2 slower) compared > with 8.2 so whats a bigger question in my mind. >=20 > Regards > Steve >=20 All of our 8.2-STABLE with ncpu >=3D 4 compile the OS in half the time a compilation of FreeBSD 9/10 is needed to. I guess this is due to the huge LLVM contribution which is now part of the source tree. Even if you allow building a whole LLVM suite (and not even pieces of it as in FreeBSD standard for CLANG purposes), it takes another q0 to 20 minutes, depending on the architecture of the underlying host. Building kernel or worl, taking time and show then the invers of that number isn't a good idea, in my opinion. Therefore I like "artificial" benchmarks: have a set of programs that can be compiled and take the time if compilation time is important. Well, your one-shot test would show, that there is indeed a marginal advantage of SCHED_ULE, if the number of cores is big enough (as said to be n > 2 in this thread). But I'm a bit disappointed about the very small advantage on that 24 core hog. Oliver --------------enigCCAE13C8844593D3BFB4F812 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQEcBAEBAgAGBQJO6jfTAAoJEOgBcD7A/5N88mQIAKwvLo61jfJiVhjqNhN81imb HcZAcGb0Apy+ijC/Q9v9HjQbWSoPk3HeENyt5iYXV421WoYXy94qP0vO67yV0H2K 10UbVfIh5XmYfz89WdIjS4HvdmPKSjYnsZ17m93sJY4oWS8iO9nDwi1Z+K9h2LaD 6jN2hEFuGk547Yr3nTbKnXlm04CKp8sG+XIzbDhwDu+uB7FdkHH9NDi2w3Ki2f52 zSozarItiCqb/Ec0DAtGHAUoE25W5Tg+CshqZgdJXzWfFVrIbI+uXNx1AxhN4t91 rJ014tPDG+4UnqTprJ+5EtxO1Hpcqn7SmZ4jZfWE5aCmxkvXaE3koZvkzzpphRo= =r3lZ -----END PGP SIGNATURE----- --------------enigCCAE13C8844593D3BFB4F812-- From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 18:35:48 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6CFC31065673 for ; Thu, 15 Dec 2011 18:35:48 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1DBDF8FC0A for ; Thu, 15 Dec 2011 18:35:47 +0000 (UTC) Received: by vcbfk1 with SMTP id fk1so3041137vcb.13 for ; Thu, 15 Dec 2011 10:35:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=2AlMjn6CxxbBgoVYL7CStHaRl2qG88b4zva2K1TTroc=; b=IozSrMog/mt1j+wgGQuRVp4rgnds8/VYI9At0SDz4wIHpfLqULQUoPLI1ilruuxVce 0r3t81IdSDmasQoqkCRadSxHGWq5SxFR3NHvxsQQXGnqS4wQ5fXJYkcE21kFzvjszSA+ 4Gfle2IoCwqR2TWDlNLapfsqZ9442wwCeb4ww= MIME-Version: 1.0 Received: by 10.220.148.146 with SMTP id p18mr880503vcv.46.1323972381673; Thu, 15 Dec 2011 10:06:21 -0800 (PST) Received: by 10.220.231.10 with HTTP; Thu, 15 Dec 2011 10:06:21 -0800 (PST) In-Reply-To: <4EEA3556.7030105@zedat.fu-berlin.de> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9F546.6060503@freebsd.org> <4EEA3556.7030105@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 10:06:21 -0800 Message-ID: From: Freddie Cash To: "O. Hartmann" Content-Type: text/plain; charset=UTF-8 Cc: Michael Larabel , FreeBSD Stable Mailing List , Current FreeBSD , Daniel Kalchev , Michael Ross , freebsd-performance@freebsd.org, Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 18:35:48 -0000 On Thu, Dec 15, 2011 at 9:58 AM, O. Hartmann wrote: > Am 12/15/11 14:51, schrieb Daniel Kalchev: >> >> On Dec 15, 2011, at 3:25 PM, Stefan Esser wrote: >> >>> Am 15.12.2011 11:10, schrieb Michael Larabel: >>>> No, the same hardware was used for each OS. >>>> >>>> In terms of the software, the stock software stack for each OS was used. >>> >>> Just curious: Why did you choose ZFS on FreeBSD, while UFS2 (with >>> journaling enabled) should be an obvious choice since it is more similar >>> in concept to ext4 and since that is what most FreeBSD users will use >>> with FreeBSD? >> >> >> Or perhaps, since it is "server" Linux distribution, use ZFS on Linux as well. With identical tuning on both Linux and FreeBSD. Having the same FS used by both OS will help make the comparison more sensible for FS I/O. >> >> Daniel_______________________________________________ > > Since ZFS in Linux can only be achieved via FUSE (ad far as I know), it > is legitimate to compare ZFS and ext4. It would be much more competetive > to compare Linux BTRFS and FreeBSD ZFS. There is a separate kernel module for ZFS that can be installed, giving you proper kernel-level support for ZFS on Linux. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 19:02:48 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 21D4E106564A; Thu, 15 Dec 2011 19:02:48 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4B6198FC13; Thu, 15 Dec 2011 19:02:45 +0000 (UTC) Received: by faaf16 with SMTP id f16so3819416faa.13 for ; Thu, 15 Dec 2011 11:02:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=SgZAVHUFCa9dzNQ7v791Vzanwzpvg9L2LKkf09pCqIA=; b=dKua/AxeaedYAKKhiqDkk6XJxGID2B/hN+gJs34qdsLEwXFhEbp8i6LudZbQ24sAiC emCaA+2CgFaBfDoGho7LhILfS1yOSBqRKyrFY441CQIwjKkM2OZlTWueIClf4uVwua1j AO1bSgwuR+b01OV9+sKbXUZ7dMjZuPkDAQzxE= MIME-Version: 1.0 Received: by 10.180.96.103 with SMTP id dr7mr7874751wib.16.1323975764332; Thu, 15 Dec 2011 11:02:44 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Thu, 15 Dec 2011 11:02:44 -0800 (PST) In-Reply-To: <20111215174857.GA28551@icarus.home.lan> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111213073615.GA69641@icarus.home.lan> <20111215174857.GA28551@icarus.home.lan> Date: Thu, 15 Dec 2011 20:02:44 +0100 X-Google-Sender-Auth: CHfHol8uWmtjFqXYPz0qUtT3ARA Message-ID: From: Attilio Rao To: Jeremy Chadwick Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 19:02:48 -0000 2011/12/15 Jeremy Chadwick : > On Thu, Dec 15, 2011 at 05:26:27PM +0100, Attilio Rao wrote: >> 2011/12/13 Jeremy Chadwick : >> > On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: >> >> > Not fully right, boinc defaults to run on idprio 31 so this isn't a= n >> >> > issue. And yes, there are cases where SCHED_ULE shows much better >> >> > performance then SCHED_4BSD. ??[...] >> >> >> >> Do we have any proof at hand for such cases where SCHED_ULE performs >> >> much better than SCHED_4BSD? Whenever the subject comes up, it is >> >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu= > >> >> 2. But in the end I see here contradictionary statements. People >> >> complain about poor performance (especially in scientific environment= s), >> >> and other give contra not being the case. >> >> >> >> Within our department, we developed a highly scalable code for planet= ary >> >> science purposes on imagery. It utilizes present GPUs via OpenCL if >> >> present. Otherwise it grabs as many cores as it can. >> >> By the end of this year I'll get a new desktop box based on Intels ne= w >> >> Sandy Bridge-E architecture with plenty of memory. If the colleague w= ho >> >> developed the code is willing performing some benchmarks on the same >> >> hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most >> >> recent Suse. For FreeBSD I intent also to look for performance with b= oth >> >> different schedulers available. >> > >> > This is in no way shape or form the same kind of benchmark as what >> > you're planning to do, but I thought I'd throw it out there for folks = to >> > take in as they see fit. >> > >> > I know folks were focused mainly on buildworld. >> > >> > I personally would find it interesting if someone with a higher-end >> > system (e.g. 2 physical CPUs, with 6 or 8 cores per CPU) was to do the >> > same test (changing -jX to -j{numofcores} of course). >> > >> > -- >> > | Jeremy Chadwick ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??jdc a= t parodius.com | >> > | Parodius Networking ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? http://www.paro= dius.com/ | >> > | UNIX Systems Administrator ?? ?? ?? ?? ?? ?? ?? ?? ?? Mountain View,= CA, US | >> > | Making life hard for others since 1977. ?? ?? ?? ?? ?? ?? ?? PGP 4BD= 6C0CB | >> > >> > >> > sched_ule >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> > - time make -j2 buildworld >> > ??1689.831u 229.328s 18:46.20 170.4% 6566+2051k 432+4264io 4565pf+0w >> > - time make -j2 buildkernel >> > ??640.542u 87.737s 9:01.38 134.5% 6490+1920k 134+5968io 0pf+0w >> > >> > >> > sched_4bsd >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> > - time make -j2 buildworld >> > ??1662.793u 206.908s 17:12.02 181.1% 6578+2054k 23750+4271io 6451pf+0w >> > - time make -j2 buildkernel >> > ??638.717u 76.146s 8:34.90 138.8% 6530+1927k 6415+5903io 0pf+0w >> > >> > >> > software >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> > * sched_ule test: ??FreeBSD 8.2-STABLE, Thu Dec ??1 04:37:29 PST 2011 >> > * sched_4bsd test: FreeBSD 8.2-STABLE, Mon Dec 12 22:42:54 PST 2011 >> >> Hi Jeremy, >> thanks for the time you spent on this. >> >> However, I wanted to ask/let you note 3 things: >> 1) Did you use 2 different code base for the test? (one updated on >> December 1 and another one on December 12) > > No; src-all (/usr/src on this system) was not updated between December > 1st and December 12th PST. =C2=A0I do believe I updated it today (15th PS= T). > I can/will obviously hold off so that we have a consistent code base for > comparing numbers between schedulers during buildworld and/or > buildkernel. > >> 2) Please note that you should have repeated this test several times >> (basically until you don't get a standard deviation which is >> acceptable with ministat) and report the ministat output > > This is the first time I have heard of ministat(1). =C2=A0I'm pretty sure= I > see what it's for and how it applies to this situation, but boy that man > page could use some clarification (I have 3 people looking at this thing > right now trying to figure out what means what in the graph :-) ). > Anyway, graph or not, I see the point. > > Regarding multiple tests: yup, you're absolutely right, the only way to > do it would be to run a sequence of tests repeatedly (probably 10 per > scheduler). =C2=A0Reboots and rm -fr /usr/obj/* would be required after e= ach > test too, to guarantee empty kernel caches (of all types) consistently > every time. > > What I posted was supposed to give people just a "general idea" if there > was any gigantic difference between the two, and there really isn't. > But, as others have stated (and you below), buildworld may not be an > effective way to "benchmark" what we're trying to test. > > Hence me wondering exactly what would make for a good test. =C2=A0Example= : > > 1. Run + background some program that "beats on things" (I really don't > know what; creation/deletion of threads? =C2=A0CPU benchmark? =C2=A0bonni= e++?), > with output going to /dev/null. > 2. Run + background "time make -j2 buildworld" with output going to /dev/= null > 3. Record/save output from "time". > 4. rm -fr /usr/obj && shutdown -r now > 5. Repeat all steps ~10 times > 6. Adjust kernel configuration file to use other scheduler > 7. Repeat steps 1-5. > > What I'm trying to figure out is what #1 and #2 should be in the above > example. > >> 3) The difference is less than 2% which I suspect is really >> statistically unuseful/the same > > Understood. > >> I'm not really even surprised ULE is not faster than 4BSD in this case >> because usually buildworld/buildkernel tests are driven for the vast >> majority by I/O overhead rather than scheduler capacity. It would be >> more interesting to analyze how buildworld does while another type of >> workload is going on. > > Yup, agreed/understood, hence me trying to find out what would classify > as a good stress test for all of this. > > I have a testbed system in my garage which I could set up to literally > do all of this in a loop, meaning automate the entire above process and > just let it go, writing stderr from time to a file (which wouldn't skew > the results at all). > > Let me know what #1 and #2 above, re: "the workloads", should be and > I'll be happy to set it up. My idea, in order to gather meaningful datas for both ULE and 4BSD would be to see how well they behave in the futher situation: - 2 concurrent interactive workloads - 2 concurrent cpu-intensive workloads - mixed and having the number of threads for both varying as: N/2, N, N + small_amount (1 or 2 or 3, etc), N*2 (where N is the number of available CPUs) which automatically translates into: - 2 concurrent interactive and intensive (A and B workloads): * A N/2 threads, B N/2 threads * A N threads, B N/2 threads * A N + small_amount, B N/2 threads * A N*2 threads, B N/2 threads * A N threads, B N threads * A N + small_amount, B N threads * A N*2 threads, B N threads * A N + small_amount, B N + small_amount threads * A N*2 threads, B N + small_amount threads * A N*2 threads, B N*2 threads For the mixed case, instead, we should try all the 16 combinations possibly and it is likely the most interesting case, to be honest. About the workload, we could use: interactives: buildworld and bonnie++ (I'm not totally sure if bonnie++ let you decides how many threads to run, but I'm sure we can replace with something that really does that) cpu-intensive: dnetc and SOMETHINGELSE (please propose something that can be setup very easilly!) mixed case: buildworld and dnetc About the environment I'd suggest the following things: - Try to boot with a maximum of 16 CPUs. I'm sure past that point TLB shootdown overhead is going to be too overwhelming, make doesn't really scale well, and also there could be too much contention on vm_page_lock_queue for interactive threads. - Try to reduce the I/O effect by using tmpfs as a storage for in and out datas when working out the benchmark - Use 10.0 with both kerneland and userland totally debug-free (please recall to set MALLOC_PRODUCTION in jemalloc) and always at the same svn revision, with the only change being the scheduler switch and the number of threads changing during the runs About the test itself I'd suggest the following things: - After every test combination, please reboot the machine (like, after you have tested the A N/2 threads and B N/2 threads case on sched_4bsd, reboot the machine before to do A N threads and B N/2 threads) - For every test combination I suggest to run the workloads 4 times, discard the first one (but keep the value!) and ministat the other three. Showing the "uncached" case against the average cached one will give much more indication than expected. - Expect a standard deviation from ministat to be 95% (or beyond) to be val= uable - For every difference in performance we find we should likely start worry about if it is as or bigger than 3% and being very concerned from 5% to above I think we already have some datas of ULE being broken in some cases (like George's and Steven's case) but we really need to characterize more, I think. Now, I understand this seems a gigantic work but I think there is much people which is interested in working on this and we may scatter these tests around, to different testers, to find meaningful datas. If it was me, I would start with comparisons involving all the N and N + small_amount cases which should be the most interesting. Do you have questions? Thanks, Attilio --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 19:12:22 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4A8E810657C0; Thu, 15 Dec 2011 19:12:22 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id F24A48FC1A; Thu, 15 Dec 2011 19:12:21 +0000 (UTC) Received: by iakl21 with SMTP id l21so5868523iak.13 for ; Thu, 15 Dec 2011 11:12:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type; bh=7d6VqVruejPBYew39OdFfyLubY4RE3DWCquvTgnBdc0=; b=iCLiQ2NOnZCCmLJ6004uiCusi7ys59g2DGY/cAbS2RFqPYjJiR28xGmUsoI0I362XX YhvIuqQlNZmpnXBWTDtSuWtPawczV5JIxKSmGfN2yJrRdmj+UoXdPrdbUiCQgGanrs/q FBgG7n6xUenL6Tt6v3QcZOk38/sVLiGO/HSlU= Received: by 10.50.173.74 with SMTP id bi10mr3986530igc.4.1323974814418; Thu, 15 Dec 2011 10:46:54 -0800 (PST) MIME-Version: 1.0 Sender: utisoft@gmail.com Received: by 10.231.199.18 with HTTP; Thu, 15 Dec 2011 10:46:22 -0800 (PST) In-Reply-To: <4EEA3556.7030105@zedat.fu-berlin.de> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9F546.6060503@freebsd.org> <4EEA3556.7030105@zedat.fu-berlin.de> From: Chris Rees Date: Thu, 15 Dec 2011 18:46:22 +0000 X-Google-Sender-Auth: un-4pXRVXneCogocUAc0frKt3Bo Message-ID: To: "O. Hartmann" Content-Type: text/plain; charset=ISO-8859-1 Cc: Michael Larabel , FreeBSD Stable Mailing List , Current FreeBSD , Daniel Kalchev , Michael Ross , freebsd-performance@freebsd.org, Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 19:12:22 -0000 On 15 December 2011 17:58, O. Hartmann wrote: > Since ZFS in Linux can only be achieved via FUSE (ad far as I know), it > is legitimate to compare ZFS and ext4. It would be much more competetive > to compare Linux BTRFS and FreeBSD ZFS. > Er... does ext4 guarantee data integrity? You're not comparing like with like; please do some research on the point of ZFS before asserting that they're fair comparisons. A fair(er) comparison could be ext4 with UFS+soft-updates. Chris From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 19:12:31 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E6B241065893 for ; Thu, 15 Dec 2011 19:12:31 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id BEE0E8FC0A for ; Thu, 15 Dec 2011 19:12:31 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id 4919246B51; Thu, 15 Dec 2011 14:12:31 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id CDEFBB942; Thu, 15 Dec 2011 14:12:30 -0500 (EST) From: John Baldwin To: freebsd-stable@freebsd.org Date: Thu, 15 Dec 2011 13:51:28 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <201112090913.CAA03333@lariat.net> <4EE223CD.2020709@my.gd> In-Reply-To: <4EE223CD.2020709@my.gd> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201112151351.28647.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 15 Dec 2011 14:12:30 -0500 (EST) Cc: Subject: Re: Two problems still present in RC3 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 19:12:32 -0000 On Friday, December 09, 2011 10:05:49 am Damien Fleuriot wrote: > > On 12/9/11 10:13 AM, Brett Glass wrote: > > FreeBSD 9.0-RC3 is looking good, but I'm still encountering two problems. > > > > Firstly, when I try to configure VLANs in /etc/rc.conf, I'm getting > > errors. For example, if I use > > > > vlans_re0="1 2" > > ip_addrs_re0_1="192.168.0.1-4/16" > > ip_addrs_re0_2="10.0.0.0/24" > > > > to create two VLANs on the interface re0, I get error messages saying > > that "create" commands (presumably using ifconfig) have failed. The > > interfaces SEEM to be configured correctly, but the messages -- which > > must be coming from scripts called by /etc/netstart -- are troubling. > > > > Secondly, there's still some strangeness in the sc terminal emulation. > > When I run jove, the status line at the bottom of the screen isn't > > entirely in reverse video as it should be. Only parts of it are, and the > > highlighting changes -- seemingly at random -- as I work. > > > > Neither of these is likely to be a showstopper (so long as the first > > won't cause me networking problems I haven't observed yet), but both are > > probably worth looking into. > > > > > I have never seen this way of configuring VLANs. It's new. I added it in the last year or so to match the way wlan devices are created. It also can do sane things like auto-destroy associated vlans when a device is stopped, etc. However, I have not tested them with /etc/netstart. They do work fine during a normal startup. I can't see obvious reasons as to why they would fail though via /etc/netstart. The stuff that handles vlans_ will auto-load if_vlan.ko if it is needed, but perhaps that isn't working from single user mode when using /etc/netstart? Also, do the vlans work fine if you let a box boot normally? -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 19:46:36 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DB89C106564A; Thu, 15 Dec 2011 19:46:36 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm2.ukr.net (fsm2.ukr.net [195.214.192.121]) by mx1.freebsd.org (Postfix) with ESMTP id 6820F8FC0A; Thu, 15 Dec 2011 19:46:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=EK+mEIr5wXLNRK+o+gkQwhNeZz7gH4y//yqgkqZMT9E=; b=ClJh5RpI1ShemCy0/yCT1WdqtbQbx66RQR01kStLiG7jjpjg0o65EgTq+l9kHG5f9+Lqoj3t6KYv0fWIB2MseglX6MT+mqfCfOH76g3f/DV83LKXtbLlAFtKDILOT7cLpZNA/x4v5v3vlcozZxllSG5ElG/AKj7dHzTvN4DJ7SM=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm2.ukr.net with esmtpsa ID 1RbHGD-000ME3-L6 ; Thu, 15 Dec 2011 21:46:33 +0200 Date: Thu, 15 Dec 2011 21:46:27 +0200 From: Ivan Klymenko To: Attilio Rao Message-ID: <20111215214627.16f472bf@nonamehost.> In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111213073615.GA69641@icarus.home.lan> <20111215174857.GA28551@icarus.home.lan> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Jeremy Chadwick Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 19:46:37 -0000 =D0=92 Thu, 15 Dec 2011 20:02:44 +0100 Attilio Rao =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > 2011/12/15 Jeremy Chadwick : > > On Thu, Dec 15, 2011 at 05:26:27PM +0100, Attilio Rao wrote: > >> 2011/12/13 Jeremy Chadwick : > >> > On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: > >> >> > Not fully right, boinc defaults to run on idprio 31 so this > >> >> > isn't an issue. And yes, there are cases where SCHED_ULE > >> >> > shows much better performance then SCHED_4BSD. ??[...] > >> >> > >> >> Do we have any proof at hand for such cases where SCHED_ULE > >> >> performs much better than SCHED_4BSD? Whenever the subject > >> >> comes up, it is mentioned, that SCHED_ULE has better > >> >> performance on boxes with a ncpu > 2. But in the end I see here > >> >> contradictionary statements. People complain about poor > >> >> performance (especially in scientific environments), and other > >> >> give contra not being the case. > >> >> > >> >> Within our department, we developed a highly scalable code for > >> >> planetary science purposes on imagery. It utilizes present GPUs > >> >> via OpenCL if present. Otherwise it grabs as many cores as it > >> >> can. By the end of this year I'll get a new desktop box based > >> >> on Intels new Sandy Bridge-E architecture with plenty of > >> >> memory. If the colleague who developed the code is willing > >> >> performing some benchmarks on the same hardware platform, we'll > >> >> benchmark bot FreeBSD 9.0/10.0 and the most recent Suse. For > >> >> FreeBSD I intent also to look for performance with both > >> >> different schedulers available. > >> > > >> > This is in no way shape or form the same kind of benchmark as > >> > what you're planning to do, but I thought I'd throw it out there > >> > for folks to take in as they see fit. > >> > > >> > I know folks were focused mainly on buildworld. > >> > > >> > I personally would find it interesting if someone with a > >> > higher-end system (e.g. 2 physical CPUs, with 6 or 8 cores per > >> > CPU) was to do the same test (changing -jX to -j{numofcores} of > >> > course). > >> > > >> > -- > >> > | Jeremy > >> > Chadwick ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??jdc at > >> > parodius.com | | Parodius > >> > Networking ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? > >> > http://www.parodius.com/ | | UNIX Systems > >> > Administrator ?? ?? ?? ?? ?? ?? ?? ?? ?? Mountain View, CA, US | > >> > | Making life hard for others since 1977. ?? ?? ?? ?? ?? ?? ?? > >> > PGP 4BD6C0CB | > >> > > >> > > >> > sched_ule > >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >> > - time make -j2 buildworld > >> > ??1689.831u 229.328s 18:46.20 170.4% 6566+2051k 432+4264io > >> > 4565pf+0w > >> > - time make -j2 buildkernel > >> > ??640.542u 87.737s 9:01.38 134.5% 6490+1920k 134+5968io 0pf+0w > >> > > >> > > >> > sched_4bsd > >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >> > - time make -j2 buildworld > >> > ??1662.793u 206.908s 17:12.02 181.1% 6578+2054k 23750+4271io > >> > 6451pf+0w > >> > - time make -j2 buildkernel > >> > ??638.717u 76.146s 8:34.90 138.8% 6530+1927k 6415+5903io 0pf+0w > >> > > >> > > >> > software > >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >> > * sched_ule test: ??FreeBSD 8.2-STABLE, Thu Dec ??1 04:37:29 PST > >> > 2011 > >> > * sched_4bsd test: FreeBSD 8.2-STABLE, Mon Dec 12 22:42:54 PST > >> > 2011 > >> > >> Hi Jeremy, > >> thanks for the time you spent on this. > >> > >> However, I wanted to ask/let you note 3 things: > >> 1) Did you use 2 different code base for the test? (one updated on > >> December 1 and another one on December 12) > > > > No; src-all (/usr/src on this system) was not updated between > > December 1st and December 12th PST. =C2=A0I do believe I updated it > > today (15th PST). I can/will obviously hold off so that we have a > > consistent code base for comparing numbers between schedulers > > during buildworld and/or buildkernel. > > > >> 2) Please note that you should have repeated this test several > >> times (basically until you don't get a standard deviation which is > >> acceptable with ministat) and report the ministat output > > > > This is the first time I have heard of ministat(1). =C2=A0I'm pretty > > sure I see what it's for and how it applies to this situation, but > > boy that man page could use some clarification (I have 3 people > > looking at this thing right now trying to figure out what means > > what in the graph :-) ). Anyway, graph or not, I see the point. > > > > Regarding multiple tests: yup, you're absolutely right, the only > > way to do it would be to run a sequence of tests repeatedly > > (probably 10 per scheduler). =C2=A0Reboots and rm -fr /usr/obj/* would > > be required after each test too, to guarantee empty kernel caches > > (of all types) consistently every time. > > > > What I posted was supposed to give people just a "general idea" if > > there was any gigantic difference between the two, and there really > > isn't. But, as others have stated (and you below), buildworld may > > not be an effective way to "benchmark" what we're trying to test. > > > > Hence me wondering exactly what would make for a good test. > > =C2=A0Example: > > > > 1. Run + background some program that "beats on things" (I really > > don't know what; creation/deletion of threads? =C2=A0CPU benchmark? > > =C2=A0bonnie++?), with output going to /dev/null. > > 2. Run + background "time make -j2 buildworld" with output going > > to /dev/null 3. Record/save output from "time". > > 4. rm -fr /usr/obj && shutdown -r now > > 5. Repeat all steps ~10 times > > 6. Adjust kernel configuration file to use other scheduler > > 7. Repeat steps 1-5. > > > > What I'm trying to figure out is what #1 and #2 should be in the > > above example. > > > >> 3) The difference is less than 2% which I suspect is really > >> statistically unuseful/the same > > > > Understood. > > > >> I'm not really even surprised ULE is not faster than 4BSD in this > >> case because usually buildworld/buildkernel tests are driven for > >> the vast majority by I/O overhead rather than scheduler capacity. > >> It would be more interesting to analyze how buildworld does while > >> another type of workload is going on. > > > > Yup, agreed/understood, hence me trying to find out what would > > classify as a good stress test for all of this. > > > > I have a testbed system in my garage which I could set up to > > literally do all of this in a loop, meaning automate the entire > > above process and just let it go, writing stderr from time to a > > file (which wouldn't skew the results at all). > > > > Let me know what #1 and #2 above, re: "the workloads", should be and > > I'll be happy to set it up. >=20 > My idea, in order to gather meaningful datas for both ULE and 4BSD > would be to see how well they behave in the futher situation: > - 2 concurrent interactive workloads > - 2 concurrent cpu-intensive workloads > - mixed >=20 > and having the number of threads for both varying as: N/2, N, N + > small_amount (1 or 2 or 3, etc), N*2 (where N is the number of > available CPUs) which automatically translates into: >=20 > - 2 concurrent interactive and intensive (A and B workloads): > * A N/2 threads, B N/2 threads > * A N threads, B N/2 threads > * A N + small_amount, B N/2 threads > * A N*2 threads, B N/2 threads > * A N threads, B N threads > * A N + small_amount, B N threads > * A N*2 threads, B N threads > * A N + small_amount, B N + small_amount threads > * A N*2 threads, B N + small_amount threads > * A N*2 threads, B N*2 threads >=20 > For the mixed case, instead, we should try all the 16 combinations > possibly and it is likely the most interesting case, to be honest. >=20 > About the workload, we could use: > interactives: buildworld and bonnie++ (I'm not totally sure if > bonnie++ let you decides how many threads to run, but I'm sure we can > replace with something that really does that) > cpu-intensive: dnetc and SOMETHINGELSE (please propose something that > can be setup very easilly!) > mixed case: buildworld and dnetc >=20 > About the environment I'd suggest the following things: > - Try to boot with a maximum of 16 CPUs. I'm sure past that point TLB > shootdown overhead is going to be too overwhelming, make doesn't > really scale well, and also there could be too much contention on > vm_page_lock_queue for interactive threads. > - Try to reduce the I/O effect by using tmpfs as a storage for in and > out datas when working out the benchmark > - Use 10.0 with both kerneland and userland totally debug-free (please > recall to set MALLOC_PRODUCTION in jemalloc) and always at the same > svn revision, with the only change being the scheduler switch and the > number of threads changing during the runs >=20 > About the test itself I'd suggest the following things: > - After every test combination, please reboot the machine (like, after > you have tested the A N/2 threads and B N/2 threads case on > sched_4bsd, reboot the machine before to do A N threads and B N/2 > threads) > - For every test combination I suggest to run the workloads 4 times, > discard the first one (but keep the value!) and ministat the other > three. Showing the "uncached" case against the average cached one will > give much more indication than expected. > - Expect a standard deviation from ministat to be 95% (or beyond) to > be valuable > - For every difference in performance we find we should likely start > worry about if it is as or bigger than 3% and being very concerned > from 5% to above >=20 > I think we already have some datas of ULE being broken in some cases > (like George's and Steven's case) but we really need to characterize > more, I think. >=20 > Now, I understand this seems a gigantic work but I think there is much > people which is interested in working on this and we may scatter these > tests around, to different testers, to find meaningful datas. >=20 > If it was me, I would start with comparisons involving all the N and N > + small_amount cases which should be the most interesting. >=20 > Do you have questions? >=20 > Thanks, > Attilio >=20 >=20 Perhaps it makes sense to co-write a script to automate these actions? And place it in /usr/src/tools/sched/... From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 20:51:29 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id DD05A106564A for ; Thu, 15 Dec 2011 20:51:29 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id CE0B514DCFE; Thu, 15 Dec 2011 20:51:28 +0000 (UTC) Message-ID: <4EEA5DD0.1040009@FreeBSD.org> Date: Thu, 15 Dec 2011 12:51:28 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@FreeBSD.org X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: swi4: clock taking 40% cpu?!? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 20:51:29 -0000 Howdy, Web server under heavy'ish load (7 on a 2 cpu system) running 8.2-RELEASE-p4 i386 I'm seeing this: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root -32 - 0K 112K WAIT 0 129:01 39.99% {swi4: clock} Any ideas why the clock should be taking so much cpu? HZ=100 if that makes a difference ... Doug -- [^L] Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 20:58:07 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DBB52106564A; Thu, 15 Dec 2011 20:58:07 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id 90BBC8FC12; Thu, 15 Dec 2011 20:58:07 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.5/8.14.4) with ESMTP id pBFKw2ea013553; Thu, 15 Dec 2011 15:58:03 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4EEA5F5C.8080503@sentex.net> Date: Thu, 15 Dec 2011 15:58:04 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Attilio Rao References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> <4EE8D607.1000504@sentex.net> <4EEA227E.7080704@sentex.net> <4EEA25BB.7040706@sentex.net> In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.71 on IPv6:2607:f3e0:0:1::12 Cc: Ivan Klymenko , mdf@freebsd.org, Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 20:58:08 -0000 On 12/15/2011 11:56 AM, Attilio Rao wrote: > So, as very first thing, can you try the following: > - Same codebase, etc. etc. > - Make the test 4 times, discard the first and ministat for the other 3 > - Reboot > - Change the steal_thresh value > - Make the test 4 times, discard the first and ministat for the other 3 > > Then report discarded values and the ministated one and we will have > more informations I guess > (also, I don't think devfs contention should play a role here, thus > nevermind about it for now). Results and data at http://www.tancsa.com/ule-bsd.html ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 21:04:23 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42FF91065675 for ; Thu, 15 Dec 2011 21:04:23 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA11.westchester.pa.mail.comcast.net (qmta11.westchester.pa.mail.comcast.net [76.96.59.211]) by mx1.freebsd.org (Postfix) with ESMTP id 060E88FC08 for ; Thu, 15 Dec 2011 21:04:22 +0000 (UTC) Received: from omta10.westchester.pa.mail.comcast.net ([76.96.62.28]) by QMTA11.westchester.pa.mail.comcast.net with comcast id 9YtR1i0040cZkys5BZ4P1r; Thu, 15 Dec 2011 21:04:23 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta10.westchester.pa.mail.comcast.net with comcast id 9Z4N1i00e1t3BNj3WZ4NLD; Thu, 15 Dec 2011 21:04:23 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 38D30102C19; Thu, 15 Dec 2011 13:04:21 -0800 (PST) Date: Thu, 15 Dec 2011 13:04:21 -0800 From: Jeremy Chadwick To: Doug Barton Message-ID: <20111215210421.GA33083@icarus.home.lan> References: <4EEA5DD0.1040009@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EEA5DD0.1040009@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@FreeBSD.org Subject: Re: swi4: clock taking 40% cpu?!? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 21:04:23 -0000 On Thu, Dec 15, 2011 at 12:51:28PM -0800, Doug Barton wrote: > Howdy, > > Web server under heavy'ish load (7 on a 2 cpu system) running > 8.2-RELEASE-p4 i386 I'm seeing this: > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 12 root -32 - 0K 112K WAIT 0 129:01 39.99% {swi4: clock} > > Any ideas why the clock should be taking so much cpu? HZ=100 if that > makes a difference ... Could be wrong, but I believe this correlates with IRQ 4. What does vmstat -i show for a total and rate for irq4 if you run it, wait a few seconds, then run it again? Does the number greatly/rapidly increase? Shot in the dark here, but the only thing I can think of that might cause this is software being extremely aggressive with calls to things like gettimeofday(2) or clock_gettime(2). Really not sure. ntpd maybe (unlikely but possible)? Sort of grasping at straws here. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 21:30:11 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6C4421065675; Thu, 15 Dec 2011 21:30:11 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 4C5D98FC12; Thu, 15 Dec 2011 21:30:09 +0000 (UTC) Received: by werb13 with SMTP id b13so308398wer.13 for ; Thu, 15 Dec 2011 13:30:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=n7KfzfVyFNQXWNxFW6vjVaEbwJSYP2hpwEXCEg172hY=; b=JeYX+K+bT1XoVpq9v5x0AF2oW1Z7WZgDunB3J3Ss1LT9ZreedL1C7sd//L+D/rb/Ny WaQXzTbYVuAwiXTS6rxgJlwlkfB4EtRzyjEJnLIYxLiD0npcroxDSutPkDzVDVCuPz5X VVriBCK/hiC2kvAcyATssUPls8PpRMJMCenyI= MIME-Version: 1.0 Received: by 10.216.131.152 with SMTP id m24mr2521388wei.56.1323984609072; Thu, 15 Dec 2011 13:30:09 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Thu, 15 Dec 2011 13:30:08 -0800 (PST) In-Reply-To: <4EEA5F5C.8080503@sentex.net> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> <4EE8D607.1000504@sentex.net> <4EEA227E.7080704@sentex.net> <4EEA25BB.7040706@sentex.net> <4EEA5F5C.8080503@sentex.net> Date: Thu, 15 Dec 2011 22:30:08 +0100 X-Google-Sender-Auth: -u6X1A6ozE_YndJdHRmklZbhh6k Message-ID: From: Attilio Rao To: Mike Tancsa Content-Type: text/plain; charset=UTF-8 Cc: Ivan Klymenko , mdf@freebsd.org, Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 21:30:11 -0000 2011/12/15 Mike Tancsa : > On 12/15/2011 11:56 AM, Attilio Rao wrote: >> So, as very first thing, can you try the following: >> - Same codebase, etc. etc. >> - Make the test 4 times, discard the first and ministat for the other 3 >> - Reboot >> - Change the steal_thresh value >> - Make the test 4 times, discard the first and ministat for the other 3 >> >> Then report discarded values and the ministated one and we will have >> more informations I guess >> (also, I don't think devfs contention should play a role here, thus >> nevermind about it for now). > > > Results and data at > > http://www.tancsa.com/ule-bsd.html I'm not totally sure, what does burnP6 do? is it a CPU-bound workload? Also, how many threads are spanked in your case for parallel bzip2? Also, it would be very good if you could arrange these tests against newer -CURRENT (with userland and kerneland debugging off). Thanks a lot of your hard work, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 21:35:43 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 97CC7106566B; Thu, 15 Dec 2011 21:35:43 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from email2.allantgroup.com (email2.emsphone.com [199.67.51.116]) by mx1.freebsd.org (Postfix) with ESMTP id 4311C8FC12; Thu, 15 Dec 2011 21:35:42 +0000 (UTC) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by email2.allantgroup.com (8.14.4/8.14.4) with ESMTP id pBFLZenw043742 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 Dec 2011 15:35:40 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.5/8.14.5) with ESMTP id pBFLZeLW064284 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 Dec 2011 15:35:40 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.5/8.14.5/Submit) id pBFLZd3w064283; Thu, 15 Dec 2011 15:35:39 -0600 (CST) (envelope-from dan) Date: Thu, 15 Dec 2011 15:35:39 -0600 From: Dan Nelson To: Jeremy Chadwick Message-ID: <20111215213539.GJ53453@dan.emsphone.com> References: <4EEA5DD0.1040009@FreeBSD.org> <20111215210421.GA33083@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111215210421.GA33083@icarus.home.lan> X-OS: FreeBSD 8.2-STABLE User-Agent: Mutt/1.5.21 (2010-09-15) X-Virus-Scanned: clamav-milter 0.97.2 at email2.allantgroup.com X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.6 (email2.allantgroup.com [199.67.51.78]); Thu, 15 Dec 2011 15:35:40 -0600 (CST) X-Scanned-By: MIMEDefang 2.68 on 199.67.51.78 Cc: Doug Barton , freebsd-stable@freebsd.org Subject: Re: swi4: clock taking 40% cpu?!? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 21:35:43 -0000 In the last episode (Dec 15), Jeremy Chadwick said: > On Thu, Dec 15, 2011 at 12:51:28PM -0800, Doug Barton wrote: > > Web server under heavy'ish load (7 on a 2 cpu system) running > > 8.2-RELEASE-p4 i386 I'm seeing this: > > > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > > 12 root -32 - 0K 112K WAIT 0 129:01 39.99% {swi4: clock} > > > > Any ideas why the clock should be taking so much cpu? HZ=100 if that > > makes a difference ... > > Could be wrong, but I believe this correlates with IRQ 4. What does > vmstat -i show for a total and rate for irq4 if you run it, wait a few > seconds, then run it again? Does the number greatly/rapidly increase? That would be "irq4" in that case, though. "swi4" is just a software interrupt thread, and "clock" is the softclock callout handler. There are both KTR and DTrace logging functions in kern_timeout.c, so you could use either one to get a handle on what's eating your CPU. Busy-looping "procstat -k 12" for a few seconds might get you some useful stacks, as well. > Shot in the dark here, but the only thing I can think of that might > cause this is software being extremely aggressive with calls to things > like gettimeofday(2) or clock_gettime(2). Really not sure. ntpd maybe > (unlikely but possible)? Sort of grasping at straws here. -- Dan Nelson dnelson@allantgroup.com From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 21:45:43 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A93961065701 for ; Thu, 15 Dec 2011 21:45:43 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta14.emeryville.ca.mail.comcast.net (qmta14.emeryville.ca.mail.comcast.net [76.96.27.212]) by mx1.freebsd.org (Postfix) with ESMTP id 8F1758FC1A for ; Thu, 15 Dec 2011 21:45:41 +0000 (UTC) Received: from omta17.emeryville.ca.mail.comcast.net ([76.96.30.73]) by qmta14.emeryville.ca.mail.comcast.net with comcast id 9WrN1i0041afHeLAEZlaFh; Thu, 15 Dec 2011 21:45:34 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta17.emeryville.ca.mail.comcast.net with comcast id 9ZCw1i00q1t3BNj8dZCxCE; Thu, 15 Dec 2011 21:12:57 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id D39DD102C19; Thu, 15 Dec 2011 13:45:39 -0800 (PST) Date: Thu, 15 Dec 2011 13:45:39 -0800 From: Jeremy Chadwick To: Dan Nelson Message-ID: <20111215214539.GA33888@icarus.home.lan> References: <4EEA5DD0.1040009@FreeBSD.org> <20111215210421.GA33083@icarus.home.lan> <20111215213539.GJ53453@dan.emsphone.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111215213539.GJ53453@dan.emsphone.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Doug Barton , freebsd-stable@freebsd.org Subject: Re: swi4: clock taking 40% cpu?!? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 21:45:43 -0000 On Thu, Dec 15, 2011 at 03:35:39PM -0600, Dan Nelson wrote: > In the last episode (Dec 15), Jeremy Chadwick said: > > On Thu, Dec 15, 2011 at 12:51:28PM -0800, Doug Barton wrote: > > > Web server under heavy'ish load (7 on a 2 cpu system) running > > > 8.2-RELEASE-p4 i386 I'm seeing this: > > > > > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > > > 12 root -32 - 0K 112K WAIT 0 129:01 39.99% {swi4: clock} > > > > > > Any ideas why the clock should be taking so much cpu? HZ=100 if that > > > makes a difference ... > > > > Could be wrong, but I believe this correlates with IRQ 4. What does > > vmstat -i show for a total and rate for irq4 if you run it, wait a few > > seconds, then run it again? Does the number greatly/rapidly increase? > > That would be "irq4" in that case, though. "swi4" is just a software > interrupt thread, and "clock" is the softclock callout handler. Yep, over my head (on the latter part). The correlation I made between swi4 and IRQ 4 was based on one of our systems. Note "swi4: clock sio", which to me meant "sioX is associated with swi4", so I looked up what sio0 was associated with and it was irq4. I guess it's just total chance that swi4 and IRQ 4 happen to both have the number 4 in them; sorry for the confusion/mistake and thanks for educating me. top: 13 root 1 -32 - 0K 8K WAIT 0 91:03 0.00% [swi4: clock sio] vmstat -i: interrupt total rate irq4: sio0 715 0 > There are both KTR and DTrace logging functions in kern_timeout.c, so you > could use either one to get a handle on what's eating your CPU. I highly doubt Doug's systems have DTrace built in to them, given that on the version of FreeBSD he's using, building it requires manually typing in "make WITH_CTF=1 buildkernel" (you cannot add this variable to make.conf or src.conf, other stuff will break -- this works on 9.x because of "a horrible kludge hack to make work" (not an exact quote but fairly close)). No admin is going to remember to type that in manually every time, and only during the buildkernel phase (not buildworld, not not installworld, and not installkernel). Grumble. :-) Also FWIW: I had no idea about KTR (what it was, its existence, etc.) until right this moment. I figured all this time it was shorthand for "ktrace". Sheesh, all this other neat debugging stuff! > Busy-looping "procstat -k 12" for a few seconds might get you some useful stacks, as > well. I guess that's why I'm surprised there isn't a process or series of processes on the machine which have, say, slightly high CPU (maybe 1% per process/child) and when ktrace'd show they're calling things like gettimeofday() a *lot*. I dealt with this exact situation at work not too long ago, where a developer decided to stick that call in a while(1) loop with conditionals that never slept/waited (but did only a little bit else) then wondered "what the problem was". Be nice to your systems! ;-) -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 21:48:03 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 189551065673; Thu, 15 Dec 2011 21:48:03 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 6763B8FC22; Thu, 15 Dec 2011 21:48:02 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so4721837wgb.31 for ; Thu, 15 Dec 2011 13:48:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=01rUC02AA1IBImtEMcYSDqr1FhgQzKU+JGgR5l659IE=; b=mPbuaR7qYL9qQqflmhEG/FOaQEfmuj02b98bHd+SanqMl7dERFG5KaTYVMQkAfsKKC e9OvIZ9LOWmPEN83dZkwSRZvOCAyuriuV1wOtiGP0CSMe4UoPSY9hhbEcSBUuUaCQu5O 0RbaaZ6XLpaZFEyiHTZGLh6JB77EHRvg/zljI= MIME-Version: 1.0 Received: by 10.180.95.136 with SMTP id dk8mr3940463wib.11.1323984306922; Thu, 15 Dec 2011 13:25:06 -0800 (PST) Received: by 10.223.158.129 with HTTP; Thu, 15 Dec 2011 13:25:06 -0800 (PST) In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9F546.6060503@freebsd.org> <4EEA3556.7030105@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 13:25:06 -0800 Message-ID: From: Kevin Oberman To: Chris Rees Content-Type: text/plain; charset=ISO-8859-1 Cc: Michael Larabel , FreeBSD Stable Mailing List , Current FreeBSD , Daniel Kalchev , Michael Ross , freebsd-performance@freebsd.org, "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 21:48:03 -0000 On Thu, Dec 15, 2011 at 10:46 AM, Chris Rees wrote: > On 15 December 2011 17:58, O. Hartmann wrote: >> Since ZFS in Linux can only be achieved via FUSE (ad far as I know), it >> is legitimate to compare ZFS and ext4. It would be much more competetive >> to compare Linux BTRFS and FreeBSD ZFS. >> > > > Er... does ext4 guarantee data integrity? > > You're not comparing like with like; please do some research on the > point of ZFS before asserting that they're fair comparisons. > > A fair(er) comparison could be ext4 with UFS+soft-updates. Wouldn't UFS+SUJ be the closest atch? -- R. Kevin Oberman, Network Engineer E-mail: kob6558@gmail.com From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 21:50:19 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E41021065676; Thu, 15 Dec 2011 21:50:18 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 83D078FC0A; Thu, 15 Dec 2011 21:50:18 +0000 (UTC) Received: by iakl21 with SMTP id l21so6168701iak.13 for ; Thu, 15 Dec 2011 13:50:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=BWWRcz+Kc66W7cNXwfHMy0GIhek/G+cxthNvX7X5F+g=; b=rhPZgMsEUlXFQ3KD7ENAkEkmSvbqwVupBDcz60MmKUAK7HfwVHuGsAE5barqs2IceG rezYeu0VWwSIyHP3mUI+WqGgRwMd2E2rPx133SwN7+TkjzA+VZqG2u8uOqDChM1HcdeR jXUMnh1gNeXuMSq17yDUS5mNB2guec456xqwA= MIME-Version: 1.0 Received: by 10.50.87.167 with SMTP id az7mr4458428igb.64.1323985817944; Thu, 15 Dec 2011 13:50:17 -0800 (PST) Sender: utisoft@gmail.com Received: by 10.231.199.18 with HTTP; Thu, 15 Dec 2011 13:50:16 -0800 (PST) Received: by 10.231.199.18 with HTTP; Thu, 15 Dec 2011 13:50:16 -0800 (PST) In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9F546.6060503@freebsd.org> <4EEA3556.7030105@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 21:50:16 +0000 X-Google-Sender-Auth: 3H_H0RfZRLtgeuGQ66WdKuNcthA Message-ID: From: Chris Rees To: Kevin Oberman Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Michael Larabel , FreeBSD Stable Mailing List , Current FreeBSD , Daniel Kalchev , Michael Ross , freebsd-performance@freebsd.org, "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 21:50:19 -0000 On 15 Dec 2011 21:25, "Kevin Oberman" wrote: > > On Thu, Dec 15, 2011 at 10:46 AM, Chris Rees wrote: > > On 15 December 2011 17:58, O. Hartmann wrote: > >> Since ZFS in Linux can only be achieved via FUSE (ad far as I know), it > >> is legitimate to compare ZFS and ext4. It would be much more competetive > >> to compare Linux BTRFS and FreeBSD ZFS. > >> > > > > > > Er... does ext4 guarantee data integrity? > > > > You're not comparing like with like; please do some research on the > > point of ZFS before asserting that they're fair comparisons. > > > > A fair(er) comparison could be ext4 with UFS+soft-updates. > > Wouldn't UFS+SUJ be the closest atch? Yup. Thanks. Chris From owner-freebsd-stable@FreeBSD.ORG Thu Dec 15 21:55:55 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C8E1F1065670; Thu, 15 Dec 2011 21:55:55 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) by mx1.freebsd.org (Postfix) with ESMTP id A1D4A8FC16; Thu, 15 Dec 2011 21:55:55 +0000 (UTC) Received: from troutmask.apl.washington.edu (localhost.apl.washington.edu [127.0.0.1]) by troutmask.apl.washington.edu (8.14.5/8.14.5) with ESMTP id pBFLttKt087668; Thu, 15 Dec 2011 13:55:55 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.14.5/8.14.5/Submit) id pBFLtsUA087667; Thu, 15 Dec 2011 13:55:54 -0800 (PST) (envelope-from sgk) Date: Thu, 15 Dec 2011 13:55:54 -0800 From: Steve Kargl To: Attilio Rao Message-ID: <20111215215554.GA87606@troutmask.apl.washington.edu> References: <4EE1EAFE.3070408@m5p.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: Andrey Chernov , George Mitchell , Doug Barton , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 21:55:55 -0000 On Thu, Dec 15, 2011 at 05:25:51PM +0100, Attilio Rao wrote: > > I basically went through all the e-mail you just sent and identified 4 > real report on which we could work on and summarizied in the attached > Excel file. > I'd like that George, Steve, Doug, Andrey and Mike possibly review the > few datas there and add more, if they want, or make more important > clarifications in particular about the Xorg presence (or rather not) > in their workload. Your summary of my observations appears correct. I have grabbed an up-to-date /usr/src, built and installed world, and built and installed a new kernel on one of the nodes in my cluster. It has CPU: Dual Core AMD Opteron(tm) Processor 280 (2392.65-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x20f12 Family = f Model = 21 Stepping = 2 Features=0x178bfbff Features2=0x1 AMD Features=0xe2500800 AMD Features2=0x3 real memory = 17179869184 (16384 MB) avail memory = 16269832192 (15516 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 2 package(s) x 2 core(s) I can perform new tests with both ULE and 4BSD, but you'll need to be precise in the information you want collected (and how to collect the data) due to the rather limited amount of time I currently have. To summarize my workload, on the master node on my cluster I start a job that will send N slave jobs to the node of interest. The slaves perform nearly identical cpu-bound floating point computations, so the expectation is that each slave should take nearly the same amount of cpu-time to complete its task. Communication occurs between only the master and a slave at the start of the process and when it finishes. The communication is over GigE ipv4 internal network. The slaves do not read or write to disk. -- Steve From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 02:43:23 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA05E106566C for ; Fri, 16 Dec 2011 02:43:23 +0000 (UTC) (envelope-from cpghost@cordula.ws) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 3BB358FC19 for ; Fri, 16 Dec 2011 02:43:22 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so5141756wgb.31 for ; Thu, 15 Dec 2011 18:43:22 -0800 (PST) MIME-Version: 1.0 Received: by 10.216.131.223 with SMTP id m73mr1957844wei.52.1324001503354; Thu, 15 Dec 2011 18:11:43 -0800 (PST) Received: by 10.223.83.142 with HTTP; Thu, 15 Dec 2011 18:11:43 -0800 (PST) X-Originating-IP: [93.221.183.169] In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215004205.GA11556@icarus.home.lan> Date: Fri, 16 Dec 2011 03:11:43 +0100 Message-ID: From: "C. P. Ghost" To: Tom Evans Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 02:43:23 -0000 On Thu, Dec 15, 2011 at 10:44 AM, Tom Evans wrote: > Real time scheduler changing would be insane! I was thinking that > both/any/all schedulers could be compiled into the kernel, and the > choice of which one to use becomes a boot time configuration. You > don't have to recompile the kernel to change timecounter. Right. Switching the scheduler on the fly may be thinkable though. I could imagine a syscall that would suspend all scheduling, convert the bookkeeping data of one scheduler into the other scheduler's, and transfer control to the other scheduler. Of course, that would require some heavy hacking, as I would imagine that "cross-scheduler surgery" would result in a pretty hard to debug kernel (at least during development). A more general solution could even be a separate userland scheduler process a la L4 [*], but since we don't have lightweight IPC in the kernel (yet, or never), it would require even heavier black wizardry. But nice and flexible it would be. ;-) [*] Refs: - https://github.com/l4ka/pistachio - http://www.systems.ethz.ch/education/past-courses/fall-2010/aos/lectures/wk13-scheduling-print.pdf Regards, -cpghost. -- Cordula's Web. http://www.cordula.ws/ From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 05:41:44 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 87945106566C; Fri, 16 Dec 2011 05:41:44 +0000 (UTC) (envelope-from lacombar@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id B50698FC13; Fri, 16 Dec 2011 05:41:42 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so5358899wgb.31 for ; Thu, 15 Dec 2011 21:41:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=zYwTYYZjl5AzeYGUNZgksYv7U7ZRA1lgZiv53sHsLqY=; b=lCJZFKYrEgsc+rMVUFognpRqr6QWFYval5NLQOlVtyvVvEADUmR92ZFDun6eKFiSOq KLm+xWA8Z+WgFKHaKSywQBzTgwJ/VRBsNE0t39SIxj635aAtBY/TbXWCGyCI3pZ+QPIO +cFk/VR358rN8JR/6hsq1ompIKU8K/SCqIOXA= MIME-Version: 1.0 Received: by 10.180.18.233 with SMTP id z9mr10392844wid.0.1324014101856; Thu, 15 Dec 2011 21:41:41 -0800 (PST) Received: by 10.180.99.226 with HTTP; Thu, 15 Dec 2011 21:41:41 -0800 (PST) In-Reply-To: <4EE9A2A0.80607@zedat.fu-berlin.de> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> Date: Fri, 16 Dec 2011 00:41:41 -0500 Message-ID: From: Arnaud Lacombe To: "O. Hartmann" Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-performance@freebsd.org, Current FreeBSD , FreeBSD Stable Mailing List , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 05:41:44 -0000 Hi, On Thu, Dec 15, 2011 at 2:32 AM, O. Hartmann wrote: > Just saw this shot benchmark on Phoronix dot com today: > > http://www.phoronix.com/scan.php?page=news_item&px=MTAyNzA > it might be worth highlighting that despite Oracle Linux 6.1 Server is using a kernel + compiler almost 2 years old, it still manages to out-perform the bleeding edge FreeBSD :-) Now, from what I've read so far in this thread, it seems that a lot of people are still in abnegation... my 0.2c, - Arnaud > It may be worth to discuss the sad performance of FBSD in some parts of > the benchmark. A difference of a factor 10 or 100 is simply far beyond > disapointing, it is more than inacceptable and by just reading those > benchmarks, I'd like to drop thinking of using FreeBSD even as a backend > server in scientific and business environments. In detail, some of the > SciMark benches look disappointing. The overall image can't help over > the fact that in C-Ray FreeBSD is better performing. > > From the compiler, I'd like say there couldn't be a drop of more than 10 > - 15% in performance - but not 10 or 100 times. > > I'm just thinking about the discussion of SCHED_ULE and all the saur > spots we discussed when I stumbled over the test. > > Regards, > Oliver > From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 06:55:49 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 431BE1065679 for ; Fri, 16 Dec 2011 06:55:49 +0000 (UTC) (envelope-from lists@rewt.org.uk) Received: from abby.lhr1.as41113.net (abby.lhr1.as41113.net [91.208.177.20]) by mx1.freebsd.org (Postfix) with ESMTP id EF3E08FC16 for ; Fri, 16 Dec 2011 06:55:48 +0000 (UTC) Received: from jasmine.internethq (unknown [91.208.177.192]) by abby.lhr1.as41113.net (Postfix) with ESMTP id D61A92283A for ; Fri, 16 Dec 2011 06:44:46 +0000 (UTC) Received: from [172.16.11.44] (jwh-laptop.internethq [172.16.11.44]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by jasmine.internethq (Postfix) with ESMTPS id 2D9B41187EA00; Fri, 16 Dec 2011 07:43:33 +0000 (GMT) Message-ID: <4EEAE8DF.40303@rewt.org.uk> Date: Fri, 16 Dec 2011 06:44:47 +0000 From: Joe Holden User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Arnaud Lacombe References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Stable Mailing List , freebsd-performance@freebsd.org, Current FreeBSD , "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 06:55:49 -0000 Arnaud Lacombe wrote: > Hi, > > On Thu, Dec 15, 2011 at 2:32 AM, O. Hartmann > wrote: >> Just saw this shot benchmark on Phoronix dot com today: >> >> http://www.phoronix.com/scan.php?page=news_item&px=MTAyNzA >> > it might be worth highlighting that despite Oracle Linux 6.1 Server is > using a kernel + compiler almost 2 years old, it still manages to > out-perform the bleeding edge FreeBSD :-) > serenity# gcc --version gcc (GCC) 4.2.1 20070831 patched [FreeBSD] serenity# uname -r 9.0-RC3 > Now, from what I've read so far in this thread, it seems that a lot of > people are still in abnegation... > > my 0.2c, > - Arnaud > >> It may be worth to discuss the sad performance of FBSD in some parts of >> the benchmark. A difference of a factor 10 or 100 is simply far beyond >> disapointing, it is more than inacceptable and by just reading those >> benchmarks, I'd like to drop thinking of using FreeBSD even as a backend >> server in scientific and business environments. In detail, some of the >> SciMark benches look disappointing. The overall image can't help over >> the fact that in C-Ray FreeBSD is better performing. >> >> From the compiler, I'd like say there couldn't be a drop of more than 10 >> - 15% in performance - but not 10 or 100 times. >> >> I'm just thinking about the discussion of SCHED_ULE and all the saur >> spots we discussed when I stumbled over the test. >> >> Regards, >> Oliver >> > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 07:06:13 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01B791065672; Fri, 16 Dec 2011 07:06:13 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id A11568FC08; Fri, 16 Dec 2011 07:06:12 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RbRru-0006Wn-Rh>; Fri, 16 Dec 2011 08:06:10 +0100 Received: from e178018085.adsl.alicedsl.de ([85.178.18.85] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RbRru-00022J-Md>; Fri, 16 Dec 2011 08:06:10 +0100 Message-ID: <4EEAEDE1.50604@zedat.fu-berlin.de> Date: Fri, 16 Dec 2011 08:06:09 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Joe Holden References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EEAE8DF.40303@rewt.org.uk> In-Reply-To: <4EEAE8DF.40303@rewt.org.uk> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig39A77F392D735EB75795DD67" X-Originating-IP: 85.178.18.85 Cc: freebsd-performance@freebsd.org, Current FreeBSD , FreeBSD Stable Mailing List , Jeremy Chadwick , Arnaud Lacombe Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 07:06:13 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig39A77F392D735EB75795DD67 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 12/16/11 07:44, Joe Holden wrote: > Arnaud Lacombe wrote: >> Hi, >> >> On Thu, Dec 15, 2011 at 2:32 AM, O. Hartmann >> wrote: >>> Just saw this shot benchmark on Phoronix dot com today: >>> >>> http://www.phoronix.com/scan.php?page=3Dnews_item&px=3DMTAyNzA >>> >> it might be worth highlighting that despite Oracle Linux 6.1 Server is= >> using a kernel + compiler almost 2 years old, it still manages to >> out-perform the bleeding edge FreeBSD :-) >> > serenity# gcc --version > gcc (GCC) 4.2.1 20070831 patched [FreeBSD] >=20 > serenity# uname -r > 9.0-RC3 >=20 For the underlying OS, as far as I know, the compiler hasn't as much impact as on userland software since autovectorization and other neat things are not used during system build. =46rom my experience using gcc 4.2 or 4.4/4.5 does not have an impact beyond 3% when SSE isn't explicetly enforced. More interesting is the performance gain due to the architecture. I think it would be very easy for M. Larabel to repeat this benchmark with a "bleeding edge" Ubuntu or Suse as well. And since FreeBSD 9.0 can be compiled with CLANG, it should be possible to compare both also with "bleeding edge" compilers, say FreeBSD 9/CLANG, Ubuntu 12/gcc 4.6.2. >> Now, from what I've read so far in this thread, it seems that a lot of= >> people are still in abnegation... >> >> my 0.2c, >> - Arnaud >> >>> It may be worth to discuss the sad performance of FBSD in some parts = of >>> the benchmark. A difference of a factor 10 or 100 is simply far beyon= d >>> disapointing, it is more than inacceptable and by just reading those >>> benchmarks, I'd like to drop thinking of using FreeBSD even as a back= end >>> server in scientific and business environments. In detail, some of th= e >>> SciMark benches look disappointing. The overall image can't help over= >>> the fact that in C-Ray FreeBSD is better performing. >>> >>> From the compiler, I'd like say there couldn't be a drop of more than= 10 >>> - 15% in performance - but not 10 or 100 times. >>> >>> I'm just thinking about the discussion of SCHED_ULE and all the saur >>> spots we discussed when I stumbled over the test. >>> >>> Regards, >>> Oliver --------------enig39A77F392D735EB75795DD67 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQEcBAEBAgAGBQJO6u3hAAoJEOgBcD7A/5N8TCsH/1Aqj3f2K3XC7u6sDI8GlXn6 OK0v5A6UrlGHHWGz6mrJ60EeH8406T/e2eA1E0iRJotQwGdr0Rvpcm+J0bxcqom8 uBVJ/yXLFyiGT3GZR7t27/wrTRXRV9yYlxqaYs9zvTf2e9rUO4ttqx69yNV5SuSI 9wzkqqA8AmctorRrpyj2wVt0iNUFzFFPSBz/REj9vJOjdFPGdqWJwKUVDeEBQrny q/4lZjhmNX5qeeC2/enceYRgN3FjeYjSqHJ2JHw7qKPnYWYF3r7/J7A0A/es2G6b KQjBTs5xfUvnVwyu2gCQoFpNQ92S5kiq6KDZs+RQ6jaQUxBrEwdWHruOGLj2OIg= =+AbL -----END PGP SIGNATURE----- --------------enig39A77F392D735EB75795DD67-- From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 08:03:31 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2DD5D1065672 for ; Fri, 16 Dec 2011 08:03:31 +0000 (UTC) (envelope-from kamikaze@bsdforen.de) Received: from mail.server1.bsdforen.de (bsdforen.de [82.193.243.81]) by mx1.freebsd.org (Postfix) with ESMTP id E7E4C8FC0C for ; Fri, 16 Dec 2011 08:03:30 +0000 (UTC) Received: from mobileKamikaze.norad (HSI-KBW-091-089-161-008.hsi2.kabel-badenwuerttemberg.de [91.89.161.8]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.server1.bsdforen.de (Postfix) with ESMTPSA id 9319B7E882 for ; Fri, 16 Dec 2011 08:45:19 +0100 (CET) Message-ID: <4EEAF70E.1040104@bsdforen.de> Date: Fri, 16 Dec 2011 08:45:18 +0100 From: Dominic Fandrey User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: battery display broken X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 08:03:31 -0000 It seems something broke with the battery display. Last night it showed 94% remaining capacity for more than 2 hours. Afterwards I docked the machine (HP6510b) and rebooted it. Since then more than 8 hours have passed, but it still shows 16% (the LED indicators state that the battery is full and no longer charging). # acpiconf -i b Design capacity: 4703 mAh Last full capacity: 4703 mAh Technology: secondary (rechargeable) Design voltage: 10800 mV Capacity (warn): 236 mAh Capacity (low): 48 mAh Low/warn granularity: 100 mAh Warn/full granularity: 100 mAh Model number: Primary Serial number: 00835 2010/01/05 Type: LIon OEM info: Hewlett-Packard State: charging Remaining capacity: 16% Remaining time: unknown Present rate: 3351 mA (39585 mW) Present voltage: 11813 mV The chipset is Intel 82801 from the ICH8 family. I'm running RELENG_9/amd64. -- A: Because it fouls the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail? From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 08:12:24 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D51EB106566B for ; Fri, 16 Dec 2011 08:12:24 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 6D9C48FC16 for ; Fri, 16 Dec 2011 08:12:24 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 9EB1D7300B; Fri, 16 Dec 2011 09:11:45 +0100 (CET) Date: Fri, 16 Dec 2011 09:11:45 +0100 From: Luigi Rizzo To: "C. P. Ghost" Message-ID: <20111216081145.GA76297@onelab2.iet.unipi.it> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215004205.GA11556@icarus.home.lan> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: Tom Evans , freebsd-stable@freebsd.org, Jeremy Chadwick Subject: switching schedulers (Re: SCHED_ULE should not be the default) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 08:12:25 -0000 On Fri, Dec 16, 2011 at 03:11:43AM +0100, C. P. Ghost wrote: > On Thu, Dec 15, 2011 at 10:44 AM, Tom Evans wrote: > > Real time scheduler changing would be insane! I was thinking that > > both/any/all schedulers could be compiled into the kernel, and the > > choice of which one to use becomes a boot time configuration. You > > don't have to recompile the kernel to change timecounter. > > Right. > > Switching the scheduler on the fly may be thinkable though. > I could imagine a syscall that would suspend all scheduling, > convert the bookkeeping data of one scheduler into the other > scheduler's, and transfer control to the other scheduler. Of > course, that would require some heavy hacking, as I would > imagine that "cross-scheduler surgery" would result in a pretty > hard to debug kernel (at least during development). Since the subject has come up a few times: back in 2002 (boy, it's almost 10 years ago!) we did implement switchable schedulers on FreeBSD 4.x UP, and the diffs and a bit of documentation are still online, and probably the architecture could be reused even now or for the SMP case. announcement and brief description http://kerneltrap.org/node/349 the patch referred in there http://info.iet.unipi.it/~luigi/ps_sched.20020719a.diff The interesting part is probably the definition of the methods that schedulers should implement (see struct _sched_interface ). The switch from one scheduler to another was implemented with a sysctl. This calls the sched_move() method of the current (i.e. old) scheduler, which extracts all ready processes from its own "queues" (however they are implemented) and reinserts them onto the new scheduler's "queues" using its (new) setrunqueue() method. You don't need to bother for blocked process as the scheduler doesn't know much about them. I am not preserving the thread's dynamic "priority" (think of accumulated work, affinity etc.) when switching schedulers, as that is expected to be an infrequent event, and so in the end it doesn't really matter -- at a switch, threads are inserted in the scheduler as newly created ones, using only the static priority as a parameter. At the time I did not address the SMP case for several reasons, but they are all gone now: - did not have a suitable test system - SMP support was still in a state of flux - i did not understand the KSE concept - and i did not have an algorithm for proportional share scheduling (the actual goal of the project) in an SMP context. cheers luigi > > A more general solution could even be a separate userland > scheduler process a la L4 [*], but since we don't have lightweight > IPC in the kernel (yet, or never), it would require even heavier > black wizardry. But nice and flexible it would be. ;-) > > [*] Refs: > - https://github.com/l4ka/pistachio > - http://www.systems.ethz.ch/education/past-courses/fall-2010/aos/lectures/wk13-scheduling-print.pdf > > Regards, > -cpghost. > > -- > Cordula's Web. http://www.cordula.ws/ > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 10:46:36 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8558B106566C for ; Fri, 16 Dec 2011 10:46:36 +0000 (UTC) (envelope-from se@freebsd.org) Received: from nm6-vm0.bullet.mail.sp2.yahoo.com (nm6-vm0.bullet.mail.sp2.yahoo.com [98.139.91.206]) by mx1.freebsd.org (Postfix) with SMTP id 5DBC98FC08 for ; Fri, 16 Dec 2011 10:46:36 +0000 (UTC) Received: from [98.139.91.66] by nm6.bullet.mail.sp2.yahoo.com with NNFMP; 16 Dec 2011 10:46:36 -0000 Received: from [208.71.42.206] by tm6.bullet.mail.sp2.yahoo.com with NNFMP; 16 Dec 2011 10:46:36 -0000 Received: from [127.0.0.1] by smtp217.mail.gq1.yahoo.com with NNFMP; 16 Dec 2011 10:46:35 -0000 X-Yahoo-Newman-Id: 977711.98480.bm@smtp217.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: 2lUakIEVM1kneLJNsjYYoqebdMFdkMHBhveZGfGBOsn4XGy Owk6nnn5Tcd7g8283JunWtIXe03_dcEt7HqwX7i6Xm_h9FcoVdw52ZdefSVS gde1oFe4_25VWsZoNh4SyFrB1Ki_4L5DJ4H1p8_E4SslzbC8qhrZE05FoIa4 NpG1beMdWvWrk2UO0ltZkIa8Q4lRniXKjDey.KCmMoTcfou8m1cpyIbd9Kcu U4EojigYBMs72.8YK9l_n1VH0i.uAxjxjfqte1Ba2wfyboloOe7fbbtUlzEY M_ZAIzK7ziRkSQ79w_9iYpCvdsKJSkVkW6SM2.g2sbHVs6nvL.XNXieumXhp 5pJlxYjQfYFsVRANOnxADYDucDHIKHmlfvpQO7.BFgDJRzcficeZfeHcjLAq EaurHQKgItuft X-Yahoo-SMTP: iDf2N9.swBDAhYEh7VHfpgq0lnq. Received: from [192.168.119.20] (se@81.173.157.6 with plain) by smtp217.mail.gq1.yahoo.com with SMTP; 16 Dec 2011 02:46:35 -0800 PST Message-ID: <4EEB218B.1090209@freebsd.org> Date: Fri, 16 Dec 2011 11:46:35 +0100 From: Stefan Esser User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Luigi Rizzo References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215004205.GA11556@icarus.home.lan> <20111216081145.GA76297@onelab2.iet.unipi.it> In-Reply-To: <20111216081145.GA76297@onelab2.iet.unipi.it> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Tom Evans , freebsd-stable@freebsd.org, "C. P. Ghost" , Jeremy Chadwick Subject: Re: switching schedulers (Re: SCHED_ULE should not be the default) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 10:46:36 -0000 Am 16.12.2011 09:11, schrieb Luigi Rizzo: > The interesting part is probably the definition of the methods that > schedulers should implement (see struct _sched_interface ). > > The switch from one scheduler to another was implemented with a > sysctl. This calls the sched_move() method of the current (i.e. > old) scheduler, which extracts all ready processes from its own > "queues" (however they are implemented) and reinserts them onto the > new scheduler's "queues" using its (new) setrunqueue() method. You > don't need to bother for blocked process as the scheduler doesn't > know much about them. > > I am not preserving the thread's dynamic "priority" (think of > accumulated work, affinity etc.) when switching > schedulers, as that is expected to be an infrequent event, and > so in the end it doesn't really matter -- at a switch, threads > are inserted in the scheduler as newly created ones, using only > the static priority as a parameter. I think this is OK for user processes (which will receive reasonable relative priorities after running a fraction of a second, anyway). But I'm not sure whether it is possible to use static priorities for (real-time) kernel threads, where priority inversion may occur, if the current dynamic (relative) thread priorities are not preserved. But not only the relative priorities of the existing processes must be preserved, new kernel threads must be created with matching (relative) priorities. This means, that the schedulers may be switched at any time, but the priority values should be portable between schedulers to prevent dead-lock (or illegal order of execution?) of threads (AFAICT). Regards, STefan From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 10:55:00 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2EA53106566B; Fri, 16 Dec 2011 10:55:00 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-ww0-f42.google.com (mail-ww0-f42.google.com [74.125.82.42]) by mx1.freebsd.org (Postfix) with ESMTP id 62C2A8FC1A; Fri, 16 Dec 2011 10:54:59 +0000 (UTC) Received: by wgbds13 with SMTP id ds13so2866781wgb.1 for ; Fri, 16 Dec 2011 02:54:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=XyEIxMMq4/9l31BQXKMWLuyKLUZiqoXczwT5f7NDMno=; b=s67ButLlyjpBdJwR5ttaf2OHIVq7EQWEGzQavWNOJ4GZHm72+SeTTVSnnkmhC9ejV5 E0DNl11XJyHwxn9Sn7ElEAGLAcVZtDLxYp+IDSG8OIJpfUoTNbZNtspTwsdR8g9V6k17 9oKlUxCrt/xqvDWZFv4Fy6iIZENzGGgLYTbyw= MIME-Version: 1.0 Received: by 10.180.74.211 with SMTP id w19mr11794028wiv.7.1324032896461; Fri, 16 Dec 2011 02:54:56 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Fri, 16 Dec 2011 02:54:55 -0800 (PST) In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> Date: Fri, 16 Dec 2011 11:54:55 +0100 X-Google-Sender-Auth: KA3O_OU4bWP9K1T7FdTNQyuu6uY Message-ID: From: Attilio Rao To: Arnaud Lacombe Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD Stable Mailing List , freebsd-performance@freebsd.org, Current FreeBSD , "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 10:55:00 -0000 2011/12/16 Arnaud Lacombe : > Hi, > > On Thu, Dec 15, 2011 at 2:32 AM, O. Hartmann > wrote: >> Just saw this shot benchmark on Phoronix dot com today: >> >> http://www.phoronix.com/scan.php?page=3Dnews_item&px=3DMTAyNzA >> > it might be worth highlighting that despite Oracle Linux 6.1 Server is > using a kernel + compiler almost 2 years old, it still manages to > out-perform the bleeding edge FreeBSD :-) > > Now, from what I've read so far in this thread, it seems that a lot of > people are still in abnegation... > > my 0.2c, > =C2=A0- Arnaud Said by someone which really thinks passing __FILE__ and __LINE__ to kernel function is going to give a mesaurable performance penalty is really hilarious however :) It is crystal clear you really don't understand how to make reliable benchmarks (and likely you don't really have a grasp of nowaday's machine contention points), so why you keep talking about it? It would be more valuable for you and whatever project you follow if you spend your time coding and making real benchmarking. Attilio --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 11:14:26 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D0571065673; Fri, 16 Dec 2011 11:14:26 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id CED428FC16; Fri, 16 Dec 2011 11:14:25 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so5864105wgb.31 for ; Fri, 16 Dec 2011 03:14:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=bR9AOidlE7MbU8fkvdHWIXr/OXepMtQeDMqZOVV3JTM=; b=dlKGUkVMF4RvDLhfeWAUl733AjOCSZ4ZAvgRE3FGnuSP7XPaMvXGKp7bzPIO1zbok6 vvOQzaeBWkYlSOnWINH3aIInGkLMqp9fl+5voQXF1k4ZqFd9SAtOilIlYIs007KMNBZY 8bkLrNuWpBbaX0VexGHzHRgFovihWLwhdKdrg= MIME-Version: 1.0 Received: by 10.227.197.70 with SMTP id ej6mr5312698wbb.13.1324034064488; Fri, 16 Dec 2011 03:14:24 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Fri, 16 Dec 2011 03:14:24 -0800 (PST) In-Reply-To: <20111215215554.GA87606@troutmask.apl.washington.edu> References: <4EE1EAFE.3070408@m5p.com> <20111215215554.GA87606@troutmask.apl.washington.edu> Date: Fri, 16 Dec 2011 12:14:24 +0100 X-Google-Sender-Auth: bVjdbKzvcGOQiaRRmYO4IIS0Ddg Message-ID: From: Attilio Rao To: Steve Kargl Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Andrey Chernov , George Mitchell , Doug Barton , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 11:14:26 -0000 2011/12/15 Steve Kargl : > On Thu, Dec 15, 2011 at 05:25:51PM +0100, Attilio Rao wrote: >> >> I basically went through all the e-mail you just sent and identified 4 >> real report on which we could work on and summarizied in the attached >> Excel file. >> I'd like that George, Steve, Doug, Andrey and Mike possibly review the >> few datas there and add more, if they want, or make more important >> clarifications in particular about the Xorg presence (or rather not) >> in their workload. > > Your summary of my observations appears correct. > > I have grabbed an up-to-date /usr/src, built and > installed world, and built and installed a new > kernel on one of the nodes in my cluster. =C2=A0It > has > > CPU: Dual Core AMD Opteron(tm) Processor 280 (2392.65-MHz K8-class CPU) > =C2=A0Origin =3D "AuthenticAMD" =C2=A0Id =3D 0x20f12 =C2=A0Family =3D f = =C2=A0Model =3D 21 =C2=A0Stepping =3D 2 > =C2=A0Features=3D0x178bfbff =C2=A0MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> > =C2=A0Features2=3D0x1 > =C2=A0AMD Features=3D0xe2500800 > =C2=A0AMD Features2=3D0x3 > real memory =C2=A0=3D 17179869184 (16384 MB) > avail memory =3D 16269832192 (15516 MB) > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > FreeBSD/SMP: 2 package(s) x 2 core(s) > > I can perform new tests with both ULE and 4BSD, but you'll > need to be precise in the information you want collected > (and how to collect the data) due to the rather limited > amount of time I currently have. It seems a perfect environment, just please make sure you made a debug-free userland (setting MALLOC_PRODUCTION in jemalloc basically). The first thing is, can you try reproducing your case? As far as I got it, for you it was enough to run N + small_amount of CPU-bound threads to show performance penalty, so I'd ask you to start with using dnetc or just your preferred cpu-bound workload and verify you can reproduce the issue. As it happens, please monitor the threads bouncing and CPU utilization via 'top' (you don't need to be 100% precise, jut to get an idea, and keep an eye on things like excessive threads migration, thread binding obsessity, low throughput on CPU). One note: if your workloads need to do I/O please use a tempfs or memory storage to do so, in order to reduce I/O effects at all. Also, verify this doesn't happen with 4BSD scheduler, just in case. Finally, if the problem is still in place, please recompile your kernel by adding: options KTR options KTR_ENTRIES=3D262144 options KTR_COMPILE=3D(KTR_SCHED) options KTR_MASK=3D(KTR_SCHED) And reproduce the issue. When you are in the middle of the scheduling issue go with: # ktrdump -ctf > ktr-ule-problem-YOURNAME.out and send to the mailing list along with your dmesg and the informations on the CPU utilization you gathered by top(1). That should cover it all, but if you have further questions, please just go ahead. Thanks, Attilio --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 13:08:04 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 779E71065673 for ; Fri, 16 Dec 2011 13:08:04 +0000 (UTC) (envelope-from se@freebsd.org) Received: from nm25.bullet.mail.sp2.yahoo.com (nm25.bullet.mail.sp2.yahoo.com [98.139.91.95]) by mx1.freebsd.org (Postfix) with SMTP id 4DA7B8FC18 for ; Fri, 16 Dec 2011 13:08:04 +0000 (UTC) Received: from [98.139.91.65] by nm25.bullet.mail.sp2.yahoo.com with NNFMP; 16 Dec 2011 13:08:03 -0000 Received: from [208.71.42.194] by tm5.bullet.mail.sp2.yahoo.com with NNFMP; 16 Dec 2011 13:08:03 -0000 Received: from [127.0.0.1] by smtp205.mail.gq1.yahoo.com with NNFMP; 16 Dec 2011 13:08:03 -0000 X-Yahoo-Newman-Id: 652125.75823.bm@smtp205.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: Cu40vcsVM1kJrvHsrx71G96hiB690PcmeMkTDiHI9oBRkqt YWQJaFGRuJPY_TmwKWgyr7BGneCUFJfTRWT5iARHmc7ipc3nV8AdrgUR6vMF 0_kZwnzdRuhsmVfiVpdbXwhjVTNtG1RIFXPfs.cKU1t.gSbYV_9iwkz9pIWd IyvnUhhhOzaeIR1NSHXaPLRHZlM56AkBCfixeJsZQv0ftPmgFdUCitG1dzob x0n836_d7U3mCZQCH9oqFmwwjVIXbvJZ11uzhOZMo2poGINkNcjXRSSDZ6WZ KzltOPyargb9EWBqH9VhESRphb4ZVOV209e2NI1lY9x1tYLt2J3XeXz0dofn hgdaFsTQGBIFmyQFIOexA2.4FSL8wzJ7pRguDO1USB0ub2c9r8Dn_oceFLya KKlwUheSvUcU2 X-Yahoo-SMTP: iDf2N9.swBDAhYEh7VHfpgq0lnq. Received: from [192.168.119.20] (se@81.173.157.6 with plain) by smtp205.mail.gq1.yahoo.com with SMTP; 16 Dec 2011 05:08:03 -0800 PST Message-ID: <4EEB42B1.1000506@freebsd.org> Date: Fri, 16 Dec 2011 14:08:01 +0100 From: Stefan Esser User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: "O. Hartmann" References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EEAE8DF.40303@rewt.org.uk> <4EEAEDE1.50604@zedat.fu-berlin.de> In-Reply-To: <4EEAEDE1.50604@zedat.fu-berlin.de> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Joe Holden , FreeBSD Stable Mailing List , Current FreeBSD , Arnaud Lacombe , freebsd-performance@freebsd.org, Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 13:08:04 -0000 Am 16.12.2011 08:06, schrieb O. Hartmann: > For the underlying OS, as far as I know, the compiler hasn't as much > impact as on userland software since autovectorization and other neat > things are not used during system build. > > From my experience using gcc 4.2 or 4.4/4.5 does not have an impact > beyond 3% when SSE isn't explicetly enforced. Well, but the compute intensive tests showed performance variance of a few percents only, IIRC. The big differences were in the parts that heavily depend on file system and buffer cache concepts (i.e. the low limit on dirty buffers in FreeBSD, which is very beneficial in real world situations; do you remember the first few releases of SunOS-4, which heavily suffered in interactive performance due to a naive unified buffer cache VM system that did not limit the amount of dirty buffers? It caused interactive shells to be swapped out within seconds on systems with background jobs writing to disk). > More interesting is the performance gain due to the architecture. I > think it would be very easy for M. Larabel to repeat this benchmark with > a "bleeding edge" Ubuntu or Suse as well. And since FreeBSD 9.0 can be > compiled with CLANG, it should be possible to compare both also with > "bleeding edge" compilers, say FreeBSD 9/CLANG, Ubuntu 12/gcc 4.6.2. Clang may be considered "bleeding edge", but in quite a different way than gcc-4.6.2. While the latter can look back on 2 decades of development, clang is still in a state where feature completeness (and bug-to-bug compatibility with GCC ;-) is much more important than performance. there is much promise of powerful optimizations becoming available in clang once it is mature, but just now expect GCC 4.6.2 to deliver 5% to 10% higher performance than clang. But as stated before: To exclude compiler dependencies just run the Linux binaries on FreeBSD. There is slight emulation overhead and Glibc is not particularly optimized for FreeBSD, but this will still provide more useful results. And the tests should be selected to represent reasonable real-world scenarios. Server programs tested on otherwise idle systems and running for just a few seconds (not reaching equilibrium during the majority of the test period) are not representative at all (again: if your goal is to compare server performance). Regards, STefan From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 15:07:47 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A8BDE1065675; Fri, 16 Dec 2011 15:07:47 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 4609C8FC21; Fri, 16 Dec 2011 15:07:47 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 3CA247300A; Fri, 16 Dec 2011 16:24:09 +0100 (CET) Date: Fri, 16 Dec 2011 16:24:09 +0100 From: Luigi Rizzo To: Stefan Esser Message-ID: <20111216152409.GA79938@onelab2.iet.unipi.it> References: <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215004205.GA11556@icarus.home.lan> <20111216081145.GA76297@onelab2.iet.unipi.it> <4EEB218B.1090209@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EEB218B.1090209@freebsd.org> User-Agent: Mutt/1.4.2.3i Cc: Tom Evans , freebsd-stable@freebsd.org, "C. P. Ghost" , Jeremy Chadwick Subject: Re: switching schedulers (Re: SCHED_ULE should not be the default) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 15:07:47 -0000 On Fri, Dec 16, 2011 at 11:46:35AM +0100, Stefan Esser wrote: > Am 16.12.2011 09:11, schrieb Luigi Rizzo: > > The interesting part is probably the definition of the methods that > > schedulers should implement (see struct _sched_interface ). > > > > The switch from one scheduler to another was implemented with a > > sysctl. This calls the sched_move() method of the current (i.e. > > old) scheduler, which extracts all ready processes from its own > > "queues" (however they are implemented) and reinserts them onto the > > new scheduler's "queues" using its (new) setrunqueue() method. You > > don't need to bother for blocked process as the scheduler doesn't > > know much about them. > > > > I am not preserving the thread's dynamic "priority" (think of > > accumulated work, affinity etc.) when switching > > schedulers, as that is expected to be an infrequent event, and > > so in the end it doesn't really matter -- at a switch, threads > > are inserted in the scheduler as newly created ones, using only > > the static priority as a parameter. > > I think this is OK for user processes (which will receive reasonable > relative priorities after running a fraction of a second, anyway). > > But I'm not sure whether it is possible to use static priorities for > (real-time) kernel threads, where priority inversion may occur, if the > current dynamic (relative) thread priorities are not preserved. the word "priority" is too overloaded in this context, as it mixes configuration information (which i called "static priority", and would be really better characterized as the "service parameters" you specify when you start a new thread) and scheduler state ("dynamic priority" in a priority based scheduler, but other schedulers have different state info, such as tickets, virtual times, deadlines, cpu affinity and so on). What i meant to say is that the way i implemented it (and i believe it is almost the only practical way), on a change of scheduler, all processes are requeued as if they had just started. Then it will be the active scheduler the one who can change the initial state according the evolution of the system (changing priorities, tickets, virtual times, deadlines, etc.) > But not only the relative priorities of the existing processes must be > preserved, new kernel threads must be created with matching (relative) > priorities. This means, that the schedulers may be switched at any time, > but the priority values should be portable between schedulers to prevent > dead-lock (or illegal order of execution?) of threads (AFAICT). This issue (i think you have in mind priority inheritance, priority inversion and related issues) is almost irrelevant in FreeBSD, and i am really sorry to see that it comes up so frequently in discussions and sometimes also in documentation related to process schedulers. Apart from bugs in the implementation (see Bruce Evans' email from a few days ago), our CPU schedulers are a collection of heuristics without formally proved properties. So, as much as we can trust developers to come up with effective solutions: - we cannot rely on priorities for correctness (mutual exclusion or deadlock avoidance); - we don't have any support for real time guarantees; - average performance (which is why some of our priority-based schedulers may decide to implement priority inheritance) is not affected by events as infrequent as changing schedulers. cheers luigi From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 15:56:42 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8918F106564A for ; Fri, 16 Dec 2011 15:56:42 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) by mx1.freebsd.org (Postfix) with ESMTP id D6A798FC14 for ; Fri, 16 Dec 2011 15:56:41 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id pBGFStqZ010222; Sat, 17 Dec 2011 02:28:55 +1100 (EST) (envelope-from smithi@nimnet.asn.au) Date: Sat, 17 Dec 2011 02:28:55 +1100 (EST) From: Ian Smith To: Dominic Fandrey In-Reply-To: <4EEAF70E.1040104@bsdforen.de> Message-ID: <20111217000216.Y64681@sola.nimnet.asn.au> References: <4EEAF70E.1040104@bsdforen.de> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-stable@freebsd.org Subject: Re: battery display broken X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 15:56:42 -0000 On Fri, 16 Dec 2011, Dominic Fandrey wrote: > It seems something broke with the battery display. Last night it > showed 94% remaining capacity for more than 2 hours. > > Afterwards I docked the machine (HP6510b) and rebooted it. Since then > more than 8 hours have passed, but it still shows 16% (the LED indicators > state that the battery is full and no longer charging). > > # acpiconf -i b > Design capacity: 4703 mAh > Last full capacity: 4703 mAh > Technology: secondary (rechargeable) > Design voltage: 10800 mV > Capacity (warn): 236 mAh > Capacity (low): 48 mAh > Low/warn granularity: 100 mAh > Warn/full granularity: 100 mAh > Model number: Primary > Serial number: 00835 2010/01/05 > Type: LIon > OEM info: Hewlett-Packard > State: charging > Remaining capacity: 16% > Remaining time: unknown > Present rate: 3351 mA (39585 mW) > Present voltage: 11813 mV At least four things can go wrong. The battery charging circuit might be broken (my first T23 failed that way after 5 years); the battery might just need 'conditioning' (discharged to exhaustion, beyond normal low-battery shutdown, then fully charged - perhaps twice), to reset its internal Coulomb Counter; the CC chip may be faulty; or the battery itself may be failing / have failed, usually one cell first. Is the battery hot at this stage? If that 'Present rate' is correct, a 3.35A/39.6W charge should tend to overheat the battery over time, if charging continues beyond full capacity, which may indicate a bad cell. 11.8V seems too low for a fully-charged 10.8V nominal LIon battery. I have several 4.0 and 4.4Ah like the below, which charge to ~12.4V, and only get down to 11.8V while discharging, at around 85% nom. capacity. Interesting that your LEDs display a different view; perhaps BIOS + EC just monitors voltage and cuts charge, but then what's reporting that fairly high charge rate? I'd expect the Embedded Controller to be doing that .. any dmesg indications of ACPI problems talking to the EC? The 16% is likely from the battery's onboard coulomb counter, but then so might be the (bogus?) charge rate report. All speculative, I know .. # acpiconf -i 0 # Thinkpad T23, 8.2-R, older but ok battery. Design capacity: 43200 mWh Last full capacity: 31850 mWh Technology: secondary (rechargeable) Design voltage: 10800 mV Capacity (warn): 2160 mWh Capacity (low): 432 mWh Low/warn granularity: 1 mWh Warn/full granularity: 1 mWh Model number: IBM-02K7026 Serial number: 932 Type: LION OEM info: Panasonic State: high Remaining capacity: 100% Remaining time: unknown Present rate: 0 mW Present voltage: 12386 mV > The chipset is Intel 82801 from the ICH8 family. I'm running > RELENG_9/amd64. All that said, I don't know specifically how HP do things, or what normal full charge voltage is expected. Tried another battery? cheers, Ian From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 16:30:40 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7EC401065673; Fri, 16 Dec 2011 16:30:40 +0000 (UTC) (envelope-from lacombar@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 71C908FC22; Fri, 16 Dec 2011 16:30:39 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so6371922wgb.31 for ; Fri, 16 Dec 2011 08:30:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=5/cArYsjdM6kpusmLhCLeswP/2xGAu77ztNND69nTLs=; b=PLWlgjxgUj4kZvi0O0YfJbWBPqKffMuhFMOXqlT9zWuqrux+rmkjgcD26fVJkmebT3 Qvs9INeAkBQkOEgz9cfkrXU7LXV3mxsNEgYoaQDd6tU+o0LY045iiUve5J6+cMhjHKaQ b/1lP1Yp3Og7LTlFa8v24s7g6A5XM9/f9AdGw= MIME-Version: 1.0 Received: by 10.216.139.96 with SMTP id b74mr923397wej.10.1324053038541; Fri, 16 Dec 2011 08:30:38 -0800 (PST) Received: by 10.180.99.226 with HTTP; Fri, 16 Dec 2011 08:30:38 -0800 (PST) In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> Date: Fri, 16 Dec 2011 11:30:38 -0500 Message-ID: From: Arnaud Lacombe To: Attilio Rao Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD Stable Mailing List , freebsd-performance@freebsd.org, Current FreeBSD , "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 16:30:40 -0000 Hi, [resend on the ml, my bad] On Fri, Dec 16, 2011 at 5:54 AM, Attilio Rao wrote: > 2011/12/16 Arnaud Lacombe : >> Hi, >> >> On Thu, Dec 15, 2011 at 2:32 AM, O. Hartmann >> wrote: >>> Just saw this shot benchmark on Phoronix dot com today: >>> >>> http://www.phoronix.com/scan.php?page=3Dnews_item&px=3DMTAyNzA >>> >> it might be worth highlighting that despite Oracle Linux 6.1 Server is >> using a kernel + compiler almost 2 years old, it still manages to >> out-perform the bleeding edge FreeBSD :-) >> >> Now, from what I've read so far in this thread, it seems that a lot of >> people are still in abnegation... >> >> my 0.2c, >> =A0- Arnaud > > Said by someone which really thinks passing __FILE__ and __LINE__ to > kernel function is going to give a mesaurable performance penalty is > really hilarious however :) > You are right, the rest of the kernel's subsystem are so sluggish, fragile and half baked that this would barely improve anything... That will be my last word in this thread. - Arnaud From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 16:43:30 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 059511065678; Fri, 16 Dec 2011 16:43:30 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6D9898FC25; Fri, 16 Dec 2011 16:43:28 +0000 (UTC) Received: by vcbfk1 with SMTP id fk1so4457132vcb.13 for ; Fri, 16 Dec 2011 08:43:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=XEYl0wTkXG1IL4Pjffk3B4QbOIqIq8MwBXCjEJejHas=; b=t5UL9rRd2wH80a7awhSuAQtR2gPpgGt1YkLMuQo1cX4ZODtkP4Gbp2zmZklJWH7gHR ygKGdYq/kK6LQPXHL4abExs7pz/NAprYwGQcM1HXwN5G84wllrAdsqGcfbLGiqiJ1M7E u294CtOfDrASMI8oa0OvTjStJCS+qVjagzXvU= MIME-Version: 1.0 Received: by 10.220.151.204 with SMTP id d12mr4077535vcw.40.1324053808493; Fri, 16 Dec 2011 08:43:28 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.52.26.50 with HTTP; Fri, 16 Dec 2011 08:43:27 -0800 (PST) In-Reply-To: <4EEB42B1.1000506@freebsd.org> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EEAE8DF.40303@rewt.org.uk> <4EEAEDE1.50604@zedat.fu-berlin.de> <4EEB42B1.1000506@freebsd.org> Date: Fri, 16 Dec 2011 08:43:27 -0800 X-Google-Sender-Auth: KjC5an4W3kNycKQhm4CA4y8apPY Message-ID: From: Adrian Chadd To: Stefan Esser Content-Type: text/plain; charset=ISO-8859-1 Cc: Joe Holden , FreeBSD Stable Mailing List , Current FreeBSD , Arnaud Lacombe , freebsd-performance@freebsd.org, "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 16:43:30 -0000 Can someone please write up a nice, concise blog post somewhere outlining all of this? Extra bonus points if it's a blog that is picked up by blogs.freebsdish.org and/or some of the other BSD sites. Guys/girls/fuzzy things - this is 2011; people look at shiny blog sites with graphs rather than mailing lists. Sorry, we lost that battle. :) Adrian From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 17:04:06 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F3F2B106566B for ; Fri, 16 Dec 2011 17:04:05 +0000 (UTC) (envelope-from oliver.pntr@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id B46B48FC13 for ; Fri, 16 Dec 2011 17:04:05 +0000 (UTC) Received: by ggnp1 with SMTP id p1so4501563ggn.13 for ; Fri, 16 Dec 2011 09:04:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=oivGKRXCgagGxQdl2yRuji/OdFplZ7WY81vCR+S7nzE=; b=NnUEfk61+Mrjkf0cP0pXR3oeWR0/orOGysJ9sXjRL6QziG1NTUCmIDq/XD5Km1LMjD rWwZLhJglS0Wz+RFPb37fExL1a2i3TRqvFbSudFlDFRhR7/pEeoGv8rXTvXanviYDMXC EOT/G0PnzQEsTj2xXlGSar855JhhOi8Y92rno= MIME-Version: 1.0 Received: by 10.182.185.66 with SMTP id fa2mr4610893obc.67.1324055044983; Fri, 16 Dec 2011 09:04:04 -0800 (PST) Received: by 10.182.74.168 with HTTP; Fri, 16 Dec 2011 09:04:04 -0800 (PST) In-Reply-To: <20111217000216.Y64681@sola.nimnet.asn.au> References: <4EEAF70E.1040104@bsdforen.de> <20111217000216.Y64681@sola.nimnet.asn.au> Date: Fri, 16 Dec 2011 18:04:04 +0100 Message-ID: From: Oliver Pinter To: Ian Smith Content-Type: text/plain; charset=ISO-8859-1 Cc: Dominic Fandrey , freebsd-stable@freebsd.org Subject: Re: battery display broken X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 17:04:06 -0000 Some others have problem with HP laptop and freebsd battery indicator. kern/162859: [acpi] ACPI battery/acline monitoring partialy working (switching) On 12/16/11, Ian Smith wrote: > On Fri, 16 Dec 2011, Dominic Fandrey wrote: > > It seems something broke with the battery display. Last night it > > showed 94% remaining capacity for more than 2 hours. > > > > Afterwards I docked the machine (HP6510b) and rebooted it. Since then > > more than 8 hours have passed, but it still shows 16% (the LED indicators > > state that the battery is full and no longer charging). > > > > # acpiconf -i b > > Design capacity: 4703 mAh > > Last full capacity: 4703 mAh > > Technology: secondary (rechargeable) > > Design voltage: 10800 mV > > Capacity (warn): 236 mAh > > Capacity (low): 48 mAh > > Low/warn granularity: 100 mAh > > Warn/full granularity: 100 mAh > > Model number: Primary > > Serial number: 00835 2010/01/05 > > Type: LIon > > OEM info: Hewlett-Packard > > State: charging > > Remaining capacity: 16% > > Remaining time: unknown > > Present rate: 3351 mA (39585 mW) > > Present voltage: 11813 mV > > At least four things can go wrong. The battery charging circuit might > be broken (my first T23 failed that way after 5 years); the battery > might just need 'conditioning' (discharged to exhaustion, beyond normal > low-battery shutdown, then fully charged - perhaps twice), to reset its > internal Coulomb Counter; the CC chip may be faulty; or the battery > itself may be failing / have failed, usually one cell first. > > Is the battery hot at this stage? If that 'Present rate' is correct, a > 3.35A/39.6W charge should tend to overheat the battery over time, if > charging continues beyond full capacity, which may indicate a bad cell. > > 11.8V seems too low for a fully-charged 10.8V nominal LIon battery. I > have several 4.0 and 4.4Ah like the below, which charge to ~12.4V, and > only get down to 11.8V while discharging, at around 85% nom. capacity. > > Interesting that your LEDs display a different view; perhaps BIOS + EC > just monitors voltage and cuts charge, but then what's reporting that > fairly high charge rate? I'd expect the Embedded Controller to be doing > that .. any dmesg indications of ACPI problems talking to the EC? > > The 16% is likely from the battery's onboard coulomb counter, but then > so might be the (bogus?) charge rate report. All speculative, I know .. > > # acpiconf -i 0 # Thinkpad T23, 8.2-R, older but ok battery. > Design capacity: 43200 mWh > Last full capacity: 31850 mWh > Technology: secondary (rechargeable) > Design voltage: 10800 mV > Capacity (warn): 2160 mWh > Capacity (low): 432 mWh > Low/warn granularity: 1 mWh > Warn/full granularity: 1 mWh > Model number: IBM-02K7026 > Serial number: 932 > Type: LION > OEM info: Panasonic > State: high > Remaining capacity: 100% > Remaining time: unknown > Present rate: 0 mW > Present voltage: 12386 mV > > > > The chipset is Intel 82801 from the ICH8 family. I'm running > > RELENG_9/amd64. > > All that said, I don't know specifically how HP do things, or what > normal full charge voltage is expected. Tried another battery? > > cheers, Ian > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 19:51:51 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 06C40106564A for ; Fri, 16 Dec 2011 19:51:51 +0000 (UTC) (envelope-from meyersh@morningside.edu) Received: from mail-tul01m020-f182.google.com (mail-tul01m020-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id D14238FC12 for ; Fri, 16 Dec 2011 19:51:50 +0000 (UTC) Received: by obbwd18 with SMTP id wd18so863678obb.13 for ; Fri, 16 Dec 2011 11:51:50 -0800 (PST) Received: by 10.182.111.69 with SMTP id ig5mr5029471obb.9.1324065110197; Fri, 16 Dec 2011 11:51:50 -0800 (PST) MIME-Version: 1.0 Received: by 10.182.143.67 with HTTP; Fri, 16 Dec 2011 11:51:09 -0800 (PST) From: Shaun Meyer Date: Fri, 16 Dec 2011 13:51:09 -0600 Message-ID: To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=UTF-8 Subject: net/samba36 winbindd core dump on group lookup X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 19:51:51 -0000 freebsd-stable@, I'm building Samba using net/samba36 with the following config: LDAP=on "With LDAP support" ADS=on "With Active Directory support" CUPS=off "With CUPS printing support" WINBIND=on "With WinBIND support" SWAT=off "With SWAT WebGUI" ACL_SUPPORT=on "With ACL support" AIO_SUPPORT=off "With Asyncronous IO support" FAM_SUPPORT=off "With File Alteration Monitor" SYSLOG=off "With Syslog support" QUOTAS=on "With Disk quota support" UTMP=off "With UTMP accounting support" PAM_SMBPASS=off "With PAM authentication vs passdb backends" DNSUPDATE=off "With dynamic DNS update(require ADS)" AVAHI=off "With Bonjour service discovery support" EXP_MODULES=on "With experimental modules" POPT=on "With system-wide POPT library" IPV6=on "With IPv6 support" MAX_DEBUG=off "With maximum debugging" SMBTORTURE=off "With smbtorture" I am able to join winbind to our 2003 domain, no problem. I'm using idmap_rid(8) backend and am using nss_winbind to enumerate uid/gid's. This set up runs fine for a few minutes (10?) until winbind begins dumping core when I do a group look up something like `getent group` or `id`. Before the core dumping begins: `wbinfo -g` returns all groups, `getent passwd` returns all users, `getent group` returns all groups, and `id [user]` returns all information as it should. After the core dumping begins: `wbinfo -g` returns all groups, `getent passwd` returns all users, `getent group` or `id [user]` causes a core dump in winbindd. The daemon continues running however. After the core dumping routine starts, I have to do some gyrations of `rm -rf /var/db/samba /usr/local/etc/samba`, restarting services, and joining/leaving the domain to get another ~10 minutes of functioning winbind. With MAX_DEBUG=on, I tried to use gdb on the winbindd.core file which shows lines like "#1627 0x0000000000000000 in ?? ()" and similar. I'm guessing this is to do with other system libs that are not compiled with -g, but I'm over my head here on FreeBSD system specifics. So I have two questions: 1) Will a buildworld with debugging symbols get me more meaningful output from gdb `backtrace full` and is this the best way? 2) Has anyone else experienced similar issues with net/samba36? I can reproduce this bug on both 9.0-RC2 and 8.2-RELEASE-p3. Your advice is appreciated, -- Shaun Meyer From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 20:11:36 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BAD661065672 for ; Fri, 16 Dec 2011 20:11:36 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA11.westchester.pa.mail.comcast.net (qmta11.westchester.pa.mail.comcast.net [76.96.59.211]) by mx1.freebsd.org (Postfix) with ESMTP id 6876E8FC19 for ; Fri, 16 Dec 2011 20:11:36 +0000 (UTC) Received: from omta12.westchester.pa.mail.comcast.net ([76.96.62.44]) by QMTA11.westchester.pa.mail.comcast.net with comcast id 9w0l1i0060xGWP85BwBcMn; Fri, 16 Dec 2011 20:11:36 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta12.westchester.pa.mail.comcast.net with comcast id 9wBb1i01A1t3BNj3YwBcQo; Fri, 16 Dec 2011 20:11:36 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 1F272102C19; Fri, 16 Dec 2011 12:11:34 -0800 (PST) Date: Fri, 16 Dec 2011 12:11:34 -0800 From: Jeremy Chadwick To: Shaun Meyer Message-ID: <20111216201134.GA55497@icarus.home.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: net/samba36 winbindd core dump on group lookup X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 20:11:36 -0000 On Fri, Dec 16, 2011 at 01:51:09PM -0600, Shaun Meyer wrote: > freebsd-stable@, > > I'm building Samba using net/samba36 with the following config: > LDAP=on "With LDAP support" > ADS=on "With Active Directory support" > CUPS=off "With CUPS printing support" > WINBIND=on "With WinBIND support" > SWAT=off "With SWAT WebGUI" > ACL_SUPPORT=on "With ACL support" > AIO_SUPPORT=off "With Asyncronous IO support" > FAM_SUPPORT=off "With File Alteration Monitor" > SYSLOG=off "With Syslog support" > QUOTAS=on "With Disk quota support" > UTMP=off "With UTMP accounting support" > PAM_SMBPASS=off "With PAM authentication vs passdb backends" > DNSUPDATE=off "With dynamic DNS update(require ADS)" > AVAHI=off "With Bonjour service discovery support" > EXP_MODULES=on "With experimental modules" > POPT=on "With system-wide POPT library" > IPV6=on "With IPv6 support" > MAX_DEBUG=off "With maximum debugging" > SMBTORTURE=off "With smbtorture" > > I am able to join winbind to our 2003 domain, no problem. I'm using > idmap_rid(8) backend and am using nss_winbind to enumerate uid/gid's. > > This set up runs fine for a few minutes (10?) until winbind begins > dumping core when I do a group look up something like `getent group` > or `id`. > > Before the core dumping begins: `wbinfo -g` returns all groups, > `getent passwd` returns all users, `getent group` returns all groups, > and `id [user]` returns all information as it should. > > After the core dumping begins: `wbinfo -g` returns all groups, `getent > passwd` returns all users, `getent group` or `id [user]` causes a core > dump in winbindd. The daemon continues running however. > > After the core dumping routine starts, I have to do some gyrations of > `rm -rf /var/db/samba /usr/local/etc/samba`, restarting services, and > joining/leaving the domain to get another ~10 minutes of functioning > winbind. > > With MAX_DEBUG=on, I tried to use gdb on the winbindd.core file which > shows lines like "#1627 0x0000000000000000 in ?? ()" and similar. I'm > guessing this is to do with other system libs that are not compiled > with -g, but I'm over my head here on FreeBSD system specifics. > > So I have two questions: > 1) Will a buildworld with debugging symbols get me more meaningful > output from gdb `backtrace full` and is this the best way? > > 2) Has anyone else experienced similar issues with net/samba36? I can > reproduce this bug on both 9.0-RC2 and 8.2-RELEASE-p3. > > Your advice is appreciated, Regarding MAX_DEBUG and trying to get a binary that's built with debugging support -- you are correct. Samba also has dependencies, as you're familiar with, and all of those would need to be built with debugging support as well. Usually when there's a core with a calling stack that's very long and all zeros, somewhere down at the bottom there is something remotely coherent. Keep in mind, however, that a software crash can sometimes completely smash the contents of the calling stack, which makes after-the-fact debugging very difficult. I would strongly advise opening up a ticket with the Samba folks, as well as posting this to freebsd-ports instead of freebsd-stable. The processes which are coring are purely Samba-specific. Remember, ports are third-party software, so starting there (with the Samba folks) would be good. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 20:53:02 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E83A1106566C; Fri, 16 Dec 2011 20:53:02 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 3FEA78FC14; Fri, 16 Dec 2011 20:53:02 +0000 (UTC) Received: by vcbfk1 with SMTP id fk1so4776964vcb.13 for ; Fri, 16 Dec 2011 12:53:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=SnNGMqmt8JyxMGwI1jG2Sv/EUHL3hiPmyhOItOIld94=; b=exooHyoQgBWEI5AAKnOy9HtZsO7MeyHsiM4ysou67p6r77MaXoDtPONr65kX0JwofV RG7Rdh5oUsYbtbpQtHGkodXURHxneQRVNYCgj18V4O8+CkAcwtTIShwNIgN2xo98SSwo piJieLhDmQRJVUoMdApQOck/qcCArhRSB6IPI= MIME-Version: 1.0 Received: by 10.220.231.73 with SMTP id jp9mr4894765vcb.50.1324068781728; Fri, 16 Dec 2011 12:53:01 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.52.26.50 with HTTP; Fri, 16 Dec 2011 12:53:01 -0800 (PST) In-Reply-To: <20111216152409.GA79938@onelab2.iet.unipi.it> References: <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215004205.GA11556@icarus.home.lan> <20111216081145.GA76297@onelab2.iet.unipi.it> <4EEB218B.1090209@freebsd.org> <20111216152409.GA79938@onelab2.iet.unipi.it> Date: Fri, 16 Dec 2011 12:53:01 -0800 X-Google-Sender-Auth: WO_stQudUO5wEr9XVIz1FtBzs_Y Message-ID: From: Adrian Chadd To: Luigi Rizzo Content-Type: text/plain; charset=ISO-8859-1 Cc: Tom Evans , "C. P. Ghost" , freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: switching schedulers (Re: SCHED_ULE should not be the default) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 20:53:03 -0000 Hi all, Can someone load a kernel module dynamically at boot-time? Ie, instead of compiling it in, can 4bsd/ule be loaded as a KLD at boot-time, so the user can just change by rebooting? That may be an acceptable solution for now. Adrian From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 21:39:02 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 0CA52106566C; Fri, 16 Dec 2011 21:39:02 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 5C11114F7DE; Fri, 16 Dec 2011 21:39:01 +0000 (UTC) Message-ID: <4EEBBA75.2090508@FreeBSD.org> Date: Fri, 16 Dec 2011 13:39:01 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Adrian Chadd References: <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215004205.GA11556@icarus.home.lan> <20111216081145.GA76297@onelab2.iet.unipi.it> <4EEB218B.1090209@freebsd.org> <20111216152409.GA79938@onelab2.iet.unipi.it> In-Reply-To: X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Tom Evans , "C. P. Ghost" , Luigi Rizzo , Jeremy Chadwick , freebsd-stable@freebsd.org Subject: Re: switching schedulers (Re: SCHED_ULE should not be the default) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 21:39:02 -0000 On 12/16/2011 12:53, Adrian Chadd wrote: > Hi all, > > Can someone load a kernel module dynamically at boot-time? > > Ie, instead of compiling it in, can 4bsd/ule be loaded as a KLD at > boot-time, so the user can just change by rebooting? > > That may be an acceptable solution for now. That, or a loader.conf tunable (which in the case of making them modules would basically amount to the same thing, right?). I've heard several really smart people with rather convincing explanations of why ULE is not the right choice for default for 2 cores or less. If we could ship one kernel with both schedulers available it should be simple to modify the installer to choose the right one and put the right stuff in loader.conf. Doug -- [^L] Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 21:40:56 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C48F106564A for ; Fri, 16 Dec 2011 21:40:56 +0000 (UTC) (envelope-from talon@lpthe.jussieu.fr) Received: from shiva.jussieu.fr (shiva.jussieu.fr [134.157.0.129]) by mx1.freebsd.org (Postfix) with ESMTP id B4F5F8FC23 for ; Fri, 16 Dec 2011 21:40:54 +0000 (UTC) Received: from parthe.lpthe.jussieu.fr (parthe.lpthe.jussieu.fr [134.157.10.1]) by shiva.jussieu.fr (8.14.4/jtpda-5.4) with ESMTP id pBGLeRJ0070501 for ; Fri, 16 Dec 2011 22:40:40 +0100 (CET) X-Ids: 165 Received: from [192.168.1.101] (sge91-2-82-227-32-26.fbx.proxad.net [82.227.32.26]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client did not present a certificate) by parthe.lpthe.jussieu.fr (Postfix) with ESMTPSA id A7E5F20C97 for ; Fri, 16 Dec 2011 22:40:26 +0100 (CET) From: Michel Talon Date: Fri, 16 Dec 2011 22:40:26 +0100 Message-Id: <1350C7A0-BE58-4C34-804A-A6A3C1C61761@lpthe.jussieu.fr> To: freebsd-stable@freebsd.org Mime-Version: 1.0 (Apple Message framework v1251.1) X-Mailer: Apple Mail (2.1251.1) X-Miltered: at jchkmail.jussieu.fr with ID 4EEBBACB.00A by Joe's j-chkmail (http : // j-chkmail dot ensmp dot fr)! X-j-chkmail-Enveloppe: 4EEBBACB.00A/134.157.10.1/parthe.lpthe.jussieu.fr/parthe.lpthe.jussieu.fr/ Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: switching schedulers (Re: SCHED_ULE should not be the default) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 21:40:56 -0000 Adrian Chadd said: > Hi all, >=20 > Can someone load a kernel module dynamically at boot-time? >=20 > Ie, instead of compiling it in, can 4bsd/ule be loaded as a KLD at > boot-time, so the user can just change by rebooting? >=20 > That may be an acceptable solution for now. As Luigi explained, the problem is not to have code for both schedulers = residing in the=20 kernel, the problem is to migrate processes from one scheduler to the = other. -- Michel Talon talon@lpthe.jussieu.fr From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 21:42:50 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A14DC106568F for ; Fri, 16 Dec 2011 21:42:50 +0000 (UTC) (envelope-from talon@lpthe.jussieu.fr) Received: from shiva.jussieu.fr (shiva.jussieu.fr [134.157.0.129]) by mx1.freebsd.org (Postfix) with ESMTP id 31A0C8FC20 for ; Fri, 16 Dec 2011 21:42:49 +0000 (UTC) Received: from parthe.lpthe.jussieu.fr (parthe.lpthe.jussieu.fr [134.157.10.1]) by shiva.jussieu.fr (8.14.4/jtpda-5.4) with ESMTP id pBGLQTVZ039260 ; Fri, 16 Dec 2011 22:26:29 +0100 (CET) X-Ids: 164 Received: from [192.168.1.101] (sge91-2-82-227-32-26.fbx.proxad.net [82.227.32.26]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client did not present a certificate) by parthe.lpthe.jussieu.fr (Postfix) with ESMTPSA id 0719F20C97; Fri, 16 Dec 2011 22:26:27 +0100 (CET) From: Michel Talon Date: Fri, 16 Dec 2011 22:26:27 +0100 Message-Id: <343FCC0E-C72D-4AE8-B730-5A3DE1046420@lpthe.jussieu.fr> To: freebsd-stable@freebsd.org Mime-Version: 1.0 (Apple Message framework v1251.1) X-Mailer: Apple Mail (2.1251.1) X-Miltered: at jchkmail.jussieu.fr with ID 4EEBB785.001 by Joe's j-chkmail (http : // j-chkmail dot ensmp dot fr)! X-j-chkmail-Enveloppe: 4EEBB785.001/134.157.10.1/parthe.lpthe.jussieu.fr/parthe.lpthe.jussieu.fr/ Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "O. Hartmann" Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 21:42:50 -0000 O Hartmann says: > For the underlying OS, as far as I know, the compiler hasn't as much > impact as on userland software since autovectorization and other neat > things are not used during system build. >=20 > =46rom my experience using gcc 4.2 or 4.4/4.5 does not have an impact > beyond 3% when SSE isn't explicetly enforced. >=20 > More interesting is the performance gain due to the architecture. I > think it would be very easy for M. Larabel to repeat this benchmark = with > a "bleeding edge" Ubuntu or Suse as well. And since FreeBSD 9.0 can = be > compiled with CLANG, it should be possible to compare both also with > "bleeding edge" compilers, say FreeBSD 9/CLANG, Ubuntu 12/gcc 4.6.2. My experience is that using gcc 4.6 gives *much* better performance than = using the obsolete gcc that is in FreeBSD and much better performance than clang. After all = you have to pay the price=20 for stupidities such as being GPL free. Or you can see it otherwise, you = can compete on the most GPL free system, or the best working system. As for the ZFS versus = ext3 performance, here also if you try to sell FreeBSD on features which are supposed to have = extraordinary benefits don't be surprised=20 when testers use these features and find horrendous performance issues. -- Michel Talon talon@lpthe.jussieu.fr From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 21:51:27 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id B8595106564A for ; Fri, 16 Dec 2011 21:51:27 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 3437D14F901; Fri, 16 Dec 2011 21:51:27 +0000 (UTC) Message-ID: <4EEBBD5E.50603@FreeBSD.org> Date: Fri, 16 Dec 2011 13:51:26 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Michel Talon References: <1350C7A0-BE58-4C34-804A-A6A3C1C61761@lpthe.jussieu.fr> In-Reply-To: <1350C7A0-BE58-4C34-804A-A6A3C1C61761@lpthe.jussieu.fr> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: switching schedulers (Re: SCHED_ULE should not be the default) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 21:51:27 -0000 On 12/16/2011 13:40, Michel Talon wrote: > Adrian Chadd said: > > >> Hi all, >> >> Can someone load a kernel module dynamically at boot-time? >> >> Ie, instead of compiling it in, can 4bsd/ule be loaded as a KLD at >> boot-time, so the user can just change by rebooting? >> >> That may be an acceptable solution for now. > > As Luigi explained, the problem is not to have code for both > schedulers residing in the kernel, the problem is to migrate > processes from one scheduler to the other. I think dynamically switching schedulers on a running system and loading one or the other at boot time are different problems, are they not? Doug -- [^L] Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 22:16:54 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3A53C1065673 for ; Fri, 16 Dec 2011 22:16:54 +0000 (UTC) (envelope-from talon@lpthe.jussieu.fr) Received: from shiva.jussieu.fr (shiva.jussieu.fr [134.157.0.129]) by mx1.freebsd.org (Postfix) with ESMTP id 9685C8FC16 for ; Fri, 16 Dec 2011 22:16:53 +0000 (UTC) Received: from parthe.lpthe.jussieu.fr (parthe.lpthe.jussieu.fr [134.157.10.1]) by shiva.jussieu.fr (8.14.4/jtpda-5.4) with ESMTP id pBGMGQMp075242 ; Fri, 16 Dec 2011 23:16:39 +0100 (CET) X-Ids: 165 Received: from [192.168.1.101] (sge91-2-82-227-32-26.fbx.proxad.net [82.227.32.26]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client did not present a certificate) by parthe.lpthe.jussieu.fr (Postfix) with ESMTPSA id 107FA1FEEA; Fri, 16 Dec 2011 23:16:24 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1251.1) From: Michel Talon In-Reply-To: <4EEBBD5E.50603@FreeBSD.org> Date: Fri, 16 Dec 2011 23:16:24 +0100 Message-Id: <91A8A86F-4D83-4C6F-8E27-B74204C6ACF9@lpthe.jussieu.fr> References: <1350C7A0-BE58-4C34-804A-A6A3C1C61761@lpthe.jussieu.fr> <4EEBBD5E.50603@FreeBSD.org> To: Doug Barton X-Mailer: Apple Mail (2.1251.1) X-Miltered: at jchkmail.jussieu.fr with ID 4EEBC33A.000 by Joe's j-chkmail (http : // j-chkmail dot ensmp dot fr)! X-j-chkmail-Enveloppe: 4EEBC33A.000/134.157.10.1/parthe.lpthe.jussieu.fr/parthe.lpthe.jussieu.fr/ Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@FreeBSD.org Subject: Re: switching schedulers (Re: SCHED_ULE should not be the default) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 22:16:54 -0000 Le 16 d=E9c. 2011 =E0 22:51, Doug Barton a =E9crit : > On 12/16/2011 13:40, Michel Talon wrote: >> Adrian Chadd said: >>=20 >>=20 >>> Hi all, >>>=20 >>> Can someone load a kernel module dynamically at boot-time? >>>=20 >>> Ie, instead of compiling it in, can 4bsd/ule be loaded as a KLD at=20= >>> boot-time, so the user can just change by rebooting? >>>=20 >>> That may be an acceptable solution for now. >>=20 >> As Luigi explained, the problem is not to have code for both >> schedulers residing in the kernel, the problem is to migrate >> processes from one scheduler to the other. >=20 > I think dynamically switching schedulers on a running system and = loading > one or the other at boot time are different problems, are they not? >=20 Of course, you are perfectly right., and i had misunderstood Adrian's = post. But if the problem is only to change scheduler by rebooting, i think it is no more expensive to compile a kernel with the other = scheduler. Or is it that people never compile kernels nowadays? The ability to switch scheduler on a = running machine would certainly be a more desirable way to test the best adaptation of = the system to the load. To come back to the problems in question about ULE i must say i don't = see obvious=20 malfunctions for my own use (i had some problems of this sort long ago, = but they disappeared with more recent FreeBSD). >=20 > Doug >=20 >=20 -- Michel Talon talon@lpthe.jussieu.fr From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 22:28:27 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 7E26C1065670 for ; Fri, 16 Dec 2011 22:28:27 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 254CA152490; Fri, 16 Dec 2011 22:27:42 +0000 (UTC) Message-ID: <4EEBC5DD.7080904@FreeBSD.org> Date: Fri, 16 Dec 2011 14:27:41 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Michel Talon References: <1350C7A0-BE58-4C34-804A-A6A3C1C61761@lpthe.jussieu.fr> <4EEBBD5E.50603@FreeBSD.org> <91A8A86F-4D83-4C6F-8E27-B74204C6ACF9@lpthe.jussieu.fr> In-Reply-To: <91A8A86F-4D83-4C6F-8E27-B74204C6ACF9@lpthe.jussieu.fr> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org Subject: Re: switching schedulers (Re: SCHED_ULE should not be the default) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 22:28:27 -0000 On 12/16/2011 14:16, Michel Talon wrote: > Of course, you are perfectly right., and i had misunderstood Adrian's > post. Happens to the best of us. :) > But if the problem is only to change scheduler by rebooting, i think > it is no more expensive to compile a kernel with the other scheduler. > Or is it that people never compile kernels nowadays? That's part of it. For my money the other 2 big problems are first that we'd like to make it as easy on the 'make release' and installer processes as possible. I imagine (although I would not object to being proven wrong) that 1 kernel with knobs is easier to manage and less resource intensive than 2 kernels that differ only by this 1 feature. The other big problem is freebsd-update. While I assume that logic could be built into the system to handle this issue, if the guts can be built into the kernel itself why not do that instead? Of lesser, but not insignificant consideration is the possibility that at some point we'll have more than 2 scheduler options. Doug -- [^L] Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 22:43:32 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34EFB1065673; Fri, 16 Dec 2011 22:43:32 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id E82128FC08; Fri, 16 Dec 2011 22:43:31 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 206387300A; Fri, 16 Dec 2011 23:59:54 +0100 (CET) Date: Fri, 16 Dec 2011 23:59:54 +0100 From: Luigi Rizzo To: Doug Barton Message-ID: <20111216225954.GA83026@onelab2.iet.unipi.it> References: <1350C7A0-BE58-4C34-804A-A6A3C1C61761@lpthe.jussieu.fr> <4EEBBD5E.50603@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EEBBD5E.50603@FreeBSD.org> User-Agent: Mutt/1.4.2.3i Cc: freebsd-stable@freebsd.org, Michel Talon Subject: Re: switching schedulers (Re: SCHED_ULE should not be the default) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 22:43:32 -0000 On Fri, Dec 16, 2011 at 01:51:26PM -0800, Doug Barton wrote: > On 12/16/2011 13:40, Michel Talon wrote: > > Adrian Chadd said: > > > > > >> Hi all, > >> > >> Can someone load a kernel module dynamically at boot-time? > >> > >> Ie, instead of compiling it in, can 4bsd/ule be loaded as a KLD at > >> boot-time, so the user can just change by rebooting? > >> > >> That may be an acceptable solution for now. > > > > As Luigi explained, the problem is not to have code for both > > schedulers residing in the kernel, the problem is to migrate > > processes from one scheduler to the other. > > I think dynamically switching schedulers on a running system and loading > one or the other at boot time are different problems, are they not? Runtime switching is a superset of loading as a module at boot time. In both cases you need to implement a generic interface between the scheduler and the rest of the system. The good thing, compared to 2002, is that now the abstraction exists, it is made by all functions and variables named sched_*() in sched_4bsd.c and sched_ule.c I see there is a small number of #ifdef SCHED_ULE in a couple of files, but probably it can be fixed. I believe all is needed for dynamic scheduler loading is to create function pointers for all these names, and initialize them when one of the scheduler modules is loaded. After that, runtime switching shouldn't require a lot of work either. The architecture and implementation i posted earlier (repeated below for convenience) should work, with just a bit of attention at locking the scheduler during a switch. References: http://kerneltrap.org/node/349 http://info.iet.unipi.it/~luigi/ps_sched.20020719a.diff It really looks much easier than i thought initially. cheers luigi From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 22:57:14 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B6B191065670; Fri, 16 Dec 2011 22:57:14 +0000 (UTC) (envelope-from crmartin@sgi.com) Received: from relay.sgi.com (relay3.sgi.com [192.48.152.1]) by mx1.freebsd.org (Postfix) with ESMTP id A0B6C8FC1A; Fri, 16 Dec 2011 22:57:14 +0000 (UTC) Received: from xmail.sgi.com (pv-excas2-dc21.corp.sgi.com [137.38.102.196]) by relay3.corp.sgi.com (Postfix) with ESMTP id C0EE9AC003; Fri, 16 Dec 2011 14:38:40 -0800 (PST) Received: from [10.3.0.220] (10.3.0.220) by xmail.sgi.com (137.38.102.30) with Microsoft SMTP Server (TLS) id 14.1.339.1; Fri, 16 Dec 2011 16:38:39 -0600 Message-ID: <4EEBC86F.1030907@sgi.com> Date: Fri, 16 Dec 2011 15:38:39 -0700 From: Charlie Martin Organization: Silicon Graphics, Inc. User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.24) Gecko/20111101 SUSE/3.1.16 Lightning/1.0b2 Thunderbird/3.1.16 MIME-Version: 1.0 To: Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.3.0.220] Cc: freebsd-current@FreeBSD.org, "Peter W. Morreale" Subject: Spinlock panic in FreeBSD 7 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 22:57:14 -0000 (This was originally posted to freebsd-hackers, I'm reposting following email suggestions.) We've observed a panic in FreeBSD 7 (7.2-PRERELEASE FreeBSD) several times that we've not been able to track down. Upgrading is not an option at this time. Does this look at all familiar to anyone? Here's an example stack trace after panic: spin lock 0xffffffff8086bdc0 (smp rendezvous) held by 0xffffff0006d1f000 (tid 100060) too long panic: spin lock held too long cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at 0xffffffff8019120a = db_trace_self_wrapper+0x2a panic() at 0xffffffff80308797 = panic+0x187 _mtx_lock_spin_failed() at 0xffffffff802fbda9 = _mtx_lock_spin_failed+0x39 _mtx_lock_spin() at 0xffffffff802fbe4e = _mtx_lock_spin+0x9e _mtx_lock_spin_flags() at 0xffffffff802fc354 = _mtx_lock_spin_flags+0x104 smp_rendezvous_cpus() at 0xffffffff80340cb3 = smp_rendezvous_cpus+0xd3 xcall() at 0xffffffff80ad755e = xcall+0x3e cyclic_remove_here() at 0xffffffff80ad7715 = cyclic_remove_here+0x1a5 cyclic_remove() at 0xffffffff80ad7a0f = cyclic_remove+0x5f profile_disable() at 0xffffffff80acf0e5 = profile_disable+0x15 dtrace_state_destroy() at 0xffffffff80adfabd = dtrace_state_destroy+0x35d dtrace_close() at 0xffffffff80adffed = dtrace_close+0x8d devfs_close() at 0xffffffff802a825d = devfs_close+0x2dd vn_close() at 0xffffffff8039cb06 = vn_close+0xb6 vn_closefile() at 0xffffffff8039cc00 = vn_closefile+0x80 devfs_close_f() at 0xffffffff802a5738 = devfs_close_f+0x28 fdrop() at 0xffffffff802d98bb = fdrop+0xdb closef() at 0xffffffff802db2f9 = closef+0x29 fdfree() at 0xffffffff802dc061 = fdfree+0x161 exit1() at 0xffffffff802e56b2 = exit1+0x2c2 sigexit() at 0xffffffff8030a86f = sigexit+0x8f postsig() at 0xffffffff8030b6ce = postsig+0x38e ast() at 0xffffffff803425f7 = ast+0x337 Xfast_syscall() at 0xffffffff80494efd = Xfast_syscall+0xdd --- syscall (32, FreeBSD ELF64, getsockname), rip = 0x800df4d5c, rsp = 0x7fffffffe398, rbp = 0x5 --- KDB: enter: panic The panic always shows up from a syscall, and almost always from syscall 32, getsockname, but we've also observed it with syscall 5. -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: CRMartin@sgi.com Website: www.sgi.com From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 23:47:14 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 380B110656D0 for ; Fri, 16 Dec 2011 23:47:14 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 625A8B2522; Fri, 16 Dec 2011 23:47:13 +0000 (UTC) Message-ID: <4EEBD880.6010408@FreeBSD.org> Date: Fri, 16 Dec 2011 15:47:12 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Luigi Rizzo References: <1350C7A0-BE58-4C34-804A-A6A3C1C61761@lpthe.jussieu.fr> <4EEBBD5E.50603@FreeBSD.org> <20111216225954.GA83026@onelab2.iet.unipi.it> In-Reply-To: <20111216225954.GA83026@onelab2.iet.unipi.it> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org, Michel Talon Subject: Re: switching schedulers (Re: SCHED_ULE should not be the default) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 23:47:14 -0000 On 12/16/2011 14:59, Luigi Rizzo wrote: > It really looks much easier than i thought initially. Awesome! -- [^L] Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Fri Dec 16 23:57:47 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 76074106566B for ; Fri, 16 Dec 2011 23:57:45 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 7D1FB14E0E4; Fri, 16 Dec 2011 23:57:44 +0000 (UTC) Message-ID: <4EEBDAF8.20200@FreeBSD.org> Date: Fri, 16 Dec 2011 15:57:44 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Christian Weisgerber References: In-Reply-To: X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: 7-STABLE: mergemaster tzsetup question X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 23:57:47 -0000 On 12/03/2011 07:24, Christian Weisgerber wrote: > Every time I run mergemaster(8) on 7.4-STABLE, I'm now presented > with > > *** There is no /var/db/zoneinfo file to update /etc/localtime. > You should run tzsetup > > Running tzsetup(8) does however not create /var/db/zoneinfo, so > mergemaster will prompt the next time, too. I guess I can just > ignore it, but it seems weird that mergemaster would keep nagging > about this. > > Where is /var/db/zoneinfo supposed to come from? > > I also notice that mergemaster can issue tzsetup arguments -C and > -r, but tzsetup doesn't support those. Once again, my apologies for assuming that my esteemed colleagues had done the responsible thing and MFC'ed their own work. I have resolved this issue by going back and doing 3 1/2 years of MFCs for tzsetup(8), which now makes it identical to the code in stable/8. If you update your src tree and then update tzsetup you should no longer experience this problem. Doug -- [^L] Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Sat Dec 17 10:57:05 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B8CA11065670 for ; Sat, 17 Dec 2011 10:57:05 +0000 (UTC) (envelope-from randy@psg.com) Received: from ran.psg.com (ran.psg.com [IPv6:2001:418:1::36]) by mx1.freebsd.org (Postfix) with ESMTP id 9E3868FC0A for ; Sat, 17 Dec 2011 10:57:05 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=rair.psg.com.psg.com) by ran.psg.com with esmtp (Exim 4.76 (FreeBSD)) (envelope-from ) id 1Rbrwv-000Gfw-0r for freebsd-stable@freebsd.org; Sat, 17 Dec 2011 10:57:05 +0000 Date: Sat, 17 Dec 2011 02:57:04 -0800 Message-ID: From: Randy Bush To: FreeBSD Stable User-Agent: Wanderlust/2.15.9 (Almost Unreal) Emacs/22.3 Mule/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Subject: 8.2->9.prerel: gmirror failed with error 19 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 10:57:05 -0000 8.2 system fully updated as of 2011.12.14 it can reboot quite happily csup to RELENG_9 make buildworld make kernel boot single root mount waiting for: usbus4 uhub4: 8 ports with 8 removable, self powered Trying to mount root from ufs:/dev/mirror/boota [rw]... mountroot: waiting for device /dev/mirror/boota ... Mounting from ufs:/dev/mirror/boota failed with error 19. neither 9 nor 8 would boot without ending up here the only way out was via loader OK unload OK load boot/kernel.old/kernel OK load boot/kernel.old/geom_mirror.ko OK set kern.geom.part.check_integrity=0 OK set vfs.root.mountfrom.options=rw OK boot -s this would only work with the 8.2 /boot/kernel.old but not with the 9.fresh /boot/kernel randy From owner-freebsd-stable@FreeBSD.ORG Sat Dec 17 11:03:38 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B3F2A106564A; Sat, 17 Dec 2011 11:03:38 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id CD07E8FC0C; Sat, 17 Dec 2011 11:03:37 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA02947; Sat, 17 Dec 2011 13:03:31 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Rbs38-0009Xs-Qc; Sat, 17 Dec 2011 13:03:30 +0200 Message-ID: <4EEC7700.3020700@FreeBSD.org> Date: Sat, 17 Dec 2011 13:03:28 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111206 Thunderbird/8.0 MIME-Version: 1.0 To: Charlie Martin References: <4EEBC86F.1030907@sgi.com> In-Reply-To: <4EEBC86F.1030907@sgi.com> X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-current@FreeBSD.org, freebsd-stable@FreeBSD.org, "Peter W. Morreale" Subject: Re: Spinlock panic in FreeBSD 7 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 11:03:38 -0000 on 17/12/2011 00:38 Charlie Martin said the following: > (This was originally posted to freebsd-hackers, I'm reposting following email > suggestions.) > > We've observed a panic in FreeBSD 7 (7.2-PRERELEASE FreeBSD) several times that > we've not been able to track down. Upgrading is not an option at this time. > > Does this look at all familiar to anyone? Here's an example stack trace after > panic: See r219502. > spin lock 0xffffffff8086bdc0 (smp rendezvous) held by 0xffffff0006d1f000 (tid > 100060) too long > panic: spin lock held too long > cpuid = 0 > KDB: stack backtrace: > db_trace_self_wrapper() at 0xffffffff8019120a = db_trace_self_wrapper+0x2a > panic() at 0xffffffff80308797 = panic+0x187 > _mtx_lock_spin_failed() at 0xffffffff802fbda9 = _mtx_lock_spin_failed+0x39 > _mtx_lock_spin() at 0xffffffff802fbe4e = _mtx_lock_spin+0x9e > _mtx_lock_spin_flags() at 0xffffffff802fc354 = _mtx_lock_spin_flags+0x104 > smp_rendezvous_cpus() at 0xffffffff80340cb3 = smp_rendezvous_cpus+0xd3 > xcall() at 0xffffffff80ad755e = xcall+0x3e > cyclic_remove_here() at 0xffffffff80ad7715 = cyclic_remove_here+0x1a5 > cyclic_remove() at 0xffffffff80ad7a0f = cyclic_remove+0x5f > profile_disable() at 0xffffffff80acf0e5 = profile_disable+0x15 > dtrace_state_destroy() at 0xffffffff80adfabd = dtrace_state_destroy+0x35d > dtrace_close() at 0xffffffff80adffed = dtrace_close+0x8d > devfs_close() at 0xffffffff802a825d = devfs_close+0x2dd > vn_close() at 0xffffffff8039cb06 = vn_close+0xb6 > vn_closefile() at 0xffffffff8039cc00 = vn_closefile+0x80 > devfs_close_f() at 0xffffffff802a5738 = devfs_close_f+0x28 > fdrop() at 0xffffffff802d98bb = fdrop+0xdb > closef() at 0xffffffff802db2f9 = closef+0x29 > fdfree() at 0xffffffff802dc061 = fdfree+0x161 > exit1() at 0xffffffff802e56b2 = exit1+0x2c2 > sigexit() at 0xffffffff8030a86f = sigexit+0x8f > postsig() at 0xffffffff8030b6ce = postsig+0x38e > ast() at 0xffffffff803425f7 = ast+0x337 > Xfast_syscall() at 0xffffffff80494efd = Xfast_syscall+0xdd > --- syscall (32, FreeBSD ELF64, getsockname), rip = 0x800df4d5c, rsp = > 0x7fffffffe398, rbp = 0x5 --- > KDB: enter: panic > > The panic always shows up from a syscall, and almost always from syscall 32, > getsockname, but we've also observed it with syscall 5. -- Andriy Gapon From owner-freebsd-stable@FreeBSD.ORG Sat Dec 17 11:22:12 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 02358106566B for ; Sat, 17 Dec 2011 11:22:12 +0000 (UTC) (envelope-from ae@FreeBSD.org) Received: from [127.0.0.1] (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 176191501F6; Sat, 17 Dec 2011 11:22:10 +0000 (UTC) Message-ID: <4EEC7B54.4070407@FreeBSD.org> Date: Sat, 17 Dec 2011 15:21:56 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111203 Thunderbird/8.0 MIME-Version: 1.0 To: Randy Bush References: In-Reply-To: X-Enigmail-Version: 1.1.2 OpenPGP: id=10C8A17A Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig34EFCDDAF9FB6617372E58C3" Cc: FreeBSD Stable Subject: Re: 8.2->9.prerel: gmirror failed with error 19 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 11:22:12 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig34EFCDDAF9FB6617372E58C3 Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: quoted-printable On 17.12.2011 14:57, Randy Bush wrote: > neither 9 nor 8 would boot without ending up here > the only way out was via loader >=20 > OK unload > OK load boot/kernel.old/kernel > OK load boot/kernel.old/geom_mirror.ko > OK set kern.geom.part.check_integrity=3D0 > OK set vfs.root.mountfrom.options=3Drw > OK boot -s >=20 > this would only work with the 8.2 /boot/kernel.old but not with the > 9.fresh /boot/kernel You should try boot in verbose mode and, probably, you will see the cause of your problem. --=20 WBR, Andrey V. Elsukov --------------enig34EFCDDAF9FB6617372E58C3 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iQEcBAEBAgAGBQJO7HtaAAoJEAHF6gQQyKF6cBYH/AlX62z4/+7dkjnbcBx2GTvJ 6V8m/1J2W0Q3gqx18u8N+naeAruIyiz2mIs7V2EadDyAIZEMFW0zdsTpOlypiBx/ ebF4CAIrdjwoMEEKiKIla5IgHW/Yu+7gwYzKCK3SeEWVOAuEXDmgLcTS0s5T5cut UIeqY30KND+LmsPw0qntHxcXSPEilXLt5jCsHPTpBfMpjDCIXT9ls2+h2VJeqbYE Qlx1K2Nu52+aZAf5CkOfXWxJjVGqWa5l5EY7J512q5wY233HCg6uhxxVhrsQCo5q AahLoIdJsMSwxTLdDfS+FB6Ff563nLIDxHQj+dkIfjpWPaFAN7/nk9Q5zRfMwtw= =ADJI -----END PGP SIGNATURE----- --------------enig34EFCDDAF9FB6617372E58C3-- From owner-freebsd-stable@FreeBSD.ORG Sat Dec 17 15:31:00 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4427E106566C; Sat, 17 Dec 2011 15:31:00 +0000 (UTC) (envelope-from randy@psg.com) Received: from ran.psg.com (ran.psg.com [IPv6:2001:418:1::36]) by mx1.freebsd.org (Postfix) with ESMTP id E24968FC16; Sat, 17 Dec 2011 15:30:59 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=rair.psg.com.psg.com) by ran.psg.com with esmtp (Exim 4.76 (FreeBSD)) (envelope-from ) id 1RbwDz-000H4Y-CC; Sat, 17 Dec 2011 15:30:59 +0000 Date: Sat, 17 Dec 2011 07:30:59 -0800 Message-ID: From: Randy Bush To: "Andrey V. Elsukov" In-Reply-To: <4EEC7B54.4070407@FreeBSD.org> References: <4EEC7B54.4070407@FreeBSD.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) Emacs/22.3 Mule/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Cc: FreeBSD Stable Subject: Re: 8.2->9.prerel: gmirror failed with error 19 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 15:31:00 -0000 >> neither 9 nor 8 would boot without ending up here >> the only way out was via loader >> >> OK unload >> OK load boot/kernel.old/kernel >> OK load boot/kernel.old/geom_mirror.ko >> OK set kern.geom.part.check_integrity=0 >> OK set vfs.root.mountfrom.options=rw >> OK boot -s >> >> this would only work with the 8.2 /boot/kernel.old but not with the >> 9.fresh /boot/kernel > > You should try boot in verbose mode and, probably, you will see the > cause of your problem. sorry, there was nothing tasty i could see randy OK lsmod 0x200000: /boot/kernel/kernel (elf kernel, 0x8f6af8) modules: x86bios.1 io.1 ufs.1 kernel_mac_support.4 if_vlan.3 ether.1 sysvshm.1 sysvsem.1 sysvmsg.1 firmware.1 kernel.900044 cd9660.1 isa.1 pseudofs.1 procfs.1 usb_quirk.1 ukbd.1 uhid.1 uhub.1 usb.1 usb_linux.1 umass.1 random.1 pci.1 null.1 mem.1 ata_via.1 ata_sis.1 ata_sii.1 ata_serverworks.1 ata_promise.1 ata_nvidia.1 ata_netcell.1 ata_national.1 ata_micron.1 ata_marvell.1 ata_jmicron.1 ata_ite.1 ata_intel.1 ata_highpoint.1 ata_cyrix.1 ata_cypress.1 ata_cenatek.1 ata_ati.1 ata_amd.1 ata_adaptec.1 ata_ali.1 ata_acard.1 ata_ahci.1 atapci.1 ata.1 acpi_pci.1 acpi.1 cam.1 0xaf7000: /boot/kernel/geom_mirror.ko (elf obj module, 0x215f0) 0xb185f0: /boot/zfs/zpool.cache (/boot/zfs/zpool.cache, 0x7a4) OK boot -v -s SMAP type=01 base=0000000000000000 len=000000000009d000 SMAP type=02 base=000000000009d000 len=0000000000003000 SMAP type=02 base=00000000000e4000 len=000000000001c000 SMAP type=01 base=0000000000100000 len=00000000dfde0000 SMAP type=03 base=00000000dfee0000 len=0000000000009000 SMAP type=04 base=00000000dfee9000 len=0000000000017000 SMAP type=02 base=00000000dff00000 len=0000000000100000 SMAP type=02 base=00000000f0000000 len=0000000004000000 SMAP type=02 base=00000000fec00000 len=0000000000010000 SMAP type=02 base=00000000fee00000 len=0000000000001000 SMAP type=02 base=00000000ff000000 len=0000000001000000 SMAP type=01 base=0000000100000000 len=0000000020000000 Table 'FACP' at 0xdfee8e51 Table 'MCFG' at 0xdfee8ec5 Table 'APIC' at 0xdfee8f01 APIC: Found table at 0xdfee8f01 APIC: Using the MADT enumerator. MADT: Found CPU APIC ID 0 ACPI ID 0: enabled SMP: Added CPU 0 (AP) MADT: Found CPU APIC ID 1 ACPI ID 1: enabled SMP: Added CPU 1 (AP) Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-PRERELEASE #18: Tue Dec 13 12:20:57 GMT 2011 root@work0.psg.com:/usr/obj/usr/src/sys/WORK0 amd64 Table 'FACP' at 0xdfee8e51 Table 'MCFG' at 0xdfee8ec5 Table 'APIC' at 0xdfee8f01 Table 'BOOT' at 0xdfee8f75 Table 'ASF!' at 0xdfee8f9d Table 'SSDT' at 0xdfee25e0 ACPI: No SRAT table found Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff80b1a000. Preloaded elf obj module "/boot/kernel/geom_mirror.ko" at 0xffffffff80b1a208. Preloaded /boot/zfs/zpool.cache "/boot/zfs/zpool.cache" at 0xffffffff80b1a8b8. Calibrating TSC clock ... TSC clock: 1995039920 Hz CPU: Intel(R) Pentium(R) Dual CPU E2180 @ 2.00GHz (1995.04-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6fd Family = 6 Model = f Stepping = 13 Features=0xbfebfbff Features2=0xe39d AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 4294967296 (4096 MB) Physical memory chunk(s): 0x0000000000001000 - 0x0000000000098fff, 622592 bytes (152 pages) 0x0000000000100000 - 0x00000000001fffff, 1048576 bytes (256 pages) 0x0000000000b4a000 - 0x00000000d7757fff, 3602964480 bytes (879630 pages) 0x0000000100000000 - 0x000000011ffeffff, 536805376 bytes (131056 pages) avail memory = 4107489280 (3917 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: INTR: Adding local APIC 1 as a target FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 x86bios: IVT 0x000000-0x0004ff at 0xfffffe0000000000 x86bios: SSEG 0x001000-0x001fff at 0xffffff8000215000 x86bios: EBDA 0x09d000-0x09ffff at 0xfffffe000009d000 x86bios: ROM 0x0a0000-0x0fefff at 0xfffffe00000a0000 APIC: CPU 0 has ACPI ID 0 APIC: CPU 1 has ACPI ID 1 ULE: setup cpu 0 ULE: setup cpu 1 ACPI: RSDP 0xf5e20 00014 (v00 PTLTD ) ACPI: RSDT 0xdfee25a4 0003C (v01 PTLTD RSDT 06040000 LTP 00000000) ACPI: FACP 0xdfee8e51 00074 (v01 INTEL 06040000 PTL 00000003) ACPI: DSDT 0xdfee39cc 05485 (v01 INTEL GLENWOOD 06040000 MSFT 0100000E) ACPI: FACS 0xdfee9fc0 00040 ACPI: MCFG 0xdfee8ec5 0003C (v01 PTLTD MCFG 06040000 LTP 00000000) ACPI: APIC 0xdfee8f01 00074 (v01 PTLTD ? APIC 06040000 LTP 00000000) ACPI: BOOT 0xdfee8f75 00028 (v01 PTLTD $SBFTBL$ 06040000 LTP 00000001) ACPI: ASF! 0xdfee8f9d 00063 (v32 CETP CETP 06040000 PTL 00000001) ACPI: SSDT 0xdfee25e0 013EC (v01 PmRef CpuPm 00003000 INTL 20050228) MADT: Found IO APIC ID 2, Interrupt 0 at 0xfec00000 ioapic0: Routing external 8259A's -> intpin 0 MADT: Found IO APIC ID 3, Interrupt 24 at 0xfec10000 MADT: Interrupt override: source 0, irq 2 ioapic0: Routing IRQ 0 -> intpin 2 MADT: Interrupt override: source 9, irq 9 ioapic0: intpin 9 trigger: level lapic0: Routing NMI -> LINT1 lapic0: LINT1 trigger: edge lapic0: LINT1 polarity: high lapic1: Routing NMI -> LINT1 lapic1: LINT1 trigger: edge lapic1: LINT1 polarity: high ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard cpu0 BSP: ID: 0x00000000 VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00000200 err: 0x000000f0 pmc: 0x00010400 io: mem: null: random: acpi0: on motherboard PCIe: Memory Mapped configuration base @ 0xf0000000 ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48 acpi0: Power Button (fixed) ACPI timer: 1/1 1/1 1/1 1/1 1/1 1/1 1/1 1/1 1/1 0/1 -> 9 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: on acpi0 cpu0: switching to generic Cx mode cpu1: on acpi0 pci_link0: Index IRQ Rtd Ref IRQs Initial Probe 0 7 N 0 3 10 11 14 15 Validation 0 255 N 0 3 10 11 14 15 After Disable 0 255 N 0 3 10 11 14 15 pci_link1: Index IRQ Rtd Ref IRQs Initial Probe 0 11 N 0 3 10 11 14 15 Validation 0 11 N 0 3 10 11 14 15 After Disable 0 255 N 0 3 10 11 14 15 pci_link2: Index IRQ Rtd Ref IRQs Initial Probe 0 5 N 0 3 10 11 14 15 Validation 0 255 N 0 3 10 11 14 15 After Disable 0 255 N 0 3 10 11 14 15 pci_link3: Index IRQ Rtd Ref IRQs Initial Probe 0 11 N 0 3 10 11 14 15 Validation 0 11 N 0 3 10 11 14 15 After Disable 0 255 N 0 3 10 11 14 15 pci_link4: Index IRQ Rtd Ref IRQs Initial Probe 0 255 N 0 3 10 11 14 15 Validation 0 255 N 0 3 10 11 14 15 After Disable 0 255 N 0 3 10 11 14 15 pci_link5: Index IRQ Rtd Ref IRQs Initial Probe 0 255 N 0 3 10 11 14 15 Validation 0 255 N 0 3 10 11 14 15 After Disable 0 255 N 0 3 10 11 14 15 pci_link6: Index IRQ Rtd Ref IRQs Initial Probe 0 255 N 0 3 10 11 14 15 Validation 0 255 N 0 3 10 11 14 15 After Disable 0 255 N 0 3 10 11 14 15 pci_link7: Index IRQ Rtd Ref IRQs Initial Probe 0 10 N 0 3 10 11 14 15 Validation 0 10 N 0 3 10 11 14 15 After Disable 0 255 N 0 3 10 11 14 15 pcib0: port 0xcf8-0xcff on acpi0 pcib0: decoding 4 range 0-0xcf7 pcib0: decoding 3 range 0xa0000-0xbffff pcib0: decoding 3 range 0xcc000-0xcffff pcib0: decoding 3 range 0xd0000-0xd3fff pcib0: decoding 3 range 0xd4000-0xd7fff pcib0: decoding 3 range 0xd8000-0xdbfff pcib0: decoding 3 range 0xdc000-0xdffff pcib0: decoding 3 range 0xe0000-0xe3fff pcib0: decoding 3 range 0xe0000000-0xefffffff pcib0: decoding 4 range 0xd00-0xfdff pci0: on pcib0 pci0: domain=0, physical bus=0 found-> vendor=0x8086, dev=0x2778, revid=0xc0 domain=0, bus=0, slot=0, func=0 class=06-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0106, statreg=0x2090, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) found-> vendor=0x8086, dev=0x2779, revid=0xc0 domain=0, bus=0, slot=1, func=0 class=06-04-00, hdrtype=0x01, mfdev=0 cmdreg=0x0107, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x45 (17250 ns), maxlat=0x00 (0 ns) intpin=a, irq=7 powerspec 2 supports D0 D3 current D0 MSI supports 1 message pcib0: matched entry for 0.1.INTA pcib0: slot 1 INTA hardwired to IRQ 16 found-> vendor=0x8086, dev=0x27d0, revid=0x01 domain=0, bus=0, slot=28, func=0 class=06-04-00, hdrtype=0x01, mfdev=1 cmdreg=0x0007, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x04 (1000 ns), maxlat=0x00 (0 ns) intpin=a, irq=11 powerspec 2 supports D0 D3 current D0 MSI supports 1 message pcib0: matched entry for 0.28.INTA pcib0: slot 28 INTA hardwired to IRQ 17 found-> vendor=0x8086, dev=0x27e0, revid=0x01 domain=0, bus=0, slot=28, func=4 class=06-04-00, hdrtype=0x01, mfdev=1 cmdreg=0x0007, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x04 (1000 ns), maxlat=0x00 (0 ns) intpin=a, irq=11 powerspec 2 supports D0 D3 current D0 MSI supports 1 message pcib0: matched entry for 0.28.INTA pcib0: slot 28 INTA hardwired to IRQ 17 found-> vendor=0x8086, dev=0x27e2, revid=0x01 domain=0, bus=0, slot=28, func=5 class=06-04-00, hdrtype=0x01, mfdev=1 cmdreg=0x0007, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x04 (1000 ns), maxlat=0x00 (0 ns) intpin=b, irq=7 powerspec 2 supports D0 D3 current D0 MSI supports 1 message pcib0: matched entry for 0.28.INTB pcib0: slot 28 INTB hardwired to IRQ 16 found-> vendor=0x8086, dev=0x27c8, revid=0x01 domain=0, bus=0, slot=29, func=0 class=0c-03-00, hdrtype=0x00, mfdev=1 cmdreg=0x0005, statreg=0x0280, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=10 map[20]: type I/O Port, range 32, base 0x3000, size 5, enabled pcib0: allocated type 4 (0x3000-0x301f) for rid 20 of pci0:0:29:0 pcib0: matched entry for 0.29.INTA pcib0: slot 29 INTA hardwired to IRQ 23 found-> vendor=0x8086, dev=0x27c9, revid=0x01 domain=0, bus=0, slot=29, func=1 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0280, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=b, irq=11 map[20]: type I/O Port, range 32, base 0x3020, size 5, enabled pcib0: allocated type 4 (0x3020-0x303f) for rid 20 of pci0:0:29:1 pcib0: matched entry for 0.29.INTB pcib0: slot 29 INTB hardwired to IRQ 19 found-> vendor=0x8086, dev=0x27ca, revid=0x01 domain=0, bus=0, slot=29, func=2 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0280, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=c, irq=5 map[20]: type I/O Port, range 32, base 0x3040, size 5, enabled pcib0: allocated type 4 (0x3040-0x305f) for rid 20 of pci0:0:29:2 pcib0: matched entry for 0.29.INTC pcib0: slot 29 INTC hardwired to IRQ 18 found-> vendor=0x8086, dev=0x27cb, revid=0x01 domain=0, bus=0, slot=29, func=3 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0280, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=d, irq=7 map[20]: type I/O Port, range 32, base 0x3060, size 5, enabled pcib0: allocated type 4 (0x3060-0x307f) for rid 20 of pci0:0:29:3 pcib0: matched entry for 0.29.INTD pcib0: slot 29 INTD hardwired to IRQ 16 found-> vendor=0x8086, dev=0x27cc, revid=0x01 domain=0, bus=0, slot=29, func=7 class=0c-03-20, hdrtype=0x00, mfdev=0 cmdreg=0x0106, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=10 powerspec 2 supports D0 D3 current D0 map[10]: type Memory, range 32, base 0xe8600000, size 10, enabled pcib0: allocated type 3 (0xe8600000-0xe86003ff) for rid 10 of pci0:0:29:7 pcib0: matched entry for 0.29.INTA pcib0: slot 29 INTA hardwired to IRQ 23 found-> vendor=0x8086, dev=0x244e, revid=0xe1 domain=0, bus=0, slot=30, func=0 class=06-04-01, hdrtype=0x01, mfdev=0 cmdreg=0x0107, statreg=0x0010, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x0c (3000 ns), maxlat=0x00 (0 ns) found-> vendor=0x8086, dev=0x27b8, revid=0x01 domain=0, bus=0, slot=31, func=0 class=06-01-00, hdrtype=0x00, mfdev=1 cmdreg=0x0007, statreg=0x0210, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) found-> vendor=0x8086, dev=0x27df, revid=0x01 domain=0, bus=0, slot=31, func=1 class=01-01-8a, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0280, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=255 pcib0: allocated type 4 (0x1f0-0x1f7) for rid 10 of pci0:0:31:1 pcib0: allocated type 4 (0x3f6-0x3f6) for rid 14 of pci0:0:31:1 pcib0: allocated type 4 (0x170-0x177) for rid 18 of pci0:0:31:1 pcib0: allocated type 4 (0x376-0x376) for rid 1c of pci0:0:31:1 map[20]: type I/O Port, range 32, base 0x30a0, size 4, enabled pcib0: allocated type 4 (0x30a0-0x30af) for rid 20 of pci0:0:31:1 found-> vendor=0x8086, dev=0x27c1, revid=0x01 domain=0, bus=0, slot=31, func=2 class=01-06-01, hdrtype=0x00, mfdev=0 cmdreg=0x0007, statreg=0x02b0, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=b, irq=11 powerspec 2 supports D0 D3 current D0 MSI supports 1 message map[10]: type I/O Port, range 32, base 0x30e8, size 3, enabled pcib0: allocated type 4 (0x30e8-0x30ef) for rid 10 of pci0:0:31:2 map[14]: type I/O Port, range 32, base 0x30dc, size 2, enabled pcib0: allocated type 4 (0x30dc-0x30df) for rid 14 of pci0:0:31:2 map[18]: type I/O Port, range 32, base 0x30e0, size 3, enabled pcib0: allocated type 4 (0x30e0-0x30e7) for rid 18 of pci0:0:31:2 map[1c]: type I/O Port, range 32, base 0x30d8, size 2, enabled pcib0: allocated type 4 (0x30d8-0x30db) for rid 1c of pci0:0:31:2 map[20]: type I/O Port, range 32, base 0x30b0, size 4, enabled pcib0: allocated type 4 (0x30b0-0x30bf) for rid 20 of pci0:0:31:2 map[24]: type Memory, range 32, base 0xe8600400, size 10, enabled pcib0: allocated type 3 (0xe8600400-0xe86007ff) for rid 24 of pci0:0:31:2 pcib0: matched entry for 0.31.INTB pcib0: slot 31 INTB hardwired to IRQ 19 found-> vendor=0x8086, dev=0x27da, revid=0x01 domain=0, bus=0, slot=31, func=3 class=0c-05-00, hdrtype=0x00, mfdev=0 cmdreg=0x0101, statreg=0x0280, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=b, irq=11 map[20]: type I/O Port, range 32, base 0x1100, size 5, enabled pcib0: allocated type 4 (0x1100-0x111f) for rid 20 of pci0:0:31:3 pcib0: matched entry for 0.31.INTB pcib0: slot 31 INTB hardwired to IRQ 19 pcib1: irq 16 at device 1.0 on pci0 pcib1: domain 0 pcib1: secondary bus 1 pcib1: subordinate bus 1 pcib1: no prefetched decode pci1: on pcib1 pci1: domain=0, physical bus=1 pcib2: irq 17 at device 28.0 on pci0 pcib0: allocated type 3 (0xe8100000-0xe81fffff) for rid 20 of pcib2 pcib2: domain 0 pcib2: secondary bus 9 pcib2: subordinate bus 10 pcib2: memory decode 0xe8100000-0xe81fffff pcib2: no prefetched decode pci9: on pcib2 pci9: domain=0, physical bus=9 found-> vendor=0x8086, dev=0x032c, revid=0x09 domain=0, bus=9, slot=0, func=0 class=06-04-00, hdrtype=0x01, mfdev=1 cmdreg=0x0107, statreg=0x0010, cachelnsz=16 (dwords) lattimer=0x00 (0 ns), mingnt=0x04 (1000 ns), maxlat=0x00 (0 ns) powerspec 2 supports D0 D3 current D0 MSI supports 1 message, 64 bit found-> vendor=0x8086, dev=0x0326, revid=0x09 domain=0, bus=9, slot=0, func=1 class=08-00-20, hdrtype=0x00, mfdev=1 cmdreg=0x0106, statreg=0x0010, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) powerspec 2 supports D0 D3 current D0 map[10]: type Memory, range 32, base 0xe8100000, size 12, enabled pcib2: allocated memory range (0xe8100000-0xe8100fff) for rid 10 of pci0:9:0:1 pcib3: at device 0.0 on pci9 pcib3: domain 0 pcib3: secondary bus 10 pcib3: subordinate bus 10 pcib3: no prefetched decode pci10: on pcib3 pci10: domain=0, physical bus=10 pcib4: irq 17 at device 28.4 on pci0 pcib0: allocated type 4 (0x4000-0x4fff) for rid 1c of pcib4 pcib0: allocated type 3 (0xe8000000-0xe80fffff) for rid 20 of pcib4 pcib4: domain 0 pcib4: secondary bus 13 pcib4: subordinate bus 13 pcib4: I/O decode 0x4000-0x4fff pcib4: memory decode 0xe8000000-0xe80fffff pcib4: no prefetched decode pci13: on pcib4 pci13: domain=0, physical bus=13 found-> vendor=0x8086, dev=0x108c, revid=0x03 domain=0, bus=13, slot=0, func=0 class=02-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0107, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=7 powerspec 2 supports D0 D3 current D0 MSI supports 1 message, 64 bit map[10]: type Memory, range 32, base 0xe8000000, size 17, enabled pcib4: allocated memory range (0xe8000000-0xe801ffff) for rid 10 of pci0:13:0:0 map[18]: type I/O Port, range 32, base 0x4000, size 5, enabled pcib4: allocated I/O port range (0x4000-0x401f) for rid 18 of pci0:13:0:0 pcib4: matched entry for 13.0.INTA pcib4: slot 0 INTA hardwired to IRQ 16 em0: port 0x4000-0x401f mem 0xe8000000-0xe801ffff irq 16 at device 0.0 on pci13 em0: attempting to allocate 1 MSI vectors (1 supported) msi: routing MSI IRQ 256 to local APIC 0 vector 49 em0: using IRQ 256 for MSI em0: Using an MSI interrupt em0: bpf attached em0: Ethernet address: 00:30:48:91:d7:08 pcib5: irq 16 at device 28.5 on pci0 pcib0: allocated type 4 (0x5000-0x5fff) for rid 1c of pcib5 pcib0: allocated type 3 (0xe8200000-0xe82fffff) for rid 20 of pcib5 pcib5: domain 0 pcib5: secondary bus 14 pcib5: subordinate bus 14 pcib5: I/O decode 0x5000-0x5fff pcib5: memory decode 0xe8200000-0xe82fffff pcib5: no prefetched decode pci14: on pcib5 pci14: domain=0, physical bus=14 found-> vendor=0x8086, dev=0x109a, revid=0x00 domain=0, bus=14, slot=0, func=0 class=02-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0107, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=11 powerspec 2 supports D0 D3 current D0 MSI supports 1 message, 64 bit map[10]: type Memory, range 32, base 0xe8200000, size 17, enabled pcib5: allocated memory range (0xe8200000-0xe821ffff) for rid 10 of pci0:14:0:0 map[18]: type I/O Port, range 32, base 0x5000, size 5, enabled pcib5: allocated I/O port range (0x5000-0x501f) for rid 18 of pci0:14:0:0 pcib5: matched entry for 14.0.INTA pcib5: slot 0 INTA hardwired to IRQ 17 em1: port 0x5000-0x501f mem 0xe8200000-0xe821ffff irq 17 at device 0.0 on pci14 em1: attempting to allocate 1 MSI vectors (1 supported) msi: routing MSI IRQ 257 to local APIC 0 vector 50 em1: using IRQ 257 for MSI em1: Using an MSI interrupt em1: bpf attached em1: Ethernet address: 00:30:48:91:d7:09 uhci0: port 0x3000-0x301f irq 23 at device 29.0 on pci0 ioapic0: routing intpin 23 (PCI IRQ 23) to lapic 0 vector 51 usbus0: on uhci0 usbus0: bpf attached uhci0: usbpf: Attached uhci1: port 0x3020-0x303f irq 19 at device 29.1 on pci0 ioapic0: routing intpin 19 (PCI IRQ 19) to lapic 0 vector 52 usbus1: on uhci1 usbus1: bpf attached uhci1: usbpf: Attached uhci2: port 0x3040-0x305f irq 18 at device 29.2 on pci0 ioapic0: routing intpin 18 (PCI IRQ 18) to lapic 0 vector 53 usbus2: on uhci2 usbus2: bpf attached uhci2: usbpf: Attached uhci3: port 0x3060-0x307f irq 16 at device 29.3 on pci0 ioapic0: routing intpin 16 (PCI IRQ 16) to lapic 0 vector 54 usbus3: on uhci3 usbus3: bpf attached uhci3: usbpf: Attached ehci0: mem 0xe8600000-0xe86003ff irq 23 at device 29.7 on pci0 usbus4: EHCI version 1.0 usbus4: on ehci0 usbus4: bpf attached ehci0: usbpf: Attached pcib6: at device 30.0 on pci0 pcib0: allocated type 4 (0x6000-0x6fff) for rid 1c of pcib6 pcib0: allocated type 3 (0xe8300000-0xe83fffff) for rid 20 of pcib6 pcib0: allocated type 3 (0xe0000000-0xe7ffffff) for rid 24 of pcib6 pcib6: domain 0 pcib6: secondary bus 15 pcib6: subordinate bus 15 pcib6: I/O decode 0x6000-0x6fff pcib6: memory decode 0xe8300000-0xe83fffff pcib6: prefetched decode 0xe0000000-0xe7ffffff pcib6: Subtractively decoded bridge. pci15: on pcib6 pci15: domain=0, physical bus=15 found-> vendor=0x1002, dev=0x515e, revid=0x02 domain=0, bus=15, slot=0, func=0 class=03-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0387, statreg=0x0290, cachelnsz=8 (dwords) lattimer=0x42 (1980 ns), mingnt=0x08 (2000 ns), maxlat=0x00 (0 ns) intpin=a, irq=7 powerspec 2 supports D0 D1 D2 D3 current D0 map[10]: type Prefetchable Memory, range 32, base 0xe0000000, size 27, enabled pcib6: allocated prefetch range (0xe0000000-0xe7ffffff) for rid 10 of pci0:15:0:0 map[14]: type I/O Port, range 32, base 0x6000, size 8, enabled pcib6: allocated I/O port range (0x6000-0x60ff) for rid 14 of pci0:15:0:0 map[18]: type Memory, range 32, base 0xe8300000, size 16, enabled pcib6: allocated memory range (0xe8300000-0xe830ffff) for rid 18 of pci0:15:0:0 pcib6: matched entry for 15.0.INTA pcib6: slot 0 INTA hardwired to IRQ 16 vgapci0: port 0x6000-0x60ff mem 0xe0000000-0xe7ffffff,0xe8300000-0xe830ffff irq 16 at device 0.0 on pci15 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30a0-0x30af at device 31.1 on pci0 ata0: on atapci0 ioapic0: routing intpin 14 (ISA IRQ 14) to lapic 0 vector 55 atapci1: port 0x30e8-0x30ef,0x30dc-0x30df,0x30e0-0x30e7,0x30d8-0x30db,0x30b0-0x30bf mem 0xe8600400-0xe86007ff irq 19 at device 31.2 on pci0 atapci1: AHCI called from vendor specific driver atapci1: AHCI v1.10 controller with 4 3Gbps ports, PM not supported atapci1: Caps: 64bit NCQ ALP AL CLO 3Gbps PMD SSC PSC 32cmd 4ports ata2: on atapci1 ata3: on atapci1 ata4: on atapci1 ata5: on atapci1 pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atrtc0: registered as a time-of-day clock (resolution 1000000us, adjustment 0.500000000s) ioapic0: routing intpin 8 (ISA IRQ 8) to lapic 0 vector 56 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 ioapic0: routing intpin 2 (ISA IRQ 0) to lapic 0 vector 57 Event timer "i8254" frequency 1193182 Hz quality 100 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 kbd0: atkbd0, generic (0), config:0x0, flags:0x3f0000 ioapic0: routing intpin 1 (ISA IRQ 1) to lapic 0 vector 58 atkbd0: [GIANT-LOCKED] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 0 vector 59 uart0: fast interrupt uart0: console (9600,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 0 vector 60 uart1: fast interrupt acpi0: wakeup code va 0xffffff81a3854000 pa 0x4000 pcib0: allocated type 3 (0xa0000-0xa07ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa0800-0xa0fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa1000-0xa17ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa1800-0xa1fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa2000-0xa27ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa2800-0xa2fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa3000-0xa37ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa3800-0xa3fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa4000-0xa47ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa4800-0xa4fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa5000-0xa57ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa5800-0xa5fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa6000-0xa67ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa6800-0xa6fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa7000-0xa77ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa7800-0xa7fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa8000-0xa87ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa8800-0xa8fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa9000-0xa97ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa9800-0xa9fff) for rid 0 of orm0 pcib0: allocated type 3 (0xaa000-0xaa7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xaa800-0xaafff) for rid 0 of orm0 pcib0: allocated type 3 (0xab000-0xab7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xab800-0xabfff) for rid 0 of orm0 pcib0: allocated type 3 (0xac000-0xac7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xac800-0xacfff) for rid 0 of orm0 pcib0: allocated type 3 (0xad000-0xad7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xad800-0xadfff) for rid 0 of orm0 pcib0: allocated type 3 (0xae000-0xae7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xae800-0xaefff) for rid 0 of orm0 pcib0: allocated type 3 (0xaf000-0xaf7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xaf800-0xaffff) for rid 0 of orm0 pcib0: allocated type 3 (0xb0000-0xb07ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb0800-0xb0fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb1000-0xb17ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb1800-0xb1fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb2000-0xb27ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb2800-0xb2fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb3000-0xb37ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb3800-0xb3fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb4000-0xb47ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb4800-0xb4fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb5000-0xb57ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb5800-0xb5fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb6000-0xb67ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb6800-0xb6fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb7000-0xb77ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb7800-0xb7fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb8000-0xb87ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb8800-0xb8fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb9000-0xb97ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb9800-0xb9fff) for rid 0 of orm0 pcib0: allocated type 3 (0xba000-0xba7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xba800-0xbafff) for rid 0 of orm0 pcib0: allocated type 3 (0xbb000-0xbb7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xbb800-0xbbfff) for rid 0 of orm0 pcib0: allocated type 3 (0xbc000-0xbc7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xbc800-0xbcfff) for rid 0 of orm0 pcib0: allocated type 3 (0xbd000-0xbd7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xbd800-0xbdfff) for rid 0 of orm0 pcib0: allocated type 3 (0xbe000-0xbe7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xbe800-0xbefff) for rid 0 of orm0 pcib0: allocated type 3 (0xbf000-0xbf7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xbf800-0xbffff) for rid 0 of orm0 pcib0: allocated type 3 (0xcc000-0xcc7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xcc800-0xccfff) for rid 0 of orm0 pcib0: allocated type 3 (0xcd000-0xcd7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xcd800-0xcdfff) for rid 0 of orm0 pcib0: allocated type 3 (0xce000-0xce7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xce800-0xcefff) for rid 0 of orm0 pcib0: allocated type 3 (0xcf000-0xcf7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xcf800-0xcffff) for rid 0 of orm0 pcib0: allocated type 3 (0xd0000-0xd07ff) for rid 0 of orm0 pcib0: allocated type 3 (0xd0800-0xd0fff) for rid 0 of orm0 pcib0: allocated type 3 (0xd1000-0xd17ff) for rid 0 of orm0 pcib0: allocated type 3 (0xd1800-0xd1fff) for rid 0 of orm0 pcib0: allocated type 3 (0xd2000-0xd27ff) for rid 0 of orm0 pcib0: allocated type 3 (0xd2800-0xd2fff) for rid 0 of orm0 pcib0: allocated type 3 (0xd3000-0xd37ff) for rid 0 of orm0 pcib0: allocated type 3 (0xd3800-0xd3fff) for rid 0 of orm0 pcib0: allocated type 3 (0xd4000-0xd47ff) for rid 0 of orm0 pcib0: allocated type 3 (0xd4800-0xd4fff) for rid 0 of orm0 pcib0: allocated type 3 (0xd5000-0xd57ff) for rid 0 of orm0 pcib0: allocated type 3 (0xd5800-0xd5fff) for rid 0 of orm0 pcib0: allocated type 3 (0xd6000-0xd67ff) for rid 0 of orm0 pcib0: allocated type 3 (0xd6800-0xd6fff) for rid 0 of orm0 pcib0: allocated type 3 (0xd7000-0xd77ff) for rid 0 of orm0 pcib0: allocated type 3 (0xd7800-0xd7fff) for rid 0 of orm0 pcib0: allocated type 3 (0xd8000-0xd87ff) for rid 0 of orm0 pcib0: allocated type 3 (0xd8800-0xd8fff) for rid 0 of orm0 pcib0: allocated type 3 (0xd9000-0xd97ff) for rid 0 of orm0 pcib0: allocated type 3 (0xd9800-0xd9fff) for rid 0 of orm0 pcib0: allocated type 3 (0xda000-0xda7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xda800-0xdafff) for rid 0 of orm0 pcib0: allocated type 3 (0xdb000-0xdb7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xdb800-0xdbfff) for rid 0 of orm0 pcib0: allocated type 3 (0xdc000-0xdc7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xdc800-0xdcfff) for rid 0 of orm0 pcib0: allocated type 3 (0xdd000-0xdd7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xdd800-0xddfff) for rid 0 of orm0 pcib0: allocated type 3 (0xde000-0xde7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xde800-0xdefff) for rid 0 of orm0 pcib0: allocated type 3 (0xdf000-0xdf7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xdf800-0xdffff) for rid 0 of orm0 pcib0: allocated type 3 (0xe0000-0xe07ff) for rid 0 of orm0 pcib0: allocated type 3 (0xe0800-0xe0fff) for rid 0 of orm0 pcib0: allocated type 3 (0xe1000-0xe17ff) for rid 0 of orm0 pcib0: allocated type 3 (0xe1800-0xe1fff) for rid 0 of orm0 pcib0: allocated type 3 (0xe2000-0xe27ff) for rid 0 of orm0 pcib0: allocated type 3 (0xe2800-0xe2fff) for rid 0 of orm0 pcib0: allocated type 3 (0xe3000-0xe37ff) for rid 0 of orm0 pcib0: allocated type 3 (0xe3800-0xe3fff) for rid 0 of orm0 isa_probe_children: disabling PnP devices atkbdc: atkbdc0 already exists; skipping it atrtc: atrtc0 already exists; skipping it sc: sc0 already exists; skipping it uart: uart0 already exists; skipping it uart: uart1 already exists; skipping it isa_probe_children: probing non-PnP devices sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sc0: fb0, kbd0, terminal emulator: scteken (teken terminal) vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 pcib0: allocated type 4 (0x3c0-0x3df) for rid 0 of vga0 pcib0: allocated type 3 (0xa0000-0xbffff) for rid 0 of vga0 fdc0 failed to probe at port 0x3f0 irq 6 drq 2 on isa0 ppc0 failed to probe at irq 7 on isa0 isa_probe_children: probing PnP devices est0: enabling SpeedStep est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr a1a0a1a06000a1a device_attach: est0 attach returned 6 p4tcc0: on cpu0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr a1a0a1a06000a1a device_attach: est1 attach returned 6 p4tcc1: on cpu1 Device configuration finished. procfs registered lapic: Divisor 2, Frequency 99752007 Hz Timecounters tick every 1.000 msec vlan: initialized, using hash tables with chaining lo0: bpf attached usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 usbus4: 480Mbps High Speed USB v2.0 ata0: reset tp1 mask=03 ostat0=7f ostat1=50 ata0: stat0=0x7f err=0x7f lsb=0x7f msb=0x7f ata0: stat0=0x00 err=0x00 lsb=0x00 msb=0x00 ata0: stat1=0x00 err=0x01 lsb=0x14 msb=0xeb ata0: reset tp2 stat0=00 stat1=00 devices=0x20000 (aprobe0:ata0:0:1:0): SIGNATURE: eb14 ugen4.1: at usbus4 uhub4: on usbus4 uhub0: 2 ports with 2 removable, self powered ata2: AHCI reset... ata2: hard reset ... uhub1: 2 ports with 2 removable, self powered ata2: SATA connect time=10ms status=00000113 ata2: ready wait time=0ms uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered ata2: software reset port 0... ata2: ready wait time=0ms ata2: SIGNATURE: 00000101 ata2: AHCI reset done: devices=00000001 (aprobe0:ata2:0:0:0): SIGNATURE: 0000 ata3: AHCI reset... ata3: hard reset ... ata3: SATA connect time=10ms status=00000113 ata3: ready wait time=0ms ata3: software reset port 0... ata3: ready wait time=0ms ata3: SIGNATURE: 00000101 ata3: AHCI reset done: devices=00000001 (aprobe1:ata3:0:0:0): SIGNATURE: 0000 ata4: AHCI reset... ata4: hard reset ... ata4: SATA connect time=10ms status=00000113 ata4: ready wait time=0ms ata4: software reset port 0... ata4: ready wait time=0ms ata4: SIGNATURE: 00000101 ata4: AHCI reset done: devices=00000001 (aprobe2:ata4:0:0:0): SIGNATURE: 0000 ata5: AHCI reset... ata5: hard reset ... ata5: SATA connect time=10ms status=00000113 ata5: ready wait time=0ms ata5: software reset port 0... ata5: ready wait time=0ms ata5: SIGNATURE: 00000101 ata5: AHCI reset done: devices=00000001 (aprobe0:ata5:0:0:0): SIGNATURE: 0000 pass0 at ata0 bus 0 scbus0 target 1 lun 0 pass0: Removable CD-ROM SCSI-0 device pass0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) pass1 at ata2 bus 0 scbus1 target 0 lun 0 pass1: ATA-7 SATA 1.x device pass1: Serial Number 6QF1VVC0 pass1: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) pass2 at ata3 bus 0 scbus2 target 0 lun 0 pass2: ATA-7 SATA 1.x device pass2: Serial Number 6QF3RPZC pass2: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) pass3 at ata4 bus 0 scbus3 target 0 lun 0 pass3: ATA-7 SATA 1.x device pass3: Serial Number 6QF1P8LH pass3: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) pass4 at ata5 bus 0 scbus4 target 0 lun 0 pass4: ATA-7 SATA 1.x device pass4: Serial Number 6QF1WN9R pass4: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) pass0 at ata0 bus 0 scbus0 target 1 lun 0 pass0: Removable CD-ROM SCSI-0 device pass0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) pass1 at ata2 bus 0 scbus1 target 0 lun 0 pass1: ATA-7 SATA 1.x device pass1: Serial Number 6QF1VVC0 pass1: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) pass2 at ata3 bus 0 scbus2 target 0 lun 0 pass2: ATA-7 SATA 1.x device pass2: Serial Number 6QF3RPZC pass2: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) pass3 at ata4 bus 0 scbus3 target 0 lun 0 pass3: ATA-7 SATA 1.x device pass3: Serial Number 6QF1P8LH pass3: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) pass4 at ata5 bus 0 scbus4 target 0 lun 0 pass4: ATA-7 SATA 1.x device pass4: Serial Number 6QF1WN9R pass4: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) SMP: AP CPU #1 Launched! cpu1 AP: ID: 0x01000000 VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 SMP: passed TSC synchronization test TSC timecounter discards lower 7 bit(s) Timecounter "TSC-low" frequency 15586249 Hz quality 1000 Root mount waiting for: usbus4 uhub4: 8 ports with 8 removable, self powered Trying to mount root from ufs:/dev/mirror/boota [rw]... mountroot: waiting for device /dev/mirror/boota ... Mounting from ufs:/dev/mirror/boota failed with error 19. Loader variables: vfs.root.mountfrom=ufs:/dev/mirror/boota vfs.root.mountfrom.options=rw Manual root filesystem specification: : [options] Mount using filesystem and with the specified (optional) option list. eg. ufs:/dev/da0s1a zfs:tank cd9660:/dev/acd0 ro (which is equivalent to: mount -t cd9660 -o ro /dev/acd0 /) ? List valid disk boot devices . Yield 1 second (for background tasks) Abort manual input mountroot> panic: mountroot: unable to (re-)mount root. cpuid = 1 Uptime: 24s Automatic reboot in 15 seconds - press a key on the console to abort Rebooting... From owner-freebsd-stable@FreeBSD.ORG Sat Dec 17 16:20:39 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D35F1106566B; Sat, 17 Dec 2011 16:20:39 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8969A8FC15; Sat, 17 Dec 2011 16:20:39 +0000 (UTC) Received: by iadj38 with SMTP id j38so3002394iad.13 for ; Sat, 17 Dec 2011 08:20:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=3b7Bx18EAujUMxSdwbU8n6v8XnrV4fXcACtM/9Ppvks=; b=SJqqYBTtisFvUcDtK6liFV+NKzExrN/C1nz4PdsGF73Bo7CQQ/iipfWyYPdi5+lCmL 0zmcHO22/OBbyN2wl4lfJthyK/ii/bjOQgeYdVnoOe+cAemt45h2nAGkrxpwqB8wlN8C 3P0y4Zf59aovxUKGTUvkOumlSM+c7PPWLSi8o= Received: by 10.50.85.136 with SMTP id h8mr15979537igz.56.1324138838968; Sat, 17 Dec 2011 08:20:38 -0800 (PST) Received: from DataIX.net (24-247-9-230.dhcp.aldl.mi.charter.com. [24.247.9.230]) by mx.google.com with ESMTPS id l28sm45946009ibc.3.2011.12.17.08.20.36 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 17 Dec 2011 08:20:37 -0800 (PST) Sender: Jason Hellenthal Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id pBHGKYHV018224 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 17 Dec 2011 11:20:34 -0500 (EST) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id pBHGKTcx018212; Sat, 17 Dec 2011 11:20:29 -0500 (EST) (envelope-from jhell@DataIX.net) Date: Sat, 17 Dec 2011 11:20:29 -0500 From: Jason Hellenthal To: Doug Barton Message-ID: <20111217162029.GA3875@DataIX.net> References: <4EEA5DD0.1040009@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EEA5DD0.1040009@FreeBSD.org> Cc: freebsd-stable@freebsd.org Subject: Re: swi4: clock taking 40% cpu?!? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 16:20:39 -0000 On Thu, Dec 15, 2011 at 12:51:28PM -0800, Doug Barton wrote: > Howdy, > > Web server under heavy'ish load (7 on a 2 cpu system) running > 8.2-RELEASE-p4 i386 I'm seeing this: > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 12 root -32 - 0K 112K WAIT 0 129:01 39.99% {swi4: clock} > > Any ideas why the clock should be taking so much cpu? HZ=100 if that > makes a difference ... > > Without NTPd running test the following. apply "/usr/bin/time -ph sleep %1" 300 600 900 If the results are skewed quite a bit then your system may benefit from a different HZ than what you have set. I have seen systems that require a HZ of 350 and as weird as it sounds NTPd may be tasting the clock too much just to try and keep time. -- ;s =; From owner-freebsd-stable@FreeBSD.ORG Sat Dec 17 16:23:12 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10D45106566C; Sat, 17 Dec 2011 16:23:12 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id BB43A8FC14; Sat, 17 Dec 2011 16:23:11 +0000 (UTC) Received: by iadj38 with SMTP id j38so3006042iad.13 for ; Sat, 17 Dec 2011 08:23:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=Y7W6wRVchrTIFv36x537CgJKSyfJxmryT+wnBB3bRi8=; b=C7XDZsFvQqTZnDjsuMhnEANRvJsbPJmnZNnyQ9SY3W2AUvAPLyiGb5huGVJvlJvsh3 nW7/RDNl3clt9PX/Vn8BaO9NAvYQho2j33w6H5wU46cKqc3EV3pNzrCiGk0kvbDgsvI0 S/AcHs8n1kRDsQtLeLLeBK/7Hdt9M7WnTJcG0= Received: by 10.50.207.40 with SMTP id lt8mr16447665igc.43.1324138991258; Sat, 17 Dec 2011 08:23:11 -0800 (PST) Received: from DataIX.net (24-247-9-230.dhcp.aldl.mi.charter.com. [24.247.9.230]) by mx.google.com with ESMTPS id wn9sm9076866igc.6.2011.12.17.08.23.09 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 17 Dec 2011 08:23:10 -0800 (PST) Sender: Jason Hellenthal Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id pBHGN7uo021589 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 17 Dec 2011 11:23:07 -0500 (EST) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id pBHGN2mX021588; Sat, 17 Dec 2011 11:23:02 -0500 (EST) (envelope-from jhell@DataIX.net) Date: Sat, 17 Dec 2011 11:23:02 -0500 From: Jason Hellenthal To: Doug Barton Message-ID: <20111217162302.GB3875@DataIX.net> References: <4EEA5DD0.1040009@FreeBSD.org> <20111217162029.GA3875@DataIX.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111217162029.GA3875@DataIX.net> Cc: freebsd-stable@freebsd.org Subject: Re: swi4: clock taking 40% cpu?!? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 16:23:12 -0000 Should also mention the kern.sched may be playing a part in this too. On Sat, Dec 17, 2011 at 11:20:29AM -0500, Jason Hellenthal wrote: > > > On Thu, Dec 15, 2011 at 12:51:28PM -0800, Doug Barton wrote: > > Howdy, > > > > Web server under heavy'ish load (7 on a 2 cpu system) running > > 8.2-RELEASE-p4 i386 I'm seeing this: > > > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > > 12 root -32 - 0K 112K WAIT 0 129:01 39.99% {swi4: clock} > > > > Any ideas why the clock should be taking so much cpu? HZ=100 if that > > makes a difference ... > > > > > > Without NTPd running test the following. > > > apply "/usr/bin/time -ph sleep %1" 300 600 900 > > If the results are skewed quite a bit then your system may benefit from a different HZ than what you have set. I have seen systems that require a HZ of 350 and as weird as it sounds NTPd may be tasting the clock too much just to try and keep time. > > -- > ;s =; -- ;s =; From owner-freebsd-stable@FreeBSD.ORG Sat Dec 17 16:31:44 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 34844106566B for ; Sat, 17 Dec 2011 16:31:44 +0000 (UTC) (envelope-from ae@FreeBSD.org) Received: from [127.0.0.1] (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 08A6E158711; Sat, 17 Dec 2011 16:31:42 +0000 (UTC) Message-ID: <4EECC3E1.2000005@FreeBSD.org> Date: Sat, 17 Dec 2011 20:31:29 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111203 Thunderbird/8.0 MIME-Version: 1.0 To: Randy Bush References: <4EEC7B54.4070407@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.1.2 OpenPGP: id=10C8A17A Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig68205149F02E85619AA55912" Cc: FreeBSD Stable Subject: Re: 8.2->9.prerel: gmirror failed with error 19 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 16:31:44 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig68205149F02E85619AA55912 Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: quoted-printable On 17.12.2011 19:30, Randy Bush wrote: > FreeBSD 9.0-PRERELEASE #18: Tue Dec 13 12:20:57 GMT 2011 > root@work0.psg.com:/usr/obj/usr/src/sys/WORK0 amd64 Could you also show your kernel config? > pass0 at ata0 bus 0 scbus0 target 1 lun 0 > pass0: Removable CD-ROM SCSI-0 device=20 > pass0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) > pass1 at ata2 bus 0 scbus1 target 0 lun 0 > pass1: ATA-7 SATA 1.x device > pass1: Serial Number 6QF1VVC0 > pass1: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) > pass2 at ata3 bus 0 scbus2 target 0 lun 0 > pass2: ATA-7 SATA 1.x device > pass2: Serial Number 6QF3RPZC > pass2: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) > pass3 at ata4 bus 0 scbus3 target 0 lun 0 > pass3: ATA-7 SATA 1.x device > pass3: Serial Number 6QF1P8LH > pass3: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) > pass4 at ata5 bus 0 scbus4 target 0 lun 0 > pass4: ATA-7 SATA 1.x device > pass4: Serial Number 6QF1WN9R > pass4: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) > pass0 at ata0 bus 0 scbus0 target 1 lun 0 > pass0: Removable CD-ROM SCSI-0 device=20 > pass0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) > pass1 at ata2 bus 0 scbus1 target 0 lun 0 > pass1: ATA-7 SATA 1.x device > pass1: Serial Number 6QF1VVC0 > pass1: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) > pass2 at ata3 bus 0 scbus2 target 0 lun 0 > pass2: ATA-7 SATA 1.x device > pass2: Serial Number 6QF3RPZC > pass2: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) > pass3 at ata4 bus 0 scbus3 target 0 lun 0 > pass3: ATA-7 SATA 1.x device > pass3: Serial Number 6QF1P8LH > pass3: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) > pass4 at ata5 bus 0 scbus4 target 0 lun 0 > pass4: ATA-7 SATA 1.x device > pass4: Serial Number 6QF1WN9R > pass4: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) > SMP: AP CPU #1 Launched! I don't see that you disk drives are detected. What you are getting if you type "?" in the following command prompt? >=20 > ? List valid disk boot devices > . Yield 1 second (for background tasks) > Abort manual input --=20 WBR, Andrey V. Elsukov --------------enig68205149F02E85619AA55912 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iQEcBAEBAgAGBQJO7MPlAAoJEAHF6gQQyKF66zcIAMFuy6O2oW14/oGl4Im04umq 9ymmVdbnNsAk42OcckcNjWazk2XeWyq1kUp+51PXjbxAew0Q+LppOP6etj1hAsxj u+ZW9sVix1TwEj+qZzd+J1hsyuKMs+RwPHF2t5xhd3nO4wvJjlXOxbMJzbib14Hx 71Jw1EzaVPwTvguEFJQT6DBnkU4thGFQvFmuv9YwLVnQbGkVrb08otkheU3AKgkg HrRQMEvRSwKD/LUVENW3yqPxvxwHRmN+QJsXNopWNm+AYprgLtY2z+1FbdvQGy6f UbVKQJUyA+gqoiZd2/mV7xpX/9J9WOAUaOwuSb5DYWhRtCdPnkt21xQLT9WwhP8= =kQNl -----END PGP SIGNATURE----- --------------enig68205149F02E85619AA55912-- From owner-freebsd-stable@FreeBSD.ORG Sat Dec 17 17:33:27 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E75211065670 for ; Sat, 17 Dec 2011 17:33:27 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (ip-3-2-0-2.r20.asbnva02.us.ce.gin.ntt.net [IPv6:2001:418:0:5000::16]) by mx1.freebsd.org (Postfix) with ESMTP id 7F2228FC0A for ; Sat, 17 Dec 2011 17:33:27 +0000 (UTC) Received: from wonderland.m5p.com (wonderland.m5p.com [IPv6:2001:418:3fd::19]) by mailhost.m5p.com (8.14.4/8.14.4) with ESMTP id pBHHXLHw054328; Sat, 17 Dec 2011 12:33:26 -0500 (EST) (envelope-from george+freebsd@m5p.com) Message-ID: <4EECD261.2080208@m5p.com> Date: Sat, 17 Dec 2011 12:33:21 -0500 From: George Mitchell User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111127 Thunderbird/8.0 MIME-Version: 1.0 To: Oliver Pinter References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (mailhost.m5p.com [IPv6:2001:418:3fd::f7]); Sat, 17 Dec 2011 12:33:26 -0500 (EST) X-Scanned-By: MIMEDefang 2.72 on IPv6:2001:418:3fd::f7 Cc: freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 17:33:28 -0000 On 12/14/11 21:05, Oliver Pinter wrote: > [...] > Hi! > > Can you try with this settings: > > op@opn ~> sysctl kern.sched. > kern.sched.cpusetsize: 8 > kern.sched.preemption: 0 > kern.sched.name: ULE > kern.sched.slice: 13 > kern.sched.interact: 30 > kern.sched.preempt_thresh: 224 > kern.sched.static_boost: 152 > kern.sched.idlespins: 10000 > kern.sched.idlespinthresh: 16 > kern.sched.affinity: 1 > kern.sched.balance: 1 > kern.sched.balance_interval: 133 > kern.sched.steal_htt: 1 > kern.sched.steal_idle: 1 > kern.sched.steal_thresh: 1 > kern.sched.topology_spec: > > 0, 1 > > > 0, 1 > > > > > [...] Sorry I didn't try this earlier, but I had time this morning. Apparently you can't change kern.sched.preemption without recompiling, so I did that. It didn't help, and subjectively it made interactive performance worse. I changed preempt_thresh and observed no difference. There were only a couple of small differences between your other settings and the 9.0-PRERELEASE defaults. Summing up for the record, in my original test: 1. It doesn't matter whether X is running or not. 2. The problem is not limited to two or fewer CPUs. (It also happens for me on a six-CPU system.) 3. It doesn't require nCPU + 1 compute-bound processes, just nCPU. With nCPU compute-bound processes running, with SCHED_ULE, any other process that is interactive (which to me means frequently waiting for I/O) gets ABYSMAL performance -- over an order of magnitude worse than it gets with SCHED_4BSD under the same conditions. -- George From owner-freebsd-stable@FreeBSD.ORG Sat Dec 17 18:47:39 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE17C106566B; Sat, 17 Dec 2011 18:47:38 +0000 (UTC) (envelope-from randy@psg.com) Received: from ran.psg.com (ran.psg.com [IPv6:2001:418:1::36]) by mx1.freebsd.org (Postfix) with ESMTP id C01FE8FC12; Sat, 17 Dec 2011 18:47:38 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=rair.psg.com.psg.com) by ran.psg.com with esmtp (Exim 4.76 (FreeBSD)) (envelope-from ) id 1RbzII-000HPq-0v; Sat, 17 Dec 2011 18:47:38 +0000 Date: Sat, 17 Dec 2011 10:47:37 -0800 Message-ID: From: Randy Bush To: "Andrey V. Elsukov" In-Reply-To: <4EECC3E1.2000005@FreeBSD.org> References: <4EEC7B54.4070407@FreeBSD.org> <4EECC3E1.2000005@FreeBSD.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) Emacs/22.3 Mule/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Cc: FreeBSD Stable Subject: Re: 8.2->9.prerel: gmirror failed with error 19 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 18:47:39 -0000 >> FreeBSD 9.0-PRERELEASE #18: Tue Dec 13 12:20:57 GMT 2011 >> root@work0.psg.com:/usr/obj/usr/src/sys/WORK0 amd64 > Could you also show your kernel config? cpu HAMMER ident WORK0 makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options INET6 # IPv6 communications protocols options SCTP # Stream Control Transmission Protocol options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories #options UFS_GJOURNAL # Enable gjournal-based UFS journaling #options MD_ROOT # MD is a potential root device #options NFSCL # New Network Filesystem Client #options NFSD # New Network Filesystem Server #options NFSLOCKD # Network Lock Manager #options NFS_ROOT # NFS usable as /, requires NFSCL #options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options COMPAT_FREEBSD32 # Compatible with i386 binaries #options COMPAT_FREEBSD4 # Compatible with FreeBSD4 #options COMPAT_FREEBSD5 # Compatible with FreeBSD5 #options COMPAT_FREEBSD6 # Compatible with FreeBSD6 #options COMPAT_FREEBSD7 # Compatible with FreeBSD7 options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework #options KDTRACE_FRAME # Ensure frames are compiled in #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel #options KDB # Kernel debugger related code #options KDB_TRACE # Print a stack trace for a panic # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcessor Kernel # CPU frequency control device cpufreq # Bus support. device acpi device pci # Floppy drives #device fdc # ATA controllers #device ahci # AHCI-compatible SATA controllers device ata # Legacy ATA/SATA controllers options ATA_CAM # Handle legacy controllers with CAM options ATA_STATIC_ID # Static device numbering #device mvs # Marvell 88SX50XX/88SX60XX/88SX70XX/SoC SATA #device siis # SiliconImage SiI3124/SiI3132/SiI3531 SATA # SCSI Controllers #device ahc # AHA2940 and onboard AIC7xxx devices #options AHC_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~128k to driver. #device ahd # AHA39320/29320 and onboard AIC79xx devices #options AHD_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~215k to driver. #device esp # AMD Am53C974 (Tekram DC-390(T)) #device hptiop # Highpoint RocketRaid 3xxx series #device isp # Qlogic family #device ispfw # Firmware for QLogic HBAs- normally a module #device mpt # LSI-Logic MPT-Fusion #device mps # LSI-Logic MPT-Fusion 2 #device ncr # NCR/Symbios Logic #device sym # NCR/Symbios Logic (newer chipsets + those of `ncr') #device trm # Tekram DC395U/UW/F DC315U adapters #device adv # Advansys SCSI adapters #device adw # Advansys wide SCSI adapters #device aic # Adaptec 15[012]x SCSI adapters, AIC-6[23]60. #device bt # Buslogic/Mylex MultiMaster SCSI adapters # ATA/SCSI peripherals device scbus # SCSI bus (required for ATA/SCSI) #device ch # SCSI media changers #device da # Direct Access (disks) #device sa # Sequential Access (tape etc) #device cd # CD device pass # Passthrough device (direct ATA/SCSI access) #device ses # SCSI Environmental Services (and SAF-TE) # RAID controllers interfaced to the SCSI subsystem #device amr # AMI MegaRAID #device arcmsr # Areca SATA II RAID #XXX it is not 64-bit clean, -scottl #device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID #device ciss # Compaq Smart RAID 5* #device dpt # DPT Smartcache III, IV - See NOTES for options #device hptmv # Highpoint RocketRAID 182x #device hptrr # Highpoint RocketRAID 17xx, 22xx, 23xx, 25xx #device iir # Intel Integrated RAID #device ips # IBM (Adaptec) ServeRAID #device mly # Mylex AcceleRAID/eXtremeRAID #device twa # 3ware 9000 series PATA/SATA RAID # RAID controllers #device aac # Adaptec FSA RAID #device aacp # SCSI passthrough for aac (requires CAM) #device ida # Compaq Smart RAID #device mfi # LSI MegaRAID SAS #device mlx # Mylex DAC960 family #XXX pointer/int warnings #device pst # Promise Supertrak SX6000 #device twe # 3ware ATA RAID #device tws # LSI 3ware 9750 SATA+SAS 6Gb/s RAID controller # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard #device psm # PS/2 mouse #device kbdmux # keyboard multiplexer device vga # VGA video card driver #device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc #options SC_PIXEL_MODE # add support for the raster text mode #device agp # support several AGP chipsets # PCCARD (PCMCIA) support # PCMCIA and cardbus bridge support #device cbb # cardbus (yenta) bridge #device pccard # PC Card (16-bit) bus #device cardbus # CardBus (32-bit) bus # Serial (COM) ports device uart # Generic UART driver # Parallel port #device ppc #device ppbus # Parallel port bus (required) #device lpt # Printer #device plip # TCP/IP over parallel #device ppi # Parallel port interface device #device vpo # Requires scbus and da #device puc # Multi I/O cards and multi-channel UARTs # PCI Ethernet NICs. #device bxe # Broadcom BCM57710/BCM57711/BCM57711E 10Gb Ethernet #device de # DEC/Intel DC21x4x (``Tulip'') device em # Intel PRO/1000 Gigabit Ethernet Family #device igb # Intel PRO/1000 PCIE Server Gigabit Family #device ixgbe # Intel PRO/10GbE PCIE Ethernet Family #device le # AMD Am7900 LANCE and Am79C9xx PCnet #device ti # Alteon Networks Tigon I/II gigabit Ethernet #device txp # 3Com 3cR990 (``Typhoon'') #device vx # 3Com 3c590, 3c595 (``Vortex'') # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! #device miibus # MII bus support #device ae # Attansic/Atheros L2 FastEthernet #device age # Attansic/Atheros L1 Gigabit Ethernet #device alc # Atheros AR8131/AR8132 Ethernet #device ale # Atheros AR8121/AR8113/AR8114 Ethernet #device bce # Broadcom BCM5706/BCM5708 Gigabit Ethernet #device bfe # Broadcom BCM440x 10/100 Ethernet #device bge # Broadcom BCM570xx Gigabit Ethernet #device dc # DEC/Intel 21143 and various workalikes #device et # Agere ET1310 10/100/Gigabit Ethernet #device fxp # Intel EtherExpress PRO/100B (82557, 82558) #device jme # JMicron JMC250 Gigabit/JMC260 Fast Ethernet #device lge # Level 1 LXT1001 gigabit Ethernet #device msk # Marvell/SysKonnect Yukon II Gigabit Ethernet #device nfe # nVidia nForce MCP on-board Ethernet #device nge # NatSemi DP83820 gigabit Ethernet #device nve # nVidia nForce MCP on-board Ethernet Networking #device pcn # AMD Am79C97x PCI 10/100 (precedence over 'le') #device re # RealTek 8139C+/8169/8169S/8110S #device rl # RealTek 8129/8139 #device sf # Adaptec AIC-6915 (``Starfire'') #device sge # Silicon Integrated Systems SiS190/191 #device sis # Silicon Integrated Systems SiS 900/SiS 7016 #device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet #device ste # Sundance ST201 (D-Link DFE-550TX) #device stge # Sundance/Tamarack TC9021 gigabit Ethernet #device tl # Texas Instruments ThunderLAN #device tx # SMC EtherPower II (83c170 ``EPIC'') #device vge # VIA VT612x gigabit Ethernet #device vr # VIA Rhine, Rhine II #device wb # Winbond W89C840F #device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'') # ISA Ethernet NICs. pccard NICs included. #device cs # Crystal Semiconductor CS89x0 NIC # 'device ed' requires 'device miibus' #device ed # NE[12]000, SMC Ultra, 3c503, DS8390 cards #device ex # Intel EtherExpress Pro/10 and Pro/10+ #device ep # Etherlink III based cards #device fe # Fujitsu MB8696x based cards #device sn # SMC's 9000 series of Ethernet chips #device xe # Xircom pccard Ethernet # Wireless NIC cards #device wlan # 802.11 support #options IEEE80211_DEBUG # enable debug msgs #options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's #options IEEE80211_SUPPORT_MESH # enable 802.11s draft support #device wlan_wep # 802.11 WEP support #device wlan_ccmp # 802.11 CCMP support #device wlan_tkip # 802.11 TKIP support #device wlan_amrr # AMRR transmit rate control algorithm #device an # Aironet 4500/4800 802.11 wireless NICs. #device ath # Atheros NIC's #device ath_pci # Atheros pci/cardbus glue #device ath_hal # pci/cardbus chip support #options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors #device ath_rate_sample # SampleRate tx rate control for ath #device bwi # Broadcom BCM430x/BCM431x wireless NICs. #device bwn # Broadcom BCM43xx wireless NICs. #device ipw # Intel 2100 wireless NICs. #device iwi # Intel 2200BG/2225BG/2915ABG wireless NICs. #device iwn # Intel 4965/1000/5000/6000 wireless NICs. #device malo # Marvell Libertas wireless NICs. #device mwl # Marvell 88W8363 802.11n wireless NICs. #device ral # Ralink Technology RT2500 wireless NICs. #device wi # WaveLAN/Intersil/Symbol 802.11 wireless NICs. #device wpi # Intel 3945ABG wireless NICs. # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device vlan # 802.1Q VLAN support device tun # Packet tunnel. device pty # BSD-style compatibility pseudo ttys device md # Memory "disks" #device gif # IPv6 and IPv4 tunneling #device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter # USB support #options USB_DEBUG # enable debug msgs device uhci # UHCI PCI->USB interface #device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device usb # USB Bus (required) #device udbp # USB Double Bulk Pipe devices (needs netgraph) device uhid # "Human Interface Devices" device ukbd # Keyboard #device ulpt # Printer device umass # Disks/Mass storage - Requires scbus and da #device ums # Mouse #device urio # Diamond Rio 500 MP3 player # USB Serial devices #device u3g # USB-based 3G modems (Option, Huawei, Sierra) #device uark # Technologies ARK3116 based serial adapters #device ubsa # Belkin F5U103 and compatible serial adapters #device uftdi # For FTDI usb serial adapters #device uipaq # Some WinCE based devices #device uplcom # Prolific PL-2303 serial adapters #device uslcom # SI Labs CP2101/CP2102 serial adapters #device uvisor # Visor and Palm devices #device uvscom # USB serial support for DDI pocket's PHS # USB Ethernet, requires miibus #device aue # ADMtek USB Ethernet #device axe # ASIX Electronics USB Ethernet #device cdce # Generic USB over Ethernet #device cue # CATC USB Ethernet #device kue # Kawasaki LSI USB Ethernet #device rue # RealTek RTL8150 USB Ethernet #device udav # Davicom DM9601E USB # USB Wireless #device rum # Ralink Technology RT2501USB wireless NICs #device run # Ralink Technology RT2700/RT2800/RT3000 NICs. #device uath # Atheros AR5523 wireless NICs #device upgt # Conexant/Intersil PrismGT wireless NICs. #device ural # Ralink Technology RT2500USB wireless NICs #device urtw # Realtek RTL8187B/L wireless NICs #device zyd # ZyDAS zd1211/zd1211b wireless NICs # FireWire support #device firewire # FireWire bus code # sbp(4) works for some systems but causes boot failure on others #device sbp # SCSI over FireWire (Requires scbus and da) #device fwe # Ethernet over FireWire (non-standard!) #device fwip # IP over FireWire (RFC 2734,3146) #device dcons # Dumb console driver #device dcons_crom # Configuration ROM for dcons # Sound support #device sound # Generic sound driver (required) #device snd_es137x # Ensoniq AudioPCI ES137x #device snd_hda # Intel High Definition Audio #device snd_ich # Intel, NVidia and other ICH AC'97 Audio #device snd_uaudio # USB Audio #device snd_via8233 # VIA VT8233x Audio From owner-freebsd-stable@FreeBSD.ORG Sat Dec 17 20:52:32 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 877DB1065676 for ; Sat, 17 Dec 2011 20:52:32 +0000 (UTC) (envelope-from boland37@xs4all.nl) Received: from smtp-vbr2.xs4all.nl (smtp-vbr2.xs4all.nl [194.109.24.22]) by mx1.freebsd.org (Postfix) with ESMTP id 1DC1C8FC14 for ; Sat, 17 Dec 2011 20:52:31 +0000 (UTC) Received: from charlemagne.boland.org (59-36-215.ftth.xms.internl.net [82.215.36.59]) (authenticated bits=0) by smtp-vbr2.xs4all.nl (8.13.8/8.13.8) with ESMTP id pBHKaw6o048145 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO) for ; Sat, 17 Dec 2011 21:36:59 +0100 (CET) (envelope-from boland37@xs4all.nl) Message-ID: <4EECFD6A.2030905@xs4all.nl> Date: Sat, 17 Dec 2011 21:36:58 +0100 From: Michiel Boland User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.24) Gecko/20111110 Thunderbird/3.1.16 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by XS4ALL Virus Scanner Subject: fsck_ufs out of swapspace X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 20:52:32 -0000 FreeBSD 9.0-PRERELEASE locked up while into some heavy I/O and failed to shut down properly, so I had to power-cycle. After it came back up it said Starting file system checks: ** SU+J Recovering /dev/ada0a ** Reading 33554432 byte journal from inode 4. swap_pager: out of swap space swap_pager_getswapspace(16): failed pid 67 (fsck_ufs), uid 0, was killed: out of swap space fsck: /dev/ada0a: Killed: 9 Script /etc/rc.d/fsck running Unknown error; help! ERROR: ABORTING BOOT (sending SIGTERM to parent)! The only way to continue was to do a full fsck (with no journal) This is a Sun Blade 100 (sparc64) with 768M of RAM. So the fsck is taking up all of this? That can't be right. What can I do to troubleshoot this further? Cheers Michiel From owner-freebsd-stable@FreeBSD.ORG Sat Dec 17 21:13:30 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B2D49106566C for ; Sat, 17 Dec 2011 21:13:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id B96A78FC13 for ; Sat, 17 Dec 2011 21:13:29 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id XAA09521; Sat, 17 Dec 2011 23:13:18 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Rc1ZG-0009tX-Fh; Sat, 17 Dec 2011 23:13:18 +0200 Message-ID: <4EED05EC.8050103@FreeBSD.org> Date: Sat, 17 Dec 2011 23:13:16 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111206 Thunderbird/8.0 MIME-Version: 1.0 To: George Mitchell References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <4EECD261.2080208@m5p.com> In-Reply-To: <4EECD261.2080208@m5p.com> X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org, Oliver Pinter Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 21:13:30 -0000 on 17/12/2011 19:33 George Mitchell said the following: > Summing up for the record, in my original test: > 1. It doesn't matter whether X is running or not. > 2. The problem is not limited to two or fewer CPUs. (It also happens > for me on a six-CPU system.) > 3. It doesn't require nCPU + 1 compute-bound processes, just nCPU. > > With nCPU compute-bound processes running, with SCHED_ULE, any other > process that is interactive (which to me means frequently waiting for > I/O) gets ABYSMAL performance -- over an order of magnitude worse than > it gets with SCHED_4BSD under the same conditions. I definitely do not see anything like this. Specifically: - with X - with 2 CPUs - with nCPU and/or nCPU + 1 compute-bound processes - with SCHED_ULE obviously :-) I do not get "abysmal" performance for I/O active tasks. Perhaps there is something specific that you would want me to run and measure. -- Andriy Gapon From owner-freebsd-stable@FreeBSD.ORG Sat Dec 17 21:20:46 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6175A1065670; Sat, 17 Dec 2011 21:20:46 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1BCD88FC1B; Sat, 17 Dec 2011 21:20:45 +0000 (UTC) Received: by iadj38 with SMTP id j38so3414978iad.13 for ; Sat, 17 Dec 2011 13:20:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=4vUDdwQDffP/qjDVonCsMYVaBsQ2sgUR8jcGy6ehRXg=; b=I8S4/FHGkrM0Oij3ohhQRckhFcHZNmUBtmI9esXyjDIN7jv4nXGxruYmGH1AMOsc83 +1yqes6/FXonW9YRjvo9mwO5vl2j3FRe16ot1kWCcWZKsZQvahriRrxxuH+OI4yup8r6 O2zuPEgnlPC1KrnEmmGgnYBOlRzauzkEZF0t8= MIME-Version: 1.0 Received: by 10.50.186.200 with SMTP id fm8mr17217739igc.93.1324156845457; Sat, 17 Dec 2011 13:20:45 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.42.243.65 with HTTP; Sat, 17 Dec 2011 13:20:45 -0800 (PST) In-Reply-To: <4EED05EC.8050103@FreeBSD.org> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <4EECD261.2080208@m5p.com> <4EED05EC.8050103@FreeBSD.org> Date: Sat, 17 Dec 2011 13:20:45 -0800 X-Google-Sender-Auth: e4a9Kg0YMq-bNc3EyQkPsxKCCuE Message-ID: From: Adrian Chadd To: Andriy Gapon Content-Type: text/plain; charset=ISO-8859-1 Cc: George Mitchell , freebsd-stable@freebsd.org, Oliver Pinter Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 21:20:46 -0000 Erm, just as a random question - since device drivers (and GEOM) run as separate threads, has anyone looked into what kind of effects the scheduler has on these? I definitely have measurable throughput/responsiveness differences between ULE and 4BSD (and preempt/non-preempt on 4BSD) on my MIPS boards when they're bridging traffic. I wonder if there's something strange going on with the scheduling and preemption of driver netisrs, taskqueues, the fast interrupt handlers, etc. This may -not- be a userland specific problem.. Adrian From owner-freebsd-stable@FreeBSD.ORG Sat Dec 17 22:00:21 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E5E010656B3; Sat, 17 Dec 2011 22:00:21 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id C68A68FC21; Sat, 17 Dec 2011 22:00:18 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id AAA10029; Sun, 18 Dec 2011 00:00:16 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Rc2Ii-0009wA-G0; Sun, 18 Dec 2011 00:00:16 +0200 Message-ID: <4EED10EF.1030108@FreeBSD.org> Date: Sun, 18 Dec 2011 00:00:15 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111206 Thunderbird/8.0 MIME-Version: 1.0 To: Adrian Chadd References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <4EECD261.2080208@m5p.com> <4EED05EC.8050103@FreeBSD.org> In-Reply-To: X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: George Mitchell , freebsd-stable@FreeBSD.org, Oliver Pinter Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 22:00:21 -0000 on 17/12/2011 23:20 Adrian Chadd said the following: > Erm, just as a random question - since device drivers (and GEOM) run > as separate threads, has anyone looked into what kind of effects the > scheduler has on these? > > I definitely have measurable throughput/responsiveness differences > between ULE and 4BSD (and preempt/non-preempt on 4BSD) on my MIPS > boards when they're bridging traffic. I wonder if there's something > strange going on with the scheduling and preemption of driver netisrs, > taskqueues, the fast interrupt handlers, etc. > > This may -not- be a userland specific problem.. That's an interesting idea. From the recent discussion about USB I can conclude that USB threads run at higher priority than GEOM threads: PI_NET/PI_DISK vs PRIBIO. The former is from the ithread range, the latter is from the regular kernel range. Maybe it would make sense to give the GEOM threads a priority from the ithread range too - given their role and importance. -- Andriy Gapon