Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 2 Dec 2007 00:37:32 +0300
From:      Alexey Vlasov <renton@df.ru>
To:        freebsd-stable@freebsd.org
Subject:   Re: 2 x quad-core system is slower that 2 x dual core on FreeBSD
Message-ID:  <20071201213732.GA16638@cannabis.dataforce.net>

next in thread | raw e-mail | index | archive | help

--SUOF0GtieIMvvwua
Content-Type: text/plain; charset=koi8-r
Content-Disposition: inline

Hi,

It seems that I'm not the only one who faced the problem that FreeBSD is
non productive on multiprocessors platforms.

I use OS Linux on my hosting for web-servers, base for all servers is
the same m/b S5000PAL ( SR1500), 2 quad kernel cpu  Xeon E5320 or E5345,
8Gb RAM. I decided to install FreeBSD 6.2 i386 on one of the servers,
and the result was totally non productive.
Used software,  Apache 2.2.6 (worker) as frontend Proxy, backend Apache
2.2.3 (prefork)

By the time we understood,that something was wrong with FreeBSD, there
had already been placed about 10 sites with high-capacity and about
hundred of usual ones. And this was the limit for FreeBSD. It came along
with a great amount of Context Switches, about hundred thousands.
I attached the log what was then happening with FreeBSD.
After playing with ab (ApacheBenchmark) options, it turned out that even
with the following options you can totally down the server:

./ab -n 100 -c 20 -t 30 http://somesite-freebsd.com

I copied at the same time somesite.com (php scripts) to Linux server,
launched ab with the same options, and saw that it has no influence on
work of the server. (And by the way there work about 1.5 virtual hosts
on that server)

All options for Apache on Linux and FreeBSD are the same:


FreeBSD:

This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking somesite-freebsd.com (be patient)
Finished 29 requests


Server Software:        Apache/2.2.3
Server Hostname:        somesite-freebsd.com
Server Port:            80

Document Path:          /ab/
Document Length:        41450 bytes

Concurrency Level:      20
Time taken for tests:   30.44765 seconds
Complete requests:      29
Failed requests:        22
   (Connect: 0, Length: 22, Exceptions: 0)
Write errors:           0
Total transferred:      1529557 bytes
HTML transferred:       1513497 bytes
Requests per second:    0.97 [#/sec] (mean)
Time per request:       20720.527 [ms] (mean)
Time per request:       1036.026 [ms] (mean, across all concurrent
requests)
Transfer rate:          49.69 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0 1760 1503.4   3002    3002
Processing:   866 13328 9460.5  13853   26246
Waiting:      139 2286 2319.0   1129    6764
Total:        871 15089 10642.6  16855   29248

Percentage of the requests served within a certain time (ms)
  50%  16705
  66%  22670
  75%  25439
  80%  26342
  90%  29160
  95%  29188
  98%  29248
  99%  29248
 100%  29248 (longest request)


Linux: (the same site)
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking linux.1gb.ru (be patient)
Finished 814 requests

Server Software:        Apache/2.2.3
Server Hostname:        somesite-linux.com
Server Port:            80

Document Path:          /ab/
Document Length:        41451 bytes

Concurrency Level:      20
Time taken for tests:   30.3216 seconds
Complete requests:      814
Failed requests:        759
   (Connect: 0, Length: 759, Exceptions: 0)
Write errors:           0
Non-2xx responses:      1
Total transferred:      34430291 bytes
HTML transferred:       34126461 bytes
Requests per second:    27.13 [#/sec] (mean)
Time per request:       737.180 [ms] (mean)
Time per request:       36.859 [ms] (mean, across all concurrent
requests)
Transfer rate:          1120.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   214  725 575.2    557    5001
Waiting:       41  265 376.0    131    3280
Total:        214  725 575.2    557    5001

Percentage of the requests served within a certain time (ms)
  50%    557
  66%    716
  75%    863
  80%    967
  90%   1398
  95%   1749
  98%   2529
  99%   3064
 100%   5001 (longest request)

# cat /etc/sysctl.conf
security.bsd.see_other_uids=0
kern.maxfiles=204800
kern.maxfilesperproc=202400

kernel:
machine         i386
cpu             I686_CPU
ident           F1RNT1

options         PAE
options         SMP

options         SCHED_4BSD
options         PREEMPTION
options         INET
options         FFS
options         SOFTUPDATES
options         UFS_ACL
options         UFS_DIRHASH
options         NULLFS
options         MD_ROOT
options         CD9660
options         PROCFS
options         PSEUDOFS
options         GEOM_GPT
options         GEOM_LABEL
options         GEOM_MIRROR
options         COMPAT_43
options         COMPAT_FREEBSD4
options         COMPAT_FREEBSD5
options         SCSI_DELAY=5000
options         KTRACE
options         SYSVSHM
options         SYSVMSG
options         SYSVSEM
options         _KPOSIX_PRIORITY_SCHEDULING
options         KBD_INSTALL_CDEV
options         ADAPTIVE_GIANT

options         LIBICONV
options         NETSMB
options         NETSMBCRYPTO
options         LIBMCHAIN
options         SMBFS

options         IPFIREWALL
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=1000
options         IPFIREWALL_DEFAULT_TO_ACCEPT
options         IPFIREWALL_FORWARD
options         IPDIVERT
options         IPSTEALTH
options         TCPDEBUG
options         DUMMYNET

options         IPSEC
options         IPSEC_DEBUG

options         ACCEPT_FILTER_DATA
options         ACCEPT_FILTER_HTTP

# cat /etc/make.conf
CPUTYPE?=nocona

CFLAGS=-O2 -pipe
CXXFLAGS+=-fconserve-space
COPTFLAGS=-O2 -pipe
NO_CPU_CFLAGS=false
NO_CPU_COPTFLAGS=false
NO_MODULES=

--
BRGDS. Alexey Vlasov.

--SUOF0GtieIMvvwua
Content-Type: text/plain; charset=koi8-r
Content-Disposition: attachment; filename="logs.txt"

last pid: 31039;  load averages: 51.33, 26.31, 18.09
up 15+22:58:54  16:47:30
692 processes: 80 running, 608 sleeping, 2 stopped, 1 zombie, 1 lock
CPU states:  2.4% user,  0.0% nice, 89.1% system,  1.6% interrupt,  6.9%
idle
Mem: 1587M Active, 5614M Inact, 416M Wired, 369M Cache, 112M Buf, 14M
Free
Swap: 16G Total, 46M Used, 16G Free

PID USERNAME       THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
31027 w_istok2_d424    1  -4    0 76240K 36532K ufs    4   0:00  9.74% httpd
30826 w_nova-i_7638    1  -4    0 79340K 40116K ufs    4   0:02  7.78% httpd
31025 w_shop2y_b290    1  -4    0 76212K 36512K RUN    7   0:00  7.70% httpd
28349 w_urtest_7dc3    1  -4    0 79616K 39496K RUN    4   1:27  7.32% httpd
30801 w_new-vw_5e17    1  -4    0 76092K 37184K ufs    5   0:02  7.25% httpd
30914 w_avtolo_3761    1  -4    0 76316K 37304K RUN    7   0:01  7.18% httpd
29952 w_svekon_9de8    1  -4    0 77976K 42916K RUN    6   0:11  7.04% httpd
29875 w_lancer_12cf    1  -4    0 79584K 41408K RUN    6   0:12  6.94% httpd
29818 w_iprit-_be41    1  -4    0 77340K 39880K RUN    6   0:13  6.92% httpd
30811 w_dfase__508a    1  -4    0 76456K 37928K RUN    4   0:02  6.91% httpd
30200 w_iprit-_be41    1  -4    0 76292K 37260K ufs    1   0:07  6.86% httpd
30445 w_new-vw_5e17    1 119    0 77048K 40196K CPU3   0   0:04  6.74% httpd
30025 w_forum-_89b3    1  -4    0 76824K 41240K RUN    4   0:10  6.71% httpd
30687 w_new-vw_5e17    1  -4    0 76232K 38616K RUN    1   0:02  6.69% httpd
30561 w_avtolo_3761    1 105    0 76512K 39500K select 2   0:03  6.67% httpd
30519 w_new-vw_5e17    1  -4    0 76784K 39396K ufs    4   0:04  6.62% httpd
31018 w_tucson_bd87    1  -4    0 76000K 36464K RUN    6   0:00  6.62% httpd
30867 w_as-vid_ac7e    1  -4    0 76356K 37452K RUN    6   0:01  6.60% httpd
30997 w_rfscla_06fc    1  -4    0 76820K 38652K RUN    2   0:01  6.60% httpd
31015 w_tucson_bd87    1  -4    0 76024K 36908K RUN    1   0:00  6.59% httpd
30532 w_shop2y_b290    1  -4    0 76904K 38232K ufs    4   0:04  6.55% httpd
30936 w_tucson_bd87    1  -4    0 77836K 39740K RUN    6   0:01  6.53% httpd
30376 w_vannay_4606    1  -4    0 78252K 41596K RUN    1   0:05  6.53% httpd
30452 w_new-vw_5e17    1  -4    0 77356K 40124K RUN    6   0:04  6.50% httpd
30201 w_berega_3091    1  -4    0 81636K 42992K RUN    4   0:07  6.46% httpd
30840 w_forum-_89b3    1  -4    0 76412K 38752K RUN    4   0:02  6.44% httpd
30809 w_rm-for_636d    1  -4    0 76148K 38028K RUN    6   0:02  6.43% httpd
30700 w_ott-gu_2def    1  -4    0 76888K 39272K RUN    4   0:02  6.40% httpd
29971 w_mobi-m_2e46    1  -4    0 76872K 41484K RUN    4   0:05  6.40% httpd
31016 w_wplane_ebed    1  -4    0 76320K 37056K RUN    7   0:00  6.39% httpd
30839 w_lyubar_f1e2    1  -4    0 77332K 39788K RUN    4   0:02  6.36% httpd
31004 w_digitr_77d4    1  -4    0 76016K 36656K CPU5   5   0:01  6.36% httpd

# vmstat

procs      memory      page                    disks     faults      cpu
 r b w     avm    fre  flt  re  pi  po  fr  sr ad4 ad6   in   sy  cs us sy id
110 9 0 10884836 384132 1167   2   2   1 1340 273   0   0   19  809 1282 13  9 78
110 10 0 10950600 382964 4530   0   0   0 3968   0   1  13 3925 35129 162137  9 87  4
131 9 0 10855252 387356 3552   0   2   0 4782   0   1   9 2505 27529 166233  5 92  3
129 11 0 10924792 379056 6723   0   4   0 4658   0   1  26 4015 29908 162047  3 94  3
122 10 1 10936140 367864 5289   0   2   0 2145   0   0  17 3788 27790 162641  3 95  3
66 55 0 10807504 378600 4236   0   4   0 6818   0   1   6 3164 26302 167760  2 95  2
128 11 0 10815488 374308 4092   0   3   0 2910   0   1  13 2962 32777 167559  2 93  5
80 42 1 10784792 377988 4592   0   3   0 5821   0   5  13 3296 28117 163950  3 94  2
64 62 0 10785528 377076 5640   0   3   0 5238   0   1   6 2890 31425 163924  3 94  3
52 77 0 10656100 386120 2994   0   1   0 4768   0   1   8 2427 23684 159985  2 95  3
70 52 0 10591528 393632 3381   0   0   0 5335   0   1   4 3010 27593 158225  3 94  3
105 9 0 10549508 402488 5426   0   0   0 7436   0   2   7 2318 31753 168856  3 94  4
77 40 0 10554208 397512 5848   0   0   0 4448   0   1   9 2275 31423 156746  3 92  5
114 9 4 10483152 403624 3369   0   0   0 4974   0   0   5 2243 27000 160225  2 93  5
109 9 0 10550332 394796 4972   0   1   0 2708   0   1  11 2515 26056 158628  3 92  5
125 9 0 10518288 423240 4256   0   3   0 11099   0   1   9 2779 28773 158967  3 93  4
118 11 0 10529144 415176 5887   4   3   0 3670   0   1  11 2595 30282 159381  2 95  3
116 9 0 10497144 418012 3407   0   1   0 3929   0   1  11 2815 24890 159056  2 93  5
119 9 0 10463764 419260 3391   0   0   0 3797   0   1   9 2558 28508 159086  2 94  3
 5 116 0 10436304 420156 4263   0   0   0 4368   0  24   8 3348 27563 159025  3 94  4
84 43 0 10563104 412824 5012   0   0   0 3630   0   2  32 2658 32205 158341  1 94  5
82 59 0 10620056 415188 6158   4   2   0 4858   0  11  11 2743 30929 159186  3 94  4
87 39 0 10453904 431328 3867   0   0   0 8894   0   1   5 2560 28903 168316  2 95  3
102 37 2 10642596 417724 6146   0   1   0 3285   0   5  11 2396 35490 158978  2 95  4
145 10 0 10470216 461180 5202   4   0   0 15462   0  77  10 2522 30525 163784  4 93  3
122 12 0 10507604 452820 4561   0   0  50 2341   0 138   6 2368 31650 158428  3 94  3
108 29 0 10669764 439152 4808   0   0   0 1294   0   4   8 3456 27089 166675  2 95  3
124 10 0 10631948 444132 6171   0   0   0 7239   0   6   6 2875 31822 167193  4 94  3


    7 users    Load 54.35 28.49 23.64                  Nov 27 17:04

Mem:KB    REAL            VIRTUAL                     VN PAGER  SWAP
PAGER
        Tot   Share      Tot    Share    Free         in  out     in out
Act  927804   34556  8233412    53740  489892 count
All 8071840  267008983194432  3143860         pages
                                                                 Interrupts
Proc:r  p  d  s  w    Csw  Trp  Sys  Int  Sof  Flt   1189 cow   16584
total
    89     9454      153k 533325086 2022  309 5066 425580 wire        1:
atkb
                                                  1499680 act
14: ata
90.0%Sys   0.6%Intr  2.9%User  0.0%Nice  6.4%Idl  5776020 inact   546
18: em0
|    |    |    |    |    |    |    |    |    |     425236 cache    31
20: ata
=============================================>>     64968 free   2001
cpu0: time
                                                          daefr  2002
cpu1: time
Namei         Name-cache    Dir-cache                2333 prcfr  2001
cpu2: time
    Calls     hits    %     hits    %                     react  2001
cpu3: time
   207699   207669  100        5    0                     pdwak  2000
cpu7: time
                                     3514 zfod            pdpgs  2001
cpu4: time
Disks   ad4   ad6   ad8              3463 ozfod           intrn  2000
cpu6: time
KB/t  16.00  4.20  6.90                98 %slo-z   114784 buf    2001
cpu5: time
tps      13     1     1              3077 tfree       229 dirtybuf
MB/s   0.21  0.00  0.01                            100000 desiredvnodes
% busy    0     1     1                             89091 numvnodes
                                                    24971 freevnodes

--SUOF0GtieIMvvwua--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20071201213732.GA16638>