Date: Sat, 12 Sep 1998 09:50:01 -0700 (PDT) From: "Jeffrey J. Mountin" <jeff-ml@mountin.net> To: freebsd-bugs@FreeBSD.ORG Subject: Re: misc/7873: poor initial configuration and documentation of kernel resources frustrating! Message-ID: <199809121650.JAA11077@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR misc/7873; it has been noted by GNATS.
From: "Jeffrey J. Mountin" <jeff-ml@mountin.net>
To: josh@zcompany.com, freebsd-gnats-submit@FreeBSD.ORG
Cc: freebsd-isp@FreeBSD.ORG
Subject: Re: misc/7873: poor initial configuration and documentation of
kernel resources frustrating!
Date: Sat, 12 Sep 1998 11:41:52 -0500
At 02:46 PM 9/9/98 -0700, josh@zcompany.com wrote:
>>Description:
>The capabilities database (/etc/login.conf) and the kernel resource
>limitations are very poorly documented and caused us no end of grief
>for a long time trying to extract the maximum performance from our web
>servers.
>>How-To-Repeat:
>install a stock freebsd system and naively change maxusers and the
>login class for your web server user and assume all is well :)
>>Fix:
>Before 3.0 is finalized, how about a generic "high performance" switch.
>In specific, I find that we need to do the following to keep apache from
>complaining with "resource not available errors":
>
>add these to the kernel:
>options "CHILD_MAX=1024"
>options "OPEN_MAX=1024"
>options "SOMAXCONN=4096"
>options "MAXMEM=(whatever is applicable)"
>maxusers 512
Bruce Evans has been shaking his finger at this "old style tweaking" and here is his latest:
----The following reply was made to PR kern/7831; it has been noted by GNATS.
From: Bruce Evans <bde@zeta.org.au>
To: freebsd-gnats-submit@FreeBSD.ORG, Roy.Nicholl@ASGtechnologies.com
Cc: Subject: Re: kern/7831: Inclusion of kernel variable SOMAXCONN as a configurable option
Date: Fri, 4 Sep 1998 21:06:22 +1000
>>Description:
>In certain situations [i.e. high volume proxy server] it is necessary to adjust the value of
>SOMAXCONN upward from its default of 128.
Use the kern.ipc.somaxconn variable. SOMAXCONN shouldn't even be visible
(since it it just misleading if someone has changed kern.ipc.somaxconn).
Bruce
----
I'm not running -current, but I have 3 of the 4 options to change with login.conf and can recompile with a higher maxusers as needed. While 2.2.7 may not have the kern.ipc.somaxconn, a 'sysctl -a' and only that will list kern.somaxconn=X.
>and give the user the web server runs as the login class "root"
>
>These are poorly documented, and given the amount of floundering I've
>seen in the mailing list, I'd say a TON of other people run into these
>problems. I don't even know if there is more I should do to get better
>performance... ??? No documentation I've found in several months of
>searching, so how am I to know?
Don't know about a "TON", but been there done that:
from login.conf(5)
These resource limit entries actually specify both the maximum and cur-
rent limits (see getrlimit(2)). The current (soft) limit is the one nor-
from getrlimit(2)
SEE ALSO
csh(1), quota(1), quotactl(2), sigaltstack(2), sigvec(2), sysctl(3)
My beef would be there is no reference for day to day tools like sysctl limits, at least the man page for limits references login.conf
from limits(1)
SEE ALSO
csh(1), env(1), limit(1), sh(1), getrlimit(2), setrlimit(2),
login_cap(3), login.conf(5)
Grrrraaaahhhh!
I'm used to man page hopping to find what I want and even resorted to checking /usr/includes for hints (almost started in the source code), but this was impossible until another old message from Bruce Evans (re bin/5306) tipped me off that there is no re ason to tweak the kernel, so I went hunting.
>FreeBSD really needs a way to easily get full performance from the
>system. I personally don't mind the kernel taking up the extra memory,
>it's a lot bigger hassle to figure out how to keep the stupid thing
>from crashing and how to keep apache from running into resource
>limitations.
>
>I'm not trying to be an ass, but I really think that this is a big
>problem with FreeBSD. I almost punted on FreeBSD and went the route
>of getting all of our CGI working under Linux because I couldn't get
>the OS to simply run at it's full potential, and I really think this
>should be treated as a bug. All operating systems have issues like
>this (Linux and file descriptors/process, for instance), but to see
>FreeBSD have more built in and undocumented restrictions and limitations
>than Windows NT is really depressing :)
Not to rant and rave about the issue, but I agree. There really isn't any documentation for building up a high traffic/load/processes web server, only the lists and the rules of tweaking have changed and I still don't have some things straight regarding the login.conf thing.
The first mention of login.conf was in 2.2.2 and it bit me right after upgrading to 2.2.5, which wasn't fun. Not sure what exactly changed and I don't plan to install 2.2.2 on a machine to find out exactly. What I do recall is that I was a bit perturbe d by this sudden change in behaviour. Whether classes were enforced or the limits for the default or root dropped, couldn't spawn and cannot forks errors started showing up in the thousands in Apache's logs. It never crashed, but I think the limits may have contributed to the delinquency of Apache's children, which only occured during peak times. This has been a periodic issue for myself since 2.2.5 and compiling in gdb (as suggested by Gary Palmer) didn't help when the process promptly exited upon c onnection. Since tweaking things like below there have no more runaways and my speculation is that the file descriptor limit was the main factor, since it was the most recently checked and adjusted.
Login.conf is just a hassle for serious servers. Must admit that average joe won't have any problems and those that do are better equiped to figure it out, but it's still a pain.
(This should part should probably drift over to ISP as well, so I'm CC'ing)
Since I have some time, I think a Virtual Hosting tutorial would be an idea. Plenty of tweaking has been done on the new server I built for a client. Here are some snipits.
The kernel was initally built with:
maxusers 128
options CHILD_MAX=256
options OPEN_MAX=256
options SOMAXCONN=256
options NMBCLUSTERS=4096
Apache started generating spawn and fork errors after it's first weekend up and the traffic started building to weekday levels. Duh! Forgot about /etc/login.class and the root class (since Apache is started as root).
root:\
:path=~/bin /bin /sbin /usr/bin /usr/sbin /usr/local/bin /usr/local/sbin:\
:cputime=infinity:\
:datasize=infinity:\
:stacksize=infinity:\
:memorylocked=infinity:\
:memoryuse=infinity:\
:filesize=infinity:\
:coredumpsize=infinity:\
:openfiles=infinity:\
:maxproc=infinity:\
:memoryuse-cur=32M:\
:maxproc-cur=64:\
:openfiles-cur=1024:\
:priority=0:\
:requirehome@:\
:umask=022:\
:tc=auth-root-defaults:
Both memoryuse-cur and maxproc-cur were double to see if problem still existed. No effect, so openfiles-cur was doubled and that fixed it.
After finding out about sysctl and checking values the options that were added to the kernel were removed. All is still well, but I noticed at some time in the future the openfiles value might need to be increased. Recompiled a kernel on a development server and compared values:
maxusers=128
kern.maxproc: 2068
kern.maxprocperuid: 2067
kern.maxfiles: 4136
kern.maxfilesperproc: 4136
maxusers=256
kern.maxproc: 4116
kern.maxprocperuid: 4115
kern.maxfiles: 8232
kern.maxfilesperproc: 8232
That should allow for plenty of growth. The current peak for file descriptors reached 1680 total with 1509 for user www alone with 7 PERL processes running.
Other values that may be of concern, but not changable in 2.2.7 without recompiling are:
kern.somaxconn: 128
kern.maxsockbuf: 262144
Not sure if the 2nd is related to the NMBCLUSTERS option. Anyone?
The peak that I have seen for 'netstat -m' is:
104/564 mbuf clusters in use
1157 Kbytes allocated to network (20% in use)
This is definately higher than releases prior to 2.2.2 could handle and the NMBCLUSTERS option had to be used.
Other options that I have seen recommended are:
options "MAXDSIZ=(256*1024*1024)"
options "DFLDSIZ=(256*1024*1024)"
Never used either and login.conf should(?) replace these. Not they are still in LINT.
What I ended up with in login.conf was user www having it's own class:
www:\
:path=/bin /sbin /usr/bin /usr/sbin /usr/local/bin /usr/local/sbin:\
:cputime=infinity:\
:datasize=infinity:\
:stacksize=infinity:\
:memorylocked=infinity:\
:memoryuse=infinity:\
:filesize=infinity:\
:coredumpsize=infinity:\
:openfiles=infinity:\
:maxproc=infinity:\
:memoryuse-cur=64:\
:maxproc-cur=128:\
:openfiles-cur=2048:\
:priority=0:\
This easily handled 101 Apache processes in testing and 78 in real life. All pages are server parsed and there is extensive use of PERL CGI on this site. With a PPro200 and 256MB RAM it has room for growth.
Some things to note about the login.conf entry. The 3 values with -cur should NOT exceed or equal what you system has available. There should be some leeway to prevent the server from crashing. Between the amount of RAM and sysctl, enough should be le ft for core processes.
Hmmm... should tweak this a bit, the following were stripped from root's values:
:priority=0:\
:requirehome@:\
:umask=022:\
:tc=auth-root-defaults:
In some cases someone may wish to increase or decrease the priority. The umask may be a good idea for scripts that create files. User www does not have a valid home dir and requirehome is one more way to nail down no logins along with tc
auth-www-defaults:\
:auth-login=krb_skey_or_passwd,passwd,kerberos,skey:
Changing this is a bit fuzzin the man pages and takes some work to track down, but I can't seem to find it for the moment. Something like:
auth-www-defaults:\
:auth=none:
Or is none supposed to be "null" or "reject" instead? logingetstyle() isn't clear (to me at least). It would be a good idea to add an example for rejecting ALL logins to login.conf, which would be another way to lock out users...
Uh, scratch that I forgot about /sbin/nologin, but a bit more detail on login_style would be helpful. In login_cap(3) it has a nice little section under logingetstyle():
This scheme allows the administrator to determine the
types of authorisation methods accepted by the sys-
tem, depending on the means by which the access oc-
curs. For example, the administrator may require
skey or kerberos as the authentication method used
for access to the system via the network, and stan-
dard methods via direct dialup or console logins,
significantly reducing the risk of password discovery
by "snooping" network packets.
Not much in the way of examples around:
auth-root-defaults:\
:auth-login=krb_skey_or_passwd,passwd,kerberos,skey:\
:auth-rlogin=krb_or_skey,kerberos,skey:
Is there an 'auth-ftp' or 'auth-telnet' to further descriminate?
Overall I'd say the original issue and related issues are just a matter of documentation. And we all know how well that is going. :/
Any comments suggestions would be appreciated and help me work out a tutorial for kernel tweaking for a web server, which is lacking and frequently asked about on the lists. Not to mention that Apache changed a bit with the 1.3.0 release.
cheers!
Jeff Mountin - Unix Systems TCP/IP networking
jeff@mountin.net
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-bugs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199809121650.JAA11077>