Date: Mon, 24 Jun 2002 14:52:21 -0700 (PDT) From: Patrick Thomas <root@utility.clubscholarship.com> To: Terry Lambert <tlambert2@mindspring.com> Cc: Nielsen <nielsen@memberwebs.com>, <hackers@freebsd.org> Subject: Re: (jail) problem and a (possible) solution ? Message-ID: <20020624143448.J68572-100000@utility.clubscholarship.com> In-Reply-To: <3D178E89.AB893E72@mindspring.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> It's obvious that you are running a large number of httpd's; the
Yes, we are running a lot of httpd's:
ps auxw | grep httpd | wc -l = 288
> The way to cross-check this would be to run a continuous "netstat -m",
> e.g.:
Funny you should ask :) I was already doing that. Here is the output
from a `netstat -m` run once per minute - the machine crashed sometime in
the next 30-60 seconds after I got this output:
524/2576/34816 mbufs in use (current/peak/max):
500 mbufs allocated to data
24 mbufs allocated to packet headers
273/2254/8704 mbuf clusters in use (current/peak/max)
5152 Kbytes allocated to network (19% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
> Basically, if you have any denials, or if the number of mbuf
> clusters gets really large, then you could have a problem.
Do you think it is reasonable that the above netstat -m output could,
within 30 or so seconds, ramp up to the bad situation you are describing ?
Because it looks fairly benign to me...
I have three questions:
1. Forgetting about my paticular problem for a moment, let's say you have
to tune a machine to run 200+ httpd servers along with another 800 misc.
processes, etc. What do you suggest setting, just to be safe (again, as a
precaution - forgetting that in reality I am tryig to fix a sick machine)
So far I have only tuned:
In my kernel: maxusers=256 (was 512, change to 256 didn't help)
options SHMMAXPGS=16384
options SHMMAX="(SHMMAXPGS*PAGE_SIZE+1)"
options SHMSEG=256
options SEMMNI=384
options SEMMNS=768
options SEMMNU=384
options SEMMAP=384
(all this SHM and SEM stuff is to run multiple postgres')
and at boot time:
sysctl -w jail.sysvipc_allowed=1
sysctl -w kern.ipc.shmall=65535
sysctl -w kern.ipc.shmmax=134217728
sysctl -w net.inet.tcp.syncookies=0
So anything obvious I am missing that you would tune for a 200+ http + 800
other processes machine?
2. Let's say I was being targeted by that effective attack you spoke
of...any way to immunize myself ?
3. You spoke of:
> # sysctl -a | grep tcp | grep space
> net.inet.tcp.sendspace: 32768
> net.inet.tcp.recvspace: 65536
>
> I guess the best way to deal with this would be to drop the size
> of the send or receive queues, until it didn't consume all your
> memory. In general, the size of these queues is supposed to be
> a *maximum*, not a *mean*, so the number of sockets possible,
> times the maximum total of both, will often exceed the amount of
> available mbuf space.
a) are you saying to collect these sysctls regularly and
try to see their values right at the crash ?
b) where do I "drop the size of the send or receive queues" ?
(sysctl or kernel setting?)
thank you very much. I will try to get a full `ps` tonight when it
crashes again :(
--PT
>
> An interesting attack that is moderately effective on FreeBSD
> boxes is to send with a very large size, and not send one of
> the fragments (e.g. the second one) to prevent fragment
> reassembly, and therefore saturate the reassembly queue. The
> Linux UDP NFS client code does this unintentionally, but you
> could believe that someone might be doing it intentionally,
> as well, which would also work against TCP. It's doubtful that
> you are being hit by a FreeBSD targetted attack, however.
>
> -- Terry
>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020624143448.J68572-100000>