Date: Thu, 15 Apr 1999 15:42:08 -0700 From: "David Schwartz" <davids@webmaster.com> To: "Daniel C. Sobral" <dcs@newsguy.com> Cc: <chat@FreeBSD.ORG> Subject: RE: swap-related problems Message-ID: <000201be8791$31c7add0$021d85d1@whenever.youwant.to> In-Reply-To: <371664D4.5337858C@newsguy.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> First, all this discussion centers on a program P being able to > allocate all available memory. If a program doesn't try to do that, > it will never run into a malloc(3) returning NULL, pre-allocate or > not. Are we agreed on this? What does malloc returning NULL have to do with anything? *sigh* We're talking about a well-behaved program, P, that gets killed because of overcommitting caused by some combination of its behavior and the behavior of other processes. > Second, run program P, using pre-allocation. At some point in the > execution of P, all memory will allocated to it, and there will be > no more free memory available. Correct? No. It's possible that P will allocate all the memory it needs at startup, and it will be a very small amount. What do you mean by "all memory will allocated to it"? > Ok, let's assume that program P will *not* get killed under any > circunstances. That is not true, but let's suppose we changed > FreeBSD to make it so. > > Well, we have a lot of programs running, none of them using > pre-allocation. Theoretically, it would be possible that none of > them ever demanded any more memory. Please, convince yourself that > this is very unlikely. > > We can safely assume that a program is bound to request memory it > hasn't used before. At the very least, there are all sorts of > processes getting spawned automatically to do all sorts of things. > There are data being read and written to disk, there are networks > packets being received and transmitted. There are cron jobs running. > There is syslogd. Etc. > > So, something needs to get killed. Is that something essential to > you? Is syslogd essential? inetd? Maybe the terminal login asking > for password on your console, or your shell? Well, protect all > essential processes from getting killed. It is false that something needs to get killed. It's entirely possible that had the operating system anticipated this situation and handled it more smoothly, say by failing to fork or failing to malloc, other well-behaved programs would have reduced their memory load. The idea is to make it possible for well behaved programs to avoid this situation by anticipating it earlier, the moment the operating system began to overcommit. For example, one of the programs written by the company I work for has an internal memory tracker. We pre-allocate a few megabytes of memory to use as an emergency pool. If the operating system returns NULL when we call malloc, we defer memory-intensive tasks for later. In extreme cases, we may refuse to accept new incoming connections. Because of that, we can avoid running into the situation where something needs to be killed. The problem is, if the operating system incorrectly assumes that every malloc or fork is vital, it sets itself up for a situation later where a copy-on-write will result in a process needing to be killed. This need is solely the result of the operating system causing mallocs and forks to succeed in the past where their failure may not have been fatal to anything. You are assuming that processes can do better than the kernel on a failed allocation. This is so obviously false that I can't believe that you are even advancing it. > So, what we have now? Non-essential processes, which can get killed. > Kill all of them. The remaining memory may or may not be enough. > Maybe P was run in a particular low memory use situation, and there > simply isn't enough space to run all essential processes. Let's make > this easy, though. Assume that killing the non-essential processes > gives you _at least_ the exact amount of memory you need to be able > to run without any essential process getting killed. Let's call > these non-essential processes N. > > Is this a sufficient solution? Yes. But there is a catch. > > You'll notice that you *need* to run N before running P. If N is not > running at the time you run P, then you'll end up killing essential > processes. If you can't see why, go back to the paragraph were I > defined N, and think of possible alternate scenarios, and see where > they got you. > > Well, you should realize then that you need to discover a set of > processes N, which doesn't do anything essential to you, which you > must run before running P. This is the same thing as defining a > maximum size for P, only harder. > > Got it now? No. A process can handle a failed malloc or fork by doing something much less drastic than death. The kernel cannot. DS To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-chat" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?000201be8791$31c7add0$021d85d1>