Date: Tue, 5 Feb 2008 13:23:30 +0100 From: "Zbigniew Szalbot" <zszalbot@gmail.com> To: lachlan@lkla.org Cc: freebsd-questions <freebsd-questions@freebsd.org> Subject: Re: Memory Error using Mailman on FreeBSD. How to debug? Message-ID: <94136a2c0802050423k2b8ebb9es58c7619e8e9de3ef@mail.gmail.com> In-Reply-To: <1153.137.153.0.37.1202210274.squirrel@sm.lkla.org> References: <1153.137.153.0.37.1202210274.squirrel@sm.lkla.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello, Barry just answered on Mailman list saying that the memory fault may be in Python for that matter.... [quote] > Now to just work out the root cause of the memory errors ... It's important to remember that Python's email parsing code sucks the entire message text into memory and keeps large attachments in memory after parsing too. There could be a number of problems associated with this, including a memory leak that causes the raw text to be kept even after parsing, or that the attachment is just to big to be kept in memory after parsing. There have been talks over the years about how to modify the email package's api's so that it would be possible to store large attachments in a separate file outside of core memory, but this has yet to be done. Now would be a good time to look into it again, if we have any hopes of getting something like this into Python 2.6 and 3.0. However, this isn't the right place to discuss such changes -- those should happen on Python's email sig: http://www.python.org/sigs/email-sig Cheers, - -Barry [/quote] Zbigniew Szalbot 2008/2/5, Lachlan Michael <lachlan@lkla.org>: > I have a question about debugging a memory error on FreeBSD. > > When a user sends an e-mail with an attachment above about 500kB to a very > small mailing list (4 members), Mailman on my server aborts processing > with the error > MemoryError : out of memory > > After talking on the Mailman list, the advice was that python and Mailman > are just returning the fact that enough memory can't be allocated, and it > could be a FreeBSD problem. Specifically, > > > MemoryError is beyond the control of both Mailman and Python. Python is > > requesting additional memory form the OS for the process and is being > > denied. This is some kind of hardware or OS limitation. It could be > > that the OS itself is running out of real or virtual memory or the > > process is running into some process limit. This is a FreeBSD question. > > and some sympathy in that > > > 500KB is not a big message. This should not be happening. I am only > > guessing, but I think there must be some per-process memory limitation > > in your OS that is set way too low. > > I have run memtest previously and found no problems, also no problems when > upgrading ports or world, so I believe the RAM is ok. The machine is > otherwise lightly loaded. > > Although I believe I have a very vanilla flavored FreeBSD machine, I > probably have some setting or parameter incorrectly set. Any hints on how > to debug this problem and where to look at? > > Machine Details (i386, 1G RAM) > ========================== > # sysctl -a | grep version > kern.version: FreeBSD 6.3-PRERELEASE #0: Wed Jan 2 18:46:42 JST 2008 > # sysctl -a | grep mem > memdesc 1 4K - 1 4096 > vm.kmem_size: 335544320 > vm.kmem_size_max: 335544320 > vm.kmem_size_scale: 3 > > hw.physmem: 1052151808 > hw.usermem: 881700864 > hw.realmem: 1065287680 > > # limits > Resource limits (current): > cputime infinity secs > filesize infinity kB > datasize 524288 kB > stacksize 65536 kB > coredumpsize infinity kB > memoryuse infinity kB > memorylocked infinity kB > maxprocesses 5547 > openfiles 11095 > sbsize infinity bytes > vmemoryuse infinity kB > > For memory problems it seems that vmstat is used, so I'll include output > when the problem occurs (i.e. sending a large attachment) and also when > sending a normal mail to exactly the same list. As far as I can see, there > seems to be plenty of physical memory available in both cases. > > vmstat when problem occurs* (receiving large attachment in mail) > ======================================================== > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ad4 cd0 in sy cs us sy id > 1 3 0 355056 225104 0 0 0 0 0 0 0 0 137 275 344 0 1 99 > 0 3 0 355056 225104 127 0 0 0 103 0 0 0 154 444 379 0 3 97 > 0 3 0 355056 225104 93 0 0 0 69 0 0 0 157 419 387 0 2 98 > 0 3 0 355056 225104 127 0 0 0 103 0 0 0 165 459 404 1 1 99 > 0 3 0 355056 225100 186 0 0 0 138 0 3 0 185 543 451 0 1 99 > 0 3 0 355056 225100 127 0 0 0 103 0 0 0 196 465 467 0 3 97 > 0 3 0 356044 224556 341 0 0 0 138 0 0 0 203 980 482 0 3 97 > 0 3 0 356068 224332 141 0 0 0 103 0 1 0 643 1775 1356 0 > 1 99 > 0 3 0 356068 224080 0 0 0 0 0 0 2 0 714 1673 1504 1 > 1 97 > 0 3 0 356068 223872 127 0 0 0 103 0 1 0 729 1854 1527 1 > 4 95 > 0 3 0 356068 223664 0 0 0 0 0 0 2 0 735 1675 1545 0 > 2 98 > 0 3 0 355060 223340 2850 0 0 0 2898 0 38 0 307 5751 1055 13 > 6 81 > *1 3 0 355140 223168 11192 0 0 0 10850 0 6 0 244 17322 821 > 63 17 20 > 0 3 0 354688 223524 917 0 0 0 981 0 3 0 196 668 473 1 4 95 > 0 3 0 354688 223524 93 0 0 0 69 0 0 0 162 407 399 0 1 99 > 0 3 1 354688 223524 34 0 0 0 34 0 0 0 166 312 415 0 1 99 > 1 3 0 354688 223524 93 0 0 0 69 0 0 0 163 423 400 0 2 98 > 0 3 0 354688 223524 34 0 0 0 34 0 0 0 149 324 372 0 1 99 > 0 3 0 354688 223524 0 0 0 0 4 0 7 0 166 271 411 0 1 99 > 0 3 0 354688 223524 34 0 0 0 34 0 0 0 170 305 410 1 1 98 > > vmstat output when no problem** doing the same task > ============================================== > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ad4 cd0 in sy cs us sy id > 1 3 0 354688 224360 211 0 0 0 181 1 0 0 126 544 335 2 1 98 > 0 3 0 354688 224360 96 0 0 0 69 0 0 0 147 412 365 1 1 98 > 0 3 0 354688 224360 127 0 0 0 103 0 0 0 146 447 367 0 1 99 > 0 3 0 354688 224360 186 0 0 0 138 0 0 0 154 598 383 0 1 99 > 0 3 0 354688 224360 127 0 0 0 103 0 2 0 162 462 400 1 4 96 > 0 3 0 354688 224360 0 0 0 0 4 0 6 0 159 281 397 0 1 99 > 0 2 1 354688 224360 318 0 0 0 247 0 0 0 171 688 425 0 1 99 > 0 2 1 354688 224360 93 0 0 0 69 0 0 0 169 409 411 1 1 98 > 0 3 0 354688 224360 127 0 0 0 103 0 0 0 170 464 427 0 2 98 > 0 3 0 354688 224360 186 0 0 0 138 0 0 0 151 548 374 1 1 98 > 0 3 0 354688 224360 127 0 0 0 103 0 0 0 150 445 375 0 2 98 > 0 3 0 354688 224360 93 0 0 0 69 0 0 0 144 403 356 0 1 99 > 0 3 0 354688 224360 34 0 0 0 34 0 4 0 156 323 401 0 2 98 > 0 3 0 354688 224360 186 0 0 0 138 0 0 0 178 555 425 0 2 98 > 0 3 0 354688 224360 127 0 0 0 103 0 0 0 171 456 417 1 1 97 > 0 3 0 354688 224360 93 0 0 0 69 0 0 0 175 406 417 1 1 98 > 0 3 0 354688 224344 1775 0 0 0 1614 0 18 0 246 5384 649 6 > 5 89 > 1 4 0 366760 214072 2959 0 0 0 293 0 22 0 260 8265 698 35 4 61 > **1 4 0 396320 186384 7154 0 0 0 203 0 8 0 248 13599 615 87 > 13 0 > 0 4 0 398500 184404 683 0 0 0 148 0 8 0 181 1634 439 53 3 44 > 0 4 0 398500 184396 37 0 0 0 35 0 0 0 170 347 407 0 2 98 > 0 4 0 398536 184376 98 0 0 0 69 0 0 0 168 537 411 0 1 99 > 0 4 0 398536 184376 128 0 0 0 104 0 0 0 156 520 393 0 1 99 > 0 4 0 398536 184376 186 0 0 0 138 0 0 0 159 536 389 1 1 98 > 1 4 0 398536 184376 128 0 0 0 104 0 3 0 166 448 411 0 1 99 > 0 4 0 398536 184376 93 0 0 0 69 0 0 0 151 414 372 0 1 99 > 0 4 0 398536 184376 128 0 0 0 104 0 0 0 153 437 373 1 1 98 > 0 4 0 398536 184376 186 0 0 0 138 0 0 0 157 550 393 0 3 97 > 0 4 0 398544 184368 130 0 0 0 104 0 0 0 178 545 428 1 1 97 > 0 4 0 399100 184184 61 0 0 0 0 0 0 0 163 379 397 0 1 99 > > Regards, > > Lachlan > > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?94136a2c0802050423k2b8ebb9es58c7619e8e9de3ef>