Date: Fri, 8 Feb 2008 07:55:05 -0800 From: Mark Sapiro <mark@msapiro.net> To: Alex Zbyslaw <xfb52@dial.pipex.com>, lachlan@lkla.org Cc: freebsd-questions@freebsd.org Subject: Re: Memory Error using Mailman on FreeBSD. How to debug? Message-ID: <PC1760200802080755050187c2f9c880@msapiro> In-Reply-To: <47AC4E08.1060801@dial.pipex.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Alex Zbyslaw wrote: > >You are also getting a stack trace from python when it exits with the >"out of memory" error. ktrace is just showing python printing the stuff >- it may be that the error also ends up in a log file somewhere - don't >know where mailman logs, sorry. From that stack trace it should be >possible to figure out which line of the python is actually causing that >memory request. My bet is on one of the cPickle lines, but it would be >nice to see the stack trace "raw" so to speak. Maybe that stack trace >would help someone on the mailman list suggest something else. Hi, I'm the Mailman guy on this thread. >From the errors that were reported on mailman-users@python.org, there are two different manifestations that we've seen, resulting in different tracebacks (logged in Mailman's 'error' log). Here's the flow. With a message of 1.5 to 2 MB in size, the message is piped from the MTA to Mailman's 'post' script which creates a queue entry which is a file containing two python pickles. The first just contains the raw message text and the second is a small bit of metadata about the message/entry. This process which is a pipe spawned by the MTA works. Next, Mailman's IncomingRunner picks up the queue entry and hits the MemoryError exception in the call to cPickle to read the file and unpickle the raw message. This is before it has done any processing of the message, so the MemoryError is only related to unpickling the raw text if the text is large enough. With a message of 300 to 500 KB, the process is the same, but the unpickling of the queuue entry into message text and metadata succeeds. The MemoryError occurs in this case in the Python email library as it is parsing the message text into an email message object. This process is known to be a memory hog in that at some point the entire raw message text and the full message object (which is at least as big as the message text) will be resident at the same time. I.e., it takes more than twice as much memory to parse a message as to store it. Note that these messages are not really large. Most Mailman installations can handle much larger messages without problem. >Did you already try sending a different kind of attachment that's the >same kind of size (a bit bigger would be better). Maybe it's something >about the attachment itself that's causing the issue? In the first case above, it is almost certainly just the size of the message text. I'm certain the same thing would occur with a simple text/plain message if it had a 1.5MB body. >As a final resort, if none of the above resolves or leads to clues, I >would try uninstalling python2.5 and installing python2.4 *just in >case*. I'm assuming that you only have python for mailman. (If you >have real python users then it's trickier. You can install multiple >versions of python but possibly not from ports. But python always >compiled cleanly from tarball on FreeBSD for me. I can offer some help >with that process if you really need it). That would be worth a try. There could be a problem in the underlying Python. I don't know how FreeBSD packages Mailman, but the source distribution installs a pythonlib directory in Mailman's tree with its own Python email library which is compatible with Python back to 2.1, so if this is the case with FreeBSD, downgrading Python wouldn't change the email packege, but it still might help. >I can't help thinking that 500Kb is a very small attachment and I can't >really see why it would legitimately cause a request for so much memory >that your settings aren't handling it. Agreed. >A quick look at the mailman web site shows that you can run qrunner from >the command line - couldn't immediately find the man page though. bin/qrunner --help >If >you could somehow queue up the email with Mailman switched off, you >could run qrunner by hand and then you'd definitely get the python >backtrace. Maybe the mailman list, or a mailman admin here, can help >with that, if you need it. We see the Python traceback, so we don't need to do that. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?PC1760200802080755050187c2f9c880>