Date: Tue, 19 Dec 2000 07:21:00 +1300 From: "Dan Langille" <dan@langille.org> To: Joseph Scott <joseph@randomnetworks.com> Cc: freebsd-hackers@FreeBSD.ORG, freebsd-ports@FreeBSD.ORG Subject: Re: processing incoming mail messages (FreshPorts 2) Message-ID: <200012181820.HAA18715@ducky.nz.freebsd.org> In-Reply-To: <Pine.BSF.4.21.0012181119070.71411-100000@mothra.ecs.csus.edu> References: <200012161822.HAA03654@ducky.nz.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 18 Dec 2000, at 11:22, Joseph Scott wrote: > > On Sun, 17 Dec 2000, Dan Langille wrote: > > # Which list would be more appropriate for asking advice on designing a > # mail processing strategy for FreshPorts 2 (i.e. processing all of > # cvs-all, not just the ports)? > # > # I'm looking for recommendations and guidance on how to capture the > # incoming messages and process them one at a time. As opposed to > # starting a separate perl script for each message (which is the the > # existing strategy and is usually fine, except when large numbers of > # messages turn up in a short period of time). > > If you don't want to process a message the instant it comes in > (via feeding it to a perl script or what ever) you'll need to setup some > sort of queue, then have a cron job come through and process the queue. Thanks. Since posting my original message, the proposed design has drifted to this strategy (which I posted to the FreshPorts develop list last night): On Mon, 18 Dec 2000, Dan Langille wrote: > ok folks, brain pick time: I'm thinking about redoing the FreshPorts > mail-database interface. At present, each incoming email from cvs-all > initiates a separate perl script. Not nice if you get say 100 emails all > in 1 minute for some reason. Now I'm thinking of using procmail to > dump each email to a separate file on disk. Then processing each > file thus freeing up the MTA ASAP. Sound good so far? > > Suggestions? Ideas? As for processing the files once they are on disk, I like the idea of processing the messages ASAP. Therefore, I'd like to have a daemon / script sitting there running all the time. This process is notified when a new message arrives (in the above case, that would be part of the procmail delivery process). This process would then "wake up" and process all available message files one at a time. Perhaps moving each processed file to an archive. When there are no more message files, it would stop and wait to be notified again. > If the problem is load then another approach would be to heavily > nice(1) the perl script the is launched when a commit mail comes in. That's already done. It's just the volume which can occur. If a large number of messages arrive at once. Starting up 50 perl jobs on the box can stress it a bit. It also makes better sense to process the messages in the order in which they arrive rather than concurrently. A side effect will be less change of database lockouts or conflicts during updates. cheers -- Dan Langille The FreeBSD Diary - http://www.freebsddiary.org/ NZ ADSL - http://www.unixathome.org/adsl/ NZ Broadband - http://www.unixathome.org/broadband/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-ports" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200012181820.HAA18715>