From owner-svn-ports-all@freebsd.org Fri Feb 24 21:15:55 2017 Return-Path: Delivered-To: svn-ports-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1E867CEB952; Fri, 24 Feb 2017 21:15:55 +0000 (UTC) (envelope-from bofh@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9A06D1C4B; Fri, 24 Feb 2017 21:15:54 +0000 (UTC) (envelope-from bofh@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v1OLFrii062923; Fri, 24 Feb 2017 21:15:53 GMT (envelope-from bofh@FreeBSD.org) Received: (from bofh@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v1OLFqfo062912; Fri, 24 Feb 2017 21:15:52 GMT (envelope-from bofh@FreeBSD.org) Message-Id: <201702242115.v1OLFqfo062912@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bofh set sender to bofh@FreeBSD.org using -f From: Muhammad Moinur Rahman Date: Fri, 24 Feb 2017 21:15:52 +0000 (UTC) To: ports-committers@freebsd.org, svn-ports-all@freebsd.org, svn-ports-head@freebsd.org Subject: svn commit: r434749 - in head/mail/dbmail: . files X-SVN-Group: ports-head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-ports-all@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the ports tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Feb 2017 21:15:55 -0000 Author: bofh Date: Fri Feb 24 21:15:52 2017 New Revision: 434749 URL: https://svnweb.freebsd.org/changeset/ports/434749 Log: mail/dbmail: adopt latest fixes from git: - login_disabled option before starttls for pop3 - fix compiler warnings for GCC5 - Fix IMAP mailbox maintanence - prevent assertion in p_string_erase - improve crypt authentication, also don't segfault when spasswd is empty - simplify log_query_time duration logic - Disconnect IMAP clients if only few free FDs left - Add primary key constraint to dbmail_authlog - Rework temporary connection failures - Give sensible default for retry 120s - Add retries for binding and searching - Bump search timeout to 60s - Increase ldap timeout to 600s 10 mins - Refactor deprecated functions - Get timeout from config - Remove redundant event_assign - Remove deprecated non functioning g_mem_profile - Add definition for authldap_free - Revert inadvertent event_assign removal - Reduce failed LDAP connection for search to error - Update LDAP to non deprecated search - Clear the ldap connection - Update ldap deprecated unbind - Fix typo - Update to ldap_unbind_ext_s and remove redundant sigaction - Rebalance commit rollback - Ensure mailbox2dbmail is using Python 2 - Tidy mailbox2dbmail man page - Update description of pid file location in server man page - Boundaries fixups ordering of parts do not add newline on - Prepend headers during delivery - Allow for systems that don't use proc PR: 210274 Submitted by: fluffy Added: head/mail/dbmail/files/patch-0001-login_disabled-option-before-starttls-for-pop3 (contents, props changed) head/mail/dbmail/files/patch-0002-fix-compiler-warnings-for-GCC5 (contents, props changed) head/mail/dbmail/files/patch-0003-Fix-IMAP-mailbox-maintanence (contents, props changed) head/mail/dbmail/files/patch-0004-prevent-assertion-in-p_string_erase (contents, props changed) head/mail/dbmail/files/patch-0005-improve-crypt-authentication (contents, props changed) head/mail/dbmail/files/patch-0006-simplify-log_query_time-duration-logic (contents, props changed) head/mail/dbmail/files/patch-0007-Disconnect-IMAP-clients-if-only-few-free-FDs-left (contents, props changed) head/mail/dbmail/files/patch-0008-Add-primary-key-constraint-to-dbmail_authlog (contents, props changed) head/mail/dbmail/files/patch-0009-Rework-temporary-connection-failures (contents, props changed) head/mail/dbmail/files/patch-0010-Give-sensible-default-for-retry-120s (contents, props changed) head/mail/dbmail/files/patch-0011-Add-retries-for-binding-and-searching (contents, props changed) head/mail/dbmail/files/patch-0012-Bump-search-timeout-to-60s (contents, props changed) head/mail/dbmail/files/patch-0013-Increase-ldap-timeout-to-600s-10-mins (contents, props changed) head/mail/dbmail/files/patch-0014-Refactor-deprecared-functions (contents, props changed) head/mail/dbmail/files/patch-0015-Get-timeout-from-config (contents, props changed) head/mail/dbmail/files/patch-0016-Remove-redundant-event_assign (contents, props changed) head/mail/dbmail/files/patch-0017-Remove-deprecated-non-functioning-g_mem_profile (contents, props changed) head/mail/dbmail/files/patch-0018-Add-definition-for-authldap_free (contents, props changed) head/mail/dbmail/files/patch-0019-Revert-inadvertant-event_assign-removal (contents, props changed) head/mail/dbmail/files/patch-0020-Reduce-failed-LDAP-connection-for-search-to-error (contents, props changed) head/mail/dbmail/files/patch-0021-Update-LDAP-to-non-deprecated-search (contents, props changed) head/mail/dbmail/files/patch-0022-Clear-the-ldap-connection (contents, props changed) head/mail/dbmail/files/patch-0023-Update-ldap-deprecated-unbind (contents, props changed) head/mail/dbmail/files/patch-0024-Fix-typo (contents, props changed) head/mail/dbmail/files/patch-0025-Update-to-ldap_unbind_ext_s-and-remove-redundant-sigaction (contents, props changed) head/mail/dbmail/files/patch-0026-Rebalance-commit-rollback (contents, props changed) head/mail/dbmail/files/patch-0027-ensure-mailbox2dbmail-is-using-Python-2 (contents, props changed) head/mail/dbmail/files/patch-0028-tidy-mailbox2dbmail-man-page (contents, props changed) head/mail/dbmail/files/patch-0029-update-description-of-pid-file-location-in-server-man-page (contents, props changed) head/mail/dbmail/files/patch-0031-boundaries-fixups-ordering-of-parts-do-not-add-newline-on- (contents, props changed) head/mail/dbmail/files/patch-0032-10083-prepend-headers-during-delivery (contents, props changed) head/mail/dbmail/files/patch-0099-Allow-for-systems-that-don-t-use-proc (contents, props changed) Modified: head/mail/dbmail/Makefile head/mail/dbmail/files/patch-acinclude.m4 head/mail/dbmail/files/patch-src_clientbase.c head/mail/dbmail/files/patch-src_dbmail.h.in Modified: head/mail/dbmail/Makefile ============================================================================== --- head/mail/dbmail/Makefile Fri Feb 24 20:54:40 2017 (r434748) +++ head/mail/dbmail/Makefile Fri Feb 24 21:15:52 2017 (r434749) @@ -32,11 +32,11 @@ SIEVE_DESC= Sieve mail sorting language OPTIONS_SUB= yes INSTALL_TARGET= install-strip -USE_AUTOTOOLS= autoheader autoconf USE_LDCONFIG= ${PREFIX}/lib/dbmail USE_RC_SUBR= dbmail-pop3d dbmail-lmtpd dbmail-imapd dbmail-timsieved -USE_OPENSSL= yes -USES= gmake libtool pkgconfig shebangfix tar:bzip2 +USES= autoreconf gmake libtool pkgconfig shebangfix ssl tar:bzip2 +GNU_CONFIGURE= yes + SHEBANG_FILES= man/fixsp.pl CPPFLAGS+= -I${LOCALBASE}/include Added: head/mail/dbmail/files/patch-0001-login_disabled-option-before-starttls-for-pop3 ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/mail/dbmail/files/patch-0001-login_disabled-option-before-starttls-for-pop3 Fri Feb 24 21:15:52 2017 (r434749) @@ -0,0 +1,77 @@ +From ec74380db22e3d92641c972aa53823d02a56a5ad Mon Sep 17 00:00:00 2001 +From: Paul J Stevens +Date: Wed, 18 Feb 2015 21:26:38 +0100 +Subject: [PATCH 01/33] login_disabled option before starttls for pop3 + +--- + dbmail.conf | 5 +++++ + src/pop3.c | 16 ++++++++++++++++ + 2 files changed, 21 insertions(+) + +diff --git dbmail.conf dbmail.conf +index 946e064..bf16d16 100644 +--- dbmail.conf ++++ dbmail.conf +@@ -227,6 +227,11 @@ port = 110 + # You can set an alternate banner to display when connecting to the service + # banner = DBMAIL pop3 server ready to rock + ++# If TLS is enabled, login before starttls is normally ++# allowed. Use login_disabled=yes to change this ++# ++# login_disabled = no ++ + # + # If yes, allows SMTP access from the host IP connecting by POP3. + # This requires addition configuration of your MTA +diff --git src/pop3.c src/pop3.c +index cc0863b..d03f71a 100644 +--- src/pop3.c ++++ src/pop3.c +@@ -366,6 +366,7 @@ int pop3(ClientSession_T *session, const char *buffer) + int found = 0; + //int indx = 0; + int validate_result; ++ bool login_disabled = FALSE; + uint64_t result, top_lines, top_messageid, user_idnr; + unsigned char *md5_apop_he; + struct message *msg; +@@ -431,6 +432,15 @@ int pop3(ClientSession_T *session, const char *buffer) + } + } + ++ if (state == CLIENTSTATE_INITIAL_CONNECT) { ++ if (server_conf->ssl) { ++ Field_T val; ++ GETCONFIGVALUE("login_disabled", "POP", val); ++ if (SMATCH(val, "yes")) ++ login_disabled = TRUE; ++ } ++ } ++ + switch (cmdtype) { + + case POP3_QUIT: +@@ -459,6 +469,9 @@ int pop3(ClientSession_T *session, const char *buffer) + if (state != CLIENTSTATE_INITIAL_CONNECT) + return pop3_error(session, "-ERR wrong command mode\r\n"); + ++ if (login_disabled && ! session->ci->sock->ssl_state) ++ return pop3_error(session, "-ERR try STLS\r\n"); ++ + if (session->username != NULL) { + /* reset username */ + g_free(session->username); +@@ -478,6 +491,9 @@ int pop3(ClientSession_T *session, const char *buffer) + if (state != CLIENTSTATE_INITIAL_CONNECT) + return pop3_error(session, "-ERR wrong command mode\r\n"); + ++ if (login_disabled && ! session->ci->sock->ssl_state) ++ return pop3_error(session, "-ERR try STLS\r\n"); ++ + if (session->password != NULL) { + g_free(session->password); + session->password = NULL; +-- +2.10.1 (Apple Git-78) + Added: head/mail/dbmail/files/patch-0002-fix-compiler-warnings-for-GCC5 ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/mail/dbmail/files/patch-0002-fix-compiler-warnings-for-GCC5 Fri Feb 24 21:15:52 2017 (r434749) @@ -0,0 +1,346 @@ +From c3badbdb282a5ffb411ee2b5062ed345896fb149 Mon Sep 17 00:00:00 2001 +From: Paul J Stevens +Date: Fri, 8 Jan 2016 18:42:24 +0100 +Subject: [PATCH 02/33] fix compiler warnings for GCC5 + +--- + src/dm_db.c | 26 +++++++++++++------------- + src/dm_mailbox.c | 6 +++--- + src/dm_mailboxstate.c | 4 ++-- + src/dm_message.c | 12 ++++++------ + src/dm_misc.c | 6 +++--- + src/dm_sievescript.c | 2 +- + src/maintenance.c | 2 +- + src/modules/authsql.c | 12 ++++++------ + src/pop3.c | 2 +- + 9 files changed, 36 insertions(+), 36 deletions(-) + +diff --git src/dm_db.c src/dm_db.c +index 8d72214..feb0b34 100644 +--- src/dm_db.c ++++ src/dm_db.c +@@ -1047,7 +1047,7 @@ int db_check_version(void) + /* test existence of usermap table */ + int db_use_usermap(void) + { +- int use_usermap = TRUE; ++ volatile int use_usermap = TRUE; + Connection_T c = db_con_get(); + TRY + if (! db_query(c, db_get_sql(SQL_TABLE_EXISTS), DBPFX, "usermap")) +@@ -1214,7 +1214,7 @@ static int dm_quota_user_validate(uint64_t user_idnr, uint64_t msg_size) + int dm_quota_rebuild_user(uint64_t user_idnr) + { + Connection_T c; ResultSet_T r; volatile int t = DM_SUCCESS; +- uint64_t quotum = 0; ++ volatile uint64_t quotum = 0; + + c = db_con_get(); + TRY +@@ -1257,7 +1257,7 @@ int dm_quota_rebuild() + + GList *quota = NULL; + struct used_quota *q; +- int i = 0; ++ volatile int i = 0; + int result; + + c = db_con_get(); +@@ -1375,7 +1375,7 @@ int db_get_reply_body(uint64_t user_idnr, char **reply_body) + uint64_t db_get_useridnr(uint64_t message_idnr) + { + Connection_T c; ResultSet_T r; +- uint64_t user_idnr = 0; ++ volatile uint64_t user_idnr = 0; + c = db_con_get(); + TRY + r = db_query(c, "SELECT %smailboxes.owner_idnr FROM %smailboxes, %smessages " +@@ -1397,7 +1397,7 @@ uint64_t db_get_useridnr(uint64_t message_idnr) + int db_log_ip(const char *ip) + { + Connection_T c; ResultSet_T r; PreparedStatement_T s; volatile int t = DM_SUCCESS; +- uint64_t id = 0; ++ volatile uint64_t id = 0; + + c = db_con_get(); + TRY +@@ -1444,8 +1444,8 @@ int db_empty_mailbox(uint64_t user_idnr, int only_empty) + Connection_T c; ResultSet_T r; volatile int t = DM_SUCCESS; + GList *mboxids = NULL; + uint64_t *id; +- unsigned i = 0; +- int result = 0; ++ volatile unsigned i = 0; ++ volatile int result = 0; + + c = db_con_get(); + +@@ -2165,7 +2165,7 @@ static int mailboxes_by_regex(uint64_t user_idnr, int only_subscribed, const cha + char *spattern; + char *namespace, *username; + GString *qs = NULL; +- int n_rows = 0; ++ volatile int n_rows = 0; + PreparedStatement_T stmt; + int prml; + +@@ -2862,7 +2862,7 @@ int db_movemsg(uint64_t mailbox_to, uint64_t mailbox_from) + #define EXPIRE_DAYS 3 + int db_mailbox_has_message_id(uint64_t mailbox_idnr, const char *messageid) + { +- int rows = 0; ++ volatile int rows = 0; + Connection_T c; ResultSet_T r; PreparedStatement_T s; + char expire[DEF_FRAGSIZE], partial[DEF_FRAGSIZE]; + INIT_QUERY; +@@ -2909,7 +2909,7 @@ int db_mailbox_has_message_id(uint64_t mailbox_idnr, const char *messageid) + static uint64_t message_get_size(uint64_t message_idnr) + { + Connection_T c; ResultSet_T r; +- uint64_t size = 0; ++ volatile uint64_t size = 0; + + c = db_con_get(); + TRY +@@ -3443,8 +3443,8 @@ int db_usermap_resolve(ClientBase_T *ci, const char *username, char *real_userna + { + char clientsock[DM_SOCKADDR_LEN]; + const char *userid = NULL, *sockok = NULL, *sockno = NULL, *login = NULL; +- unsigned row = 0; +- int result = TRUE; ++ volatile unsigned row = 0; ++ volatile int result = TRUE; + int score, bestscore = -1; + char *bestlogin = NULL, *bestuserid = NULL; + Connection_T c; ResultSet_T r; PreparedStatement_T s; +@@ -3653,7 +3653,7 @@ int db_user_validate(ClientBase_T *ci, const char *pwfield, uint64_t *user_idnr, + { + int is_validated = 0; + char salt[13], cryptres[35]; +- int t = FALSE; ++ volatile int t = FALSE; + char dbpass[COLUMN_WIDTH+1]; + char encode[COLUMN_WIDTH+1]; + char hashstr[FIELDSIZE]; +diff --git src/dm_mailbox.c src/dm_mailbox.c +index 3558b8e..6f4b99c 100644 +--- src/dm_mailbox.c ++++ src/dm_mailbox.c +@@ -181,7 +181,7 @@ static int _mimeparts_dump(DbmailMailbox *self, GMimeStream *ostream) + uint64_t msgid, physid, *id; + DbmailMessage *m; + GTree *uids; +- int count = 0; ++ volatile int count = 0; + PreparedStatement_T stmt; + Connection_T c; + ResultSet_T r; +@@ -475,7 +475,7 @@ char * dbmail_mailbox_sorted_as_string(DbmailMailbox *self) + uint64_t *msn; + + l = g_list_first(self->sorted); +- if (! g_list_length(l)>0) ++ if (! (g_list_length(l) > 0)) + return s; + + t = g_string_new(""); +@@ -1224,7 +1224,7 @@ static GTree * mailbox_search(DbmailMailbox *self, search_key *s) + char partial[DEF_FRAGSIZE]; + Connection_T c; ResultSet_T r; PreparedStatement_T st; + GTree *ids; +- char *inset = NULL; ++ volatile char *inset = NULL; + + GString *t; + String_T q; +diff --git src/dm_mailboxstate.c src/dm_mailboxstate.c +index 2ef3fd3..723689a 100644 +--- src/dm_mailboxstate.c ++++ src/dm_mailboxstate.c +@@ -977,7 +977,7 @@ int MailboxState_hasPermission(T M, uint64_t userid, const char *right_flag) + if (! owner_id) { + result = db_get_mailbox_owner(mboxid, &owner_id); + MailboxState_setOwner(M, owner_id); +- if (! result > 0) ++ if (! (result > 0)) + return result; + } + +@@ -1116,7 +1116,7 @@ int MailboxState_build_recent(T M) + return 0; + } + +-static long long int _update_recent(GList *slices, uint64_t seq) ++static long long int _update_recent(volatile GList *slices, uint64_t seq) + { + INIT_QUERY; + Connection_T c; +diff --git src/dm_message.c src/dm_message.c +index 066634a..9d30d52 100644 +--- src/dm_message.c ++++ src/dm_message.c +@@ -361,11 +361,11 @@ static DbmailMessage * _mime_retrieve(DbmailMessage *self) + ResultSet_T r; + char internal_date[SQL_INTERNALDATE_LEN]; + GMimeContentType *mimetype = NULL; +- int prevdepth, depth = 0, row = 0; ++ volatile int prevdepth, depth = 0, row = 0; + volatile int t = FALSE; +- gboolean got_boundary = FALSE, prev_boundary = FALSE, is_header = TRUE, prev_header, finalized=FALSE; +- gboolean prev_is_message = FALSE, is_message = FALSE; +- String_T m = NULL, n = NULL; ++ volatile gboolean got_boundary = FALSE, prev_boundary = FALSE, is_header = TRUE, prev_header, finalized=FALSE; ++ volatile gboolean prev_is_message = FALSE, is_message = FALSE; ++ volatile String_T m = NULL, n = NULL; + const void *blob; + Field_T frag; + +@@ -530,7 +530,7 @@ static gboolean store_mime_multipart(GMimeObject *object, DbmailMessage *m, cons + { + const char *boundary; + const char *preface = NULL, *postface = NULL; +- int n, i, c; ++ int n = 0, i, c; + + g_return_val_if_fail(GMIME_IS_OBJECT(object), TRUE); + +@@ -1977,7 +1977,7 @@ DbmailMessage * dbmail_message_construct(DbmailMessage *self, + + static int get_mailbox_from_filters(DbmailMessage *message, uint64_t useridnr, const char *mailbox, char *into, size_t into_n) + { +- int t = FALSE; ++ volatile int t = FALSE; + uint64_t anyone = 0; + PreparedStatement_T stmt; + Connection_T c; +diff --git src/dm_misc.c src/dm_misc.c +index e6ca9a0..1294930 100644 +--- src/dm_misc.c ++++ src/dm_misc.c +@@ -1029,7 +1029,7 @@ int g_tree_merge(GTree *a, GTree *b, int condition) + + type=g_strdup("AND"); + +- if (! g_tree_nnodes(a) > 0) ++ if (! (g_tree_nnodes(a) > 0)) + break; + + /* delete from A all keys not in B */ +@@ -1053,7 +1053,7 @@ int g_tree_merge(GTree *a, GTree *b, int condition) + case IST_SUBSEARCH_OR: + type=g_strdup("OR"); + +- if (! g_tree_nnodes(b) > 0) ++ if (! (g_tree_nnodes(b) > 0)) + break; + + merger->tree = a; +@@ -1081,7 +1081,7 @@ int g_tree_merge(GTree *a, GTree *b, int condition) + case IST_SUBSEARCH_NOT: + type=g_strdup("NOT"); + +- if (! g_tree_nnodes(b) > 0) ++ if (! (g_tree_nnodes(b) > 0)) + break; + + keys = g_tree_keys(b); +diff --git src/dm_sievescript.c src/dm_sievescript.c +index e163413..80f333d 100644 +--- src/dm_sievescript.c ++++ src/dm_sievescript.c +@@ -132,7 +132,7 @@ int dm_sievescript_list(uint64_t user_idnr, GList **scriptlist) + + int dm_sievescript_rename(uint64_t user_idnr, char *scriptname, char *newname) + { +- int active = 0; ++ volatile int active = 0; + Connection_T c; ResultSet_T r; PreparedStatement_T s; volatile int t = FALSE; + assert(scriptname); + +diff --git src/maintenance.c src/maintenance.c +index 2e46453..b4a020b 100644 +--- src/maintenance.c ++++ src/maintenance.c +@@ -1012,7 +1012,7 @@ int do_migrate(int migrate_limit) + { + Connection_T c; ResultSet_T r; + int id = 0; +- int count = 0; ++ volatile int count = 0; + DbmailMessage *m; + + qprintf ("Migrate legacy 2.2.x messageblks to mimeparts...\n"); +diff --git src/modules/authsql.c src/modules/authsql.c +index 49e1dc9..8e4829e 100644 +--- src/modules/authsql.c ++++ src/modules/authsql.c +@@ -97,7 +97,7 @@ int auth_getclientid(uint64_t user_idnr, uint64_t * client_idnr) + { + assert(client_idnr != NULL); + *client_idnr = 0; +- C c; R r; int t = TRUE; ++ C c; R r; volatile int t = TRUE; + + c = db_con_get(); + TRY +@@ -118,7 +118,7 @@ int auth_getmaxmailsize(uint64_t user_idnr, uint64_t * maxmail_size) + { + assert(maxmail_size != NULL); + *maxmail_size = 0; +- C c; R r; int t = TRUE; ++ C c; R r; volatile int t = TRUE; + + c = db_con_get(); + TRY +@@ -361,7 +361,7 @@ uint64_t auth_md5_validate(ClientBase_T *ci UNUSED, char *username, + uint64_t user_idnr = 0; + const char *dbpass; + C c; R r; +- int t = FALSE; ++ volatile int t = FALSE; + + /* lookup the user_idnr */ + if (! auth_user_exists(username, &user_idnr)) +@@ -430,7 +430,7 @@ char *auth_get_userid(uint64_t user_idnr) + + int auth_check_userid(uint64_t user_idnr) + { +- C c; R r; gboolean t = TRUE; ++ C c; R r; volatile gboolean t = TRUE; + + c = db_con_get(); + TRY +@@ -570,7 +570,7 @@ int auth_addalias_ext(const char *alias, + + int auth_removealias(uint64_t user_idnr, const char *alias) + { +- C c; S s; gboolean t = FALSE; ++ C c; S s; volatile gboolean t = FALSE; + + c = db_con_get(); + TRY +@@ -590,7 +590,7 @@ int auth_removealias(uint64_t user_idnr, const char *alias) + + int auth_removealias_ext(const char *alias, const char *deliver_to) + { +- C c; S s; gboolean t = FALSE; ++ C c; S s; volatile gboolean t = FALSE; + + c = db_con_get(); + TRY +diff --git src/pop3.c src/pop3.c +index d03f71a..b7106d3 100644 +--- src/pop3.c ++++ src/pop3.c +@@ -99,7 +99,7 @@ static int db_createsession(uint64_t user_idnr, ClientSession_T * session) + { + Connection_T c; ResultSet_T r; volatile int t = DM_SUCCESS; + struct message *tmpmessage; +- int message_counter = 0; ++ volatile int message_counter = 0; + const char *query_result; + uint64_t mailbox_idnr; + INIT_QUERY; +-- +2.10.1 (Apple Git-78) + Added: head/mail/dbmail/files/patch-0003-Fix-IMAP-mailbox-maintanence ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/mail/dbmail/files/patch-0003-Fix-IMAP-mailbox-maintanence Fri Feb 24 21:15:52 2017 (r434749) @@ -0,0 +1,81 @@ +From f2d27f8727ae4159d356d63c1af6ac1a60b1261a Mon Sep 17 00:00:00 2001 +From: Pavlo Lavrenenko +Date: Thu, 21 May 2015 11:42:56 +0300 +Subject: [PATCH 03/33] Fix IMAP mailbox maintanence + +Update message's mailbox id in a separate explicit transaction to prevent +SQLException: ORA-01453 error on the next db_begin_transaction() call +(see db_mailbox_seq_update() next to 'UPDATE dbmail_messages SET mailbox_idnr' +query). +--- + src/dm_db.c | 6 ++++++ + src/dm_db.h | 1 + + src/maintenance.c | 8 ++------ + 3 files changed, 9 insertions(+), 6 deletions(-) + +diff --git src/dm_db.c src/dm_db.c +index feb0b34..313b33f 100644 +--- src/dm_db.c ++++ src/dm_db.c +@@ -4290,6 +4290,12 @@ void db_message_set_seq(uint64_t message_id, uint64_t seq) + END_TRY; + } + ++int db_move_message(uint64_t message_id, uint64_t mailbox_id) ++{ ++ return db_update("UPDATE %smessages SET mailbox_idnr = %" PRIu64 " WHERE message_idnr = %" PRIu64 "", ++ DBPFX, mailbox_id, message_id); ++} ++ + int db_rehash_store(void) + { + GList *ids = NULL; +diff --git src/dm_db.h src/dm_db.h +index 57aa256..84595d2 100644 +--- src/dm_db.h ++++ src/dm_db.h +@@ -777,6 +777,7 @@ char * db_returning(const char *s); + + uint64_t db_mailbox_seq_update(uint64_t mailbox_id, uint64_t message_id); + void db_message_set_seq(uint64_t message_id, uint64_t seq); ++int db_move_message(uint64_t message_id, uint64_t mailbox_id); + + int db_rehash_store(void); + +diff --git src/maintenance.c src/maintenance.c +index b4a020b..24b818c 100644 +--- src/maintenance.c ++++ src/maintenance.c +@@ -1184,7 +1184,7 @@ int do_erase_old(int days, char * mbtrash_name) + /* Move message to Trash if the message is in INBOX mailbox and date less then passed date. */ + int do_move_old (int days, char * mbinbox_name, char * mbtrash_name) + { +- Connection_T c; ResultSet_T r; ResultSet_T r1; PreparedStatement_T s; PreparedStatement_T s1; PreparedStatement_T s2; ++ Connection_T c; ResultSet_T r; ResultSet_T r1; PreparedStatement_T s; PreparedStatement_T s1; + int skip = 1; + char expire [DEF_FRAGSIZE]; + uint64_t mailbox_to; +@@ -1200,9 +1200,7 @@ int do_move_old (int days, char * mbinbox_name, char * mbtrash_name) + "WHERE mb.name = ? AND msg.status < %d " + "AND phys.internal_date < %s", + DBPFX, DBPFX, DBPFX, MESSAGE_STATUS_DELETE, expire); +- + s1 = db_stmt_prepare(c, "SELECT mailbox_idnr FROM %smailboxes WHERE owner_idnr = ? AND name = ?", DBPFX); +- s2 = db_stmt_prepare(c, "UPDATE %smessages SET mailbox_idnr = ? WHERE message_idnr = ?", DBPFX); + + db_stmt_set_str(s, 1, mbinbox_name); + +@@ -1225,9 +1223,7 @@ int do_move_old (int days, char * mbinbox_name, char * mbtrash_name) + } + + if (!skip) { +- db_stmt_set_u64(s2,1,mailbox_to); +- db_stmt_set_u64(s2,2,id); +- db_stmt_exec(s2); ++ db_move_message(id, mailbox_to); + db_mailbox_seq_update(mailbox_to, 0); + db_mailbox_seq_update(mailbox_from, 0); + } +-- +2.10.1 (Apple Git-78) + Added: head/mail/dbmail/files/patch-0004-prevent-assertion-in-p_string_erase ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/mail/dbmail/files/patch-0004-prevent-assertion-in-p_string_erase Fri Feb 24 21:15:52 2017 (r434749) @@ -0,0 +1,41 @@ +From b6f28ffa54e6533fd0d8676248d0c9bc6d4bd859 Mon Sep 17 00:00:00 2001 +From: Paul J Stevens +Date: Sun, 10 Jan 2016 20:01:31 +0100 +Subject: [PATCH 04/33] prevent assertion in p_string_erase + +--- + src/dm_imapsession.c | 9 +++++---- + 1 file changed, 5 insertions(+), 4 deletions(-) + +diff --git src/dm_imapsession.c src/dm_imapsession.c +index dbf1834..67eae60 100644 +--- src/dm_imapsession.c ++++ src/dm_imapsession.c +@@ -870,19 +870,20 @@ static void _imap_send_part(ImapSession *self, GMimeObject *part, body_fetch *bo + } else { + char *tmp = imap_get_logical_part(part,type); + String_T str = p_string_new(self->pool, tmp); ++ size_t len = p_string_len(str); + g_free(tmp); + +- if (p_string_len(str) < 1) { ++ if (len < 1) { + dbmail_imap_session_buff_printf(self, "] NIL"); + } else { + uint64_t cnt = 0; + if (bodyfetch->octetcnt > 0) { +- cnt = get_dumpsize(bodyfetch, p_string_len(str)); ++ cnt = get_dumpsize(bodyfetch, len); + dbmail_imap_session_buff_printf(self, "]<%" PRIu64 "> {%" PRIu64 "}\r\n", bodyfetch->octetstart, cnt); +- p_string_erase(str,0,bodyfetch->octetstart); ++ p_string_erase(str,0,min(bodyfetch->octetstart,len)); + p_string_truncate(str,cnt); + } else { +- dbmail_imap_session_buff_printf(self, "] {%" PRIu64 "}\r\n", p_string_len(str)); ++ dbmail_imap_session_buff_printf(self, "] {%" PRIu64 "}\r\n", len); + } + dbmail_imap_session_buff_printf(self,"%s", p_string_str(str)); + } +-- +2.10.1 (Apple Git-78) + Added: head/mail/dbmail/files/patch-0005-improve-crypt-authentication ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/mail/dbmail/files/patch-0005-improve-crypt-authentication Fri Feb 24 21:15:52 2017 (r434749) @@ -0,0 +1,38 @@ +From b4b82aca1dd1c8aece722b8370da02b715e4bb53 Mon Sep 17 00:00:00 2001 +From: Paul J Stevens +Date: Wed, 10 Feb 2016 09:14:41 +0100 +Subject: [PATCH 05/33] improve crypt authentication + +also don't segfault when spasswd is empty +--- + src/dm_db.c | 7 ++++++- + 1 file changed, 6 insertions(+), 1 deletion(-) + +diff --git src/dm_db.c src/dm_db.c +index 313b33f..bfe9601 100644 +--- src/dm_db.c ++++ src/dm_db.c +@@ -3687,6 +3687,10 @@ int db_user_validate(ClientBase_T *ci, const char *pwfield, uint64_t *user_idnr, + return t; + + if (! t) return FALSE; ++ if (! strlen(dbpass)) { ++ TRACE(TRACE_INFO, "Empty password for [%" PRIu64 "] in [%s]", *user_idnr, pwfield); ++ return FALSE; ++ } + + if (SMATCH(encode, "")) { + TRACE(TRACE_DEBUG, "validating using plaintext passwords"); +@@ -3699,7 +3703,8 @@ int db_user_validate(ClientBase_T *ci, const char *pwfield, uint64_t *user_idnr, + + if (SMATCH(encode, "crypt")) { + TRACE(TRACE_DEBUG, "validating using crypt() encryption"); +- is_validated = (strcmp((const char *) crypt(password, dbpass), dbpass) == 0) ? 1 : 0; ++ strncpy(salt, dbpass, 2); ++ is_validated = (strcmp((const char *) crypt(password, salt), dbpass) == 0) ? 1 : 0; + } else if (SMATCH(encode, "md5")) { + /* get password */ + if (strncmp(dbpass, "$1$", 3)) { // no match +-- +2.10.1 (Apple Git-78) + Added: head/mail/dbmail/files/patch-0006-simplify-log_query_time-duration-logic ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/mail/dbmail/files/patch-0006-simplify-log_query_time-duration-logic Fri Feb 24 21:15:52 2017 (r434749) @@ -0,0 +1,119 @@ +From f5b73b342b4a6cae1bb15340e5da7330e29319c1 Mon Sep 17 00:00:00 2001 +From: Paul J Stevens +Date: Sat, 5 Mar 2016 18:43:12 +0100 +Subject: [PATCH 06/33] simplify log_query_time duration logic + +--- + src/dm_db.c | 16 ++++++++-------- + src/dm_misc.c | 12 ++++++++++++ + src/dm_misc.h | 2 ++ + test/check_dbmail_db.c | 27 +++++++++++++++++++++++++++ + 4 files changed, 49 insertions(+), 8 deletions(-) + +diff --git src/dm_db.c src/dm_db.c +index bfe9601..429d0ae 100644 +--- src/dm_db.c ++++ src/dm_db.c +@@ -354,14 +354,14 @@ void db_con_clear(Connection_T c) + + void log_query_time(char *query, struct timeval before, struct timeval after) + { +- double elapsed = ((double)after.tv_sec + ((double)after.tv_usec / 1000000)) - ((double)before.tv_sec + ((double)before.tv_usec / 1000000)); +- TRACE(TRACE_DATABASE, "last query took [%.3f] seconds", elapsed); +- if (elapsed > (double)db_params.query_time_warning) +- TRACE(TRACE_WARNING, "slow query [%s] took [%.3f] seconds", query, elapsed); +- else if (elapsed > (double)db_params.query_time_notice) +- TRACE(TRACE_NOTICE, "slow query [%s] took [%.3f] seconds", query, elapsed); +- else if (elapsed > (double)db_params.query_time_info) +- TRACE(TRACE_INFO, "slow query [%s] took [%.3f] seconds", query, elapsed); ++ unsigned int elapsed = (unsigned int)diff_time(before, after); ++ TRACE(TRACE_DATABASE, "last query took [%d] seconds", elapsed); ++ if (elapsed > db_params.query_time_warning) ++ TRACE(TRACE_WARNING, "slow query [%s] took [%d] seconds", query, elapsed); ++ else if (elapsed > db_params.query_time_notice) ++ TRACE(TRACE_NOTICE, "slow query [%s] took [%d] seconds", query, elapsed); ++ else if (elapsed > db_params.query_time_info) ++ TRACE(TRACE_INFO, "slow query [%s] took [%d] seconds", query, elapsed); + return; + } + +diff --git src/dm_misc.c src/dm_misc.c +index 1294930..e27ef34 100644 +--- src/dm_misc.c ++++ src/dm_misc.c +@@ -2422,4 +2422,16 @@ void uint64_free(void *data) + mempool_push(small_pool, data, sizeof(uint64_t)); + } + ++/* ++ * calculate the difference between two timeval values ++ * as number of seconds, using default rounding ++ */ ++int diff_time(struct timeval before, struct timeval after) ++{ ++ int tbefore = before.tv_sec * 1000000 + before.tv_usec; ++ int tafter = after.tv_sec * 1000000 + after.tv_usec; ++ int tdiff = tafter - tbefore; ++ return (int)rint((double)tdiff/1000000); ++} ++ + +diff --git src/dm_misc.h src/dm_misc.h +index a5dd04f..9660dfa 100644 +--- src/dm_misc.h ++++ src/dm_misc.h +@@ -178,4 +178,6 @@ gchar * get_crlf_encoded_opt(const gchar *string, int dots); + void strip_crlf(char *buffer); + void uint64_free(void *); + ++int diff_time(struct timeval before, struct timeval after); ++ + #endif +diff --git test/check_dbmail_db.c test/check_dbmail_db.c +index 47c34be..e6cb170 100644 +--- test/check_dbmail_db.c ++++ test/check_dbmail_db.c +@@ -414,6 +414,32 @@ START_TEST(test_db_get_sql) + } + END_TEST + ++START_TEST(test_diff_time) ++{ ++ struct timeval before, after; ++ int diff; ++ ++ before.tv_sec = 1; before.tv_usec = 0; ++ after.tv_sec = 2; after.tv_usec = 0; ++ diff = diff_time(before, after); ++ fail_unless(diff == 1); ++ ++ before.tv_sec = 1; before.tv_usec = 1000000 - 1; ++ after.tv_sec = 2; after.tv_usec = 0; ++ diff = diff_time(before, after); ++ fail_unless(diff == 0); ++ ++ before.tv_sec = 1; before.tv_usec = 500001; ++ after.tv_sec = 2; after.tv_usec = 0; ++ diff = diff_time(before, after); ++ fail_unless(diff == 0); ++ ++ before.tv_sec = 1; before.tv_usec = 499999; ++ after.tv_sec = 2; after.tv_usec = 0; ++ diff = diff_time(before, after); ++ fail_unless(diff == 1); ++} ++END_TEST + + Suite *dbmail_db_suite(void) + { +@@ -435,6 +461,7 @@ Suite *dbmail_db_suite(void) + tcase_add_test(tc_db, test_mailbox_match_new); + tcase_add_test(tc_db, test_db_findmailbox_by_regex); + tcase_add_test(tc_db, test_db_get_sql); ++ tcase_add_test(tc_db, test_diff_time); + + return s; + } +-- +2.10.1 (Apple Git-78) + Added: head/mail/dbmail/files/patch-0007-Disconnect-IMAP-clients-if-only-few-free-FDs-left ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/mail/dbmail/files/patch-0007-Disconnect-IMAP-clients-if-only-few-free-FDs-left Fri Feb 24 21:15:52 2017 (r434749) @@ -0,0 +1,148 @@ +From 6b7eccfec4f76b7d9d1f865caf741ff3214b5964 Mon Sep 17 00:00:00 2001 +From: Pavlo Lavrenenko +Date: Thu, 2 Jun 2016 08:18:51 +0300 +Subject: [PATCH 07/33] Disconnect IMAP clients if only few free FDs left (#37) + +After network connection to DB server goes down the processing of IMAP session +stalls: DB connection pool becomes exhausted as active connections do not +close till TCP timeout kicks in (true at least for Oracle). While DBMail still +accepts incoming connections it quickly reaches the RLIMIT_NOFILE and becomes +unresponsive. Send BYE response if the number of opened FDs reaches the +RLIMIT_NOFILE value. +--- + src/dbmail.h.in | 5 +++++ + src/dm_misc.c | 20 ++++++++++++++++++++ + src/dm_misc.h | 8 ++++++++ + src/imap4.c | 23 ++++++++++++++++++++++- + 4 files changed, 55 insertions(+), 1 deletion(-) + +diff --git src/dbmail.h.in src/dbmail.h.in +index d826dc3..17215ef 100644 +--- src/dbmail.h.in ++++ src/dbmail.h.in +@@ -69,12 +69,14 @@ + #include + #include + #include ++#include + #include + #include + #include + #include + #include + #include ++#include + #include + #include + #include +@@ -252,6 +254,9 @@ + /* input reading linelimit */ + #define MAX_LINESIZE (64*1024) + ++/* minumun number of free file descriptors required to run the daemon */ ++#define FREE_DF_THRESHOLD 16 ++ + /* string length for query */ + #define DEF_QUERYSIZE (32*1024) + #define DEF_FRAGSIZE 256 +diff --git src/dm_misc.c src/dm_misc.c +index e27ef34..e795de1 100644 +--- src/dm_misc.c ++++ src/dm_misc.c +@@ -104,6 +104,26 @@ int drop_privileges(char *newuser, char *newgroup) + return 0; + } + ++int get_opened_fd_count(void) ++{ ++ DIR* dir = NULL; ++ struct dirent* entry = NULL; ++ char buf[32]; ++ int fd_count = 0; ++ ++ snprintf(buf, 32, "/proc/%i/fd/", getpid()); ++ ++ dir = opendir(buf); ++ if (dir == NULL) ++ return -1; ++ ++ while ((entry = readdir(dir)) != NULL) ++ fd_count++; ++ closedir(dir); ++ ++ return fd_count - 2; /* exclude '.' and '..' entries */ ++} ++ + void create_unique_id(char *target, uint64_t message_idnr) + { + char md5_str[FIELDSIZE]; +diff --git src/dm_misc.h src/dm_misc.h +index 9660dfa..b6cf24f 100644 +--- src/dm_misc.h ++++ src/dm_misc.h +@@ -45,6 +45,14 @@ void g_string_maybe_shrink(GString *s); + int drop_privileges(char *newuser, char *newgroup); + + /** ++ \brief get the number of opened files (requires /proc mounted) ++ \return ++ - -1 on error ++ - number of opened files ++*/ ++int get_opened_fd_count(void); ++ ++/** + * \brief create a unique id for a message (used for pop, stored per message) + * \param target target string. Length should be UID_SIZE + * \param message_idnr message_idnr of message +diff --git src/imap4.c src/imap4.c +index 0532f2e..e523edc 100644 +--- src/imap4.c ++++ src/imap4.c +@@ -351,6 +351,12 @@ static void send_greeting(ImapSession *session) + dbmail_imap_session_set_state(session, CLIENTSTATE_NON_AUTHENTICATED); + } + ++static void disconnect_user(ImapSession *session) ++{ ++ imap_session_printf(session, "* BYE [Service unavailable.]\r\n"); ++ imap_handle_abort(session); ++} ++ + /* + * the default timeout callback */ + +@@ -601,6 +607,8 @@ int imap_handle_connection(client_sock *c) + { + ImapSession *session; + ClientBase_T *ci; ++ struct rlimit fd_limit; ++ int fd_count; + + ci = client_init(c); + +@@ -617,7 +625,20 @@ int imap_handle_connection(client_sock *c) + Capa_remove(session->capa, "LOGINDISABLED"); + } + +- send_greeting(session); ++ fd_count = get_opened_fd_count(); ++ if (fd_count < 0 || getrlimit(RLIMIT_NPROC, &fd_limit) < 0) { ++ TRACE(TRACE_ERR, ++ "[%p] failed to retrieve fd limits, dropping client connection", ++ session); ++ disconnect_user(session); ++ } else if (fd_limit.rlim_cur - fd_count < FREE_DF_THRESHOLD) { ++ TRACE(TRACE_WARNING, ++ "[%p] fd count [%d], fd limit [%d], fd threshold [%d]: dropping client connection", ++ session, fd_count, fd_limit.rlim_cur, FREE_DF_THRESHOLD); ++ disconnect_user(session); ++ } else { ++ send_greeting(session); ++ } + + reset_callbacks(session); + +-- +2.10.1 (Apple Git-78) + Added: head/mail/dbmail/files/patch-0008-Add-primary-key-constraint-to-dbmail_authlog ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/mail/dbmail/files/patch-0008-Add-primary-key-constraint-to-dbmail_authlog Fri Feb 24 21:15:52 2017 (r434749) @@ -0,0 +1,26 @@ +From 024ba19d1c83bee22b6b5901d2fff586d9391cf3 Mon Sep 17 00:00:00 2001 +From: Alan Hicks +Date: Tue, 24 Feb 2015 11:27:45 +0000 +Subject: [PATCH 08/33] Add primary key constraint to dbmail_authlog + +--- + sql/postgresql/create_tables.pgsql | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +diff --git sql/postgresql/create_tables.pgsql sql/postgresql/create_tables.pgsql +index da3b694..45740fd 100644 +--- sql/postgresql/create_tables.pgsql ++++ sql/postgresql/create_tables.pgsql +@@ -46,7 +46,8 @@ CREATE TABLE dbmail_authlog ( + dst_port INT8, + status VARCHAR(32) DEFAULT 'active', + bytes_rx INT8 DEFAULT '0' NOT NULL, +- bytes_tx INT8 DEFAULT '0' NOT NULL ++ bytes_tx INT8 DEFAULT '0' NOT NULL, ++ PRIMARY KEY (id) + ); + + +-- +2.10.1 (Apple Git-78) + Added: head/mail/dbmail/files/patch-0009-Rework-temporary-connection-failures ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/mail/dbmail/files/patch-0009-Rework-temporary-connection-failures Fri Feb 24 21:15:52 2017 (r434749) @@ -0,0 +1,162 @@ +From 9d093f76a82c88647df1cd0ef08b3e8f82e26abb Mon Sep 17 00:00:00 2001 +From: Alan Hicks +Date: Fri, 30 Sep 2016 17:17:05 +0100 +Subject: [PATCH 09/33] Rework temporary connection failures + +--- + src/modules/authldap.c | 102 ++++++++++++++++++++++++++++++++----------------- + 1 file changed, 67 insertions(+), 35 deletions(-) + +diff --git src/modules/authldap.c src/modules/authldap.c +index 19802c8..2073768 100644 +--- src/modules/authldap.c ++++ src/modules/authldap.c +@@ -110,18 +110,48 @@ static gpointer authldap_once(gpointer UNUSED data) + return (gpointer)NULL; + } + +-/* +- lookup thread-local ldap connection +-*/ ++/* ++ * ldap_con_get() ++ * ++ * Lookup thread-local ldap connection and bind using config credentials ++ * retrying a few times if the server is temporarily unavailable ++ * ++ * returns connection on success, NULL on failure ++ */ + static LDAP * ldap_con_get(void) + { +- LDAP * c = (LDAP *)g_static_private_get(&ldap_conn_key); +- if (! c) { +- authldap_connect(); +- c = (LDAP *)g_static_private_get(&ldap_conn_key); ++ LDAP * ld = (LDAP *)g_static_private_get(&ldap_conn_key); ++ if (ld) { ++ TRACE(TRACE_DEBUG, "connection [%p]", ld); ++ return ld; + } +- TRACE(TRACE_DEBUG, "connection [%p]", c); +- return c; ++ int c = 0; ++ int err = -1; // Start wanting success ++ while (err != 0 && c++ < 5) { ++ // Loop until success or too many retries ++ TRACE(TRACE_DEBUG, "No connection trying [%d]", c); ++ ++ err = authldap_connect(); ++ ++ switch (err) { ++ case LDAP_SUCCESS: ++ ld = (LDAP *)g_static_private_get(&ldap_conn_key); ++ TRACE(TRACE_DEBUG, "connection [%p]", ld); ++ break; ++ case LDAP_SERVER_DOWN: ++ TRACE(TRACE_WARNING, "LDAP gone away: %s. Trying to reconnect(%d/5).", ldap_err2string(err),c); ++ sleep(2); // reconnect failed. wait before trying again ++ break; ++ default: ++ TRACE(TRACE_ERR, "LDAP error(%d): %s", err, ldap_err2string(err)); ++ break; ++ } ++ } ++ if (! ld) { ++ TRACE(TRACE_ERR, "Unable to connect to LDAP giving up"); ++ } ++ TRACE(TRACE_DEBUG, "connection [%p]", ld); ++ return ld; + } + + /* +@@ -140,6 +170,13 @@ static void authldap_free(gpointer data) + sigaction(SIGPIPE, &oldact, 0); + } + ++/* ++ * auth_ldap_bind() ++ * ++ * Bind to server using config credentials ++ * *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***