From owner-svn-src-stable-9@FreeBSD.ORG Thu Oct 10 09:42:42 2013 Return-Path: Delivered-To: svn-src-stable-9@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 99FE7871; Thu, 10 Oct 2013 09:42:42 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 76C4E2D5E; Thu, 10 Oct 2013 09:42:42 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.7/8.14.7) with ESMTP id r9A9gg6v018316; Thu, 10 Oct 2013 09:42:42 GMT (envelope-from glebius@svn.freebsd.org) Received: (from glebius@localhost) by svn.freebsd.org (8.14.7/8.14.5/Submit) id r9A9gfVa018310; Thu, 10 Oct 2013 09:42:41 GMT (envelope-from glebius@svn.freebsd.org) Message-Id: <201310100942.r9A9gfVa018310@svn.freebsd.org> From: Gleb Smirnoff Date: Thu, 10 Oct 2013 09:42:41 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r256257 - stable/9/lib/libfetch X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable-9@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SVN commit messages for only the 9-stable src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Oct 2013 09:42:42 -0000 Author: glebius Date: Thu Oct 10 09:42:41 2013 New Revision: 256257 URL: http://svnweb.freebsd.org/changeset/base/256257 Log: Merge from head r253514,r253680,r253805,r254650: --------------------------------------------------------------------- r253514 | des | 2013-07-21 10:59:56 +0400 (вс, 21 июл 2013) | 5 lines Use the correct request syntax for proxied (tunneled) HTTPS requests. PR: bin/180666 MFC after: 3 days --------------------------------------------------------------------- r253680 | des | 2013-07-26 19:53:43 +0400 (пт, 26 июл 2013) | 7 lines Implement certificate verification, and many other SSL-related imrovements; complete details in the PR. PR: kern/175514 Submitted by: Michael Gmelin MFC after: 1 week --------------------------------------------------------------------- r253805 | des | 2013-07-30 17:07:55 +0400 (вт, 30 июл 2013) | 5 lines Include an Accept header in requests. PR: kern/180917 MFC after: 1 week --------------------------------------------------------------------- r254650 | des | 2013-08-22 11:43:36 +0400 (чт, 22 авг 2013) | 8 lines Even though it doesn't really make sense in the context of a CONNECT request, RFC 2616 14.23 mandates the presence of the Host: header in all HTTP 1.1 requests. PR: kern/181445 Submitted by: Kimo MFC after: 3 days --------------------------------------------------------------------- Reviewed by: des Modified: stable/9/lib/libfetch/common.c stable/9/lib/libfetch/common.h stable/9/lib/libfetch/fetch.3 stable/9/lib/libfetch/http.c Directory Properties: stable/9/lib/libfetch/ (props changed) Modified: stable/9/lib/libfetch/common.c ============================================================================== --- stable/9/lib/libfetch/common.c Thu Oct 10 09:32:27 2013 (r256256) +++ stable/9/lib/libfetch/common.c Thu Oct 10 09:42:41 2013 (r256257) @@ -1,5 +1,6 @@ /*- * Copyright (c) 1998-2011 Dag-Erling Smørgrav + * Copyright (c) 2013 Michael Gmelin * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -47,6 +48,10 @@ __FBSDID("$FreeBSD$"); #include #include +#ifdef WITH_SSL +#include +#endif + #include "fetch.h" #include "common.h" @@ -317,15 +322,488 @@ fetch_connect(const char *host, int port return (conn); } +#ifdef WITH_SSL +/* + * Convert characters A-Z to lowercase (intentionally avoid any locale + * specific conversions). + */ +static char +fetch_ssl_tolower(char in) +{ + if (in >= 'A' && in <= 'Z') + return (in + 32); + else + return (in); +} + +/* + * isalpha implementation that intentionally avoids any locale specific + * conversions. + */ +static int +fetch_ssl_isalpha(char in) +{ + return ((in >= 'A' && in <= 'Z') || (in >= 'a' && in <= 'z')); +} + +/* + * Check if passed hostnames a and b are equal. + */ +static int +fetch_ssl_hname_equal(const char *a, size_t alen, const char *b, + size_t blen) +{ + size_t i; + + if (alen != blen) + return (0); + for (i = 0; i < alen; ++i) { + if (fetch_ssl_tolower(a[i]) != fetch_ssl_tolower(b[i])) + return (0); + } + return (1); +} + +/* + * Check if domain label is traditional, meaning that only A-Z, a-z, 0-9 + * and '-' (hyphen) are allowed. Hyphens have to be surrounded by alpha- + * numeric characters. Double hyphens (like they're found in IDN a-labels + * 'xn--') are not allowed. Empty labels are invalid. + */ +static int +fetch_ssl_is_trad_domain_label(const char *l, size_t len, int wcok) +{ + size_t i; + + if (!len || l[0] == '-' || l[len-1] == '-') + return (0); + for (i = 0; i < len; ++i) { + if (!isdigit(l[i]) && + !fetch_ssl_isalpha(l[i]) && + !(l[i] == '*' && wcok) && + !(l[i] == '-' && l[i - 1] != '-')) + return (0); + } + return (1); +} + +/* + * Check if host name consists only of numbers. This might indicate an IP + * address, which is not a good idea for CN wildcard comparison. + */ +static int +fetch_ssl_hname_is_only_numbers(const char *hostname, size_t len) +{ + size_t i; + + for (i = 0; i < len; ++i) { + if (!((hostname[i] >= '0' && hostname[i] <= '9') || + hostname[i] == '.')) + return (0); + } + return (1); +} + +/* + * Check if the host name h passed matches the pattern passed in m which + * is usually part of subjectAltName or CN of a certificate presented to + * the client. This includes wildcard matching. The algorithm is based on + * RFC6125, sections 6.4.3 and 7.2, which clarifies RFC2818 and RFC3280. + */ +static int +fetch_ssl_hname_match(const char *h, size_t hlen, const char *m, + size_t mlen) +{ + int delta, hdotidx, mdot1idx, wcidx; + const char *hdot, *mdot1, *mdot2; + const char *wc; /* wildcard */ + + if (!(h && *h && m && *m)) + return (0); + if ((wc = strnstr(m, "*", mlen)) == NULL) + return (fetch_ssl_hname_equal(h, hlen, m, mlen)); + wcidx = wc - m; + /* hostname should not be just dots and numbers */ + if (fetch_ssl_hname_is_only_numbers(h, hlen)) + return (0); + /* only one wildcard allowed in pattern */ + if (strnstr(wc + 1, "*", mlen - wcidx - 1) != NULL) + return (0); + /* + * there must be at least two more domain labels and + * wildcard has to be in the leftmost label (RFC6125) + */ + mdot1 = strnstr(m, ".", mlen); + if (mdot1 == NULL || mdot1 < wc || (mlen - (mdot1 - m)) < 4) + return (0); + mdot1idx = mdot1 - m; + mdot2 = strnstr(mdot1 + 1, ".", mlen - mdot1idx - 1); + if (mdot2 == NULL || (mlen - (mdot2 - m)) < 2) + return (0); + /* hostname must contain a dot and not be the 1st char */ + hdot = strnstr(h, ".", hlen); + if (hdot == NULL || hdot == h) + return (0); + hdotidx = hdot - h; + /* + * host part of hostname must be at least as long as + * pattern it's supposed to match + */ + if (hdotidx < mdot1idx) + return (0); + /* + * don't allow wildcards in non-traditional domain names + * (IDN, A-label, U-label...) + */ + if (!fetch_ssl_is_trad_domain_label(h, hdotidx, 0) || + !fetch_ssl_is_trad_domain_label(m, mdot1idx, 1)) + return (0); + /* match domain part (part after first dot) */ + if (!fetch_ssl_hname_equal(hdot, hlen - hdotidx, mdot1, + mlen - mdot1idx)) + return (0); + /* match part left of wildcard */ + if (!fetch_ssl_hname_equal(h, wcidx, m, wcidx)) + return (0); + /* match part right of wildcard */ + delta = mdot1idx - wcidx - 1; + if (!fetch_ssl_hname_equal(hdot - delta, delta, + mdot1 - delta, delta)) + return (0); + /* all tests succeded, it's a match */ + return (1); +} + +/* + * Get numeric host address info - returns NULL if host was not an IP + * address. The caller is responsible for deallocation using + * freeaddrinfo(3). + */ +static struct addrinfo * +fetch_ssl_get_numeric_addrinfo(const char *hostname, size_t len) +{ + struct addrinfo hints, *res; + char *host; + + host = (char *)malloc(len + 1); + memcpy(host, hostname, len); + host[len] = '\0'; + memset(&hints, 0, sizeof(hints)); + hints.ai_family = PF_UNSPEC; + hints.ai_socktype = SOCK_STREAM; + hints.ai_protocol = 0; + hints.ai_flags = AI_NUMERICHOST; + /* port is not relevant for this purpose */ + getaddrinfo(host, "443", &hints, &res); + free(host); + return res; +} + +/* + * Compare ip address in addrinfo with address passes. + */ +static int +fetch_ssl_ipaddr_match_bin(const struct addrinfo *lhost, const char *rhost, + size_t rhostlen) +{ + const void *left; + + if (lhost->ai_family == AF_INET && rhostlen == 4) { + left = (void *)&((struct sockaddr_in*)(void *) + lhost->ai_addr)->sin_addr.s_addr; +#ifdef INET6 + } else if (lhost->ai_family == AF_INET6 && rhostlen == 16) { + left = (void *)&((struct sockaddr_in6 *)(void *) + lhost->ai_addr)->sin6_addr; +#endif + } else + return (0); + return (!memcmp(left, (const void *)rhost, rhostlen) ? 1 : 0); +} + +/* + * Compare ip address in addrinfo with host passed. If host is not an IP + * address, comparison will fail. + */ +static int +fetch_ssl_ipaddr_match(const struct addrinfo *laddr, const char *r, + size_t rlen) +{ + struct addrinfo *raddr; + int ret; + char *rip; + + ret = 0; + if ((raddr = fetch_ssl_get_numeric_addrinfo(r, rlen)) == NULL) + return 0; /* not a numeric host */ + + if (laddr->ai_family == raddr->ai_family) { + if (laddr->ai_family == AF_INET) { + rip = (char *)&((struct sockaddr_in *)(void *) + raddr->ai_addr)->sin_addr.s_addr; + ret = fetch_ssl_ipaddr_match_bin(laddr, rip, 4); +#ifdef INET6 + } else if (laddr->ai_family == AF_INET6) { + rip = (char *)&((struct sockaddr_in6 *)(void *) + raddr->ai_addr)->sin6_addr; + ret = fetch_ssl_ipaddr_match_bin(laddr, rip, 16); +#endif + } + + } + freeaddrinfo(raddr); + return (ret); +} + +/* + * Verify server certificate by subjectAltName. + */ +static int +fetch_ssl_verify_altname(STACK_OF(GENERAL_NAME) *altnames, + const char *host, struct addrinfo *ip) +{ + const GENERAL_NAME *name; + size_t nslen; + int i; + const char *ns; + + for (i = 0; i < sk_GENERAL_NAME_num(altnames); ++i) { +#if OPENSSL_VERSION_NUMBER < 0x10000000L + /* + * This is a workaround, since the following line causes + * alignment issues in clang: + * name = sk_GENERAL_NAME_value(altnames, i); + * OpenSSL explicitly warns not to use those macros + * directly, but there isn't much choice (and there + * shouldn't be any ill side effects) + */ + name = (GENERAL_NAME *)SKM_sk_value(void, altnames, i); +#else + name = sk_GENERAL_NAME_value(altnames, i); +#endif + ns = (const char *)ASN1_STRING_data(name->d.ia5); + nslen = (size_t)ASN1_STRING_length(name->d.ia5); + + if (name->type == GEN_DNS && ip == NULL && + fetch_ssl_hname_match(host, strlen(host), ns, nslen)) + return (1); + else if (name->type == GEN_IPADD && ip != NULL && + fetch_ssl_ipaddr_match_bin(ip, ns, nslen)) + return (1); + } + return (0); +} + +/* + * Verify server certificate by CN. + */ +static int +fetch_ssl_verify_cn(X509_NAME *subject, const char *host, + struct addrinfo *ip) +{ + ASN1_STRING *namedata; + X509_NAME_ENTRY *nameentry; + int cnlen, lastpos, loc, ret; + unsigned char *cn; + + ret = 0; + lastpos = -1; + loc = -1; + cn = NULL; + /* get most specific CN (last entry in list) and compare */ + while ((lastpos = X509_NAME_get_index_by_NID(subject, + NID_commonName, lastpos)) != -1) + loc = lastpos; + + if (loc > -1) { + nameentry = X509_NAME_get_entry(subject, loc); + namedata = X509_NAME_ENTRY_get_data(nameentry); + cnlen = ASN1_STRING_to_UTF8(&cn, namedata); + if (ip == NULL && + fetch_ssl_hname_match(host, strlen(host), cn, cnlen)) + ret = 1; + else if (ip != NULL && fetch_ssl_ipaddr_match(ip, cn, cnlen)) + ret = 1; + OPENSSL_free(cn); + } + return (ret); +} + +/* + * Verify that server certificate subjectAltName/CN matches + * hostname. First check, if there are alternative subject names. If yes, + * those have to match. Only if those don't exist it falls back to + * checking the subject's CN. + */ +static int +fetch_ssl_verify_hname(X509 *cert, const char *host) +{ + struct addrinfo *ip; + STACK_OF(GENERAL_NAME) *altnames; + X509_NAME *subject; + int ret; + + ret = 0; + ip = fetch_ssl_get_numeric_addrinfo(host, strlen(host)); + altnames = X509_get_ext_d2i(cert, NID_subject_alt_name, + NULL, NULL); + + if (altnames != NULL) { + ret = fetch_ssl_verify_altname(altnames, host, ip); + } else { + subject = X509_get_subject_name(cert); + if (subject != NULL) + ret = fetch_ssl_verify_cn(subject, host, ip); + } + + if (ip != NULL) + freeaddrinfo(ip); + if (altnames != NULL) + GENERAL_NAMES_free(altnames); + return (ret); +} + +/* + * Configure transport security layer based on environment. + */ +static void +fetch_ssl_setup_transport_layer(SSL_CTX *ctx, int verbose) +{ + long ssl_ctx_options; + + ssl_ctx_options = SSL_OP_ALL | SSL_OP_NO_TICKET; + if (getenv("SSL_ALLOW_SSL2") == NULL) + ssl_ctx_options |= SSL_OP_NO_SSLv2; + if (getenv("SSL_NO_SSL3") != NULL) + ssl_ctx_options |= SSL_OP_NO_SSLv3; + if (getenv("SSL_NO_TLS1") != NULL) + ssl_ctx_options |= SSL_OP_NO_TLSv1; + if (verbose) + fetch_info("SSL options: %x", ssl_ctx_options); + SSL_CTX_set_options(ctx, ssl_ctx_options); +} + + +/* + * Configure peer verification based on environment. + */ +static int +fetch_ssl_setup_peer_verification(SSL_CTX *ctx, int verbose) +{ + X509_LOOKUP *crl_lookup; + X509_STORE *crl_store; + const char *ca_cert_file, *ca_cert_path, *crl_file; + + if (getenv("SSL_NO_VERIFY_PEER") == NULL) { + ca_cert_file = getenv("SSL_CA_CERT_FILE") != NULL ? + getenv("SSL_CA_CERT_FILE") : "/etc/ssl/cert.pem"; + ca_cert_path = getenv("SSL_CA_CERT_PATH"); + if (verbose) { + fetch_info("Peer verification enabled"); + if (ca_cert_file != NULL) + fetch_info("Using CA cert file: %s", + ca_cert_file); + if (ca_cert_path != NULL) + fetch_info("Using CA cert path: %s", + ca_cert_path); + } + SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, + fetch_ssl_cb_verify_crt); + SSL_CTX_load_verify_locations(ctx, ca_cert_file, + ca_cert_path); + if ((crl_file = getenv("SSL_CRL_FILE")) != NULL) { + if (verbose) + fetch_info("Using CRL file: %s", crl_file); + crl_store = SSL_CTX_get_cert_store(ctx); + crl_lookup = X509_STORE_add_lookup(crl_store, + X509_LOOKUP_file()); + if (crl_lookup == NULL || + !X509_load_crl_file(crl_lookup, crl_file, + X509_FILETYPE_PEM)) { + fprintf(stderr, + "Could not load CRL file %s\n", + crl_file); + return (0); + } + X509_STORE_set_flags(crl_store, + X509_V_FLAG_CRL_CHECK | + X509_V_FLAG_CRL_CHECK_ALL); + } + } + return (1); +} + +/* + * Configure client certificate based on environment. + */ +static int +fetch_ssl_setup_client_certificate(SSL_CTX *ctx, int verbose) +{ + const char *client_cert_file, *client_key_file; + + if ((client_cert_file = getenv("SSL_CLIENT_CERT_FILE")) != NULL) { + client_key_file = getenv("SSL_CLIENT_KEY_FILE") != NULL ? + getenv("SSL_CLIENT_KEY_FILE") : client_cert_file; + if (verbose) { + fetch_info("Using client cert file: %s", + client_cert_file); + fetch_info("Using client key file: %s", + client_key_file); + } + if (SSL_CTX_use_certificate_chain_file(ctx, + client_cert_file) != 1) { + fprintf(stderr, + "Could not load client certificate %s\n", + client_cert_file); + return (0); + } + if (SSL_CTX_use_PrivateKey_file(ctx, client_key_file, + SSL_FILETYPE_PEM) != 1) { + fprintf(stderr, + "Could not load client key %s\n", + client_key_file); + return (0); + } + } + return (1); +} + +/* + * Callback for SSL certificate verification, this is called on server + * cert verification. It takes no decision, but informs the user in case + * verification failed. + */ +int +fetch_ssl_cb_verify_crt(int verified, X509_STORE_CTX *ctx) +{ + X509 *crt; + X509_NAME *name; + char *str; + + str = NULL; + if (!verified) { + if ((crt = X509_STORE_CTX_get_current_cert(ctx)) != NULL && + (name = X509_get_subject_name(crt)) != NULL) + str = X509_NAME_oneline(name, 0, 0); + fprintf(stderr, "Certificate verification failed for %s\n", + str != NULL ? str : "no relevant certificate"); + OPENSSL_free(str); + } + return (verified); +} + +#endif /* * Enable SSL on a connection. */ int -fetch_ssl(conn_t *conn, int verbose) +fetch_ssl(conn_t *conn, const struct url *URL, int verbose) { #ifdef WITH_SSL int ret, ssl_err; + X509_NAME *name; + char *str; /* Init the SSL library and context */ if (!SSL_library_init()){ @@ -339,8 +817,14 @@ fetch_ssl(conn_t *conn, int verbose) conn->ssl_ctx = SSL_CTX_new(conn->ssl_meth); SSL_CTX_set_mode(conn->ssl_ctx, SSL_MODE_AUTO_RETRY); + fetch_ssl_setup_transport_layer(conn->ssl_ctx, verbose); + if (!fetch_ssl_setup_peer_verification(conn->ssl_ctx, verbose)) + return (-1); + if (!fetch_ssl_setup_client_certificate(conn->ssl_ctx, verbose)) + return (-1); + conn->ssl = SSL_new(conn->ssl_ctx); - if (conn->ssl == NULL){ + if (conn->ssl == NULL) { fprintf(stderr, "SSL context creation failed\n"); return (-1); } @@ -353,22 +837,35 @@ fetch_ssl(conn_t *conn, int verbose) return (-1); } } + conn->ssl_cert = SSL_get_peer_certificate(conn->ssl); - if (verbose) { - X509_NAME *name; - char *str; + if (conn->ssl_cert == NULL) { + fprintf(stderr, "No server SSL certificate\n"); + return (-1); + } - fprintf(stderr, "SSL connection established using %s\n", + if (getenv("SSL_NO_VERIFY_HOSTNAME") == NULL) { + if (verbose) + fetch_info("Verify hostname"); + if (!fetch_ssl_verify_hname(conn->ssl_cert, URL->host)) { + fprintf(stderr, + "SSL certificate subject doesn't match host %s\n", + URL->host); + return (-1); + } + } + + if (verbose) { + fetch_info("SSL connection established using %s", SSL_get_cipher(conn->ssl)); - conn->ssl_cert = SSL_get_peer_certificate(conn->ssl); name = X509_get_subject_name(conn->ssl_cert); str = X509_NAME_oneline(name, 0, 0); - printf("Certificate subject: %s\n", str); - free(str); + fetch_info("Certificate subject: %s", str); + OPENSSL_free(str); name = X509_get_issuer_name(conn->ssl_cert); str = X509_NAME_oneline(name, 0, 0); - printf("Certificate issuer: %s\n", str); - free(str); + fetch_info("Certificate issuer: %s", str); + OPENSSL_free(str); } return (0); @@ -726,6 +1223,22 @@ fetch_close(conn_t *conn) if (--conn->ref > 0) return (0); +#ifdef WITH_SSL + if (conn->ssl) { + SSL_shutdown(conn->ssl); + SSL_set_connect_state(conn->ssl); + SSL_free(conn->ssl); + conn->ssl = NULL; + } + if (conn->ssl_ctx) { + SSL_CTX_free(conn->ssl_ctx); + conn->ssl_ctx = NULL; + } + if (conn->ssl_cert) { + X509_free(conn->ssl_cert); + conn->ssl_cert = NULL; + } +#endif ret = close(conn->sd); free(conn->cache.buf); free(conn->buf); Modified: stable/9/lib/libfetch/common.h ============================================================================== --- stable/9/lib/libfetch/common.h Thu Oct 10 09:32:27 2013 (r256256) +++ stable/9/lib/libfetch/common.h Thu Oct 10 09:42:41 2013 (r256257) @@ -87,7 +87,10 @@ int fetch_bind(int, int, const char *) conn_t *fetch_connect(const char *, int, int, int); conn_t *fetch_reopen(int); conn_t *fetch_ref(conn_t *); -int fetch_ssl(conn_t *, int); +#ifdef WITH_SSL +int fetch_ssl_cb_verify_crt(int, X509_STORE_CTX*); +#endif +int fetch_ssl(conn_t *, const struct url *, int); ssize_t fetch_read(conn_t *, char *, size_t); int fetch_getln(conn_t *); ssize_t fetch_write(conn_t *, const char *, size_t); Modified: stable/9/lib/libfetch/fetch.3 ============================================================================== --- stable/9/lib/libfetch/fetch.3 Thu Oct 10 09:32:27 2013 (r256256) +++ stable/9/lib/libfetch/fetch.3 Thu Oct 10 09:42:41 2013 (r256257) @@ -1,5 +1,6 @@ .\"- -.\" Copyright (c) 1998-2011 Dag-Erling Smørgrav +.\" Copyright (c) 1998-2013 Dag-Erling Smørgrav +.\" Copyright (c) 2013 Michael Gmelin .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without @@ -25,7 +26,7 @@ .\" .\" $FreeBSD$ .\" -.Dd September 27, 2011 +.Dd July 30, 2013 .Dt FETCH 3 .Os .Sh NAME @@ -392,6 +393,60 @@ method in a manner consistent with the r library, .Fn fetchPutHTTP is currently unimplemented. +.Sh HTTPS SCHEME +Based on HTTP SCHEME. +By default the peer is verified using the CA bundle located in +.Pa /etc/ssl/cert.pem . +The file may contain multiple CA certificates. +A common source of a current CA bundle is +.Pa \%security/ca_root_nss . +.Pp +The CA bundle used for peer verification can be changed by setting the +environment variables +.Ev SSL_CA_CERT_FILE +to point to a concatenated bundle of trusted certificates and +.Ev SSL_CA_CERT_PATH +to point to a directory containing hashes of trusted CAs (see +.Xr verify 1 ) . +.Pp +A certificate revocation list (CRL) can be used by setting the +environment variable +.Ev SSL_CRL_FILE +(see +.Xr crl 1 ) . +.Pp +Peer verification can be disabled by setting the environment variable +.Ev SSL_NO_VERIFY_PEER . +Note that this also disables CRL checking. +.Pp +By default the service identity is verified according to the rules +detailed in RFC6125 (also known as hostname verification). +This feature can be disabled by setting the environment variable +.Ev SSL_NO_VERIFY_HOSTNAME . +.Pp +Client certificate based authentication is supported. +The environment variable +.Ev SSL_CLIENT_CERT_FILE +should be set to point to a file containing key and client certificate +to be used in PEM format. In case the key is stored in a separate +file, the environment variable +.Ev SSL_CLIENT_KEY_FILE +can be set to point to the key in PEM format. +In case the key uses a password, the user will be prompted on standard +input (see +.Xr PEM 3 ) . +.Pp +By default +.Nm libfetch +allows SSLv3 and TLSv1 when negotiating the connecting with the remote +peer. +You can change this behavior by setting the environment variable +.Ev SSL_ALLOW_SSL2 +to allow SSLv2 (not recommended) and +.Ev SSL_NO_SSL3 +or +.Ev SSL_NO_TLS1 +to disable the respective methods. .Sh AUTHENTICATION Apart from setting the appropriate environment variables and specifying the user name and password in the URL or the @@ -516,6 +571,15 @@ variable is set. Same as .Ev FTP_PROXY , for compatibility. +.It Ev HTTP_ACCEPT +Specifies the value of the +.Va Accept +header for HTTP requests. +If empty, no +.Va Accept +header is sent. +The default is +.Dq */* . .It Ev HTTP_AUTH Specifies HTTP authorization parameters as a colon-separated list of items. @@ -579,6 +643,31 @@ which proxies should not be used. Same as .Ev NO_PROXY , for compatibility. +.It Ev SSL_ALLOW_SSL2 +Allow SSL version 2 when negotiating the connection (not recommended). +.It Ev SSL_CA_CERT_FILE +CA certificate bundle containing trusted CA certificates. +Default value: +.Pa /etc/ssl/cert.pem . +.It Ev SSL_CA_CERT_PATH +Path containing trusted CA hashes. +.It Ev SSL_CLIENT_CERT_FILE +PEM encoded client certificate/key which will be used in +client certificate authentication. +.It Ev SSL_CLIENT_KEY_FILE +PEM encoded client key in case key and client certificate +are stored separately. +.It Ev SSL_CRL_FILE +File containing certificate revocation list. +.It Ev SSL_NO_SSL3 +Don't allow SSL version 3 when negotiating the connection. +.It Ev SSL_NO_TLS1 +Don't allow TLV version 1 when negotiating the connection. +.It Ev SSL_NO_VERIFY_HOSTNAME +If set, do not verify that the hostname matches the subject of the +certificate presented by the server. +.It Ev SSL_NO_VERIFY_PEER +If set, do not verify the peer certificate against trusted CAs. .El .Sh EXAMPLES To access a proxy server on @@ -610,6 +699,19 @@ as follows: .Bd -literal -offset indent NO_PROXY=localhost,127.0.0.1 .Ed +.Pp +Access HTTPS website without any certificate verification whatsoever: +.Bd -literal -offset indent +SSL_NO_VERIFY_PEER=1 +SSL_NO_VERIFY_HOSTNAME=1 +.Ed +.Pp +Access HTTPS website using client certificate based authentication +and a private CA: +.Bd -literal -offset indent +SSL_CLIENT_CERT_FILE=/path/to/client.pem +SSL_CA_CERT_FILE=/path/to/myca.pem +.Ed .Sh SEE ALSO .Xr fetch 1 , .Xr ftpio 3 , @@ -678,7 +780,8 @@ with numerous suggestions and contributi .An Hajimu Umemoto Aq ume@FreeBSD.org , .An Henry Whincup Aq henry@techiebod.com , .An Jukka A. Ukkonen Aq jau@iki.fi , -.An Jean-Fran\(,cois Dockes Aq jf@dockes.org +.An Jean-Fran\(,cois Dockes Aq jf@dockes.org , +.An Michael Gmelin Aq freebsd@grem.de and others. It replaces the older .Nm ftpio @@ -688,7 +791,9 @@ and .An Jordan K. Hubbard Aq jkh@FreeBSD.org . .Pp This manual page was written by -.An Dag-Erling Sm\(/orgrav Aq des@FreeBSD.org . +.An Dag-Erling Sm\(/orgrav Aq des@FreeBSD.org +and +.An Michael Gmelin Aq freebsd@grem.de . .Sh BUGS Some parts of the library are not yet implemented. The most notable @@ -717,6 +822,10 @@ implemented, superfluous at this site" i .Fn fetchStatFTP does not check that the result of an MDTM command is a valid date. .Pp +In case password protected keys are used for client certificate based +authentication the user is prompted for the password on each and every +fetch operation. +.Pp The man page is incomplete, poorly written and produces badly formatted text. .Pp Modified: stable/9/lib/libfetch/http.c ============================================================================== --- stable/9/lib/libfetch/http.c Thu Oct 10 09:32:27 2013 (r256256) +++ stable/9/lib/libfetch/http.c Thu Oct 10 09:42:41 2013 (r256257) @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2000-2011 Dag-Erling Smørgrav + * Copyright (c) 2000-2013 Dag-Erling Smørgrav * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -1400,6 +1400,8 @@ http_connect(struct url *URL, struct url if (strcasecmp(URL->scheme, SCHEME_HTTPS) == 0 && purl) { http_cmd(conn, "CONNECT %s:%d HTTP/1.1", URL->host, URL->port); + http_cmd(conn, "Host: %s:%d", + URL->host, URL->port); http_cmd(conn, ""); if (http_get_reply(conn) != HTTP_OK) { fetch_close(conn); @@ -1408,7 +1410,7 @@ http_connect(struct url *URL, struct url http_get_reply(conn); } if (strcasecmp(URL->scheme, SCHEME_HTTPS) == 0 && - fetch_ssl(conn, verbose) == -1) { + fetch_ssl(conn, URL, verbose) == -1) { fetch_close(conn); /* grrr */ errno = EAUTH; @@ -1581,7 +1583,7 @@ http_request(struct url *URL, const char if (verbose) fetch_info("requesting %s://%s%s", url->scheme, host, url->doc); - if (purl) { + if (purl && strcasecmp(URL->scheme, SCHEME_HTTPS) != 0) { http_cmd(conn, "%s %s://%s%s HTTP/1.1", op, url->scheme, host, url->doc); } else { @@ -1664,6 +1666,12 @@ http_request(struct url *URL, const char } /* other headers */ + if ((p = getenv("HTTP_ACCEPT")) != NULL) { + if (*p != '\0') + http_cmd(conn, "Accept: %s", p); + } else { + http_cmd(conn, "Accept: */*"); + } if ((p = getenv("HTTP_REFERER")) != NULL && *p != '\0') { if (strcasecmp(p, "auto") == 0) http_cmd(conn, "Referer: %s://%s%s",