Date: Mon, 30 Jun 2008 19:54:30 +0200 From: Dominic Fandrey <kamikaze@bsdforen.de> To: freebsd-hackers@freebsd.org Subject: deflate and gzip support for libfetch over http Message-ID: <48691DD6.9000306@bsdforen.de>
next in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format. --------------050207020908070903030703 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit I wanted to benchmark a http-connection using fetch in a shell-script. I came to realize that fetch doesn't support compressed streams when I force-fed it the compressed stream by ignoring the absence of Accept-Encoding. This doesn't influence my benchmarking, but I think fetch should have this feature. I hacked together deflate support for the http part of libfetch. Next on my todo list is proper error handling, gzip support, code clean up and general code clean up in http.c (in order of priority). I'd love to get some feedback, do you consider this useful? Does it work on your system? Would there be a chance of getting the finalized version into SVN? The attached patch applies to RELENG_7. Probably everywhere else, too. Because I think libfetch development has been at a halt for some time. Regards, Dominic --------------050207020908070903030703 Content-Type: text/plain; name="patch-libfetch.diff.txt" Content-Transfer-Encoding: base64 Content-Disposition: inline; filename="patch-libfetch.diff.txt" LS0tIHNyYy9saWIvbGliZmV0Y2gvaHR0cC5jLm9yaWcJMjAwOC0wNi0yOSAxNToyODo1OC4w MDAwMDAwMDAgKzAyMDAKKysrIHNyYy9saWIvbGliZmV0Y2gvaHR0cC5jCTIwMDgtMDYtMzAg MTk6Mzg6NTcuMDAwMDAwMDAwICswMjAwCkBAIC03NSw2ICs3NSw3IEBACiAjaW5jbHVkZSA8 c3RyaW5nLmg+CiAjaW5jbHVkZSA8dGltZS5oPgogI2luY2x1ZGUgPHVuaXN0ZC5oPgorI2lu Y2x1ZGUgPHpsaWIuaD4KIAogI2luY2x1ZGUgPG5ldGluZXQvaW4uaD4KICNpbmNsdWRlIDxu ZXRpbmV0L3RjcC5oPgpAQCAtMTA1LDcgKzEwNiw2IEBACiAKICNkZWZpbmUgSFRUUF9FUlJP Uih4eXopICgoeHl6KSA+IDQwMCAmJiAoeHl6KSA8IDU5OSkKIAotCiAvKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioKICAqIEkvTyBmdW5jdGlvbnMgZm9yIGRlY29kaW5nIGNodW5rZWQgc3Ry ZWFtcwogICovCkBAIC0xMjYsNiArMTI2LDE2IEBACiAjZW5kaWYKIH07CiAKK3N0cnVjdCB6 bGliaW8KK3sKKwlGSUxFCQkqc291cmNlOwkvKiB0aGUgaHR0cCBjb25uZWN0aW9uIHRvIHJl YWQgZnJvbSAqLworCXpfc3RyZWFtCSpzdHJlYW07CS8qIHRoZSB6bGliIHN0cmVhbSB0byBy ZWFkIHRoZSAqLworCQkJCQkvKiB1bmNvbXByZXNzZWQgZGF0YSBmcm9tICovCisJY2hhcgkJ aW5bNjU1MzZdOwkvKiByZWFkIGJ1ZmZlciAqLworfTsKKwordHlwZWRlZiBGSUxFICogKCpm dW5vcGVuX2Z1bmN0aW9uKShjb25uX3QgKiwgaW50KTsKKwogLyoKICAqIEdldCBuZXh0IGNo dW5rIGhlYWRlcgogICovCkBAIC0zMDIsMTAgKzMxMiw1MCBAQAogfQogCiAvKgorICogUmVh ZCBmdW5jdGlvbiBmb3IgZGVmbGF0ZSBjb21wcmVzc2VkIGRhdGEuCisgKi8KK3N0YXRpYyBp bnQKK2h0dHBfaW5mbGF0ZV9yZWFkZm4odm9pZCAqdiwgY2hhciAqYnVmLCBpbnQgbGVuKQor eworCXN0cnVjdCB6bGliaW8gKmlvID0gKHN0cnVjdCB6bGliaW8gKil2OworCWludCBzdGF0 dXM7CisKKwkvKiBPbmx5IHJlYWQgaWYgdGhlIGxhc3QgcmVhZCBjaHVuayBoYXMgY29tcGxl dGVseSBiZWVuIGZsdXNoZWQuICovCisJaWYgKGlvLT5zdHJlYW0tPmF2YWlsX2luID09IDAp IHsKKwkJaW8tPnN0cmVhbS0+YXZhaWxfaW4gPSBmcmVhZChpby0+aW4sIHNpemVvZihjaGFy KSwgc2l6ZW9mKGlvLT5pbiksIGlvLT5zb3VyY2UpOworCisJCS8qIEZvcndhcmQgZXJyb3Jz IGFuZCBlb2YgKi8KKwkJaWYgKGlvLT5zdHJlYW0tPmF2YWlsX2luIDw9IDApCisJCQlyZXR1 cm4gaW8tPnN0cmVhbS0+YXZhaWxfaW47CisKKwkJaW8tPnN0cmVhbS0+bmV4dF9pbiA9IGlv LT5pbjsKKwl9CisJCisJaW8tPnN0cmVhbS0+YXZhaWxfb3V0ID0gbGVuOworCWlvLT5zdHJl YW0tPm5leHRfb3V0ID0gYnVmOworCXN0YXR1cyA9IGluZmxhdGUoaW8tPnN0cmVhbSwgWl9T WU5DX0ZMVVNIKTsKKworCXJldHVybiAobGVuIC0gaW8tPnN0cmVhbS0+YXZhaWxfb3V0KTsK K30KKworLyoKKyAqIENsb3NlIGZ1bmN0aW9uIGZvciBkZWZsYXRlIGNvbXByZXNzZWQgZGF0 YQorICovCitzdGF0aWMgaW50CitodHRwX2luZmxhdGVfY2xvc2Vmbih2b2lkICp2KQorewor CXN0cnVjdCB6bGliaW8gKmlvID0gKHN0cnVjdCB6bGliaW8gKil2OworCisJKHZvaWQpaW5m bGF0ZUVuZChpby0+c3RyZWFtKTsKKwlmcmVlKGlvLT5zdHJlYW0pOworCXJldHVybiAoZmNs b3NlKGlvLT5zb3VyY2UpKTsKK30KKworLyoKICAqIFdyYXAgYSBmaWxlIGRlc2NyaXB0b3Ig dXAKICAqLwogc3RhdGljIEZJTEUgKgotaHR0cF9mdW5vcGVuKGNvbm5fdCAqY29ubiwgaW50 IGNodW5rZWQpCitodHRwX2Z1bm9wZW5fcmF3KGNvbm5fdCAqY29ubiwgaW50IGNodW5rZWQp CiB7CiAJc3RydWN0IGh0dHBpbyAqaW87CiAJRklMRSAqZjsKQEAgLTMxNiw3ICszNjYsNyBA QAogCX0KIAlpby0+Y29ubiA9IGNvbm47CiAJaW8tPmNodW5rZWQgPSBjaHVua2VkOwotCWYg PSBmdW5vcGVuKGlvLCBodHRwX3JlYWRmbiwgaHR0cF93cml0ZWZuLCBOVUxMLCBodHRwX2Ns b3NlZm4pOworCWYgPSBmdW5vcGVuKGlvLCAmaHR0cF9yZWFkZm4sICZodHRwX3dyaXRlZm4s IE5VTEwsICZodHRwX2Nsb3NlZm4pOwogCWlmIChmID09IE5VTEwpIHsKIAkJZmV0Y2hfc3lz ZXJyKCk7CiAJCWZyZWUoaW8pOwpAQCAtMzI1LDYgKzM3NSw1MCBAQAogCXJldHVybiAoZik7 CiB9CiAKKy8qCisgKiBXcmFwIGEgZmlsZSBkZXNjcmlwdG9yIHVwIGFyb3VuZCB0aGUgemxp cCBpbmZsYXRlIGNvbW1hbmQKKyAqLworc3RhdGljIEZJTEUgKgoraHR0cF9mdW5vcGVuX2lu ZmxhdGUoY29ubl90ICpjb25uLCBpbnQgY2h1bmtlZCkKK3sKKwlzdHJ1Y3QgemxpYmlvICpp bzsKKwlGSUxFICpmOworCisJaWYgKChpbyA9IGNhbGxvYygxLCBzaXplb2YoKmlvKSkpID09 IE5VTEwpIHsKKwkJZmV0Y2hfc3lzZXJyKCk7CisJCXJldHVybiAoTlVMTCk7CisJfQorCQor CWlvLT5zb3VyY2UgPSBodHRwX2Z1bm9wZW5fcmF3KGNvbm4sIGNodW5rZWQpOworCisJaWYg KChpby0+c3RyZWFtID0gY2FsbG9jKDEsIHNpemVvZigqKGlvLT5zdHJlYW0pKSkpID09IE5V TEwpIHsKKwkJZmV0Y2hfc3lzZXJyKCk7CisJCWZyZWUoaW8pOworCQlyZXR1cm4gKE5VTEwp OworCX0KKworCWlvLT5zdHJlYW0tPnphbGxvYyA9IFpfTlVMTDsKKwlpby0+c3RyZWFtLT56 ZnJlZSA9IFpfTlVMTDsKKwlpby0+c3RyZWFtLT5vcGFxdWUgPSBaX05VTEw7CisJaW8tPnN0 cmVhbS0+YXZhaWxfaW4gPSAwOworCWlvLT5zdHJlYW0tPm5leHRfaW4gPSBaX05VTEw7CisJ aWYgKGluZmxhdGVJbml0Mihpby0+c3RyZWFtLCAtTUFYX1dCSVRTKSAhPSBaX09LKSB7CisJ CWZldGNoX3N5c2VycigpOworCQlmcmVlKGlvLT5zb3VyY2UpOworCQlmcmVlKGlvKTsKKwkJ cmV0dXJuIChOVUxMKTsKKwl9CisKKwlmID0gZnVub3BlbihpbywgJmh0dHBfaW5mbGF0ZV9y ZWFkZm4sIE5VTEwsIE5VTEwsICZodHRwX2luZmxhdGVfY2xvc2Vmbik7CisKKwlpZiAoZiA9 PSBOVUxMKSB7CisJCWZldGNoX3N5c2VycigpOworCQlmcmVlKGlvLT5zb3VyY2UpOworCQlm cmVlKGlvKTsKKwkJcmV0dXJuIChOVUxMKTsKKwl9CisJcmV0dXJuIGY7Cit9CiAKIC8qKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKgogICogSGVscGVyIGZ1bmN0aW9ucyBmb3IgdGFsa2luZyB0 byB0aGUgc2VydmVyIGFuZCBwYXJzaW5nIGl0cyByZXBsaWVzCkBAIC0zMzYsNiArNDMwLDcg QEAKIAloZHJfZXJyb3IgPSAtMSwKIAloZHJfZW5kID0gMCwKIAloZHJfdW5rbm93biA9IDEs CisJaGRyX2NvbnRlbnRfZW5jb2RpbmcsCiAJaGRyX2NvbnRlbnRfbGVuZ3RoLAogCWhkcl9j b250ZW50X3JhbmdlLAogCWhkcl9sYXN0X21vZGlmaWVkLApAQCAtMzQ5LDYgKzQ0NCw3IEBA CiAJaGRyX3QJCSBudW07CiAJY29uc3QgY2hhcgkqbmFtZTsKIH0gaGRyX25hbWVzW10gPSB7 CisJeyBoZHJfY29udGVudF9lbmNvZGluZywJCSJDb250ZW50LUVuY29kaW5nIiB9LAogCXsg aGRyX2NvbnRlbnRfbGVuZ3RoLAkJIkNvbnRlbnQtTGVuZ3RoIiB9LAogCXsgaGRyX2NvbnRl bnRfcmFuZ2UsCQkiQ29udGVudC1SYW5nZSIgfSwKIAl7IGhkcl9sYXN0X21vZGlmaWVkLAkJ Ikxhc3QtTW9kaWZpZWQiIH0sCkBAIC00OTYsNiArNTkyLDI0IEBACiB9CiAKIC8qCisgKiBQ YXJzZSBhIGNvbnRlbnQtZW5jb2RpbmcgaGVhZGVyCisgKi8KK2Z1bm9wZW5fZnVuY3Rpb24K K2h0dHBfcGFyc2VfZW5jb2RpbmcoY29uc3QgY2hhciAqcCwgb2ZmX3QgKmxlbmd0aCkKK3sK KwlvZmZfdCBsZW4gPSBzaXplb2YoImd6aXAiKTsKKwkvKiBub3QgeWV0IHN1cHBvcnRlZAor CWlmIChzdHJuY21wKHAsICJnemlwIiwgbGVuKSAgPT0gMCkKKwkJcmV0dXJuICZodHRwIF9m dW5vcGVuX2d1bnppcDsgKi8KKworCWxlbiA9IHNpemVvZigiZGVmbGF0ZSIpOworCWlmIChz dHJuY21wKHAsICJkZWZsYXRlIiwgbGVuKSA9PSAwKQorCQlyZXR1cm4gJmh0dHBfZnVub3Bl bl9pbmZsYXRlOworCisJcmV0dXJuICZodHRwX2Z1bm9wZW5fcmF3OworfQorCisvKgogICog UGFyc2UgYSBjb250ZW50LWxlbmd0aCBoZWFkZXIKICAqLwogc3RhdGljIGludApAQCAtODAw LDYgKzkxNCw3IEBACiAJY29ubl90ICpjb25uOwogCXN0cnVjdCB1cmwgKnVybCwgKm5ldzsK IAlpbnQgY2h1bmtlZCwgZGlyZWN0LCBuZWVkX2F1dGgsIG5vcmVkaXJlY3QsIHZlcmJvc2U7 CisJZnVub3Blbl9mdW5jdGlvbiBmdW5vcGVuZm47CiAJaW50IGUsIGksIG4sIHZhbDsKIAlv ZmZfdCBvZmZzZXQsIGNsZW5ndGgsIGxlbmd0aCwgc2l6ZTsKIAl0aW1lX3QgbXRpbWU7CkBA IC04MzQsNiArOTQ5LDcgQEAKIAkJbGVuZ3RoID0gLTE7CiAJCXNpemUgPSAtMTsKIAkJbXRp bWUgPSAwOworCQlmdW5vcGVuZm4gPSAmaHR0cF9mdW5vcGVuX3JhdzsKIAogCQkvKiBjaGVj ayBwb3J0ICovCiAJCWlmICghdXJsLT5wb3J0KQpAQCAtOTE5LDYgKzEwMzUsNyBAQAogCQkJ aHR0cF9jbWQoY29ubiwgIlVzZXItQWdlbnQ6ICVzICIgX0xJQkZFVENIX1ZFUiwgZ2V0cHJv Z25hbWUoKSk7CiAJCWlmICh1cmwtPm9mZnNldCA+IDApCiAJCQlodHRwX2NtZChjb25uLCAi UmFuZ2U6IGJ5dGVzPSVsbGQtIiwgKGxvbmcgbG9uZyl1cmwtPm9mZnNldCk7CisJCWh0dHBf Y21kKGNvbm4sICJBY2NlcHQtRW5jb2Rpbmc6IGRlZmxhdGUiKTsKIAkJaHR0cF9jbWQoY29u biwgIkNvbm5lY3Rpb246IGNsb3NlIik7CiAJCWh0dHBfY21kKGNvbm4sICIiKTsKIApAQCAt OTk5LDYgKzExMTYsOSBAQAogCQkJY2FzZSBoZHJfZXJyb3I6CiAJCQkJaHR0cF9zZXRlcnIo SFRUUF9QUk9UT0NPTF9FUlJPUik7CiAJCQkJZ290byBvdWNoOworCQkJY2FzZSBoZHJfY29u dGVudF9lbmNvZGluZzoKKwkJCQlmdW5vcGVuZm4gPSBodHRwX3BhcnNlX2VuY29kaW5nKHAs ICZsZW5ndGgpOworCQkJCWJyZWFrOwogCQkJY2FzZSBoZHJfY29udGVudF9sZW5ndGg6CiAJ CQkJaHR0cF9wYXJzZV9sZW5ndGgocCwgJmNsZW5ndGgpOwogCQkJCWJyZWFrOwpAQCAtMTEx OSw3ICsxMjM5LDkgQEAKIAogCS8qIGZpbGwgaW4gc3RhdHMgKi8KIAlpZiAodXMpIHsKLQkJ dXMtPnNpemUgPSBzaXplOworCQkvKiBXZSBvbmx5IGtub3cgdGhlIHJlYWwgb3V0cHV0IHNp emUgZm9yIHVuY29tcHJlc3NlZCBkYXRhLiAqLworCQlpZiAoZnVub3BlbmZuID09ICZodHRw X2Z1bm9wZW5fcmF3KQorCQkJdXMtPnNpemUgPSBzaXplOwogCQl1cy0+YXRpbWUgPSB1cy0+ bXRpbWUgPSBtdGltZTsKIAl9CiAKQEAgLTExMzQsNyArMTI1Niw3IEBACiAJVVJMLT5sZW5n dGggPSBjbGVuZ3RoOwogCiAJLyogd3JhcCBpdCB1cCBpbiBhIEZJTEUgKi8KLQlpZiAoKGYg PSBodHRwX2Z1bm9wZW4oY29ubiwgY2h1bmtlZCkpID09IE5VTEwpIHsKKwlpZiAoKGYgPSAo KmZ1bm9wZW5mbikoY29ubiwgY2h1bmtlZCkpID09IE5VTEwpIHsKIAkJZmV0Y2hfc3lzZXJy KCk7CiAJCWdvdG8gb3VjaDsKIAl9Ci0tLSBzcmMvbGliL2xpYmZldGNoL01ha2VmaWxlLm9y aWcJMjAwOC0wNi0yOSAyMzo0NTozMi4wMDAwMDAwMDAgKzAyMDAKKysrIHNyYy9saWIvbGli ZmV0Y2gvTWFrZWZpbGUJMjAwOC0wNi0yOSAyMzo0MzoxMS4wMDAwMDAwMDAgKzAyMDAKQEAg LTIwLDYgKzIwLDggQEAKIExEQUREPQkJLWxzc2wgLWxjcnlwdG8KIC5lbmRpZgogCitMREFE RCs9CQktbHoKKwogQ0ZMQUdTKz0JLURGVFBfQ09NQklORV9DV0RTIC1sCiAKIENTVEQ/PQkJ Yzk5Cg== --------------050207020908070903030703--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?48691DD6.9000306>