Date: Thu, 13 May 2021 10:45:50 -0400 From: mike tancsa <mike@sentex.net> To: freebsd-fs <freebsd-fs@freebsd.org> Subject: speeding up zfs send | recv Message-ID: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net>
next in thread | raw e-mail | index | archive | help
For offsite storage, I have been doing a zfs send across a 10G link and noticed something I don't understand with respect to speed.=C2=A0 I have = a number of datasets I am sending, one which contains a great many Maildirs / files and others that a few large 30-60G files (vm disk images).=C2=A0 When I am sending the data set with the mail spool (many s= mall files and directories), transfer tends to be markedly slower.=C2=A0 Looki= ng at the cacti graph, it seems to hover around 500Mb/s through an aes128-gcm cipher when sending the mail spool, vs sending the dataset that has the VMs on it, around 2.5Gb/s (both on a 5min average)... Why would the mail spool send be so slow compared to the sends where datasets only have a few large files ? One thing I am wondering is, could it be due to the amount of snapshots I have ? For each, I have about 60-100 snapshots. I am only sending a copy based on the latest snapshot, but I guess that's a lot of calculations to go through in order to get a complete image. However, I would have thought that would impact both types of datasets equally ? e.g. on my oldest mailspool snapshot, I see 60G of difference from the oldest snapshot on a dataset that's about 600GB in size By contrast, the dataset with VM images, is 300G and the oldest snapshot shows just 16G of difference and has a total of 93 snapshots. Is there anything I can do to speed up the send ? The recv side has lots of spare CPU. I dont see the disk blocking at all.=C2=A0 The sending side= is pretty busy, but I would imagine equally busy across all data sets. sender is a recent RELENG_12, recv side is RELENG_13 as a side note, is zstd ever nice!=C2=A0 On a different dataset that has = a lot of big ass json files. I am seeing refcompressratio=C2=A0 22.15x=C2=A0= =C2=A0 vs 13.19x=C2=A0 for the old lz4. =C2=A0=C2=A0=C2=A0 ---Mike
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?866d6937-a4e8-bec3-d61b-07df3065fca9>