Date: Mon, 30 Jul 2012 18:56:00 +0100 From: "Clayton Milos" <clay@milos.co.za> To: <freebsd-stable@freebsd.org> Subject: GELI Only running at the speed of 1 thread Message-ID: <032101cd6e7c$a6f5a950$f4e0fbf0$@milos.co.za>
next in thread | raw e-mail | index | archive | help
Hi Pawel / All
I have a system with 2 AMD Opteron 285's in it with the second cores
disabled so effectively a 2 CPU single core each setup.
I run the following:
#> kldload geom_zero
#> geli onetime -s 4096 /dev/gzero
#> dd if=/dev/gzero of=/dev/null bs=1m count=1k
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 0.507832 secs (2114364037 bytes/sec)
#> dd if=/dev/gzero.eli of=/dev/null bs=1m count=1k
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 15.919784 secs (67447009 bytes/sec)
67 MByte/sec is about right for the CPU. Problem is that if I run 2 of those
processes simultaneously then they both get over 60MB/s each. This looks to
me like the geli process isn't threading correctly.
Systat looks like this so you can see there are 2 processes but each is only
using about half of each CPU:
/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
Load Average |||||
/0% /10 /20 /30 /40 /50 /60 /70 /80 /90 /100
root idle XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
root g_eli[0] g XXXXXXXXXXXXXXXXXXXXXXXXXXX
root idle XXXXXXXXXXXXXXXXXXXXXXX
root g_eli[1] g XXXXXXXXXXXXXXXXXXXXXXX
root geom X
Top is also showing that only half of the CPU power is being used:
last pid: 3081; load averages: 1.36, 0.69, 0.40
up 0+00:14:13 18:45:15
53 processes: 1 running, 51 sleeping, 1 zombie
CPU: 0.2% user, 0.0% nice, 49.6% system, 0.4% interrupt, 49.8% idle
Mem: 819M Active, 91M Inact, 759M Wired, 2208K Cache, 635M Buf, 14G Free
Swap: 16G Total, 16G Free
With the tunables I've only changed the cache limit because I have the ram
for it and I want to use it for an 8 TB array.
#> sysctl kern.geom.eli
kern.geom.eli.batch: 0
kern.geom.eli.threads: 0
kern.geom.eli.overwrites: 5
kern.geom.eli.visible_passphrase: 0
kern.geom.eli.tries: 3
kern.geom.eli.debug: 0
kern.geom.eli.version: 6
kern.geom.eli.key_cache_misses: 0
kern.geom.eli.key_cache_hits: 4725333
kern.geom.eli.key_cache_limit: 32768
My /boot/loader.conf is:
ng_bpf_load="YES"
amdtemp_load="YES"
ubsec_load="YES"
vm.kmem_size="1024M"
vm.kmem_size_max="1024M"
vfs.zfs.arc_max="600M"
vfs.zfs.vdev.cache.size="8M"
vfs.zfs.txg.timeout="5"
kern.maxvnodes="250000"
kern.geom.eli.key_cache_limit="32768"
hint.lapic.1.disabled="1"
hint.lapic.2.disabled="1"
I have the ubsec in there because I have a BCM5823 Crypto Accelerator
installed. It does not support AES and is not used by GELI:
Kernel: GEOM_ELI: Device gzero.eli created.
kernel: GEOM_ELI: Encryption: AES-XTS 128
kernel: GEOM_ELI: Crypto: software
Any advice would be greatly appreciated. As I said it looks like GELI isn't
threading properly. Is it because of a setting I am running?
//Clay
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?032101cd6e7c$a6f5a950$f4e0fbf0$>