Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 8 Jan 2013 18:42:25 +0100
From:      Nicolas Rachinsky <fbsd-mas-0@ml.turing-complete.org>
To:        freebsd-fs@FreeBSD.org
Subject:   slowdown of zfs (tx->tx)
Message-ID:  <20130108174225.GA17260@mid.pc5.i.0x5.de>

next in thread | raw e-mail | index | archive | help
Hallo,

we have a problem that started to begin on one of our backup servers
recently.

We noticed that backups took an absurd amount of time (we aborted them
after several hours when the same backup usually takes minutes). We
first considered a disk broken and kicked it. But that didn't change
anything.

About one third of the rsync invocations end in a state where top
shows mostly tx->tx as the state. It seems that other rsync instances
that run at the same time or are started while one rsync is in this
state do also get into this state.

These rsyncs can be killed, but it takes a while (several seconds
or tens of seconds).

Repeating the same rsync invocation afterwards works (sometimes).

There is almost no disk activity during this time.

What can I do to debug or avoid this?




Some information:

The backups are taken with rsync (and --fake-super, but a patched
version that does not read extended attributes, since it seems
writing them all the time is faster than reading them).

sync is disabled for the whole pool.

root uses UFS and is on another set of disks (together with swap).



zpool status
  pool: pool1
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
  scan: scrub canceled on Fri Jan  4 10:31:35 2013
config:

        NAME                      STATE     READ WRITE CKSUM
        pool1                     DEGRADED     0     0     0
          raidz2-0                DEGRADED     0     0     0
            ada5                  ONLINE       0     0     0
            ada8                  ONLINE       0     0     0
            ada2                  ONLINE       0     0     0
            ada3                  ONLINE       0     0     0
            11846390416703086268  UNAVAIL      0     0     0  was /dev/dsk/ada1
            ada6                  ONLINE       0     0     0
            ada0                  ONLINE       0     0     1
            ada7                  ONLINE       0     0     0
            ada4                  ONLINE       0     0     3

errors: No known data errors


8.3-RELEASE-p5
with
http://svnweb.freebsd.org/base?view=revision&revision=240345
and
http://svnweb.freebsd.org/base?view=revision&revision=240632
applied

amd64 with 8G of RAM


Thanks in advance

Nicolas



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130108174225.GA17260>