From owner-svn-src-head@freebsd.org Thu Dec 14 22:11:37 2017 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C92FE918E9; Thu, 14 Dec 2017 22:11:37 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CDD537BE3E; Thu, 14 Dec 2017 22:11:36 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id vBEMBZrE013778; Thu, 14 Dec 2017 22:11:35 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id vBEMBZwf013772; Thu, 14 Dec 2017 22:11:35 GMT (envelope-from markj@FreeBSD.org) Message-Id: <201712142211.vBEMBZwf013772@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Thu, 14 Dec 2017 22:11:35 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r326861 - in head/tests/sys/geom/class: . mirror X-SVN-Group: head X-SVN-Commit-Author: markj X-SVN-Commit-Paths: in head/tests/sys/geom/class: . mirror X-SVN-Commit-Revision: 326861 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Dec 2017 22:11:37 -0000 Author: markj Date: Thu Dec 14 22:11:35 2017 New Revision: 326861 URL: https://svnweb.freebsd.org/changeset/base/326861 Log: Add some basic tests for gmirror read and write error handling. MFC after: 2 weeks Sponsored by: Dell EMC Isilon Added: head/tests/sys/geom/class/mirror/10_test.sh (contents, props changed) head/tests/sys/geom/class/mirror/11_test.sh (contents, props changed) head/tests/sys/geom/class/mirror/12_test.sh (contents, props changed) head/tests/sys/geom/class/mirror/13_test.sh (contents, props changed) Modified: head/tests/sys/geom/class/geom_subr.sh head/tests/sys/geom/class/mirror/Makefile Modified: head/tests/sys/geom/class/geom_subr.sh ============================================================================== --- head/tests/sys/geom/class/geom_subr.sh Thu Dec 14 20:48:50 2017 (r326860) +++ head/tests/sys/geom/class/geom_subr.sh Thu Dec 14 22:11:35 2017 (r326861) @@ -20,6 +20,16 @@ attach_md() echo $test_md } +detach_md() +{ + local test_md unit + + test_md=$1 + unit=${test_md#md} + mdconfig -d -u $unit || exit + sed -i '' "/^${test_md}$/d" $TEST_MDS_FILE || exit +} + geom_test_cleanup() { local test_md @@ -38,6 +48,7 @@ if [ $(id -u) -ne 0 ]; then echo '1..0 # SKIP tests must be run as root' exit 0 fi + # If the geom class isn't already loaded, try loading it. if ! kldstat -q -m g_${class}; then if ! geom ${class} load; then Added: head/tests/sys/geom/class/mirror/10_test.sh ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/tests/sys/geom/class/mirror/10_test.sh Thu Dec 14 22:11:35 2017 (r326861) @@ -0,0 +1,69 @@ +#!/bin/sh +# $FreeBSD$ + +# Test handling of read errors. + +. $(dirname $0)/conf.sh + +echo 1..3 + +set -e + +ddbs=2048 +regreadfp="debug.fail_point.g_mirror_regular_request_read" +m1=$(mktemp $base.XXXXXX) +m2=$(mktemp $base.XXXXXX) + +dd if=/dev/random of=$m1 bs=$ddbs count=1024 >/dev/null 2>&1 +dd if=/dev/zero of=$m2 bs=$ddbs count=1024 >/dev/null 2>&1 + +us0=$(attach_md -t vnode -f $m1) +us1=$(attach_md -t vnode -f $m2) + +gmirror label $name /dev/$us0 +gmirror insert $name /dev/$us1 +devwait +syncwait + +tmp1=$(mktemp $base.XXXXXX) +tmp2=$(mktemp $base.XXXXXX) + +EIO=5 +# gmirror should retry a failed read from the other mirror. +sysctl ${regreadfp}="1*return(${EIO})" +dd if=/dev/mirror/$name of=$tmp1 iseek=256 bs=$ddbs count=1 >/dev/null 2>&1 +dd if=/dev/$us1 of=$tmp2 iseek=256 bs=$ddbs count=1 >/dev/null 2>&1 +sysctl ${regreadfp}='off' + +if cmp -s $tmp1 $tmp2; then + echo "ok 1" +else + echo "not ok 1" +fi + +# Make sure that one of the mirrors was marked broken. +genid1=$(gmirror dump /dev/$us0 | awk '/^[[:space:]]*genid: /{print $2}') +genid2=$(gmirror dump /dev/$us1 | awk '/^[[:space:]]*genid: /{print $2}') +if [ $genid1 -eq $(($genid2 + 1)) -o $genid2 -eq $(($genid1 + 1)) ]; then + echo "ok 2" +else + echo "not ok 2" +fi + +# Force a retaste of the disconnected component. +if [ $(gmirror status -s $name | awk '{print $3}') = $us0 ]; then + detach_md $us1 + us1=$(attach_md -t vnode -f $m2) +else + detach_md $us0 + us0=$(attach_md -t vnode -f $m1) +fi + +# Make sure that the component wasn't re-added to the gmirror. +if [ $(gmirror status -s $name | wc -l) -eq 1 ]; then + echo "ok 3" +else + echo "not ok 3" +fi + +rm -f $m1 $m2 $tmp1 $tmp2 Added: head/tests/sys/geom/class/mirror/11_test.sh ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/tests/sys/geom/class/mirror/11_test.sh Thu Dec 14 22:11:35 2017 (r326861) @@ -0,0 +1,84 @@ +#!/bin/sh +# $FreeBSD$ + +# Test handling of read errors. + +. $(dirname $0)/conf.sh + +echo 1..4 + +set -e + +ddbs=2048 +regreadfp="debug.fail_point.g_mirror_regular_request_read" +m1=$(mktemp $base.XXXXXX) +m2=$(mktemp $base.XXXXXX) + +dd if=/dev/random of=$m1 bs=$ddbs count=1024 >/dev/null 2>&1 +dd if=/dev/zero of=$m2 bs=$ddbs count=1024 >/dev/null 2>&1 + +us0=$(attach_md -t vnode -f $m1) +us1=$(attach_md -t vnode -f $m2) + +gmirror label $name /dev/$us0 +gmirror insert $name /dev/$us1 +devwait +syncwait + +tmp1=$(mktemp $base.XXXXXX) +tmp2=$(mktemp $base.XXXXXX) + +ENXIO=6 +# gmirror has special handling for ENXIO. It does not mark the failed component +# as broken, allowing it to rejoin the mirror automatically when it appears. +sysctl ${regreadfp}="1*return(${ENXIO})" +dd if=/dev/mirror/$name of=$tmp1 iseek=512 bs=$ddbs count=1 >/dev/null 2>&1 +dd if=/dev/$us1 of=$tmp2 iseek=512 bs=$ddbs count=1 >/dev/null 2>&1 +sysctl ${regreadfp}='off' + +if cmp -s $tmp1 $tmp2; then + echo "ok 1" +else + echo "not ok 1" +fi + +# Verify that the genids still match after ENXIO. +genid1=$(gmirror dump /dev/$us0 | awk '/^[[:space:]]*genid: /{print $2}') +genid2=$(gmirror dump /dev/$us1 | awk '/^[[:space:]]*genid: /{print $2}') +if [ $genid1 -eq $genid2 ]; then + echo "ok 2" +else + echo "not ok 2" +fi + +# Trigger a syncid bump. +dd if=/dev/zero of=/dev/mirror/$name bs=$ddbs count=1 >/dev/null 2>&1 + +# The ENXIO+write should have caused a syncid bump. +syncid1=$(gmirror dump /dev/$us0 | awk '/^[[:space:]]*syncid: /{print $2}') +syncid2=$(gmirror dump /dev/$us1 | awk '/^[[:space:]]*syncid: /{print $2}') +if [ $syncid1 -eq $(($syncid2 + 1)) -o $syncid2 -eq $(($syncid1 + 1)) ]; then + echo "ok 3" +else + echo "not ok 3" +fi + +# Force a retaste of the disconnected component. +if [ $(gmirror status -s $name | awk '{print $3}') = $us0 ]; then + detach_md $us1 + us1=$(attach_md -t vnode -f $m2) +else + detach_md $us0 + us0=$(attach_md -t vnode -f $m1) +fi + +# Make sure that the retaste caused the mirror to automatically be re-added. +if [ $(gmirror status -s $name | wc -l) -eq 2 ]; then + echo "ok 4" +else + echo "not ok 4" +fi + +syncwait + +rm -f $m1 $m2 $tmp1 $tmp2 Added: head/tests/sys/geom/class/mirror/12_test.sh ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/tests/sys/geom/class/mirror/12_test.sh Thu Dec 14 22:11:35 2017 (r326861) @@ -0,0 +1,68 @@ +#!/bin/sh +# $FreeBSD$ + +# Test handling of write errors. + +. $(dirname $0)/conf.sh + +echo 1..3 + +set -e + +ddbs=2048 +regwritefp="debug.fail_point.g_mirror_regular_request_write" +m1=$(mktemp $base.XXXXXX) +m2=$(mktemp $base.XXXXXX) + +dd if=/dev/zero of=$m1 bs=$ddbs count=1024 >/dev/null 2>&1 +dd if=/dev/zero of=$m2 bs=$ddbs count=1024 >/dev/null 2>&1 + +us0=$(attach_md -t vnode -f $m1) +us1=$(attach_md -t vnode -f $m2) + +gmirror label $name /dev/$us0 /dev/$us1 +devwait + +tmp1=$(mktemp $base.XXXXXX) +tmp2=$(mktemp $base.XXXXXX) +dd if=/dev/random of=$tmp1 bs=$ddbs count=1 >/dev/null 2>&1 + +EIO=5 +# gmirror should kick one of the mirrors out after hitting EIO. +sysctl ${regwritefp}="1*return(${EIO})" +dd if=$tmp1 of=/dev/mirror/$name bs=$ddbs count=1 >/dev/null 2>&1 +dd if=/dev/mirror/$name of=$tmp2 bs=$ddbs count=1 >/dev/null 2>&1 +sysctl ${regwritefp}='off' + +if cmp -s $tmp1 $tmp2; then + echo "ok 1" +else + echo "not ok 1" +fi + +# Make sure that one of the mirrors was marked broken. +genid1=$(gmirror dump /dev/$us0 | awk '/^[[:space:]]*genid: /{print $2}') +genid2=$(gmirror dump /dev/$us1 | awk '/^[[:space:]]*genid: /{print $2}') +if [ $genid1 -eq $(($genid2 + 1)) -o $genid2 -eq $(($genid1 + 1)) ]; then + echo "ok 2" +else + echo "not ok 2" +fi + +# Force a retaste of the disconnected component. +if [ $(gmirror status -s $name | awk '{print $3}') = $us0 ]; then + detach_md $us1 + us1=$(attach_md -t vnode -f $m2) +else + detach_md $us0 + us0=$(attach_md -t vnode -f $m1) +fi + +# Make sure that the component wasn't re-added to the gmirror. +if [ $(gmirror status -s $name | wc -l) -eq 1 ]; then + echo "ok 3" +else + echo "not ok 3" +fi + +rm -f $m1 $m2 $tmp1 $tmp2 Added: head/tests/sys/geom/class/mirror/13_test.sh ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/tests/sys/geom/class/mirror/13_test.sh Thu Dec 14 22:11:35 2017 (r326861) @@ -0,0 +1,81 @@ +#!/bin/sh +# $FreeBSD$ + +# Test handling of write errors. + +. $(dirname $0)/conf.sh + +echo 1..4 + +set -e + +ddbs=2048 +regwritefp="debug.fail_point.g_mirror_regular_request_write" +m1=$(mktemp $base.XXXXXX) +m2=$(mktemp $base.XXXXXX) + +dd if=/dev/random of=$m1 bs=$ddbs count=1024 >/dev/null 2>&1 +dd if=/dev/zero of=$m2 bs=$ddbs count=1024 >/dev/null 2>&1 + +us0=$(attach_md -t vnode -f $m1) +us1=$(attach_md -t vnode -f $m2) + +gmirror label $name /dev/$us0 /dev/$us1 +devwait + +tmp1=$(mktemp $base.XXXXXX) +tmp2=$(mktemp $base.XXXXXX) + +dd if=/dev/random of=$tmp1 bs=$ddbs count=1 >/dev/null 2>&1 + +ENXIO=6 +# gmirror has special handling for ENXIO. It does not mark the failed component +# as broken, allowing it to rejoin the mirror automatically when it appears. +sysctl ${regwritefp}="1*return(${ENXIO})" +dd if=$tmp1 of=/dev/mirror/$name bs=$ddbs count=1 >/dev/null 2>&1 +dd if=/dev/mirror/$name of=$tmp2 bs=$ddbs count=1 >/dev/null 2>&1 +sysctl ${regwritefp}='off' + +if cmp -s $tmp1 $tmp2; then + echo "ok 1" +else + echo "not ok 1" +fi + +# Verify that the genids still match after ENXIO. +genid1=$(gmirror dump /dev/$us0 | awk '/^[[:space:]]*genid: /{print $2}') +genid2=$(gmirror dump /dev/$us1 | awk '/^[[:space:]]*genid: /{print $2}') +if [ $genid1 -eq $genid2 ]; then + echo "ok 2" +else + echo "not ok 2" +fi + +# The ENXIO should have caused a syncid bump. +syncid1=$(gmirror dump /dev/$us0 | awk '/^[[:space:]]*syncid: /{print $2}') +syncid2=$(gmirror dump /dev/$us1 | awk '/^[[:space:]]*syncid: /{print $2}') +if [ $syncid1 -eq $(($syncid2 + 1)) -o $syncid2 -eq $(($syncid1 + 1)) ]; then + echo "ok 3" +else + echo "not ok 3" +fi + +# Force a retaste of the disconnected component. +if [ $(gmirror status -s $name | awk '{print $3}') = $us0 ]; then + detach_md $us1 + us1=$(attach_md -t vnode -f $m2) +else + detach_md $us0 + us0=$(attach_md -t vnode -f $m1) +fi + +# Make sure that the retaste caused the mirror to automatically be re-added. +if [ $(gmirror status -s $name | wc -l) -eq 2 ]; then + echo "ok 4" +else + echo "not ok 4" +fi + +syncwait + +rm -f $m1 $m2 $tmp1 $tmp2 Modified: head/tests/sys/geom/class/mirror/Makefile ============================================================================== --- head/tests/sys/geom/class/mirror/Makefile Thu Dec 14 20:48:50 2017 (r326860) +++ head/tests/sys/geom/class/mirror/Makefile Thu Dec 14 22:11:35 2017 (r326861) @@ -13,6 +13,10 @@ TAP_TESTS_SH+= 6_test TAP_TESTS_SH+= 7_test TAP_TESTS_SH+= 8_test TAP_TESTS_SH+= 9_test +TAP_TESTS_SH+= 10_test +TAP_TESTS_SH+= 11_test +TAP_TESTS_SH+= 12_test +TAP_TESTS_SH+= 13_test ${PACKAGE}FILES+= conf.sh