From owner-freebsd-fs@FreeBSD.ORG Sun Jul 3 20:15:45 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63C44106564A for ; Sun, 3 Jul 2011 20:15:45 +0000 (UTC) (envelope-from kristoph.bauer@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 343288FC0C for ; Sun, 3 Jul 2011 20:15:44 +0000 (UTC) Received: by iwr19 with SMTP id 19so5399953iwr.13 for ; Sun, 03 Jul 2011 13:15:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=mdCoaPaWnTj9GpvjbYWXUvCy8RpBmof7h9OCJ261Ooc=; b=oHH5gApb4gTMF4esQAqFBynKdcRpQR0i9/DdK6naFePMwhNqJj8hibEOzKJXQRx3JD QShTWIJlAnPTbhNBiNoJS9YgDHnFYT/u+dPGUZEXuafvdVpkYpotQ8uXAW4KFL7QqTnE QrPJshJZVVSwaBlf7+xX9EZkEqCi01ghoES40= MIME-Version: 1.0 Received: by 10.231.47.207 with SMTP id o15mr5015230ibf.35.1309722496947; Sun, 03 Jul 2011 12:48:16 -0700 (PDT) Received: by 10.231.43.68 with HTTP; Sun, 3 Jul 2011 12:48:16 -0700 (PDT) Date: Sun, 3 Jul 2011 13:48:16 -0600 Message-ID: From: Kris Bauer To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: ZFS on geli X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Jul 2011 20:15:45 -0000 Hello, Related to a thread that I saw in freebsd-stable ( http://lists.freebsd.org/pipermail/freebsd-stable/2011-June/063145.html ), I wondered about the security of ZFS on geli. When using a zpool that is backed by geli disks, is there information contained in the unencrypted zpool.cache that would directly correspond to encrypted zfs metadata in a predictable location on the geli disks? Thanks. From owner-freebsd-fs@FreeBSD.ORG Mon Jul 4 11:07:01 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 581CC1065674 for ; Mon, 4 Jul 2011 11:07:01 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 455088FC18 for ; Mon, 4 Jul 2011 11:07:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p64B71kl040420 for ; Mon, 4 Jul 2011 11:07:01 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p64B70N4040417 for freebsd-fs@FreeBSD.org; Mon, 4 Jul 2011 11:07:00 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 4 Jul 2011 11:07:00 GMT Message-Id: <201107041107.p64B70N4040417@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2011 11:07:01 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o o kern/157929 fs [nfs] NFS slow read o kern/157728 fs [zfs] zfs (v28) incremental receive may leave behind t o kern/157722 fs [geli] unable to newfs a geli encrypted partition o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156933 fs [zfs] ZFS receive after read on readonly=on filesystem o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156168 fs [nfs] [panic] Kernel panic under concurrent access ove o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs o kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 o kern/154447 fs [zfs] [panic] Occasional panics - solaris assert somew p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153847 fs [nfs] [panic] Kernel panic from incorrect m_free in nf o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153520 fs [zfs] Boot from GPT ZFS root on HP BL460c G1 unstable o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small p kern/152488 fs [tmpfs] [patch] mtime of file updated when only inode o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o kern/151845 fs [smbfs] [patch] smbfs should be upgraded to support Un o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/151111 fs [zfs] vnodes leakage during zfs unmount o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/150207 fs zpool(1): zpool import -d /dev tries to open weird dev o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o bin/148296 fs [zfs] [loader] [patch] Very slow probe in /usr/src/sys o kern/148204 fs [nfs] UDP NFS causes overload o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147790 fs [zfs] zfs set acl(mode|inherit) fails on existing zfs o kern/147560 fs [zfs] [boot] Booting 8.1-PRERELEASE raidz system take o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an o bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142914 fs [zfs] ZFS performance degradation over time o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140134 fs [msdosfs] write and fsck destroy filesystem integrity o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139597 fs [patch] [tmpfs] tmpfs initializes va_gen but doesn't u o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133174 fs [msdosfs] [patch] msdosfs must support multibyte inter o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs f kern/130133 fs [panic] [zfs] 'kmem_map too small' caused by make clea o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs f kern/127375 fs [zfs] If vm.kmem_size_max>"1073741823" then write spee o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi f kern/126703 fs [panic] [zfs] _mtx_lock_sleep: recursed on non-recursi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files f sparc/123566 fs [zfs] zpool import issue: EOVERFLOW o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F f kern/120210 fs [zfs] [panic] reboot after panic: solaris assert: arc_ o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] [iconv] mount_msdosfs: msdosfs_iconv: Operat o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/33464 fs [ufs] soft update inconsistencies after system crash o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 232 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Jul 4 15:38:42 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9675F106566C for ; Mon, 4 Jul 2011 15:38:42 +0000 (UTC) (envelope-from gosand1982@yahoo.com) Received: from nm21-vm4.bullet.mail.ne1.yahoo.com (nm21-vm4.bullet.mail.ne1.yahoo.com [98.138.91.181]) by mx1.freebsd.org (Postfix) with SMTP id 48FDB8FC15 for ; Mon, 4 Jul 2011 15:38:42 +0000 (UTC) Received: from [98.138.90.56] by nm21.bullet.mail.ne1.yahoo.com with NNFMP; 04 Jul 2011 15:38:41 -0000 Received: from [98.138.88.238] by tm9.bullet.mail.ne1.yahoo.com with NNFMP; 04 Jul 2011 15:38:41 -0000 Received: from [127.0.0.1] by omp1038.mail.ne1.yahoo.com with NNFMP; 04 Jul 2011 15:38:41 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 722758.68186.bm@omp1038.mail.ne1.yahoo.com Received: (qmail 7380 invoked by uid 60001); 4 Jul 2011 15:38:41 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1309793921; bh=+/VyIHvgZXzTUEKGPMVB/zQ2lpGdVWpus3RmDwDS4O0=; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:MIME-Version:Content-Type; b=4jPdowOhQD0YzIBQilhpso3HZapXWUkSNLvZRKpyhihXu2w77inHnWBZjiIq79no4qaBeZS9toTotskZsRDEFIK2Qb/piVBDWGFlW1CRB4nx4/T28sPt/sjxlUp/96bJI8x9YrFpAa5cMqhkaftE7DXnUBNh+6/pzX33VVqC9YQ= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:MIME-Version:Content-Type; b=GxPoih3ufqGeXYzICO8BMp567/gZKg9VkRCnWDpfLegZ53PKc/2Y75Ho2AZvw16mKREsUsL/Cs/4RtoKIEfsivDuET0rlOzIyAPoMUvKP8Zudf9jWRBo67oOsZUg09mQ+t6Fjaw9Alm9PGSN+or1LbBH6FeB8tI7bfHeUsrucqU=; X-YMail-OSG: 7pGVR.EVM1lUUB0Wky390viqRSAvUUSc.k8Y8zg6a5gTkjf znOGjU_x8DqAyJq2glxWPB5dDJtnczXu4UADMQNi8NFxknaoZnHnl5rnoSzv BxHsxSNAY.7XReirkv9fjWB_EXnwOywy25UPR9kkqHJkxG2aFElA15Mb7XHZ BuyHWd0Vk257ef0cpu67MrVvAejn6rLIW._4_79aknNFO_v7Mwo0Zp.Ku7Mr 036O9f7j6S3nGigDh1z4ESiUvIkLMn1VrQQoS2GbfB4hkhEK1RcDHJke1VJS a2uhA9UoK4aurFMNTS7.gHiRYpix3poBKg1ar6PgWROAPWkdBojY- Received: from [173.164.238.34] by web120016.mail.ne1.yahoo.com via HTTP; Mon, 04 Jul 2011 08:38:41 PDT X-Mailer: YahooMailRC/572 YahooMailWebService/0.8.112.307740 Message-ID: <1309793921.2618.YahooMailRC@web120016.mail.ne1.yahoo.com> Date: Mon, 4 Jul 2011 08:38:41 -0700 (PDT) From: George Sanders To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailman-Approved-At: Mon, 04 Jul 2011 19:15:51 +0000 Subject: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2011 15:38:42 -0000 If I understand correctly, the interesting thing about a Sun x4500 (a "thumper") is that every one of the 48 disks has a direct path to the system board, allowing for full, independent throughput from every single drive. The downside, in 2011, is that it is a SATA2 system @ 3.0gbps. So, is this still an interesting system ? Is it still difficult to put together a system with 48 independent paths to the board, like the Thumper ? Assuming a raidZ3 configuration, does that drive independence help me more than being stuck at 3.0gbps hurts me ? Or would a "modern" system with a more typical drive contention scheme, but running at 6.0gbps, be superior ? Just curious ... From owner-freebsd-fs@FreeBSD.ORG Mon Jul 4 19:40:33 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 85F0D1065676 for ; Mon, 4 Jul 2011 19:40:33 +0000 (UTC) (envelope-from mj@feral.com) Received: from ns1.feral.com (ns1.feral.com [192.67.166.1]) by mx1.freebsd.org (Postfix) with ESMTP id 4E75C8FC12 for ; Mon, 4 Jul 2011 19:40:33 +0000 (UTC) Received: from [192.168.135.103] (c-24-7-47-62.hsd1.ca.comcast.net [24.7.47.62]) (authenticated bits=0) by ns1.feral.com (8.14.4/8.14.4) with ESMTP id p64JLEH9038248 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Mon, 4 Jul 2011 12:21:15 -0700 (PDT) (envelope-from mj@feral.com) Message-ID: <4E1212A7.70405@feral.com> Date: Mon, 04 Jul 2011 12:21:11 -0700 From: Matthew Jacob Organization: Feral Software User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.18) Gecko/20110616 Thunderbird/3.1.11 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <1309793921.2618.YahooMailRC@web120016.mail.ne1.yahoo.com> In-Reply-To: <1309793921.2618.YahooMailRC@web120016.mail.ne1.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (ns1.feral.com [192.67.166.1]); Mon, 04 Jul 2011 12:21:15 -0700 (PDT) X-Mailman-Approved-At: Mon, 04 Jul 2011 20:26:23 +0000 Subject: Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2011 19:40:33 -0000 On 7/4/2011 8:38 AM, George Sanders wrote: > > If I understand correctly, the interesting thing about a Sun x4500 (a "thumper") > is that every one of the 48 disks has a direct path to the system board, > allowing for full, independent throughput from every single drive. > > The downside, in 2011, is that it is a SATA2 system @ 3.0gbps. > > So, is this still an interesting system ? Is it still difficult to put together > a system with 48 independent paths to the board, like the Thumper ? > > Assuming a raidZ3 configuration, does that drive independence help me more than > being stuck at 3.0gbps hurts me ? Or would a "modern" system with a more > typical drive contention scheme, but running at 6.0gbps, be superior ? > > J IMO, until you go with flash, even 1.5Gbps is more than adequate, particularly for independent busses. I mean, you're not going to really make use of speeds greater than rotational+density, right? From owner-freebsd-fs@FreeBSD.ORG Mon Jul 4 20:38:26 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 15188106566C for ; Mon, 4 Jul 2011 20:38:26 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [IPv6:2a01:348:0:15:5d59:5c40:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id 9E5F78FC26 for ; Mon, 4 Jul 2011 20:38:25 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id 98FC7E6C6B; Mon, 4 Jul 2011 21:38:24 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=mail; bh=XmXq6Vu+wDZT lNH1WXvq3T5bn8o=; b=EL1O0hVBo/ZodtB5bU/wziTrqHOPVjMixHeLRf8COC8s PwhFmg0aSmMHDb+kzUd8unpfcNDHGOIvMT4KwlIpK/5lCh3QsbkO4E9rbZCbQzxL gZpzMmSkyO4B0x024Q0I10udSoBAApMCY9/TpEBc2Gnk0i7wWgU/Y9668IA6DyQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=mail; b=WWPX+g VCi9ICDEh+g62+PE5Xz7azztgmAPC7NieQnvTWFpqsUFC0kwfZNmUZH3EzNL6pAh K6WM3eIvAD49YfVbhZ4ebxkT6xuM6crMPC6jLLfxhDzwc+sWohOp46insJ4x7md6 8QtJ0k/OblIHk5orzuK0+tLhIDOpoKcb5mGM0= Received: from [192.168.1.87] (188-222-18-231.zone13.bethere.co.uk [188.222.18.231]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id 7532FE63F5; Mon, 4 Jul 2011 21:38:24 +0100 (BST) Message-ID: <4E1224BE.1020508@cran.org.uk> Date: Mon, 04 Jul 2011 21:38:22 +0100 From: Bruce Cran User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: Matthew Jacob References: <1309793921.2618.YahooMailRC@web120016.mail.ne1.yahoo.com> <4E1212A7.70405@feral.com> In-Reply-To: <4E1212A7.70405@feral.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2011 20:38:26 -0000 On 04/07/2011 20:21, Matthew Jacob wrote: > > IMO, until you go with flash, even 1.5Gbps is more than adequate, > particularly for independent busses. I mean, you're not going to > really make use of speeds greater than rotational+density, right? With protocol overhead, I think 1.5 Gb is a bit of a limit - for example my Samsung disks can read data at 155 MB/s. -- Bruce Cran From owner-freebsd-fs@FreeBSD.ORG Mon Jul 4 21:15:27 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8FA931065678 for ; Mon, 4 Jul 2011 21:15:27 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta04.westchester.pa.mail.comcast.net (qmta04.westchester.pa.mail.comcast.net [76.96.62.40]) by mx1.freebsd.org (Postfix) with ESMTP id 3AB018FC13 for ; Mon, 4 Jul 2011 21:15:26 +0000 (UTC) Received: from omta22.westchester.pa.mail.comcast.net ([76.96.62.73]) by qmta04.westchester.pa.mail.comcast.net with comcast id 3xCx1h0031ap0As54xFT8C; Mon, 04 Jul 2011 21:15:27 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta22.westchester.pa.mail.comcast.net with comcast id 3xFQ1h00Q1t3BNj3ixFRps; Mon, 04 Jul 2011 21:15:26 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 0051F102C36; Mon, 4 Jul 2011 14:15:22 -0700 (PDT) Date: Mon, 4 Jul 2011 14:15:22 -0700 From: Jeremy Chadwick To: Bruce Cran Message-ID: <20110704211522.GA43675@icarus.home.lan> References: <1309793921.2618.YahooMailRC@web120016.mail.ne1.yahoo.com> <4E1212A7.70405@feral.com> <4E1224BE.1020508@cran.org.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4E1224BE.1020508@cran.org.uk> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, Matthew Jacob Subject: Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2011 21:15:27 -0000 On Mon, Jul 04, 2011 at 09:38:22PM +0100, Bruce Cran wrote: > On 04/07/2011 20:21, Matthew Jacob wrote: > > > >IMO, until you go with flash, even 1.5Gbps is more than adequate, > >particularly for independent busses. I mean, you're not going to > >really make use of speeds greater than rotational+density, right? > > With protocol overhead, I think 1.5 Gb is a bit of a limit - for > example my Samsung disks can read data at 155 MB/s. Agreed. Likewise, Matthew, where did you get 1.5Gbps from? SATA revision 2 is 3.0Gbps. 1.5Gbps is SATA revision 1, which isn't on the Sun x4500, nor is it what the George mentioned to begin with. The x4500 uses Marvell 88SX6081 controllers, which are SATA rev 2. Quoting him: >>> If I understand correctly, the interesting thing about a Sun x4500 (a >>> "thumper") is that every one of the 48 disks has a direct path to the >>> system board, allowing for full, independent throughput from every >>> single drive. >>> >>> The downside, in 2011, is that it is a SATA2 system @ 3.0gbps. George, I wouldn't worry about SATA rev 2 being a bottleneck. However, you should probably be made aware of the fact that the on-board Marvell SATA controllers are PCI-X, so effectively each of your controllers is limited to around 1GByte/sec worth of bandwidth: http://techreport.com/discussions.x/13849 Eight (8) mechanical drives which do 120-130MByte/sec connected to one controller could hit this bottleneck. SSDs would hit the bottleneck easily. So if you're looking for "an insane speed demon", this system is probably too old. And as I understand it, exceeding PCI/PCI-X bandwidth can have weird effects on the system. If you don't plan on using the entire capacity of the x4500 (e.g. only using 24 disks or some such), ensuring you "distribute the load" of the disks across the controllers (say, 3 disks per controller) would guarantee you don't hit this bottleneck on any of your controllers. I have no idea how to determine what physical bay is wired to what physical controller on the x4500. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Mon Jul 4 21:44:29 2011 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9DB26106566C; Mon, 4 Jul 2011 21:44:29 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 756C18FC1F; Mon, 4 Jul 2011 21:44:29 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p64LiTaK029937; Mon, 4 Jul 2011 21:44:29 GMT (envelope-from rmacklem@freefall.freebsd.org) Received: (from rmacklem@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p64LiSQg029933; Mon, 4 Jul 2011 21:44:28 GMT (envelope-from rmacklem) Date: Mon, 4 Jul 2011 21:44:28 GMT Message-Id: <201107042144.p64LiSQg029933@freefall.freebsd.org> To: christian@teleservice.net, rmacklem@FreeBSD.org, freebsd-fs@FreeBSD.org From: rmacklem@FreeBSD.org Cc: Subject: Re: kern/157929: [nfs] NFS slow read X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2011 21:44:29 -0000 Synopsis: [nfs] NFS slow read State-Changed-From-To: open->feedback State-Changed-By: rmacklem State-Changed-When: Mon Jul 4 21:43:18 UTC 2011 State-Changed-Why: I have emailed the individual and am waiting for more information w.r.t. the slow NFS read rate he is experiencing. http://www.freebsd.org/cgi/query-pr.cgi?pr=157929 From owner-freebsd-fs@FreeBSD.ORG Tue Jul 5 07:18:07 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 440171065675 for ; Tue, 5 Jul 2011 07:18:07 +0000 (UTC) (envelope-from gerrit@pmp.uni-hannover.de) Received: from mrelay1.uni-hannover.de (mrelay1.uni-hannover.de [130.75.2.106]) by mx1.freebsd.org (Postfix) with ESMTP id C18C78FC19 for ; Tue, 5 Jul 2011 07:18:06 +0000 (UTC) Received: from www.pmp.uni-hannover.de (www.pmp.uni-hannover.de [130.75.117.2]) by mrelay1.uni-hannover.de (8.14.4/8.14.4) with ESMTP id p6570ds2007736; Tue, 5 Jul 2011 09:00:41 +0200 Received: from pmp.uni-hannover.de (unknown [130.75.117.3]) by www.pmp.uni-hannover.de (Postfix) with SMTP id 8A65F10A; Tue, 5 Jul 2011 09:00:39 +0200 (CEST) Date: Tue, 5 Jul 2011 09:00:39 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Jeremy Chadwick Message-Id: <20110705090039.17321256.gerrit@pmp.uni-hannover.de> In-Reply-To: <20110704211522.GA43675@icarus.home.lan> References: <1309793921.2618.YahooMailRC@web120016.mail.ne1.yahoo.com> <4E1212A7.70405@feral.com> <4E1224BE.1020508@cran.org.uk> <20110704211522.GA43675@icarus.home.lan> Organization: Albert-Einstein-Institut (MPI =?ISO-8859-1?Q?f=FCr?= Gravitationsphysik & IGP =?ISO-8859-1?Q?Universit=E4t?= Hannover) X-Mailer: Sylpheed 3.0.3 (GTK+ 2.22.1; amd64-portbld-freebsd8.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-PMX-Version: 5.5.9.395186, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2011.7.5.64514 Cc: freebsd-fs@freebsd.org, Matthew Jacob Subject: Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gerrit.kuehn@aei.mpg.de List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2011 07:18:07 -0000 On Mon, 4 Jul 2011 14:15:22 -0700 Jeremy Chadwick wrote about Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...): JC> Likewise, Matthew, where did you get 1.5Gbps from? SATA revision 2 is JC> 3.0Gbps. 1.5Gbps is SATA revision 1, which isn't on the Sun x4500, nor JC> is it what the George mentioned to begin with. The x4500 uses Marvell JC> 88SX6081 controllers, which are SATA rev 2. Quoting him: Thinking about the controllers: I am still looking for a viable replacement of LSI1068e-based cards for home-built ZFS servers. This LSI chipset does not support drives larger than 2TB which has become a serious limitation for me. I read that the LSI2008 is supposed to be some kind of follow-up, and it should be supported with recent FreeBSD's mps driver. However, I could not find any cheap (non-hw-raid) cards with 8 channels (or more) like there are with the old LSI1068 chipset. Are there any recommendations? cu Gerrit From owner-freebsd-fs@FreeBSD.ORG Tue Jul 5 08:05:03 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A95C106564A for ; Tue, 5 Jul 2011 08:05:03 +0000 (UTC) (envelope-from michael@fuckner.net) Received: from dedihh.fuckner.net (dedihh.fuckner.net [81.209.183.161]) by mx1.freebsd.org (Postfix) with ESMTP id AD6CA8FC14 for ; Tue, 5 Jul 2011 08:05:02 +0000 (UTC) Received: from dedihh.fuckner.net (localhost [127.0.0.1]) by dedihh.fuckner.net (Postfix) with ESMTP id 0883B11BB7; Tue, 5 Jul 2011 09:54:57 +0200 (CEST) X-Virus-Scanned: amavisd-new at fuckner.net Received: from dedihh.fuckner.net ([127.0.0.1]) by dedihh.fuckner.net (dedihh.fuckner.net [127.0.0.1]) (amavisd-new, port 10024) with SMTP id KTF7MtA1N3uc; Tue, 5 Jul 2011 09:54:51 +0200 (CEST) Received: from c64.rebootking.de (e176137029.adsl.alicedsl.de [85.176.137.29]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by dedihh.fuckner.net (Postfix) with ESMTPSA id 4326B11BAF; Tue, 5 Jul 2011 09:54:51 +0200 (CEST) Message-ID: <4E12C34A.8030409@fuckner.net> Date: Tue, 05 Jul 2011 09:54:50 +0200 From: Michael Fuckner User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110428 Fedora/3.1.10-1.fc15 Lightning/1.0b3pre Thunderbird/3.1.10 MIME-Version: 1.0 To: gerrit.kuehn@aei.mpg.de References: <1309793921.2618.YahooMailRC@web120016.mail.ne1.yahoo.com> <4E1212A7.70405@feral.com> <4E1224BE.1020508@cran.org.uk> <20110704211522.GA43675@icarus.home.lan> <20110705090039.17321256.gerrit@pmp.uni-hannover.de> In-Reply-To: <20110705090039.17321256.gerrit@pmp.uni-hannover.de> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, Matthew Jacob Subject: Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2011 08:05:03 -0000 On 07/05/2011 09:00 AM, Gerrit K=FChn wrote: > However, I could not find any > cheap (non-hw-raid) cards with 8 channels (or more) like there are with= the > old LSI1068 chipset. Are there any recommendations? LSI 9211-8i- but I don't know anything about the price. Integrated Raid is currently not supported, but they work as HBA with 8.2-STABLE. Regards, Michael! From owner-freebsd-fs@FreeBSD.ORG Tue Jul 5 08:10:48 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 69024106566B for ; Tue, 5 Jul 2011 08:10:48 +0000 (UTC) (envelope-from gerrit@pmp.uni-hannover.de) Received: from mrelay1.uni-hannover.de (mrelay1.uni-hannover.de [130.75.2.106]) by mx1.freebsd.org (Postfix) with ESMTP id EECC98FC0A for ; Tue, 5 Jul 2011 08:10:47 +0000 (UTC) Received: from www.pmp.uni-hannover.de (www.pmp.uni-hannover.de [130.75.117.2]) by mrelay1.uni-hannover.de (8.14.4/8.14.4) with ESMTP id p658Adgs012343; Tue, 5 Jul 2011 10:10:40 +0200 Received: from pmp.uni-hannover.de (unknown [130.75.117.3]) by www.pmp.uni-hannover.de (Postfix) with SMTP id 22AE410A; Tue, 5 Jul 2011 10:10:39 +0200 (CEST) Date: Tue, 5 Jul 2011 10:10:39 +0200 From: Gerrit =?UTF-8?B?S8O8aG4=?= To: Michael Fuckner Message-Id: <20110705101039.701cc042.gerrit@pmp.uni-hannover.de> In-Reply-To: <4E12C34A.8030409@fuckner.net> References: <1309793921.2618.YahooMailRC@web120016.mail.ne1.yahoo.com> <4E1212A7.70405@feral.com> <4E1224BE.1020508@cran.org.uk> <20110704211522.GA43675@icarus.home.lan> <20110705090039.17321256.gerrit@pmp.uni-hannover.de> <4E12C34A.8030409@fuckner.net> Organization: Albert-Einstein-Institut (MPI =?UTF-8?B?ZsO8cg==?= Gravitationsphysik & IGP =?UTF-8?B?VW5pdmVyc2l0w6R0?= Hannover) X-Mailer: Sylpheed 3.0.3 (GTK+ 2.22.1; amd64-portbld-freebsd8.1) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-PMX-Version: 5.5.9.395186, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2011.7.5.80015 Cc: freebsd-fs@freebsd.org Subject: Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gerrit.kuehn@aei.mpg.de List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2011 08:10:48 -0000 On Tue, 05 Jul 2011 09:54:50 +0200 Michael Fuckner wrote about Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...): MF> LSI 9211-8i- but I don't know anything about the price. Something around 200,-€ as far as I can see. This boils down to more than 20,-€/disk which is about twice the price of a cheap LSI1068 card (around 11,-€/disk). If you build machines with 24, 36 or even more disks, this can become quite an issue. MF> Integrated Raid is currently not supported, but they work as HBA with MF> 8.2-STABLE. I guess the integrated raid is the problem when it comes to the price. I'm looking for something without integrated raid, which is usually much cheaper. Thanks for the hint, anyway. cu Gerrit From owner-freebsd-fs@FreeBSD.ORG Tue Jul 5 08:37:51 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E1399106566B for ; Tue, 5 Jul 2011 08:37:51 +0000 (UTC) (envelope-from michael@fuckner.net) Received: from dedihh.fuckner.net (dedihh.fuckner.net [81.209.183.161]) by mx1.freebsd.org (Postfix) with ESMTP id 983F28FC0A for ; Tue, 5 Jul 2011 08:37:51 +0000 (UTC) Received: from dedihh.fuckner.net (localhost [127.0.0.1]) by dedihh.fuckner.net (Postfix) with ESMTP id 54DA211F83; Tue, 5 Jul 2011 10:37:49 +0200 (CEST) X-Virus-Scanned: amavisd-new at fuckner.net Received: from dedihh.fuckner.net ([127.0.0.1]) by dedihh.fuckner.net (dedihh.fuckner.net [127.0.0.1]) (amavisd-new, port 10024) with SMTP id PYYE69R+zrSy; Tue, 5 Jul 2011 10:37:43 +0200 (CEST) Received: from c64.rebootking.de (e176137029.adsl.alicedsl.de [85.176.137.29]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by dedihh.fuckner.net (Postfix) with ESMTPSA id AC62E11F78; Tue, 5 Jul 2011 10:37:43 +0200 (CEST) Message-ID: <4E12CD57.7030403@fuckner.net> Date: Tue, 05 Jul 2011 10:37:43 +0200 From: Michael Fuckner User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110428 Fedora/3.1.10-1.fc15 Lightning/1.0b3pre Thunderbird/3.1.10 MIME-Version: 1.0 To: gerrit.kuehn@aei.mpg.de References: <1309793921.2618.YahooMailRC@web120016.mail.ne1.yahoo.com> <4E1212A7.70405@feral.com> <4E1224BE.1020508@cran.org.uk> <20110704211522.GA43675@icarus.home.lan> <20110705090039.17321256.gerrit@pmp.uni-hannover.de> <4E12C34A.8030409@fuckner.net> <20110705101039.701cc042.gerrit@pmp.uni-hannover.de> In-Reply-To: <20110705101039.701cc042.gerrit@pmp.uni-hannover.de> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2011 08:37:52 -0000 On 07/05/2011 10:10 AM, Gerrit K=C3=BChn wrote: > On Tue, 05 Jul 2011 09:54:50 +0200 Michael Fuckner > wrote about Re: Are thumpers still interesting in 2011 ? (raidz3 on x45= 00 > @ 3.0gbps ...): >=20 > MF> LSI 9211-8i- but I don't know anything about the price. >=20 > Something around 200,-=E2=82=AC as far as I can see. This boils down to= more than > 20,-=E2=82=AC/disk which is about twice the price of a cheap LSI1068 ca= rd (around > 11,-=E2=82=AC/disk). Of Course 3GBit is cheaper than 6GBit- 6GBit is new. > If you build machines with 24, 36 or even more disks, this > can become quite an issue. if you have so many disks I would recommend a chassis with expander Backplane (like Supermicro 846E16). You have 4x6GBit to the backplane which should be enough even if you connect 24 HDD. > MF> Integrated Raid is currently not supported, but they work as HBA wi= th > MF> 8.2-STABLE. >=20 > I guess the integrated raid is the problem when it comes to the price. = I'm > looking for something without integrated raid, which is usually much > cheaper. Thanks for the hint, anyway. LSI 3081E-R can also do integrated raid as well- 3801E with external connectors can't do this because of the missing NVRAM Chip. IMHO integrated raid it is supported in FreeBSD. Regards, Michael! From owner-freebsd-fs@FreeBSD.ORG Tue Jul 5 09:07:27 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3901B106566B for ; Tue, 5 Jul 2011 09:07:27 +0000 (UTC) (envelope-from gerrit@pmp.uni-hannover.de) Received: from mrelay1.uni-hannover.de (mrelay1.uni-hannover.de [130.75.2.106]) by mx1.freebsd.org (Postfix) with ESMTP id BD3BF8FC0A for ; Tue, 5 Jul 2011 09:07:26 +0000 (UTC) Received: from www.pmp.uni-hannover.de (www.pmp.uni-hannover.de [130.75.117.2]) by mrelay1.uni-hannover.de (8.14.4/8.14.4) with ESMTP id p6597LhW015963; Tue, 5 Jul 2011 11:07:23 +0200 Received: from pmp.uni-hannover.de (unknown [130.75.117.3]) by www.pmp.uni-hannover.de (Postfix) with SMTP id 9F7A610A; Tue, 5 Jul 2011 11:07:21 +0200 (CEST) Date: Tue, 5 Jul 2011 11:07:21 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Michael Fuckner Message-Id: <20110705110721.7506f649.gerrit@pmp.uni-hannover.de> In-Reply-To: <4E12CD57.7030403@fuckner.net> References: <1309793921.2618.YahooMailRC@web120016.mail.ne1.yahoo.com> <4E1212A7.70405@feral.com> <4E1224BE.1020508@cran.org.uk> <20110704211522.GA43675@icarus.home.lan> <20110705090039.17321256.gerrit@pmp.uni-hannover.de> <4E12C34A.8030409@fuckner.net> <20110705101039.701cc042.gerrit@pmp.uni-hannover.de> <4E12CD57.7030403@fuckner.net> Organization: Albert-Einstein-Institut (MPI =?ISO-8859-1?Q?f=FCr?= Gravitationsphysik & IGP =?ISO-8859-1?Q?Universit=E4t?= Hannover) X-Mailer: Sylpheed 3.0.3 (GTK+ 2.22.1; amd64-portbld-freebsd8.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-PMX-Version: 5.5.9.395186, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2011.7.5.85714 Cc: freebsd-fs@freebsd.org, gerrit.kuehn@aei.mpg.de Subject: Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gerrit.kuehn@aei.mpg.de List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2011 09:07:27 -0000 On Tue, 05 Jul 2011 10:37:43 +0200 Michael Fuckner wrote about Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...): MF> if you have so many disks I would recommend a chassis with expander MF> Backplane (like Supermicro 846E16). You have 4x6GBit to the backplane MF> which should be enough even if you connect 24 HDD. I think then I get an SFF8087 lane (or even infiniband, which is even more expensive) with 24 drives connected and face the same issue: which controller do I connect this to? cu Gerrit From owner-freebsd-fs@FreeBSD.ORG Tue Jul 5 10:00:23 2011 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 57A211065670 for ; Tue, 5 Jul 2011 10:00:23 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 2E4F88FC15 for ; Tue, 5 Jul 2011 10:00:23 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p65A0NQI034559 for ; Tue, 5 Jul 2011 10:00:23 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p65A0Ntg034557; Tue, 5 Jul 2011 10:00:23 GMT (envelope-from gnats) Date: Tue, 5 Jul 2011 10:00:23 GMT Message-Id: <201107051000.p65A0Ntg034557@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Dominic Fandrey Cc: Subject: Re: kern/140134: [msdosfs] write and fsck destroy filesystem integrity X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Dominic Fandrey List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2011 10:00:23 -0000 The following reply was made to PR kern/140134; it has been noted by GNATS. From: Dominic Fandrey To: bug-followup@FreeBSD.org, kamikaze@bsdforen.de Cc: Subject: Re: kern/140134: [msdosfs] write and fsck destroy filesystem integrity Date: Tue, 05 Jul 2011 11:38:24 +0200 I request closing. This hasn't happened in a very long time and a lot of code has changed since. Regards -- A: Because it fouls the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail? From owner-freebsd-fs@FreeBSD.ORG Tue Jul 5 12:17:48 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0223D106564A for ; Tue, 5 Jul 2011 12:17:48 +0000 (UTC) (envelope-from michael@fuckner.net) Received: from dedihh.fuckner.net (dedihh.fuckner.net [81.209.183.161]) by mx1.freebsd.org (Postfix) with ESMTP id AB7108FC1B for ; Tue, 5 Jul 2011 12:17:47 +0000 (UTC) Received: from dedihh.fuckner.net (localhost [127.0.0.1]) by dedihh.fuckner.net (Postfix) with ESMTP id DAB2B182C9; Tue, 5 Jul 2011 14:17:46 +0200 (CEST) X-Virus-Scanned: amavisd-new at fuckner.net Received: from dedihh.fuckner.net ([127.0.0.1]) by dedihh.fuckner.net (dedihh.fuckner.net [127.0.0.1]) (amavisd-new, port 10024) with SMTP id 0IrAiPhUwb1w; Tue, 5 Jul 2011 14:17:40 +0200 (CEST) Received: from c64.rebootking.de (e176137029.adsl.alicedsl.de [85.176.137.29]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by dedihh.fuckner.net (Postfix) with ESMTPSA id 1A277182BF; Tue, 5 Jul 2011 14:17:40 +0200 (CEST) Message-ID: <4E1300E3.9070602@fuckner.net> Date: Tue, 05 Jul 2011 14:17:39 +0200 From: Michael Fuckner User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110428 Fedora/3.1.10-1.fc15 Lightning/1.0b3pre Thunderbird/3.1.10 MIME-Version: 1.0 To: gerrit.kuehn@aei.mpg.de References: <1309793921.2618.YahooMailRC@web120016.mail.ne1.yahoo.com> <4E1212A7.70405@feral.com> <4E1224BE.1020508@cran.org.uk> <20110704211522.GA43675@icarus.home.lan> <20110705090039.17321256.gerrit@pmp.uni-hannover.de> <4E12C34A.8030409@fuckner.net> <20110705101039.701cc042.gerrit@pmp.uni-hannover.de> <4E12CD57.7030403@fuckner.net> <20110705110721.7506f649.gerrit@pmp.uni-hannover.de> In-Reply-To: <20110705110721.7506f649.gerrit@pmp.uni-hannover.de> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2011 12:17:48 -0000 On 07/05/2011 11:07 AM, Gerrit K=FChn wrote: Hi Gerrit, > I think then I get an SFF8087 lane (or even infiniband, which is even > more expensive) with 24 drives connected and face the same issue: which > controller do I connect this to? You mean like CX4 Connectors for SAS or Infiniband over CX4/ QSFP Connectors? I'd stick with LSI 9211-4i + 846E16 Chassis (http://www.supermicro.nl/products/chassis/4U/846/SC846E16-R1200.cfm) SAS Controllers (HBA or RAID) typicially use Chips with 8 external Ports. If you buy a Controller with more ports, they solder an Expanderchip to the raidcard (LSI SAS2x36, http://lsi.com/products/storagecomponents/Pages/LSISAS2x36.aspx or PMC PM8005, http://www.pmc-sierra.com/products/details/pm8005/). So in both ways (HBA with 24 ports, single cables to each HDD) or HBA with 4/8 ports, Expanderchip on the backplane), the topology is identical- and the expandersolution has less cabeling which makes the whole system more reliable. Regards, Michael! From owner-freebsd-fs@FreeBSD.ORG Tue Jul 5 12:20:55 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 60E4E1065672 for ; Tue, 5 Jul 2011 12:20:55 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 1C7FD8FC15 for ; Tue, 5 Jul 2011 12:20:54 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1Qe4cX-0004sa-Qm for freebsd-fs@freebsd.org; Tue, 05 Jul 2011 14:20:53 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 05 Jul 2011 14:20:53 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 05 Jul 2011 14:20:53 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Tue, 05 Jul 2011 14:20:39 +0200 Lines: 17 Message-ID: References: <1309793921.2618.YahooMailRC@web120016.mail.ne1.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.12) Gecko/20101102 Thunderbird/3.1.6 In-Reply-To: <1309793921.2618.YahooMailRC@web120016.mail.ne1.yahoo.com> X-Enigmail-Version: 1.1.2 Subject: Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2011 12:20:55 -0000 On 04/07/2011 17:38, George Sanders wrote: > > > If I understand correctly, the interesting thing about a Sun x4500 (a "thumper") > is that every one of the 48 disks has a direct path to the system board, > allowing for full, independent throughput from every single drive. > > The downside, in 2011, is that it is a SATA2 system @ 3.0gbps. > > So, is this still an interesting system ? Is it still difficult to put together > a system with 48 independent paths to the board, like the Thumper ? It's a bit outdated, but here are the plans for a do-it-yourself slightly scaled down Thumper-like server: http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ From owner-freebsd-fs@FreeBSD.ORG Tue Jul 5 14:30:14 2011 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B6617106566B for ; Tue, 5 Jul 2011 14:30:14 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id A6C2E8FC17 for ; Tue, 5 Jul 2011 14:30:14 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p65EUEFX092495 for ; Tue, 5 Jul 2011 14:30:14 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p65EUEEC092490; Tue, 5 Jul 2011 14:30:14 GMT (envelope-from gnats) Date: Tue, 5 Jul 2011 14:30:14 GMT Message-Id: <201107051430.p65EUEEC092490@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Nathan Bowyer Cc: Subject: Re: kern/156781: [zfs] zfs is losing the snapshot directory, X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Nathan Bowyer List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2011 14:30:14 -0000 The following reply was made to PR kern/156781; it has been noted by GNATS. From: Nathan Bowyer To: bug-followup@FreeBSD.org, bmeyer@mesoft.com.au Cc: Subject: Re: kern/156781: [zfs] zfs is losing the snapshot directory, Date: Tue, 5 Jul 2011 08:56:27 -0500 --0015174c11fca119b604a752d913 Content-Type: text/plain; charset=ISO-8859-1 I have had this occur on amd64 with 8.2-RELEASE. I also came to the conclusion that it takes a reboot to fix the issue. Is there any activity on this bug? Thanks, Nathan --0015174c11fca119b604a752d913 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I have had this occur on amd64 with 8.2-RELEASE. =A0I also came to the conc= lusion that it takes a reboot to fix the issue.

Is there= any activity on this bug?

Thanks,
Natha= n
--0015174c11fca119b604a752d913-- From owner-freebsd-fs@FreeBSD.ORG Tue Jul 5 19:19:43 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E5E95106564A for ; Tue, 5 Jul 2011 19:19:43 +0000 (UTC) (envelope-from cjr@cruwe.de) Received: from cruwe.de (cruwe.de [188.40.164.98]) by mx1.freebsd.org (Postfix) with ESMTP id 98EB18FC13 for ; Tue, 5 Jul 2011 19:19:43 +0000 (UTC) Received: from cruwe.de (unknown [127.0.0.4]) by cruwe.de (Postfix) with ESMTP id 1A7991C07B for ; Tue, 5 Jul 2011 20:59:54 +0200 (CEST) Received: by cruwe.de (Postfix, from userid 65534) id EFB871C07A; Tue, 5 Jul 2011 20:59:53 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.cruwe.de X-Spam-Level: X-Spam-Status: No, score=-1.0 required=4.0 tests=ALL_TRUSTED autolearn=unavailable version=3.3.1 Received: from dijkstra (p5B37A364.dip.t-dialin.net [91.55.163.100]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by cruwe.de (Postfix) with ESMTPSA id 18EF51C06F for ; Tue, 5 Jul 2011 20:59:52 +0200 (CEST) Date: Tue, 5 Jul 2011 20:59:48 +0200 From: "Christopher J. Ruwe" To: Message-ID: <20110705205948.7eba5b40@dijkstra> X-Mailer: Claws Mail 3.7.9 (GTK+ 2.22.1; amd64-portbld-freebsd8.2) Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/_j1YzQJ_1Mk_pGM4s2u4/Gq" X-Virus-Scanned: ClamAV on mail.cruwe.de using ClamSMTP Subject: zpool-zfs'es on a GELI-encrypted volume are not mounted at boot [patch included] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2011 19:19:44 -0000 --MP_/_j1YzQJ_1Mk_pGM4s2u4/Gq Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline I run my notebook under FreeBSD 8.2-stable, r223699. I have setup my disks with ZFS so that I boot from a very small rpool and mount datasets, among these /usr from another pool configured on top of an AES encrypted GELI. When installing a new world using this setup, it is necessary to manually adapt /etc/rc.d/mountcritlocal, mountcritlocal_start() to do a zfs mount -a. Failing to do so causes my rootpool to be mounted (which follows from rc.conf), then the GELI volume to be unlocked. After this, the boot routine hangs, as /usr (which resides) on the encrypted vol, which is not mounted, as the canonical zfs mounts are mounted before GELI. I cannot imagine that I am the only one to run ZFSes on an encrypted GELI volume. Am I booting this setup in an inadvisable way, so that I need to run into problems? If not, then it might be an idea to include a zfs mount -a in mountcritlocal in the canonical rc.d-setup. Am I getting this right or could you please comment? Thank you, cheers, -- Christopher J. Ruwe TZ GMT + 2 --MP_/_j1YzQJ_1Mk_pGM4s2u4/Gq Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=zfs-mountcritlocal.patch *** /usr/src/etc/rc.d/mountcritlocal 2011-06-30 21:37:46.097575355 +0200 --- /etc/rc.d/mountcritlocal 2011-07-01 18:03:43.518493334 +0200 *************** *** 36,41 **** --- 36,42 ---- done mount_excludes=${mount_excludes%,} mount -a -t ${mount_excludes} + zfs mount -a err=$? check_startmsgs && echo '.' --MP_/_j1YzQJ_1Mk_pGM4s2u4/Gq-- From owner-freebsd-fs@FreeBSD.ORG Tue Jul 5 22:50:08 2011 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 737FB106566B for ; Tue, 5 Jul 2011 22:50:08 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 64B538FC14 for ; Tue, 5 Jul 2011 22:50:08 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p65Mo845049766 for ; Tue, 5 Jul 2011 22:50:08 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p65Mo8oT049765; Tue, 5 Jul 2011 22:50:08 GMT (envelope-from gnats) Date: Tue, 5 Jul 2011 22:50:08 GMT Message-Id: <201107052250.p65Mo8oT049765@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Brendon Meyer Cc: Subject: Re: kern/156781: [zfs] zfs is losing the snapshot directory, X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Brendon Meyer List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2011 22:50:08 -0000 The following reply was made to PR kern/156781; it has been noted by GNATS. From: Brendon Meyer To: Nathan Bowyer Cc: bug-followup@FreeBSD.org Subject: Re: kern/156781: [zfs] zfs is losing the snapshot directory, Date: Wed, 6 Jul 2011 08:23:32 +1000 On 05/07/2011, at 11:56 PM, Nathan Bowyer wrote: > I have had this occur on amd64 with 8.2-RELEASE. I also came to the = conclusion that it takes a reboot to fix the issue. > Is there any activity on this bug? Not sure to be honest. I'm going actually in the process of building a = new testing environment for a updated mail server and will probably be = able to give a more definitive answer a bit later (the build will be = done later this week and then I'll be letting my test scrips run mad on = it for a week or so). =20 The fact that you have had the same problems with AMD means that the = problem isn't SPARC specific. I am guessing that it is probably a = commit that was made between to the zfs or solaris code between Mon Feb = 28 16:07:24 EST 2011 and Sun Apr 24 21:28:12 EST 2011 (those are two = systems I have that are very similarly configured but the earlier one = doesn't have the problem whereas the later one does). =20 /BGM= From owner-freebsd-fs@FreeBSD.ORG Tue Jul 5 22:57:42 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DAA7F106564A; Tue, 5 Jul 2011 22:57:42 +0000 (UTC) (envelope-from smckay@internode.on.net) Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by mx1.freebsd.org (Postfix) with ESMTP id 1040F8FC14; Tue, 5 Jul 2011 22:57:41 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av0EACuSE0520E0N/2dsb2JhbAA1HqgNeL9HjUaGNgSifg Received: from ppp118-208-77-13.lns20.bne4.internode.on.net (HELO dungeon.home) ([118.208.77.13]) by ipmail06.adl6.internode.on.net with ESMTP; 06 Jul 2011 08:12:22 +0930 Received: from dungeon.home (localhost [127.0.0.1]) by dungeon.home (8.14.4/8.14.3) with ESMTP id p65Mfr9M002216; Wed, 6 Jul 2011 08:41:53 +1000 (EST) (envelope-from smckay@internode.on.net) Received: (from root@localhost) by dungeon.home (8.14.4/8.14.3/Submit) id p65MfqVA002215; Wed, 6 Jul 2011 08:41:52 +1000 (EST) (envelope-from mckay) Date: Wed, 6 Jul 2011 08:41:52 +1000 (EST) Message-Id: <201107052241.p65MfqVA002215@dungeon.home> From: Stephen McKay To: freebsd-fs@freebsd.org References: <201103081425.p28EPQtM002115@dungeon.home> Cc: Stephen McKay Subject: Constant minor ZFS corruption, probably solved X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2011 22:57:42 -0000 Perhaps you remember me struggling with a small but continuous amount of corruption on ZFS volumes with a new server we had built at work. Well, it's been 4 months now and although the machine sat in a corner for most of that time, I've now done enough tests so that I'm 90% certain what the problem is: Seagate's caching firmware. The specific drive model we are using is the 2TB Green ST2000DL003-9VT166 (CC32 firmware), their first 4KB sector drive model as far as I know. I would not be surprised to see bugs in Seagate's first stab at their ambitious "smart align" technology. Why do I think it's the firmware? I have a testing regime that involves copying about 1.2TB of data to the server via ssh while simultaneously keeping the server disks busy reading and seeking with repeated runs of find. After each copy, I run a scrub. Any copy plus scrub sequence that completes with zero checksum errors is a success. Any checksum errors at all mean it's a fail. With write caching enabled, every run has failed, and I've done this 20 or so times (with various other settings tweaked to no avail). But with write caching disabled (using kern.cam.ada.write_cache=0 for the onboard ports and "sdparm --clear=WCE" for the mps attached drives), I have run my test successfully 6 times in a row. To be sure that this was not a fluke, I turned write caching back on, producing checksum errors, then back off and have completed a further 4 successful runs. I can't see this as anything except bugs in the drive firmware, but I'm open to other reasonable interpretations. Have I missed anything? The only reason I'm 90% sure and not 100% is that I'm chasing very low error counts per run: around 20KB to 100KB wrong out of 1.2TB, which is between 1 and 8 millionths of 1 percent bad data. When trying to close the gap between 1 millionth of a percent and zero, any experiment could be just noise. :-( I really should dig up some of my old statistics textbooks and work out how many runs I must do before I can claim "certainty". Is there a statistician in the house? I should also note that the speed of the machine seems undiminished by disabling write caching. I left NCQ enabled and this seems to allow everything to run at full speed, at least under ZFS. If the machine had been crippled by disabling write caching, I would have believed that the general slowdown was lowering the stress on some dodgy part or that some race condition was being avoided. But with no noticeable slowdown? I have to conclude that the fault is just in the disks since that's the part directly affected by the setting. Except... Except for that last niggling doubt that when chasing 1 millionth of 1 percent bugs, even a tiny perturbation in operation may be nudging the system past a software or (other) hardware bug. Or, to put it another way, I'm certain that disabling write caching has given us a stable machine. And I'm 90% certain that it's because of bugs in Seagate's cache firmware. I hope someone else can replicate this and settle the issue. I am running -current from 17 June (though earlier versions made no difference, at least for failing tests, which was all I knew how to do until recently). The same failures occurred when running 8.2-release. Changing from ZFS v15 to ZFS v28 made no difference. A recap of our hardware, which has been slightly rearranged for recent testing, though none of it was replaced: Asus P7F-E (includes 6 3Gb/s SATA ports) PIKE2008 (8 port SAS card based on LSI2008 chip, supports 6Gb/s) Xeon X3440 (2.53GHz 4core with hyperthreading) Chenbro CSPC-41416AB rackmount case 2x 2GB 1333MHz ECC DDR3 RAM (Corsair) 2x Seagate ST3500418AS 500GB normal disks, for OS booting (on the PIKE) 12x Seagate ST2000DL003 2TB "green" disks (yes, with 4kB sectors) (6 disks on the onboard Intel SATA controller using ahci driver, 6 disks on the PIKE using the mps driver) The pool is arranged as 2 x 6 disk raidz2 vdevs, one vdev entirely on the onboard controller and one on the PIKE. ZFS has been set to use ashift=12 to align for lying 4kB sector drives. Jeremy has already said his piece about aiming too low on hardware but I've made many reliable small servers and workstations from ASUS motherboards, ECC RAM and consumer level disks. The occasional failure to achieve a stable cheap system occurs roughly as often as the failure rate for expensive gear, though my sample size for expensive gear is small. We weren't expecting to build a machine to eclipse all others, just a reliable disk store of a decent size. For our next machine, we might be more conservative on our disk purchases though, since it is becoming clear that all disk manufacturers are cutting even more corners on consumer level drives, more or less forcing raid and zfs users to use full speed (full heat, full noise, full power draw, full price) disks. BTW, if anyone has a direct line into Seagate or Western Digital I'd really love to know why nobody is following the standard written specificially for 4K sector drives. Why report the wrong physical sector size if no OS that uses that data gets it wrong? And where is the jumper for "I want native 4KB sectors with no translation"? Surely the engineers made one of those *before* they made the mind bending sector shuffling version? Cheers, Stephen. PS Jeremy, I swapped in various 2 and 4 port disk controllers during testing (none remain in there now). A cheap JMicron JMB363 card and a modestly priced Adaptec 1430SA worked well, but another cheap card using a Silicon Image 3132 chip produced silent corruption on reads. I've seen a small number of posts claiming the 3132 is known for silent corruption when both channels are busy, but you recommend this chip. Is there a way to tell the good cards from the bad ones? Is FreeBSD missing a driver (errata) fix? Or are there actually no good SiI 3132 cards? I'm keeping my 3132 card in the "bad hardware - driver testing only" bucket. From owner-freebsd-fs@FreeBSD.ORG Wed Jul 6 00:21:55 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5DEEF106564A for ; Wed, 6 Jul 2011 00:21:55 +0000 (UTC) (envelope-from jwd@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 358B38FC13 for ; Wed, 6 Jul 2011 00:21:55 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p660Lt0K041668 for ; Wed, 6 Jul 2011 00:21:55 GMT (envelope-from jwd@freefall.freebsd.org) Received: (from jwd@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p660LtEt041667 for freebsd-fs@FreeBSD.org; Wed, 6 Jul 2011 00:21:55 GMT (envelope-from jwd) Date: Wed, 6 Jul 2011 00:21:55 +0000 From: John To: freebsd-fs@FreeBSD.org Message-ID: <20110706002155.GB12224@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: Subject: avl_find() panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2011 00:21:55 -0000 Hi folks, I have a system that panic'd this morning, 4 day old current (2011-07-01_11.45pm). Message typed in from the console immediately after reboot. OS on ufs, data volumes on zfs. ZFS filesystem version 5 ZFS storage pool version 28 panic: avl_find() succedded inside avl_find() Unfortunately, I don't have a traceback for this. The comment in avl.c makes it seem like the avl code is enforcing uniqueness in calling code, esp. where it talks about kernel vs userland. I'll followup with more info if this replicates. Thoughts welcome. -John From owner-freebsd-fs@FreeBSD.ORG Wed Jul 6 00:29:07 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26321106564A for ; Wed, 6 Jul 2011 00:29:05 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta13.emeryville.ca.mail.comcast.net (qmta13.emeryville.ca.mail.comcast.net [76.96.27.243]) by mx1.freebsd.org (Postfix) with ESMTP id 0F5508FC19 for ; Wed, 6 Jul 2011 00:29:04 +0000 (UTC) Received: from omta17.emeryville.ca.mail.comcast.net ([76.96.30.73]) by qmta13.emeryville.ca.mail.comcast.net with comcast id 4NDd1h0041afHeLADQV2RJ; Wed, 06 Jul 2011 00:29:02 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta17.emeryville.ca.mail.comcast.net with comcast id 4Qak1h01X1t3BNj8dQallH; Wed, 06 Jul 2011 00:34:46 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id D555D102C36; Tue, 5 Jul 2011 17:29:00 -0700 (PDT) Date: Tue, 5 Jul 2011 17:29:00 -0700 From: Jeremy Chadwick To: John Message-ID: <20110706002900.GA70126@icarus.home.lan> References: <20110706002155.GB12224@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110706002155.GB12224@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@FreeBSD.org, freebsd-current@freebsd.org Subject: Re: avl_find() panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2011 00:29:07 -0000 On Wed, Jul 06, 2011 at 12:21:55AM +0000, John wrote: > I have a system that panic'd this morning, 4 day old current > (2011-07-01_11.45pm). Message typed in from the console immediately > after reboot. OS on ufs, data volumes on zfs. > > ZFS filesystem version 5 > ZFS storage pool version 28 > panic: avl_find() succedded inside avl_find() > > Unfortunately, I don't have a traceback for this. > > The comment in avl.c makes it seem like the avl code is enforcing > uniqueness in calling code, esp. where it talks about kernel vs > userland. > > I'll followup with more info if this replicates. Cross-posting is generally shunned, but since this is a current thing, adding freebsd-current to the CC list. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Wed Jul 6 00:51:34 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 524B0106566B; Wed, 6 Jul 2011 00:51:34 +0000 (UTC) (envelope-from jwd@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 38DA78FC1B; Wed, 6 Jul 2011 00:51:34 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p660pYZV069625; Wed, 6 Jul 2011 00:51:34 GMT (envelope-from jwd@freefall.freebsd.org) Received: (from jwd@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p660pYl7069624; Wed, 6 Jul 2011 00:51:34 GMT (envelope-from jwd) Date: Wed, 6 Jul 2011 00:51:34 +0000 From: John To: freebsd-current@freebsd.org, freebsd-fs@freebsd.org Message-ID: <20110706005134.GA60868@FreeBSD.org> References: <20110618043100.GA40745@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110618043100.GA40745@FreeBSD.org> User-Agent: Mutt/1.4.2.3i Cc: Subject: Re: LOR: vfs_lookup.c:501 ffs_vnops.c:261 vfs_subr.c:2134 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2011 00:51:34 -0000 Hi Folks, I just updated this particular systems to current as of this evening (2011-07-05 7:22pm EDT) and am still seeing this LOR. lock order reversal: 1st 0xfffffe003c893818 ufs (ufs) @ /usr/src.2011-07-05_7.22pm_EDT/sys/kern/vfs_lookup.c:501 2nd 0xffffff9f0c7eeef8 bufwait (bufwait) @ /usr/src.2011-07-05_7.22pm_EDT/sys/ufs/ffs/ffs_vnops.c:261 3rd 0xfffffe003c8689f8 ufs (ufs) @ /usr/src.2011-07-05_7.22pm_EDT/sys/kern/vfs_subr.c:2134 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x807 __lockmgr_args() at __lockmgr_args+0xd42 ffs_lock() at ffs_lock+0x8c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x47 vget() at vget+0x7b vfs_hash_get() at vfs_hash_get+0xd5 ffs_vgetf() at ffs_vgetf+0x48 softdep_sync_buf() at softdep_sync_buf+0x568 ffs_syncvnode() at ffs_syncvnode+0x293 ffs_truncate() at ffs_truncate+0x4c4 ufs_direnter() at ufs_direnter+0x6ed ufs_makeinode() at ufs_makeinode+0x250 VOP_CREATE_APV() at VOP_CREATE_APV+0x8d vn_open_cred() at vn_open_cred+0x46a kern_openat() at kern_openat+0x17f syscallenter() at syscallenter+0x1aa syscall() at syscall+0x4c Xfast_syscall() at Xfast_syscall+0xdd --- syscall (5, FreeBSD ELF64, open), rip = 0x800936b9c, rsp = 0x7fffffffdac8, rbp = 0 --- Cross-posting to -fs for more visibility. Thoughts welcome. -John ----- John's Original Message ----- > Hi folks, > > I'm seeing the following LOR in dmesg after my latest update this evening. > > # uname -a > FreeBSD zfscarp3p 9.0-CURRENT FreeBSD 9.0-CURRENT #0: Fri Jun 17 22:36:45 EDT 2011 root@zfscarp3p/usr/obj/usr/src.2011-06-17_9.36pm_EDT/sys/GENERIC amd6 > > WARNING: WITNESS option enabled, expect reduced performance. > Trying to mount root from ufs:/dev/da56s1a [rw]... > lock order reversal: > 1st 0xfffffe003d2f9278 ufs (ufs) @ /usr/src.2011-06-17_9.36pm_EDT/sys/kern/vfs_lookup.c:501 > 2nd 0xffffff9f0c7f0e58 bufwait (bufwait) @ /usr/src.2011-06-17_9.36pm_EDT/sys/ufs/ffs/ffs_vnops.c:261 > 3rd 0xfffffe003d371278 ufs (ufs) @ /usr/src.2011-06-17_9.36pm_EDT/sys/kern/vfs_subr.c:2134 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > kdb_backtrace() at kdb_backtrace+0x37 > _witness_debugger() at _witness_debugger+0x2e > witness_checkorder() at witness_checkorder+0x807 > __lockmgr_args() at __lockmgr_args+0xd42 > ffs_lock() at ffs_lock+0x8c > VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b > _vn_lock() at _vn_lock+0x47 > vget() at vget+0x7b > vfs_hash_get() at vfs_hash_get+0xd5 > ffs_vgetf() at ffs_vgetf+0x48 > softdep_sync_buf() at softdep_sync_buf+0x56a > ffs_syncvnode() at ffs_syncvnode+0x293 > ffs_truncate() at ffs_truncate+0x4c4 > ufs_direnter() at ufs_direnter+0x6ed > ufs_makeinode() at ufs_makeinode+0x250 > VOP_CREATE_APV() at VOP_CREATE_APV+0x8d > vn_open_cred() at vn_open_cred+0x46a > kern_openat() at kern_openat+0x17f > syscallenter() at syscallenter+0x1aa > syscall() at syscall+0x4c > Xfast_syscall() at Xfast_syscall+0xdd > --- syscall (5, FreeBSD ELF64, open), rip = 0x800936b7c, rsp = 0x7fffffffdac8, rbp = 0 --- > bce1: Gigabit link up! > > Only seems to happen once at boot time. > > Thanks, > John From owner-freebsd-fs@FreeBSD.ORG Wed Jul 6 01:15:44 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 927061065674 for ; Wed, 6 Jul 2011 01:15:44 +0000 (UTC) (envelope-from jwd@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7FA218FC17; Wed, 6 Jul 2011 01:15:44 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p661FicL087918; Wed, 6 Jul 2011 01:15:44 GMT (envelope-from jwd@freefall.freebsd.org) Received: (from jwd@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p661FiGe087917; Wed, 6 Jul 2011 01:15:44 GMT (envelope-from jwd) Date: Wed, 6 Jul 2011 01:15:44 +0000 From: John To: Rick Macklem , freebsd-fs@freebsd.org Message-ID: <20110706011544.GA69706@FreeBSD.org> References: <20110609133805.GA78874@FreeBSD.org> <1069270455.338453.1307636209760.JavaMail.root@erie.cs.uoguelph.ca> <20110610125939.GA69616@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110610125939.GA69616@FreeBSD.org> User-Agent: Mutt/1.4.2.3i Cc: Subject: Re: New NFS server stress test hang X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2011 01:15:44 -0000 ----- John's Original Message ----- > ----- Rick Macklem's Original Message ----- > > John De wrote: > > > ----- Rick Macklem's Original Message ----- > > > > John De wrote: > > > > > Hi, > > > > > > > > > > We've been running some stress tests of the new nfs server. > > > > > The system is at r222531 (head), 9 clients, two mounts each > > > > > to the server: > > > > > > > > > > mount_nfs -o > > > > > udp,nfsv3,rsize=32768,wsize=32768,noatime,nolockd,acregmin=1,acregmax=2,acdirmin=1,acdirmax=2,negnametimeo=2 > > > > > ${servera}:/vol/datsrc /c/$servera/vol/datsrc > > > > > mount_nfs -o > > > > > udp,nfsv3,rsize=32768,wsize=32768,noatime,nolockd,acregmin=1,acregmax=2,acdirmin=1,acdirmax=2,negnametimeo=0 > > > > > ${servera}:/vol/datgen /c/$servera/vol/datgen > > > > > > > > > > > > > > > The system is still up & responsive, simply no nfs services > > > > > are working. All (200) threads appear to be active, but not > > > > > doing anything. The debugger is not compiled into this kernel. > > > > > We can run any other tracing commands desired. We can also > > > > > rebuild the kernel with the debugger enabled for any kernel > > > > > debugging needed. > > > > > > > > > > --- long logs deleted --- > > > > > > > > How about a: > > > > ps axHlww <-- With the "H" we'll see what the nfsd server threads > > > > are up to > > > > procstat -kka > > > > > > > > Oh, and a couple of nfsstats a few seconds apart. It's what the > > > > counts > > > > are changing by that might tell us what is going on. (You can use > > > > "-z" > > > > to zero them out, if you have an nfsstat built from recent sources.) > > > > > > > > Also, does a new NFS mount attempt against the server do anything? > > > > > > > > Thanks in advance for help with this, rick > > > > > > Hi Rick, > > > > > > Here's the output. In general, the nfsd processes appear to be in > > > either nfsrvd_getcache(35 instances) or nfsrvd_updatecache(164) > > > sleeping on > > > "nfssrc". The server numbers don't appear to be moving. A showmount > > > from a > > > client system works, but a mount does not (see below). > > > > Please try the attached patch and let me know if it helps. When I looked > > I found several places where the rc_flag variable was being fiddled without the > > mutex held. I suspect one of these resulted in the RC_LOCKED flag not > > getting cleared, so all the threads got stuck waiting on it. > > > > The patch is at: > > http://people.freebsd.org/~rmacklem/cache.patch > > in case it gets eaten by the list handler. > > Thanks for digging into this, rick > > Hi Rick, > > Patch applied. The system has been up and running for about > 16 hours now and so far it's still handling the load quite nicely. > > last pid: 15853; load averages: 5.36, 4.64, 4.48 up 0+16:08:16 > 08:48:07 > 72 processes: 7 running, 65 sleeping > CPU: % user, % nice, % system, % interrupt, % idle > Mem: 22M Active, 3345M Inact, 79G Wired, 9837M Buf, 11G Free > Swap: > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 2049 root 26 52 0 10052K 1712K CPU3 3 97:21 942.24% nfsd > > I'll followup again in 24 hours with another status. > > Any performance related numbers/knobs we can provide that might > be of interest? > > Thanks Rick. > > -John Hi Rick, We've run the nfs share patchs and produced some numbers, no errors seen. A few questions about times. Four sets of repeated runs, each set run for about 2 days, with the following nfs options: tcp,nfsv3,rsize=32768,wsize=32768,noatime,nolockd,acregmin=1,acregmax=2,acdirmin=1,acdirmax=2,negnametimeo=0 tcp,nfsv3,rsize=32768,wsize=32768,noatime,nolockd,acregmin=1,acregmax=2,acdirmin=1,acdirmax=2,negnametimeo=2 udp,nfsv3,rsize=32768,wsize=32768,noatime,nolockd,acregmin=1,acregmax=2,acdirmin=1,acdirmax=2,negnametimeo=0 udp,nfsv3,rsize=32768,wsize=32768,noatime,nolockd,acregmin=1,acregmax=2,acdirmin=1,acdirmax=2,negnametimeo=2 negnametimeo=2 for /dat (src), 0 for /datgen (obj). Switched out tcp and udp, otherwise left options the same. stock = stock system nolock = share patches Run Median build time (min) Max build time (min) Tcp.nolock: 11.2 11.6 Tcp.stock: 13.6 20.8 (Some of these ran over cooling issues, ie: additional heat) Udp.nolock: 14.9 15.3 Udp.stock: 20.6 20.7 Average nfsd cpu usage: Tcp.nolock: 197.46 Tcp.stock: 164.872 Udp.nolock: 374.656 Udp.stock: 339.156 These cpu numbers seem a bit confusing. Udp seems to have more overhead. The share patches seem faster walkclock timewise, but use more cpu. Thoughts? Thanks, John From owner-freebsd-fs@FreeBSD.ORG Wed Jul 6 10:35:09 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2E0E41065672 for ; Wed, 6 Jul 2011 10:35:09 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id DB79A8FC0C for ; Wed, 6 Jul 2011 10:35:08 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1QePRi-0004jQ-Mp for freebsd-fs@freebsd.org; Wed, 06 Jul 2011 12:35:06 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 06 Jul 2011 12:35:06 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 06 Jul 2011 12:35:06 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Wed, 06 Jul 2011 12:34:53 +0200 Lines: 20 Message-ID: References: <1309217450.43651.YahooMailRC@web120014.mail.ne1.yahoo.com> <20110628010822.GA41399@icarus.home.lan> <1309302840.88674.YahooMailRC@web120004.mail.ne1.yahoo.com> <20110628234723.GA63965@icarus.home.lan> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.12) Gecko/20101102 Thunderbird/3.1.6 In-Reply-To: <20110628234723.GA63965@icarus.home.lan> X-Enigmail-Version: 1.1.2 Subject: Re: Improving old-fashioned UFS2 performance with lots of inodes... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2011 10:35:09 -0000 On 29/06/2011 01:47, Jeremy Chadwick wrote: >> unfortunately, so for now we will use UFS2, and as I said ... it seems a shame >> that UFS2 cannot use system RAM for any good purpose... >> >> Or can it ? Anyone ? > > Like I said: the only person (I know of) who could answer this would be > Kirk McKusick. I'm not well-versed in the inner workings and design of > filesystems; Kirk would be. I'm not sure who else "knows" UFS around > here. UFS will use all your memory for caching, there's no known issues here. Of course, you still need to read all this data in to be cached. As Jeremy said, even ZFS will not help you with huge file systems without some work. You could read this: http://en.wikipedia.org/wiki/Sharding and simply replace "databases" with "file systems" and "tables" with "directories" :) From owner-freebsd-fs@FreeBSD.ORG Wed Jul 6 14:32:20 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B636D106566B for ; Wed, 6 Jul 2011 14:32:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 5B8C18FC0C for ; Wed, 6 Jul 2011 14:32:19 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap4EAMtwFE6DaFvO/2dsb2JhbABTG4QnpDiIerITkROBK4N/gQwEkkGQWA X-IronPort-AV: E=Sophos;i="4.65,487,1304308800"; d="scan'208";a="126202526" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 06 Jul 2011 10:32:18 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id F2584B3F09; Wed, 6 Jul 2011 10:32:18 -0400 (EDT) Date: Wed, 6 Jul 2011 10:32:18 -0400 (EDT) From: Rick Macklem To: John Message-ID: <1060425320.262543.1309962738973.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <20110706011544.GA69706@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE7 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: Re: New NFS server stress test hang X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2011 14:32:20 -0000 John De wrote: ----- Original Message ----- > ----- John's Original Message ----- > > ----- Rick Macklem's Original Message ----- > > > John De wrote: > > > > ----- Rick Macklem's Original Message ----- > > > > > John De wrote: > > > > > > Hi, > > > > > > > > > > > > We've been running some stress tests of the new nfs server. > > > > > > The system is at r222531 (head), 9 clients, two mounts each > > > > > > to the server: > > > > > > > > > > > > mount_nfs -o > > > > > > udp,nfsv3,rsize=32768,wsize=32768,noatime,nolockd,acregmin=1,acregmax=2,acdirmin=1,acdirmax=2,negnametimeo=2 > > > > > > ${servera}:/vol/datsrc /c/$servera/vol/datsrc > > > > > > mount_nfs -o > > > > > > udp,nfsv3,rsize=32768,wsize=32768,noatime,nolockd,acregmin=1,acregmax=2,acdirmin=1,acdirmax=2,negnametimeo=0 > > > > > > ${servera}:/vol/datgen /c/$servera/vol/datgen > > > > > > > > > > > > > > > > > > The system is still up & responsive, simply no nfs services > > > > > > are working. All (200) threads appear to be active, but not > > > > > > doing anything. The debugger is not compiled into this > > > > > > kernel. > > > > > > We can run any other tracing commands desired. We can also > > > > > > rebuild the kernel with the debugger enabled for any kernel > > > > > > debugging needed. > > > > > > > > > > > > --- long logs deleted --- > > > > > > > > > > How about a: > > > > > ps axHlww <-- With the "H" we'll see what the nfsd server > > > > > threads > > > > > are up to > > > > > procstat -kka > > > > > > > > > > Oh, and a couple of nfsstats a few seconds apart. It's what > > > > > the > > > > > counts > > > > > are changing by that might tell us what is going on. (You can > > > > > use > > > > > "-z" > > > > > to zero them out, if you have an nfsstat built from recent > > > > > sources.) > > > > > > > > > > Also, does a new NFS mount attempt against the server do > > > > > anything? > > > > > > > > > > Thanks in advance for help with this, rick > > > > > > > > Hi Rick, > > > > > > > > Here's the output. In general, the nfsd processes appear to be > > > > in > > > > either nfsrvd_getcache(35 instances) or nfsrvd_updatecache(164) > > > > sleeping on > > > > "nfssrc". The server numbers don't appear to be moving. A > > > > showmount > > > > from a > > > > client system works, but a mount does not (see below). > > > > > > Please try the attached patch and let me know if it helps. When I > > > looked > > > I found several places where the rc_flag variable was being > > > fiddled without the > > > mutex held. I suspect one of these resulted in the RC_LOCKED flag > > > not > > > getting cleared, so all the threads got stuck waiting on it. > > > > > > The patch is at: > > > http://people.freebsd.org/~rmacklem/cache.patch > > > in case it gets eaten by the list handler. > > > Thanks for digging into this, rick > > > > Hi Rick, > > > > Patch applied. The system has been up and running for about > > 16 hours now and so far it's still handling the load quite nicely. > > > > last pid: 15853; load averages: 5.36, 4.64, 4.48 up 0+16:08:16 > > 08:48:07 > > 72 processes: 7 running, 65 sleeping > > CPU: % user, % nice, % system, % interrupt, % idle > > Mem: 22M Active, 3345M Inact, 79G Wired, 9837M Buf, 11G Free > > Swap: > > > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > > 2049 root 26 52 0 10052K 1712K CPU3 3 97:21 942.24% nfsd > > > > I'll followup again in 24 hours with another status. > > > > Any performance related numbers/knobs we can provide that might > > be of interest? > > > > Thanks Rick. > > > > -John > > Hi Rick, > > We've run the nfs share patchs and produced some numbers, no errors > seen. A few questions about times. > > Four sets of repeated runs, each set run for about 2 days, with the > following nfs options: > > tcp,nfsv3,rsize=32768,wsize=32768,noatime,nolockd,acregmin=1,acregmax=2,acdirmin=1,acdirmax=2,negnametimeo=0 > tcp,nfsv3,rsize=32768,wsize=32768,noatime,nolockd,acregmin=1,acregmax=2,acdirmin=1,acdirmax=2,negnametimeo=2 > udp,nfsv3,rsize=32768,wsize=32768,noatime,nolockd,acregmin=1,acregmax=2,acdirmin=1,acdirmax=2,negnametimeo=0 > udp,nfsv3,rsize=32768,wsize=32768,noatime,nolockd,acregmin=1,acregmax=2,acdirmin=1,acdirmax=2,negnametimeo=2 > > negnametimeo=2 for /dat (src), 0 for /datgen (obj). Switched out tcp > and udp, otherwise left options the same. > > stock = stock system > nolock = share patches > I assume this means without/with the patch I sent you. If not, the following will be completely bogus. > Run Median build time (min) Max build time (min) > Tcp.nolock: 11.2 11.6 > Tcp.stock: 13.6 20.8 (Some of these ran over cooling issues, ie: > additional heat) > Udp.nolock: 14.9 15.3 > Udp.stock: 20.6 20.7 > > Average nfsd cpu usage: > > Tcp.nolock: 197.46 > Tcp.stock: 164.872 > Udp.nolock: 374.656 > Udp.stock: 339.156 > > These cpu numbers seem a bit confusing. Udp seems to have more > overhead. The share patches > seem faster walkclock timewise, but use more cpu. > Ok, well without the patch, you were simply getting incorrect/buggy behaviour, so I don't think those stats mean much. (I won't try and guess what the buggy behaviour really was. The fun part of any caching mechanism is that most bugs just affect perf.:-) W.r.t. UDP vs TCP... for TCP, it is only necessary to keep the last few (often only the last one) request(s) on the TCP connection, since the only client side RPC retry will occur much later, for the case that a TCP connection is broken by a long network partitioning or similar. For UDP, it must keep all the replies for at least 1sec after being sent, since UDP can drop requests/replies at any time. This is going to be more work and use more mbufs etc. > Thoughts? > One final comment. For TCP, larger rsize/wsize than 32768 should work. If you don't specify them as mount arguments, it will use the largest supported by the client and server. Currently this is MAX_BSIZE == 64K for FreeBSD, but I hope to try cranking that up soon. For Solaris10, it is 1Mbyte. The only situation I am aware of where larger rsize/wsize will result in poorer perf. is when the network fabric can't handle the larger bursts of data traffic. (Since you are running 32K UDP, I don't think that will be an issue for you. If it happens, it is usually obvious, in that perf. drop by an order of magnitude. --> reads or writes take so long you think the server has crashed.) rick From owner-freebsd-fs@FreeBSD.ORG Thu Jul 7 01:01:21 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 98F5A106566C for ; Thu, 7 Jul 2011 01:01:21 +0000 (UTC) (envelope-from gosand1982@yahoo.com) Received: from nm19-vm4.bullet.mail.ne1.yahoo.com (nm19-vm4.bullet.mail.ne1.yahoo.com [98.138.91.179]) by mx1.freebsd.org (Postfix) with SMTP id 4A70C8FC0C for ; Thu, 7 Jul 2011 01:01:20 +0000 (UTC) Received: from [98.138.90.52] by nm19.bullet.mail.ne1.yahoo.com with NNFMP; 07 Jul 2011 01:01:20 -0000 Received: from [98.138.89.245] by tm5.bullet.mail.ne1.yahoo.com with NNFMP; 07 Jul 2011 01:01:20 -0000 Received: from [127.0.0.1] by omp1059.mail.ne1.yahoo.com with NNFMP; 07 Jul 2011 01:01:20 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 580923.15698.bm@omp1059.mail.ne1.yahoo.com Received: (qmail 49283 invoked by uid 60001); 7 Jul 2011 01:01:20 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1310000480; bh=8SesShJKZ2wjxbOLXY94eQ5L+YlGDgpKnMPPQPHyZrI=; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:MIME-Version:Content-Type; b=vYHKJbBirGeF3eVfP+1eqKOmVMK0iNpLEcI6+Aq2OKSGOUVewhaVFobCedbAGTtpdLi8yh1mgCehhJmDIob/Af1ICaXP21wGB8OvmlF1Kk2I16BxULjxDqE+4bbGnXiozfgqPf5/RvEWvHRDan95IpnpsQ1FwqZTSMpeXE2eRgw= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:MIME-Version:Content-Type; b=E4cZdjZT3DNnCN5htd4YAh780BdWTFLoouA93zZbF6lxtDpK9DWUDXLdHp9ZJQ8Y/Gx0X7Z01n8x/qpnmsVwA/lZQP4sHKTyYJvNN/ebd4ZzB1oEh4mDYXoOlPlJNNvU3BqeChbPoTqMl5vtv49hCwHlYSJhJPFW7MOAqo+0CC8=; X-YMail-OSG: gr.61hIVM1lsj0D9uapiNmm0.N4XoYB2eeDpZcJF6XE3o5C 2NNVVzjgW5iWUiJFCrRhs3oDQ78xzpy5cYN.5aiwy2NY8hXOc_TGQH3W.P75 C7ngoTqTWz3cwkZh4XJrrVhUw47j4G9vZFlVRhDFN7Jv_EsXtZ.ReT5HB7P7 4GlamR79rFT51I2pCsITQlIIvXx4fDLRysf_UaODAkiCMV3Ym33MR6LghaHr 4EYCSKFjrWCIKnK0f5aXHb91pLLZjMFCaBaGhfd8ozGZK8AC3MsfZVJ8mSmx gxXIqUYcdHddwBFDFC7LX6gB9NwomtBJZ0Jn5XIhDJL2iTdU4huM- Received: from [75.210.254.42] by web120004.mail.ne1.yahoo.com via HTTP; Wed, 06 Jul 2011 18:01:20 PDT X-Mailer: YahooMailRC/572 YahooMailWebService/0.8.112.307740 Message-ID: <1310000480.20036.YahooMailRC@web120004.mail.ne1.yahoo.com> Date: Wed, 6 Jul 2011 18:01:20 -0700 (PDT) From: George Sanders To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: RE: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2011 01:01:21 -0000 Gerrit Khun wrote: I am still looking for a viable replacement of LSI1068e-based cards for home-built ZFS servers. This LSI chipset does not support drives larger than 2TB which has become a serious limitation for me. I read that the LSI2008 is supposed to be some kind of follow-up, and it should be supported with recent FreeBSD's mps driver. However, I could not find any cheap (non-hw-raid) cards with 8 channels (or more) like there are with the old LSI1068 chipset. Are there any recommendations? Is this indeed the case ? The six controllers that are built into a "thumper" (a x4500) cannot see space beyond 2TB on a single disk ? This is very good information - thank you - as I was considering using an old x4500 with 3TB disk drives... Is there really no firmware update - from either LSI or Sun - that fixes this ? From owner-freebsd-fs@FreeBSD.ORG Thu Jul 7 07:57:15 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1ADAF106566B for ; Thu, 7 Jul 2011 07:57:15 +0000 (UTC) (envelope-from gerrit@pmp.uni-hannover.de) Received: from mrelay1.uni-hannover.de (mrelay1.uni-hannover.de [130.75.2.106]) by mx1.freebsd.org (Postfix) with ESMTP id A2E8C8FC0A for ; Thu, 7 Jul 2011 07:57:14 +0000 (UTC) Received: from www.pmp.uni-hannover.de (www.pmp.uni-hannover.de [130.75.117.2]) by mrelay1.uni-hannover.de (8.14.4/8.14.4) with ESMTP id p677vBFF017262; Thu, 7 Jul 2011 09:57:12 +0200 Received: from pmp.uni-hannover.de (unknown [130.75.117.3]) by www.pmp.uni-hannover.de (Postfix) with SMTP id 6A54710A; Thu, 7 Jul 2011 09:57:11 +0200 (CEST) Date: Thu, 7 Jul 2011 09:57:11 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: George Sanders Message-Id: <20110707095711.dfa208b5.gerrit@pmp.uni-hannover.de> In-Reply-To: <1310000480.20036.YahooMailRC@web120004.mail.ne1.yahoo.com> References: <1310000480.20036.YahooMailRC@web120004.mail.ne1.yahoo.com> Organization: Albert-Einstein-Institut (MPI =?ISO-8859-1?Q?f=FCr?= Gravitationsphysik & IGP =?ISO-8859-1?Q?Universit=E4t?= Hannover) X-Mailer: Sylpheed 3.0.3 (GTK+ 2.22.1; amd64-portbld-freebsd8.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-PMX-Version: 5.5.9.395186, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2011.7.7.74514 Cc: freebsd-fs@freebsd.org Subject: Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gerrit.kuehn@aei.mpg.de List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2011 07:57:15 -0000 On Wed, 6 Jul 2011 18:01:20 -0700 (PDT) George Sanders wrote about RE: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...): GS> Is this indeed the case ? The six controllers that are built into a GS> "thumper" (a x4500) cannot see space beyond 2TB on a single disk ? I don't know what is built into Thumpers. If it is based on LSI1068 chipsets, 2TB is the limit. At least that is what I was told by several support people and also found out myself by doing some Google and by trying out several firmware versions available. It is a hardware limitation of the chipset that cannot be circumvented by a firmware update. However, one of the supportes I talked to claimed that the limitation only existed for SATA drives and SAS would be working beyond 2TB. I cannot say if this is really true, because I did not get any technical explanation why this should be different with SAS drives, and I do not have any SAS drives here to test. GS> This is very good information - thank you - as I was considering using GS> an old x4500 with 3TB disk drives... So just be careful and try it with one drive first before buying some dozen of them... :-) GS> Is there really no firmware update - from either LSI or Sun - that GS> fixes this ? I was told it is a hardware limitation and thus cannot be fixed. cu Gerrit From owner-freebsd-fs@FreeBSD.ORG Thu Jul 7 14:22:11 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D154B106564A for ; Thu, 7 Jul 2011 14:22:11 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 931D18FC16 for ; Thu, 7 Jul 2011 14:22:11 +0000 (UTC) Received: by gwb15 with SMTP id 15so493818gwb.13 for ; Thu, 07 Jul 2011 07:22:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=TaBCt269EvUiTvszG8ienpPLRuOlKjuKq7ubsXlWg48=; b=OSm4Z2hypR36ORuq/TXkCEbo1xz8bE/L260Oauz960XBpj9FIKziyNTppXBWSRiWtZ sKidoka894+hpoo7tMpqFEy2O7H/TX9Ipf7bEcsBb/HcwUcMbV7jKCUwm18vf1WE647G D5WGNY7EMGeX9ldMY4gQ4kkJu0zXK8LSkD+7s= MIME-Version: 1.0 Received: by 10.91.160.18 with SMTP id m18mr975439ago.124.1310048530507; Thu, 07 Jul 2011 07:22:10 -0700 (PDT) Received: by 10.90.15.24 with HTTP; Thu, 7 Jul 2011 07:22:10 -0700 (PDT) In-Reply-To: <1310000480.20036.YahooMailRC@web120004.mail.ne1.yahoo.com> References: <1310000480.20036.YahooMailRC@web120004.mail.ne1.yahoo.com> Date: Thu, 7 Jul 2011 07:22:10 -0700 Message-ID: From: Freddie Cash To: George Sanders Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: Are thumpers still interesting in 2011 ? (raidz3 on x4500 @ 3.0gbps ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2011 14:22:11 -0000 On Wed, Jul 6, 2011 at 6:01 PM, George Sanders wrote: > > I am still looking for a viable replacement of LSI1068e-based cards for > home-built ZFS servers. This LSI chipset does not support drives larger > than 2TB > > which has become a serious limitation for me. I read that the LSI2008 is > supposed to be some kind of follow-up, and it should be supported with > recent > FreeBSD's mps driver. However, I could not find any cheap (non-hw-raid) > cards > with 8 channels (or more) like there are with the old LSI1068 chipset. Are > there > > any recommendations? > SuperMicro AOC-USAS2-L8i is an 8-port muiltilane controller using the LSI2008 chipset. It's the upgrade to the venerable AOC-USAS-L8i based on the LSI1068 chipset. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Thu Jul 7 14:25:32 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 96F9C106566B for ; Thu, 7 Jul 2011 14:25:32 +0000 (UTC) (envelope-from rickvanderzwet@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5AF5D8FC0C for ; Thu, 7 Jul 2011 14:25:32 +0000 (UTC) Received: by yxl31 with SMTP id 31so499179yxl.13 for ; Thu, 07 Jul 2011 07:25:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:from:date:x-google-sender-auth:message-id :subject:to:content-type; bh=DQR8yORF8jrUyXsl1glNDShpoa/+bJzGk8UUXsxdWmQ=; b=q2sHD4eaueKJHQ8nLDA8VLur1YL0iPoiBRvZrSEmlFSSQq7X+WmgU+geLULTBc8NcE OHgO9lOqoZDTUlBnWbf5e0J65dDOVp/MNbzJufiitqDqMNLGw8hhpX+lT2fcTYM3Fwf6 3kb7spJN/ZSzp3R1maugl10SSq5DBJV7nWc9Q= Received: by 10.236.190.73 with SMTP id d49mr1012417yhn.517.1310047092067; Thu, 07 Jul 2011 06:58:12 -0700 (PDT) MIME-Version: 1.0 Sender: rickvanderzwet@gmail.com Received: by 10.236.161.106 with HTTP; Thu, 7 Jul 2011 06:57:52 -0700 (PDT) From: Rick van der Zwet Date: Thu, 7 Jul 2011 15:57:52 +0200 X-Google-Sender-Auth: ts002m5kDUE1-7PCRnjy1r6VimU Message-ID: To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: fsck_ufs a 2TB partition with 256MB RAM stalls X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2011 14:25:32 -0000 I want to build a file server with limited power usage, so I got myself an ALIX alix2d13 which has 256MB DDR RAM. I connected a 2TB USB2.0 disk to the alix2d13 to be used as storage. The file system gets corrupted due to power failure, which is likely going to happen when running Solar Power in The Netherlands, I cannot fix it anymore cause the fsck_ufs never to complete. This actually makes sense as the recommendation [1] says ``1TB storage needs 1GB of RAM for fsck_ufs''. I can partition my disk in 16 parts of 128GB to work around the matter, but I rather keep one large partition. Any recommendations of getting this solved or will the smaller partitions be my only friend? Br. /Rick [1] http://groups.google.com/group/lucky.freebsd.stable/msg/4f1dcc954ecb7392 -- http://rickvanderzwet.nl From owner-freebsd-fs@FreeBSD.ORG Thu Jul 7 21:22:23 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 962C5106566B for ; Thu, 7 Jul 2011 21:22:23 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 28F998FC0A for ; Thu, 7 Jul 2011 21:22:22 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1Qew1X-0008GX-1u for freebsd-fs@freebsd.org; Thu, 07 Jul 2011 23:22:15 +0200 Received: from 195.225.157.86 ([195.225.157.86]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 07 Jul 2011 23:22:15 +0200 Received: from c.kworr by 195.225.157.86 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 07 Jul 2011 23:22:15 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Volodymyr Kostyrko Date: Fri, 08 Jul 2011 00:22:01 +0300 Lines: 95 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: 195.225.157.86 User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:5.0) Gecko/20110702 Thunderbird/5.0 Subject: everlasting log device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2011 21:22:23 -0000 Hi all. When I get my hands on SSD device I tried to setup a log/cache partition for my pools. Everything works fine until one day I realized that I have a better place to stick this SSD in. I have upgraded system from RELENG_8_2 to RELENG_8 and tried to remove devices. From my two pools one was successfully freed from log/cache devices yet another one refuses to live without log device: # zpool upgrade This system is currently running ZFS pool version 28. All pools are formatted using this version. # zfs upgrade This system is currently running ZFS filesystem version 5. All filesystems are formatted with the current version. # zpool status pool: utwig state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-2Q scan: resilvered 0 in 0h21m with 0 errors on Sat Jul 2 15:07:35 2011 config: NAME STATE READ WRITE CKSUM utwig DEGRADED 0 0 0 mirror-0 ONLINE 0 0 0 gptid/ecb17af1-9119-11df-bb0b-00304f4e6d80 ONLINE 0 0 0 gptid/03aed1f5-95a3-11df-bb0b-00304f4e6d80 ONLINE 0 0 0 logs gptid/231b9002-a4a5-11e0-a114-3f386a87752c UNAVAIL 0 0 0 cannot open errors: No known data errors pool: utwig-sas state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM utwig-sas ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 aacd1 ONLINE 0 0 0 aacd2 ONLINE 0 0 0 errors: No known data errors # zpool remove utwig gptid/231b9002-a4a5-11e0-a114-3f386a87752c && echo good good And nothing changes - system needs that partition. One more weird thing. # zpool iostat -v utwig capacity operations bandwidth pool alloc free read write read write -------------------------------------- ----- ----- ----- ----- ----- ----- utwig 284G 172G 41 70 272K 793K mirror 284G 172G 41 70 272K 793K gptid/ecb17af1-9119-11df-bb0b-00304f4e6d80 - - 8 27 456K 794K gptid/03aed1f5-95a3-11df-bb0b-00304f4e6d80 - - 8 27 459K 794K gptid/231b9002-a4a5-11e0-a114-3f386a87752c 148K 3,97G 0 0 0 0 -------------------------------------- ----- ----- ----- ----- ----- ----- System claims that this log device has 148K data. Is this the size of unwritten data? The number is still the same when booting into single user mode and doesn't change at all. Can I remove this log device? Should I recreate the pool to get rid of this behavior? -- Sphinx of black quartz judge my vow. From owner-freebsd-fs@FreeBSD.ORG Thu Jul 7 22:00:26 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2365F1065672 for ; Thu, 7 Jul 2011 22:00:26 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id DD93E8FC0A for ; Thu, 7 Jul 2011 22:00:25 +0000 (UTC) Received: by iyb11 with SMTP id 11so1670274iyb.13 for ; Thu, 07 Jul 2011 15:00:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=Bob69Rm+AwSioqky1GtSzpWikoXpDcoJ4aIwFWMToEA=; b=YQ2OOtUuqfzW5bjSU56eX5OlNIYqPdyPk4gOV3dfDoL0VwUP3qWW/WoVxetQBv72Rf cANZn5zkAwgFTS53vXqsGqEiw9NkMaFCiGUtksHmwkiAVLfjxPtpZ56okjKEZuxqdz1Q HZaCIgm4JvdHx5NM553cn6zyW2025juA1l6P8= Received: by 10.42.131.71 with SMTP id y7mr1337343ics.315.1310076025222; Thu, 07 Jul 2011 15:00:25 -0700 (PDT) Received: from DataIX.net ([99.181.128.71]) by mx.google.com with ESMTPS id v16sm5667408ibe.17.2011.07.07.15.00.23 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 07 Jul 2011 15:00:23 -0700 (PDT) Sender: "J. Hellenthal" Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id p67M0Kop081788 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 7 Jul 2011 18:00:20 -0400 (EDT) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id p67M0JcF081783; Thu, 7 Jul 2011 18:00:20 -0400 (EDT) (envelope-from jhell@DataIX.net) Date: Thu, 7 Jul 2011 18:00:19 -0400 From: Jason Hellenthal To: Volodymyr Kostyrko Message-ID: <20110707220019.GA79464@DataIX.net> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Cc: freebsd-fs@freebsd.org Subject: Re: everlasting log device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2011 22:00:26 -0000 On Fri, Jul 08, 2011 at 12:22:01AM +0300, Volodymyr Kostyrko wrote: > Hi all. > > When I get my hands on SSD device I tried to setup a log/cache partition > for my pools. Everything works fine until one day I realized that I have > a better place to stick this SSD in. I have upgraded system from > RELENG_8_2 to RELENG_8 and tried to remove devices. From my two pools > one was successfully freed from log/cache devices yet another one > refuses to live without log device: > > # zpool upgrade > This system is currently running ZFS pool version 28. > > All pools are formatted using this version. > > # zfs upgrade > This system is currently running ZFS filesystem version 5. > > All filesystems are formatted with the current version. > > # zpool status > pool: utwig > state: DEGRADED > status: One or more devices could not be opened. Sufficient replicas > exist for > the pool to continue functioning in a degraded state. > action: Attach the missing device and online it using 'zpool online'. > see: http://www.sun.com/msg/ZFS-8000-2Q > scan: resilvered 0 in 0h21m with 0 errors on Sat Jul 2 15:07:35 2011 > config: > > NAME STATE READ > WRITE CKSUM > utwig DEGRADED 0 > 0 0 > mirror-0 ONLINE 0 > 0 0 > gptid/ecb17af1-9119-11df-bb0b-00304f4e6d80 ONLINE 0 > 0 0 > gptid/03aed1f5-95a3-11df-bb0b-00304f4e6d80 ONLINE 0 > 0 0 > logs > gptid/231b9002-a4a5-11e0-a114-3f386a87752c UNAVAIL 0 > 0 0 cannot open > > errors: No known data errors > > pool: utwig-sas > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > utwig-sas ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > aacd1 ONLINE 0 0 0 > aacd2 ONLINE 0 0 0 > > errors: No known data errors > > # zpool remove utwig gptid/231b9002-a4a5-11e0-a114-3f386a87752c && echo good > good > > And nothing changes - system needs that partition. > > One more weird thing. > > # zpool iostat -v utwig > capacity operations > bandwidth > pool alloc free read write > read write > -------------------------------------- ----- ----- ----- ----- > ----- ----- > utwig 284G 172G 41 70 > 272K 793K > mirror 284G 172G 41 70 > 272K 793K > gptid/ecb17af1-9119-11df-bb0b-00304f4e6d80 - - 8 > 27 456K 794K > gptid/03aed1f5-95a3-11df-bb0b-00304f4e6d80 - - 8 > 27 459K 794K > gptid/231b9002-a4a5-11e0-a114-3f386a87752c 148K 3,97G 0 > 0 0 0 > -------------------------------------- ----- ----- ----- ----- > ----- ----- > > System claims that this log device has 148K data. Is this the size of > unwritten data? The number is still the same when booting into single > user mode and doesn't change at all. > > Can I remove this log device? Should I recreate the pool to get rid of > this behavior? > If you have the possibility to re-create the pool then Id definately suggest it. If you remove this device (physically) your pool will not be operable unfortunately there is still somehting missing to allow SLOGs to be removed from a running pool yet, what that might be is beyond me at this time. You might try to export the pool then boot into single user mode and reimport the pool and try the removal procedure but I raelly dont think that will help you. Good luck. From owner-freebsd-fs@FreeBSD.ORG Thu Jul 7 22:44:28 2011 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 86E0F106564A for ; Thu, 7 Jul 2011 22:44:28 +0000 (UTC) (envelope-from mailreturn@smtp.ymlp15.net) Received: from smtp.ymlp15.net (smtp.ymlp15.net [87.237.8.246]) by mx1.freebsd.org (Postfix) with SMTP id 9D1AD8FC16 for ; Thu, 7 Jul 2011 22:44:27 +0000 (UTC) Received: (qmail 6894 invoked by uid 0); 7 Jul 2011 22:44:24 -0000 Date: Fri, 8 Jul 2011 00:44:24 +0200 To: fs@freebsd.org From: Improved Attitude To Work Message-ID: X-YMLPcode: p82a+1598+322236 MIME-Version: 1.0 Content-Type: text/plain; charset = "utf-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Management Training on "Attitudinal Re-Orientation" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: invitation@thewodia.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2011 22:44:28 -0000 --------------------------------------------------------------------------= ------ This email newsletter was sent to you in graphical HTML format. If you're seeing this version, your email program prefers plain text = emails. You can read the original version online: http://ymlp95.net/zJfGCc --------------------------------------------------------------------------= ------ WODIA Training Institute (WOTI) Management Training andSkills Development The Boat House, 21, Ogunnusi Road, Ogba, Ikeja, Lagos.www.thewodia.org/woti.html ( http://www.thewodia.org/woti.html ), Email : support@thewodia.org,invitation@thewodia.org Telephone : +234 802 307 9485, +234 813 375 4358 INVITATION TO MANAGEMENT TRAINING PROGRAM Investment in People- TOPIC:Attitudinal Re-orientation And Best Practices In Work Environment . Date : JULY 19-20, 2011 (2 days) Venue : WODIA Training Institute (WOTI) The Boat House, 21, Ogunnusi Road, (AVIS Petrol Station Bus-Stop), Ogba, Ikeja, Lagos. Nigeria. Fee : N20,000:00 (Twenty thousand Naira only)per Head. Fees include Tuition, Resource Materials, Tea/Coffee Break,Lunch, Certificate of Participation and Group Photograph. Time : 9.00a.m daily Participation:Confirm Space Availabilitybefore payment by calling +234 802 307 9485 OR +234 813 375 4358. You could also send an Email to:support@thewodia.org, invitation@thewodia.org ------------------------- Payment Instructions : After confirmation of space availability, payment should be made into any Branch of Zenith Bank PLC. account number 6011822855 in favour of WODIA Training Institute (WOTI) before program date. TRAINING DETAILS: Objectives: The Two-day Intensive Management Training Program on Attitudinal re-orientation and Best Practices in Work Environment is intended to : - -encouragepositive attitude to work and discourage non-challant attitude towards organizational goals; - educate the participants on the benefits of loyalty to their organizations; -promote learning skills of the attendees and update their ethical standards at work place; -enhance human capital output; -boost personal development and improve workers' enthusiasm towards their official responsibilities; -discourage conflict of interest that diverts worker's concentration from official work; and - distinguish official behaviour from informal attitude that diminishes productivity. COURSE OUTLINES : * Developing Positive Attitude To Work * Meeting Organization=E2=80=99s Expectations and Targets * Work Orientation and Employment Contract * Conflict of Interest and Ethical issues at Work Place * Work Ethics, Mental Alertness and Commitment to organizational Goals. WHO SHOULD ATTEND: Attendance open to both Male and Female who are Middle-Level Officers in Public, Private, Corporate and Voluntary Establishments. Accreditation : WOTI is Certified by the Centre for Management Development (CMD), Nigeria as an Accredited Management Training Institution. Recource Persons will be WOTI Training Consultants and others to be invited from the Industrial Training Fund (ITF), Centre for Management Development (CMD) and other National and International Consulting Agencies. Training Aids will include but not limited to projectors and screen, flip charts, magic boards, computers and other multi-media and audio-visual aids. Methodology will involve lecture, role play, demonstration, discussion and other methods suitable for the target audience. Program will be highly inclusive and participatory. Thanks for Reading, ------------------------- Gladys Osagie, Program Coordinator, WODIA Training Institute (WOTI), www.thewodia.org/woti.html, EMAIL : support@thewodia.org +234 802 307 9485 Forward this email to a friend ( = http://ymlp95.net/forward.php?id=3DJfGCc&e=3Dfs@freebsd.org ) _____________________________ Unsubscribe / Change Profile: = http://ymlp95.net/u.php?id=3Dgewmweugsgmeeemhgubyq Powered by YourMailingListProvider From owner-freebsd-fs@FreeBSD.ORG Thu Jul 7 23:39:23 2011 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 507631065670; Thu, 7 Jul 2011 23:39:23 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 28A2A8FC12; Thu, 7 Jul 2011 23:39:23 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p67NdNIu022504; Thu, 7 Jul 2011 23:39:23 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p67NdNoX022500; Thu, 7 Jul 2011 23:39:23 GMT (envelope-from linimon) Date: Thu, 7 Jul 2011 23:39:23 GMT Message-Id: <201107072339.p67NdNoX022500@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/158711: [ffs] [panic] panic in ffs_blkfree and ffs_valloc X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2011 23:39:23 -0000 Old Synopsis: panic in ffs_blkfree and ffs_valloc New Synopsis: [ffs] [panic] panic in ffs_blkfree and ffs_valloc Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Thu Jul 7 23:38:28 UTC 2011 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=158711 From owner-freebsd-fs@FreeBSD.ORG Fri Jul 8 04:43:00 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E6052106566B for ; Fri, 8 Jul 2011 04:43:00 +0000 (UTC) (envelope-from jwd@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D2F9C8FC0C; Fri, 8 Jul 2011 04:43:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p684h0Br010281; Fri, 8 Jul 2011 04:43:00 GMT (envelope-from jwd@freefall.freebsd.org) Received: (from jwd@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p684h0rJ010280; Fri, 8 Jul 2011 04:43:00 GMT (envelope-from jwd) Date: Fri, 8 Jul 2011 04:43:00 +0000 From: John To: freebsd-fs@freebsd.org Message-ID: <20110708044300.GA2130@FreeBSD.org> References: <20110707220019.GA79464@DataIX.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110707220019.GA79464@DataIX.net> User-Agent: Mutt/1.4.2.3i Cc: Jason Hellenthal , Volodymyr Kostyrko Subject: Re: everlasting log device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2011 04:43:01 -0000 ----- Jason Hellenthal's Original Message ----- > > > On Fri, Jul 08, 2011 at 12:22:01AM +0300, Volodymyr Kostyrko wrote: > > Hi all. > > > > When I get my hands on SSD device I tried to setup a log/cache partition > > for my pools. Everything works fine until one day I realized that I have > > a better place to stick this SSD in. I have upgraded system from > > RELENG_8_2 to RELENG_8 and tried to remove devices. From my two pools > > one was successfully freed from log/cache devices yet another one > > refuses to live without log device: > > > > # zpool upgrade > > This system is currently running ZFS pool version 28. > > > > All pools are formatted using this version. > > > > # zfs upgrade > > This system is currently running ZFS filesystem version 5. > > > > All filesystems are formatted with the current version. > > > > # zpool status > > pool: utwig > > state: DEGRADED > > status: One or more devices could not be opened. Sufficient replicas > > exist for > > the pool to continue functioning in a degraded state. > > action: Attach the missing device and online it using 'zpool online'. > > see: http://www.sun.com/msg/ZFS-8000-2Q > > scan: resilvered 0 in 0h21m with 0 errors on Sat Jul 2 15:07:35 2011 > > config: > > > > NAME STATE READ > > WRITE CKSUM > > utwig DEGRADED 0 > > 0 0 > > mirror-0 ONLINE 0 > > 0 0 > > gptid/ecb17af1-9119-11df-bb0b-00304f4e6d80 ONLINE 0 > > 0 0 > > gptid/03aed1f5-95a3-11df-bb0b-00304f4e6d80 ONLINE 0 > > 0 0 > > logs > > gptid/231b9002-a4a5-11e0-a114-3f386a87752c UNAVAIL 0 > > 0 0 cannot open > > > > errors: No known data errors > > > > pool: utwig-sas > > state: ONLINE > > scan: none requested > > config: > > > > NAME STATE READ WRITE CKSUM > > utwig-sas ONLINE 0 0 0 > > mirror-0 ONLINE 0 0 0 > > aacd1 ONLINE 0 0 0 > > aacd2 ONLINE 0 0 0 > > > > errors: No known data errors > > > > # zpool remove utwig gptid/231b9002-a4a5-11e0-a114-3f386a87752c && echo good > > good > > > > And nothing changes - system needs that partition. > > > > One more weird thing. > > > > # zpool iostat -v utwig > > capacity operations > > bandwidth > > pool alloc free read write > > read write > > -------------------------------------- ----- ----- ----- ----- > > ----- ----- > > utwig 284G 172G 41 70 > > 272K 793K > > mirror 284G 172G 41 70 > > 272K 793K > > gptid/ecb17af1-9119-11df-bb0b-00304f4e6d80 - - 8 > > 27 456K 794K > > gptid/03aed1f5-95a3-11df-bb0b-00304f4e6d80 - - 8 > > 27 459K 794K > > gptid/231b9002-a4a5-11e0-a114-3f386a87752c 148K 3,97G 0 > > 0 0 0 > > -------------------------------------- ----- ----- ----- ----- > > ----- ----- > > > > System claims that this log device has 148K data. Is this the size of > > unwritten data? The number is still the same when booting into single > > user mode and doesn't change at all. > > > > Can I remove this log device? Should I recreate the pool to get rid of > > this behavior? > > > > If you have the possibility to re-create the pool then Id definately > suggest it. > > If you remove this device (physically) your pool will not be operable > unfortunately there is still somehting missing to allow SLOGs to be > removed from a running pool yet, what that might be is beyond me at this > time. You might try to export the pool then boot into single user mode > and reimport the pool and try the removal procedure but I raelly dont > think that will help you. > > Good luck. I have the same issue. Easy to ignore most of the time. Really annoying at others. Haven't figured out a way to fix/avoid it yet. This is running a current systems just a few days old. It's been around for awhile though. # zpool iostat -v capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- pool1 4.95G 971G 1 0 4.40K 803 raidz1 4.95G 971G 1 0 4.40K 803 da0 - - 0 0 3.19K 287 da1 - - 0 0 3.17K 287 da2 - - 0 0 3.20K 279 da3 - - 0 0 3.17K 279 da4 - - 0 0 3.20K 287 da5 - - 0 0 3.20K 299 da6 - - 0 0 3.17K 289 hast/md0 0 250M 0 0 0 0 hast/md1 4K 250M 0 0 0 0 -John From owner-freebsd-fs@FreeBSD.ORG Fri Jul 8 05:09:32 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9EAF21065670 for ; Fri, 8 Jul 2011 05:09:32 +0000 (UTC) (envelope-from c.kworr@gmail.com) Received: from mail-fx0-f44.google.com (mail-fx0-f44.google.com [209.85.161.44]) by mx1.freebsd.org (Postfix) with ESMTP id 2E2AF8FC08 for ; Fri, 8 Jul 2011 05:09:31 +0000 (UTC) Received: by fxe6 with SMTP id 6so1471458fxe.17 for ; Thu, 07 Jul 2011 22:09:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=ZBR+OtBO5bCdCgyrvnIHw+QoYQ+hOygbtCEMRyRxRik=; b=it5SoH5HP9BBvNKkErWaoo9WRBenr22LwEorQRivktN+Hi7WnyyX8gbY8h+MjEGjEC mH7YyKTWZKMkuTxtQLVOKXwZoVO1as3oaz2aOJ2e3U0cKXz/wh9LGM0o/jyfFdlD7+VI iEcr+Xfwl3vrHXrGlKjhNfTkmanMz16QrZff4= Received: by 10.223.145.22 with SMTP id b22mr2539290fav.95.1310101771027; Thu, 07 Jul 2011 22:09:31 -0700 (PDT) Received: from limbo.lan ([195.225.157.86]) by mx.google.com with ESMTPS id 15sm146863fau.0.2011.07.07.22.09.28 (version=SSLv3 cipher=OTHER); Thu, 07 Jul 2011 22:09:29 -0700 (PDT) Message-ID: <4E169107.6040109@gmail.com> Date: Fri, 08 Jul 2011 08:09:27 +0300 From: Volodymyr Kostyrko User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:5.0) Gecko/20110702 Thunderbird/5.0 MIME-Version: 1.0 To: Jason Hellenthal References: <20110707220019.GA79464@DataIX.net> In-Reply-To: <20110707220019.GA79464@DataIX.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: everlasting log device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2011 05:09:32 -0000 08.07.2011 01:00, Jason Hellenthal wrote: >> System claims that this log device has 148K data. Is this the size of >> unwritten data? The number is still the same when booting into single >> user mode and doesn't change at all. >> >> Can I remove this log device? Should I recreate the pool to get rid of >> this behavior? >> > > If you have the possibility to re-create the pool then Id definately > suggest it. > > If you remove this device (physically) your pool will not be operable > unfortunately there is still somehting missing to allow SLOGs to be > removed from a running pool yet, what that might be is beyond me at this > time. You might try to export the pool then boot into single user mode > and reimport the pool and try the removal procedure but I raelly dont > think that will help you. Well I have already removed device physically and the pool continues to work despite yelling about missing log device. I already tried booting from flash drive and importing/exporting pool - this doesn't work. I also tried to replace this log device with another one. Replacing works and I can remove old device but the new one will now be unkillable. Am I right that working without external log device is almost the same as working with disabled ZIL? -- Sphinx of black quartz judge my vow. From owner-freebsd-fs@FreeBSD.ORG Fri Jul 8 06:52:54 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D9C5106566B for ; Fri, 8 Jul 2011 06:52:54 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 940388FC15 for ; Fri, 8 Jul 2011 06:52:53 +0000 (UTC) Received: by wwe6 with SMTP id 6so1578552wwe.31 for ; Thu, 07 Jul 2011 23:52:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=kXlvuLR5FLXjTPqmDAlfBrDkmtg0ILTmmqm782EFNzc=; b=W1NIWuicsBGWzAX6tMyEQamG1hIRECOMvgld4qyTVJ18t23iRo+kKhcObrJ9ds4MLm IMRyYNfl0S5jucKVaWMA6pAGz/GTrFoehTep1LKPAdv8f4TLLLkAVsd9vX/pTPnaSm32 BkBKvOU+sLosz4ZKJ/EC/hADyicTnlDG8yfo0= MIME-Version: 1.0 Received: by 10.216.68.132 with SMTP id l4mr1480097wed.41.1310107972239; Thu, 07 Jul 2011 23:52:52 -0700 (PDT) Sender: artemb@gmail.com Received: by 10.216.135.169 with HTTP; Thu, 7 Jul 2011 23:52:52 -0700 (PDT) In-Reply-To: <4E169107.6040109@gmail.com> References: <20110707220019.GA79464@DataIX.net> <4E169107.6040109@gmail.com> Date: Thu, 7 Jul 2011 23:52:52 -0700 X-Google-Sender-Auth: pw9a-vov2XKbM8MK2_2-Ms4z8I0 Message-ID: From: Artem Belevich To: Volodymyr Kostyrko Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org Subject: Re: everlasting log device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2011 06:52:54 -0000 On Thu, Jul 7, 2011 at 10:09 PM, Volodymyr Kostyrko wrote: > Well I have already removed device physically and the pool continues to work > despite yelling about missing log device. I already tried booting from flash > drive and importing/exporting pool - this doesn't work. I also tried to > replace this log device with another one. Replacing works and I can remove > old device but the new one will now be unkillable. You may try booting OpenSolaris/Illumos and see if it can do a better job removing ZIL. --Artem From owner-freebsd-fs@FreeBSD.ORG Fri Jul 8 08:08:36 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4D293106566B for ; Fri, 8 Jul 2011 08:08:36 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0FD3D8FC19 for ; Fri, 8 Jul 2011 08:08:35 +0000 (UTC) Received: by iwr19 with SMTP id 19so2037338iwr.13 for ; Fri, 08 Jul 2011 01:08:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:content-transfer-encoding :in-reply-to; bh=U9BMyrRDzmOM41AWmQnLLLwfyyVBF+BF+QOXgMjUDs0=; b=FjNQEi+uasqArnf/NKqz77MpxZuRi/oBBSdr5xwlqvcQvr2mmYIR+7aX0ywz+Bx5yg ojURy5r9NsOIM8b8f8IFjrQvvIOs9nxRkQo+ZBhqQBYppDet4qS8MK5O+mI2dKCDR+bU we7UwXMAej0SJsMjBYFdI0zp/zQ2kpTj7x2aA= Received: by 10.42.91.139 with SMTP id p11mr1554718icm.402.1310112514158; Fri, 08 Jul 2011 01:08:34 -0700 (PDT) Received: from DataIX.net (adsl-99-181-128-71.dsl.klmzmi.sbcglobal.net [99.181.128.71]) by mx.google.com with ESMTPS id e1sm10687224icv.20.2011.07.08.01.08.31 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 08 Jul 2011 01:08:32 -0700 (PDT) Sender: "J. Hellenthal" Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id p6888T9o050567 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 8 Jul 2011 04:08:29 -0400 (EDT) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id p6888Tcj050566; Fri, 8 Jul 2011 04:08:29 -0400 (EDT) (envelope-from jhell@DataIX.net) Date: Fri, 8 Jul 2011 04:08:29 -0400 From: Jason Hellenthal To: Volodymyr Kostyrko Message-ID: <20110708080828.GA49288@DataIX.net> References: <20110707220019.GA79464@DataIX.net> <4E169107.6040109@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <4E169107.6040109@gmail.com> Cc: freebsd-fs@freebsd.org Subject: Re: everlasting log device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2011 08:08:36 -0000 On Fri, Jul 08, 2011 at 08:09:27AM +0300, Volodymyr Kostyrko wrote: > 08.07.2011 01:00, Jason Hellenthal wrote: >=20 > >> System claims that this log device has 148K data. Is this the size of > >> unwritten data? The number is still the same when booting into single > >> user mode and doesn't change at all. > >> > >> Can I remove this log device? Should I recreate the pool to get rid of > >> this behavior? > >> > > > > If you have the possibility to re-create the pool then Id definately > > suggest it. > > > > If you remove this device (physically) your pool will not be operable > > unfortunately there is still somehting missing to allow SLOGs to be > > removed from a running pool yet, what that might be is beyond me at this > > time. You might try to export the pool then boot into single user mode > > and reimport the pool and try the removal procedure but I raelly dont > > think that will help you. >=20 > Well I have already removed device physically and the pool continues to= =20 > work despite yelling about missing log device. I already tried booting=20 > from flash drive and importing/exporting pool - this doesn't work. I=20 > also tried to replace this log device with another one. Replacing works= =20 > and I can remove old device but the new one will now be unkillable. >=20 > Am I right that working without external log device is almost the same=20 > as working with disabled ZIL? >=20 What was the reason or solution for which you added the slog in the first place that you were trying to solve. I would assume it would be for speed but if that's the case and your workloads are not fsync() high amounts of data then there is absolutely no point other than debugging to make FreeBSD better! ;) I don't know what size your device was that you allotted to the slog but hope you didn't go far beyond 256MB yes that's meant to say 'MB'. There is some long discussions about slog/ZIL devices in earlier threads on these lists so you'll have to refer to them to understand what I am talking about here. Cache devices can be removed any time, and depending on the case can improve or decrease performance. ZFS is a large beast to tackle with a lot of different ways you can configure a system and a slew of options at accomplishing that. So in the least words there was a lot of documentation that was put out about it that can aid in determining how YOU should configure your pool and give you somewhat of a description of what things do and how they interact with other system processes. Besides that there is a slew of other information by some well known people on the freebsd threads that will also come in handy, check out the following. Google "site:lists.freebsd.org zfs, slog, zil, cache" Other than that it seems one layer of the system is not talking well to ZFS when its told to do something and it would be really swell if this could be found and exterminated with extreme prejudice. From owner-freebsd-fs@FreeBSD.ORG Fri Jul 8 13:10:04 2011 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 25CE51065674 for ; Fri, 8 Jul 2011 13:10:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 118C58FC08 for ; Fri, 8 Jul 2011 13:10:04 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p68DA3XX019284 for ; Fri, 8 Jul 2011 13:10:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p68DA3Nj019275; Fri, 8 Jul 2011 13:10:03 GMT (envelope-from gnats) Date: Fri, 8 Jul 2011 13:10:03 GMT Message-Id: <201107081310.p68DA3Nj019275@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Martin Birgmeier Cc: Subject: Re: kern/131342: [nfs] mounting/unmounting of disks causes NFS to fail X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Martin Birgmeier List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2011 13:10:04 -0000 The following reply was made to PR kern/131342; it has been noted by GNATS. From: Martin Birgmeier To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/131342: [nfs] mounting/unmounting of disks causes NFS to fail Date: Fri, 08 Jul 2011 15:00:03 +0200 This is a friendly reminder that some kind soul with knowledge of the relevant kernel parts look into this... the error can easily be reproduced. I just had it on a 7.4 system which did heavy reading from an 8.2 server. When I mounted something on the server, the client got a "Permission denied" reply. So, to recap the scenario: 7.4 NFS client 8.2 NFS server client mounts a fs from the server (via IPv4, might be interesting to look at http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/151681, too, but that is unrelated) client does heavy i/o on the mounted fs server does a mount (on its side, in this case it was from an md device) --> error: client gets back some NFS error (in this case "permission denied") From owner-freebsd-fs@FreeBSD.ORG Fri Jul 8 14:08:26 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74207106566C for ; Fri, 8 Jul 2011 14:08:26 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id 24D288FC14 for ; Fri, 8 Jul 2011 14:08:25 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id p68E8NpS026867; Fri, 8 Jul 2011 09:08:25 -0500 (CDT) Date: Fri, 8 Jul 2011 09:08:23 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Volodymyr Kostyrko In-Reply-To: <4E169107.6040109@gmail.com> Message-ID: References: <20110707220019.GA79464@DataIX.net> <4E169107.6040109@gmail.com> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Fri, 08 Jul 2011 09:08:25 -0500 (CDT) Cc: freebsd-fs@freebsd.org Subject: Re: everlasting log device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2011 14:08:26 -0000 On Fri, 8 Jul 2011, Volodymyr Kostyrko wrote: > > Am I right that working without external log device is almost the same as > working with disabled ZIL? No. Zfs should use the main store for its log if a dedicated log device is not available. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Fri Jul 8 19:58:05 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AEF3B106564A for ; Fri, 8 Jul 2011 19:58:05 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 6799D8FC0C for ; Fri, 8 Jul 2011 19:58:04 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap0EAE1gF06DaFvO/2dsb2JhbABUhEKkBrkZkDuBK4QAgQ0EkkyQYA X-IronPort-AV: E=Sophos;i="4.65,500,1304308800"; d="scan'208";a="130282992" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 08 Jul 2011 15:58:04 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 2D1C2B3E95; Fri, 8 Jul 2011 15:58:04 -0400 (EDT) Date: Fri, 8 Jul 2011 15:58:04 -0400 (EDT) From: Rick Macklem To: Martin Birgmeier Message-ID: <1489296886.367033.1310155084168.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <201107081310.p68DA3Nj019275@freefall.freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@FreeBSD.org Subject: Re: kern/131342: [nfs] mounting/unmounting of disks causes NFS to fail X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2011 19:58:05 -0000 Martin Birgmeier wrote: > The following reply was made to PR kern/131342; it has been noted by > GNATS. > > From: Martin Birgmeier > To: bug-followup@FreeBSD.org > Cc: > Subject: Re: kern/131342: [nfs] mounting/unmounting of disks causes > NFS to > fail > Date: Fri, 08 Jul 2011 15:00:03 +0200 > > This is a friendly reminder that some kind soul with knowledge of the > relevant kernel parts look into this... the error can easily be > reproduced. I just had it on a 7.4 system which did heavy reading from > an 8.2 server. When I mounted something on the server, the client got > a > "Permission denied" reply. > > So, to recap the scenario: > > 7.4 NFS client > 8.2 NFS server > client mounts a fs from the server (via IPv4, might be interesting to > look at http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/151681, too, > but > that is unrelated) > client does heavy i/o on the mounted fs > server does a mount (on its side, in this case it was from an md > device) > > --> error: client gets back some NFS error (in this case "permission > denied") > I just made a quick attempt and wasn't able to reproduce this. I mounted/unmounted a UFS volume on the server (both to a subdir of the exported volume and to a directory outside of the exported volume) while a client as accessing an exported fs and didn't get an error. Could the way you mount the volume on the server somehow end up renumbering the exported volumes? If so, the fsid in the file handle will no longer be able to vfs_busyfs(fsid); - and then the mount point will be broken until remounted by the client. I don't use anything like geom and don't use ZFS. Since you can reproduce this easily, I'd suggest that you: 1 - look to make sure drives (the st_dev value returned by stat(2)) aren't being renumbered by the mount. (If they are, then that has to be avoided if an NFS export is to still work.) 2 - Try mounting/unmounting something else, to see if it is md specific. Also, does it only happen when there is a heavy load generated by the client or all the time? (If only under heavy load, it may be a mount list locking bug, since that's the only place where a mount of a non-exported volume on the server will affect the exported mounts, as far as I can see.) I don't mind looking at a packet trace (you can email me the file generated by "tcpdump -s 0 -w host " when run on the server, but only if you can reproduce it without the heavy client load. (If only reproduced when there is a heavy client load a packet trace would be too big and probably not useful, since the bug is likely some race related to the mount list.) rick ps: I assume you are referring to mounts that worked before the server mount and not a case where the new mount was supposed to be exported. From owner-freebsd-fs@FreeBSD.ORG Fri Jul 8 20:12:41 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E914F1065670 for ; Fri, 8 Jul 2011 20:12:41 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id A58C58FC17 for ; Fri, 8 Jul 2011 20:12:41 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap0EALtjF06DaFvO/2dsb2JhbABUhEKkBrkhkDuBK4QAgQ0EkkyQYA X-IronPort-AV: E=Sophos;i="4.65,500,1304308800"; d="scan'208";a="126570988" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 08 Jul 2011 16:12:40 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id BBC34B3F42; Fri, 8 Jul 2011 16:12:40 -0400 (EDT) Date: Fri, 8 Jul 2011 16:12:40 -0400 (EDT) From: Rick Macklem To: Martin Birgmeier Message-ID: <55849302.367556.1310155960717.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <201107081310.p68DA3Nj019275@freefall.freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@FreeBSD.org Subject: Re: kern/131342: [nfs] mounting/unmounting of disks causes NFS to fail X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2011 20:12:42 -0000 Martin Birgmeier wrote: > The following reply was made to PR kern/131342; it has been noted by > GNATS. > > From: Martin Birgmeier > To: bug-followup@FreeBSD.org > Cc: > Subject: Re: kern/131342: [nfs] mounting/unmounting of disks causes > NFS to > fail > Date: Fri, 08 Jul 2011 15:00:03 +0200 > > This is a friendly reminder that some kind soul with knowledge of the > relevant kernel parts look into this... the error can easily be > reproduced. I just had it on a 7.4 system which did heavy reading from > an 8.2 server. When I mounted something on the server, the client got > a > "Permission denied" reply. > > So, to recap the scenario: > > 7.4 NFS client > 8.2 NFS server > client mounts a fs from the server (via IPv4, might be interesting to > look at http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/151681, too, > but > that is unrelated) > client does heavy i/o on the mounted fs > server does a mount (on its side, in this case it was from an md > device) > > --> error: client gets back some NFS error (in this case "permission > denied") > Oh, and one more question... Is the error persistent (ie. is the client mount unusable until remounted) or does the mount point work after the mount/unmount of the other volume has completed? If it just happens when the other volume is unmounted/mounted, make sure that you aren't using the "soft" option for your client mounts. ("soft" implies that an RPC fails after a timeout, and an unmount/mount of another volume could delay the RPC for a while, until the mount list is unlocked.) rick From owner-freebsd-fs@FreeBSD.ORG Sat Jul 9 19:42:50 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E567106564A for ; Sat, 9 Jul 2011 19:42:50 +0000 (UTC) (envelope-from Martin.Birgmeier@aon.at) Received: from email.aon.at (nat-warsl417-01.aon.at [195.3.96.119]) by mx1.freebsd.org (Postfix) with ESMTP id BF7EE8FC0A for ; Sat, 9 Jul 2011 19:42:49 +0000 (UTC) Received: (qmail 30404 invoked from network); 9 Jul 2011 19:16:06 -0000 Received: from smarthub82.res.a1.net (HELO email.aon.at) ([172.18.1.202]) (envelope-sender ) by fallback44.highway.telekom.at (qmail-ldap-1.03) with SMTP for ; 9 Jul 2011 19:16:06 -0000 Received: (qmail 3982 invoked from network); 9 Jul 2011 19:16:03 -0000 X-Spam-Checker-Version: SpamAssassin 3.2.0 (2007-05-01) on WARSBL507.highway.telekom.at X-Spam-Level: Received: from 91-113-4-34.adsl.highway.telekom.at (HELO gandalf.xyzzy) ([91.113.4.34]) (envelope-sender ) by smarthub82.res.a1.net (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 9 Jul 2011 19:16:02 -0000 Received: from atpcdvvc.xyzzy (atpcdvvc.xyzzy [IPv6:fec0:0:0:4d42::84]) by gandalf.xyzzy (8.14.4/8.14.4) with ESMTP id p69JG17C080908; Sat, 9 Jul 2011 21:16:02 +0200 (CEST) (envelope-from Martin.Birgmeier@aon.at) Message-ID: <4E18A8F1.5060102@aon.at> Date: Sat, 09 Jul 2011 21:16:01 +0200 From: Martin Birgmeier User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:5.0) Gecko/20110708 Thunderbird/5.0 MIME-Version: 1.0 To: Rick Macklem References: <1489296886.367033.1310155084168.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <1489296886.367033.1310155084168.JavaMail.root@erie.cs.uoguelph.ca> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org Subject: Re: kern/131342: [nfs] mounting/unmounting of disks causes NFS to fail X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Jul 2011 19:42:50 -0000 Thank you for looking into this - answers below. On 07/08/11 21:58, Rick Macklem wrote: > Martin Birgmeier wrote: >> The following reply was made to PR kern/131342; it has been noted by >> GNATS. >> >> From: Martin Birgmeier >> To: bug-followup@FreeBSD.org >> Cc: >> Subject: Re: kern/131342: [nfs] mounting/unmounting of disks causes >> NFS to >> fail >> Date: Fri, 08 Jul 2011 15:00:03 +0200 >> >> This is a friendly reminder that some kind soul with knowledge of the >> relevant kernel parts look into this... the error can easily be >> reproduced. I just had it on a 7.4 system which did heavy reading from >> an 8.2 server. When I mounted something on the server, the client got >> a >> "Permission denied" reply. >> >> So, to recap the scenario: >> >> 7.4 NFS client >> 8.2 NFS server >> client mounts a fs from the server (via IPv4, might be interesting to >> look at http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/151681, too, >> but >> that is unrelated) >> client does heavy i/o on the mounted fs >> server does a mount (on its side, in this case it was from an md >> device) >> >> --> error: client gets back some NFS error (in this case "permission >> denied") >> > I just made a quick attempt and wasn't able to reproduce this. I mounted/unmounted > a UFS volume on the server (both to a subdir of the exported volume and to a > directory outside of the exported volume) while a client as accessing an exported fs > and didn't get an error. You'll need to be doing heavy NFS i/o from the client to the server while mounting/unmounting something on the server in order to reproduce the problem. > > Could the way you mount the volume on the server somehow end up renumbering the > exported volumes? If so, the fsid in the file handle will no longer be able to > vfs_busyfs(fsid); > - and then the mount point will be broken until remounted by the client. I am sorry, but I don't know how to check this (# of the exported volume). On the other hand, I do not believe it does - see below. > > I don't use anything like geom and don't use ZFS. I had this problem earlier when I wasn't using ZFS, so it does not seem to be specific to ZFS. However, now the server is (also) running ZFS (root is on UFS). > > Since you can reproduce this easily, I'd suggest that you: > 1 - look to make sure drives (the st_dev value returned by stat(2)) aren't being > renumbered by the mount. (If they are, then that has to be avoided if an NFS > export is to still work.) > 2 - Try mounting/unmounting something else, to see if it is md specific. I seem to remember that it's not only confined to adding an md-backed mount on the server, but that I also had this with CDROM mounts (mounting a CD on the server would result in a client error). I'd need to check that, but it might take a while. > > Also, does it only happen when there is a heavy load generated by the client or > all the time? (If only under heavy load, it may be a mount list locking bug, since > that's the only place where a mount of a non-exported volume on the server will > affect the exported mounts, as far as I can see.) I am quite sure it is mostly under heavy load; see also below. > > I don't mind looking at a packet trace (you can email me the file generated by > "tcpdump -s 0 -w host" when run on the server, but only if > you can reproduce it without the heavy client load. (If only reproduced when there > is a heavy client load a packet trace would be too big and probably not useful, > since the bug is likely some race related to the mount list.) Maybe I manage to reproduce it and cut it down sufficiently - this might take a while, though. > > rick > ps: I assume you are referring to mounts that worked before the server mount > and not a case where the new mount was supposed to be exported. That's clear. > > Oh, and one more question... > Is the error persistent (ie. is the client mount unusable until remounted) > or does the mount point work after the mount/unmount of the other volume > has completed? This seems to be a crucial question: in fact, after the single error event (which typically halts the heavy NFS i/o, therefore changes the situation - cf the question about load above), the mount continues to work perfectly. So referring to your question about renumbering above, I'd guess no, it does not get renumbered. > > If it just happens when the other volume is unmounted/mounted, make sure > that you aren't using the "soft" option for your client mounts. ("soft" > implies that an RPC fails after a timeout, and an unmount/mount of > another volume could delay the RPC for a while, until the mount list > is unlocked.) I am not using soft mounts. > > rick Regards, Martin p.s. Sending this also to freebsd-fs, but since I'm currently not subscribed, this might not make it through.