From owner-freebsd-geom@FreeBSD.ORG Sun Sep 21 03:44:05 2008 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6DB7106564A for ; Sun, 21 Sep 2008 03:44:05 +0000 (UTC) (envelope-from bsd@fluffles.net) Received: from mail.fluffles.net (fluffles.net [80.69.95.190]) by mx1.freebsd.org (Postfix) with ESMTP id 6BD628FC12 for ; Sun, 21 Sep 2008 03:44:05 +0000 (UTC) (envelope-from bsd@fluffles.net) Received: from [10.10.0.2] (cust.95.160.adsl.cistron.nl [195.64.95.160]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: info@fluffles.net) by mail.fluffles.net (Postfix) with ESMTP id C88F4B29D66; Sun, 21 Sep 2008 05:45:03 +0200 (CEST) Message-ID: <48D5C389.6040209@fluffles.net> Date: Sun, 21 Sep 2008 05:46:17 +0200 From: "fluffles.net" User-Agent: Thunderbird 2.0.0.16 (X11/20080724) MIME-Version: 1.0 To: Andrei Kolu References: <200805201423.56220.antik@bsd.ee> <4832D989.60607@quip.cz> <200805211121.53354.antik@bsd.ee> In-Reply-To: <200805211121.53354.antik@bsd.ee> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-geom@freebsd.org Subject: Re: gjournal misconfiguration X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Sep 2008 03:44:05 -0000 Andrei Kolu wrote: >>> I created geom journal according to this documentation: >>> http://www.freebsd.org/doc/en/books/handbook/geom-gjournal.html >>> >>> I already had partition on second drive (da1s1d) >>> > to me: /dev/da1s1d.journal == /dev/da1s1.journald > > does it make any sense? > Hi Andrei! After some puzzling i realised that your issues with two names for your journal can actually make sense. If my theory is correct, the last sector of da1s1d and da1s1 are the same physical location on disk. Meaning that the d-parition covers the whole /dev/da1s1 device without leaving space for the metasector geom_journal needs. You then created a gjournal on it, warning you of an existing filesystem. You may have thought "i don't need this filesystem anymore" and disregarded the warning. The journal works, but now *two* journals are detected: one on the da1s1 device and one on the da1s1d device, since they both point to the same physical location where gjournal stores its data. So your problem is - i think - actually a partitioning problem, causing unwanted behavior. If you want to use BSD partitions or disklabels (a,d,e,f suffix) you should have labeled /dev/da1s1.journal and NOT /dev/da1s1 which you probably did. I guess you first created /dev/da1s1, then created bsd partitions on it and got /dev/da1s1d and then created a geom_journal on /dev/da1s1d. Geoms work like a container in a container. The /dev/da1s1.journal device (when used with seperate journal storage) is one sector smaller than the underlying /dev/da1s1 device. This keeps the metadata seperate, you can stack geom modules on top of eachother. In your procedure, you have 'violated' the sovereignty of the /dev/da1s1d device probably when you created this d-label, which points to the same last sector as /dev/da1s1. To use journaling on /dev/da1s1d, you should have made /dev/da1s1d at least one sector smaller, when using "bsdlabel -e /dev/da1s1". Then you would have had no problems, only a little lower performance due to stripe misalignment. You should definately resolve this. Here's my suggestion, but it causes all loss of data on the da1 device: First, unmount everything similar to /dev/da1, then: # unload kernel module (important!) kldunload geom_journal # check geom_journal is unloaded kldstat # clear gjournal metasector gjournal clear /dev/da1 gjournal clear /dev/da1s1 gjournal clear /dev/da1s1d # clear all disklabels dd if=/dev/zero of=/dev/da1 bs=1m count=1 # load gjournal gjournal load # create geom_journal device on /dev/da1s1 and using /dev/da1s2 as journal storage: gjournal label /dev/da1s1 /dev/das2 A second option would be to use /dev/da1 in Dangerously Dedicated mode, my favorite, causing other operating systems than FreeBSD to see only junk on the drive and overwrite with a partition table, like Windows does with its "Initialize disk?" question, people click yes and gone is your precious first 20MB of data :) For FreeBSD it works fine though. The advantage is higher performance because you do not have a misalignment anymore, aside from the 64KiB offset UFS2 is hardcoded to. This second option would look like on of these: gjournal label /dev/da1 # OR: gjournal label /dev/da1 /dev/some/other/journal/storage/provider After this step you have /dev/da1.journal or /dev/da1s1.journal depending on the choice you made. Now just apply a filesystem: newfs -J /dev/da1s1.journal and mount using async and optionally noatime: mount -o async,noatime /dev/da1.journal /mnt (change da1 to da1s1 if applicable) And you're done basically. If you really want, you can create labels like /dev/da1s1.journala and /dev/da1s1.journalb - note the last letter indicates the bsd partition/disklabel. Personally i recommend you use DD mode (/dev/da1.journal as filesystem) unless you have good reason not to. You can't do this procedure if this is your system disk, ofcourse. I assume it is not. Regards, Veronica From owner-freebsd-geom@FreeBSD.ORG Mon Sep 22 06:30:35 2008 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5D7EF1065670 for ; Mon, 22 Sep 2008 06:30:35 +0000 (UTC) (envelope-from antik@bsd.ee) Received: from sorbesgroup.com (mail.sorbesgroup.com [217.159.241.118]) by mx1.freebsd.org (Postfix) with ESMTP id DB4258FC12 for ; Mon, 22 Sep 2008 06:30:34 +0000 (UTC) (envelope-from antik@bsd.ee) Received: from localhost (localhost.localdomain [127.0.0.1]) by sorbesgroup.com (Postfix) with ESMTP id E264B3C503D1; Mon, 22 Sep 2008 09:03:10 +0300 (EEST) Received: from sorbesgroup.com ([127.0.0.1]) by localhost (sorbesgroup.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 28784-02; Mon, 22 Sep 2008 09:03:09 +0300 (EEST) Received: from [127.0.0.1] (unknown [192.168.0.80]) by sorbesgroup.com (Postfix) with ESMTP id 3A30C3C503D3; Mon, 22 Sep 2008 09:03:08 +0300 (EEST) Message-ID: <48D7366B.5090004@bsd.ee> Date: Mon, 22 Sep 2008 09:08:43 +0300 From: Andrei Kolu User-Agent: Thunderbird 2.0.0.16 (Windows/20080708) MIME-Version: 1.0 To: "fluffles.net" References: <200805201423.56220.antik@bsd.ee> <4832D989.60607@quip.cz> <200805211121.53354.antik@bsd.ee> <48D5C389.6040209@fluffles.net> In-Reply-To: <48D5C389.6040209@fluffles.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: amavisd-new at localhost Cc: freebsd-geom@freebsd.org Subject: Re: gjournal misconfiguration X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Sep 2008 06:30:35 -0000 fluffles.net wrote: > Andrei Kolu wrote: > >>>> I created geom journal according to this documentation: >>>> http://www.freebsd.org/doc/en/books/handbook/geom-gjournal.html >>>> >>>> I already had partition on second drive (da1s1d) >>>> >>>> > > >> to me: /dev/da1s1d.journal == /dev/da1s1.journald >> >> does it make any sense? >> >> > > Hi Andrei! > > After some puzzling i realised that your issues with two names for your > journal can actually make sense. If my theory is correct, the last > sector of da1s1d and da1s1 are the same physical location on disk. > Meaning that the d-parition covers the whole /dev/da1s1 device without > leaving space for the metasector geom_journal needs. You then created a > gjournal on it, warning you of an existing filesystem. You may have > thought "i don't need this filesystem anymore" and disregarded the > warning. The journal works, but now *two* journals are detected: one on > the da1s1 device and one on the da1s1d device, since they both point to > the same physical location where gjournal stores its data. So your > problem is - i think - actually a partitioning problem, causing unwanted > behavior. > > If you want to use BSD partitions or disklabels (a,d,e,f suffix) you > should have labeled /dev/da1s1.journal and NOT /dev/da1s1 which you > probably did. I guess you first created /dev/da1s1, then created bsd > partitions on it and got /dev/da1s1d and then created a geom_journal on > /dev/da1s1d. Geoms work like a container in a container. The > /dev/da1s1.journal device (when used with seperate journal storage) is > one sector smaller than the underlying /dev/da1s1 device. This keeps the > metadata seperate, you can stack geom modules on top of eachother. In > your procedure, you have 'violated' the sovereignty of the /dev/da1s1d > device probably when you created this d-label, which points to the same > last sector as /dev/da1s1. To use journaling on /dev/da1s1d, you should > have made /dev/da1s1d at least one sector smaller, when using "bsdlabel > -e /dev/da1s1". Then you would have had no problems, only a little lower > performance due to stripe misalignment. > > I made da1s1d with sysinstall- I'd expect sysinstall to behave not create such a mess. Then: # gjournal label -f /dev/da1s1d # gjournal load # newfs -O 2 -J /dev/da1s1d.journal # mount /dev/da1s1d.journal /data Those are exact steps I made, according to this manual: http://www.freebsd.org/doc/en/books/handbook/geom-gjournal.html The real question is: Why it worked from command line without any warning message but failed after reboot and changed journaled filesystem labels? Do we need this .journal extension at all? For example we can live with softupdates without separate /dev/ad4s1a.softupdates name extension. > You can't do this procedure if this is your system disk, ofcourse. I > assume it is not. > > I wonder if I use swap space for journaling how does journaling information survive raw swap operations? IIRC then NTFS filesystem stores journaling information in a middle of partition for faster access and can survive harsh conditions- current FreeBSD last sector usage is a bit problematic. Have anyone tried journaling on geom mirror? Andrei From owner-freebsd-geom@FreeBSD.ORG Mon Sep 22 11:06:53 2008 Return-Path: Delivered-To: freebsd-geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 83FBA1065694 for ; Mon, 22 Sep 2008 11:06:53 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 740668FC23 for ; Mon, 22 Sep 2008 11:06:53 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m8MB6r2w015371 for ; Mon, 22 Sep 2008 11:06:53 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m8MB6qpQ015367 for freebsd-geom@FreeBSD.org; Mon, 22 Sep 2008 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 22 Sep 2008 11:06:52 GMT Message-Id: <200809221106.m8MB6qpQ015367@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-geom@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-geom@FreeBSD.org X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Sep 2008 11:06:53 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/126902 geom [geom] [geom_label] Kernel panic during install boot o kern/124973 geom [gjournal] [patch] boot order affects geom_journal con o kern/124969 geom gvinum(8): gvinum raid5 plex does not detect missing s o kern/124294 geom [geom] gmirror(8) have inappropriate logic when workin o kern/124130 geom [gmirror][usb] gmirror fails to start usb devices that o kern/123962 geom [panic] gjournal(8): gjournal (455Gb data, 8Gb journal o kern/123630 geom [patch] [gmirror] gmirror doesnt allow the original dr o kern/123122 geom [geom] GEOM / gjournal kernel lock f kern/122415 geom [geom] UFS labels are being constantly created and rem o kern/122067 geom [panic]: Geom crashed during boot o kern/121559 geom [patch] [geom] geom label class allows to create inacc o kern/121364 geom [gmirror] Removing all providers create a "zombie" mir o kern/120231 geom [geom] GEOM_CONCAT error adding second drive o kern/120044 geom [msdosfs] [geom] incorrect MSDOSFS label fries adminis o kern/120021 geom net-p2p/qbittorrent crashes system when it works thoug o kern/119743 geom [geom] geom label for cds is keeped after dismount and f kern/115547 geom [geom] [patch] [request] let GEOM Eli get password fro o kern/114532 geom [geom] GEOM_MIRROR shows up in kldstat even if compile o kern/113957 geom [gmirror] gmirror is intermittently reporting a degrad o kern/113885 geom [gmirror] [patch] improved gmirror balance algorithm o kern/113837 geom [geom] unable to access 1024 sector size storage o kern/113419 geom [geom] geom fox multipathing not failing back p bin/110705 geom gmirror(8) control utility does not exit with correct o kern/107707 geom [geom] [patch] [request] add new class geom_xbox360 to o kern/104389 geom [geom] [patch] sys/geom/geom_dump.c doesn't encode XML o kern/98034 geom [geom] dereference of NULL pointer in acd_geom_detach o kern/94632 geom [geom] Kernel output resets input while GELI asks for o kern/90582 geom [geom] [panic] Restore cause panic string (ffs_blkfree o bin/90093 geom fdisk(8) incapable of altering in-core geometry a kern/89660 geom [vinum] [patch] [panic] due to g_malloc returning null o kern/89546 geom [geom] GEOM error s kern/89102 geom [geom] [panic] panic when forced unmount FS from unplu o kern/87544 geom [gbde] mmaping large files on a gbde filesystem deadlo o kern/84556 geom [geom] GBDE-encrypted swap causes panic at shutdown o kern/79251 geom [2TB] newfs fails on 2.6TB gbde device o kern/79035 geom [vinum] gvinum unable to create a striped set of mirro o bin/78131 geom gbde(8) "destroy" not working. s kern/73177 geom kldload geom_* causes panic due to memory exhaustion 38 problems total. From owner-freebsd-geom@FreeBSD.ORG Tue Sep 23 13:31:01 2008 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA029106567F for ; Tue, 23 Sep 2008 13:31:01 +0000 (UTC) (envelope-from news@nermal.rz1.convenimus.net) Received: from dd17730.kasserver.com (dd17730.kasserver.com [85.13.138.103]) by mx1.freebsd.org (Postfix) with ESMTP id A80DB8FC23 for ; Tue, 23 Sep 2008 13:31:01 +0000 (UTC) (envelope-from news@nermal.rz1.convenimus.net) Received: from nermal.rz1.convenimus.net (unknown [87.230.127.0]) by dd17730.kasserver.com (Postfix) with ESMTP id 7AFB11871FFF4 for ; Tue, 23 Sep 2008 15:18:18 +0200 (CEST) Received: by nermal.rz1.convenimus.net (Postfix, from userid 8) id 4AC7A15224; Tue, 23 Sep 2008 14:37:02 +0200 (CEST) To: freebsd-geom@freebsd.org Path: not-for-mail From: Christian Baer Newsgroups: gmane.os.freebsd.devel.geom Date: Tue, 23 Sep 2008 15:18:04 +0200 Organization: Convenimus Projekt Lines: 16 Message-ID: NNTP-Posting-Host: nermal.rz1.convenimus.net Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-Trace: nermal.rz1.convenimus.net 1222173422 47411 192.168.100.10 (23 Sep 2008 12:37:02 GMT) X-Complaints-To: abuse@convenimus.net NNTP-Posting-Date: Tue, 23 Sep 2008 12:37:02 +0000 (UTC) User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.16) Gecko/20080708 Thunderbird/2.0.0.16 Mnenhy/0.7.5.0 Subject: Big file systems with geli X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2008 13:31:02 -0000 Hey there folks! After reading up a little on encryption modes, I was wondering, how large may a geli-encrypted file system be and still be cosidered being secure? There are suggestions not to go above 1TB with a single key in XTS mode[1]. geli uses cbc, IIRC, which is a simpler approach. So I was wondering, what size might be safe here. 1TB file systems can even be on single drives (Seagate has 1,5TB drives) and RAIDs could be even bigger. Since ffs can go way beyond that, a hint concerning geli would be nice. Regards, Chris [1] http://en.wikipedia.org/wiki/Disk_encryption_theory#XTS From owner-freebsd-geom@FreeBSD.ORG Tue Sep 23 14:01:08 2008 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BF2E1065671 for ; Tue, 23 Sep 2008 14:01:08 +0000 (UTC) (envelope-from os@rsu.ru) Received: from mail.r61.net (mail.r61.net [195.208.245.249]) by mx1.freebsd.org (Postfix) with ESMTP id CDABA8FC0A for ; Tue, 23 Sep 2008 14:01:07 +0000 (UTC) (envelope-from os@rsu.ru) Received: from [195.208.252.154] (brain.cc.rsu.ru [195.208.252.154]) (authenticated bits=0) by mail.r61.net (8.14.1/8.14.1) with ESMTP id m8NDZWIK006592 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Tue, 23 Sep 2008 17:35:32 +0400 (MSD) (envelope-from os@rsu.ru) From: Oleg Sharoiko To: freebsd-geom@freebsd.org Content-Type: text/plain Date: Tue, 23 Sep 2008 17:35:32 +0400 Message-Id: <1222176932.22343.54.camel@brain.cc.rsu.ru> Mime-Version: 1.0 X-Mailer: Evolution 2.22.3.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Subject: Kernel panic with gmultipath X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2008 14:01:08 -0000 Hi! I'm getting reproducable panics with geom_multipath. As soon as active provider has been removed from system any I/O activity on consumer will trigger kernel panic. I'll continue debugging this problem and I'll greatly appreciate any help with it, because my knowledge of GEOM subsystem are somewhat limited. Fatal trap 12: page fault while in kernel mode fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x8:0xffffffff801f8006 stack pointer = 0x10:0xffffffffabe63bc0 frame pointer = 0x10:0xffffff000342f680 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 4 (g_down) trap number = 12 panic: page fault (kgdb) bt #0 doadump () at pcpu.h:194 #1 0x0000000000000004 in ?? () #2 0xffffffff80235d18 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #3 0xffffffff80236150 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #4 0xffffffff80397284 in trap_fatal (frame=0xc, eva=Variable "eva" is not available.) at /usr/src/sys/amd64/amd64/trap.c:724 #5 0xffffffff80397665 in trap_pfault (frame=0xffffffffabe63b10, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:641 #6 0xffffffff80397f44 in trap (frame=0xffffffffabe63b10) at /usr/src/sys/amd64/amd64/trap.c:410 #7 0xffffffff80380dee in calltrap () at /usr/src/sys/amd64/amd64/exception.S:169 #8 0xffffffff801f8006 in g_io_request (bp=0xffffff0003d3c1b0, cp=0xffffff000342f680) at /usr/src/sys/geom/geom_io.c:364 #9 0xffffffff801f892d in g_io_schedule_down (tp=Variable "tp" is not available. ) at /usr/src/sys/geom/geom_io.c:540 #10 0xffffffff801f8d38 in g_down_procbody () at /usr/src/sys/geom/geom_kern.c:118 #11 0xffffffff8021a663 in fork_exit ( callout=0xffffffff801f8ce0 , arg=0x0, frame=0xffffffffabe63c80) at /usr/src/sys/kern/kern_fork.c:781 #12 0xffffffff803811be in fork_trampoline () ---Type to continue, or q to quit--- at /usr/src/sys/amd64/amd64/exception.S:415 #13 0x0000000000000000 in ?? () #14 0x0000000000000000 in ?? () #15 0x0000000000000001 in ?? () (kgdb) list *0xffffffff801f8006 0xffffffff801f8006 is in g_io_request (/usr/src/sys/geom/geom_io.c:364). 359 KASSERT(bp->bio_length % cp->provider->sectorsize == 0, 360 ("wrong length %jd for sectorsize %u", 361 bp->bio_length, cp->provider->sectorsize)); 362 } 363 364 g_trace(G_T_BIO, "bio_request(%p) from %p(%s) to %p(%s) cmd %d", 365 bp, cp, cp->geom->name, pp, pp->name, bp->bio_cmd); 366 367 bp->bio_from = cp; 368 bp->bio_to = pp; (kgdb) frame 8 #8 0xffffffff801f8006 in g_io_request (bp=0xffffff0003d3c1b0, cp=0xffffff000342f680) at /usr/src/sys/geom/geom_io.c:364 364 g_trace(G_T_BIO, "bio_request(%p) from %p(%s) to %p(%s) cmd %d", (kgdb) print pp $1 = (struct g_provider *) 0x0 (kgdb) print *cp $2 = {geom = 0xffffff000367a300, consumer = {le_next = 0x0, le_prev = 0xffffff0003ddf008}, provider = 0x0, consumers = {le_next = 0x0, le_prev = 0xffffff000342f720}, acr = 0, acw = 0, ace = 0, spoiled = 0, stat = 0xffffff00033b4d80, nstart = 24279, nend = 24279, private = 0xffffff000342f600, index = 2} -- Oleg Sharoiko. Software and Network Engineer Computer Center of Rostov State University. From owner-freebsd-geom@FreeBSD.ORG Tue Sep 23 21:32:30 2008 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 16224106568B for ; Tue, 23 Sep 2008 21:32:30 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from noop.in-addr.com (mail.in-addr.com [IPv6:2001:470:8:162::1]) by mx1.freebsd.org (Postfix) with ESMTP id E2D938FC1B for ; Tue, 23 Sep 2008 21:32:29 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by noop.in-addr.com with local (Exim 4.54 (FreeBSD)) id 1KiFUg-000Brs-QQ; Tue, 23 Sep 2008 17:32:26 -0400 Date: Tue, 23 Sep 2008 17:32:26 -0400 From: Gary Palmer To: Oleg Sharoiko Message-ID: <20080923213226.GC60230@in-addr.com> References: <1222176932.22343.54.camel@brain.cc.rsu.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1222176932.22343.54.camel@brain.cc.rsu.ru> Cc: freebsd-geom@freebsd.org Subject: Re: Kernel panic with gmultipath X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2008 21:32:30 -0000 On Tue, Sep 23, 2008 at 05:35:32PM +0400, Oleg Sharoiko wrote: > Hi! > > I'm getting reproducable panics with geom_multipath. As soon as active > provider has been removed from system any I/O activity on consumer will > trigger kernel panic. I'll continue debugging this problem and I'll > greatly appreciate any help with it, because my knowledge of GEOM > subsystem are somewhat limited. What version of FreeBSD are you running? Regards, Gary From owner-freebsd-geom@FreeBSD.ORG Wed Sep 24 06:24:24 2008 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E51BB1065674 for ; Wed, 24 Sep 2008 06:24:24 +0000 (UTC) (envelope-from os@rsu.ru) Received: from mail.r61.net (mail.r61.net [195.208.245.249]) by mx1.freebsd.org (Postfix) with ESMTP id 6406E8FC0A for ; Wed, 24 Sep 2008 06:24:24 +0000 (UTC) (envelope-from os@rsu.ru) Received: from mind.local (os.adsl.r61.net [195.208.243.95]) (authenticated bits=0) by mail.r61.net (8.14.1/8.14.1) with ESMTP id m8O6OLph008226 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 24 Sep 2008 10:24:22 +0400 (MSD) (envelope-from os@rsu.ru) Message-ID: <48D9DD10.5070600@rsu.ru> Date: Wed, 24 Sep 2008 10:24:16 +0400 From: Oleg Sharoiko User-Agent: Thunderbird 2.0.0.16 (Macintosh/20080707) MIME-Version: 1.0 To: Gary Palmer References: <1222176932.22343.54.camel@brain.cc.rsu.ru> <20080923213226.GC60230@in-addr.com> In-Reply-To: <20080923213226.GC60230@in-addr.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-geom@freebsd.org Subject: Re: Kernel panic with gmultipath X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2008 06:24:25 -0000 Gary Palmer wrote: > On Tue, Sep 23, 2008 at 05:35:32PM +0400, Oleg Sharoiko wrote: > >> Hi! >> >> I'm getting reproducable panics with geom_multipath. As soon as active >> provider has been removed from system any I/O activity on consumer will >> trigger kernel panic. I'll continue debugging this problem and I'll >> greatly appreciate any help with it, because my knowledge of GEOM >> subsystem are somewhat limited. >> > > What version of FreeBSD are you running? > Forgot to mention that, sorry. This happens with both 7.0-Release and RELENG_7 checked out on 2008-09-18. I'll try HEAD today later, nevertheless I'd like to try fixing this for RELENG_7 to use multipath in production. -- Oleg From owner-freebsd-geom@FreeBSD.ORG Thu Sep 25 05:20:09 2008 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F74A1065686 for ; Thu, 25 Sep 2008 05:20:09 +0000 (UTC) (envelope-from joshua.dunham@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.239]) by mx1.freebsd.org (Postfix) with ESMTP id 258C88FC1F for ; Thu, 25 Sep 2008 05:20:09 +0000 (UTC) (envelope-from joshua.dunham@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so242070rvf.43 for ; Wed, 24 Sep 2008 22:20:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=k4F1crNOfz10SLYEs8mJwdXAzlOxfzFwqQ/LXdLnESU=; b=hCqaoYe9M8/4XqAV6+L01PwOJGoALsH8l0k2mWC9W5BP/TXzcj7HzyIfcPf6tM8IXF JZNjL7cxuMsOJd4bOWY4F5Z8gSwpcd/s09oGHANIrEuF5hja4QhIznCH4nTDraHU0fNy TaZSmjqmpcMGizS6iD3cKRO3GUXiBgP0Chxp0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=gy+5RMLZe9JvMlU/NM8pHDXXkw1aSUrdpczGRXaQdleZXndrn350HgMCN3gY+hhJMF TXzpmQW62xxbdg5QrftSd/rgYthYg0vJvceEFeWEwnThQaxoG2sGkIW3njz45T1Yipyt Ts4DvoLhckJP6Qk/ExcFJr8pMMpc7t3ME8zyA= Received: by 10.140.249.20 with SMTP id w20mr3900713rvh.189.1222318458463; Wed, 24 Sep 2008 21:54:18 -0700 (PDT) Received: by 10.140.202.4 with HTTP; Wed, 24 Sep 2008 21:54:18 -0700 (PDT) Message-ID: Date: Thu, 25 Sep 2008 00:54:18 -0400 From: "Joshua Dunham" To: freebsd-geom@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: After crash geom::graid5 may not resync correctly. Geli complains missing metadata X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Sep 2008 05:20:09 -0000 Hi Guys, So, I'm a graid5 / geli newbie and having some trouble. Here's the scoop. Last week I had a perfectly running system which included 6x 500GB WD sata drives in a geom::graid5 array with geli managing the encryption layer. 2x sata controllers that are the same as when system was working 100% so I assume there should be no conflict now. My transfer rates started crawling so I popped into ssh and immediately read that one of the discs was throwing UDMA xfer errors. Well I rebooted and changed the BIOS settings for the drive and tried to boot back into freebsd. The system then started freezing after a few seconds of starting a resync. I called the drive junk, shut down the system, and ordered a spare. Once it came I replaced the drive and booted into freebsd, formatted the disc using the raid filesystem and rebooted. The system recognized the new drive as one that was out of sync and immediately started to re-sync it. After it was done resynchronizing I went to attach the encryption layer and here is where the trouble started. I get an error message about missing metadata. NAS:~# geli attach /dev/raid5/storage Cannot read metadata from /dev/raid5/storage: Invalid argument. In troubleshooting this problem I have learned about the geli dump command, but it's probably too late now. I also heard from various sources that it is very bad to swap the sata cables on a raid system. I don't think I swapped any cables, but the BIOS has a section where I can define the HD's of the system for booting etc. When you remove a drive the one below it (ad4 let's say) will shift up (to ad6 lets say). This has probably happened, but does it really kill the array? I'd really assume since it's raid5 the metadata for graid5 would keep the discs in the correct order as it stores the Disk number in the metadata. Also, since geli is on a raid5 I'd also assume that the geli metadata would not go missing if the raid5 rebuilds correctly. Please Help! Here are some stats about the geom::graid5 array if it's helpful. NAS:~# graid5 list Geom name: storage State: COMPLETE CALM Status: Total=6, Online=6 Type: AUTOMATIC Pending: (wqp 0 // 0) Stripesize: 131072 MemUse: 0 (msl 0) Newest: -1 ID: 68917578 Providers: 1. Name: raid5/storage Mediasize: 2500539187200 (2.3T) Sectorsize: 512 Mode: r0w0e0 Consumers: 1. Name: ad20 Mediasize: 500107862016 (466G) Sectorsize: 512 Mode: r1w1e1 DiskNo: 4 Error: No 2. Name: ad18 Mediasize: 500107862016 (466G) Sectorsize: 512 Mode: r1w1e1 DiskNo: 1 Error: No 3. Name: ad16 Mediasize: 500107862016 (466G) Sectorsize: 512 Mode: r1w1e1 DiskNo: 0 Error: No 4. Name: ad14 Mediasize: 500107862016 (466G) Sectorsize: 512 Mode: r1w1e1 DiskNo: 5 Error: No 5. Name: ad12 Mediasize: 500107862016 (466G) Sectorsize: 512 Mode: r1w1e1 DiskNo: 3 Error: No 6. Name: ad10 Mediasize: 500107862016 (466G) Sectorsize: 512 Mode: r1w1e1 DiskNo: 2 Error: No NAS:~# graid5 dump ad10 Metadata on ad10: Magic string: GEOM::RAID5 Metadata version: 2 Device name: storage Device ID: 68917578 Disk number: 2 Total number of disks: 6 Provider Size: 500107862016 Verified: -1 State: 0 Stripe size: 131072 Newest: 4294967295 NoHot: No Hardcoded provider: ## graid5 dump adXX output looks exactly the same besides the 'Disk number:' output for all devices. Any advice you guys can give to rescue the data would be soooo appreciated, you have no idea. -Joshua From owner-freebsd-geom@FreeBSD.ORG Sat Sep 27 14:57:41 2008 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF3AC1065686 for ; Sat, 27 Sep 2008 14:57:41 +0000 (UTC) (envelope-from auryn@zirakzigil.org) Received: from aurynhome1sv1.zirakzigil.org (mail.zirakzigil.org [82.63.178.63]) by mx1.freebsd.org (Postfix) with SMTP id DD5AC8FC14 for ; Sat, 27 Sep 2008 14:57:39 +0000 (UTC) (envelope-from auryn@zirakzigil.org) Received: (qmail 98748 invoked by uid 98); 27 Sep 2008 14:30:57 -0000 Received: from 192.168.229.11 by aurynhome1sv1.zirakzigil.org (envelope-from , uid 89) with qmail-scanner-1.25 ( Clear:RC:1(192.168.229.11):. Processed in 0.043597 secs); 27 Sep 2008 14:30:57 -0000 X-Qmail-Scanner-Mail-From: auryn@zirakzigil.org via aurynhome1sv1.zirakzigil.org X-Qmail-Scanner: 1.25 (Clear:RC:1(192.168.229.11):. Processed in 0.043597 secs) Received: from unknown (HELO aurynhome1ws2.zirakzigil.org) (postmaster@zirakzigil.org@192.168.229.11) by 0 with SMTP; 27 Sep 2008 14:30:57 -0000 Message-ID: <48DE439C.4050505@zirakzigil.org> Date: Sat, 27 Sep 2008 16:30:52 +0200 From: Giulio Ferro User-Agent: Thunderbird 2.0.0.0 (X11/20070513) MIME-Version: 1.0 To: freebsd-geom@freebsd.org, pjd@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Problems with journal? X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Sep 2008 14:57:42 -0000 I'm experiencing very serious delay issues in 2 production servers. The server are dual intel quad core o.s. : freebsd 7 stable (yesterday) amd64, mfi driver for 2 physical controllers: 1) two sas disks in mirror (mfi0) 2) one dell md1000 disk enclosure with 15 sata disks in raid 60 (total ~7 TB) (mfi1) I created a single partition for the second disk, I put the geom_journal on it, I formatted it with ufs and mounted it on /data directory. from /etc/fstab /dev/mfid1.journal /data ufs rw,async 2 0 Everything seemed to work nice until I put the servers in production under a heavy load. In fact, I found that something was wrong when the standard maintenance script /etc/periodic/daily/450.status-security would very nearly freeze the server at night. When this happened the server would get incredibly slow, and it took minutes for me to ssh login and kill the process hanging the system. The process which hung the system was a simple find on the /data partition. During this the machine became unresponsive, and the db which stays on the same partition didn't serve requests any more. During this hang a "top" command would show a 20% system and 100% for the find process. In order to check whether the problem was with the journal, I enabled debugging: sysctl kern.geom.journal.debug=1 I started the "killer" script and waited. Nothing happened for several minutes, then the problem would show its ugly face again and here is what I got in the logs: (normal behavior) Sep 27 15:20:38 dcsdata1 kernel: GEOM_JOURNAL[1]: Sync time of /data: 0.043653s Sep 27 15:20:38 dcsdata1 kernel: GEOM_JOURNAL[1]: BIO_FLUSH time of mfid1: 0.000001s Sep 27 15:20:38 dcsdata1 kernel: GEOM_JOURNAL[1]: Suspend time of /data: 0.087174s Sep 27 15:20:38 dcsdata1 kernel: GEOM_JOURNAL[1]: Starting copy of journal. Sep 27 15:20:38 dcsdata1 kernel: GEOM_JOURNAL[1]: Switch time of mfid1: 0.003278s Sep 27 15:20:38 dcsdata1 kernel: GEOM_JOURNAL[1]: Entire switch time: 0.172709s Sep 27 15:20:38 dcsdata1 kernel: GEOM_JOURNAL[1]: Data has been copied. Sep 27 15:20:48 dcsdata1 kernel: GEOM_JOURNAL[1]: Entire switch time: 0.000005s Sep 27 15:20:58 dcsdata1 kernel: GEOM_JOURNAL[1]: Entire switch time: 0.000005s Sep 27 15:21:08 dcsdata1 kernel: GEOM_JOURNAL[1]: Msync time of /data: 0.029507s Sep 27 15:21:08 dcsdata1 kernel: GEOM_JOURNAL[1]: Sync time of /data: 0.053562s Sep 27 15:21:08 dcsdata1 kernel: GEOM_JOURNAL[1]: BIO_FLUSH time of mfid1: 0.000002s Sep 27 15:21:08 dcsdata1 kernel: GEOM_JOURNAL[1]: Suspend time of /data: 0.086662s Sep 27 15:21:08 dcsdata1 kernel: GEOM_JOURNAL[1]: Starting copy of journal. Sep 27 15:21:08 dcsdata1 kernel: GEOM_JOURNAL[1]: Switch time of mfid1: 0.003178s Sep 27 15:21:08 dcsdata1 kernel: GEOM_JOURNAL[1]: Entire switch time: 0.183835s Sep 27 15:21:08 dcsdata1 kernel: GEOM_JOURNAL[1]: Data has been copied. (starts abnormal behavior) Sep 27 15:24:40 dcsdata1 kernel: GEOM_JOURNAL[1]: Suspend time of /data: 181.380868s Sep 27 15:24:40 dcsdata1 kernel: GEOM_JOURNAL[1]: Starting copy of journal. Sep 27 15:24:40 dcsdata1 kernel: GEOM_JOURNAL[1]: Switch time of mfid1: 0.003105s Sep 27 15:24:40 dcsdata1 kernel: GEOM_JOURNAL[1]: Entire switch time: 181.473032s Sep 27 15:24:40 dcsdata1 kernel: GEOM_JOURNAL[1]: Data has been copied. Sep 27 15:24:50 dcsdata1 kernel: GEOM_JOURNAL[1]: Entire switch time: 0.000007s Sep 27 15:25:00 dcsdata1 kernel: GEOM_JOURNAL[1]: Msync time of /data: 0.029305s Sep 27 15:25:00 dcsdata1 kernel: GEOM_JOURNAL[1]: Sync time of /data: 0.045913s Sep 27 15:25:00 dcsdata1 kernel: GEOM_JOURNAL[1]: BIO_FLUSH time of mfid1: 0.000001s After this no more logs would show, and after a long time I finally killed the "find" process. In the logs I got this: Sep 27 15:42:30 dcsdata1 kernel: GEOM_JOURNAL[1]: Suspend time of /data: 1050.197022s Sep 27 15:42:30 dcsdata1 kernel: GEOM_JOURNAL[1]: Starting copy of journal. Sep 27 15:42:30 dcsdata1 kernel: GEOM_JOURNAL[1]: Switch time of mfid1: 0.003276s Sep 27 15:42:30 dcsdata1 kernel: GEOM_JOURNAL[1]: Entire switch time: 1050.282609s Sep 27 15:42:30 dcsdata1 kernel: GEOM_JOURNAL[1]: Data has been copied. and then everything got normal again: Sep 27 15:42:40 dcsdata1 kernel: GEOM_JOURNAL[1]: Msync time of /data: 0.026546s Sep 27 15:42:40 dcsdata1 kernel: GEOM_JOURNAL[1]: Sync time of /data: 0.043528s Sep 27 15:42:40 dcsdata1 kernel: GEOM_JOURNAL[1]: BIO_FLUSH time of mfid1: 0.000002s Sep 27 15:42:40 dcsdata1 kernel: GEOM_JOURNAL[1]: Suspend time of /data: 0.041300s Sep 27 15:42:40 dcsdata1 kernel: GEOM_JOURNAL[1]: Starting copy of journal. Sep 27 15:42:40 dcsdata1 kernel: GEOM_JOURNAL[1]: Switch time of mfid1: 0.005283s Sep 27 15:42:40 dcsdata1 kernel: GEOM_JOURNAL[1]: Entire switch time: 0.133709s Sep 27 15:42:40 dcsdata1 kernel: GEOM_JOURNAL[1]: Data has been copied. Sep 27 15:42:50 dcsdata1 kernel: GEOM_JOURNAL[1]: Msync time of /data: 0.025201s Sep 27 15:42:50 dcsdata1 kernel: GEOM_JOURNAL[1]: Sync time of /data: 0.036892s Sep 27 15:42:50 dcsdata1 kernel: GEOM_JOURNAL[1]: BIO_FLUSH time of mfid1: 0.000001s Sep 27 15:42:50 dcsdata1 kernel: GEOM_JOURNAL[1]: Suspend time of /data: 0.036874s Sep 27 15:42:50 dcsdata1 kernel: GEOM_JOURNAL[1]: Starting copy of journal. Sep 27 15:42:50 dcsdata1 kernel: GEOM_JOURNAL[1]: Switch time of mfid1: 0.002075s Sep 27 15:42:50 dcsdata1 kernel: GEOM_JOURNAL[1]: Entire switch time: 0.108044s Sep 27 15:42:50 dcsdata1 kernel: GEOM_JOURNAL[1]: Data has been copied. Sep 27 15:43:01 dcsdata1 kernel: GEOM_JOURNAL[1]: Msync time of /data: 0.026738s Sep 27 15:43:01 dcsdata1 kernel: GEOM_JOURNAL[1]: Sync time of /data: 0.036973s Sep 27 15:43:01 dcsdata1 kernel: GEOM_JOURNAL[1]: BIO_FLUSH time of mfid1: 0.000001s Sep 27 15:43:01 dcsdata1 kernel: GEOM_JOURNAL[1]: Suspend time of /data: 0.035504s Sep 27 15:43:01 dcsdata1 kernel: GEOM_JOURNAL[1]: Starting copy of journal. Sep 27 15:43:01 dcsdata1 kernel: GEOM_JOURNAL[1]: Switch time of mfid1: 0.003822s Sep 27 15:43:01 dcsdata1 kernel: GEOM_JOURNAL[1]: Entire switch time: 0.110049s Sep 27 15:43:01 dcsdata1 kernel: GEOM_JOURNAL[1]: Data has been copied. From what I can see the mfi driver and the hardware below it are not to blame for the abysmal performance, but the journal probably is. I hope any of you can help me look in the right direction, and point me to any further tests to try or tunable to set... Thanks in advance.