Date: Mon, 14 Jun 2010 18:07:51 +0300 From: Mikolaj Golub <to.my.trociny@gmail.com> To: Pawel Jakub Dawidek <pjd@FreeBSD.org> Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST Message-ID: <868w6hwt2w.fsf@kopusha.home.net> In-Reply-To: <20100614095044.GH1721@garage.freebsd.pl> (Pawel Jakub Dawidek's message of "Mon, 14 Jun 2010 11:50:44 %2B0200") References: <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 14 Jun 2010 11:50:44 +0200 Pawel Jakub Dawidek wrote: PJD> On Mon, Jun 14, 2010 at 10:28:15AM +0300, Mikolaj Golub wrote: >> >> On Sun, 13 Jun 2010 12:24:01 +0200 Pawel Jakub Dawidek wrote: >> >> >> Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. >> >> PJD> This is the most important bit from the primary node. >> >> PJD> The header either does not contain 'seq' field or this field is 0. It >> PJD> can only be 0 if you have old kernel. With recent kernel geom_gate.ko >> PJD> was modified to start seq at 1, so this should not happen. >> >> I am a bit confused how this seq is supposed to work. For sync thread. I have >> set up hast on 8-STABLE (before I used it on 9-CURRENT only) and have the same >> issue as hiroshi@ does. I have added >> >> pjdlog_debug(2, "remote_send: seq is %llu.", (uint64_t)ggio->gctl_seq); >> >> after >> >> nv_add_uint64(nv, (uint64_t)ggio->gctl_seq, "seq"); >> >> in primary/remote_send thread and observe the following: PJD> [...] PJD> Could you find where exactly it looses proper value? PJD> I found that in ggate_recv_thread() after ioctl(2), gctl_seq has PJD> expected value, but I'm not setup to test it further quickly. I suppose ggate_recv_thread() is ok but as I wrote earlier I am concerned about sync thread. I have added additional prints and how it looks: Jun 14 17:47:50 zhuzha hastd: [storage] (primary) Device hast/storage recovered. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) init_environment: hio: Ox28441580; seq: 0. ... Jun 14 17:47:50 zhuzha hastd: [storage] (primary) init_environment: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) init_environment: hio: Ox284f2640; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) ggate_recv: Taking free request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) ggate_recv: (0x284f2640) Got free request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) ggate_recv 1: hio: Ox284f2640; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) ggate_recv: (0x284f2640) Waiting for request from the kernel. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) ggate_recv 2: hio: Ox284f2640; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send: Taking request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) local_send: Taking request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) ggate_send: Taking request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_recv: No requests, waiting. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_guard: Checking connections. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) Synchronization started. 10485760 bytes to go. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_guard: Connection to tcp4://192.168.120.6 is ok. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync: Taking free request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync: (0x284f2600) Got free request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 1: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 2: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 3: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync: (0x284f2600) Sending sync request: READ(0, 131072). Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync: (0x284f2600) Moving request to the send queue. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 4: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 5: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) local_send: (0x284f2600) Got request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) local_send 1: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) local_send: Taking request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 6: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 7: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync: (0x284f2600) Sending sync request: WRITE(0, 131072). Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync: (0x284f2600) Moving request to the send queue. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 8: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync: (0x284f2600) Moving request to the send queues. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 9: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send: (0x284f2600) Got request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send 1: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send 3: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send: seq is 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send: (0x284f2600) Moving request to the recv queue. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send 7: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send: Taking request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) Header contains no 'seq' field. Jun 14 17:47:50 zhuzha kernel: Jun 14 17:47:50 zhuzha hastd: [storage] (primary) Header contains no 'seq' field. So ggate_recv takes free just initilized hio (0x284f2640) from the free queue and is waiting for the data from the kernel. At this time sync thread starts syncronization, takes another just initilized hio (0x284f2600) from the free queue and puts it to remote_send, so the request is sent with seq == 0. -- Mikolaj Golub
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?868w6hwt2w.fsf>