Date: Fri, 8 Jan 2016 20:59:59 -0500 (EST) From: Rick Macklem <rmacklem@uoguelph.ca> To: Jeff Darcy <jdarcy@redhat.com> Cc: Raghavendra G <raghavendra@gluster.com>, freebsd-fs <freebsd-fs@freebsd.org>, Hubbard Jordan <jkh@ixsystems.com>, Xavier Hernandez <xhernandez@datalab.es>, Gluster Devel <gluster-devel@gluster.org> Subject: Re: [Gluster-devel] FreeBSD port of GlusterFS racks up a lot of CPU usage Message-ID: <981529129.154244852.1452304799182.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <1924941590.6473225.1452248249994.JavaMail.zimbra@redhat.com> References: <571237035.145690509.1451437960464.JavaMail.zimbra@uoguelph.ca> <20151230103152.GS13942@ndevos-x240.usersys.redhat.com> <2D8C2729-D556-479B-B4E2-66E1BB222F41@ixsystems.com> <1083933309.146084334.1451517977647.JavaMail.zimbra@uoguelph.ca> <CADRNtgStOg8UZfxNt-SzvvPf7d1J7CC_gi49ww3BbixU0Ey-rg@mail.gmail.com> <568F6D07.6070500@datalab.es> <CADRNtgRM17Eg3Z=LWifVNo=ai72dMiEVRKS3RwNfQ-dK7Pspew@mail.gmail.com> <1924941590.6473225.1452248249994.JavaMail.zimbra@redhat.com>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_154244850_659138303.1452304799181
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Jeff Darcy wrote:
> > > I don't know anything about gluster's poll implementation so I may
> > > be totally wrong, but would it be possible to use an eventfd (or a
> > > pipe if eventfd is not supported) to signal the need to add more
> > > file descriptors to the poll call ?
> > >
> > >
> > > The poll call should listen on this new fd. When we need to change
> > > the fd list, we should simply write to the eventfd or pipe from
> > > another thread. This will cause the poll call to return and we will
> > > be able to change the fd list without having a short timeout nor
> > > having to decide on any trade-off.
> >
> >
> > Thats a nice idea. Based on my understanding of why timeouts are being
> > used, this approach can work.
>
> The own-thread code which preceded the current poll implementation did
> something similar, using a pipe fd to be woken up for new *outgoing*
> messages. That code still exists, and might provide some insight into
> how to do this for the current poll code.
I took a look at event-poll.c and found something interesting...
- A pipe called "breaker" is already set up by event_pool_new_poll() and
closed by event_pool_destroy_poll(), however it never gets used for anything.
So, I added a few lines of code that writes a byte to it whenever the list of
file descriptors is changed and read when poll() returns, if its revents is set.
I also changed the timeout to -1 (infinity) and it seems to work for a trivial
test.
--> Btw, I also noticed the "changed" variable gets set to 1 on a change, but
never reset to 0. I didn't change this, since it looks "racey". (ie. I
think you could easily get a race between a thread that clears it and one
that adds a new fd.)
A slightly safer version of the patch would set a long (100msec ??) timeout instead
of -1.
Anyhow, I've attached the patch in case anyone would like to try it and will
create a bug report for this after I've had more time to test it.
(I only use a couple of laptops, so my testing will be minimal.)
Thanks for all the help, rick
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
>
------=_Part_154244850_659138303.1452304799181
Content-Type: text/x-patch; name=gluster-poll.patch
Content-Disposition: attachment; filename=gluster-poll.patch
Content-Transfer-Encoding: base64
LS0tIGdsdXN0ZXJmcy0zLjcuNi9saWJnbHVzdGVyZnMvc3JjL2V2ZW50LXBvbGwuYy5zYXYJMjAx
Ni0wMS0wNiAxNTo1ODowMy41MjIyODYwMDAgLTA1MDAKKysrIGdsdXN0ZXJmcy0zLjcuNi9saWJn
bHVzdGVyZnMvc3JjL2V2ZW50LXBvbGwuYwkyMDE2LTAxLTA4IDE3OjQ1OjI4LjA3ODAyNDAwMCAt
MDUwMApAQCAtMTgwLDYgKzE4MCwxNiBAQCBldmVudF9wb29sX25ld19wb2xsIChpbnQgY291bnQs
IGludCBldmVuCiAgICAgICAgIHJldHVybiBldmVudF9wb29sOwogfQogCitzdGF0aWMgdm9pZAor
ZXZlbnRfcG9vbF9jaGFuZ2VkIChzdHJ1Y3QgZXZlbnRfcG9vbCAqZXZlbnRfcG9vbCkKK3sKKwor
ICAgICAgICBldmVudF9wb29sLT5jaGFuZ2VkID0gMTsKKyAgICAgICAgLyogV3JpdGUgYSBieXRl
IGludG8gdGhlIGJyZWFrZXIgcGlwZSB0byB3YWtlIHVwIHBvbGwoKS4gKi8KKyAgICAgICAgaWYg
KGV2ZW50X3Bvb2wtPmJyZWFrZXJbMV0gPj0gMCkKKyAgICAgICAgICAgICAgICB3cml0ZShldmVu
dF9wb29sLT5icmVha2VyWzFdLCAiWCIsIDEpOworfQorCiAKIHN0YXRpYyBpbnQKIGV2ZW50X3Jl
Z2lzdGVyX3BvbGwgKHN0cnVjdCBldmVudF9wb29sICpldmVudF9wb29sLCBpbnQgZmQsCkBAIC0y
NDQsNyArMjU0LDcgQEAgZXZlbnRfcmVnaXN0ZXJfcG9sbCAoc3RydWN0IGV2ZW50X3Bvb2wgKgog
ICAgICAgICAgICAgICAgICAgICAgICAgYnJlYWs7CiAgICAgICAgICAgICAgICAgfQogCi0gICAg
ICAgICAgICAgICAgZXZlbnRfcG9vbC0+Y2hhbmdlZCA9IDE7CisgICAgICAgICAgICAgICAgZXZl
bnRfcG9vbF9jaGFuZ2VkKGV2ZW50X3Bvb2wpOwogCiAgICAgICAgIH0KIHVubG9jazoKQEAgLTI3
NSw3ICsyODUsNyBAQCBldmVudF91bnJlZ2lzdGVyX3BvbGwgKHN0cnVjdCBldmVudF9wb29sCiAg
ICAgICAgICAgICAgICAgfQogCiAgICAgICAgICAgICAgICAgZXZlbnRfcG9vbC0+cmVnW2lkeF0g
PSAgZXZlbnRfcG9vbC0+cmVnWy0tZXZlbnRfcG9vbC0+dXNlZF07Ci0gICAgICAgICAgICAgICAg
ZXZlbnRfcG9vbC0+Y2hhbmdlZCA9IDE7CisgICAgICAgICAgICAgICAgZXZlbnRfcG9vbF9jaGFu
Z2VkKGV2ZW50X3Bvb2wpOwogICAgICAgICB9CiB1bmxvY2s6CiAgICAgICAgIHB0aHJlYWRfbXV0
ZXhfdW5sb2NrICgmZXZlbnRfcG9vbC0+bXV0ZXgpOwpAQCAtMzUwLDcgKzM2MCw3IEBAIGV2ZW50
X3NlbGVjdF9vbl9wb2xsIChzdHJ1Y3QgZXZlbnRfcG9vbCAKICAgICAgICAgICAgICAgICB9CiAK
ICAgICAgICAgICAgICAgICBpZiAocG9sbF9pbiArIHBvbGxfb3V0ID4gLTIpCi0gICAgICAgICAg
ICAgICAgICAgICAgICBldmVudF9wb29sLT5jaGFuZ2VkID0gMTsKKyAgICAgICAgICAgICAgICAg
ICAgICAgIGV2ZW50X3Bvb2xfY2hhbmdlZChldmVudF9wb29sKTsKICAgICAgICAgfQogdW5sb2Nr
OgogICAgICAgICBwdGhyZWFkX211dGV4X3VubG9jayAoJmV2ZW50X3Bvb2wtPm11dGV4KTsKQEAg
LTQ0OCw2ICs0NTgsNyBAQCBldmVudF9kaXNwYXRjaF9wb2xsIChzdHJ1Y3QgZXZlbnRfcG9vbCAq
CiAgICAgICAgIGludCAgICAgICAgICAgICAgc2l6ZSA9IDA7CiAgICAgICAgIGludCAgICAgICAg
ICAgICAgaSA9IDA7CiAgICAgICAgIGludCAgICAgICAgICAgICAgcmV0ID0gLTE7CisgICAgICAg
IGNoYXIgICAgICAgICAgICAgeDsKIAogICAgICAgICBHRl9WQUxJREFURV9PUl9HT1RPICgiZXZl
bnQiLCBldmVudF9wb29sLCBvdXQpOwogCkBAIC00NzIsNyArNDgzLDcgQEAgZXZlbnRfZGlzcGF0
Y2hfcG9sbCAoc3RydWN0IGV2ZW50X3Bvb2wgKgogICAgICAgICAgICAgICAgIHNpemUgPSBldmVu
dF9kaXNwYXRjaF9wb2xsX3Jlc2l6ZSAoZXZlbnRfcG9vbCwgdWZkcywgc2l6ZSk7CiAgICAgICAg
ICAgICAgICAgdWZkcyA9IGV2ZW50X3Bvb2wtPmV2Y2FjaGU7CiAKLSAgICAgICAgICAgICAgICBy
ZXQgPSBwb2xsICh1ZmRzLCBzaXplLCAxKTsKKyAgICAgICAgICAgICAgICByZXQgPSBwb2xsICh1
ZmRzLCBzaXplLCAtMSk7CiAKICAgICAgICAgICAgICAgICBpZiAocmV0ID09IDApCiAgICAgICAg
ICAgICAgICAgICAgICAgICAvKiB0aW1lb3V0ICovCkBAIC00ODIsNyArNDkzLDEzIEBAIGV2ZW50
X2Rpc3BhdGNoX3BvbGwgKHN0cnVjdCBldmVudF9wb29sICoKICAgICAgICAgICAgICAgICAgICAg
ICAgIC8qIHN5cyBjYWxsICovCiAgICAgICAgICAgICAgICAgICAgICAgICBjb250aW51ZTsKIAot
ICAgICAgICAgICAgICAgIGZvciAoaSA9IDA7IGkgPCBzaXplOyBpKyspIHsKKyAgICAgICAgICAg
ICAgICBpZiAodWZkc1swXS5yZXZlbnRzICE9IDAgJiYgZXZlbnRfcG9vbC0+YnJlYWtlclswXSA+
PSAwKSB7CisgICAgICAgICAgICAgICAgICAgICAgICAvKiBKdXN0IHJlYWQgYWxsIHRoZSBqdW5r
IGluIHRoZSBicmVha2VyIHBpcGUuICovCisgICAgICAgICAgICAgICAgICAgICAgICB3aGlsZSAo
cmVhZChldmVudF9wb29sLT5icmVha2VyWzBdLCAmeCwgMSkgPiAwKQorICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICA7CisgICAgICAgICAgICAgICAgfQorCisgICAgICAgICAgICAgICAg
Zm9yIChpID0gMTsgaSA8IHNpemU7IGkrKykgewogICAgICAgICAgICAgICAgICAgICAgICAgaWYg
KCF1ZmRzW2ldLnJldmVudHMpCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbnRp
bnVlOwogCg==
------=_Part_154244850_659138303.1452304799181--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?981529129.154244852.1452304799182.JavaMail.zimbra>