Date: Sat, 7 Oct 2017 20:34:06 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Ian Lepore <ian@freebsd.org>, "freebsd-current@freebsd.org" <freebsd-current@freebsd.org> Subject: Re: RFC how to use kernel procs/threads efficiently Message-ID: <YQXPR0101MB099790E37DE1BCB023AA130FDD760@YQXPR0101MB0997.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <1507317060.86205.268.camel@freebsd.org> References: <YQXPR0101MB099752292CCAC9E8A72C1E96DD710@YQXPR0101MB0997.CANPRD01.PROD.OUTLOOK.COM>, <1507317060.86205.268.camel@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Ian Lepore wrote: >On Fri, 2017-10-06 at 19:02 +0000, Rick Macklem wrote: >> Hi, >> >> I have now dropped the client side of Flexible File Layout for pNFS into= head >> and I believe it is basically working. >> Currently when talking to mirrored DS servers, it does the Write and Com= mit >> RPCs to the mirrors serially. This works, but is inefficient w.r.t. elap= sed to to >> completion. >> >> To do them concurrently, I need separate kernel processes/threads to do = them. >> I can think of two ways to do this: >> 1 - The code that I have running in projects/pnfs-planb-server for the p= NFS server >> side does a kproc_create() to create a kernel process that does th= e RPC and >> then krpc_exit()s. >> - This was easy to code and works. However, I am concerned that th= ere is >> going to be excessive overheads from doing all the kproc_create(= )s and >> kproc_exit()s? >> Anyone know if these calls will result in large overheads? >> 2 - I haven't coded this, but the other way I can think of to do this is= to >> create a pool of threads (kthread_create() is sufficient in this c= ase, I >> think?) and then hand each RPC to an available thread so it can do= the RPC. >> - Other than a little more complex coding, the main issue I see wi= th this one >> is "How many threads and when to create more/less of them.". >> >> Anyhow, any comments w.r.t. the merits of either of the above approaches >> (or a suggestion of other ways to do this) would be appreciated, rick > >taskqueue(9) is an existing mechanism to enqueue functions to execute >asynch using a pool of threads, but it doesn't answer the scalability >questions. In fact it may make them harder, inasmuch as I don't think >there's a mechanism to dynamically adjust the number of threads after >first calling taskqueue_start_threads(). Hmm, yes. Thanks for the pointer. I hadn't read "man taskqueue" until now. The kernel RPC doesn't use this and I suspect that it is because of what yo= u said w.r.t. dynamically adjusting the # of threads. However, it does save "hand coding" the queues for #2 and I'm lazy (plus don't believe reinventing the wheel is the best plan). I think I will try using taskqueue and just have a sysctl for #of-threads. (Actually most of the code ends up the same, because basically they all end up with a function with a single argument that does the RPC. The only difference is what call starts the RPC.) Anyone else have comments? rick=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQXPR0101MB099790E37DE1BCB023AA130FDD760>