Date: Sat, 6 Apr 2013 11:11:47 -0700 From: John-Mark Gurney <jmg@funkthat.com> To: Bakul Shah <bakul@bitblocks.com> Cc: freebsd-net@freebsd.org, Carl Shapiro <carl.shapiro@gmail.com>, Andriy Gapon <avg@freebsd.org>, FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: close(2) while accept(2) is blocked Message-ID: <20130406181147.GJ76354@funkthat.com> In-Reply-To: <20130330202249.F218BB82A@mail.bitblocks.com> References: <515475C7.6010404@FreeBSD.org> <CANVK_QgnC-pLGwh7Oad87JO_z1WmLeY3kfT9HhdpSzMnpjdNgA@mail.gmail.com> <20130329235431.32D7FB82A@mail.bitblocks.com> <20130330161434.GG76354@funkthat.com> <20130330202249.F218BB82A@mail.bitblocks.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Bakul Shah wrote this message on Sat, Mar 30, 2013 at 13:22 -0700:
> On Sat, 30 Mar 2013 09:14:34 PDT John-Mark Gurney <jmg@funkthat.com> wrote:
> >
> > As someone else pointed out in this thread, if a userland program
> > depends upon this behavior, it has a race condition in it...
> >
> > Thread 1 Thread 2 Thread 3
> > enters routine to read
> > enters routine to close
> > calls close(3)
> > open() returns 3
> > does read(3) for orignal fd
> >
> > How can the original threaded program ensure that thread 2 doesn't
> > create a new fd in between? So even if you use a lock, this won't
> > help, because as far as I know, there is no enter read and unlock
> > mutex call yet...
>
> It is worse. Consider:
>
> fd = open(file,...);
> read(fd, ...);
>
> No guarantee read() gets data from the same opened file!
> Another thread could've come along, closed fd and pointed it
> to another file. So nothing is safe. Might as well stop using
> threads, right?!
Nope, you need your threads to cooperate w/ either locks or some
ownership mechanism like token passing... As long as only one
thread own the fd, you won't have troubles...
> We are talking about cooperative threads where you don't have
> to assume the worst case. Here not being notified on a close
Multiple people have said when threads cooperate without locks, but
no one has given a good example where they do... The cases you list
below are IMO special (and extreme) cases...
We were talking about "cooperating" threads in a general purpose
langage like Java where you can not depend upon your other threads
doing limited amount of work...
> event can complicate things. As an example, I have done
> something like this in the past: A frontend process validating
> TCP connections and then passing on valid TCP connections to
> another process for actual service (via sendmsg() over a unix
> domain). All the worker threads in service process can do a
> recvmsg() on the same fd. They process whatever tcp connection
> they get. Now what happens when the frontend process is
> restarted for some reason? All the worker threads need to
> eventually reconnect to a new unix domain posted by the new
> frontend process. You can handle this multiple ways but
> terminating all the blocking syscalls on the now invalid fd is
> the simplest solution from a user perspective.
This'd make an interesting race... Not sure if it could happen, but
close(), closes the receiving fd, but another thread is in the process
of receiving an fd and puts the new fd in the close of the now closed
unix domain socket... I can draw a more detailed diagram if you want..
> > I decided long ago that this is only solvable by proper use of locking
> > and ensuring that if you call close (the syscall), that you do not have
> > any other thread that may use the fd. It's the close routine's (not
> > syscall) function to make sure it locks out other threads and all other
> > are out of the code path that will use the fd before it calls close..
>
> If you lock before close(), you have to lock before every
> other syscall on that fd. That complicates userland coding and
> slows down things when this can be handled more simply in the
> kernel.
There is "ownership" that can be passed, such as via kqueue _ONESHOT
w/ multiple threads which allows you to avoid locking and only one
thread ever owns the fd...
> Another usecase is where N worker threads all accept() on the
> same fd. Single threading using a lock defeats any performance
> gain.
In this case you still need to do something special since what happens
if one of the worker threads opens a file and/or listen/accept socket
as part of it's work?
Thread 1 Thread 2 Thread 3
about to call accept but
after flag check
sets flag that close
is going to be called
accept connect
process connection
close listen'd socket
open socket as part of
processing gets same fd
calls listen on socket
calls accept on socket
And there we have it, a race condition... You can't always guarantee
what your worker threads do, if you can, it's good, but we need to make
sure that beginner programs don't get into traps like these... They
can easily make the mistake of saying, well, since close kicks all
my threads out, I'll just do that instead of making sure that they
don't have a race condition...
> > If someone could describe how this new eject a person from read could
> > be done in a race safe way, then I'd say go ahead w/ it... Otherwise
> > we're just moving the race around, and letting people think that they
> > have solved the problem when they haven't...
>
> In general it just makes sense to notify everyone waiting on
> something that the situation has changed and they are going to
> have to wait forever. The kernel should already have the
> necessary information about which threads are sleeping on a
> fd. Wake them all up. On being awakened they see that the fd
> is no longer valid and all return with a count of data already
> read or -1 and EBADF. Doing the equivalent in userland is
> complicated.
>
> Carl has pointed out how BSD and Linux have required a
> workaround compared to Solaris and OS X (in Java and IIRC, the
> Go runtime). Seems like we have a number of usecases and this
> is something worth fixing.
I would like to know how a general purpose high level language like
Java can avoid all the races that I brought up w/o locking the fd
and, as you basicly said earlier, w/o the threads cooperating... To
me it doesn't seem possible... Now if someone could explain it, I'd
love to be proved wrong as it'd improve my knowlege of concurent
systems...
--
John-Mark Gurney Voice: +1 415 225 5579
"All that I will do, has been done, All that I have, has not."
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130406181147.GJ76354>