From owner-freebsd-threads@FreeBSD.ORG Sun Jun 15 23:06:39 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3904C37B401; Sun, 15 Jun 2003 23:06:39 -0700 (PDT) Received: from mta4.rcsntx.swbell.net (mta4.rcsntx.swbell.net [151.164.30.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id 787C243F93; Sun, 15 Jun 2003 23:06:36 -0700 (PDT) (envelope-from julian@elischer.org) Received: from elischer.org (adsl-68-120-128-240.dsl.snfc21.pacbell.net [68.120.128.240])h5G66XbB007480; Mon, 16 Jun 2003 01:06:34 -0500 (CDT) Message-ID: <3EED5D8B.80703@elischer.org> Date: Sun, 15 Jun 2003 23:02:51 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4b) Gecko/20030524 X-Accept-Language: en, hu MIME-Version: 1.0 To: Alexander Nedotsukov References: <3EE74525.7020809@mail.ru> In-Reply-To: <3EE74525.7020809@mail.ru> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: threads@freebsd.org cc: Daniel Eischen cc: "Matthew N. Dodd" Subject: Re: nvidia OpenGL and lib{thr,kse} related crash X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 06:06:39 -0000 Alexander Nedotsukov wrote: > The kernel driver compilled by me on -cuurent. The OpenGL library by > nvidia on 4.x for sure. > Can you say is this http://www.minion.de/files/machdep.c.diff supposed > to save %gs? that patch is against a version of the file int eh 4.x branch it is completely irrelevant for the head branch as only the head branch used %gs in is thread libraries. > >> > Well just give me a hint which one should be removed: > > options COMPAT_43 #Compatible with BSD 4.3 [KEEP > THIS!] > options COMPAT_FREEBSD4 #Compatible with FreeBSD4 these are just kernel ABI compaibility things. he was more refering to which libraries they were linked against. -- +------------------------------------+ ______ _ __ | __--_|\ Julian Elischer | \ U \/ / hard at work in | / \ julian@elischer.org +------>x USA \ a very strange | ( OZ ) \___ ___ | country ! +- X_.---._/ presently in San Francisco \_/ \\ v From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 13:11:01 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8E19737B401 for ; Mon, 16 Jun 2003 13:11:01 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id D8AF543FA3 for ; Mon, 16 Jun 2003 13:11:00 -0700 (PDT) (envelope-from eischen@pcnet.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5GKAxXh022156; Mon, 16 Jun 2003 16:10:59 -0400 (EDT) Date: Mon, 16 Jun 2003 16:10:59 -0400 (EDT) From: Daniel Eischen To: Andy Ritger In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: Gareth Hughes Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 20:11:01 -0000 On Sat, 14 Jun 2003, Andy Ritger wrote: > > I'd like to add a few comments to what has recently been said here > and on freebsd-current to clarify what the %gs register is used > for by the NVIDIA driver and what ideally would be done to allow > the NVIDIA driver to coexist with FreeBSD threading implementations. > > The NVIDIA driver does not need %gs to maintain internal state in the > kernel; rather, it uses it to maintain fast thread local data for the > OpenGL implementation, where it is critically important to have fast > (single instruction) access to such data. Take 2. After thinking a bit more... I guess OpenGL (or OpenGL implementation interfaces to the NVIDIA driver) doesn't have thread-safe interfaces. Instead of trying to change the interface of the OS, why not change (or add) OpenGL or NVIDIA driver interfaces so that there is not a need for TLS? -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 14:22:42 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DD38337B405 for ; Mon, 16 Jun 2003 14:22:41 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4325443FB1 for ; Mon, 16 Jun 2003 14:22:38 -0700 (PDT) (envelope-from ARitger@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 14:25:45 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 14:22:24 -0700 Received: from nvidia.com (BERNSTEIN [172.16.224.128]) by mail-sc-0.nvidia.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id MJR3VW6Y; Mon, 16 Jun 2003 14:22:22 -0700 From: Andy Ritger To: Daniel Eischen Date: Mon, 16 Jun 2003 17:54:47 -0400 (EDT) In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: Gareth Hughes Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 21:22:42 -0000 On Mon, 16 Jun 2003, Daniel Eischen wrote: > On Sat, 14 Jun 2003, Andy Ritger wrote: > > > > I'd like to add a few comments to what has recently been said here > > and on freebsd-current to clarify what the %gs register is used > > for by the NVIDIA driver and what ideally would be done to allow > > the NVIDIA driver to coexist with FreeBSD threading implementations. > > > > The NVIDIA driver does not need %gs to maintain internal state in the > > kernel; rather, it uses it to maintain fast thread local data for the > > OpenGL implementation, where it is critically important to have fast > > (single instruction) access to such data. > > Take 2. After thinking a bit more... > > I guess OpenGL (or OpenGL implementation interfaces to the > NVIDIA driver) doesn't have thread-safe interfaces. Instead > of trying to change the interface of the OS, why not change > (or add) OpenGL or NVIDIA driver interfaces so that there > is not a need for TLS? Thanks, Dan. Sorry for the slow response. I had hoped to put some tests together to have some hard numbers to backup my argument. Unfortunately, I had to put those tests on hold. The issue is really that each thread has its own rendering context, and an OpenGL implementation needs fast access to that thread-local rendering context data. OpenGL is a state machine. Applications call commands like: glColor3f(1.0, 0.0, 0.0); /* set the current color */ glTexCoord2f(1.0, 1.0); /* set the current texture coordinate */ glVertex3f(0.0, 0.0, 0.0); /* draw a vertex, using the current * accumulated state in the rendering * context */ Also, the affect that an OpenGL command has may vary, based on previously accumulated state, or different modes of operation that have been enabled. A common implementation technique to deal with this is to use dispatch tables that get "plugged in" based on the mode of operation. Because modes of operation are enabled by a thread for the rendering context that it is currently using, these dispatch tables must be thread specific. Consider this example: void glFoo(GLint bar) { gl_dispatch_t *dispatch = GET_CURRENT_DISPATCH(); dispatch->Foo(bar); } where the current dispatch table is fetched from thread-local storage. Having to perform a function call to do this lookup can severely impact performance, particularly when the function that will be executed after the lookup is less than ten instructions long. OpenGL's immediate mode rendering API falls into this category, which is used heavily in workstation applications and benchmarks like Viewperf. To give you an idea of how this dispatch mechanism can be implemented, here is how it could be implemented on Linux using the new TLS implementation: glFoo: mov %gs:__gl_dispatch@ntpoff, %eax jmp *__foo_offset(%eax) Here, we have a __thread variable "__gl_dispatch" which holds the current thread's dispatch table pointer. This is fetched using the Local Exec TLS access model (a single instruction per access), and the required function pointer is jumped through to execute the correct backend function. I think adding new OpenGL interfaces to eliminate the need for TLS is out of the question: this would require source code changes to applications that already work fine on other operating systems. The TLS problem has been solved numerous times already on x86; it's hard to make the case that it can't be solved this time without changing the OpenGL API. I can appreciate that you might not currently have the resources to pursue the ELF TLS mechanism that the glibc folks have implemented. Hopefully, this might be something to pursue in the future. While I have very little bandwidth to spare, I'd be willing to work with anyone else interested to investigate further the idea of ELF TLS on FreeBSD. So from an OpenGL point of view, here are several alternatives that I see for atleast the near term: - make NVIDIA's OpenGL implementation not thread-safe (just use global data rather that thread-local data) - accept the performance hit of using pthread_getspecific() on FreeBSD. From talking to other OpenGL engineers, conservative estimates of the performance impact on applications like viewperf range from 10% - 15%. I'd like to quantify that, but certainly there will be a performance penalty. both of these options are regressions from the current driver. Another alternative, which is admittedly hacky, would be for the userland KSE implementation to reserve a chunk of data (say, 16 words) at a fixed offset in the thread environment block for an OpenGL implementation to use. Thanks, - Andy > -- > Dan Eischen > From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 14:41:07 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8E35637B401 for ; Mon, 16 Jun 2003 14:41:07 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 033DE43F3F for ; Mon, 16 Jun 2003 14:41:05 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 14:44:24 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 14:41:03 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7CD@mail-sc-3.nvidia.com> From: Gareth Hughes To: Andy Ritger , Daniel Eischen Date: Mon, 16 Jun 2003 14:41:01 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 21:41:07 -0000 On Mon, 16 Jun 2003, Andy Ritger wrote: > > So from an OpenGL point of view, here are several alternatives that > I see for atleast the near term: > > - make NVIDIA's OpenGL implementation not thread-safe (just > use global data rather that thread-local data) > > - accept the performance hit of using pthread_getspecific() > on FreeBSD. From talking to other OpenGL engineers, > conservative estimates of the performance impact on > applications like viewperf range from 10% - 15%. I'd like > to quantify that, but certainly there will be a performance > penalty. And these are *very* conservative estimates -- you're essentially adding a function call into a path that is, on average, less than ten instructions per OpenGL API call, where the number of API calls per frame is upward of 3 million (3 calls per vertex, over a million vertices for some Viewperf benchmarks). The API was designed this way for a reason, and fast thread-local storage is a fundamental part of a high performance implementation. -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 15:01:00 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0208B37B401 for ; Mon, 16 Jun 2003 15:01:00 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8A15B43F3F for ; Mon, 16 Jun 2003 15:00:58 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 15:03:45 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 15:00:24 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7D1@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Julian Elischer' , Andy Ritger Date: Mon, 16 Jun 2003 15:00:23 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de cc: Daniel Eischen Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 22:01:00 -0000 On Mon, 16 Jun 2003, Julian Elischer wrote: > > It wouldn't take much for us to give you a "application specific > pointer" in the memory block that is pointed to by %gs. > > This would allow you to find your data with a single dereference > for the N:N threads library.. > > %gs ----->[threadsystem-thread-specific-data] > [ stuff ] > [ ap-specific pointe ]---->[ your data ] > [ ] > > BUT > that would only work for the 1:1 threading library What's wrong with: %gs ----->[threadsystem-thread-specific-data ] [ stuff ] [libGL stuff (fixed size, known offset)] Or, better yet, to make sure no problems arise when you change the internals of your data structures: %gs ----->[libGL stuff (16 words, say) ] [threadsystem-thread-specific-data] [ stuff ] You reserve the first 16 words of your thread data structure for us, and we're done. At least when this library is being used. > The M:N library (known as KSE) uses an extra level of > indirection already. It has a "virtual-CPU" (KSE) that is > pointer to by the %gs. that in turn knows what thread it is currently > working for. > > > %gs ----->[threadsystem-virtual-cpu-specific-data] > [ stuff ] > [ current-thread-pointer ]------\ > | > /----------------------------------------------------/ > | > \----->[threadsystem-thread-specific-data] > [ stuff ] > [ ap-specific pointe ]---->[ your data ] > [ ] > > you couldn't do both with the same code.. > > We'd have to (in both libraries) supply you with an entrypoint for a > function to return the pointer to your address. > It could be optimised to still be pretty fast, but it's certainly > non-standard.. We really want to avoid function calls. What you're describing here is essentially the ELF TLS mechanism (one flavour, at least). If you haven't done so already, you should check out Ulrich Drepper's document. Keeping this kind of thing in mind when you're developing the thread libraries is a good idea -- the ELF TLS mechanisms were designed to be both portable and fast. Both Linux and Solaris support __thread variables, with Linux supporting them on perhaps half a dozen architectures (last time I checked). > This is the problem that you see when you decide > to use a non-standard manner to do these thing however. > (By which I mean that you decided to "roll-your-own" but you have made > yourself non-portable) I'm not sure what you mean by this. Do you mean if you provide us with a magic function to get our thread-local data, we'd be non-portable? Or that because we use the ELF TLS stuff we're non-portable already? -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 15:09:02 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 610DA37B404 for ; Mon, 16 Jun 2003 15:09:02 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id BAC8943FD7 for ; Mon, 16 Jun 2003 15:09:01 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 15:11:49 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 15:08:28 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7D3@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Julian Elischer' Date: Mon, 16 Jun 2003 15:08:27 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de cc: Daniel Eischen cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 22:09:02 -0000 On Mon, 16 Jun 2003, Julian Elischer wrote: > > Unfortunatly you are making an assumption about the threads that you > are only able to make in Linux "by chance" as they did something else > for their TLS. I gather that you don't keep a local variable > pointed to your local drawing context, but just use the 'global' > style of programming but using %gs as a 'roll-your-own' version of > 'local context'. Can you just grab %gs on Microsoft? I thought they > were going to be using it for their TLS too. There is space reserved for OpenGL in the Windows thread environment block, so you can access your current context pointer at a fixed offset from %fs. I don't understand your point about local variables -- fundamentally, OpenGL has a notion of the current context. All rendering that occurs in a given thread is based on the context that is currently bound to that thread. Each thread can have its own context, although no context can be bound to more than one thread at a time. Thus, you have two options: 1) Store your pointers in global variables, and be thread-unsafe. 2) Store your pointers in some thread-safe manner, i.e., thread-local storage. If you need us to explain why this is so in greater detail, please just say the word. > > BTW > Have you looked at the speed of that? At one time it was a lot slower > to dereference things through %gs than it was to simply have a normal > register allocated to the task (e.g. %esi) as would be done normally. > I don't know if that is true with any modern machines though. Sure, you may (and I stress may) pay a few cycles of overhead for a segment prefix, but it is nothing compared to a function call. > HOWEVER.. > as far as we know we still have %fs unused.... > Maybe we could just switch registers :-) > (that too is of course non portable too but.....) Wine uses %fs. > BTW what do you do for ia64/alpha/amd64 archtectures? All these platforms now support __thread variables and ELF TLS on Linux. -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 15:20:08 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4FF4337B407 for ; Mon, 16 Jun 2003 15:20:08 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id A0A4A43FBD for ; Mon, 16 Jun 2003 15:20:07 -0700 (PDT) (envelope-from eischen@pcnet.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5GMK0Xh012314; Mon, 16 Jun 2003 18:20:00 -0400 (EDT) Date: Mon, 16 Jun 2003 18:20:00 -0400 (EDT) From: Daniel Eischen To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: Gareth Hughes cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 22:20:08 -0000 On Mon, 16 Jun 2003, Julian Elischer wrote: > > On Mon, 16 Jun 2003, Gareth Hughes wrote: > > > On Mon, 16 Jun 2003, Andy Ritger wrote: > > > > > > So from an OpenGL point of view, here are several alternatives that > > > I see for atleast the near term: > > > > > > - make NVIDIA's OpenGL implementation not thread-safe (just > > > use global data rather that thread-local data) > > > > > > - accept the performance hit of using pthread_getspecific() > > > on FreeBSD. From talking to other OpenGL engineers, > > > conservative estimates of the performance impact on > > > applications like viewperf range from 10% - 15%. I'd like > > > to quantify that, but certainly there will be a performance > > > penalty. > > > > And these are *very* conservative estimates -- you're essentially adding a > > function call into a path that is, on average, less than ten instructions > > per OpenGL API call, where the number of API calls per frame is upward of 3 I see this as a problem with the OpenGL API. You're trying to make something thread-safe that isn't by its nature. I would rather see OpenGL-MT with new interfaces that are by nature thread-safe. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 15:28:38 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B127B37B401 for ; Mon, 16 Jun 2003 15:28:38 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 122D643FAF for ; Mon, 16 Jun 2003 15:28:36 -0700 (PDT) (envelope-from eischen@pcnet.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5GMSVXh013736; Mon, 16 Jun 2003 18:28:32 -0400 (EDT) Date: Mon, 16 Jun 2003 18:28:31 -0400 (EDT) From: Daniel Eischen To: Gareth Hughes In-Reply-To: <2D32959E172B8F4D9B02F68266BE421401A6D7D1@mail-sc-3.nvidia.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Julian Elischer' cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 22:28:39 -0000 On Mon, 16 Jun 2003, Gareth Hughes wrote: > On Mon, 16 Jun 2003, Julian Elischer wrote: > > > > It wouldn't take much for us to give you a "application specific > > pointer" in the memory block that is pointed to by %gs. > > > > This would allow you to find your data with a single dereference > > for the N:N threads library.. > > > > %gs ----->[threadsystem-thread-specific-data] > > [ stuff ] > > [ ap-specific pointe ]---->[ your data ] > > [ ] > > > > BUT > > that would only work for the 1:1 threading library > > What's wrong with: > > %gs ----->[threadsystem-thread-specific-data ] > [ stuff ] > [libGL stuff (fixed size, known offset)] %gs is for the KSE in libkse. Multiple threads can run in the same KSE. > Or, better yet, to make sure no problems arise when you change the internals > of your data structures: > > %gs ----->[libGL stuff (16 words, say) ] > [threadsystem-thread-specific-data] > [ stuff ] > > You reserve the first 16 words of your thread data structure for us, and > we're done. At least when this library is being used. Again, %gs isn't per-thread; it's per-KSE. Plus, we're reserving TLS for one vendor/library. What happens when someone else comes along and wants the same thing? I'd much rather see someone push for a new OpenGL spec with better interfaces/APIs. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 15:44:26 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D936C37B401 for ; Mon, 16 Jun 2003 15:44:26 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 579DF43FAF for ; Mon, 16 Jun 2003 15:44:26 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 15:47:14 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 15:43:53 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7D8@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Daniel Eischen' , Julian Elischer Date: Mon, 16 Jun 2003 15:43:53 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 22:44:27 -0000 On Mon, 16 Jun 2003, Daniel Eischen wrote: > > I see this as a problem with the OpenGL API. You're trying > to make something thread-safe that isn't by its nature. > I would rather see OpenGL-MT with new interfaces that > are by nature thread-safe. Sorry, but this simply isn't going to happen. People smarter than you or I designed the API this way to allow for high performance implementations on a wide variety of platforms. There are very good reasons each API call doesn't take a context parameter. Imagine you have a bunch of threads, each rendering part of a scene (and thus, executing the same code). What you're suggesting amounts to the application having to track the current context in a thread-safe manner (using thread-local storage), and passing the current context into each and every OpenGL API routine. In complex scenes, made up of many millions of vertices, that can add up to a huge amount of overhead. Instead, the designers split the window-system specific part of the graphics library into its own module, and this module handles all the context management stuff (like creating, binding and deleting contexts). The OpenGL core API operates in conjuction with this window-system specifc library, but does not need to worry about the details of this side of things. Without going into a huge amount of detail, the fact that OpenGL is a state machine means things get very messy if the current context can change with every API call. The amount of validation required is prohibitive -- imagine if you were building a display list in one context, while rendering a scene with the other context, with interleaved API calls. This kind of thing prevents direct-to-hardware fastpaths. Many implementations of OpenGL (particularly the really good ones) write the data they receive from the application directly to a DMA buffer or, in the early days, directly into the graphics subsystem registers. Imagine if we have a full hardware implementation of the OpenGL machine -- a context switch probably means reading out the current values of all the state registers, saving them off somewhere, and reloading the new context's state. Not something you want to have to do on every API call you receive. A new graphics API is not the solution. As we've said, this is a well-studied and essentially solved problem. The ELF TLS standard is exactly that: a standard. It is supported by multiple operating systems on a wide variety of architectures, probably all the architectures that FreeBSD supports. Again, I urge you to look at Ulrich Drepper's document if you haven't done so already. -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 15:56:24 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D4A0D37B401 for ; Mon, 16 Jun 2003 15:56:24 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 482D543F3F for ; Mon, 16 Jun 2003 15:56:22 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 15:59:09 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 15:55:49 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7DB@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Daniel Eischen' Date: Mon, 16 Jun 2003 15:55:48 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Julian Elischer' cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 22:56:25 -0000 On Mon, 16 Jun 2003, Daniel Eischen wrote: > > Again, %gs isn't per-thread; it's per-KSE. Plus, we're reserving > TLS for one vendor/library. What happens when someone else comes > along and wants the same thing? I'd much rather see someone push > for a new OpenGL spec with better interfaces/APIs. I don't think there's a library out there that has the strict performance requirements that OpenGL does. Of course, if FreeBSD supported the ELF TLS standard, this point would be moot because applications and libraries would automatically get fast thread-local storage. If not, and another library really did need the same kind of fast TLS access, what's wrong with just allocating another static block after the libGL one? Your internal data structures would work fine, libGL would work fine because you haven't changed the location of its data block, and the new library would access its data directly. The only problem with this scheme is if you move the block, or change the way it is accessed, this would break binary compatibility. -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 16:06:57 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C51BB37B401 for ; Mon, 16 Jun 2003 16:06:57 -0700 (PDT) Received: from rwcrmhc11.attbi.com (rwcrmhc11.comcast.net [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5060243FCB for ; Mon, 16 Jun 2003 16:06:57 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (rwcrmhc11) with ESMTP id <2003061618031001300am14te>; Mon, 16 Jun 2003 18:03:11 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id LAA20228; Mon, 16 Jun 2003 11:03:10 -0700 (PDT) Date: Mon, 16 Jun 2003 11:03:09 -0700 (PDT) From: Julian Elischer To: Daniel Eischen In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: "Matthew N. Dodd" cc: Alexander Nedotsukov cc: threads@freebsd.org Subject: Re: nvidia OpenGL and lib{thr,kse} related crash X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 23:06:58 -0000 On Wed, 11 Jun 2003, Daniel Eischen wrote: > On Wed, 11 Jun 2003, Matthew N. Dodd wrote: > > > On Thu, 12 Jun 2003, Alexander Nedotsukov wrote: > > > Does some one know any nvidia developer to point them out on this fact? > > > > The NVIDIA drivers uses %gs for TLS. Ugghh! Can you be more specific about this? when you say "driver", do you mean a kernel driver or a userland program. I assume it's a userland pseudo-driver that links with pthreads to achieve its threads? Geez. why don't they use the per-thread-storage that the library provides? What makes them think that %gs is free? I think that linux will start using %gs for the TLS too as well soon. It's used for Thread-local-storage in the pthreads library too, so they trash each other's TLS. do they allocate LDT entries to back their own storage? > > That makes them pretty much useless in -current, especially since > libc_r is on the way out. > > One wonders why they use pthread_key_create() et al, and still > need TLS. > > > I suppose I should have mentioned this. > > Probably :-) > > -- > Dan Eischen > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 16:11:04 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AFAC237B401 for ; Mon, 16 Jun 2003 16:11:04 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 250F943F85 for ; Mon, 16 Jun 2003 16:11:04 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 16:13:38 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 16:10:18 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7DE@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Julian Elischer' Date: Mon, 16 Jun 2003 16:10:10 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de cc: Daniel Eischen cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 23:11:05 -0000 On Mon, 16 Jun 2003, Julian Elischer wrote: > > Both :-) > ELF TLS is a proposal and not yet (that I know of) part of > the standard. The latest copy of the System V generic ABI, found here: http://www.caldera.com/developers/gabi/ shows that the TLS stuff is indeed part of the specification now. Drepper states: One of the last additions of the generic ELF ABI was support for thread-local storage. on his homepage, just under the link to the processor-specific ELF TLS document: http://people.redhat.com/drepper/ -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 16:13:42 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D215237B401 for ; Mon, 16 Jun 2003 16:13:42 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0452543FD7 for ; Mon, 16 Jun 2003 16:13:40 -0700 (PDT) (envelope-from eischen@pcnet.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5GNDZXh019751; Mon, 16 Jun 2003 19:13:35 -0400 (EDT) Date: Mon, 16 Jun 2003 19:13:35 -0400 (EDT) From: Daniel Eischen To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: Gareth Hughes cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 23:13:43 -0000 On Mon, 16 Jun 2003, Julian Elischer wrote: > > On Mon, 16 Jun 2003, Gareth Hughes wrote: > > I'm not sure what you mean by this. Do you mean if you provide us with a > > magic function to get our thread-local data, we'd be non-portable? Or that > > because we use the ELF TLS stuff we're non-portable already? > > Both :-) > ELF TLS is a proposal and not yet (that I know of) part of the standard. > > It is however probably a good thing to support ELF TLS if it comes to be > a standard so I'm treating this as a "wakeup" but errno (for example) is > not something that is very time critical and most other uses of TLS are > not as time critical as your app appears to be. Usually this sort of > thing is done with a argument in the function calls pointing to the > local state, rather than using __thread (which is new). Right. It just seems wrong to me to be able to insert __thread after every global variable in a library and instantly make it thread-safe. > Anyone have recent numbers on times needed to change a segment > register? :-/ > It is sometimes slow to load segment registers on some > chips, so we try to not do it when we switch in userspace to > another thread (we have a pointer to the current CPU's structure and in > THAT we switch the "current thread" pointer. We COULD change things > around so that %gs points to a thread-specific data pointer and go > backwards to the "per CPU" structure, but what do we do when we I don't think we (libkse/libpthread) can do that. The %gs register has to point to the KSE mailbox so that we can clear km_curthread in one instruction. A thread can switch between KSEs so you can't atomically dereference and set curthread->kmbx->km_curthread. > have no thread running? (just the Userland thread scheduler is running > but has not selected a thread to run). I guess we could have a DUMMY > thread structure, but that suggests 2 loads of the segment register > for each contect switch in userland.. > (as opposed to 0). One to the dummy, and one to the next thread. > that could realy slow down all the other apps. So far we have avoided a dummy thread structure. We don't want to necessitate one. > It's an interesting problem. Luckily we are only just starting out > in the "kernel supported threads" game so we can still change things > relatively easily. We COULD make the C compiler generate > > > leal %gs:(TD_OFFSET) > %eax movl TLS(%eax),%eax > (or whatever the syntax is) I guess but it would generate code that > would refuse to work on lib_r and with the N:N threads. > > Also the compiled code seems to be adding its own LDT entries that would > be a bad idea if the threading system is doing that too. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 16:17:14 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7FD1737B401 for ; Mon, 16 Jun 2003 16:17:14 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id C60D943F93 for ; Mon, 16 Jun 2003 16:17:13 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 16:20:03 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 16:16:42 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7DF@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Julian Elischer' Date: Mon, 16 Jun 2003 16:16:42 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de cc: Daniel Eischen cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 23:17:14 -0000 On Mon, 16 Jun 2003, Julian Elischer wrote: > > can wine use OpenGL? > > Doesn't look like a clash to me. :-) I don't understand your point here... > what DOES make sense however is to support the __thread > concept, in a way that is efficient as it is not specific to one > rather esoteric application. I for one have no objection to making > sweeping changes if it looks as if __thread is going to solve problems > for large numbers of applications. > (though they may not be posix complient applications, it probably makes > sense to support it if there are enough of them breaking the same > rules.) Currently, given the performance requirements of OpenGL, our drivers are one of the first users of the new ELF TLS implementation on Linux, outside of libc itself. Given the fact that this is a general solution to a common problem, I strongly urge you to take it into consideration when working on your threading libraries. There are only going to be more users of __thread variables as time goes on -- you can imagine that many multithreaded user-space server applications will start using this soon, instead of the clunky pthread TLS interface. -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 16:31:25 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 148AB37B401 for ; Mon, 16 Jun 2003 16:31:25 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2EE7343FAF for ; Mon, 16 Jun 2003 16:31:24 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 16:34:11 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 16:30:50 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7E1@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Daniel Eischen' , Julian Elischer Date: Mon, 16 Jun 2003 16:30:49 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 23:31:25 -0000 On Mon, 16 Jun 2003, Daniel Eischen wrote: > > Right. It just seems wrong to me to be able to insert __thread > after every global variable in a library and instantly make > it thread-safe. Why, exactly? Surely, from a programming point of view, this is exactly what you want -- a transparent way to access your thread local data. -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 16:47:13 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2790237B401 for ; Mon, 16 Jun 2003 16:47:13 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9223343FB1 for ; Mon, 16 Jun 2003 16:47:10 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 16:49:58 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 16:46:37 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7E2@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Julian Elischer' Date: Mon, 16 Jun 2003 16:46:36 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Daniel Eischen' cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 23:47:13 -0000 On Mon, 16 Jun 2003, Julian Elischer wrote: > > I think that the problem is that the access method for TLS is dependent > on which library is used. > > [snip] > > The trouble is that each of these would require a differnt mechanism to > reach TLS and the compiler cannot know ahead of time which one to use. > > [snip] > > I may be wrong but I don't think it is a standard yet.. > especailly for the reason that we see here.. > It requires that the compiler know what threading library is in use. > > We could certainly implement efficient TLS code generation for each > library, but which one would be compiled in when you compile a .o file > that may be used with any library? Please read Ulrich Drepper's document, if you haven't done so already. You'll see that the general case of __thread variable access involves a function call to look up the variables address. There are optimizations to this access model, that allows one of the other three models to be used (ranging from a function call the first time a __thread variable is accessed, down to a single instruction per access). FreeBSD could trivially implement __thread variables with the General Dynamic model (involving a function call per access). Our driver uses the Local Exec model (single instruction per access) because GNU libc has an optimization on x86 that allows shared libraries to use this model, which is normally reserved for applications. The key thing is that they're still __thread variables, the access model depends on the compile time options used and what's available at runtime. Please, I urge you to read Drepper's document carefully. -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 16:59:39 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5A07237B401 for ; Mon, 16 Jun 2003 16:59:39 -0700 (PDT) Received: from sccrmhc11.attbi.com (sccrmhc11.comcast.net [204.127.202.55]) by mx1.FreeBSD.org (Postfix) with ESMTP id 551E143FAF for ; Mon, 16 Jun 2003 16:59:38 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc11) with ESMTP id <20030616230819011007thhbe>; Mon, 16 Jun 2003 23:08:19 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA22265; Mon, 16 Jun 2003 16:08:18 -0700 (PDT) Date: Mon, 16 Jun 2003 16:08:17 -0700 (PDT) From: Julian Elischer To: Gareth Hughes In-Reply-To: <2D32959E172B8F4D9B02F68266BE421401A6D7D3@mail-sc-3.nvidia.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: Daniel Eischen cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2003 23:59:39 -0000 On Mon, 16 Jun 2003, Gareth Hughes wrote: > > (that too is of course non portable too but.....) > > Wine uses %fs. > can wine use OpenGL? Doesn't look like a clash to me. :-) what DOES make sense however is to support the __thread concept, in a way that is efficient as it is not specific to one rather esoteric application. I for one have no objection to making sweeping changes if it looks as if __thread is going to solve problems for large numbers of applications. (though they may not be posix complient applications, it probably makes sense to support it if there are enough of them breaking the same rules.) From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 17:30:11 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1370037B401 for ; Mon, 16 Jun 2003 17:30:11 -0700 (PDT) Received: from sccrmhc13.attbi.com (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4C17443F75 for ; Mon, 16 Jun 2003 17:30:10 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc13) with ESMTP id <20030616220005016001jerte>; Mon, 16 Jun 2003 22:00:05 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id PAA21769; Mon, 16 Jun 2003 15:00:01 -0700 (PDT) Date: Mon, 16 Jun 2003 14:59:59 -0700 (PDT) From: Julian Elischer To: Gareth Hughes In-Reply-To: <2D32959E172B8F4D9B02F68266BE421401A6D7CD@mail-sc-3.nvidia.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: Daniel Eischen cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 00:30:11 -0000 On Mon, 16 Jun 2003, Gareth Hughes wrote: > On Mon, 16 Jun 2003, Andy Ritger wrote: > > > > So from an OpenGL point of view, here are several alternatives that > > I see for atleast the near term: > > > > - make NVIDIA's OpenGL implementation not thread-safe (just > > use global data rather that thread-local data) > > > > - accept the performance hit of using pthread_getspecific() > > on FreeBSD. From talking to other OpenGL engineers, > > conservative estimates of the performance impact on > > applications like viewperf range from 10% - 15%. I'd like > > to quantify that, but certainly there will be a performance > > penalty. > > And these are *very* conservative estimates -- you're essentially adding a > function call into a path that is, on average, less than ten instructions > per OpenGL API call, where the number of API calls per frame is upward of 3 > million (3 calls per vertex, over a million vertices for some Viewperf > benchmarks). The API was designed this way for a reason, and fast > thread-local storage is a fundamental part of a high performance > implementation. Unfortunatly you are making an assumption about the threads that you are only able to make in Linux "by chance" as they did something else for their TLS. I gather that you don't keep a local variable pointed to your local drawing context, but just use the 'global' style of programming but using %gs as a 'roll-your-own' version of 'local context'. Can you just grab %gs on Microsoft? I thought they were going to be using it for their TLS too. BTW Have you looked at the speed of that? At one time it was a lot slower to dereference things through %gs than it was to simply have a normal register allocated to the task (e.g. %esi) as would be done normally. I don't know if that is true with any modern machines though. HOWEVER.. as far as we know we still have %fs unused.... Maybe we could just switch registers :-) (that too is of course non portable too but.....) BTW what do you do for ia64/alpha/amd64 archtectures? > > -- > Gareth Hughes (gareth@nvidia.com) > OpenGL Developer, NVIDIA Corporation > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 17:30:13 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CCFF137B401 for ; Mon, 16 Jun 2003 17:30:13 -0700 (PDT) Received: from sccrmhc13.attbi.com (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id 26BF843FA3 for ; Mon, 16 Jun 2003 17:30:11 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc13) with ESMTP id <20030616235353016001m3q2e>; Mon, 16 Jun 2003 23:53:53 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA22610; Mon, 16 Jun 2003 16:53:52 -0700 (PDT) Date: Mon, 16 Jun 2003 16:53:51 -0700 (PDT) From: Julian Elischer To: Gareth Hughes In-Reply-To: <2D32959E172B8F4D9B02F68266BE421401A6D7E1@mail-sc-3.nvidia.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Daniel Eischen' cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 00:30:14 -0000 On Mon, 16 Jun 2003, Gareth Hughes wrote: > On Mon, 16 Jun 2003, Daniel Eischen wrote: > > > > Right. It just seems wrong to me to be able to insert __thread > > after every global variable in a library and instantly make > > it thread-safe. > > Why, exactly? Surely, from a programming point of view, this is > exactly what you want -- a transparent way to access your thread > local data. I think what Dan is saying is that there is a lot more to making a program thread-safe than just banging __thread before global variables. I think that it is sad that OpenGL has been implemented this way rather than passing an "our state" variable to each API call as is usually done. Because in the real world the market leader decides the rules and because Linux has decided to make the TLS hack standard, that means that many peolpe will use it. This means that the posix standard basically goes into the trash basket of history. If the lowest level calls took the 'current drawing context' (or whatever you call it) as an argument, then it wouldn't matter so much how it was originally derived. An occasional function call to derive it would be ok. it's the fact that you decide to derive it from first principles every time rather than caching it that is the problem. __thread can easily be derived in an efficient manner in a library independent manner using an entrypoint, but making it important to do it in 1 machine instruction is not possible in a library independent manner. I think that this is a BAD THING even if we end up having to do it somehow.. BTW in my previous comment re: wine and OpenGL does wine link with openGL? If not I'd just go for using %fs for now :-) From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 17:30:14 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9F94037B401 for ; Mon, 16 Jun 2003 17:30:14 -0700 (PDT) Received: from sccrmhc13.attbi.com (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id DFBB443FA3 for ; Mon, 16 Jun 2003 17:30:13 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc13) with ESMTP id <20030617000234016001mih4e>; Tue, 17 Jun 2003 00:02:34 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id RAA22696; Mon, 16 Jun 2003 17:02:29 -0700 (PDT) Date: Mon, 16 Jun 2003 17:02:28 -0700 (PDT) From: Julian Elischer To: Gareth Hughes In-Reply-To: <2D32959E172B8F4D9B02F68266BE421401A6D7E2@mail-sc-3.nvidia.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Daniel Eischen' cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 00:30:15 -0000 On Mon, 16 Jun 2003, Gareth Hughes wrote: > On Mon, 16 Jun 2003, Julian Elischer wrote: > > > > I think that the problem is that the access method for TLS is dependent > > on which library is used. > > > > [snip] > > > > The trouble is that each of these would require a differnt mechanism to > > reach TLS and the compiler cannot know ahead of time which one to use. > > > > [snip] > > > > I may be wrong but I don't think it is a standard yet.. > > especailly for the reason that we see here.. > > It requires that the compiler know what threading library is in use. > > > > We could certainly implement efficient TLS code generation for each > > library, but which one would be compiled in when you compile a .o file > > that may be used with any library? > > Please read Ulrich Drepper's document, if you haven't done so already. > You'll see that the general case of __thread variable access involves > a function call to look up the variables address. There are > optimizations to this access model, that allows one of the other three > models to be used (ranging from a function call the first time a > __thread variable is accessed, down to a single instruction per > access). FreeBSD could trivially implement __thread variables with > the General Dynamic model (involving a function call per access). > Our driver uses the Local Exec model (single instruction per access) > because GNU libc has an optimization on x86 that allows shared > libraries to use this model, which is normally reserved for > applications. The key thing is that they're still __thread variables, > the access model depends on the compile time options used and what's > available at runtime. Please, I urge you to read Drepper's document > carefully. > I have read most of it already. What I'm saying is that we can and probably should implement TLS using the general model to provide TLS at "sane" speed, (e.g. 5 instructions) but that I don't think we can implement the "1 instruction" version that you are asking for without breaking the binary compatibility that we currently have between our 3 pthread libraries. We can currently switch libraries between 3 very different threads libraries without recompiling the app or any other libraries involved. in fact we have a config file to the loader that specifies which one to use at run time **Per application**. Without using an entrypoint (or maybe self modifying code) (*EEK!*) I don't see how we can do it and keep that *Very useful* functionality. From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 17:34:21 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 36D9B37B401 for ; Mon, 16 Jun 2003 17:34:21 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0885F43F75 for ; Mon, 16 Jun 2003 17:34:19 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 17:37:07 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 17:33:46 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7E4@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Julian Elischer' Date: Mon, 16 Jun 2003 17:33:45 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Daniel Eischen' cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 00:34:21 -0000 On Mon, 16 Jun 2003, Julian Elischer wrote: > > I think what Dan is saying is that there is a lot more to making > a program thread-safe than just banging __thread before global > variables. No question. But, the __thread keyword is the best way to access thread local data that I've seen. > I think that it is sad that OpenGL has been implemented this way rather > than passing an "our state" variable to each API call as is usually > done. Because in the real world the market leader decides the rules > and because Linux has decided to make the TLS hack standard, that means > that many peolpe will use it. This means that the posix standard > basically goes into the trash basket of history. Please have a little faith, and trust me when I say some of the top people in the field of computer graphics, each with (now) decades of experience with software and hardware implementations of the 3D graphics pipeline, knew what they were doing when they designed the OpenGL API. It didn't become the only non-Microsoft graphics API by accident (and at the time of its release, every vendor basically had their own API). I've been working on OpenGL implementations for years, so please trust me when I say it's done this way for very, very good reasons. It seems like it'll take a lot longer for me to convince you of this, so please just take my word for it. > If the lowest level calls took the 'current drawing context' (or > whatever you call it) as an argument, then it wouldn't matter so much > how it was originally derived. An occasional function call to > derive it > would be ok. it's the fact that you decide to derive it from first > principles every time rather than caching it that is the problem. I have no idea what you're trying to say here. It's critical that an implementation of OpenGL be able to "cache away" the current context. Context switching is a very heavy-weight operation in OpenGL. There are very good reasons why this is so, as I touched upon in an earlier email. Here's another example: what if every time you switched contexts, you had to free up your video memory heap, to make room for the back/depth/stencil buffers and textures for the new context? Do you want to do this after you render every triangle? I can go on and on about why the design decisions were made. > __thread can easily be derived in an efficient manner in a library > independent manner using an entrypoint, but making it > important to do it in 1 machine instruction is not possible in a > library independent manner. I think that this is a BAD THING > even if we end up having to do it somehow.. If FreeBSD supported __thread variables (ELF TLS), then the NVIDIA driver would use them, period. If you can't provide a single instruction access model like Linux and other platforms can, then you'll simply have a slower implementation of OpenGL on your platform. It's disappointing that you're designing a system that can't support the fastest TLS access model, but, so be it. > BTW in my previous comment re: wine and OpenGL > does wine link with openGL? If not I'd just go for using %fs > for now :-) Yes, Wine uses OpenGL. -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 17:35:19 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C639837B409 for ; Mon, 16 Jun 2003 17:35:19 -0700 (PDT) Received: from rwcrmhc11.attbi.com (rwcrmhc11.comcast.net [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2D70343F75 for ; Mon, 16 Jun 2003 17:35:19 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (rwcrmhc11) with ESMTP id <2003061623344601300anepke>; Mon, 16 Jun 2003 23:34:46 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA22467; Mon, 16 Jun 2003 16:34:45 -0700 (PDT) Date: Mon, 16 Jun 2003 16:34:44 -0700 (PDT) From: Julian Elischer To: Gareth Hughes In-Reply-To: <2D32959E172B8F4D9B02F68266BE421401A6D7DB@mail-sc-3.nvidia.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Daniel Eischen' cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 00:35:20 -0000 On Mon, 16 Jun 2003, Gareth Hughes wrote: > On Mon, 16 Jun 2003, Daniel Eischen wrote: > > > > Again, %gs isn't per-thread; it's per-KSE. Plus, we're reserving > > TLS for one vendor/library. What happens when someone else comes > > along and wants the same thing? I'd much rather see someone push > > for a new OpenGL spec with better interfaces/APIs. > I think that the problem is that the access method for TLS is dependent on which library is used. In the multiplexd thread library, %gs points to the current KSE (kernel Schedulable entity) (think virtual CPU), and THAT has a pointer to the current thread. with thousands of threads going in and out of runnablility each tick (without notifying the kernel) we don't want to keep changing %gs in userland as that's slow. (there are plenty of apps that have MANY threads). These threads run entirely in userland and control switches between them at an alarming rate.. Anything that slows down context switches has a bad effect on the speed that these programs (e.g. some java implementations and programs) run. In the 1:1 (or N:N) thread library the context times are considerably larger as there is kernel interaction on each and every context switch. Overhead from switching %gs in this library would probably be buried in the noise. It would probably be an acceptable solution. In the single-streamed pthreads library (libc_r) that is currently in use, %gs is no used so your code is not colliding, but then there isn't explicit support for __thread though (curthread->TLS) would be all that is required since it is not multithreaded from a real perspective. The trouble is that each of these would require a differnt mechanism to reach TLS and the compiler cannot know ahead of time which one to use. > I don't think there's a library out there that has the strict > performance requirements that OpenGL does. Of course, if FreeBSD > supported the ELF TLS standard. I may be wrong but I don't think it is a standard yet.. especailly for the reason that we see here.. It requires that the compiler know what threading library is in use. We could certainly implement efficient TLS code generation for each library, but which one would be compiled in when you compile a .o file that may be used with any library? > this point would be moot because > applications and libraries would automatically get fast > thread-local storage. If not, and another library really did need > the same kind of fast TLS access, what's wrong with just allocating > another static block after the libGL one? Your internal data > structures would work fine, libGL would work fine because you > haven't changed the location of its data block, and the new library > would access its data directly. The only problem with this scheme > is if you move the block, or change the way it is accessed, this > would break binary compatibility. A single library is not going to get It's own block in the system thread descriptor, but it makes great sense to allocate a pointer there for the system to support TLS in a uniform manner for many applications. if we place a pointer (actually a couple) there for the .tbss and .tdata segments then we can get to those segments quickly (though possibly requiring 2 instructions) (I'm not a x86 assembler guru) I wouldn't think of adding such for a single app or library, but support for ELF TLS is probably of high enough importance that it is worth while.. as I said though.. "which library gets to support it?" the one with one indirection? the one with 2 indirections or the one with no indirections? The trick is that because it is a pre-link-time thing, it cannot handle the case where there are more than one binary interface to the threads.. That is why it is not in the posix threading moddle and probably never really will be, except in a slow form. (BTW have you looked at the speed of function calls on modern PCs, I think you'll be surprised). From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 18:12:04 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3E63137B401 for ; Mon, 16 Jun 2003 18:12:04 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7074E43F85 for ; Mon, 16 Jun 2003 18:12:03 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 18:14:51 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 18:11:30 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7E6@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Julian Elischer' Date: Mon, 16 Jun 2003 18:11:30 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Daniel Eischen' cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 01:12:04 -0000 On Mon, 16 Jun 2003, Julian Elischer wrote: > > What I'm saying is that we can and probably should implement TLS using > the general model to provide TLS at "sane" speed, (e.g. 5 instructions) > but that I don't think we can implement the "1 instruction" version that > you are asking for without breaking the binary compatibility that we > currently have between our 3 pthread libraries. We can currently switch > libraries between 3 very different threads libraries without recompiling > the app or any other libraries involved. in fact we have a config file > to the loader that specifies which one to use at run time **Per > application**. Without using an entrypoint (or maybe self modifying > code) (*EEK!*) I don't see how we can do it and keep that *Very useful* > functionality. Let me make our position crystal clear: If FreeBSD support ELF TLS and __thread variables in ANY form, our driver will use this support. If the best you can do is a function call per access, so be it. It doesn't sound like there are any other options, given the fact that you ship with three different thread libraries. -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 18:18:41 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BAD8D37B401 for ; Mon, 16 Jun 2003 18:18:41 -0700 (PDT) Received: from mx2.mail.ru (mx2.mail.ru [194.67.23.22]) by mx1.FreeBSD.org (Postfix) with ESMTP id 07B3B43FCB for ; Mon, 16 Jun 2003 18:18:41 -0700 (PDT) (envelope-from kabaev@mail.ru) Received: from [151.203.235.42] (port=55759 helo=kan.dnsalias.net) by mx2.mail.ru with esmtp id 19S57O-000JCo-00; Tue, 17 Jun 2003 05:18:39 +0400 Received: from kan.dnsalias.net (ak03@localhost [127.0.0.1]) by kan.dnsalias.net (8.12.9/8.12.9) with ESMTP id h5H1IaTI090294; Mon, 16 Jun 2003 21:18:36 -0400 (EDT) (envelope-from kan@kan.dnsalias.net) Received: (from kan@localhost) by kan.dnsalias.net (8.12.9/8.12.9/Submit) id h5H1IZb3090293; Mon, 16 Jun 2003 21:18:35 -0400 (EDT) Date: Mon, 16 Jun 2003 21:18:35 -0400 From: Alexander Kabaev To: Gareth Hughes Message-Id: <20030616211835.10d0fba4.kabaev@mail.ru> In-Reply-To: <2D32959E172B8F4D9B02F68266BE421401A6D7E6@mail-sc-3.nvidia.com> References: <2D32959E172B8F4D9B02F68266BE421401A6D7E6@mail-sc-3.nvidia.com> X-Mailer: Sylpheed version 0.9.0claws2 (GTK+ 1.2.10; i386-portbld-freebsd5.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam: Not detected cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Julian Elischer' cc: Andy Ritger cc: 'Daniel Eischen' Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 01:18:42 -0000 On Mon, 16 Jun 2003 18:11:30 -0700 Gareth Hughes wrote: > > If FreeBSD support ELF TLS and __thread variables in ANY form, our > driver will use this support. If the best you can do is a function > call per access, so be it. It doesn't sound like there are any other > options, given the fact that you ship with three different thread > libraries. Three different libraries can attempt to coordinate and reserve exactly the same TLS segment size. From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 18:35:59 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0DFB537B401 for ; Mon, 16 Jun 2003 18:35:59 -0700 (PDT) Received: from mx5.mail.ru (mx5.mail.ru [194.67.23.25]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4AFB043FD7 for ; Mon, 16 Jun 2003 18:35:58 -0700 (PDT) (envelope-from kabaev@mail.ru) Received: from [151.203.235.42] (port=55800 helo=kan.dnsalias.net) by mx5.mail.ru with esmtp id 19S5O8-0007zj-00; Tue, 17 Jun 2003 05:35:56 +0400 Received: from kan.dnsalias.net (ak03@localhost [127.0.0.1]) by kan.dnsalias.net (8.12.9/8.12.9) with ESMTP id h5H1ZtTI090395; Mon, 16 Jun 2003 21:35:55 -0400 (EDT) (envelope-from kan@kan.dnsalias.net) Received: (from kan@localhost) by kan.dnsalias.net (8.12.9/8.12.9/Submit) id h5H1ZsKl090394; Mon, 16 Jun 2003 21:35:54 -0400 (EDT) Date: Mon, 16 Jun 2003 21:35:54 -0400 From: Alexander Kabaev To: Gareth Hughes Message-Id: <20030616213554.14f45120.kabaev@mail.ru> In-Reply-To: <2D32959E172B8F4D9B02F68266BE421401A6D7DE@mail-sc-3.nvidia.com> References: <2D32959E172B8F4D9B02F68266BE421401A6D7DE@mail-sc-3.nvidia.com> X-Mailer: Sylpheed version 0.9.0claws2 (GTK+ 1.2.10; i386-portbld-freebsd5.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam: Not detected cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Julian Elischer' cc: Andy Ritger cc: Daniel Eischen Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 01:35:59 -0000 On Mon, 16 Jun 2003 16:10:10 -0700 Gareth Hughes wrote: > On Mon, 16 Jun 2003, Julian Elischer wrote: > > > > Both :-) > > ELF TLS is a proposal and not yet (that I know of) part of > > the standard. > > The latest copy of the System V generic ABI, found here: > > http://www.caldera.com/developers/gabi/ > > shows that the TLS stuff is indeed part of the specification > now. Drepper states: > > One of the last additions of the generic ELF ABI was > support for thread-local storage. > > on his homepage, just under the link to the processor-specific > ELF TLS document: > > http://people.redhat.com/drepper/ > That was very nice of them to develop a "standard" which penalises each and every threads implementation except pure kernel-based threads Linux happens to implement now. Said that, I think FreeBSD will have to follow the suit sooner or later. When the mass of Linux software using TLS reaches certain level, we'll be unable to ignore its requirements. It is sad though that we'll have to give up a some of our performance optimizations. From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 18:37:43 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E4A2A37B401 for ; Mon, 16 Jun 2003 18:37:43 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 53EBB43FD7 for ; Mon, 16 Jun 2003 18:37:43 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 18:40:31 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 18:37:11 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7E7@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Julian Elischer' Date: Mon, 16 Jun 2003 18:37:10 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Daniel Eischen' cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 01:37:44 -0000 On Mon, 16 Jun 2003, Julian Elischer wrote: > > I'm not making comments about their qualifications in graphics, > just that it's sad when teh threading interface is distorted by graphics > people.. In effect by insisting on having the TLS values accessible > at the lowest high-performace parts, they are excerting "un-natural" > pressure on the development of threads. :-/ The design of the ELF TLS spec had nothing to do with OpenGL or other graphics people. When this support was added to the Linux C library and toolchain, we started using it because it met our requirements for high performance thread-local storage. Granted, we worked with the GNU libc developers to iron out a few issues, but we had nothing to do with the specification itself. The ELF TLS spec was designed to meet the needs of a class of applications, which OpenGL happens to fall into. Fast thread-local storage is a good thing in general, beyond the scope of 3D graphics alone. > I'm saying that if it were don like this: > > __thread local_context_t *lc; > medium_level_OpenGL_function() > { > int linestodo=1000; > local_context_t *drawing_context; > int i; > > drawing_context = lc; > for (i = linestodo; i; i++) { > OpenGL_Low_level_Thingy(drawing_context, arg1, arg2); > } > > then the performance of TLS wouldn't be so crucial; > It would still be relatively ok (maybe 5 instructions) > but it wouldn't have to be 1 instruction > > In fact if they were inpplemented in the following way: > > __inline OpenGL_Low_level_Thingy(local_context_t *drawing_context, > arg1, arg2) > { > __asm "blah blah " /* load args to known regs */ > call library entrypoint /* with args in known regs...*/ > } > > you would have the fastest version of all without > any requirement for making the TLS so specialised. (purely register > transfer) Umm, you clearly don't understand what I've been talking about. Upon entry into libGL, i.e. when an OpenGL API entrypoint is called by the application, things like the current context or current dispatch table are fetched from thread-local storage. Internally, we pass these pointers around as required. So, we might have something like this: void glBegin(GLenum mode) { // Grab the current dispatch pointer from TLS __GLdispatch *dispatch = GET_CURRENT_DISPATCH(); // Call into the driver backend dispatch->Begin(mode) } or, in x86 assembly: glBegin: mov %gs:__gl_dispatch@ntpoff, %eax jmp *__begin_offset(%eax) (note that Andy mentioned this example in his original email) This would jump to a function inside the driver like this: void __internal_Begin(GLenum mode) { __GLcontext *ctx = GET_CURRENT_CONTEXT(); do_something(ctx, ...); do_something_else(ctx, ...); // and so on } Once we're inside the driver, we know what the current context or other thread-local variable's value is. Two critical points: 1) We have to fetch the value from TLS at least once per entry into the driver. 2) Some of the driver backend functions are very small, typically the more performance critical it is the smaller it is. In general, you want to avoid things like pthread_getspecific() inside the dispatch layer and your 6-instruction implementation of glColor4f or glNormal3f (which can be called millions of times per frame). > I have no intention of wanting yuo to context switch to > another thread.. I'm just saying that it's a pity you don't just > go teh route that other libraries have and make that time critical > fucntions just have the value at hand already. If the time critical function is a 6-instruction function at the top-level of the API (that is, called directly from the dispatch layer), how do you get this value other than looking it up out of thread-local storage? Caching that variable in TLS with a fast access mechanism qualifies as "keeping the value at hand" in my books. > No, I think you misunderstand.. > > A single thread would always use the same context. > I'm not saying otherwise.. > I'm just wishing that you would keep the value of its address > aroundin a local stack variable a bit more instead of deriving it > with %gs all the time. It is, once you've looked it up. Problem is, if the only work you do inside the library is copy three floats off the stack (parameters to the GL API call) into a DMA buffer, set a bit in a bitmask and return, the time spent accessing the current context becomes a large percentage of the time you spend in the library for that call, period. Understand? > If the leaf functions on OpenGL were to be implemented with > asm interfaces (you said they were hand optimised anyhow) > and the callers would cache the drawing context pointer in a local > register, then My ability to give you a TLS pointer in > > > getTLS: lea %eax,%gs(mumble) > movl mumble(%eax), %eax > ret > > would be fast enough as the cost of the extra function call would be > amortised over many low-level calls. > (actually it'd probably be faster than what you have now I think) There is no caller of these functions. In the fast paths, there is no function call, once you get inside the library. That's the whole point. 1) Application calls OpenGL function. 2) OpenGL API dispatch function looks up the dispatch pointer from TLS, and jumps through it. 2 insns. 3) Driver function looks up the current context out of TLS, copies some data into a buffer, and returns. Maybe 6 insns or so. 4) Application continues on. The cost of TLS access becomes significant when the driver is doing less than a dozen non-TLS-lookup instructions for the important API calls. -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 18:44:25 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7D48E37B401 for ; Mon, 16 Jun 2003 18:44:25 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id D603243FBF for ; Mon, 16 Jun 2003 18:44:24 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 18:47:11 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 18:43:51 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7E8@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Alexander Kabaev' Date: Mon, 16 Jun 2003 18:43:50 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Julian Elischer' cc: Andy Ritger cc: Daniel Eischen Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 01:44:25 -0000 On Mon, 16 Jun 2003, Alexander Kabaev wrote: > > That was very nice of them to develop a "standard" which penalises each > and every threads implementation except pure kernel-based threads Linux > happens to implement now. > > Said that, I think FreeBSD will have to follow the suit sooner or later. > When the mass of Linux software using TLS reaches certain level, we'll > be unable to ignore its requirements. It is sad though that we'll have > to give up a some of our performance optimizations. No, this is simply not true. If your thread library implementation is that much better than anything else out there, simply support the dynamic access models (General Dynamic and Local Dynamic) only. Both of these models require at least one function call to access __thread variables (the former requiring one per access, while the latter is one for the first __thread variable accessed in a given function). This will work just fine, no matter how you've implemented your threading library. One of the benefits of the Linux implementation is that it allows the static TLS access methods (Initial Exec and Local Exec) to be used. If your implementation fundamentally can't support these access models, then just don't support them. -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 18:55:46 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AF8AC37B401 for ; Mon, 16 Jun 2003 18:55:46 -0700 (PDT) Received: from wall.polstra.com (wall-gw.polstra.com [206.213.73.130]) by mx1.FreeBSD.org (Postfix) with ESMTP id CB52443FCB for ; Mon, 16 Jun 2003 18:55:45 -0700 (PDT) (envelope-from jdp@polstra.com) Received: from strings.polstra.com (strings.polstra.com [206.213.73.20]) by wall.polstra.com (8.12.3p2/8.12.3) with ESMTP id h5H1tjPL045108 for ; Mon, 16 Jun 2003 18:55:45 -0700 (PDT) (envelope-from jdp@polstra.com) Message-ID: X-Mailer: XFMail 1.5.4 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 Date: Mon, 16 Jun 2003 18:55:44 -0700 (PDT) From: John Polstra To: freebsd-threads@freebsd.org X-Bogosity: No, tests=bogofilter, spamicity=0.497234, version=0.11.2 Subject: getcontext() for a different thread X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 01:55:47 -0000 I've been toying with the idea of making Modula-3 use FreeBSD's native threads library. One thing that would be required is something like getcontext() that could be called to get the context of a different thread. That is needed by the garbage collector, because it has to examine all of the threads' registers in case some of them contain pointers to objects that need to be traced. Are there any major obstacles to implementing this? I see that getcontext() in the kernel already takes a "struct thread *" argument. But it calls get_mcontext(), which (at least on the i386) references "curthread" a couple of times. Is that really necessary, or could "td" be used in place of "curthread"? An other-thread getcontext would have to be passed a kernel thread handle of some sort. Is there currently anything like that which is exposed to userland? An other-thread getcontext would have to do some security checks to make sure you weren't trying to read the register of a thread that didn't belong to you. Anything else? John -- John Polstra John D. Polstra & Co., Inc. Seattle, Washington USA "Two buttocks cannot avoid friction." -- Malawi saying From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 19:02:32 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B604837B401 for ; Mon, 16 Jun 2003 19:02:32 -0700 (PDT) Received: from mx3.mail.ru (mx3.mail.ru [194.67.23.23]) by mx1.FreeBSD.org (Postfix) with ESMTP id F372543F3F for ; Mon, 16 Jun 2003 19:02:31 -0700 (PDT) (envelope-from kabaev@mail.ru) Received: from [151.203.235.42] (port=55934 helo=kan.dnsalias.net) by mx3.mail.ru with esmtp id 19S5nq-0003tX-00; Tue, 17 Jun 2003 06:02:30 +0400 Received: from kan.dnsalias.net (ak03@localhost [127.0.0.1]) by kan.dnsalias.net (8.12.9/8.12.9) with ESMTP id h5H22STI090577; Mon, 16 Jun 2003 22:02:29 -0400 (EDT) (envelope-from kan@kan.dnsalias.net) Received: (from kan@localhost) by kan.dnsalias.net (8.12.9/8.12.9/Submit) id h5H22SXV090576; Mon, 16 Jun 2003 22:02:28 -0400 (EDT) Date: Mon, 16 Jun 2003 22:02:28 -0400 From: Alexander Kabaev To: Gareth Hughes Message-Id: <20030616220228.49672fc7.kabaev@mail.ru> In-Reply-To: <2D32959E172B8F4D9B02F68266BE421401A6D7E8@mail-sc-3.nvidia.com> References: <2D32959E172B8F4D9B02F68266BE421401A6D7E8@mail-sc-3.nvidia.com> X-Mailer: Sylpheed version 0.9.0claws2 (GTK+ 1.2.10; i386-portbld-freebsd5.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam: Not detected cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Julian Elischer' cc: Andy Ritger cc: Daniel Eischen Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 02:02:33 -0000 On Mon, 16 Jun 2003 18:43:50 -0700 Gareth Hughes wrote: > No, this is simply not true. > > If your thread library implementation is that much better than > anything else out there, simply support the dynamic access models > (General Dynamic and Local Dynamic) only. You do not understand. Whether or not it is better only time will tell. ELF TLS standard is defined in a way, that only a _specific_ threads implementation can reap full benefits, most other are penalized by the segment access reloads on every thread context switch. > > One of the benefits of the Linux implementation is that it allows > the static TLS access methods (Initial Exec and Local Exec) to be > used. If your implementation fundamentally can't support these > access models, then just don't support them. So the choice is either to pay penalty implementing TLS through callable entry, or by making context switches more expensive than they have to be. I am obviously not thrilled with the standard, but I guess I'll have live with it since I have nothing better to offer on my own anyway. From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 19:23:51 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E9D8A37B401 for ; Mon, 16 Jun 2003 19:23:51 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 208B743FD7 for ; Mon, 16 Jun 2003 19:23:51 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Mon, 16 Jun 2003 19:26:37 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 19:23:17 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7EA@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Alexander Kabaev' Date: Mon, 16 Jun 2003 19:23:15 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Julian Elischer' cc: Andy Ritger cc: Daniel Eischen Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 02:23:52 -0000 On Mon, 16 Jun 2003 Alexander Kabaev wrote: > > You do not understand. Whether or not it is better only time will tell. > ELF TLS standard is defined in a way, that only a _specific_ threads > implementation can reap full benefits, most other are penalized by the > segment access reloads on every thread context switch. > > [snip] > > So the choice is either to pay penalty implementing TLS through callable > entry, or by making context switches more expensive than they have to > be. > > I am obviously not thrilled with the standard, but I guess I'll have > live with it since I have nothing better to offer on my own anyway. To save you the trouble, I'll copy the relevant information out of Drepper's document (sections 3.4.2, 4.1.2, 4.2.2). C code: extern __thread int x; &x; Assembly code, General Dynamic access model: leal x@tlsgd(%ebx),%eax call __tls_get_addr@plt C code: static __thread int x1; static __thread int x2; &x1; &x2; Assembly code, Local Dynamic access model: leal x1@tlsldm(%ebx),%eax call __tls_get_addr@plt ... leal x1@dtpoff(%eax),%edx ... leal x2@dtpoff(%eax),%edx I don't see any segment register use there, do you? That's the implementation of the dynamic TLS access models on x86 (specifically, the GNU variants, not the Sun variants). You implement __tls_get_addr() however you need to. Only the static access models require the use of %gs. If you don't want to use %gs, then go right ahead and only implement the dynamic access models. __thread variable access will be slower on FreeBSD in some cases (those that could potentially use the static access models), but you're free to implement whatever kind of thread library you want, including one that does not touch nor use the %gs segment register. -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 19:42:47 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7071337B401 for ; Mon, 16 Jun 2003 19:42:47 -0700 (PDT) Received: from mx3.mail.ru (mx3.mail.ru [194.67.23.23]) by mx1.FreeBSD.org (Postfix) with ESMTP id B926843F93 for ; Mon, 16 Jun 2003 19:42:46 -0700 (PDT) (envelope-from kabaev@mail.ru) Received: from [151.203.235.42] (port=56024 helo=kan.dnsalias.net) by mx3.mail.ru with esmtp id 19S6Qn-000Oie-00; Tue, 17 Jun 2003 06:42:45 +0400 Received: from kan.dnsalias.net (ak03@localhost [127.0.0.1]) by kan.dnsalias.net (8.12.9/8.12.9) with ESMTP id h5H2giTI090843; Mon, 16 Jun 2003 22:42:44 -0400 (EDT) (envelope-from kan@kan.dnsalias.net) Received: (from kan@localhost) by kan.dnsalias.net (8.12.9/8.12.9/Submit) id h5H2ghMW090842; Mon, 16 Jun 2003 22:42:43 -0400 (EDT) Date: Mon, 16 Jun 2003 22:42:43 -0400 From: Alexander Kabaev To: Gareth Hughes Message-Id: <20030616224243.74f151d8.kabaev@mail.ru> In-Reply-To: <2D32959E172B8F4D9B02F68266BE421401A6D7EA@mail-sc-3.nvidia.com> References: <2D32959E172B8F4D9B02F68266BE421401A6D7EA@mail-sc-3.nvidia.com> X-Mailer: Sylpheed version 0.9.0claws2 (GTK+ 1.2.10; i386-portbld-freebsd5.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam: Not detected cc: threads@freebsd.org Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 02:42:47 -0000 On Mon, 16 Jun 2003 19:23:15 -0700 Gareth Hughes wrote: > > To save you the trouble, I'll copy the relevant information out > of Drepper's document (sections 3.4.2, 4.1.2, 4.2.2). > What on Earth gives you an idea I haven't read it several times already? I'll spare you the trouble copying the code samples, please restrict yourself to section numbers only if you feel so inclined. > > I don't see any segment register use there, do you? > You see functions calls, do you? Penalty choice #2 on my list. That's the > implementation of the dynamic TLS access models on x86 > (specifically, the GNU variants, not the Sun variants). You > implement __tls_get_addr() however you need to. Only the static > access models require the use of %gs. Here you go, penalty choice #1. We are getting nowhere with this discussion. Sparing each others time by terminating it does show some promise though. From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 20:40:07 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CAB2837B401 for ; Mon, 16 Jun 2003 20:40:07 -0700 (PDT) Received: from rwcrmhc11.attbi.com (rwcrmhc11.comcast.net [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5337C43F85 for ; Mon, 16 Jun 2003 20:40:07 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (rwcrmhc11) with ESMTP id <2003061701334601300aoam4e>; Tue, 17 Jun 2003 01:33:47 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id SAA23290; Mon, 16 Jun 2003 18:33:46 -0700 (PDT) Date: Mon, 16 Jun 2003 18:33:45 -0700 (PDT) From: Julian Elischer To: Alexander Kabaev In-Reply-To: <20030616211835.10d0fba4.kabaev@mail.ru> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: Gareth Hughes cc: 'Daniel Eischen' cc: Andy Ritger Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 03:40:08 -0000 On Mon, 16 Jun 2003, Alexander Kabaev wrote: > On Mon, 16 Jun 2003 18:11:30 -0700 > Gareth Hughes wrote: > > > > > If FreeBSD support ELF TLS and __thread variables in ANY form, our > > driver will use this support. If the best you can do is a function > > call per access, so be it. It doesn't sound like there are any other > > options, given the fact that you ship with three different thread > > libraries. > Three different libraries can attempt to coordinate and reserve exactly > the same TLS segment size. It's not that the segents need to be teh same it's that we are already using %gs in 2 of them, and that they point to different things in each library :-/ We'll see what we can do.. From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 22:28:14 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5043737B401 for ; Mon, 16 Jun 2003 22:28:14 -0700 (PDT) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id AB20443F3F for ; Mon, 16 Jun 2003 22:28:13 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2injaq3.dialup.mindspring.com ([165.121.171.67] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19S90t-0006ny-00; Mon, 16 Jun 2003 22:28:12 -0700 Message-ID: <3EEEA6A9.5F8D60C3@mindspring.com> Date: Mon, 16 Jun 2003 22:27:05 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Gareth Hughes References: <2D32959E172B8F4D9B02F68266BE421401A6D7CD@mail-sc-3.nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a40317d7a3f931e2f42af2114c3122eab0a8438e0f32a48e08350badd9bab72f9c350badd9bab72f9c cc: threads@freebsd.org cc: zander@mail.minion.de cc: Daniel Eischen cc: Andy Ritger Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 05:28:14 -0000 Gareth Hughes wrote: > On Mon, 16 Jun 2003, Andy Ritger wrote: > > So from an OpenGL point of view, here are several alternatives that > > I see for atleast the near term: > > > > - make NVIDIA's OpenGL implementation not thread-safe (just > > use global data rather that thread-local data) > > > > - accept the performance hit of using pthread_getspecific() > > on FreeBSD. From talking to other OpenGL engineers, > > conservative estimates of the performance impact on > > applications like viewperf range from 10% - 15%. I'd like > > to quantify that, but certainly there will be a performance > > penalty. > > And these are *very* conservative estimates -- you're essentially adding a > function call into a path that is, on average, less than ten instructions > per OpenGL API call, where the number of API calls per frame is upward of 3 > million (3 calls per vertex, over a million vertices for some Viewperf > benchmarks). The API was designed this way for a reason, and fast > thread-local storage is a fundamental part of a high performance > implementation. What do you do on systems where you can't grab the %gs register and use it for whatever you want, because it's in use for something else? The pthread_getspecific() could probably be made into an inline, at the very least, and could potentially be made to lazy-bind %gs to the evil use to which it's currently being put. -- Terry From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 22:35:07 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B440237B401 for ; Mon, 16 Jun 2003 22:35:07 -0700 (PDT) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2C0D243F75 for ; Mon, 16 Jun 2003 22:35:07 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2injaq3.dialup.mindspring.com ([165.121.171.67] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19S97X-00000h-00; Mon, 16 Jun 2003 22:35:04 -0700 Message-ID: <3EEEA846.DD3F3C7B@mindspring.com> Date: Mon, 16 Jun 2003 22:33:58 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Gareth Hughes References: <2D32959E172B8F4D9B02F68266BE421401A6D7D1@mail-sc-3.nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4d71d00f4e433a515d330d0b32798e4aa350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Julian Elischer' cc: Andy Ritger cc: Daniel Eischen Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 05:35:08 -0000 Gareth Hughes wrote: > Or, better yet, to make sure no problems arise when you change the internals > of your data structures: > > %gs ----->[libGL stuff (16 words, say) ] > [threadsystem-thread-specific-data] > [ stuff ] > > You reserve the first 16 words of your thread data structure for us, and > we're done. At least when this library is being used. This would really suck on most systems because of the cache line misses. The thread specific data for the threading system wants to be in a cache line aligned memory location, and not exceed a cache line in length. You are running multiple threads: surely it's more important, in general, for threads context switch to be fast from your perspective? -- Terry From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 22:46:12 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 75BBB37B401 for ; Mon, 16 Jun 2003 22:46:12 -0700 (PDT) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id DF63243F75 for ; Mon, 16 Jun 2003 22:46:09 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2injaq3.dialup.mindspring.com ([165.121.171.67] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19S9IE-0001Qb-00; Mon, 16 Jun 2003 22:46:07 -0700 Message-ID: <3EEEAADC.66CA7261@mindspring.com> Date: Mon, 16 Jun 2003 22:45:00 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Gareth Hughes References: <2D32959E172B8F4D9B02F68266BE421401A6D7D8@mail-sc-3.nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4d71d00f4e433a515fd17ac834477db6c350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Daniel Eischen' cc: Andy Ritger cc: Julian Elischer Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 05:46:12 -0000 Gareth Hughes wrote: > On Mon, 16 Jun 2003, Daniel Eischen wrote: > > > > I see this as a problem with the OpenGL API. You're trying > > to make something thread-safe that isn't by its nature. > > I would rather see OpenGL-MT with new interfaces that > > are by nature thread-safe. > > Sorry, but this simply isn't going to happen. People smarter > than you or I designed the API this way to allow for high > performance implementations on a wide variety of platforms. > There are very good reasons each API call doesn't take a > context parameter. It sounds like you want to wrap your entry points, and then use -ffixed-reg to reserve one of the general purpose registers to burn as a TLS context register. If you are going to burn a register on this, it should at least be a register you actually have a right to muck with... -- Terry From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 22:56:39 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6914937B401 for ; Mon, 16 Jun 2003 22:56:39 -0700 (PDT) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8565F43FBF for ; Mon, 16 Jun 2003 22:56:38 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2injaq3.dialup.mindspring.com ([165.121.171.67] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19S9SM-0002jG-00; Mon, 16 Jun 2003 22:56:35 -0700 Message-ID: <3EEEAD50.C374C068@mindspring.com> Date: Mon, 16 Jun 2003 22:55:28 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Gareth Hughes References: <2D32959E172B8F4D9B02F68266BE421401A6D7DF@mail-sc-3.nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4f75b8175ffdaf6a02d8f6eadf9515152387f7b89c61deb1d350badd9bab72f9c350badd9bab72f9c cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Julian Elischer' cc: Andy Ritger cc: Daniel Eischen Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 05:56:39 -0000 Gareth Hughes wrote: > On Mon, 16 Jun 2003, Julian Elischer wrote: > > > > can wine use OpenGL? > > > > Doesn't look like a clash to me. :-) > > I don't understand your point here... His point is that you can feel free to use %fs, even though WINE uses %fs, so long as you do not use WINE and OpenGL in the same program. In other words, the argument that you can't use %fs because WINE does only holds water if you use OpenGL to implement WINE, or WINE to implement OpenGL. Otherwise, there's no conflict. > Currently, given the performance requirements of OpenGL, our drivers > are one of the first users of the new ELF TLS implementation on Linux, > outside of libc itself. Given the fact that this is a general solution > to a common problem, I strongly urge you to take it into consideration > when working on your threading libraries. This is problematic. It constrains the implementation to a specific architecture. The 1:1 mapping of threads in Linux and Solaris are basically a result of "N:M threads are hard to get right; let's just implement 1:1, and go shopping!". Likewise, until we know that this standard is not one of the areas SCO is claiming Linux stole from SCO, it's going to be risky to just follow Linux's lead (on anything!) for a while. > There are only going to be > more users of __thread variables as time goes on -- you can imagine > that many multithreaded user-space server applications will start using > this soon, instead of the clunky pthread TLS interface. It's "clunky" for a reason: to discourage people from writing "apartment" or "rental" model threaded application, instead of "free threaded" applications, which are safely reentrant. We can make it less expensive to use (I'm positive!), but to make it less "clunky" would just encourage evil developers to use it. To paraphrase a wise man: people a lot smarter than you or I designed the POSIX thread local storage API the way it is for a reason... ;^). -- Terry From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 22:59:22 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5833037B401 for ; Mon, 16 Jun 2003 22:59:22 -0700 (PDT) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id D008A43F3F for ; Mon, 16 Jun 2003 22:59:21 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2injaq3.dialup.mindspring.com ([165.121.171.67] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19S9V1-00034S-00; Mon, 16 Jun 2003 22:59:19 -0700 Message-ID: <3EEEADF5.6DC76F9F@mindspring.com> Date: Mon, 16 Jun 2003 22:58:13 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Gareth Hughes References: <2D32959E172B8F4D9B02F68266BE421401A6D7E1@mail-sc-3.nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4f75b8175ffdaf6a0ca4897b4cf105796548b785378294e88350badd9bab72f9c350badd9bab72f9c cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Daniel Eischen' cc: Andy Ritger cc: Julian Elischer Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 05:59:22 -0000 Gareth Hughes wrote: > On Mon, 16 Jun 2003, Daniel Eischen wrote: > > Right. It just seems wrong to me to be able to insert __thread > > after every global variable in a library and instantly make > > it thread-safe. > > Why, exactly? Surely, from a programming point of view, this is > exactly what you want -- a transparent way to access your thread > local data. No, what you want first and foremost is your code to compile and run on all platforms, without limiting your market by relying on defacto language extensions that have not been explicitly and intentionally standardized by a recognized standards body. -- Terry From owner-freebsd-threads@FreeBSD.ORG Mon Jun 16 23:35:42 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B0BDF37B401 for ; Mon, 16 Jun 2003 23:35:42 -0700 (PDT) Received: from stork.mail.pas.earthlink.net (stork.mail.pas.earthlink.net [207.217.120.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6B4A043F85 for ; Mon, 16 Jun 2003 23:35:41 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2injaq3.dialup.mindspring.com ([165.121.171.67] helo=mindspring.com) by stork.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19SA46-0002lL-00; Mon, 16 Jun 2003 23:35:35 -0700 Message-ID: <3EEEB671.2172F5D8@mindspring.com> Date: Mon, 16 Jun 2003 23:34:25 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Alexander Kabaev References: <2D32959E172B8F4D9B02F68266BE421401A6D7E6@mail-sc-3.nvidia.com> <20030616211835.10d0fba4.kabaev@mail.ru> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4fcbf903df156e85a374543cf4f6c2306666fa475841a1c7a350badd9bab72f9c350badd9bab72f9c cc: zander@mail.minion.de cc: Gareth Hughes cc: 'Julian Elischer' cc: threads@freebsd.org cc: 'Daniel Eischen' cc: Andy Ritger Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 06:35:43 -0000 Alexander Kabaev wrote: > Gareth Hughes wrote: > > If FreeBSD support ELF TLS and __thread variables in ANY form, our > > driver will use this support. If the best you can do is a function > > call per access, so be it. It doesn't sound like there are any other > > options, given the fact that you ship with three different thread > > libraries. > > Three different libraries can attempt to coordinate and reserve exactly > the same TLS segment size. Gareth's point is that there is a function call to abstract that, unless all approaches use the same mechanism. Right now, he's said that he can't use %fs, because he has a version of WINE that uses OpenGL to implement it's video, and the threads people have said that he can't use %gs because it points to a per-KSE value, not a per thread value, and there are potentially many threads associated with each KSE, so it's no good for TLS, only for KSELS. I've suggested that he use the compiler flags to reserve another *different* register (not %fs and not %gs) for his purpose; the compiler has explicit support for this. If he is wiling to burn a register in order to get this functionality, it should be one that's normally allocated for user programs to burn anyway. Julian has pointed out that the TLS data should probably be cached following a return from a context switch as a result of the user space threads scheduler, if there is one, scheduling the thread to run; so far, it seems that Gareth has mistaken what Julian meant by this (that the reload happen high up in the OpenGL code, using an explicit call, to do the register switch, so that it's available lower down; this would work with the %fs approach, without breaking WINE). In addition, it seems that Gareth's Linux approach is assuming that a thread which is involuntarily context switch will be run again, when the next quantum becomes available. This is an invalid assumption for libkse (though it's currently hacked, locally, by most people to keep non-strict POSIX applications happy), and could be an invalid assumption in Linux and libthr in FreeBSD, should the scheduler code change in the future to recalculate priority prior to giving the quantum to a particular thread. Add to this that the model that Gareth is suggesting as a defacto standard is one advocacted by a company that's sueing IBM over the Linux OS containing uniform code in 60 lines of its scheduler which might very well be related to this publication, for all we know, and the assumptions about implementation implicit in the standard, and there's a significant issue that needs to be resolve here, and I'm not just talking about the legal implications of implementing to a Caldera/SCO specification that they may feel contains their intellectual property. Minimally, we are likely talking a "first call" optimization, at a minimum, in order to set a global offset as valid relative to a register common to all three threads libraries in FreeBSD, and, potentially, two of them, if we *must* go indirect through %gs to get the TLS from the KSELS in %gs. Does that jive with what everyone else understands of thi mailing list thread so far? -- Terry From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 00:18:12 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4D65F37B401 for ; Tue, 17 Jun 2003 00:18:12 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id F27FD43FA3 for ; Tue, 17 Jun 2003 00:18:10 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5H7IADZ052914; Tue, 17 Jun 2003 00:18:10 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5H7IAxH002576; Tue, 17 Jun 2003 00:18:10 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5H7IAGh002575; Tue, 17 Jun 2003 00:18:10 -0700 (PDT) (envelope-from marcel) Date: Tue, 17 Jun 2003 00:18:10 -0700 From: Marcel Moolenaar To: threads@FreeBSD.org Message-ID: <20030617071810.GA2451@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.4i cc: gareth@nvidia.com cc: aritger@nvidia.com Subject: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 07:18:12 -0000 Guys, I haven't followed the threads related to Nvidia and TLS, but after browsing the archives it's clear that people don't understand the issue and/or each other. Consequently, people get pissed off for no good reason at all. The __thread keyword does not originate from Linux and has no relation to kernel and/or user threading. It has been introduced to work around the limitations of pthread provided mechanisms for thread local storage. It's a language extension that requires support by the compiler, linker, runtime linker and threading libraries. Whether we want to support the __thread keyword or not is unrelated to why OpenGL makes use of it. Any suggestion that OpenGL needs to change their interfaces is utterly bollocks. Likewise, suggestions made to reserve space for OpenGL is also not to the point. The whole pivoting role of %gs is purely circumstantial. You can implement TLS as implied by the __thread keyword in lots of different ways. The simple truth of the matter is that OpenGL uses the __thread keyword to create thread local variables and if we don't support the __thread keyword (we do not now), we don't support OpenGL (at least, proprietary implementations) in all cases. There's a definite advantage to supporting the __thread keyword in userland and we should add the support. It really isn't that hard, but it requires some thought and testing. In most cases you simply point your thread pointer between the control structure and the thread local segments. For dynamic TLS, Sun has come up with a mechanism that annihilates all the performance advantages and consequently made it the default model. This is where the __tls_get_addr() function call comes into play (you don't need it for the static TLS model). It sucks, but allows loading shared libraries that contain TLS. Adding support for the __thread keyword in the kernel is probably not worth it. In short: Don't bash Nvidia. What they do is not uncommon. Well, maybe in Open Source environments. So please end this thread, unless people get constructive. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 02:13:06 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0A0DA37B401 for ; Tue, 17 Jun 2003 02:13:06 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 77D3743FB1 for ; Tue, 17 Jun 2003 02:13:05 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Tue, 17 Jun 2003 02:16:22 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Tue, 17 Jun 2003 02:13:00 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D7EC@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Terry Lambert' , Alexander Kabaev Date: Tue, 17 Jun 2003 02:12:59 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Julian Elischer' cc: Andy Ritger cc: 'Daniel Eischen' Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 09:13:06 -0000 Terry Lambert wrote: > > Right now, he's said that he can't use %fs, because he has a > version of WINE that uses OpenGL to implement it's video, and > the threads people have said that he can't use %gs because it > points to a per-KSE value, not a per thread value, and there > are potentially many threads associated with each KSE, so it's > no good for TLS, only for KSELS. All versions of Wine use OpenGL for 3D graphics. > I've suggested that he use the compiler flags to reserve another > *different* register (not %fs and not %gs) for his purpose; the > compiler has explicit support for this. If he is wiling to burn > a register in order to get this functionality, it should be one > that's normally allocated for user programs to burn anyway. We're not willing to burn a register. Our libraries are not PIC for the same reason. > Julian has pointed out that the TLS data should probably be cached > following a return from a context switch as a result of the user > space threads scheduler, if there is one, scheduling the thread to > run; so far, it seems that Gareth has mistaken what Julian meant > by this (that the reload happen high up in the OpenGL code, using > an explicit call, to do the register switch, so that it's available > lower down; this would work with the %fs approach, without breaking > WINE). I must be completely misunderstanding you. What do you mean by "high level" and "low level" OpenGL code? > In addition, it seems that Gareth's Linux approach is assuming that > a thread which is involuntarily context switch will be run again, > when the next quantum becomes available. This is an invalid > assumption for libkse (though it's currently hacked, locally, by > most people to keep non-strict POSIX applications happy), and > could be an invalid assumption in Linux and libthr in FreeBSD, > should the scheduler code change in the future to recalculate > priority prior to giving the quantum to a particular thread. Why do you say that? -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 06:22:16 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C27CB37B401 for ; Tue, 17 Jun 2003 06:22:16 -0700 (PDT) Received: from rwcrmhc11.attbi.com (rwcrmhc11.comcast.net [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3574143F75 for ; Tue, 17 Jun 2003 06:22:16 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (rwcrmhc11) with ESMTP id <2003061706094101300am5i2e>; Tue, 17 Jun 2003 06:09:41 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id XAA25090; Mon, 16 Jun 2003 23:09:40 -0700 (PDT) Date: Mon, 16 Jun 2003 23:09:39 -0700 (PDT) From: Julian Elischer To: Terry Lambert In-Reply-To: <3EEEADF5.6DC76F9F@mindspring.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: Gareth Hughes cc: 'Daniel Eischen' cc: Andy Ritger Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 13:22:17 -0000 Terry, don't go overboard.. I think we can work with this.. I have some ideas.. We may be able to cut it down to 2 instructions in general and one for the N:N case using linker relaxation. It depends on how much we can hack teh linker to do similar stuff to what linux does, but notice tht on a fully dynamically linked object it looks like the Optimised 1 instruction version can't be used anyhow. (well it needs a bit more thinking but it may not be as bad as it seemed at first.) In the M:N case we can probably have the Userland thread scheduler set a pointer to the TLD blocks into the KSE mailox each time a new thread is scheduled.. %gs would be left the same but what it points to could be changed cheaply. On Mon, 16 Jun 2003, Terry Lambert wrote: > Gareth Hughes wrote: > > On Mon, 16 Jun 2003, Daniel Eischen wrote: > > > Right. It just seems wrong to me to be able to insert __thread > > > after every global variable in a library and instantly make > > > it thread-safe. > > > > Why, exactly? Surely, from a programming point of view, this is > > exactly what you want -- a transparent way to access your thread > > local data. > > No, what you want first and foremost is your code to compile and > run on all platforms, without limiting your market by relying on > defacto language extensions that have not been explicitly and > intentionally standardized by a recognized standards body. > > -- Terry > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 06:41:14 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5D49A37B401 for ; Tue, 17 Jun 2003 06:41:14 -0700 (PDT) Received: from park.rambler.ru (park.rambler.ru [81.19.64.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id D081743FA3 for ; Tue, 17 Jun 2003 06:41:12 -0700 (PDT) (envelope-from is@rambler-co.ru) Received: from is.park.rambler.ru (is.park.rambler.ru [81.19.64.102]) by park.rambler.ru (8.12.6/8.12.6) with ESMTP id h5HDfBmF051534 for ; Tue, 17 Jun 2003 17:41:11 +0400 (MSD) Date: Tue, 17 Jun 2003 17:41:11 +0400 (MSD) From: Igor Sysoev X-Sender: is@is To: threads@freebsd.org In-Reply-To: <20030617071810.GA2451@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 13:41:14 -0000 On Tue, 17 Jun 2003, Marcel Moolenaar wrote: > There's a definite advantage to supporting the __thread keyword in > userland and we should add the support. It really isn't that hard, > but it requires some thought and testing. In most cases you simply > point your thread pointer between the control structure and the > thread local segments. If the thread implementation uses gs register to point to thread specific data: gs -> [ thread specific data ] [ tls_array ] then this C code __thread int a; a = 1; can be translated to mov tls_key, %ecx mov $gs:tls_array, %eax mov (%eax,%ecx,4), %eax mov $1, (%eax) In FreeBSD we use gs to point to KSE specific data gs -> [ KSE specific data ] [ current thread ] -> [ thread specfic data ] [ tls_array ] and then same C code can be translate to mov tls_key, %ecx mov $gs:current_thread, %eax mov tls_array(%eax), %eax mov (%eax,%ecx,4), %eax mov $1, (%eax) And I think we should use this scheme not only in libkse but in libthr too to be binary compatible. Igor Sysoev http://sysoev.ru/en/ From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 09:19:02 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1CED237B407 for ; Tue, 17 Jun 2003 09:19:02 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id E169543F75 for ; Tue, 17 Jun 2003 09:19:00 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5HGImXh023336; Tue, 17 Jun 2003 12:18:48 -0400 (EDT) Date: Tue, 17 Jun 2003 12:18:48 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Gareth Hughes In-Reply-To: <2D32959E172B8F4D9B02F68266BE421401A6D7E7@mail-sc-3.nvidia.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: 'Julian Elischer' cc: Andy Ritger cc: 'Daniel Eischen' Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 16:19:02 -0000 On Mon, 16 Jun 2003, Gareth Hughes wrote: > Umm, you clearly don't understand what I've been talking about. > Upon entry into libGL, i.e. when an OpenGL API entrypoint is > called by the application, things like the current context or > current dispatch table are fetched from thread-local storage. > Internally, we pass these pointers around as required. > > So, we might have something like this: > > void glBegin(GLenum mode) > { > // Grab the current dispatch pointer from TLS > __GLdispatch *dispatch = GET_CURRENT_DISPATCH(); > > // Call into the driver backend > dispatch->Begin(mode) > } > > or, in x86 assembly: > > glBegin: > mov %gs:__gl_dispatch@ntpoff, %eax > jmp *__begin_offset(%eax) > > (note that Andy mentioned this example in his original email) > > This would jump to a function inside the driver like this: > > void __internal_Begin(GLenum mode) > { > __GLcontext *ctx = GET_CURRENT_CONTEXT(); > > do_something(ctx, ...); > do_something_else(ctx, ...); > // and so on > } But you have at least 2 levels of "get current thread" context: one in the openGl entry point (GET_CURRENT_DISPATCH()) and another in the driver (GET_CURRENT_CONTEXT()). If you are going to get some TLS in the OpenGL entry point, then make it the only TLS thingy you'll ever need. Hang all your stuff off the one TLS context and pass it (or parts of it) around to other consumers. So the driver gets called as follows: void __internal_Begin(GLenum mode, __GLcontext *ctx) { do_something(ctx, ...); do_something_else(ctx, ...); // and so on } Now when somebody comes up with thread-safe OpenGL and you have glBegin_r(GLctx *ctx, GLenum mode), you are golden: glBegin_r(GLctx *ctx, GLenum mode) { __GLdispatch *dispatch = ctx->dispatch; __GLcontext *glctx = ctx->glctx; dispatch->Begin(glctx, mode); } > Once we're inside the driver, we know what the current context > or other thread-local variable's value is. Two critical > points: > > 1) We have to fetch the value from TLS at least once per entry > into the driver. Pass what you need to the driver. You needn't have to fetch any TLS in the driver because the only way to the driver is from the OpenGL entry point -- unless I'm missing something. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 09:38:04 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9890D37B401 for ; Tue, 17 Jun 2003 09:38:04 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id E97B743FAF for ; Tue, 17 Jun 2003 09:38:03 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5HGbwXh026549; Tue, 17 Jun 2003 12:37:58 -0400 (EDT) Date: Tue, 17 Jun 2003 12:37:58 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: zander@mail.minion.de cc: Gareth Hughes cc: threads@freebsd.org cc: 'Daniel Eischen' cc: Andy Ritger Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 16:38:04 -0000 On Mon, 16 Jun 2003, Julian Elischer wrote: > Terry, don't go overboard.. Pleast let him. If it's not too cold, I'll join him for a swim :-) > On Mon, 16 Jun 2003, Terry Lambert wrote: > > > Gareth Hughes wrote: > > > On Mon, 16 Jun 2003, Daniel Eischen wrote: > > > > Right. It just seems wrong to me to be able to insert __thread > > > > after every global variable in a library and instantly make > > > > it thread-safe. > > > > > > Why, exactly? Surely, from a programming point of view, this is > > > exactly what you want -- a transparent way to access your thread > > > local data. >From a programming view, I want well thought APIs. If you are providing a library, its interfaces should be thread-safe by default. I understand that the OpenGL API is what it is, but that shouldn't stop folks from improving on it. > > > > No, what you want first and foremost is your code to compile and > > run on all platforms, without limiting your market by relying on > > defacto language extensions that have not been explicitly and > > intentionally standardized by a recognized standards body. And perhaps someone will get the urge to design an API that is thread-safe and works on all platforms without performance penalties, and this becomes a defacto API. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 09:40:59 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4975937B401 for ; Tue, 17 Jun 2003 09:40:59 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 29EE243FAF for ; Tue, 17 Jun 2003 09:40:58 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5HGevDZ055759; Tue, 17 Jun 2003 09:40:57 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5HGevst000851; Tue, 17 Jun 2003 09:40:57 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5HGerkp000850; Tue, 17 Jun 2003 09:40:53 -0700 (PDT) (envelope-from marcel) Date: Tue, 17 Jun 2003 09:40:53 -0700 From: Marcel Moolenaar To: Igor Sysoev Message-ID: <20030617164053.GD584@dhcp01.pn.xcllnt.net> References: <20030617071810.GA2451@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 16:40:59 -0000 On Tue, Jun 17, 2003 at 05:41:11PM +0400, Igor Sysoev wrote: > On Tue, 17 Jun 2003, Marcel Moolenaar wrote: > > > There's a definite advantage to supporting the __thread keyword in > > userland and we should add the support. It really isn't that hard, > > but it requires some thought and testing. In most cases you simply > > point your thread pointer between the control structure and the > > thread local segments. > > If the thread implementation uses gs register to point to thread > specific data: > > gs -> [ thread specific data ] > [ tls_array ] > > then this C code > > __thread int a; a = 1; > > can be translated to > > mov tls_key, %ecx > mov $gs:tls_array, %eax > mov (%eax,%ecx,4), %eax > mov $1, (%eax) The runtime specification of the platform generally dictates how TLS access sequences are to be generated. This influences how you want to implement your thread library. It's not the other way around. If our use of %gs on i386 conflicts with the runtime specification (extension), we need to change. I would also appreciate a less i386-oriented mindset on this list. What may not be a "true" standard on i386, is on ia64. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 11:59:17 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2AF3D37B401 for ; Tue, 17 Jun 2003 11:59:17 -0700 (PDT) Received: from web12401.mail.yahoo.com (web12401.mail.yahoo.com [216.136.173.128]) by mx1.FreeBSD.org (Postfix) with SMTP id D14C743FA3 for ; Tue, 17 Jun 2003 11:59:16 -0700 (PDT) (envelope-from ks4usa@yahoo.com) Message-ID: <20030617185916.44649.qmail@web12401.mail.yahoo.com> Received: from [12.29.176.205] by web12401.mail.yahoo.com via HTTP; Tue, 17 Jun 2003 11:59:16 PDT Date: Tue, 17 Jun 2003 11:59:16 -0700 (PDT) From: Sergey Kosyakov To: freebsd-threads@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: The first kse_create call X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 18:59:17 -0000 Hi, I'm trying to run simple program with kse. As I understood the very first kse_create call does not create KSE but just assigns the mailbox to the existing "default" KSE and makes upcall. May be I'm doing something wrong, but I never got upcall on the first kse. When I create another KSE (and another KSE group) I immediatelly get the upcall for this (second) KSE. Just interesting how does it work - may be the first KSE has special behavior? Thanks, Sergey. __________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 13:54:33 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 84D1437B401 for ; Tue, 17 Jun 2003 13:54:33 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id E5A8043FB1 for ; Tue, 17 Jun 2003 13:54:32 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5HKsVXh009128; Tue, 17 Jun 2003 16:54:32 -0400 (EDT) Date: Tue, 17 Jun 2003 16:54:31 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Sergey Kosyakov In-Reply-To: <20030617185916.44649.qmail@web12401.mail.yahoo.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: The first kse_create call X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 20:54:33 -0000 > Hi, > > I'm trying to run simple program with kse. As I understood the very > first kse_create call does not create KSE but just assigns the mailbox > to the existing "default" KSE and makes upcall. May be I'm doing > something wrong, > but I never got upcall on the first kse. When I create another KSE (and > another KSE group) I immediatelly get the upcall for this (second) KSE. > Just interesting how does it work - may be the first KSE has special > behavior? This is the correct behavior. The first kse_create() does not generate an immediate upcall. It only flags the current context as being a KSE. An upcall in this initial KSE will take place under the same conditions as other KSEs (KSE mailbox has a thread mailbox pointer and thread blocks, quantum expires, etc). Subsequent kse_create() calls will generate upcalls immediately (well, at the mercy of the kernel scheduler). -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 15:02:23 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DF51937B401 for ; Tue, 17 Jun 2003 15:02:23 -0700 (PDT) Received: from sccrmhc13.attbi.com (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4201D43F85 for ; Tue, 17 Jun 2003 15:02:23 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc13) with ESMTP id <20030617220221016001mfg3e>; Tue, 17 Jun 2003 22:02:22 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id PAA31262; Tue, 17 Jun 2003 15:02:21 -0700 (PDT) Date: Tue, 17 Jun 2003 15:02:20 -0700 (PDT) From: Julian Elischer To: Marcel Moolenaar In-Reply-To: <20030617071810.GA2451@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@FreeBSD.org cc: gareth@nvidia.com cc: aritger@nvidia.com Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 22:02:24 -0000 On Tue, 17 Jun 2003, Marcel Moolenaar wrote: > Guys, > > In short: Don't bash Nvidia. What they do is not uncommon. Well, > maybe in Open Source environments. So please end this thread, > unless people get constructive. I think its already ended.. basically: We should alwasy be able to use (on i386) the sam amethods outlined for solaris. Not quite as quick as those for Linux but more general. From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 15:09:07 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7292937B401 for ; Tue, 17 Jun 2003 15:09:07 -0700 (PDT) Received: from sccrmhc12.attbi.com (sccrmhc12.comcast.net [204.127.202.56]) by mx1.FreeBSD.org (Postfix) with ESMTP id A3E4643FBF for ; Tue, 17 Jun 2003 15:09:06 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc12) with ESMTP id <200306172209050120099l4me>; Tue, 17 Jun 2003 22:09:05 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id PAA31304; Tue, 17 Jun 2003 15:09:03 -0700 (PDT) Date: Tue, 17 Jun 2003 15:09:02 -0700 (PDT) From: Julian Elischer To: Gareth Hughes In-Reply-To: <2D32959E172B8F4D9B02F68266BE421401A6D7EC@mail-sc-3.nvidia.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Alexander Kabaev cc: zander@mail.minion.de cc: threads@freebsd.org cc: 'Daniel Eischen' cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 22:09:07 -0000 On Tue, 17 Jun 2003, Gareth Hughes wrote: > Terry Lambert wrote: > > > [a lot of stuff] Gareth, thanks for puting up with us and showing considerable patience. I think we can make an 'efficient (though possibly not quite the 1 instruction possible in a static linked linux binary) __thread implimentation. We can possibly do 1 instruction in the 1:1 library but that would leave binary compatibility with the other libraries out the window. I think that it is a pitty that the openGL ABI doesn't carry around a 'context' argument on all calls but it's not going to change and we have to use what is there. Julian From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 15:24:06 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E84F137B401 for ; Tue, 17 Jun 2003 15:24:06 -0700 (PDT) Received: from rwcrmhc12.attbi.com (rwcrmhc12.comcast.net [216.148.227.85]) by mx1.FreeBSD.org (Postfix) with ESMTP id 28E4943F3F for ; Tue, 17 Jun 2003 15:24:06 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (rwcrmhc12) with ESMTP id <2003061722240501400ad3gre>; Tue, 17 Jun 2003 22:24:05 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id PAA31406; Tue, 17 Jun 2003 15:24:04 -0700 (PDT) Date: Tue, 17 Jun 2003 15:24:03 -0700 (PDT) From: Julian Elischer To: Sergey Kosyakov In-Reply-To: <20030617185916.44649.qmail@web12401.mail.yahoo.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: The first kse_create call X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 22:24:07 -0000 The first kse_create() "converts" teh current thread/KSE to be upcall "capable" but it just returns.. all the other calls to kse_create() will also 'just return', however the new KSE/thread will upcall. in the case of the first call, there is no "other" thread. (unless the NEW_KSEGRP flag is set). On Tue, 17 Jun 2003, Sergey Kosyakov wrote: > Hi, > > I'm trying to run simple program with kse. As I understood the very > first kse_create call does not create KSE but just assigns the mailbox > to the existing "default" KSE and makes upcall. May be I'm doing > something wrong, > but I never got upcall on the first kse. When I create another KSE (and > another KSE group) I immediatelly get the upcall for this (second) KSE. > Just interesting how does it work - may be the first KSE has special > behavior? > > Thanks, > Sergey. > > > __________________________________ > Do you Yahoo!? > SBC Yahoo! DSL - Now only $29.95 per month! > http://sbc.yahoo.com > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 15:28:29 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C794B37B401; Tue, 17 Jun 2003 15:28:29 -0700 (PDT) Received: from sccrmhc12.attbi.com (sccrmhc12.comcast.net [204.127.202.56]) by mx1.FreeBSD.org (Postfix) with ESMTP id B712A43F75; Tue, 17 Jun 2003 15:28:28 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc12) with ESMTP id <200306172228270120099rkne>; Tue, 17 Jun 2003 22:28:27 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id PAA31459; Tue, 17 Jun 2003 15:28:26 -0700 (PDT) Date: Tue, 17 Jun 2003 15:28:26 -0700 (PDT) From: Julian Elischer To: deischen@freebsd.org In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Sergey Kosyakov cc: freebsd-threads@freebsd.org Subject: Re: The first kse_create call X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 22:28:30 -0000 On Tue, 17 Jun 2003, Daniel Eischen wrote: > > Hi, > > > > I'm trying to run simple program with kse. As I understood the very > > first kse_create call does not create KSE but just assigns the mailbox > > to the existing "default" KSE and makes upcall. May be I'm doing > > something wrong, > > but I never got upcall on the first kse. When I create another KSE (and > > another KSE group) I immediatelly get the upcall for this (second) KSE. > > Just interesting how does it work - may be the first KSE has special > > behavior? > > This is the correct behavior. The first kse_create() does not > generate an immediate upcall. It only flags the current context > as being a KSE. An upcall in this initial KSE will take place > under the same conditions as other KSEs (KSE mailbox has a thread > mailbox pointer and thread blocks, quantum expires, etc). > Subsequent kse_create() calls will generate upcalls immediately > (well, at the mercy of the kernel scheduler). Wellll, the kse_create() will not upcall, but the new kse that is created will upcall. The kse_create() call will just return to the original thread. > > -- > Dan Eischen > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 15:39:44 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AB52437B401 for ; Tue, 17 Jun 2003 15:39:44 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id D128D43F3F for ; Tue, 17 Jun 2003 15:39:43 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from ns1.xcllnt.net (localhost [127.0.0.1]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5HMdBDZ057728; Tue, 17 Jun 2003 15:39:11 -0700 (PDT) (envelope-from marcel@ns1.xcllnt.net) Received: (from marcel@localhost) by ns1.xcllnt.net (8.12.9/8.12.9/Submit) id h5HMdAX4057727; Tue, 17 Jun 2003 15:39:10 -0700 (PDT) Date: Tue, 17 Jun 2003 15:39:10 -0700 From: Marcel Moolenaar To: Julian Elischer Message-ID: <20030617223910.GB57040@ns1.xcllnt.net> References: <20030617071810.GA2451@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.1i cc: threads@FreeBSD.org cc: gareth@nvidia.com cc: aritger@nvidia.com Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 22:39:44 -0000 On Tue, Jun 17, 2003 at 03:02:20PM -0700, Julian Elischer wrote: > > > On Tue, 17 Jun 2003, Marcel Moolenaar wrote: > > > Guys, > > > > In short: Don't bash Nvidia. What they do is not uncommon. Well, > > maybe in Open Source environments. So please end this thread, > > unless people get constructive. > > I think its already ended.. > > basically: > We should alwasy be able to use (on i386) the sam amethods outlined for > solaris. > Not quite as quick as those for Linux but more general. I'm not sure you understand the issue (I can easily be wrong, I just don't see the evidence in your statement). To support the __thread keyword, our thread library needs to create the TLS as defined in the binary and its dependent shared libraries by virtue of the .tdata and .tbss sections/segments, based on the image of the TLS as constructed by the RTLD for the initial set of modules (created for the initial thread) and amended by TLS space defined in the dynamicly loaded libraries; and the TLS has to be created for every new thread at the time the thread itself is created. This TLS allocation has to be made accessable in accordance with runtime specifications for the supported architectures (libthr: i386 & ia64; libkse: i386 currently -- more to follow) and in line with the access sequences created by the compiler, and using the static relocations known to the static linker and dynamic relocations of which the support needs to be added to RTLD. The static TLS model requires the least amount of work: add support to allocate the TLS image for every thread creation and point the thread pointer to it in a way compatible with the runtime spec. The dynamic TLS model requires more substantial changes and involves RTLD as well. This is the model that requires __tls_get_addr(). -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 16:38:48 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9068137B401 for ; Tue, 17 Jun 2003 16:38:48 -0700 (PDT) Received: from sccrmhc11.attbi.com (sccrmhc11.comcast.net [204.127.202.55]) by mx1.FreeBSD.org (Postfix) with ESMTP id C83E843FBD for ; Tue, 17 Jun 2003 16:38:47 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc11) with ESMTP id <20030617233846011007vprae>; Tue, 17 Jun 2003 23:38:46 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA31972; Tue, 17 Jun 2003 16:38:45 -0700 (PDT) Date: Tue, 17 Jun 2003 16:38:44 -0700 (PDT) From: Julian Elischer To: Marcel Moolenaar In-Reply-To: <20030617223910.GB57040@ns1.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@FreeBSD.org cc: gareth@nvidia.com cc: aritger@nvidia.com Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 23:38:48 -0000 On Tue, 17 Jun 2003, Marcel Moolenaar wrote: > On Tue, Jun 17, 2003 at 03:02:20PM -0700, Julian Elischer wrote: > > > > > > On Tue, 17 Jun 2003, Marcel Moolenaar wrote: > > > > > Guys, > > > > > > In short: Don't bash Nvidia. What they do is not uncommon. Well, > > > maybe in Open Source environments. So please end this thread, > > > unless people get constructive. > > > > I think its already ended.. > > > > basically: > > We should alwasy be able to use (on i386) the sam amethods outlined for > > solaris. > > Not quite as quick as those for Linux but more general. > > I'm not sure you understand the issue (I can easily be wrong, I just > don't see the evidence in your statement). I understand what is needed from the thread library. I am asking for input from people who understand the toolchain :-) > To support the __thread > keyword, our thread library needs to create the TLS as defined in the > binary and its dependent shared libraries by virtue of the .tdata and > .tbss sections/segments, based on the image of the TLS as constructed > by the RTLD for the initial set of modules (created for the initial > thread) and amended by TLS space defined in the dynamicly loaded > libraries; and the TLS has to be created for every new thread at the > time the thread itself is created. Geez considerred writing shorter sentences? :-) there is no reason that all our thread libraries cannot set up TLS sections associated with each modile linked. (Assuming there is a good description as to how it should be done) The ammending need not be done up front if the dynamic model is used.. > This TLS allocation has to be made > accessable in accordance with runtime specifications for the supported > architectures (libthr: i386 & ia64; libkse: i386 currently -- more to > follow) and in line with the access sequences created by the compiler, > and using the static relocations known to the static linker and dynamic > relocations of which the support needs to be added to RTLD. exactly. Though, I might add that we don't need to follow the spec exactly. In the same way that Solaris has an "excepmtion" and has its own version of the spec we could do so too. It turns out however that the solaris-x86 spec is very close to what we want and need anyhow. Libthr and libkse (i386) can be modified to both have the same support. The spec requires that %gs:0 is the address of a pointer to the thread control block (TCB). In kse this is easily achieved by puting such a pointer at the front of the KSE mailbox. (where %gs points), and in libthr the pointer points directly to the TCB itself (as far as I see) and there could be a pointer to itself set there. for libc_r we could set up %gs to be anything we want as we don't use it as such.. anyway. the three libraries would stay binarily compatible. We obviously would need to add code to do teh initialisation of the .tdata and .tbss segments as indicated in the doc. but it all looks doable. (though rather hacky). > > The static TLS model requires the least amount of work: add support > to allocate the TLS image for every thread creation and point the > thread pointer to it in a way compatible with the runtime spec. yes We can do this. I disagree however that it requires the LEAST amount of work.. > > The dynamic TLS model requires more substantial changes and involves > RTLD as well. This is the model that requires __tls_get_addr(). By my reading of it, you can't use the static model alone. If you have dynamically loaded modules you need to be able to use the dynamic model. Anyway that's why I'm asking.. we need to have people who understand the rtld in on the discussion because we will need to do this. > > -- > Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net > From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 16:40:43 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7DD1F37B404 for ; Tue, 17 Jun 2003 16:40:43 -0700 (PDT) Received: from sccrmhc13.attbi.com (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9A6B743FBD for ; Tue, 17 Jun 2003 16:40:42 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc13) with ESMTP id <20030617234041016001lsmhe>; Tue, 17 Jun 2003 23:40:41 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA32006; Tue, 17 Jun 2003 16:40:40 -0700 (PDT) Date: Tue, 17 Jun 2003 16:40:40 -0700 (PDT) From: Julian Elischer To: Marcel Moolenaar In-Reply-To: <20030617223910.GB57040@ns1.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@FreeBSD.org cc: gareth@nvidia.com cc: aritger@nvidia.com Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 23:40:43 -0000 On Tue, 17 Jun 2003, Marcel Moolenaar wrote: thread moved to a different subject in current@freebsd.org.. thanks for listenning.. From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 16:45:03 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D59C737B401; Tue, 17 Jun 2003 16:45:03 -0700 (PDT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3247443FBD; Tue, 17 Jun 2003 16:45:00 -0700 (PDT) (envelope-from davidxu@freebsd.org) Received: from tiger (davidxu@localhost [127.0.0.1]) by freefall.freebsd.org (8.12.9/8.12.9) with SMTP id h5HNivUp043852; Tue, 17 Jun 2003 16:44:57 -0700 (PDT) (envelope-from davidxu@freebsd.org) Message-ID: <002101c3352a$e931a7f0$0701a8c0@tiger> From: "David Xu" To: "Marcel Moolenaar" , "Julian Elischer" References: <20030617071810.GA2451@dhcp01.pn.xcllnt.net> <20030617223910.GB57040@ns1.xcllnt.net> Date: Wed, 18 Jun 2003 07:48:09 +0800 MIME-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 cc: threads@freebsd.org cc: gareth@nvidia.com cc: aritger@nvidia.com Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: David Xu List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 23:45:04 -0000 ----- Original Message -----=20 From: "Marcel Moolenaar" To: "Julian Elischer" Cc: ; ; Sent: Wednesday, June 18, 2003 6:39 AM Subject: Re: Nvidia, TLS and __thread keyword -- an observation > On Tue, Jun 17, 2003 at 03:02:20PM -0700, Julian Elischer wrote: > >=20 > >=20 > > On Tue, 17 Jun 2003, Marcel Moolenaar wrote: > >=20 > > > Guys, > > >=20 > > > In short: Don't bash Nvidia. What they do is not uncommon. Well, > > > maybe in Open Source environments. So please end this thread, > > > unless people get constructive. > >=20 > > I think its already ended.. > >=20 > > basically: > > We should alwasy be able to use (on i386) the sam amethods outlined = for > > solaris.=20 > > Not quite as quick as those for Linux but more general. >=20 > I'm not sure you understand the issue (I can easily be wrong, I just > don't see the evidence in your statement). To support the __thread > keyword, our thread library needs to create the TLS as defined in the > binary and its dependent shared libraries by virtue of the .tdata and > .tbss sections/segments, based on the image of the TLS as constructed > by the RTLD for the initial set of modules (created for the initial > thread) and amended by TLS space defined in the dynamicly loaded > libraries; and the TLS has to be created for every new thread at the > time the thread itself is created. This TLS allocation has to be made > accessable in accordance with runtime specifications for the supported > architectures (libthr: i386 & ia64; libkse: i386 currently -- more to > follow) and in line with the access sequences created by the compiler, > and using the static relocations known to the static linker and = dynamic > relocations of which the support needs to be added to RTLD. >=20 > The static TLS model requires the least amount of work: add support > to allocate the TLS image for every thread creation and point the > thread pointer to it in a way compatible with the runtime spec. >=20 > The dynamic TLS model requires more substantial changes and involves > RTLD as well. This is the model that requires __tls_get_addr(). >=20 I believe this will add overhead to thread creating and destroying, How fast an RTLD can be in this case ? I can create and join 10000=20 threads in 0.2 second here on old Celeron 500 using libkse with hash table added for thread searching. Hope that while OpenGL can get benifit from __thread, it won't punish another style of program --- program does lots of thread creating and destroying and rarely needs to access thread local variables, we want to keep thread creating and destroying as cheap as possible, if we can not obey this rule, then thread creating and destroying become as heavy as process creating, so what benifit=20 can you get when it is as heavy as process ? I am not objecting the __thread idea, just notice you this problem when you try to implement it. > --=20 > Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to = "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 16:57:08 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DA69E37B401 for ; Tue, 17 Jun 2003 16:57:08 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3329143F93 for ; Tue, 17 Jun 2003 16:57:06 -0700 (PDT) (envelope-from ARitger@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Tue, 17 Jun 2003 16:59:44 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Tue, 17 Jun 2003 16:56:21 -0700 Received: from nvidia.com (BERNSTEIN [172.16.224.128]) by mail-sc-0.nvidia.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id MJR3XJRQ; Tue, 17 Jun 2003 16:56:20 -0700 From: Andy Ritger To: Julian Elischer Date: Tue, 17 Jun 2003 20:29:17 -0400 (EDT) In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Alexander Kabaev cc: zander@mail.minion.de cc: Gareth Hughes cc: threads@freebsd.org cc: 'Daniel Eischen' Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2003 23:57:09 -0000 On Tue, 17 Jun 2003, Julian Elischer wrote: > > > On Tue, 17 Jun 2003, Gareth Hughes wrote: > > > Terry Lambert wrote: > > > > > > [a lot of stuff] > > Gareth, thanks for puting up with us and showing considerable patience. And thanks for putting up with Gareth ;) > I think we can make an 'efficient (though possibly not quite the 1 > instruction possible in a static linked linux binary) __thread > implimentation. Sure. > We can possibly do 1 instruction in the 1:1 library but that would leave > binary compatibility with the other libraries out the window. I'll try to keep on eye on the technical discussion (looks like it moved to current@freebsd.org), but please keep us in the loop. For the short term, I'll probably leave the NVIDIA libGL as is (hijacking %gs), and document it as such in our README. Once the TLS decisions are worked out on FreeBSD, we'll propogate the appropriate changes to the NVIDIA OpenGL driver. Thanks, - Andy > I think that it is a pitty that the openGL ABI doesn't carry around a > 'context' argument on all calls but it's not going to change and > we have to use what is there. > > Julian > > > From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 17:36:29 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5B57037B401 for ; Tue, 17 Jun 2003 17:36:29 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8853643FD7 for ; Tue, 17 Jun 2003 17:36:28 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5I0ZuDZ058380; Tue, 17 Jun 2003 17:35:56 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5I0Zust002506; Tue, 17 Jun 2003 17:35:56 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5I0Zutd002505; Tue, 17 Jun 2003 17:35:56 -0700 (PDT) (envelope-from marcel) Date: Tue, 17 Jun 2003 17:35:56 -0700 From: Marcel Moolenaar To: David Xu Message-ID: <20030618003556.GA2440@dhcp01.pn.xcllnt.net> References: <20030617223910.GB57040@ns1.xcllnt.net> <002101c3352a$e931a7f0$0701a8c0@tiger> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <002101c3352a$e931a7f0$0701a8c0@tiger> User-Agent: Mutt/1.5.4i cc: threads@freebsd.org cc: gareth@nvidia.com cc: Julian Elischer cc: aritger@nvidia.com Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 00:36:29 -0000 On Wed, Jun 18, 2003 at 07:48:09AM +0800, David Xu wrote: > > > The static TLS model requires the least amount of work: add support > > to allocate the TLS image for every thread creation and point the > > thread pointer to it in a way compatible with the runtime spec. > > > > The dynamic TLS model requires more substantial changes and involves > > RTLD as well. This is the model that requires __tls_get_addr(). > > > > I believe this will add overhead to thread creating and destroying, > How fast an RTLD can be in this case ? In the dynamic TLS model you would like to delay the creation of the TLS space. Normally __tls_get_addr() gets used for this. In the static TLS model you allocate the TLS when you llocate the thread control structure. Thus, there's virtually no cost. However TLS accesses for the dynamic TLS model are expensive. I have some ideas about that. With some kernel support you can even create dynamic TLS with static TLS code sequences... -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 17:48:35 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 547C737B401 for ; Tue, 17 Jun 2003 17:48:35 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id F41B243F75 for ; Tue, 17 Jun 2003 17:48:33 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5I0m2DZ058419; Tue, 17 Jun 2003 17:48:02 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5I0m1st002540; Tue, 17 Jun 2003 17:48:01 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5I0m1Bp002539; Tue, 17 Jun 2003 17:48:01 -0700 (PDT) (envelope-from marcel) Date: Tue, 17 Jun 2003 17:48:01 -0700 From: Marcel Moolenaar To: Julian Elischer Message-ID: <20030618004801.GB2440@dhcp01.pn.xcllnt.net> References: <20030617223910.GB57040@ns1.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@FreeBSD.org cc: gareth@nvidia.com cc: aritger@nvidia.com Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 00:48:35 -0000 On Tue, Jun 17, 2003 at 04:38:44PM -0700, Julian Elischer wrote: > > The dynamic TLS model requires more substantial changes and involves > > RTLD as well. This is the model that requires __tls_get_addr(). > > By my reading of it, > you can't use the static model alone. > If you have dynamically loaded modules you need to be able to use the > dynamic model. Yes. If you don't support dynamic loading of shared libraries with TLS, you don't need the dynamic model. You can use the static TLS model even with shared objects, provided you allocate the TLS on thread creation. But this is mostly academic. We need to implement them both... -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 17:49:51 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5434F37B401 for ; Tue, 17 Jun 2003 17:49:51 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9B22F43FBF for ; Tue, 17 Jun 2003 17:49:50 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5I0njXh015437; Tue, 17 Jun 2003 20:49:45 -0400 (EDT) Date: Tue, 17 Jun 2003 20:49:45 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Sergey Kosyakov cc: freebsd-threads@freebsd.org Subject: Re: The first kse_create call X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 00:49:51 -0000 On Tue, 17 Jun 2003, Julian Elischer wrote: > > On Tue, 17 Jun 2003, Daniel Eischen wrote: > > This is the correct behavior. The first kse_create() does not > > generate an immediate upcall. It only flags the current context > > as being a KSE. An upcall in this initial KSE will take place > > under the same conditions as other KSEs (KSE mailbox has a thread > > mailbox pointer and thread blocks, quantum expires, etc). > > Subsequent kse_create() calls will generate upcalls immediately > > (well, at the mercy of the kernel scheduler). > > Wellll, the kse_create() will not upcall, but the new kse that is > created will upcall. Right, that's what I meant. > The kse_create() call will just return to the original thread. Yup. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Tue Jun 17 21:08:16 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BF55037B401 for ; Tue, 17 Jun 2003 21:08:16 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 21FCB43FBD for ; Tue, 17 Jun 2003 21:08:14 -0700 (PDT) (envelope-from ARitger@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Tue, 17 Jun 2003 21:06:04 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Tue, 17 Jun 2003 21:07:36 -0700 Received: from nvidia.com (BERNSTEIN [172.16.224.128]) by mail-sc-0.nvidia.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id MJR3XRTV; Tue, 17 Jun 2003 21:07:33 -0700 From: Andy Ritger To: Julian Elischer Date: Wed, 18 Jun 2003 00:40:36 -0400 (EDT) In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: Alexander Kabaev cc: zander@mail.minion.de cc: Gareth Hughes cc: 'Daniel Eischen' Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 04:08:17 -0000 On Tue, 17 Jun 2003, Julian Elischer wrote: > > > On Tue, 17 Jun 2003, Andy Ritger wrote: > > > > > > > On Tue, 17 Jun 2003, Julian Elischer wrote: > > > > > > > > > > > On Tue, 17 Jun 2003, Gareth Hughes wrote: > > > > > > > Terry Lambert wrote: > > > > > > > > > > > > [a lot of stuff] > > > > > > Gareth, thanks for puting up with us and showing considerable patience. > > > > And thanks for putting up with Gareth ;) > > > > > I think we can make an 'efficient (though possibly not quite the 1 > > > instruction possible in a static linked linux binary) __thread > > > implimentation. > > > > Sure. > > > > > We can possibly do 1 instruction in the 1:1 library but that would leave > > > binary compatibility with the other libraries out the window. > > > > I'll try to keep on eye on the technical discussion (looks like it > > moved to current@freebsd.org), but please keep us in the loop. > > > > For the short term, I'll probably leave the NVIDIA libGL as is > > (hijacking %gs), and document it as such in our README. Once the > > TLS decisions are worked out on FreeBSD, we'll propogate the > > appropriate changes to the NVIDIA OpenGL driver. > > It seems to me that what you have done is to "roll-your-own" TLS for > FreeBSD. Can you detail in such a way that we don't have to read the > entire source, what you have done (e.g. with %gs). The way the "home-brewed" TLS in the NVIDIA FreeBSD OpenGL driver currently works is that for each thread we allocate a structure to store all the thread local data we need, plug it into a selector in the LDT by calling i386_set_ldt(), and then set that thread's selector index in %gs. Retrieving the appropriate TLS data is done by looking up the structure from %gs, and looking at the offsetof the relevant field in the structure. Of course, this assumes that %gs is context-switched per thread. I assume the 1:1 threading library does something similar, plugging a structure into an LDT, and then setting the LDT index in %gs? One stopgap would be to designate some padding at a fixed location in your structure, which an OpenGL implementation can "hijack" to store its own TLS data. There were earlier objections to this hack, so I'm not sure if this is something you'd consider as a stopgap... I also don't see how this stopgap could be implemented in the other thread libraries. Thanks, - Andy > We may be able to make a "stopgap" version that can co-exist with the > threading libraries if we know exactly how you were thinking when you > did it.. (i.e. maybe we can 'co-operate' on %gx instead of > just exploding).. Real TLS may take a while if we need to rewrite parts > of our run-time linker. > > > > > > Thanks, > > - Andy > > > > > I think that it is a pitty that the openGL ABI doesn't carry around a > > > 'context' argument on all calls but it's not going to change and > > > we have to use what is there. > > > > > > > > Julian > > > > > > > > > > > > > > > > > _______________________________________________ > > freebsd-threads@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > > > From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 01:37:33 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9C46737B401 for ; Wed, 18 Jun 2003 01:37:33 -0700 (PDT) Received: from obsecurity.dyndns.org (adsl-64-169-104-32.dsl.lsan03.pacbell.net [64.169.104.32]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0317443FA3 for ; Wed, 18 Jun 2003 01:37:33 -0700 (PDT) (envelope-from kris@obsecurity.org) Received: from rot13.obsecurity.org (rot13.obsecurity.org [10.0.0.5]) by obsecurity.dyndns.org (Postfix) with ESMTP id C1FAB66D6A for ; Wed, 18 Jun 2003 01:37:32 -0700 (PDT) Received: by rot13.obsecurity.org (Postfix, from userid 1000) id AAF064DA; Wed, 18 Jun 2003 01:37:32 -0700 (PDT) Date: Wed, 18 Jun 2003 01:37:32 -0700 From: Kris Kennaway To: threads@FreeBSD.org Message-ID: <20030618083732.GA8907@rot13.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="azLHFNyN32YCQGCU" Content-Disposition: inline User-Agent: Mutt/1.4.1i Subject: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 08:37:33 -0000 --azLHFNyN32YCQGCU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Any objections? Kris Index: net/gethostnamadr.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /usr/home/ncvs/src/lib/libc/net/gethostnamadr.c,v retrieving revision 1.21 diff -u -u -r1.21 gethostnamadr.c --- net/gethostnamadr.c 24 Apr 2003 18:05:48 -0000 1.21 +++ net/gethostnamadr.c 18 Jun 2003 08:19:43 -0000 @@ -116,25 +116,6 @@ return hp; } =20 -struct hostent_data; - -/* - * Temporary function (not thread safe) - */ -int gethostbyaddr_r(const char *addr, int len, int type, - struct hostent *result, struct hostent_data *buffer) -{ - struct hostent *hp; - int ret; - if ((hp =3D gethostbyaddr(addr, len, type)) =3D=3D NULL) { - ret =3D -1; - } else { - memcpy(result, hp, sizeof(struct hostent)); - ret =3D 0; - } - return(ret); -} - void sethostent(stayopen) int stayopen; --azLHFNyN32YCQGCU Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (FreeBSD) iD8DBQE+8CTMWry0BWjoQKURArwiAJ9zzjyYyRO2E9nS2yJ8XCbGMC9jegCfYEwu miOgtcWxhukoJUdMhtCv7vo= =DVMe -----END PGP SIGNATURE----- --azLHFNyN32YCQGCU-- From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 01:42:45 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 34F7F37B401 for ; Wed, 18 Jun 2003 01:42:45 -0700 (PDT) Received: from bluejay.mail.pas.earthlink.net (bluejay.mail.pas.earthlink.net [207.217.120.218]) by mx1.FreeBSD.org (Postfix) with ESMTP id 642BC43FBF for ; Wed, 18 Jun 2003 01:42:44 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2ivfl1t.dialup.mindspring.com ([165.247.212.61] helo=mindspring.com) by bluejay.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19SYWd-00049X-00; Wed, 18 Jun 2003 01:42:40 -0700 Message-ID: <3EF02571.8A4C432D@mindspring.com> Date: Wed, 18 Jun 2003 01:40:17 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Marcel Moolenaar References: <20030617071810.GA2451@dhcp01.pn.xcllnt.net> <20030617223910.GB57040@ns1.xcllnt.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4444641a1ab8e5fde2a83e2d1721208dc548b785378294e88350badd9bab72f9c350badd9bab72f9c cc: threads@FreeBSD.org cc: gareth@nvidia.com cc: Julian Elischer cc: aritger@nvidia.com Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 08:42:45 -0000 Marcel Moolenaar wrote: > I'm not sure you understand the issue (I can easily be wrong, I just > don't see the evidence in your statement). To support the __thread > keyword, our thread library needs to create the TLS as defined in the > binary and its dependent shared libraries by virtue of the .tdata and > .tbss sections/segments, based on the image of the TLS as constructed > by the RTLD for the initial set of modules (created for the initial > thread) and amended by TLS space defined in the dynamicly loaded > libraries; and the TLS has to be created for every new thread at the > time the thread itself is created. Most of these issues can be sidestepped. The correct approach is actually the Microsoft published model, which is a rehash of another even older model: For each shared object, be it a library or dlopen'ed .so, you need: 1) Process attach (currently, .init) 2) Process detach (currently, .fini) 3) Thread attach (you are implying this with .tdata and .tbss) 4) Thread detach (you are implying this with .tdata and .tbss) Really, you want an explicit interface, rather than an implied one. This may mean "implying" the creation of .tini and .tfin sections, or some other approach, which deal with the .tdata/.tbss, or otherwise. Actually, a means of putting the relocation table in a per thread code table would also resolve the relocation issues that lead to people wanting to put locks around the RTLD references, but it's probably more correct to resolve this by serializing the thread attach/detach process, instead. > The static TLS model requires the least amount of work: add support > to allocate the TLS image for every thread creation and point the > thread pointer to it in a way compatible with the runtime spec. There would be no difference between static vs. dynamic, for the most part, if one were to use the .tini/.fin approach. The trouble you are anticipating here is actually all related to the fact of you having defined things as belonging to a data interface, rather than using accessor/mutator functions in order to operate and to hook the thread attach/detach events (attaches are implicit for any existing threads at time of load, and detaches are implicit for any existing threads at time of unload -- meaning you need to deal with it in the same fashion as create/delete, and you need to deal with out-of-order; this is a restriction you already have to live with anyway). > The dynamic TLS model requires more substantial changes and involves > RTLD as well. This is the model that requires __tls_get_addr(). I don't believe that this is true. I think the code examples omit the case where you have triggered functions to deal with explicit attach/detach events. And all such events can be made explicit, and serialized. They are rare enough that there should be almost zero cost relative to doing the same thing with static construction, which would use a linker set to gather the the .tini/.tfin function lists together for call on thread start/stop. Now some general comments: Realize that no matter how you approach this, there is going to be additional runtime overhead to thread creation/deletion for implicit TLS support, even if you get referencing it down to 1 instruction. No matter what you do, you will be paying an increased runtime penalty for thread creation/termination (join, exit, etc.) in exchange for your ability to use implicit TLS via compiler extension. I expect that high performance requirement programs that use a lot of threads will lean not to use __thread, in much the same way that C++ programmers lean not to use RTTI or exceptions -- both available language features, whose cost exceed their benefit. I believe that thread lifetime is proportional to the desirability of implicit TLS, and that thread count is inversely proportional. People writing code that expects to be used by threaded programs need to be aware of this. For example, an OpenGL interface onto a MySQL database or an LDAP server or a DNS server would likely suck, due to the thread impedence mismatch between the implementations. Likewise, thread count would make OpenGL undesiragle technology for implementing a web browser that didn't explicitly and heavily rely on a worker thread pool and the HTTP 1.1 persistent connections approach. Even then, you would still be screwed by any web servers that performed chunk-encoding, since it would rob you of the "Content-Length:" header, and mean that in order to signal end of data, they had to close the connection on you, losing you your persistence. Just some things to think about before you throw out explicit context for frame rate improvements that are unusable to any code that isn't a game or a benchmark... 8-(. -- Terry From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 01:50:12 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D91E637B407 for ; Wed, 18 Jun 2003 01:50:12 -0700 (PDT) Received: from bluejay.mail.pas.earthlink.net (bluejay.mail.pas.earthlink.net [207.217.120.218]) by mx1.FreeBSD.org (Postfix) with ESMTP id C03E443F93 for ; Wed, 18 Jun 2003 01:50:11 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2ivfl1t.dialup.mindspring.com ([165.247.212.61] helo=mindspring.com) by bluejay.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19SYds-0004lh-00; Wed, 18 Jun 2003 01:50:09 -0700 Message-ID: <3EF0272C.16D9D299@mindspring.com> Date: Wed, 18 Jun 2003 01:47:40 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Julian Elischer References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a400d9487570c1371517d48034843f82433ca473d225a0f487350badd9bab72f9c350badd9bab72f9c cc: threads@FreeBSD.org cc: Marcel Moolenaar Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 08:50:13 -0000 [ ... removed poor nVidia folks ... ] Julian Elischer wrote: > Libthr and libkse (i386) can be modified to both have the same > support. The spec requires that %gs:0 is the address of a pointer to > the thread control block (TCB). In kse this is easily achieved > by puting such a pointer at the front of the KSE mailbox. (where %gs > points), and in libthr the pointer points directly to the TCB itself > (as far as I see) and there could be a pointer to itself set there. I don't understand this statement. Specifically, %gs points to a KSE that may have multiple threads associated with it, and must therefore have multiple implicit TLS instances associated with it. You can't do this without some sort of indirection, above and beyond just %gs, unless you agree to have a 1:1 relationship between threads and KSEs. -- Terry From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 01:56:21 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AAEB337B401 for ; Wed, 18 Jun 2003 01:56:21 -0700 (PDT) Received: from bluejay.mail.pas.earthlink.net (bluejay.mail.pas.earthlink.net [207.217.120.218]) by mx1.FreeBSD.org (Postfix) with ESMTP id 197F143F75 for ; Wed, 18 Jun 2003 01:56:21 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2ivfl1t.dialup.mindspring.com ([165.247.212.61] helo=mindspring.com) by bluejay.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19SYjc-0005CV-00; Wed, 18 Jun 2003 01:56:05 -0700 Message-ID: <3EF02888.4E72CE7F@mindspring.com> Date: Wed, 18 Jun 2003 01:53:28 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: David Xu References: <20030617071810.GA2451@dhcp01.pn.xcllnt.net> <002101c3352a$e931a7f0$0701a8c0@tiger> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a400d9487570c13715de391cab7bb4f1fc387f7b89c61deb1d350badd9bab72f9c350badd9bab72f9c cc: threads@freebsd.org cc: Julian Elischer cc: Marcel Moolenaar Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 08:56:22 -0000 David Xu wrote: [ ... implicit TLS initialization and destruction ... ] > I believe this will add overhead to thread creating and destroying, Yes. It will. See my other posting for the tradeoffs. The complexity is not as bad as Marcel makes out, but it's still a bad tradeoff for most existing threaded applications and libraries, even if we thing more people will use implict TLS as time goes on. I think it's the wrong tradeoff, too. I would make them burn a general purpose register -- one that they are allowed to burn, as an application -- on FreeBSD. I have a hard time believing that any of their functions use up all the registers, even on a register-poor CISC architecture like x86; if they are that complex, then they are doing so much stuff that explicit TLS access overhead would be lost in the noise. -- Terry From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 02:07:55 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 823F837B401 for ; Wed, 18 Jun 2003 02:07:55 -0700 (PDT) Received: from bluejay.mail.pas.earthlink.net (bluejay.mail.pas.earthlink.net [207.217.120.218]) by mx1.FreeBSD.org (Postfix) with ESMTP id E1A6143FBD for ; Wed, 18 Jun 2003 02:07:54 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2ivfl1t.dialup.mindspring.com ([165.247.212.61] helo=mindspring.com) by bluejay.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19SYuq-00066A-00; Wed, 18 Jun 2003 02:07:41 -0700 Message-ID: <3EF02B40.A4BD1EF@mindspring.com> Date: Wed, 18 Jun 2003 02:05:04 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Marcel Moolenaar References: <20030617223910.GB57040@ns1.xcllnt.net> <20030618003556.GA2440@dhcp01.pn.xcllnt.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4295de0da9206727919bfdccebff7ee12a2d4e88014a4647c350badd9bab72f9c350badd9bab72f9c cc: David Xu cc: Julian Elischer cc: threads@freebsd.org Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 09:07:55 -0000 Marcel Moolenaar wrote: > On Wed, Jun 18, 2003 at 07:48:09AM +0800, David Xu wrote: > > I believe this will add overhead to thread creating and destroying, > > How fast an RTLD can be in this case ? > > In the dynamic TLS model you would like to delay the creation of > the TLS space. Normally __tls_get_addr() gets used for this. In > the static TLS model you allocate the TLS when you llocate the > thread control structure. Lazy binding in this context doesn't make a lot of sense. In the case of a dynamically linked binary, every thread will need the context sooner or later, unless you are running a heterogeneous workload; even then, the locking required at access time would be prohibitive, compared to binding it once at object load time. For modules (e.g. Apache CGI's dlopen'ed and loaded in), the same is true: you will run them eventually, when your thread ends up with a request, unless you do special dancing around creating "uncontaminated" vs. "contaminated" thread pools, and reluctantly migrate from the former to the latter -- i.e.: make special effort to keep threads uncontaminated by having had a particular dynamic object's TLS attached to it. Garbage collection would be ugly, the locking overhead would be just as great, trying to keep the TLS sections dynamically bound, and you would have much more overhead on thread teardown. Forget thread_join()! The code for the poor applicaiton to keep track of this would very much exceed the overhead of the application just passing around a context to the reentrant functions. > Thus, there's virtually no cost. However TLS accesses for the > dynamic TLS model are expensive. I have some ideas about that. > With some kernel support you can even create dynamic TLS with > static TLS code sequences... I think you can only do this if you do *not* lazy-bind, and the refrences are at fixed offsets. This means orphaning sections of TLS data any time you unload a dynamic module that formerly referenced it. Much better to make thread attach/detach as explicit as process attach/detach already is, with the .init/.fini sections that are there for the process, create corresponding ones for the threads. -- Terry From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 04:58:17 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D80EF37B401 for ; Wed, 18 Jun 2003 04:58:17 -0700 (PDT) Received: from web12404.mail.yahoo.com (web12404.mail.yahoo.com [216.136.173.131]) by mx1.FreeBSD.org (Postfix) with SMTP id 2E9D043FE5 for ; Wed, 18 Jun 2003 04:58:17 -0700 (PDT) (envelope-from ks4usa@yahoo.com) Message-ID: <20030618115817.99930.qmail@web12404.mail.yahoo.com> Received: from [12.29.176.205] by web12404.mail.yahoo.com via HTTP; Wed, 18 Jun 2003 04:58:17 PDT Date: Wed, 18 Jun 2003 04:58:17 -0700 (PDT) From: Sergey Kosyakov To: deischen@freebsd.org In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii cc: freebsd-threads@freebsd.org Subject: Re: The first kse_create call X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 11:58:18 -0000 --- Daniel Eischen wrote: > > but I never got upcall on the first kse. When I create another KSE > (and > > another KSE group) I immediatelly get the upcall for this (second) > > This is the correct behavior. The first kse_create() does not > generate an immediate upcall. It only flags the current context > as being a KSE. An upcall in this initial KSE will take place > under the same conditions as other KSEs (KSE mailbox has a thread > mailbox pointer and thread blocks, quantum expires, etc). How I can set the quantum? Is km_quantum from kse_mailbox the right place? I did not get any upcall when I set it. Also found, that "ps" and "top" do not show CPU utilization at least when one KSE with mailbox exists (5.1-RELEASE): 1036 p4 R+ 0:00.00 ./kt Process 1036 runs "printf" in loop. Thanks, Sergey. __________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 05:59:24 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E19BA37B401 for ; Wed, 18 Jun 2003 05:59:24 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 43AA743F93 for ; Wed, 18 Jun 2003 05:59:24 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5ICxNXh027481; Wed, 18 Jun 2003 08:59:23 -0400 (EDT) Date: Wed, 18 Jun 2003 08:59:23 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Kris Kennaway In-Reply-To: <20030618083732.GA8907@rot13.obsecurity.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 12:59:25 -0000 On Wed, 18 Jun 2003, Kris Kennaway wrote: > Any objections? > > Kris Why is this bogus? Do we have another gethostbyaddr_r hiding somewhere? -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 06:08:14 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DD98337B401 for ; Wed, 18 Jun 2003 06:08:14 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id D0E2643FB1 for ; Wed, 18 Jun 2003 06:08:13 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5ID8DXh028967; Wed, 18 Jun 2003 09:08:13 -0400 (EDT) Date: Wed, 18 Jun 2003 09:08:13 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Sergey Kosyakov In-Reply-To: <20030618115817.99930.qmail@web12404.mail.yahoo.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: The first kse_create call X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 13:08:15 -0000 On Wed, 18 Jun 2003, Sergey Kosyakov wrote: > > --- Daniel Eischen wrote: > > > but I never got upcall on the first kse. When I create another KSE > > (and > > > another KSE group) I immediatelly get the upcall for this (second) > > > > This is the correct behavior. The first kse_create() does not > > generate an immediate upcall. It only flags the current context > > as being a KSE. An upcall in this initial KSE will take place > > under the same conditions as other KSEs (KSE mailbox has a thread > > mailbox pointer and thread blocks, quantum expires, etc). > > How I can set the quantum? Is km_quantum from kse_mailbox the right > place? I did not get any upcall when I set it. You have to have both a thread mailbox pointer set in the KSE mailbox and you have to expire the quantum. The quantum is system plus user time. It is not real time (e.g., a nanosleep() does not expire quantum while it sleeps). quantum is in usecs. > Also found, that "ps" and "top" do not show CPU utilization at least > when one KSE with mailbox exists (5.1-RELEASE): > 1036 p4 R+ 0:00.00 ./kt > > Process 1036 runs "printf" in loop. I don't know about ps and top; there have been recent changes to reflect more accurate display of KSE processes. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 06:46:25 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 74BFE37B401 for ; Wed, 18 Jun 2003 06:46:25 -0700 (PDT) Received: from web12401.mail.yahoo.com (web12401.mail.yahoo.com [216.136.173.128]) by mx1.FreeBSD.org (Postfix) with SMTP id C824343F3F for ; Wed, 18 Jun 2003 06:46:24 -0700 (PDT) (envelope-from ks4usa@yahoo.com) Message-ID: <20030618134624.85995.qmail@web12401.mail.yahoo.com> Received: from [12.29.176.205] by web12401.mail.yahoo.com via HTTP; Wed, 18 Jun 2003 06:46:24 PDT Date: Wed, 18 Jun 2003 06:46:24 -0700 (PDT) From: Sergey Kosyakov To: Daniel Eischen In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii cc: freebsd-threads@freebsd.org Subject: Re: The first kse_create call X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 13:46:25 -0000 --- Daniel Eischen wrote: > > You have to have both a thread mailbox pointer set in the > KSE mailbox and you have to expire the quantum. The quantum > is system plus user time. It is not real time (e.g., a nanosleep() > does not expire quantum while it sleeps). I'm trying to do this: mbx.km_version=KSE_VER_0; mbx.km_func=uc_1; mbx.km_stack.ss_sp=stack_1; mbx.km_stack.ss_size=SIGSTKSZ; mbx.km_stack.ss_flags=SS_ONSTACK; ret=getcontext(&(thr_mbx.tm_context)); printf("getcontext %d\n",ret); thr_mbx.tm_context.uc_link=NULL; thr_mbx.tm_context.uc_stack.ss_sp=thr_stack_1; thr_mbx.tm_context.uc_stack.ss_size=SIGSTKSZ; thr_mbx.tm_context.uc_stack.ss_flags=SS_ONSTACK; makecontext(&(thr_mbx.tm_context),func,0); thr_mbx.tm_uticks=10; thr_mbx.tm_sticks=0; mbx.km_curthread=&thr_mbx; ret=kse_create(&mbx,0); do { ++i; }while(1); And never got upcall. (But if I put printf in the loop I got the upcall). Thanks, Sergey. __________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 09:34:33 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C910737B401 for ; Wed, 18 Jun 2003 09:34:33 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0B2F243F75 for ; Wed, 18 Jun 2003 09:34:33 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5IGYWXh000368; Wed, 18 Jun 2003 12:34:32 -0400 (EDT) Date: Wed, 18 Jun 2003 12:34:32 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Sergey Kosyakov In-Reply-To: <20030618134624.85995.qmail@web12401.mail.yahoo.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: The first kse_create call X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 16:34:34 -0000 On Wed, 18 Jun 2003, Sergey Kosyakov wrote: > --- Daniel Eischen wrote: > > > > You have to have both a thread mailbox pointer set in the > > KSE mailbox and you have to expire the quantum. The quantum > > is system plus user time. It is not real time (e.g., a nanosleep() > > does not expire quantum while it sleeps). > > I'm trying to do this: > > mbx.km_version=KSE_VER_0; > mbx.km_func=uc_1; > mbx.km_stack.ss_sp=stack_1; > mbx.km_stack.ss_size=SIGSTKSZ; > mbx.km_stack.ss_flags=SS_ONSTACK; > > ret=getcontext(&(thr_mbx.tm_context)); > printf("getcontext %d\n",ret); If thr_mbx is the initial thread mailbox, you don't need to get a context for it. The kernel will give you back its context on an upcall when the thread resumes after being blocked or after quantum expires; the thread mailbox will be in the KSE mailbox completed list. > thr_mbx.tm_context.uc_link=NULL; > thr_mbx.tm_context.uc_stack.ss_sp=thr_stack_1; > thr_mbx.tm_context.uc_stack.ss_size=SIGSTKSZ; > thr_mbx.tm_context.uc_stack.ss_flags=SS_ONSTACK; > makecontext(&(thr_mbx.tm_context),func,0); Again, you only need to do the above for new threads so that you can start them when you get an upcall and the current thread is either not runnable or you choose to swap it out for another thread. > thr_mbx.tm_uticks=10; > thr_mbx.tm_sticks=0; You (application) don't set tm_uticks or tm_sticks. The kernel does that to let you know how much time the thread has consumed. > mbx.km_curthread=&thr_mbx; > ret=kse_create(&mbx,0); > do > { > ++i; > }while(1); > > And never got upcall. (But if I put printf in the loop I got the > upcall). You're not setting km_quantum. It has to be done before the kse_create() and cannot be changed afterwards (kernel only reads it once I believe). You also have to make sure km_flags is 0. You get the upcall with a printf() because the thread eventually blocks in the kernel. The above spin loop makes no system calls and won't block. Using KSEs on your own is tricky. There is a lot to keep track of. You've got to handle signals on upcalls, keep track of whether the current thread has blocked or not, and atomically set the thread mbx pointer in the KSE when you switch to another thread (If you set the mbx pointer before you switch to the new thread, then if you get an upcall before switching, the kernel may overwrite the thread context before you've finished with it. And it might contain partial context of the KSE context). See src/lib/libpthread/i386/i386/thr_switch.S. It is much easier to just use libpthread^Wlibkse :-) -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 09:43:14 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8F7A137B401 for ; Wed, 18 Jun 2003 09:43:14 -0700 (PDT) Received: from stork.mail.pas.earthlink.net (stork.mail.pas.earthlink.net [207.217.120.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2562643F85 for ; Wed, 18 Jun 2003 09:43:14 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2ivfj3r.dialup.mindspring.com ([165.247.204.123] helo=mindspring.com) by stork.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19Sg1c-0002FW-00; Wed, 18 Jun 2003 09:43:08 -0700 Message-ID: <3EF09656.CF1614A5@mindspring.com> Date: Wed, 18 Jun 2003 09:41:58 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Daniel Eischen References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4ba9a1a37c9fbb3e8bf49df63c05ec713350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c cc: threads@freebsd.org cc: Kris Kennaway Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 16:43:14 -0000 Daniel Eischen wrote: > On Wed, 18 Jun 2003, Kris Kennaway wrote: > > Any objections? > > Why is this bogus? Do we have another gethostbyaddr_r hiding > somewhere? It lies? (_r). -- Terry From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 09:44:26 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BAD4D37B401; Wed, 18 Jun 2003 09:44:26 -0700 (PDT) Received: from sccrmhc13.attbi.com (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id DF41643F75; Wed, 18 Jun 2003 09:44:25 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc13) with ESMTP id <2003061816442401600ahhuhe>; Wed, 18 Jun 2003 16:44:25 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id JAA38731; Wed, 18 Jun 2003 09:44:23 -0700 (PDT) Date: Wed, 18 Jun 2003 09:44:23 -0700 (PDT) From: Julian Elischer To: deischen@freebsd.org In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Sergey Kosyakov cc: freebsd-threads@freebsd.org Subject: Re: The first kse_create call X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 16:44:27 -0000 Sergey, look at: /usr/src/tools/KSE/* the ksetest program just tests basic things the 'rr' program tests teh mechanism for quantum expiry for "round-robin" scheduling. On Wed, 18 Jun 2003, Daniel Eischen wrote: > On Wed, 18 Jun 2003, Sergey Kosyakov wrote: > > --- Daniel Eischen wrote: > > > > > > You have to have both a thread mailbox pointer set in the > > > KSE mailbox and you have to expire the quantum. The quantum > > > is system plus user time. It is not real time (e.g., a nanosleep() > > > does not expire quantum while it sleeps). > > > > I'm trying to do this: > > > > mbx.km_version=KSE_VER_0; > > mbx.km_func=uc_1; > > mbx.km_stack.ss_sp=stack_1; > > mbx.km_stack.ss_size=SIGSTKSZ; > > mbx.km_stack.ss_flags=SS_ONSTACK; > > > > ret=getcontext(&(thr_mbx.tm_context)); > > printf("getcontext %d\n",ret); > > If thr_mbx is the initial thread mailbox, you don't need > to get a context for it. The kernel will give you back > its context on an upcall when the thread resumes after > being blocked or after quantum expires; the thread mailbox > will be in the KSE mailbox completed list. > > > thr_mbx.tm_context.uc_link=NULL; > > thr_mbx.tm_context.uc_stack.ss_sp=thr_stack_1; > > thr_mbx.tm_context.uc_stack.ss_size=SIGSTKSZ; > > thr_mbx.tm_context.uc_stack.ss_flags=SS_ONSTACK; > > makecontext(&(thr_mbx.tm_context),func,0); > > Again, you only need to do the above for new threads so that > you can start them when you get an upcall and the current > thread is either not runnable or you choose to swap it out > for another thread. > > > thr_mbx.tm_uticks=10; > > thr_mbx.tm_sticks=0; > > You (application) don't set tm_uticks or tm_sticks. The kernel > does that to let you know how much time the thread has consumed. > > > mbx.km_curthread=&thr_mbx; > > ret=kse_create(&mbx,0); > > do > > { > > ++i; > > }while(1); > > > > And never got upcall. (But if I put printf in the loop I got the > > upcall). > > You're not setting km_quantum. It has to be done before the > kse_create() and cannot be changed afterwards (kernel only reads > it once I believe). You also have to make sure km_flags is 0. > > You get the upcall with a printf() because the thread eventually > blocks in the kernel. The above spin loop makes no system calls > and won't block. > > Using KSEs on your own is tricky. There is a lot to keep > track of. You've got to handle signals on upcalls, keep > track of whether the current thread has blocked or not, > and atomically set the thread mbx pointer in the KSE when > you switch to another thread (If you set the mbx pointer > before you switch to the new thread, then if you get > an upcall before switching, the kernel may overwrite the > thread context before you've finished with it. And it > might contain partial context of the KSE context). > See src/lib/libpthread/i386/i386/thr_switch.S. > > It is much easier to just use libpthread^Wlibkse :-) > > -- > Dan Eischen > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 10:15:52 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8574C37B401 for ; Wed, 18 Jun 2003 10:15:52 -0700 (PDT) Received: from web12406.mail.yahoo.com (web12406.mail.yahoo.com [216.136.173.133]) by mx1.FreeBSD.org (Postfix) with SMTP id B455643FAF for ; Wed, 18 Jun 2003 10:15:51 -0700 (PDT) (envelope-from ks4usa@yahoo.com) Message-ID: <20030618171551.38062.qmail@web12406.mail.yahoo.com> Received: from [12.29.176.205] by web12406.mail.yahoo.com via HTTP; Wed, 18 Jun 2003 10:15:51 PDT Date: Wed, 18 Jun 2003 10:15:51 -0700 (PDT) From: Sergey Kosyakov To: freebsd-threads@freebsd.org In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: The first kse_create call X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 17:15:52 -0000 --- Daniel Eischen wrote: > On Wed, 18 Jun 2003, Sergey Kosyakov wrote: > > --- Daniel Eischen wrote: > > > > > > You have to have both a thread mailbox pointer set in the > > > KSE mailbox and you have to expire the quantum. The quantum > > > is system plus user time. It is not real time (e.g., a > nanosleep() > > > does not expire quantum while it sleeps). > > > > I'm trying to do this: > > > > mbx.km_version=KSE_VER_0; > > mbx.km_func=uc_1; > > mbx.km_stack.ss_sp=stack_1; > > mbx.km_stack.ss_size=SIGSTKSZ; > > mbx.km_stack.ss_flags=SS_ONSTACK; > > > > ret=getcontext(&(thr_mbx.tm_context)); > > printf("getcontext %d\n",ret); > > If thr_mbx is the initial thread mailbox, you don't need > to get a context for it. The kernel will give you back > its context on an upcall when the thread resumes after > being blocked or after quantum expires; the thread mailbox > will be in the KSE mailbox completed list. > > > thr_mbx.tm_context.uc_link=NULL; > > thr_mbx.tm_context.uc_stack.ss_sp=thr_stack_1; > > thr_mbx.tm_context.uc_stack.ss_size=SIGSTKSZ; > > thr_mbx.tm_context.uc_stack.ss_flags=SS_ONSTACK; > > makecontext(&(thr_mbx.tm_context),func,0); > > Again, you only need to do the above for new threads so that > you can start them when you get an upcall and the current > thread is either not runnable or you choose to swap it out > for another thread. > > > thr_mbx.tm_uticks=10; > > thr_mbx.tm_sticks=0; > > You (application) don't set tm_uticks or tm_sticks. The kernel > does that to let you know how much time the thread has consumed. > > > mbx.km_curthread=&thr_mbx; > > ret=kse_create(&mbx,0); > > do > > { > > ++i; > > }while(1); > > > > And never got upcall. (But if I put printf in the loop I got the > > upcall). > > You're not setting km_quantum. It has to be done before the > kse_create() and cannot be changed afterwards (kernel only reads > it once I believe). You also have to make sure km_flags is 0. Yes, it works. I just was confused by your words "The quantum is system plus user time" and did not clear flags. Thanks. > > It is much easier to just use libpthread^Wlibkse :-) I'm just trying to understand can direct kse usage give additional benefits for my needs or not. Also I did not see anything like KSE/SA approach before, so now I have a chance to toy with it! Thanks, Sergey. __________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 11:17:30 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B45A737B401 for ; Wed, 18 Jun 2003 11:17:30 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 05D3C43F75 for ; Wed, 18 Jun 2003 11:17:30 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5IIHTXh016939; Wed, 18 Jun 2003 14:17:29 -0400 (EDT) Date: Wed, 18 Jun 2003 14:17:29 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Terry Lambert In-Reply-To: <3EF09656.CF1614A5@mindspring.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: Kris Kennaway Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 18:17:31 -0000 On Wed, 18 Jun 2003, Terry Lambert wrote: > Daniel Eischen wrote: > > On Wed, 18 Jun 2003, Kris Kennaway wrote: > > > Any objections? > > > > Why is this bogus? Do we have another gethostbyaddr_r hiding > > somewhere? > > It lies? (_r). If that's true, then it's a bug and eventually should be fixed. Additionally, you can't go around removing public interfaces without bumping library versions (unless said interface hasn't seen a release yet). -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 11:27:14 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9865537B401 for ; Wed, 18 Jun 2003 11:27:14 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.bayarea.net [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 798FE43F93 for ; Wed, 18 Jun 2003 11:27:13 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from ns1.xcllnt.net (localhost [127.0.0.1]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5IIQfDZ063759; Wed, 18 Jun 2003 11:26:41 -0700 (PDT) (envelope-from marcel@ns1.xcllnt.net) Received: (from marcel@localhost) by ns1.xcllnt.net (8.12.9/8.12.9/Submit) id h5IIQcde063758; Wed, 18 Jun 2003 11:26:38 -0700 (PDT) Date: Wed, 18 Jun 2003 11:26:38 -0700 From: Marcel Moolenaar To: Terry Lambert Message-ID: <20030618182638.GA63660@ns1.xcllnt.net> References: <20030617223910.GB57040@ns1.xcllnt.net> <002101c3352a$e931a7f0$0701a8c0@tiger> <20030618003556.GA2440@dhcp01.pn.xcllnt.net> <3EF02B40.A4BD1EF@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3EF02B40.A4BD1EF@mindspring.com> User-Agent: Mutt/1.5.1i cc: David Xu cc: Julian Elischer cc: threads@freebsd.org Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 18:27:14 -0000 On Wed, Jun 18, 2003 at 02:05:04AM -0700, Terry Lambert wrote: > Marcel Moolenaar wrote: > > On Wed, Jun 18, 2003 at 07:48:09AM +0800, David Xu wrote: > > > I believe this will add overhead to thread creating and destroying, > > > How fast an RTLD can be in this case ? > > > > In the dynamic TLS model you would like to delay the creation of > > the TLS space. Normally __tls_get_addr() gets used for this. In > > the static TLS model you allocate the TLS when you llocate the > > thread control structure. > > Lazy binding in this context doesn't make a lot of sense. It does. In a process with 1000 threads where 1 thread does a dlopen(), you don't want to create 999 TLS spaces if they're not going to be used. Besides time, this also is a space issue. Note also that I don't advocate what I think we should do, but what the specification is designed for. People have put some thought in it... -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 12:51:14 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4651637B401 for ; Wed, 18 Jun 2003 12:51:14 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8F0B843FB1 for ; Wed, 18 Jun 2003 12:51:13 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5IJpCXh003462; Wed, 18 Jun 2003 15:51:12 -0400 (EDT) Date: Wed, 18 Jun 2003 15:51:12 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Andy Ritger In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: Gareth Hughes Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 19:51:14 -0000 On Sat, 14 Jun 2003, Andy Ritger wrote: > The current NVIDIA FreeBSD driver only supports one threading library > on FreeBSD for thread-safe, multi-threaded OpenGL applications: the > FreeBSD port of linux-threads. The NVIDIA FreeBSD OpenGL driver uses > both the i386_set_ldt system call and %gs to support high performance > native OpenGL applications. One question. How does using %gs work in libc_r? Thread switches in libc_r use setjmp()/longjmp() neither of which save and restore %gs. If OpenGL sets %gs, libc_r will not change it when threads are switched. Is NVIDIA's OpenGL suppose to be thread-safe for libc_r? I don't see how it can be. What am I missing? -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 13:09:23 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E57EA37B407 for ; Wed, 18 Jun 2003 13:09:23 -0700 (PDT) Received: from hqemgate00.nvidia.com (hqemgate00.nvidia.com [216.228.112.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 60D5C43FE0 for ; Wed, 18 Jun 2003 13:09:23 -0700 (PDT) (envelope-from gareth@nvidia.com) Received: from mail-sc-0.nvidia.com (Not Verified[172.16.217.105]) id ; Wed, 18 Jun 2003 13:07:39 -0700 Received: by mail-sc-0.nvidia.com with Internet Mail Service (5.5.2653.19) id ; Wed, 18 Jun 2003 13:09:11 -0700 Message-ID: <2D32959E172B8F4D9B02F68266BE421401A6D803@mail-sc-3.nvidia.com> From: Gareth Hughes To: 'Daniel Eischen' , Andy Ritger Date: Wed, 18 Jun 2003 13:09:09 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain cc: threads@freebsd.org cc: zander@mail.minion.de Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 20:09:24 -0000 On Wed, 18 Jun 2003, Daniel Eischen wrote: > > On Sat, 14 Jun 2003, Andy Ritger wrote: > > The current NVIDIA FreeBSD driver only supports one threading library > > on FreeBSD for thread-safe, multi-threaded OpenGL applications: the > > FreeBSD port of linux-threads. The NVIDIA FreeBSD OpenGL driver uses > > both the i386_set_ldt system call and %gs to support high performance > > native OpenGL applications. > > One question. How does using %gs work in libc_r? Thread switches > in libc_r use setjmp()/longjmp() neither of which save and restore > %gs. If OpenGL sets %gs, libc_r will not change it when threads are > switched. > > Is NVIDIA's OpenGL suppose to be thread-safe for libc_r? I don't > see how it can be. What am I missing? To quote the quote from Andy you used: The current NVIDIA FreeBSD driver only supports one threading library on FreeBSD for thread-safe, multi-threaded OpenGL applications: the FreeBSD port of linux-threads. -- Gareth Hughes (gareth@nvidia.com) OpenGL Developer, NVIDIA Corporation From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 13:14:24 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F166D37B401 for ; Wed, 18 Jun 2003 13:14:23 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4F94443FB1 for ; Wed, 18 Jun 2003 13:14:21 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5IKEIXh007580; Wed, 18 Jun 2003 16:14:20 -0400 (EDT) Date: Wed, 18 Jun 2003 16:14:18 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Gareth Hughes In-Reply-To: <2D32959E172B8F4D9B02F68266BE421401A6D803@mail-sc-3.nvidia.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: zander@mail.minion.de cc: Andy Ritger Subject: RE: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 20:14:24 -0000 On Wed, 18 Jun 2003, Gareth Hughes wrote: > On Wed, 18 Jun 2003, Daniel Eischen wrote: > > > > On Sat, 14 Jun 2003, Andy Ritger wrote: > > > The current NVIDIA FreeBSD driver only supports one threading library > > > on FreeBSD for thread-safe, multi-threaded OpenGL applications: the > > > FreeBSD port of linux-threads. The NVIDIA FreeBSD OpenGL driver uses > > > both the i386_set_ldt system call and %gs to support high performance > > > native OpenGL applications. > > > > One question. How does using %gs work in libc_r? Thread switches > > in libc_r use setjmp()/longjmp() neither of which save and restore > > %gs. If OpenGL sets %gs, libc_r will not change it when threads are > > switched. > > > > Is NVIDIA's OpenGL suppose to be thread-safe for libc_r? I don't > > see how it can be. What am I missing? > > To quote the quote from Andy you used: > > The current NVIDIA FreeBSD driver only supports one threading library > on FreeBSD for thread-safe, multi-threaded OpenGL applications: the > FreeBSD port of linux-threads. Hmm, for some reason I was under the impression that it was supposedly thread-safe with libc_r. Oh well, go figure. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 13:19:05 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BF44937B401 for ; Wed, 18 Jun 2003 13:19:05 -0700 (PDT) Received: from wall.polstra.com (wall-gw.polstra.com [206.213.73.130]) by mx1.FreeBSD.org (Postfix) with ESMTP id CB25143FB1 for ; Wed, 18 Jun 2003 13:19:04 -0700 (PDT) (envelope-from jdp@polstra.com) Received: from strings.polstra.com (strings.polstra.com [206.213.73.20]) by wall.polstra.com (8.12.3p2/8.12.3) with ESMTP id h5IKJ3PM058736 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Wed, 18 Jun 2003 13:19:03 -0700 (PDT) (envelope-from jdp@strings.polstra.com) Received: (from jdp@localhost) by strings.polstra.com (8.12.6/8.12.6/Submit) id h5IKJ3Fa023295; Wed, 18 Jun 2003 13:19:03 -0700 (PDT) (envelope-from jdp) Date: Wed, 18 Jun 2003 13:19:03 -0700 (PDT) Message-Id: <200306182019.h5IKJ3Fa023295@strings.polstra.com> To: threads@freebsd.org From: John Polstra In-Reply-To: References: Organization: Polstra & Co., Seattle, WA X-Bogosity: No, tests=bogofilter, spamicity=0.026480, version=0.11.2 Subject: Re: The first kse_create call X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 20:19:06 -0000 In article , Daniel Eischen wrote: > quantum is in usecs. The comment in says it is msecs. Which is correct? Looking at kern_thread.c, I think you're right and the comment is wrong. John -- John Polstra John D. Polstra & Co., Inc. Seattle, Washington USA "Two buttocks cannot avoid friction." -- Malawi saying From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 14:08:13 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AAD2A37B401 for ; Wed, 18 Jun 2003 14:08:13 -0700 (PDT) Received: from obsecurity.dyndns.org (adsl-64-169-104-32.dsl.lsan03.pacbell.net [64.169.104.32]) by mx1.FreeBSD.org (Postfix) with ESMTP id F0FE843FBD for ; Wed, 18 Jun 2003 14:08:12 -0700 (PDT) (envelope-from kris@obsecurity.org) Received: from rot13.obsecurity.org (rot13.obsecurity.org [10.0.0.5]) by obsecurity.dyndns.org (Postfix) with ESMTP id B47B866BE5; Wed, 18 Jun 2003 14:08:12 -0700 (PDT) Received: by rot13.obsecurity.org (Postfix, from userid 1000) id 8E091759; Wed, 18 Jun 2003 14:08:12 -0700 (PDT) Date: Wed, 18 Jun 2003 14:08:12 -0700 From: Kris Kennaway To: Daniel Eischen Message-ID: <20030618210812.GB21622@rot13.obsecurity.org> References: <20030618083732.GA8907@rot13.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="fUYQa+Pmc3FrFX/N" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.1i cc: threads@freebsd.org cc: Kris Kennaway Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 21:08:13 -0000 --fUYQa+Pmc3FrFX/N Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jun 18, 2003 at 08:59:23AM -0400, Daniel Eischen wrote: > On Wed, 18 Jun 2003, Kris Kennaway wrote: >=20 > > Any objections? > >=20 > > Kris >=20 > Why is this bogus? Do we have another gethostbyaddr_r hiding > somewhere? Nope, but this one isn't thread-safe, and it's not prototyped anywhere (see my mail to current from a few days ago). We don't have any of the other getfoobybar_r functions either. Kris --fUYQa+Pmc3FrFX/N Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (FreeBSD) iD8DBQE+8NS8Wry0BWjoQKURAgNUAJwOcg2SH6jebZHo8n3BT4Pb486AuQCg1bqE K4DOO9crqM7Jb3gQ4Wl0p08= =LFTs -----END PGP SIGNATURE----- --fUYQa+Pmc3FrFX/N-- From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 14:12:16 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DBF6937B401 for ; Wed, 18 Jun 2003 14:12:16 -0700 (PDT) Received: from obsecurity.dyndns.org (adsl-64-169-104-32.dsl.lsan03.pacbell.net [64.169.104.32]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2526C43F93 for ; Wed, 18 Jun 2003 14:12:16 -0700 (PDT) (envelope-from kris@obsecurity.org) Received: from rot13.obsecurity.org (rot13.obsecurity.org [10.0.0.5]) by obsecurity.dyndns.org (Postfix) with ESMTP id EA93766E2B; Wed, 18 Jun 2003 14:12:15 -0700 (PDT) Received: by rot13.obsecurity.org (Postfix, from userid 1000) id D20354DA; Wed, 18 Jun 2003 14:12:15 -0700 (PDT) Date: Wed, 18 Jun 2003 14:12:15 -0700 From: Kris Kennaway To: Daniel Eischen Message-ID: <20030618211215.GC21622@rot13.obsecurity.org> References: <3EF09656.CF1614A5@mindspring.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="V0207lvV8h4k8FAm" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.1i cc: threads@freebsd.org cc: Kris Kennaway Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 21:12:17 -0000 --V0207lvV8h4k8FAm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jun 18, 2003 at 02:17:29PM -0400, Daniel Eischen wrote: > On Wed, 18 Jun 2003, Terry Lambert wrote: >=20 > > Daniel Eischen wrote: > > > On Wed, 18 Jun 2003, Kris Kennaway wrote: > > > > Any objections? > > >=20 > > > Why is this bogus? Do we have another gethostbyaddr_r hiding > > > somewhere? > >=20 > > It lies? (_r). >=20 > If that's true, then it's a bug and eventually should > be fixed. Additionally, you can't go around removing > public interfaces without bumping library versions > (unless said interface hasn't seen a release yet). It's not a public interface since it's not prototyped, but a number of ports (>27) are finding it anyway and then presumably failing to work correctly because they think they're getting a re-entrant gethostbyaddr() but are not (alternatively, because they're guessing the wrong prototype and will fail to work due to LP64 issues). Please look into the CVS history of this change - it was not mentioned in the commit log by julian, and it looks to me like it was committed accidentally (Julian hasn't responded to 2 mails enquiring about it though). Kris --V0207lvV8h4k8FAm Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (FreeBSD) iD8DBQE+8NWvWry0BWjoQKURAh2+AJ9ZMtIL2xqYYY4JWgBwV8EmJtl3JACfUn/t /qXD5SwHI4uOzoGKyrf7A2A= =8jTt -----END PGP SIGNATURE----- --V0207lvV8h4k8FAm-- From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 15:05:20 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E030537B41A; Wed, 18 Jun 2003 15:05:19 -0700 (PDT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id E052243FA3; Wed, 18 Jun 2003 15:05:01 -0700 (PDT) (envelope-from davidxu@freebsd.org) Received: from tiger (davidxu@localhost [127.0.0.1]) by freefall.freebsd.org (8.12.9/8.12.9) with SMTP id h5IM4wUp076476; Wed, 18 Jun 2003 15:04:59 -0700 (PDT) (envelope-from davidxu@freebsd.org) Message-ID: <001e01c335e6$1ccca630$0701a8c0@tiger> From: "David Xu" To: "Daniel Eischen" , "Terry Lambert" References: Date: Thu, 19 Jun 2003 06:08:09 +0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 cc: threads@freebsd.org cc: Kris Kennaway Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: David Xu List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 22:05:20 -0000 ----- Original Message -----=20 From: "Daniel Eischen" To: "Terry Lambert" Cc: ; "Kris Kennaway" Sent: Thursday, June 19, 2003 2:17 AM Subject: Re: Removal of bogus gethostbyaddr_r() > On Wed, 18 Jun 2003, Terry Lambert wrote: >=20 > > Daniel Eischen wrote: > > > On Wed, 18 Jun 2003, Kris Kennaway wrote: > > > > Any objections? > > >=20 > > > Why is this bogus? Do we have another gethostbyaddr_r hiding > > > somewhere? > >=20 > > It lies? (_r). >=20 > If that's true, then it's a bug and eventually should > be fixed. Additionally, you can't go around removing > public interfaces without bumping library versions > (unless said interface hasn't seen a release yet). >=20 I think those *_r interfaces are not very useful except bloating library. why do not improve non _r version to use thread local instead to make them reentrant between threads ? > --=20 > Dan Eischen >=20 > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to = "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 15:19:32 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A51A837B401 for ; Wed, 18 Jun 2003 15:19:32 -0700 (PDT) Received: from rwcrmhc12.attbi.com (rwcrmhc12.comcast.net [216.148.227.85]) by mx1.FreeBSD.org (Postfix) with ESMTP id 37FBE43F93 for ; Wed, 18 Jun 2003 15:19:32 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (rwcrmhc12) with ESMTP id <2003061822193101400fqcmhe>; Wed, 18 Jun 2003 22:19:31 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id PAA40890; Wed, 18 Jun 2003 15:19:31 -0700 (PDT) Date: Wed, 18 Jun 2003 15:19:30 -0700 (PDT) From: Julian Elischer To: Marcel Moolenaar In-Reply-To: <20030618182638.GA63660@ns1.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: David Xu cc: threads@freebsd.org Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 22:19:32 -0000 Marcel, are you (or do you kno of anyone else) doing anything on TLS? On Wed, 18 Jun 2003, Marcel Moolenaar wrote: > On Wed, Jun 18, 2003 at 02:05:04AM -0700, Terry Lambert wrote: > > Marcel Moolenaar wrote: > > > On Wed, Jun 18, 2003 at 07:48:09AM +0800, David Xu wrote: > > > > I believe this will add overhead to thread creating and destroying, > > > > How fast an RTLD can be in this case ? > > > > > > In the dynamic TLS model you would like to delay the creation of > > > the TLS space. Normally __tls_get_addr() gets used for this. In > > > the static TLS model you allocate the TLS when you llocate the > > > thread control structure. > > > > Lazy binding in this context doesn't make a lot of sense. > > It does. In a process with 1000 threads where 1 thread does > a dlopen(), you don't want to create 999 TLS spaces if they're > not going to be used. Besides time, this also is a space > issue. > > Note also that I don't advocate what I think we should do, but > what the specification is designed for. People have put some > thought in it... > > -- > Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net > From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 15:55:52 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0349837B401 for ; Wed, 18 Jun 2003 15:55:52 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.bayarea.net [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 33CB943FB1 for ; Wed, 18 Jun 2003 15:55:51 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from ns1.xcllnt.net (localhost [127.0.0.1]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5IMtHDZ065137; Wed, 18 Jun 2003 15:55:17 -0700 (PDT) (envelope-from marcel@ns1.xcllnt.net) Received: (from marcel@localhost) by ns1.xcllnt.net (8.12.9/8.12.9/Submit) id h5IMtHn7065136; Wed, 18 Jun 2003 15:55:17 -0700 (PDT) Date: Wed, 18 Jun 2003 15:55:17 -0700 From: Marcel Moolenaar To: Julian Elischer Message-ID: <20030618225517.GA64374@ns1.xcllnt.net> References: <20030618182638.GA63660@ns1.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.1i cc: David Xu cc: threads@freebsd.org Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 22:55:52 -0000 On Wed, Jun 18, 2003 at 03:19:30PM -0700, Julian Elischer wrote: > > Marcel, are you (or do you kno of anyone else) doing anything on TLS? Not in the context of FreeBSD. I used to be involved 2 years ago (roughly) in my daytime job and have experience from a tool-chain point of view and have been close to the fire WRT RTLD and pthread work. If we think this is important to have for 5.2, I can make time to work on it. I do ask for a good planning in that case, because there's a lot of work on my plate that I cannot yet delegate to others (read: ia64 work)... We can approach it like this (just an example to bootstrap the work): o Agree on a version of GCC we'll use during prototyping or wait for GCC to be updated in the tree. o Implement static TLS in libthr. o Test. o Implement dynamic TLS in libthr and RTLD. Concurrently implement static TLS in libkse. o Test static TLS in libkse. o Finish dynamic TLS in libthr/RTLD and start dynamic TLS in libkse. o Test libthr (finish dynamic TLS in libkse). o Test libkse. I deliberately don't want to deal with libc_r, but if people think we should make that work too than we simply don't deal with libc_r on ia64. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 16:16:03 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3C2E537B401 for ; Wed, 18 Jun 2003 16:16:03 -0700 (PDT) Received: from mail.allcaps.org (allcaps.org [216.240.173.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id CAA8943FA3 for ; Wed, 18 Jun 2003 16:16:00 -0700 (PDT) (envelope-from bsder@allcaps.org) Received: from mail.allcaps.org (localhost [127.0.0.1]) by mail.allcaps.org (Postfix) with ESMTP id F206A92FAF; Wed, 18 Jun 2003 19:16:02 -0400 (EDT) Received: from localhost (bsder@localhost)h5ING2aP011482; Wed, 18 Jun 2003 16:16:02 -0700 X-Authentication-Warning: mail.allcaps.org: bsder owned process doing -bs Date: Wed, 18 Jun 2003 16:16:02 -0700 (PDT) From: "Andrew P. Lentvorski, Jr." To: Kris Kennaway In-Reply-To: <20030618083732.GA8907@rot13.obsecurity.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@FreeBSD.org Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 23:16:03 -0000 On Wed, 18 Jun 2003, Kris Kennaway wrote: > Any objections? > > Kris Please do. This already tripped up mozilla as part of the ongoing attempt to parallelize DNS lookups. See: [Bug 70213] XP_UNIX DNS lookups are serialized http://bugzilla.mozilla.org/show_bug.cgi?id=70213 -a From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 16:18:12 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 436D437B401 for ; Wed, 18 Jun 2003 16:18:12 -0700 (PDT) Received: from h132-197-179-27.gte.com (h132-197-179-27.gte.com [132.197.179.27]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4A81B43FA3 for ; Wed, 18 Jun 2003 16:18:11 -0700 (PDT) (envelope-from ak03@gte.com) Received: from kanpc.gte.com (ak03@localhost [127.0.0.1]) h5INHXTO071359; Wed, 18 Jun 2003 19:17:33 -0400 (EDT) (envelope-from ak03@kanpc.gte.com) Received: (from ak03@localhost) by kanpc.gte.com (8.12.9/8.12.9/Submit) id h5INHSGN071358; Wed, 18 Jun 2003 19:17:28 -0400 (EDT) Date: Wed, 18 Jun 2003 19:17:28 -0400 From: Alexander Kabaev To: Marcel Moolenaar Message-Id: <20030618191728.2fc32bd9.ak03@gte.com> In-Reply-To: <20030618225517.GA64374@ns1.xcllnt.net> References: <20030618182638.GA63660@ns1.xcllnt.net> <20030618225517.GA64374@ns1.xcllnt.net> Organization: Verizon Data Services X-Mailer: Sylpheed version 0.9.0claws25 (GTK+ 1.2.10; i386-portbld-freebsd5.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit cc: David Xu cc: Julian Elischer cc: threads@freebsd.org Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 23:18:12 -0000 On Wed, 18 Jun 2003 15:55:17 -0700 Marcel Moolenaar wrote: > On Wed, Jun 18, 2003 at 03:19:30PM -0700, Julian Elischer wrote: > > > > Marcel, are you (or do you kno of anyone else) doing anything on > > TLS? > > We can approach it like this (just an example to bootstrap the > work): > > o Agree on a version of GCC we'll use during prototyping or wait > for GCC to be updated in the tree. > o Implement static TLS in libthr. > o Test. > o Implement dynamic TLS in libthr and RTLD. Concurrently implement > static TLS in libkse. > o Test static TLS in libkse. > o Finish dynamic TLS in libthr/RTLD and start dynamic TLS in libkse. > o Test libthr (finish dynamic TLS in libkse). > o Test libkse. > > I deliberately don't want to deal with libc_r, but if people think > we should make that work too than we simply don't deal with libc_r > on ia64. I announced my intention to work on rtld side of TLS support some time ago and I already have some work in progress. Interested parties can easily compile GCC 3.3 port with TLS support. Run 'make patch' in the ports directory and then edit work/gcc-3.3/gcc/configure to look for binutils 2.13 instead of 2.14 while testing for the platform TLS support. Compile and install the port. -- Alexander Kabaev From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 16:29:50 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AECA137B401 for ; Wed, 18 Jun 2003 16:29:50 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.bayarea.net [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 15A5E43F93 for ; Wed, 18 Jun 2003 16:29:50 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from ns1.xcllnt.net (localhost [127.0.0.1]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5INTIDZ065342; Wed, 18 Jun 2003 16:29:18 -0700 (PDT) (envelope-from marcel@ns1.xcllnt.net) Received: (from marcel@localhost) by ns1.xcllnt.net (8.12.9/8.12.9/Submit) id h5INTINu065341; Wed, 18 Jun 2003 16:29:18 -0700 (PDT) Date: Wed, 18 Jun 2003 16:29:18 -0700 From: Marcel Moolenaar To: Julian Elischer Message-ID: <20030618232918.GA65167@ns1.xcllnt.net> References: <20030618225517.GA64374@ns1.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.1i cc: David Xu cc: threads@freebsd.org Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 23:29:51 -0000 On Wed, Jun 18, 2003 at 04:13:16PM -0700, Julian Elischer wrote: > > > I deliberately don't want to deal with libc_r, but if people think > > we should make that work too than we simply don't deal with libc_r > > on ia64. > > unfortunatly we need to have binary compatibility between > lic_r and libkse/libthr for a while until they settle down. Ok. We'll treat ia64 as the exception then. We need to stop abusing setjmp()/longjmp() on ia64 ASAP. Whose going to look at i386 and other platforms? I mainly like to focus on ia64 while being supportive in all other cases (instead of being the driving force). -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 16:33:57 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 982CE37B401 for ; Wed, 18 Jun 2003 16:33:57 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.bayarea.net [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id EC77143F3F for ; Wed, 18 Jun 2003 16:33:56 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from ns1.xcllnt.net (localhost [127.0.0.1]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5INXPDZ065371; Wed, 18 Jun 2003 16:33:25 -0700 (PDT) (envelope-from marcel@ns1.xcllnt.net) Received: (from marcel@localhost) by ns1.xcllnt.net (8.12.9/8.12.9/Submit) id h5INXP27065370; Wed, 18 Jun 2003 16:33:25 -0700 (PDT) Date: Wed, 18 Jun 2003 16:33:25 -0700 From: Marcel Moolenaar To: Alexander Kabaev Message-ID: <20030618233325.GB65167@ns1.xcllnt.net> References: <20030618182638.GA63660@ns1.xcllnt.net> <20030618225517.GA64374@ns1.xcllnt.net> <20030618191728.2fc32bd9.ak03@gte.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030618191728.2fc32bd9.ak03@gte.com> User-Agent: Mutt/1.5.1i cc: David Xu cc: Julian Elischer cc: threads@freebsd.org Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 23:33:57 -0000 On Wed, Jun 18, 2003 at 07:17:28PM -0400, Alexander Kabaev wrote: > > I announced my intention to work on rtld side of TLS support some time > ago and I already have some work in progress. Cool! That's settled then. Do you have an estimate as to when you you have something alpha/beta for us to play with? > Interested parties can easily compile GCC 3.3 port with TLS support. > Run 'make patch' in the ports directory and then edit > work/gcc-3.3/gcc/configure to look for binutils 2.13 instead of 2.14 > while testing for the platform TLS support. Compile and install the > port. Sounds good. Let's settle for that for now. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 19:12:09 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BE60C37B401 for ; Wed, 18 Jun 2003 19:12:09 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0EE5D43F93 for ; Wed, 18 Jun 2003 19:12:09 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5J2C8Xh004553; Wed, 18 Jun 2003 22:12:08 -0400 (EDT) Date: Wed, 18 Jun 2003 22:12:08 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Kris Kennaway In-Reply-To: <20030618210812.GB21622@rot13.obsecurity.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 02:12:10 -0000 On Wed, 18 Jun 2003, Kris Kennaway wrote: > On Wed, Jun 18, 2003 at 08:59:23AM -0400, Daniel Eischen wrote: > > On Wed, 18 Jun 2003, Kris Kennaway wrote: > > > > > Any objections? > > > > > > Kris > > > > Why is this bogus? Do we have another gethostbyaddr_r hiding > > somewhere? > > Nope, but this one isn't thread-safe, and it's not prototyped anywhere > (see my mail to current from a few days ago). We don't have any of > the other getfoobybar_r functions either. Well, I guess we should probably fix the prototype and make it thread-safe, but if ports are failing because they pick it up, then I don't have a problem with removing it. What's there is simple enough to add back once it has the correct API and is really thread-safe. My $.02. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 19:18:13 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 19CA837B401 for ; Wed, 18 Jun 2003 19:18:13 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 78F6743F93 for ; Wed, 18 Jun 2003 19:18:12 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5J2IAXh005341; Wed, 18 Jun 2003 22:18:11 -0400 (EDT) Date: Wed, 18 Jun 2003 22:18:10 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: David Xu In-Reply-To: <001e01c335e6$1ccca630$0701a8c0@tiger> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: Kris Kennaway Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 02:18:13 -0000 On Thu, 19 Jun 2003, David Xu wrote: > From: "Daniel Eischen" > > If that's true, then it's a bug and eventually should > > be fixed. Additionally, you can't go around removing > > public interfaces without bumping library versions > > (unless said interface hasn't seen a release yet). > > > > I think those *_r interfaces are not very useful except > bloating library. why do not improve non _r version to use > thread local instead to make them reentrant between threads ? The _r versions are expected and part of POSIX (in general, I'll haven't checked to see if gethostbyaddr_r is part of POSIX, but it does exist in Solaris). So it's necessary to have the _r versions, at least those specified by the POSIX spec. We just need to make sure we provide the correct prototypes for them and ensure they are indeed thread-safe. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 22:59:35 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8BA8A37B401; Wed, 18 Jun 2003 22:59:35 -0700 (PDT) Received: from out003.verizon.net (out003pub.verizon.net [206.46.170.103]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8A10443F93; Wed, 18 Jun 2003 22:59:34 -0700 (PDT) (envelope-from mtm@identd.net) Received: from kokeb.ambesa.net ([138.88.125.205]) by out003.verizon.net (InterMail vM.5.01.05.33 201-253-122-126-133-20030313) with ESMTP id <20030619055933.FNBD4805.out003.verizon.net@kokeb.ambesa.net>; Thu, 19 Jun 2003 00:59:33 -0500 Date: Thu, 19 Jun 2003 01:59:32 -0400 From: Mike Makonnen To: Daniel Eischen In-Reply-To: References: <20030618210812.GB21622@rot13.obsecurity.org> X-Mailer: Sylpheed version 0.8.10 (GTK+ 1.2.10; i386-portbld-freebsd5.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Authentication-Info: Submitted using SMTP AUTH at out003.verizon.net from [138.88.125.205] at Thu, 19 Jun 2003 00:59:33 -0500 Message-Id: <20030619055933.FNBD4805.out003.verizon.net@kokeb.ambesa.net> cc: nectar@freeBSD.org cc: threads@FreeBSD.org cc: kris@obsecurity.org Subject: making nsswitch(3) thread-safe (was Re: Removal of bogus gethostbyaddr_r()) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 05:59:35 -0000 [ Jacques, I'm CC'ing you since this affects nss ] I just took a look into what it would take to make the gethostby*_r functions thread safe, and it isn't pretty. The "thread-unsafeness" goes all the way down into the individual methods for the various nsswitch(3) databases (dns, files, etc). So, the most expedient and least invasive way to go about it would be to put a mutex in the gethostby* functions that is dependent on __isthreaded being true. This, offcourse, means that the entire call-chain (from top to bottom) is locked for the duration of a call to one of these functions, which can be a considerable amount of time if it has to timeout. During this time no other thread in the application can resolve a host because it will be blocked on said mutex. To my mind the way to fix this properly is to make the nsdispatch(3) call-chain in libc thread-safe from the ground-up. This is a huge task in and of itself, but is complicated even further by the fact that whatever we do can't change the semantics of the existing user-visible functions. I think there are three points to keep in mind: 1. The nsswitch(3) database functions should be made thread-safe, but at the same time not change the thread-unsafe semantics that third-party apps might expect (through nsdispatch(3)). 2. The thread-unsafeness starts at the bottom (low-level functions) of the call-chain, in the nsswitch(3) database functions. 3. Because so many databases (hosts, passwd, group, etc) will be affected by this it would be too huge a task to do it all at once. The safest approach is probably to introduce the changes separately a few at a time. We can achieve this by Introducing a variable that gets passed down the call-chain telling the nsswitch(3) database functions whether the "destination address" (return value) will by allocated by the caller or not. If this variable is false then the routines can use a static buffer, otherwise, they will assume the caller has allocated the necessary space. Most of the functions will need a lot more work than this to make them fully thread-safe. The advantage of doing it this way is that it maintains the status-quo while allowing us to make the individual subsystems thread-safe separately and as time and resources permit. The following patch shows what I have in mind. It won't compile just yet, but it might make what I'm saying a little clearer :-) This initial phase is really just a mechanical change of argument lists. It doesn't introduce any thread-safeness on its own. It just makes it easier to introduce such changes safely on a subsytem by subsystem basis. Basically, the _nsdispatch() functionality is moved into nsdispatch_common(), which takes a va_list as its last argument instead of a variable number of arguments. It also takes one additional argument(int is_r) so callers can tell it what kind of semantics they want. Applications calling nsdispatch() get a wrapper that passes a false (0) value in the is_r argument. Callers from within libc will continue calling_nsdispatch(), which also grows an additional argument (int is_r) that callers can pass down to the nsswitch(3) database functions through nsdispatch_common(). What do you think? Should I continue with this or do you think it's bogus? The important point to keep in mind is that we have to keep the dual semantics (thread-safe and unsafe) all the way down the call-chain because only the top-level and low-level functions know the actual memory type and size that needs to be allocated for the return value to the library users. So, we can't make the low-level nsswitch(2) database functions unconditionally thread-safe and restrict the dual semantics to the top-level functions. Cheers. -- Mike Makonnen | GPG-KEY: http://www.identd.net/~mtm/mtm.asc mtm@identd.net | D228 1A6F C64E 120A A1C9 A3AA DAE1 E2AF DBCC 68B9 mtm@FreeBSD.Org| FreeBSD - The Power To Serve Index: include/nsswitch.h =================================================================== RCS file: /home/ncvs/src/include/nsswitch.h,v retrieving revision 1.3 diff -u -r1.3 nsswitch.h --- include/nsswitch.h 17 Apr 2003 14:14:21 -0000 1.3 +++ include/nsswitch.h 19 Jun 2003 03:13:29 -0000 @@ -104,13 +104,13 @@ /* * ns_dtab `method' function signature. */ -typedef int (*nss_method)(void *_retval, void *_mdata, va_list _ap); +typedef int (*nss_method)(void *_retval, void *_mdata, int is_r, va_list _ap); /* * Macro for generating method prototypes. */ #define NSS_METHOD_PROTOTYPE(method) \ - int method(void *, void *, va_list) + int method(void *, void *, int, va_list) /* * ns_dtab - `nsswitch dispatch table' Index: lib/libc/net/gethostbydns.c =================================================================== RCS file: /home/ncvs/src/lib/libc/net/gethostbydns.c,v retrieving revision 1.43 diff -u -r1.43 gethostbydns.c --- lib/libc/net/gethostbydns.c 1 May 2003 19:03:13 -0000 1.43 +++ lib/libc/net/gethostbydns.c 19 Jun 2003 03:33:43 -0000 @@ -470,7 +470,7 @@ } int -_dns_gethostbyname(void *rval, void *cb_data, va_list ap) +_dns_gethostbyname(void *rval, void *cb_data, int is_r, va_list ap) { const char *name; int af; @@ -604,7 +604,7 @@ } int -_dns_gethostbyaddr(void *rval, void *cb_data, va_list ap) +_dns_gethostbyaddr(void *rval, void *cb_data, int is_r, va_list ap) { const char *addr; /* XXX should have been def'd as u_char! */ int len, af; Index: lib/libc/net/nsdispatch.c =================================================================== RCS file: /home/ncvs/src/lib/libc/net/nsdispatch.c,v retrieving revision 1.8 diff -u -r1.8 nsdispatch.c --- lib/libc/net/nsdispatch.c 24 Apr 2003 19:57:31 -0000 1.8 +++ lib/libc/net/nsdispatch.c 19 Jun 2003 03:17:17 -0000 @@ -557,13 +557,41 @@ } -__weak_reference(_nsdispatch, nsdispatch); +__weak_reference(__nsdispatch, nsdispatch); + +int +__nsdispatch(void *retval, const ns_dtab disp_tab[], const char *database, + const char *method_name, const ns_src defaults[], ...) +{ + va_list ap; + int error; + + va_start(ap, defaults); + error = nsdispatch_common(retval, disp_tab, database, method_name, + defaults, 0, ap); + va_end(ap); + return (error); +} int _nsdispatch(void *retval, const ns_dtab disp_tab[], const char *database, - const char *method_name, const ns_src defaults[], ...) + const char *method_name, const ns_src defaults[], int is_r, ...) +{ + va_list ap; + int error; + + va_start(ap, is_r); + error = nsdispatch_common(retval, disp_tab, database, method_name, + defaults, is_r, ap); + va_end(ap); + return (error); +} + +int +nsdispatch_common(void *retval, const ns_dtab disp_tab[], const char *database, + const char *method_name, const ns_src defaults[], int is_r, + va_list ap) { - va_list ap; const ns_dbt *dbt; const ns_src *srclist; nss_method method; @@ -597,9 +625,7 @@ method = nss_method_lookup(srclist[i].name, database, method_name, disp_tab, &mdata); if (method != NULL) { - va_start(ap, defaults); - result = method(retval, mdata, ap); - va_end(ap); + result = method(retval, mdata, is_r, ap); if (result & (srclist[i].flags)) break; } From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 23:34:59 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4C74C37B401 for ; Wed, 18 Jun 2003 23:34:59 -0700 (PDT) Received: from stork.mail.pas.earthlink.net (stork.mail.pas.earthlink.net [207.217.120.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id B057343FA3 for ; Wed, 18 Jun 2003 23:34:58 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2ivfk2f.dialup.mindspring.com ([165.247.208.79] helo=mindspring.com) by stork.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19St0Z-0004UP-00; Wed, 18 Jun 2003 23:34:55 -0700 Message-ID: <3EF15947.1102D796@mindspring.com> Date: Wed, 18 Jun 2003 23:33:43 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Daniel Eischen References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4d198646b967cb52082105061744b4e11a2d4e88014a4647c350badd9bab72f9c350badd9bab72f9c cc: threads@freebsd.org cc: Kris Kennaway Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 06:34:59 -0000 Daniel Eischen wrote: > On Wed, 18 Jun 2003, Terry Lambert wrote: > > Daniel Eischen wrote: > > > On Wed, 18 Jun 2003, Kris Kennaway wrote: > > > > Any objections? > > > > > > Why is this bogus? Do we have another gethostbyaddr_r hiding > > > somewhere? > > > > It lies? (_r). > > If that's true, then it's a bug and eventually should > be fixed. Yes. It's not thread-safe, and it's not reentrant, as implied by the name. I have no idea how it made it in past all the reviewing. > Additionally, you can't go around removing > public interfaces without bumping library versions > (unless said interface hasn't seen a release yet). That's a good point; but you'd hope that the binding of shared library symbols would be RTLD_NOW at compile time and RTLD_LAZY at runtime, so the only thing that would break are programs that actually used the symbol. And those programs are broken anyway, since if they depend on the thing working, they're broken. Maybe we could just declare an "void *" by that name that was equal to zero... 8-). But if no one is going to fix it before the next release, it should be diked out, because it's fooling "configure" prgorams (and unfortunately, just diking it out of the header files isn't enough, because they look by linking, rather than by looking). Makes you want minor version numbres back... -- Terry From owner-freebsd-threads@FreeBSD.ORG Wed Jun 18 23:40:28 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1866337B401 for ; Wed, 18 Jun 2003 23:40:28 -0700 (PDT) Received: from stork.mail.pas.earthlink.net (stork.mail.pas.earthlink.net [207.217.120.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id 776CC43F93 for ; Wed, 18 Jun 2003 23:40:27 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2ivfk2f.dialup.mindspring.com ([165.247.208.79] helo=mindspring.com) by stork.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19St5c-00057n-00; Wed, 18 Jun 2003 23:40:09 -0700 Message-ID: <3EF15A83.D869E0C9@mindspring.com> Date: Wed, 18 Jun 2003 23:38:59 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Marcel Moolenaar References: <20030617223910.GB57040@ns1.xcllnt.net> <002101c3352a$e931a7f0$0701a8c0@tiger> <20030618003556.GA2440@dhcp01.pn.xcllnt.net> <3EF02B40.A4BD1EF@mindspring.com> <20030618182638.GA63660@ns1.xcllnt.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4d198646b967cb52024b77e6d0c7aa962667c3043c0873f7e350badd9bab72f9c350badd9bab72f9c cc: David Xu cc: Julian Elischer cc: threads@freebsd.org Subject: Re: Nvidia, TLS and __thread keyword -- an observation X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 06:40:28 -0000 Marcel Moolenaar wrote: > On Wed, Jun 18, 2003 at 02:05:04AM -0700, Terry Lambert wrote: > > Marcel Moolenaar wrote: > > > On Wed, Jun 18, 2003 at 07:48:09AM +0800, David Xu wrote: > > > > I believe this will add overhead to thread creating and destroying, > > > > How fast an RTLD can be in this case ? > > > > > > In the dynamic TLS model you would like to delay the creation of > > > the TLS space. Normally __tls_get_addr() gets used for this. In > > > the static TLS model you allocate the TLS when you llocate the > > > thread control structure. > > > > Lazy binding in this context doesn't make a lot of sense. > > It does. In a process with 1000 threads where 1 thread does > a dlopen(), you don't want to create 999 TLS spaces if they're > not going to be used. Besides time, this also is a space > issue. If you wanted to save space, you would not be using per thread storage in the first place. 8-). Time is only an issue if you are talking .tdata; the .tbss is all zeroed, so could be allocated as a very large block, with relatively no initialization overhead. > Note also that I don't advocate what I think we should do, but > what the specification is designed for. People have put some > thought in it... I understand the specification's intent, both the purely technical, and the political. -- Terry From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 00:14:23 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D046B37B401 for ; Thu, 19 Jun 2003 00:14:23 -0700 (PDT) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5E40843FCB for ; Thu, 19 Jun 2003 00:14:21 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2ivfk2f.dialup.mindspring.com ([165.247.208.79] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19Stch-0007md-00; Thu, 19 Jun 2003 00:14:19 -0700 Message-ID: <3EF1628A.380EB45F@mindspring.com> Date: Thu, 19 Jun 2003 00:13:14 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Daniel Eischen References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4b3d42763fbf8996744c64234924a6247666fa475841a1c7a350badd9bab72f9c350badd9bab72f9c cc: threads@freebsd.org cc: zander@mail.minion.de cc: Gareth Hughes cc: Andy Ritger Subject: Re: NVIDIA and TLS X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 07:14:24 -0000 Daniel Eischen wrote: > One question. How does using %gs work in libc_r? Thread switches > in libc_r use setjmp()/longjmp() neither of which save and restore > %gs. If OpenGL sets %gs, libc_r will not change it when threads are > switched. > > Is NVIDIA's OpenGL suppose to be thread-safe for libc_r? I don't > see how it can be. What am I missing? 1) No other users of %gs. 2) Thread running at the time of involuntary preemption is the thread that gets resumed on the next quantum, with no regard to thread priority. 3) No one (mostly) does a lot of work in signal handlers. It's not so much that it works on purpose as that it works by serendipity. -- Terry From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 00:44:24 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9ECEA37B401 for ; Thu, 19 Jun 2003 00:44:24 -0700 (PDT) Received: from puffin.mail.pas.earthlink.net (puffin.mail.pas.earthlink.net [207.217.120.139]) by mx1.FreeBSD.org (Postfix) with ESMTP id 058B743FAF for ; Thu, 19 Jun 2003 00:44:24 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2ivfk2f.dialup.mindspring.com ([165.247.208.79] helo=mindspring.com) by puffin.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19Su5S-0005DS-00; Thu, 19 Jun 2003 00:44:03 -0700 Message-ID: <3EF1697D.635165E5@mindspring.com> Date: Thu, 19 Jun 2003 00:42:53 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: David Xu References: <001e01c335e6$1ccca630$0701a8c0@tiger> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a40d8cac93775fc85e10e0569d18a4fa8a548b785378294e88350badd9bab72f9c350badd9bab72f9c cc: threads@freebsd.org cc: Kris Kennaway Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 07:44:24 -0000 David Xu wrote: > > > It lies? (_r). > > > > If that's true, then it's a bug and eventually should > > be fixed. Additionally, you can't go around removing > > public interfaces without bumping library versions > > (unless said interface hasn't seen a release yet). > > I think those *_r interfaces are not very useful except > bloating library. why do not improve non _r version to use > thread local instead to make them reentrant between threads ? Nameserver services are incredibly problematic to make thread safe. For example, you can't serialize responses, and you have to support both UDP, and, if the UDP response claims that the request was too large, you have to connect using TCP, and disconnect, etc.. As one example, suppose you have two requests outstanding, one for "A.COM" and the other for "C.NET", and you get a response from one that claims to be authoritative for the other, thus poisinging your outstanding response list? The net effect is that you are running a DNS proxy cache, with unrolled loop, in your *_r() routines, and you have to do everything a proxy cache would do on behalf of its clients. The problem with your approach here is that things like sockets aren't thread-local. It wouldn't make much sense for a multithreaded Apache to have a UDP socket per thread before it's allowed to use DNS services. One approach to resolving the issue would be to serialize the access to a request queue, and throw off some number of threads in order to handle library consumer responses, each with its own context statite (doesn't matter if if's thread local storage, or just an array of structures indexed by start order and parameter, or using a set of auto's allocated in the thread main, of the stack, which is also thread local, by definition), which would include a separate UDP socket. Doing this has a problem, though, in that you now require threads to use the interface, even if your program's a finite state automaton with multiple concurrent contexts, but no threads: you can't have an unthreaded program. The upshot of all this is that there's going to be a lot of work involved no matter how you slice it (the best suggestion so far, the last time this question came up, is to use some sort of "lookupd", similar to Apple's Darwin, and make the messaging interface thread safe and the lookupd a smart DNS proxy server). Predominant wisdom has been to let the Bind people solve this for us, when they release their Bind-9 resolver library release version (what there is of the current stuff is beta). -- Terry From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 01:44:34 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D424637B401; Thu, 19 Jun 2003 01:44:34 -0700 (PDT) Received: from stork.mail.pas.earthlink.net (stork.mail.pas.earthlink.net [207.217.120.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id 47A9043FD7; Thu, 19 Jun 2003 01:44:34 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2ivfk2f.dialup.mindspring.com ([165.247.208.79] helo=mindspring.com) by stork.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19Sv21-0001R1-00; Thu, 19 Jun 2003 01:44:34 -0700 Message-ID: <3EF177A3.4F99EC70@mindspring.com> Date: Thu, 19 Jun 2003 01:43:15 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: deischen@freebsd.org References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a48a7ada51db5154afc8d1d9a7697f0189350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c cc: David Xu cc: Kris Kennaway cc: threads@freebsd.org Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 08:44:35 -0000 Daniel Eischen wrote: > The _r versions are expected and part of POSIX (in general, > I'll haven't checked to see if gethostbyaddr_r is part of > POSIX, but it does exist in Solaris). It's not part of POSIX. Feel free to check: http://www.opengroup.org/onlinepubs/007904975/nfindex.html The final APIs have yet to be worked out by ISC, who is writing a reference implementation of a library to go with their Bind 9 reference implementation, but it is not yet final. Most of the exisitng implementations are in fact premature. There is some indication consumers are expected to use the getaddrinfo/freeaddrinfo interfaces instead. These *are* mandated by POSIX, and they *are* mandated to be thread safe, and the final API may in fact not support gethostbyaddr_r at all, with gethostbyaddr itself being deprecated for address family independent lookups using getaddrinfo, instead. See also: http://www.opengroup.org/onlinepubs/007904975/functions/getaddrinfo.html In other words, if you want a threads-safe interface, you need to update your software, not screw around making unportable interfaces even less portable. -- Terry From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 13:20:14 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7505437B401 for ; Thu, 19 Jun 2003 13:20:14 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9E87F43F85 for ; Thu, 19 Jun 2003 13:20:13 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JKKDDZ071077 for ; Thu, 19 Jun 2003 13:20:13 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JKKD17000926 for ; Thu, 19 Jun 2003 13:20:13 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5JKKDfA000925 for threads@FreeBSD.org; Thu, 19 Jun 2003 13:20:13 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 13:20:13 -0700 From: Marcel Moolenaar To: threads@FreeBSD.org Message-ID: <20030619202013.GA833@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.4i Subject: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 20:20:14 -0000 Ok, Step 1: static TLS in libthr. I have gcc33 installed and looked at the access sequences for TLS on both i386 and ia64. Then I looked at libthr to see what was needed and the first and obvious orbservation is that we need a way to figure out if the binary has a TLS template and use it if it does. If not, we probably need some minimal glue to have the TLS pointer point to something meaningful. Note again, we don't have RTLD involved. We're talking staticly linking now. What about the following: 1. The kernel already iterates over the program headers and can pass the address and size of the TLS template to the process (or RTLD) by means of the auxargs (ie have AT_TLS_ADDR and AT_TLS_SIZE). If no template exists AT_TLS_* will be zero. This prevents coding object file dependencies on thread and allows the RTLD to modify the args even in the event that the program itself does not have TLS, but libraries in the startup set do. 2. On thread creation we allocate the TLS space according to the template (or some MD specific placebo) and put a pointer to it in struct thread. For static TLS we don't have a means to lazily allocate the TLS. 3. The MD _get_curthread() and _set_curthread() will be adjusted according to the RT spec. if not currently compliant. Question: 1. What's the best way to expose the auxargs to the thread library? Alternatives? -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 13:23:50 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A3EC037B401 for ; Thu, 19 Jun 2003 13:23:50 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id A4EB343FAF for ; Thu, 19 Jun 2003 13:23:49 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JKNnDZ071106 for ; Thu, 19 Jun 2003 13:23:49 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JKNn17000949 for ; Thu, 19 Jun 2003 13:23:49 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5JKNn4c000948 for threads@freebsd.org; Thu, 19 Jun 2003 13:23:49 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 13:23:49 -0700 From: Marcel Moolenaar To: threads@freebsd.org Message-ID: <20030619202349.GB833@dhcp01.pn.xcllnt.net> References: <20030619202013.GA833@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030619202013.GA833@dhcp01.pn.xcllnt.net> User-Agent: Mutt/1.5.4i Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 20:23:51 -0000 On Thu, Jun 19, 2003 at 01:20:13PM -0700, Marcel Moolenaar wrote: > > 1. The kernel already iterates over the program headers and can > pass the address and size of the TLS template to the process > (or RTLD) by means of the auxargs (ie have AT_TLS_ADDR and > AT_TLS_SIZE). If no template exists AT_TLS_* will be zero. > This prevents coding object file dependencies on thread and ^^^^^^^^^ I mean: in the thread libraries. Sorry about that, -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 13:29:37 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 590B537B401 for ; Thu, 19 Jun 2003 13:29:37 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id A06DE43F3F for ; Thu, 19 Jun 2003 13:29:36 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5JKTZXh015866; Thu, 19 Jun 2003 16:29:36 -0400 (EDT) Date: Thu, 19 Jun 2003 16:29:35 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Marcel Moolenaar In-Reply-To: <20030619202013.GA833@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 20:29:37 -0000 On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > Ok, > > Step 1: static TLS in libthr. Can you please make this libkse first? This is the harder one, and what you may find works for libthr may not work for libkse. Also, libkse is on the TODO for 5.2-release. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 13:38:40 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4EE6537B401; Thu, 19 Jun 2003 13:38:40 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7E5ED43F75; Thu, 19 Jun 2003 13:38:39 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JKcdDZ071173; Thu, 19 Jun 2003 13:38:39 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JKcd17001007; Thu, 19 Jun 2003 13:38:39 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5JKcd4x001006; Thu, 19 Jun 2003 13:38:39 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 13:38:39 -0700 From: Marcel Moolenaar To: deischen@freebsd.org Message-ID: <20030619203839.GA968@dhcp01.pn.xcllnt.net> References: <20030619202013.GA833@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 20:38:40 -0000 On Thu, Jun 19, 2003 at 04:29:35PM -0400, Daniel Eischen wrote: > On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > > Ok, > > > > Step 1: static TLS in libthr. > > Can you please make this libkse first? This is the harder > one, and what you may find works for libthr may not work > for libkse. Also, libkse is on the TODO for 5.2-release. libthr will be the only threading library on ia64 in 5.2-R if libkse is not ported before then. Given the current status libkse is simply not a viable threading implementation for me to implement TLS in. I expect to implement TLS in libthr in such a way that the MI changes are leveraged when implementing it in libkse. Also, implementing TLS in the threads library that's conceptually less complex, makes debugging the TLS specific changes much easier and thus reduces the problem space when implementing it in libkse. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 13:42:31 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4698637B404 for ; Thu, 19 Jun 2003 13:42:31 -0700 (PDT) Received: from h132-197-179-27.gte.com (h132-197-179-27.gte.com [132.197.179.27]) by mx1.FreeBSD.org (Postfix) with ESMTP id 399BB43F85 for ; Thu, 19 Jun 2003 13:42:30 -0700 (PDT) (envelope-from ak03@gte.com) Received: from kanpc.gte.com (ak03@localhost [127.0.0.1]) h5JKgTTO019224; Thu, 19 Jun 2003 16:42:29 -0400 (EDT) (envelope-from ak03@kanpc.gte.com) Received: (from ak03@localhost) by kanpc.gte.com (8.12.9/8.12.9/Submit) id h5JKgSv6019223; Thu, 19 Jun 2003 16:42:28 -0400 (EDT) Date: Thu, 19 Jun 2003 16:42:28 -0400 From: Alexander Kabaev To: Marcel Moolenaar Message-Id: <20030619164228.6d2e7dd2.ak03@gte.com> In-Reply-To: <20030619202013.GA833@dhcp01.pn.xcllnt.net> References: <20030619202013.GA833@dhcp01.pn.xcllnt.net> Organization: Verizon Data Services X-Mailer: Sylpheed version 0.9.0claws25 (GTK+ 1.2.10; i386-portbld-freebsd5.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 20:42:31 -0000 On Thu, 19 Jun 2003 13:20:13 -0700 Marcel Moolenaar wrote: > Ok, > > Step 1: static TLS in libthr. > > I have gcc33 installed and looked at the access sequences for TLS > on both i386 and ia64. Then I looked at libthr to see what was > needed and the first and obvious orbservation is that we need a > way to figure out if the binary has a TLS template and use it if > it does. If not, we probably need some minimal glue to have the > TLS pointer point to something meaningful. Note again, we don't > have RTLD involved. We're talking staticly linking now. > > What about the following: > > 1. The kernel already iterates over the program headers and can > pass the address and size of the TLS template to the process > (or RTLD) by means of the auxargs (ie have AT_TLS_ADDR and > AT_TLS_SIZE). If no template exists AT_TLS_* will be zero. > This prevents coding object file dependencies on thread and > allows the RTLD to modify the args even in the event that the > program itself does not have TLS, but libraries in the startup > set do. Any reason why existing AT_PHDR is not enough? > 2. On thread creation we allocate the TLS space according to the > template (or some MD specific placebo) and put a pointer to it > in struct thread. For static TLS we don't have a means to > lazily allocate the TLS. > 3. The MD _get_curthread() and _set_curthread() will be adjusted > according to the RT spec. if not currently compliant. > > Question: > 1. What's the best way to expose the auxargs to the thread library? rtld can export several _magic_ symbols thread library can look for to get the info it needs. Alternatively, rtld_thread_init interface can be extended to pass required into to the library in a structure of some sort. For static binaries, startup code will have to parse AT_PHDR and store TLS attributes in a global variable or call a function defined by the thread library. If I remember correctly, libpthread people wanted to have that function anyway. > Alternatives? > > -- > Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to > "freebsd-threads-unsubscribe@freebsd.org" -- Alexander Kabaev From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 13:53:35 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 94C1337B401 for ; Thu, 19 Jun 2003 13:53:35 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id C234E43F93 for ; Thu, 19 Jun 2003 13:53:34 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5JKrYXh019572; Thu, 19 Jun 2003 16:53:34 -0400 (EDT) Date: Thu, 19 Jun 2003 16:53:34 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Marcel Moolenaar In-Reply-To: <20030619203839.GA968@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 20:53:35 -0000 On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > On Thu, Jun 19, 2003 at 04:29:35PM -0400, Daniel Eischen wrote: > > On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > > > Ok, > > > > > > Step 1: static TLS in libthr. > > > > Can you please make this libkse first? This is the harder > > one, and what you may find works for libthr may not work > > for libkse. Also, libkse is on the TODO for 5.2-release. > > libthr will be the only threading library on ia64 in 5.2-R if libkse > is not ported before then. Given the current status libkse is simply > not a viable threading implementation for me to implement TLS in. Are you doing this on i386 or ia64? i386 is our biggest base and should be the first target... You will have other help here (davidxu, kan, myself) also. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 14:05:24 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A92F637B401 for ; Thu, 19 Jun 2003 14:05:24 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id B71F843FB1 for ; Thu, 19 Jun 2003 14:05:23 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JL5NDZ071314; Thu, 19 Jun 2003 14:05:23 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JL5N17001123; Thu, 19 Jun 2003 14:05:23 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5JL5NuW001122; Thu, 19 Jun 2003 14:05:23 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 14:05:23 -0700 From: Marcel Moolenaar To: Alexander Kabaev Message-ID: <20030619210523.GA1030@dhcp01.pn.xcllnt.net> References: <20030619202013.GA833@dhcp01.pn.xcllnt.net> <20030619164228.6d2e7dd2.ak03@gte.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030619164228.6d2e7dd2.ak03@gte.com> User-Agent: Mutt/1.5.4i cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 21:05:24 -0000 On Thu, Jun 19, 2003 at 04:42:28PM -0400, Alexander Kabaev wrote: > > > > 1. The kernel already iterates over the program headers and can > > pass the address and size of the TLS template to the process > > (or RTLD) by means of the auxargs (ie have AT_TLS_ADDR and > > AT_TLS_SIZE). If no template exists AT_TLS_* will be zero. > > This prevents coding object file dependencies on thread and > > allows the RTLD to modify the args even in the event that the > > program itself does not have TLS, but libraries in the startup > > set do. > > Any reason why existing AT_PHDR is not enough? I assume that in the dynamic case the rtld will construct the TLS for the initial load set. This is what the threads library should use as the TLS template. Put differently: if the threads library looks at the object file itself, it will correctly obtain the TLS template in static binaries, but will not have the correct template in shared binaries. What is missing are the TLS templates in the shared libraries in the initial load set in that case. > ... For static binaries, startup code will have to parse AT_PHDR and > store TLS attributes in a global variable or call a function defined by > the thread library. If I remember correctly, libpthread people wanted to > have that function anyway. This is possible and removes the dependency on the kernel for passing the info in auxargs. The issue with this is that the startup code may be doing it for nothing (non-threaded binaries), or interfere in getting the right information to the threads library and thus making it rather complex. What about the following: The startup code provides a weak function (__tls_template() or something like that) that is also provided (non-weak) by the rtld. The threads library calls this function to obtain the TLS of the initial load set (which for static binaries boils down to looking up PT_TLS)? The object details are hidden in the startup code and the difference between static and dynamic is handled by binding. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 14:35:20 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 014E437B401; Thu, 19 Jun 2003 14:35:20 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 24C1A43FB1; Thu, 19 Jun 2003 14:35:19 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JLZIDZ071440; Thu, 19 Jun 2003 14:35:18 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JLZI17001221; Thu, 19 Jun 2003 14:35:18 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5JLZIdL001220; Thu, 19 Jun 2003 14:35:18 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 14:35:18 -0700 From: Marcel Moolenaar To: deischen@freebsd.org Message-ID: <20030619213518.GB1030@dhcp01.pn.xcllnt.net> References: <20030619203839.GA968@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 21:35:20 -0000 On Thu, Jun 19, 2003 at 04:53:34PM -0400, Daniel Eischen wrote: > > > > libthr will be the only threading library on ia64 in 5.2-R if libkse > > is not ported before then. Given the current status libkse is simply > > not a viable threading implementation for me to implement TLS in. > > Are you doing this on i386 or ia64? i386 is our biggest > base and should be the first target... You will have other > help here (davidxu, kan, myself) also. As I mentioned earlier, I do ia64. I expect other people to work on other patforms. I'm not pulling this wagon by myself. I have too much ia64 specific work and cross-platform work on my plate to take on more multi-platform work. There are enough i386 developers who can help out with both i386 specific and MI code. The same holds (to lesser extend) for sparc64. It should be clear by now that we don't have a bucket of ia64 developers who can take over the care for ia64 while I'm away from home... -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 14:50:51 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F406B37B401 for ; Thu, 19 Jun 2003 14:50:50 -0700 (PDT) Received: from sccrmhc12.attbi.com (sccrmhc12.comcast.net [204.127.202.56]) by mx1.FreeBSD.org (Postfix) with ESMTP id C3F4E43F93 for ; Thu, 19 Jun 2003 14:50:49 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc12) with ESMTP id <20030619215048012004um5le>; Thu, 19 Jun 2003 21:50:48 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id OAA49892; Thu, 19 Jun 2003 14:50:46 -0700 (PDT) Date: Thu, 19 Jun 2003 14:50:45 -0700 (PDT) From: Julian Elischer To: Marcel Moolenaar In-Reply-To: <20030619202013.GA833@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@FreeBSD.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 21:50:51 -0000 On Thu, 19 Jun 2003, Marcel Moolenaar wrote: Note: The following applies to ia32, but similar comments woudl apply to other achitectures. One thing to bear in mind is that if you push libthr support for TLS to the absolute maximum you will generate code that is not binary compatible with KSE (when it gets to ia64). Libthr has the advantage that the register used for identifying the current thread can be set DIRECTLY at the thread control block, where libkse will (probably) require a single indirection. (by which I mean that the register will point to the control structure for the Virtual CPU that is running (KSE) and that in turn will have a pointer to the thread that is currently running. So to keep binray compatibility we would need to not push the libthr optimisations to the absolute ends beyond what can be achieved with libkse (unless the loader code relaxer can be set up to use different relaxations depending on which thread library is linked in. (there is SOME precedence for this)). Libc_r can be modified to set the thread pointer register as needed, but it might slow it down. (I was just looking at the ia-32 docs..it's not as bad as I thought to write a segment register so it is feasible to change the segment register in libc_r purely to supplrt TLS). (in a code compatible manner.) One possible optimisation in libkse would be to place a pointer to the TLS object table in the Virtual CPU structure (KSE mailbox) whenever a new thread was scheduled on that KSE/VCPU. This would allow teh generation of code that bypassed the redirection via teh thread structure, at the expence of the extra MOV in the context switch. This would bring it closer to the order of optimisation achievable with 1:1, though the code used for relaxation may or may not be the same. We just need to decide if the cost is worth it.. > Ok, > > Step 1: static TLS in libthr. > > I have gcc33 installed and looked at the access sequences for TLS > on both i386 and ia64. Then I looked at libthr to see what was > needed and the first and obvious orbservation is that we need a > way to figure out if the binary has a TLS template and use it if > it does. If not, we probably need some minimal glue to have the > TLS pointer point to something meaningful. Note again, we don't > have RTLD involved. We're talking staticly linking now. Call me stupid but can you draw a picture of what you mean? (it's worth a thoudsand words you know :-) > > What about the following: > > 1. The kernel already iterates over the program headers and can > pass the address and size of the TLS template to the process > (or RTLD) by means of the auxargs (ie have AT_TLS_ADDR and > AT_TLS_SIZE). If no template exists AT_TLS_* will be zero. > This prevents coding object file dependencies on thread and > allows the RTLD to modify the args even in the event that the > program itself does not have TLS, but libraries in the startup > set do. I need to go out to the car and get my copy of the TLS proposal.... this supports exec-time linking but does it support run-time (i.e after exec has begun) linking? > 2. On thread creation we allocate the TLS space according to the > template (or some MD specific placebo) and put a pointer to it > in struct thread. For static TLS we don't have a means to > lazily allocate the TLS. > 3. The MD _get_curthread() and _set_curthread() will be adjusted > according to the RT spec. if not currently compliant. > > Question: > 1. What's the best way to expose the auxargs to the thread library? > > Alternatives? > > -- > Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 15:00:44 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B820B37B404 for ; Thu, 19 Jun 2003 15:00:44 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 938F043F75 for ; Thu, 19 Jun 2003 15:00:43 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JM09DZ071575; Thu, 19 Jun 2003 15:00:09 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JM0917001340; Thu, 19 Jun 2003 15:00:09 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5JM09Ju001339; Thu, 19 Jun 2003 15:00:09 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 15:00:08 -0700 From: Marcel Moolenaar To: Julian Elischer Message-ID: <20030619220008.GA1273@dhcp01.pn.xcllnt.net> References: <20030619202013.GA833@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@FreeBSD.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 22:00:45 -0000 On Thu, Jun 19, 2003 at 02:50:45PM -0700, Julian Elischer wrote: > > Libthr has the advantage that the register used for identifying the > current thread can be set DIRECTLY at the thread control block, where > libkse will (probably) require a single indirection. (by which I mean > that the register will point to the control structure for the Virtual > CPU that is running (KSE) and that in turn will have a pointer to the > thread that is currently running. On ia64, the runtime specification dictates that the TP registers points to the TLS. How the threads implementation relates this to the thread control structure (or other control structures) is up to the implementation. If libkse is ported to ia64, this will have to be the case as well. Hence the thread pointer will *not* point to the KSE. An indirection will be used to point from the TLS to the KSE. There is no other way. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 15:36:14 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D85D337B401 for ; Thu, 19 Jun 2003 15:36:14 -0700 (PDT) Received: from rwcrmhc11.attbi.com (rwcrmhc11.comcast.net [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3934743FAF for ; Thu, 19 Jun 2003 15:36:14 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (rwcrmhc11) with ESMTP id <20030619223613013008susoe>; Thu, 19 Jun 2003 22:36:13 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id PAA50226; Thu, 19 Jun 2003 15:36:13 -0700 (PDT) Date: Thu, 19 Jun 2003 15:36:11 -0700 (PDT) From: Julian Elischer To: Marcel Moolenaar In-Reply-To: <20030619220008.GA1273@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@FreeBSD.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 22:36:15 -0000 On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > On Thu, Jun 19, 2003 at 02:50:45PM -0700, Julian Elischer wrote: > > > > Libthr has the advantage that the register used for identifying the > > current thread can be set DIRECTLY at the thread control block, where > > libkse will (probably) require a single indirection. (by which I mean > > that the register will point to the control structure for the Virtual > > CPU that is running (KSE) and that in turn will have a pointer to the > > thread that is currently running. > > On ia64, the runtime specification dictates that the TP registers > points to the TLS. How the threads implementation relates this to > the thread control structure (or other control structures) is up to > the implementation. If libkse is ported to ia64, this will have to > be the case as well. Hence the thread pointer will *not* point to > the KSE. An indirection will be used to point from the TLS to the > KSE. > > There is no other way. There is always "another way". It's just whether it is worth following it.. What happens when there is no thread.. (e.g. the UTS (User Thread Scheduler) is running and has not selected a thread to run? Basically in libKSE there is what the kernel calls a thread, which is interpretted in userspace as a virtual CPU, and this is used to run user level threads. The kernel sets the thread-pointer register upcorrectly according to which of these "kernel visible threads" is running. This is a legitimate use of the thread pointer register as far as I see. In 1:1 (libthr) the two things are the same and it stops there. Under M:N, while the kernel's thread is running in the UTS there is no userland thread selected to be running on it, but the kernel still thinks it is running a thread. So from the kernel's (and other external) point of view, the KSE/VCPU is a thread. Only that thread itself knows that it is switching between different userland contexts. The thread register is set by the kernel according to which of the 'threads' it sees is about to be run. It doesn't see the userland threads and knows nothing aboyut them so it is physically incapable of setting the register to point to them. (though it COULD save and restore the value for threads that have enterred the kernel I guess, (blindly hoping that they make sense)) BTW what are the official register designations for ia64 i.e. what is reserved for what? Where is the doc that describes this? > > -- > Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net > From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 15:36:46 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9EF7A37B401 for ; Thu, 19 Jun 2003 15:36:46 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 68FD943F3F for ; Thu, 19 Jun 2003 15:36:45 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JMa9DZ071774; Thu, 19 Jun 2003 15:36:09 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JMa817001438; Thu, 19 Jun 2003 15:36:09 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5JMa81p001437; Thu, 19 Jun 2003 15:36:08 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 15:36:08 -0700 From: Marcel Moolenaar To: Julian Elischer Message-ID: <20030619223608.GB1273@dhcp01.pn.xcllnt.net> References: <20030619202013.GA833@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 22:36:46 -0000 [sorry, I overlooked the other comments] > > I have gcc33 installed and looked at the access sequences for TLS > > on both i386 and ia64. Then I looked at libthr to see what was > > needed and the first and obvious orbservation is that we need a > > way to figure out if the binary has a TLS template and use it if > > it does. If not, we probably need some minimal glue to have the > > TLS pointer point to something meaningful. Note again, we don't > > have RTLD involved. We're talking staticly linking now. > > Call me stupid but can you draw a picture of what you mean? > (it's worth a thoudsand words you know :-) Not easily. Let me try with words again. Let me know if it's more clear or not. If not, I'll see if there's a graphical representation on the net. When code contains thread local variables (by way of defining them with the __thread modifier), the compiler will reserve the space for them in the .tdata section (for initialized data) or the .tbss section (for uninitialized data or data initialized to zero). This is exactly like how the compiler reserves space for global data (using .data and .bss sections), except of course that the intend of the TLS is that each thread has its own instance. The linker combines the .tdata and .tbss sections in the same way it combines the .data and .bss sections. The end result is an executable (or library) that contains both global data and TLS. The global data is normally loaded by the kernel at program load because there's one instance per process. For each thread, the thread library has to create the TLS instance by copying the TLS image present in the executable (or constructed by the rtld). Hence the use of template. The compiler generates access sequences according to the runtime specification which in general means that all offsets to the TLS are based on some TLS base address. On ia64 the thread pointer points to the TLS and serves as the TLS base address. On other architectures there may be an indirection. This means that on ia64 the lack of TLS still requires us to allocate something for the thread pointer to point to. On other architectures this may not be the case. A typical access sequence on i386 is: 00000000 : 0: 55 push %ebp 1: 89 e5 mov %esp,%ebp 3: 65 a1 00 00 00 00 mov %gs:0x0,%eax 9: 8b 80 00 00 00 00 mov 0x0(%eax),%eax f: c9 leave 10: c3 ret At gs:0x0 is the address of the TLS and there's a relocation associated with the load from %eax: RELOCATION RECORDS FOR [.text]: OFFSET TYPE VALUE 0000000b R_386_TLS_LE i On ia64 the access sequence for this same C code is: 0000000000000000 : 0: 0b 10 00 1a 00 21 [MMI] mov r2=r13;; 6: e0 00 08 00 48 00 addl r14=0,r2 c: 00 00 04 00 nop.i 0x0;; 10: 1d 40 00 1c 10 10 [MFB] ld4 r8=[r14] 16: 00 00 00 02 00 80 nop.f 0x0 1c: 08 00 84 00 br.ret.sptk.many b0;; The thread pointer is r13 on ia64. Access to TLS is without indirection. The relocation is attached to the addl instruction: RELOCATION RECORDS FOR [.text]: OFFSET TYPE VALUE 0000000000000001 TPREL22 i > > 1. The kernel already iterates over the program headers and can > > pass the address and size of the TLS template to the process > > (or RTLD) by means of the auxargs (ie have AT_TLS_ADDR and > > AT_TLS_SIZE). If no template exists AT_TLS_* will be zero. > > This prevents coding object file dependencies on thread and > > allows the RTLD to modify the args even in the event that the > > program itself does not have TLS, but libraries in the startup > > set do. > > I need to go out to the car and get my copy of the TLS proposal.... > this supports exec-time linking but does it support run-time (i.e after > exec has begun) linking? Yes. The rtld will dynamicly construct the TLS template from the images in the ELF files in the startup set and pass this in AT_TLS_* by overriding the values (at least that was the idea). -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 15:52:58 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E67A237B4A0 for ; Thu, 19 Jun 2003 15:52:57 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 14C3943F75 for ; Thu, 19 Jun 2003 15:52:57 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JMqPDZ071847; Thu, 19 Jun 2003 15:52:25 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JMqP17001489; Thu, 19 Jun 2003 15:52:25 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5JMqO1h001488; Thu, 19 Jun 2003 15:52:24 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 15:52:24 -0700 From: Marcel Moolenaar To: Julian Elischer Message-ID: <20030619225224.GC1273@dhcp01.pn.xcllnt.net> References: <20030619220008.GA1273@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 22:52:58 -0000 On Thu, Jun 19, 2003 at 03:36:11PM -0700, Julian Elischer wrote: > > What happens when there is no thread.. (e.g. the UTS (User Thread > Scheduler) is running and has not selected a thread to run? The thread pointer has to point to the TLS for accesses to thread local data. The thread pointer can point to anything if there's no access to TLS. For consistency you can have a sentinel structure (16-bytes), that contains the indirection to mimic the common case. Or you can opt to have TP (the thread pointer) point to the KSE when there's no thread. This however can create unwanted complexity. > Basically in libKSE there is what the kernel calls a thread, which is > interpretted in userspace as a virtual CPU, and this is used to > run user level threads. The kernel sets the thread-pointer register > upcorrectly according to which of these "kernel visible threads" > is running. This is a legitimate use of the thread pointer register as > far as I see. In 1:1 (libthr) the two things are the same and it stops > there. It isn't on ia64. The thread pointer register has a predefined meaning. You cannot point it to anything else than the TLS in a process that has TLS. (if there's no TLS, the TP register is unused (=reserved)). See also: http://www.freebsd.org/platforms/ia64/refs.html I'm not sure the runtime specification pointed to there has the TLS info. I'll update the page ASAP if it's missing. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 16:03:37 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 233B037B401 for ; Thu, 19 Jun 2003 16:03:37 -0700 (PDT) Received: from sccrmhc13.attbi.com (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5569E43F93 for ; Thu, 19 Jun 2003 16:03:36 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc13) with ESMTP id <20030619230335016001kqf1e>; Thu, 19 Jun 2003 23:03:35 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA50436; Thu, 19 Jun 2003 16:03:31 -0700 (PDT) Date: Thu, 19 Jun 2003 16:03:30 -0700 (PDT) From: Julian Elischer To: Marcel Moolenaar In-Reply-To: <20030619223608.GB1273@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 23:03:37 -0000 On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > [sorry, I overlooked the other comments] > > > > I have gcc33 installed and looked at the access sequences for TLS > > > on both i386 and ia64. Then I looked at libthr to see what was > > > needed and the first and obvious orbservation is that we need a > > > way to figure out if the binary has a TLS template and use it if > > > it does. If not, we probably need some minimal glue to have the > > > TLS pointer point to something meaningful. Note again, we don't > > > have RTLD involved. We're talking staticly linking now. > > > > Call me stupid but can you draw a picture of what you mean? > > (it's worth a thoudsand words you know :-) > > Not easily. Let me try with words again. Let me know if it's > more clear or not. If not, I'll see if there's a graphical > representation on the net. > > When code contains thread local variables (by way of defining them > with the __thread modifier), the compiler will reserve the space > for them in the .tdata section (for initialized data) or the .tbss > section (for uninitialized data or data initialized to zero). This > is exactly like how the compiler reserves space for global data > (using .data and .bss sections), except of course that the intend > of the TLS is that each thread has its own instance. > > The linker combines the .tdata and .tbss sections in the same way > it combines the .data and .bss sections. The end result is an > executable (or library) that contains both global data and TLS. > The global data is normally loaded by the kernel at program load > because there's one instance per process. For each thread, the > thread library has to create the TLS instance by copying the TLS > image present in the executable (or constructed by the rtld). > Hence the use of template. > > The compiler generates access sequences according to the runtime > specification which in general means that all offsets to the TLS > are based on some TLS base address. On ia64 the thread pointer > points to the TLS and serves as the TLS base address. On other > architectures there may be an indirection. This means that on ia64 > the lack of TLS still requires us to allocate something for the > thread pointer to point to. On other architectures this may not be > the case. > > A typical access sequence on i386 is: > > 00000000 : > 0: 55 push %ebp > 1: 89 e5 mov %esp,%ebp > 3: 65 a1 00 00 00 00 mov %gs:0x0,%eax > 9: 8b 80 00 00 00 00 mov 0x0(%eax),%eax > f: c9 leave > 10: c3 ret The example you show doesn't have any offset into the TLS, just returning the base address of the dtv, correct? you still need do significant work to get from teh dtv to the address of the variable. (at least according to Figure 1.) or are you talking about something else? > > At gs:0x0 is the address of the TLS and there's a relocation > associated with the load from %eax: that's fine.. I can arrange that at the address specified by %gs:0 is a pointer to anything that is needed. What I can't arrange is that %gs:0 IS the address of the TLS. (well maybe I can.. hmmmm...) > RELOCATION RECORDS FOR [.text]: > OFFSET TYPE VALUE > 0000000b R_386_TLS_LE i > > On ia64 the access sequence for this same C code is: > > 0000000000000000 : > 0: 0b 10 00 1a 00 21 [MMI] mov r2=r13;; > 6: e0 00 08 00 48 00 addl r14=0,r2 > c: 00 00 04 00 nop.i 0x0;; > 10: 1d 40 00 1c 10 10 [MFB] ld4 r8=[r14] > 16: 00 00 00 02 00 80 nop.f 0x0 > 1c: 08 00 84 00 br.ret.sptk.many b0;; > > The thread pointer is r13 on ia64. Access to TLS is without > indirection. The relocation is attached to the addl instruction: > > RELOCATION RECORDS FOR [.text]: > OFFSET TYPE VALUE > 0000000000000001 TPREL22 i > > > > 1. The kernel already iterates over the program headers and can > > > pass the address and size of the TLS template to the process > > > (or RTLD) by means of the auxargs (ie have AT_TLS_ADDR and > > > AT_TLS_SIZE). If no template exists AT_TLS_* will be zero. > > > This prevents coding object file dependencies on thread and > > > allows the RTLD to modify the args even in the event that the > > > program itself does not have TLS, but libraries in the startup > > > set do. > > > > I need to go out to the car and get my copy of the TLS proposal.... > > this supports exec-time linking but does it support run-time (i.e after > > exec has begun) linking? > > Yes. The rtld will dynamicly construct the TLS template from the > images in the ELF files in the startup set and pass this in > AT_TLS_* by overriding the values (at least that was the idea). > But if you add a nwe library after threads already exist then you need to add those extra TLS segments to teh existing TLS objects that are attached to already running threads. The code you show can't do that.. > -- > Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net > From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 16:11:53 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0EEA537B401 for ; Thu, 19 Jun 2003 16:11:53 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5C65F43FB1 for ; Thu, 19 Jun 2003 16:11:52 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5JNBkXh009361; Thu, 19 Jun 2003 19:11:46 -0400 (EDT) Date: Thu, 19 Jun 2003 19:11:46 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Marcel Moolenaar In-Reply-To: <20030619225224.GC1273@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 23:11:53 -0000 On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > On Thu, Jun 19, 2003 at 03:36:11PM -0700, Julian Elischer wrote: > > Basically in libKSE there is what the kernel calls a thread, which is > > interpretted in userspace as a virtual CPU, and this is used to > > run user level threads. The kernel sets the thread-pointer register > > upcorrectly according to which of these "kernel visible threads" > > is running. This is a legitimate use of the thread pointer register as > > far as I see. In 1:1 (libthr) the two things are the same and it stops > > there. > > It isn't on ia64. The thread pointer register has a predefined > meaning. You cannot point it to anything else than the TLS in > a process that has TLS. (if there's no TLS, the TP register is > unused (=reserved)). > > See also: > http://www.freebsd.org/platforms/ia64/refs.html > > I'm not sure the runtime specification pointed to there has the > TLS info. I'll update the page ASAP if it's missing. Please do. All it says is the TP is the thread pointer and "The usage of this register is ABI specific. Programs conforming to these conventions may not modify this register." I don't see how using it to point to a per-KSE structure abuses its intended use. You must have a different document in mind. The one I looked at was: http://developer.intel.com/design/itanium/downloads/24535803.pdf Titled: "Itanium Software Conventions and Runtime Architecture Guide", May 2001 Or perhaps I just didn't look in the correct section... The above quote was from section 5.2. Thanks, -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 16:12:52 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D37C837B401 for ; Thu, 19 Jun 2003 16:12:52 -0700 (PDT) Received: from rwcrmhc11.attbi.com (rwcrmhc11.comcast.net [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6A7FD43FCB for ; Thu, 19 Jun 2003 16:12:52 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (rwcrmhc11) with ESMTP id <20030619231251013008ud4ae>; Thu, 19 Jun 2003 23:12:52 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA50513; Thu, 19 Jun 2003 16:12:51 -0700 (PDT) Date: Thu, 19 Jun 2003 16:12:50 -0700 (PDT) From: Julian Elischer To: Marcel Moolenaar In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 23:12:53 -0000 BTW Marcel, I think that we can get from where we are now with kse to what we need by just a little massaging of what points to what. it is possible that the register %gs could be pointed to the thread structure directly and we could derive the KSE from that. it will just make the context switches a fraction more expensive if we need to change the segment register.. The big cost is that a processin x86 can only have a limited number of segments set up in teh Local descriptor table (LDT). I forget the actual number but i vaguely remember that it is 16383 or 8191 or something.. A process could theoretically want to have more than that numbe rof threads.. By pointing to the KSE we limit ourselves to having to set up onl a small number of LDT entries which is a big saving. On ia64 we don;t need to use descriptors so there is not that limit so in effect we could point directly to the thread descriptor and let THAT point to teh VCPU mailbox in question. it's just an extra write or two at context switch time. I think we'll be just fine. From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 16:16:47 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B75AC37B401 for ; Thu, 19 Jun 2003 16:16:47 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1B4DF43FAF for ; Thu, 19 Jun 2003 16:16:47 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5JNGgXh010062; Thu, 19 Jun 2003 19:16:42 -0400 (EDT) Date: Thu, 19 Jun 2003 19:16:42 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: Marcel Moolenaar Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 23:16:48 -0000 On Thu, 19 Jun 2003, Julian Elischer wrote: > > BTW Marcel, I think that we can get from where we are now with kse to > what we need by just a little massaging of what points to what. > it is possible that the register %gs could be pointed to the thread > structure directly and we could derive the KSE from that. it will just > make the context switches a fraction more expensive if we need to > change the segment register.. > > The big cost is that a processin x86 can only have a limited number of > segments set up in teh Local descriptor table (LDT). > I forget the actual number but i vaguely remember that it is 16383 or > 8191 or something.. A process could theoretically want to have > more than that numbe rof threads.. By pointing to the KSE we limit > ourselves to having to set up onl a small number of LDT entries > which is a big saving. Yes, and we can have thousands and thousands of threads. > On ia64 we don;t need to use descriptors so there is not that limit so > in effect we could point directly to the thread descriptor and let THAT > point to teh VCPU mailbox in question. it's just an extra write or two > at context switch time. I think we'll be just fine. It is not just that. It is the cost of an ldt allocation for each thread. In libkse, we only need as many LDTs as there are KSEs. Allocating an ldt per-thread increases thread startup and teardown. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 16:22:36 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B616737B401 for ; Thu, 19 Jun 2003 16:22:35 -0700 (PDT) Received: from sccrmhc13.attbi.com (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0CAEB43F75 for ; Thu, 19 Jun 2003 16:22:35 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc13) with ESMTP id <2003061923223301600acf38e>; Thu, 19 Jun 2003 23:22:34 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA50600; Thu, 19 Jun 2003 16:22:33 -0700 (PDT) Date: Thu, 19 Jun 2003 16:22:33 -0700 (PDT) From: Julian Elischer To: Daniel Eischen In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: Marcel Moolenaar Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 23:22:36 -0000 On Thu, 19 Jun 2003, Daniel Eischen wrote: > On Thu, 19 Jun 2003, Julian Elischer wrote: > > > > BTW Marcel, I think that we can get from where we are now with kse to > > what we need by just a little massaging of what points to what. > > it is possible that the register %gs could be pointed to the thread > > structure directly and we could derive the KSE from that. it will just > > make the context switches a fraction more expensive if we need to > > change the segment register.. > > > > The big cost is that a processin x86 can only have a limited number of > > segments set up in teh Local descriptor table (LDT). > > I forget the actual number but i vaguely remember that it is 16383 or > > 8191 or something.. A process could theoretically want to have > > more than that numbe rof threads.. By pointing to the KSE we limit > > ourselves to having to set up onl a small number of LDT entries > > which is a big saving. > > Yes, and we can have thousands and thousands of threads. > > > On ia64 we don;t need to use descriptors so there is not that limit so > > in effect we could point directly to the thread descriptor and let THAT > > point to teh VCPU mailbox in question. it's just an extra write or two > > at context switch time. I think we'll be just fine. > > It is not just that. It is the cost of an ldt allocation for > each thread. In libkse, we only need as many LDTs as there > are KSEs. Allocating an ldt per-thread increases thread > startup and teardown. that's sort-of-what I was saying.. I think we are ok anyhow.. the spec for i386 is general enough so that we can use it as it is without problem. > > -- > Dan Eischen > > From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 16:30:25 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1869237B401 for ; Thu, 19 Jun 2003 16:30:25 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 671E343FDD for ; Thu, 19 Jun 2003 16:30:24 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5JNUJXh012038; Thu, 19 Jun 2003 19:30:19 -0400 (EDT) Date: Thu, 19 Jun 2003 19:30:19 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: Marcel Moolenaar Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@gdeb.com List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 23:30:25 -0000 On Thu, 19 Jun 2003, Julian Elischer wrote: > > On Thu, 19 Jun 2003, Daniel Eischen wrote: > > > > > > On ia64 we don;t need to use descriptors so there is not that limit so > > > in effect we could point directly to the thread descriptor and let THAT > > > point to teh VCPU mailbox in question. it's just an extra write or two > > > at context switch time. I think we'll be just fine. > > > > It is not just that. It is the cost of an ldt allocation for > > each thread. In libkse, we only need as many LDTs as there > > are KSEs. Allocating an ldt per-thread increases thread > > startup and teardown. > > that's sort-of-what I was saying.. > I think we are ok anyhow.. the spec for i386 is general enough so that > we can use it as it is without problem. Well, I guess you can do it two different ways for different platforms, but that increases the amount of MD code in libkse. It's nice now, because most of the KSE mailbox accesses are MI (well, some relying on ). -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 16:31:21 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2C6E337B404 for ; Thu, 19 Jun 2003 16:31:21 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4D46843FAF for ; Thu, 19 Jun 2003 16:31:19 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JNUgDZ072074; Thu, 19 Jun 2003 16:30:42 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JNUg17001597; Thu, 19 Jun 2003 16:30:42 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5JNUgHX001596; Thu, 19 Jun 2003 16:30:42 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 16:30:42 -0700 From: Marcel Moolenaar To: Julian Elischer Message-ID: <20030619233042.GD1273@dhcp01.pn.xcllnt.net> References: <20030619223608.GB1273@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 23:31:21 -0000 On Thu, Jun 19, 2003 at 04:03:30PM -0700, Julian Elischer wrote: > > > The compiler generates access sequences according to the runtime > > specification which in general means that all offsets to the TLS > > are based on some TLS base address. On ia64 the thread pointer > > points to the TLS and serves as the TLS base address. On other > > architectures there may be an indirection. This means that on ia64 > > the lack of TLS still requires us to allocate something for the > > thread pointer to point to. On other architectures this may not be > > the case. > > > > A typical access sequence on i386 is: > > > > 00000000 : > > 0: 55 push %ebp > > 1: 89 e5 mov %esp,%ebp > > 3: 65 a1 00 00 00 00 mov %gs:0x0,%eax > > 9: 8b 80 00 00 00 00 mov 0x0(%eax),%eax > > f: c9 leave > > 10: c3 ret > > The example you show doesn't have any offset into the TLS, > just returning the base address of the dtv, correct? This is an access sequence for static TLS. There's no DTV in that case. > you still need do significant work to get from teh dtv to the address of > the variable. (at least according to Figure 1.) or are you talking about > something else? The access sequence in the dynamic TLS model is more complex (and dog slow due to the call). For the same C code that yields the static TLS access sequence above: 00000000 : 0: 55 push %ebp 1: 89 e5 mov %esp,%ebp 3: 53 push %ebx 4: e8 00 00 00 00 call 9 9: 5b pop %ebx a: 81 c3 03 00 00 00 add $0x3,%ebx 10: 8d 04 1d 00 00 00 00 lea 0x0(,%ebx,1),%eax 17: e8 fc ff ff ff call 18 1c: 8b 00 mov (%eax),%eax 1e: 8b 1c 24 mov (%esp,1),%ebx 21: c9 leave 22: c3 ret The DTV fodder is abstracted y the __tls_get_addr() function. Here the relocations are: RELOCATION RECORDS FOR [.text]: OFFSET TYPE VALUE 0000000c R_386_GOTPC _GLOBAL_OFFSET_TABLE_ 00000013 R_386_TLS_GD i 00000018 R_386_PLT32 ___tls_get_addr > that's fine.. I can arrange that at the address specified by %gs:0 > is a pointer to anything that is needed. > What I can't arrange is that %gs:0 IS the address of the TLS. > (well maybe I can.. hmmmm...) On i386, gs:0x0 does not have to point to the TLS. There's no equivelent of gs:0x0 on ia64. > > > > 1. The kernel already iterates over the program headers and can > > > > pass the address and size of the TLS template to the process > > > > (or RTLD) by means of the auxargs (ie have AT_TLS_ADDR and > > > > AT_TLS_SIZE). If no template exists AT_TLS_* will be zero. > > > > This prevents coding object file dependencies on thread and > > > > allows the RTLD to modify the args even in the event that the > > > > program itself does not have TLS, but libraries in the startup > > > > set do. > > > > > > I need to go out to the car and get my copy of the TLS proposal.... > > > this supports exec-time linking but does it support run-time (i.e after > > > exec has begun) linking? > > > > Yes. The rtld will dynamicly construct the TLS template from the > > images in the ELF files in the startup set and pass this in > > AT_TLS_* by overriding the values (at least that was the idea). > > > > But if you add a nwe library after threads already exist then you need > to add those extra TLS segments to teh existing TLS objects that are > attached to already running threads. The code you show can't do that.. See above. In the static TLS model there's no support for accessing TLS in dynamicly loaded libraries. That's where the dynamic TLS model kicks in. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 16:32:10 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DE35737B401 for ; Thu, 19 Jun 2003 16:32:10 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3723843FE1 for ; Thu, 19 Jun 2003 16:32:10 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5JNW6Xh012283; Thu, 19 Jun 2003 19:32:06 -0400 (EDT) Date: Thu, 19 Jun 2003 19:32:06 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 23:32:11 -0000 On Thu, 19 Jun 2003, Julian Elischer wrote: > On Thu, 19 Jun 2003, Daniel Eischen wrote: > > > > It is not just that. It is the cost of an ldt allocation for > > each thread. In libkse, we only need as many LDTs as there > > are KSEs. Allocating an ldt per-thread increases thread > > startup and teardown. > > BTW I think we should make kse_create() create the LDT entry > and return the new segment descriptor inteh mailbox .... > > (just a thought.. (irrelevant to this discussion)) Sure, I think I agreed to this a while back. It can be back-burnered if necessary, though, since we've already got the code that does the allocations/deallocations. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 16:42:31 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1C86E37B401 for ; Thu, 19 Jun 2003 16:42:31 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id D409743F93 for ; Thu, 19 Jun 2003 16:42:29 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JNfwDZ072140; Thu, 19 Jun 2003 16:41:58 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5JNfw17001634; Thu, 19 Jun 2003 16:41:58 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5JNfw2d001633; Thu, 19 Jun 2003 16:41:58 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 16:41:58 -0700 From: Marcel Moolenaar To: Julian Elischer Message-ID: <20030619234158.GE1273@dhcp01.pn.xcllnt.net> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 23:42:31 -0000 On Thu, Jun 19, 2003 at 04:12:50PM -0700, Julian Elischer wrote: > > > BTW Marcel, I think that we can get from where we are now with kse to > what we need by just a little massaging of what points to what. > it is possible that the register %gs could be pointed to the thread > structure directly and we could derive the KSE from that. it will just > make the context switches a fraction more expensive if we need to > change the segment register.. Yes. On i386 because the indirection through %gs:0x0 allows having %gs point anything we like: the KSE. On ia64 there's room for a pointer at offset 8 in the TLS that we can use to point to whatever we like. That context switches involve an additional pointer fiddle to make sure we always have the indirection to the KSE is something I don't worry about. > On ia64 we don;t need to use descriptors so there is not that limit so > in effect we could point directly to the thread descriptor and let THAT > point to teh VCPU mailbox in question. it's just an extra write or two > at context switch time. I think we'll be just fine. Agreed. As long as we have the right abstraction in libkse (by means of MD headers/macros or whatever), we should not have to worry too much about it. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 16:53:00 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1FD9437B401 for ; Thu, 19 Jun 2003 16:53:00 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7D19B43FB1 for ; Thu, 19 Jun 2003 16:52:59 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5JNqtXh015244; Thu, 19 Jun 2003 19:52:55 -0400 (EDT) Date: Thu, 19 Jun 2003 19:52:55 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Marcel Moolenaar In-Reply-To: <20030619234158.GE1273@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 23:53:00 -0000 On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > On Thu, Jun 19, 2003 at 04:12:50PM -0700, Julian Elischer wrote: > > > > > > BTW Marcel, I think that we can get from where we are now with kse to > > what we need by just a little massaging of what points to what. > > it is possible that the register %gs could be pointed to the thread > > structure directly and we could derive the KSE from that. it will just > > make the context switches a fraction more expensive if we need to > > change the segment register.. > > Yes. On i386 because the indirection through %gs:0x0 allows having %gs > point anything we like: the KSE. On ia64 there's room for a pointer at > offset 8 in the TLS that we can use to point to whatever we like. That > context switches involve an additional pointer fiddle to make sure we > always have the indirection to the KSE is something I don't worry about. Currently, the libkse TLS pointer has to point to the KSE mailbox. It is necessary to be able to set a word in the KSE mailbox in 1 instruction. The KSE mailbox "current thread pointer" must be set to NULL to prevent upcalls. Indirecting to get to the mailbox pointer in order to set it leaves open a race condition where the TLS changes out from under us before we set it. So if TP can't point to the KSE mailbox, then there needs to be other changes in the kernel and libkse. Julian, please clarify what I've said if it doesn't seem clear. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 17:01:09 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D8EAE37B401 for ; Thu, 19 Jun 2003 17:01:09 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id E3F9D43F3F for ; Thu, 19 Jun 2003 17:01:08 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K00aDZ072232; Thu, 19 Jun 2003 17:00:36 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K00a17001752; Thu, 19 Jun 2003 17:00:36 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5K00aoE001751; Thu, 19 Jun 2003 17:00:36 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 17:00:36 -0700 From: Marcel Moolenaar To: Daniel Eischen Message-ID: <20030620000036.GG1273@dhcp01.pn.xcllnt.net> References: <20030619225224.GC1273@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 00:01:10 -0000 On Thu, Jun 19, 2003 at 07:11:46PM -0400, Daniel Eischen wrote: > > > > I'm not sure the runtime specification pointed to there has the > > TLS info. I'll update the page ASAP if it's missing. > > Please do. I updated the page. It can take a while before the WWW site is updated, so in the mean time go here: http://developer.intel.com/design/itanium/downloads/245370.htm -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 17:47:24 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5164D37B401 for ; Thu, 19 Jun 2003 17:47:24 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id AB80243F85 for ; Thu, 19 Jun 2003 17:47:23 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5K0lFXh022922; Thu, 19 Jun 2003 20:47:15 -0400 (EDT) Date: Thu, 19 Jun 2003 20:47:15 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: Julian Elischer cc: Marcel Moolenaar Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 00:47:24 -0000 On Thu, 19 Jun 2003, Daniel Eischen wrote: > On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > > On Thu, Jun 19, 2003 at 04:12:50PM -0700, Julian Elischer wrote: > > > > > > > > > BTW Marcel, I think that we can get from where we are now with kse to > > > what we need by just a little massaging of what points to what. > > > it is possible that the register %gs could be pointed to the thread > > > structure directly and we could derive the KSE from that. it will just > > > make the context switches a fraction more expensive if we need to > > > change the segment register.. > > > > Yes. On i386 because the indirection through %gs:0x0 allows having %gs > > point anything we like: the KSE. On ia64 there's room for a pointer at > > offset 8 in the TLS that we can use to point to whatever we like. That > > context switches involve an additional pointer fiddle to make sure we > > always have the indirection to the KSE is something I don't worry about. > > Currently, the libkse TLS pointer has to point to the KSE mailbox. > It is necessary to be able to set a word in the KSE mailbox in 1 > instruction. The KSE mailbox "current thread pointer" must be > set to NULL to prevent upcalls. Indirecting to get to the > mailbox pointer in order to set it leaves open a race condition > where the TLS changes out from under us before we set it. > So if TP can't point to the KSE mailbox, then there needs to > be other changes in the kernel and libkse. Perhaps we can just read and clear the TP register in order to prevent upcalls? Once read, we can access the KSE (through indirection if necessary), set km_curthread to NULL, then restore the TP register. I think that would work, except the kernel would need MD code to detect whether upcalls were disabled or not. Just thinking out loud... -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 17:58:52 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7B26937B401 for ; Thu, 19 Jun 2003 17:58:52 -0700 (PDT) Received: from mx2.mail.ru (mx2.mail.ru [194.67.23.22]) by mx1.FreeBSD.org (Postfix) with ESMTP id B83E943FAF for ; Thu, 19 Jun 2003 17:58:51 -0700 (PDT) (envelope-from kabaev@mail.ru) Received: from [141.154.214.216] (port=55638 helo=kan.dnsalias.net) by mx2.mail.ru with esmtp id 19TAEr-0000Tj-00; Fri, 20 Jun 2003 04:58:49 +0400 Received: from kan.dnsalias.net (ak03@localhost [127.0.0.1]) by kan.dnsalias.net (8.12.9/8.12.9) with ESMTP id h5K0wb4V009071; Thu, 19 Jun 2003 20:58:37 -0400 (EDT) (envelope-from kan@kan.dnsalias.net) Received: (from kan@localhost) by kan.dnsalias.net (8.12.9/8.12.9/Submit) id h5K0waQ3009070; Thu, 19 Jun 2003 20:58:37 -0400 (EDT) Date: Thu, 19 Jun 2003 20:58:36 -0400 From: Alexander Kabaev To: Marcel Moolenaar Message-Id: <20030619205836.040122e5.kabaev@mail.ru> In-Reply-To: <20030619233042.GD1273@dhcp01.pn.xcllnt.net> References: <20030619223608.GB1273@dhcp01.pn.xcllnt.net> <20030619233042.GD1273@dhcp01.pn.xcllnt.net> X-Mailer: Sylpheed version 0.9.0claws2 (GTK+ 1.2.10; i386-portbld-freebsd5.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam: Not detected cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 00:58:52 -0000 On Thu, 19 Jun 2003 16:30:42 -0700 Marcel Moolenaar wrote: > On Thu, Jun 19, 2003 at 04:03:30PM -0700, Julian Elischer wrote: > > > > > The compiler generates access sequences according to the runtime > > > specification which in general means that all offsets to the TLS > > > are based on some TLS base address. On ia64 the thread pointer > > > points to the TLS and serves as the TLS base address. On other > > > architectures there may be an indirection. This means that on ia64 > > > the lack of TLS still requires us to allocate something for the > > > thread pointer to point to. On other architectures this may not be > > > the case. > > > > > > A typical access sequence on i386 is: > > > > > > 00000000 : > > > 0: 55 push %ebp > > > 1: 89 e5 mov %esp,%ebp > > > 3: 65 a1 00 00 00 00 mov %gs:0x0,%eax > > > 9: 8b 80 00 00 00 00 mov 0x0(%eax),%eax > > > f: c9 leave > > > 10: c3 ret > > > > The example you show doesn't have any offset into the TLS, > > just returning the base address of the dtv, correct? > > This is an access sequence for static TLS. There's no DTV in that > case. > Actually, the code looks like this: mov 0x0(%eax),%eax ;; return DTV address (or null if static) mov -4(%eax), %eax ;; return first int variable in TLS. i.e. i386 is using TLS Type II, where variables are addressed using _negative_ offsets relative to thread pointer (%gs:0). From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 18:40:33 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 55ECF37B401 for ; Thu, 19 Jun 2003 18:40:33 -0700 (PDT) Received: from mta1.srv.hcvlny.cv.net (mta1.srv.hcvlny.cv.net [167.206.5.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6186143F75 for ; Thu, 19 Jun 2003 18:40:32 -0700 (PDT) (envelope-from agapon@cv-nj.com) Received: from asv17.srv.hcvlny.cv.net (asv17.srv.hcvlny.cv.net [167.206.5.171]) by mta1.srv.hcvlny.cv.net (iPlanet Messaging Server 5.2 HotFix 1.14 (built Mar 18 2003)) threads@freebsd.org; Thu, 19 Jun 2003 21:40:35 -0400 (EDT) Received: from edge.foundation.invalidasv17.srv.hcvlny.cv.net ; Thu, 19 Jun 2003 21:39:22 -0400 (EDT) Received: from cv-nj.com (localhost.foundation.invalid [127.0.0.1]) h5K1eQW3004463 for ; Thu, 19 Jun 2003 21:40:27 -0400 (EDT envelope-from agapon@cv-nj.com) Date: Thu, 19 Jun 2003 21:40:26 -0400 From: Andriy Gapon To: threads@freebsd.org Message-id: <3EF2660A.1040508@cv-nj.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii; format=flowed Content-transfer-encoding: 7BIT X-Accept-Language: en, uk, ru User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.3.1) Gecko/20030610 Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 01:40:33 -0000 Btw, what happened to the patch(es) proposed by Maxim Sobolev: http://docs.freebsd.org/cgi/getmsg.cgi?fetch=13028+0+archive/2002/freebsd-audit/20020818.freebsd-audit and in Problem Report bin/29581: http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/29581 the discussion in the mailing lists seems to have resulted in nothing, and the problem seemed to be of non-unsolvable kind. -- Andriy Gapon From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 18:57:37 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9A79B37B404; Thu, 19 Jun 2003 18:57:37 -0700 (PDT) Received: from sccrmhc13.attbi.com (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8B19043FBD; Thu, 19 Jun 2003 18:57:36 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc13) with ESMTP id <2003062001573501600t1i1ge>; Fri, 20 Jun 2003 01:57:35 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id SAA51560; Thu, 19 Jun 2003 18:57:34 -0700 (PDT) Date: Thu, 19 Jun 2003 18:57:33 -0700 (PDT) From: Julian Elischer To: deischen@freebsd.org In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: Marcel Moolenaar Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 01:57:38 -0000 On Thu, 19 Jun 2003, Daniel Eischen wrote: > On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > > On Thu, Jun 19, 2003 at 04:12:50PM -0700, Julian Elischer wrote: > > > > > > > > > BTW Marcel, I think that we can get from where we are now with kse to > > > what we need by just a little massaging of what points to what. > > > it is possible that the register %gs could be pointed to the thread > > > structure directly and we could derive the KSE from that. it will just > > > make the context switches a fraction more expensive if we need to > > > change the segment register.. > > > > Yes. On i386 because the indirection through %gs:0x0 allows having %gs > > point anything we like: the KSE. On ia64 there's room for a pointer at > > offset 8 in the TLS that we can use to point to whatever we like. That > > context switches involve an additional pointer fiddle to make sure we > > always have the indirection to the KSE is something I don't worry about. > > Currently, the libkse TLS pointer has to point to the KSE mailbox. > It is necessary to be able to set a word in the KSE mailbox in 1 > instruction. The KSE mailbox "current thread pointer" must be > set to NULL to prevent upcalls. Indirecting to get to the > mailbox pointer in order to set it leaves open a race condition > where the TLS changes out from under us before we set it. > So if TP can't point to the KSE mailbox, then there needs to > be other changes in the kernel and libkse. > > Julian, please clarify what I've said if it doesn't seem > clear. What you have said is true, but it desn't reflect on where %gs:0 points becasue the kernel doesn't use %gs to find the user KSE mailbox. it just "knows" . In the same what the UTS "just KNOWS" where the thread mailbox and KSE mailboxes are. it doesn't need to get them via %gs:0 so we can still do teh singl instruction stuff. We can set %gs befoer we set the KSE->thread pointer, while we are in the UTS. No-one will follow it until we start running as the thread because it is only used by code using the __thread keyword. In other words, the KSE_mbox->thread_mbox link is completely separate (orthogonal if you want) from the TLS pointer.. they are separate issues. The info is slightly redundant (only slightly) but they don't impact on each other. I think we are just fine.. we can set teh forst entry in the KSE mailbox to be a pointer to the TLS for teh thread we are switching in. It will ONLY be read by the thread itself so we know that it must be correct as we set it when we schedule it in.. A thread can never switch KSEs without doing so via an upcall, or otherwise involving the UTS, so we know that %gs:0 always points to a pointer to the TLS.. Everythings cool. For that matter %gs:0 need not point to the mailbox at all, since anyoen who needs to find it either has teh time to get it slowely (via wherever it DOES point), or has other ways of finding it. It comes in as the argument to the upcall for example (i.e the kernel tells it where it is). Wherever it points we can get to teh mailbox or teh TLS or both.. it's not important. It's just a shell-game with pointers. > > -- > Dan Eischen > > From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 19:33:49 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E805D37B401 for ; Thu, 19 Jun 2003 19:33:49 -0700 (PDT) Received: from pop015.verizon.net (pop015pub.verizon.net [206.46.170.172]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2AA1D43FAF for ; Thu, 19 Jun 2003 19:33:49 -0700 (PDT) (envelope-from mtm@identd.net) Received: from kokeb.ambesa.net ([138.88.125.205]) by pop015.verizon.net (InterMail vM.5.01.05.33 201-253-122-126-133-20030313) with ESMTP id <20030620023348.PAID20810.pop015.verizon.net@kokeb.ambesa.net>; Thu, 19 Jun 2003 21:33:48 -0500 Date: Thu, 19 Jun 2003 22:33:47 -0400 From: Mike Makonnen To: Andriy Gapon In-Reply-To: <3EF2660A.1040508@cv-nj.com> References: <3EF2660A.1040508@cv-nj.com> X-Mailer: Sylpheed version 0.8.10 (GTK+ 1.2.10; i386-portbld-freebsd5.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Authentication-Info: Submitted using SMTP AUTH at pop015.verizon.net from [138.88.125.205] at Thu, 19 Jun 2003 21:33:48 -0500 Message-Id: <20030620023348.PAID20810.pop015.verizon.net@kokeb.ambesa.net> cc: threads@freebsd.org Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 02:33:50 -0000 On Thu, 19 Jun 2003 21:40:26 -0400 Andriy Gapon wrote: > > Btw, what happened to the patch(es) proposed by Maxim Sobolev: > http://docs.freebsd.org/cgi/getmsg.cgi?fetch=13028+0+archive/2002/freebsd-audit/20020818.freebsd-audit > > and in Problem Report bin/29581: > http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/29581 > > the discussion in the mailing lists seems to have resulted in nothing, and the > problem seemed to be of non-unsolvable kind. I am not so sure that using pthreads thread local storage is such a good idea for a couple of reasons: 1. There is a finite amount of keys that an application can have open at any one time (PTHREAD_KEYS_MAX). On FreeBSD this is 256. So, every key we create is one less the application can use. This is really only a minor concern. Most applications don't use thread local storage and those that do don't usually need so many. 2. The order in which key destructors are called is not guaranteed. An application that uses pthreads thread local storage for its own reasons and calls one of the resolver routines from inside one of its destructors could get in trouble because the destructor for the resolver routines was called before its own. This would almost certainly lead to a memory leak in the application. However, these reservations are not show-stoppers since they can be easily mitigated by a mention in the appropriate man pages. Cheers. -- Mike Makonnen | GPG-KEY: http://www.identd.net/~mtm/mtm.asc mtm@identd.net | D228 1A6F C64E 120A A1C9 A3AA DAE1 E2AF DBCC 68B9 mtm@FreeBSD.Org| FreeBSD - The Power To Serve From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 19:47:18 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D6DBF37B401 for ; Thu, 19 Jun 2003 19:47:18 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3BF6A43F85 for ; Thu, 19 Jun 2003 19:47:18 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5K2lDXh010347; Thu, 19 Jun 2003 22:47:13 -0400 (EDT) Date: Thu, 19 Jun 2003 22:47:13 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: Marcel Moolenaar Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 02:47:19 -0000 On Thu, 19 Jun 2003, Julian Elischer wrote: > > On Thu, 19 Jun 2003, Daniel Eischen wrote: > > > On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > > > On Thu, Jun 19, 2003 at 04:12:50PM -0700, Julian Elischer wrote: > > > > > > > > > > > > BTW Marcel, I think that we can get from where we are now with kse to > > > > what we need by just a little massaging of what points to what. > > > > it is possible that the register %gs could be pointed to the thread > > > > structure directly and we could derive the KSE from that. it will just > > > > make the context switches a fraction more expensive if we need to > > > > change the segment register.. > > > > > > Yes. On i386 because the indirection through %gs:0x0 allows having %gs > > > point anything we like: the KSE. On ia64 there's room for a pointer at > > > offset 8 in the TLS that we can use to point to whatever we like. That > > > context switches involve an additional pointer fiddle to make sure we > > > always have the indirection to the KSE is something I don't worry about. > > > > Currently, the libkse TLS pointer has to point to the KSE mailbox. > > It is necessary to be able to set a word in the KSE mailbox in 1 > > instruction. The KSE mailbox "current thread pointer" must be > > set to NULL to prevent upcalls. Indirecting to get to the > > mailbox pointer in order to set it leaves open a race condition > > where the TLS changes out from under us before we set it. > > So if TP can't point to the KSE mailbox, then there needs to > > be other changes in the kernel and libkse. > > > > Julian, please clarify what I've said if it doesn't seem > > clear. > > > What you have said is true, but it desn't reflect on where %gs:0 points > becasue the kernel doesn't use %gs to find the user KSE mailbox. it just > "knows" . In the same what the UTS "just KNOWS" where the thread > mailbox and KSE mailboxes are. it doesn't need to get them via %gs:0 so > we can still do teh singl instruction stuff. We can set %gs befoer we > set the KSE->thread pointer, while we are in the UTS. No-one will follow > it until we start running as the thread because it is only used by code > using the __thread keyword. > > In other words, the KSE_mbox->thread_mbox link is completely separate > (orthogonal if you want) from the TLS pointer.. they are separate > issues. > The info is slightly redundant (only slightly) but they don't impact on > each other. > > I think we are just fine.. we can set teh forst entry in the KSE mailbox > to be a pointer to the TLS for teh thread we are switching in. > It will ONLY be read by the thread itself so we know that it must be > correct as we set it when we schedule it in.. > > A thread can never switch KSEs without doing so via an upcall, or That is exactly the problem. An upcall switching the current thread to another KSE while the UTS is in the middle of preventing upcalls. > otherwise involving the UTS, so we know that %gs:0 always points to a > pointer to the TLS.. Everythings cool. You're missing something. The UTS needs to prevent upcalls at times. The current method of doing that is by setting the KSE_mbox->thread_mbox (km_curthread) link to null. This has to be atomic in some way, because it is always possible for the thread to get swapped out (quantum, etc) and completed on another KSE. So code in the UTS that needs to do this: curthread = _get_curthread(); /* Prevent upcalls */ curkse = _get_curkse(); /* * Here is race condition. An upcall here can force the * thread to be resumed on another KSE. If that is the * case, then curkse is no longer valid and we're setting * km_curthread to NULL in the wrong KSE! */ atomic_store_ptr(&curkse->km_curthread, NULL); KSE_LOCK_ACQUIRE(curkse, &some_lock); ... Currently, the UTS can atomically clear km_curthread in 1 instruction, so there is no race condition and the above looks more like this: curthread = _get_curthread(); curkse = read_and_clear_kmbx(); /* 1 instruction */ KSE_LOCK_ACQUIRE(curkse, &some_lock); ... This isn't i386-specific. If TP is going to point to something other than the KSE mailbox, then you have to find another way to atomically stop upcalls without the KSE from changing out from under you. I think we can get around it, at least on ia64, just by clearing and restroing TP around the critical section, but the kernel needs to know the difference between TP being cleared and not. It can't just look in the KSE mailbox to see if km_curthread is NULL. So the above code would be: tp = read_and_clear_tp(); /* Some asm to do this. */ curkse = _get_curkse(tp); curthread = _get_curthread(tp); curkse->km_curthread = NULL; restore_tp(tp); > For that matter %gs:0 need not point to the mailbox at all, > since anyoen who needs to find it either has teh time to get it slowely > (via wherever it DOES point), or has other ways of finding it. Currently it _does_ have to point to the mailbox. David Xu has convinced me of that due to the above argument. It initially did _not_ point to the mailbox and that caused the exact problems I'm talking about. Trust me, we've already been down that route. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 20:02:08 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A79D537B401 for ; Thu, 19 Jun 2003 20:02:08 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0B76643F85 for ; Thu, 19 Jun 2003 20:02:08 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5K31vXh012393; Thu, 19 Jun 2003 23:01:57 -0400 (EDT) Date: Thu, 19 Jun 2003 23:01:57 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Mike Makonnen In-Reply-To: <20030620023348.PAID20810.pop015.verizon.net@kokeb.ambesa.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Andriy Gapon cc: threads@freebsd.org Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 03:02:09 -0000 On Thu, 19 Jun 2003, Mike Makonnen wrote: > On Thu, 19 Jun 2003 21:40:26 -0400 > Andriy Gapon wrote: > > > > > Btw, what happened to the patch(es) proposed by Maxim Sobolev: > > http://docs.freebsd.org/cgi/getmsg.cgi?fetch=13028+0+archive/2002/freebsd-audit/20020818.freebsd-audit > > > > and in Problem Report bin/29581: > > http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/29581 > > > > the discussion in the mailing lists seems to have resulted in nothing, and the > > problem seemed to be of non-unsolvable kind. > > I am not so sure that using pthreads thread local storage is such a good idea > for a couple of reasons: I haven't grep'd, but I'm pretty sure we use pthread keys in a couple of other places in libc. But regardless, ... > 1. There is a finite amount of keys that an application can have open at any > one time (PTHREAD_KEYS_MAX). On FreeBSD this is 256. So, every key we > create is one less the application can use. This is really only a minor > concern. Most applications don't use thread local storage and those that > do don't usually need so many. We (the implementation) are always free to reserve a few extra on top of PTHREAD_KEYS_MAX just so libc doesn't consume application keys. If you or others are concerned about that, we can do this probably pretty easily. > 2. The order in which key destructors are called is not guaranteed. An > application that uses pthreads thread local storage for its own reasons and > calls one of the resolver routines from inside one of its destructors > could get in trouble because the destructor for the resolver routines > was called before its own. This would almost certainly lead to a memory > leak in the application. Hmm, true. Libc usage of pthread functions use the single underscore versions, so we could always differentiate between application keys and libc keys. libc keys could be destroyed last. Actually, we could automatically allocate a fixed amount of key space just for the implementation to use, so that it wouldn't detract from PTHREAD_KEYS_MAX and could be easily destroyed after the other keys. We should try to handle it the same way (libthr, libkse) if possible. Shoot me some email if you ever want to address the problem. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 20:13:59 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DC36E37B401 for ; Thu, 19 Jun 2003 20:13:59 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 052E943FDF for ; Thu, 19 Jun 2003 20:13:59 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K3DQDZ072996; Thu, 19 Jun 2003 20:13:26 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K3DQ17002305; Thu, 19 Jun 2003 20:13:26 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5K3DQbV002304; Thu, 19 Jun 2003 20:13:26 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 20:13:26 -0700 From: Marcel Moolenaar To: Daniel Eischen Message-ID: <20030620031326.GA2260@dhcp01.pn.xcllnt.net> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 03:14:00 -0000 On Thu, Jun 19, 2003 at 10:47:13PM -0400, Daniel Eischen wrote: > > > For that matter %gs:0 need not point to the mailbox at all, > > since anyoen who needs to find it either has teh time to get it slowely > > (via wherever it DOES point), or has other ways of finding it. > > Currently it _does_ have to point to the mailbox. Q1: The race is only possible by the kernel changing the thread pointer? Q2: Given that libthr has been ported to ia64, what needs to be done to port libkse to ia64 (roughtly)? -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 20:22:39 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 061E637B401 for ; Thu, 19 Jun 2003 20:22:39 -0700 (PDT) Received: from pop017.verizon.net (pop017pub.verizon.net [206.46.170.210]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1FA6943FCB for ; Thu, 19 Jun 2003 20:22:38 -0700 (PDT) (envelope-from mtm@identd.net) Received: from kokeb.ambesa.net ([138.88.125.205]) by pop017.verizon.net (InterMail vM.5.01.05.33 201-253-122-126-133-20030313) with ESMTP id <20030620032237.PMLR27254.pop017.verizon.net@kokeb.ambesa.net>; Thu, 19 Jun 2003 22:22:37 -0500 Date: Thu, 19 Jun 2003 23:22:35 -0400 From: Mike Makonnen To: Daniel Eischen In-Reply-To: References: <20030620023348.PAID20810.pop015.verizon.net@kokeb.ambesa.net> X-Mailer: Sylpheed version 0.8.10 (GTK+ 1.2.10; i386-portbld-freebsd5.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Authentication-Info: Submitted using SMTP AUTH at pop017.verizon.net from [138.88.125.205] at Thu, 19 Jun 2003 22:22:36 -0500 Message-Id: <20030620032237.PMLR27254.pop017.verizon.net@kokeb.ambesa.net> cc: agapon@cv-nj.com cc: threads@freebsd.org Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 03:22:39 -0000 On Thu, 19 Jun 2003 23:01:57 -0400 (EDT) Daniel Eischen wrote: > > We should try to handle it the same way (libthr, libkse) > if possible. Shoot me some email if you ever want to > address the problem. I don't like the added complexity that they would introduce in the pthreads libraries, so I don't plan on working on it. I think it can be solved differently, but I'll have to do some work before I can say for sure. I don't object to someone else implementing it, though :-) Cheers. -- Mike Makonnen | GPG-KEY: http://www.identd.net/~mtm/mtm.asc mtm@identd.net | D228 1A6F C64E 120A A1C9 A3AA DAE1 E2AF DBCC 68B9 mtm@FreeBSD.Org| FreeBSD - The Power To Serve From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 20:39:47 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 66B4A37B401 for ; Thu, 19 Jun 2003 20:39:47 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 67DE643FAF for ; Thu, 19 Jun 2003 20:39:46 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5K3dgXh017914; Thu, 19 Jun 2003 23:39:42 -0400 (EDT) Date: Thu, 19 Jun 2003 23:39:42 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Marcel Moolenaar In-Reply-To: <20030620031326.GA2260@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 03:39:47 -0000 On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > On Thu, Jun 19, 2003 at 10:47:13PM -0400, Daniel Eischen wrote: > > > > > For that matter %gs:0 need not point to the mailbox at all, > > > since anyoen who needs to find it either has teh time to get it slowely > > > (via wherever it DOES point), or has other ways of finding it. > > > > Currently it _does_ have to point to the mailbox. > > Q1: The race is only possible by the kernel changing the thread > pointer? No, all the kernel has to do is swap out the currently running thread (quantum expiration, interrupt, page fault, etc) and send it up as completed on a different KSE. The UTS can't stop interrupts from happening in the kernel, so a lot can happen to cause the currently running KSE (and its currently running thread) to be swapped out. If the process' next KSE (within the same KSE Group) to be scheduled is not the last KSE, then the thread from the last KSE goes up as completed on the new KSE. Now the thread's notion of "what KSE am I on" is wrong (if it is cached). For the most part, this is OK. But in critical regions it is not. > Q2: Given that libthr has been ported to ia64, what needs to be done > to port libkse to ia64 (roughtly)? libpthread/arch/i386/include/ksd.h: #define _ksd_curkse ((struct kse *)KSD_GET_PTR(mbx.km_udata)) #define _ksd_curthread KSD_GET_PTR(curthread) #define _ksd_set_tmbx(value) KSD_SET_PTR(mbx.km_curthread, (void *)value) #define _ksd_get_tmbx(value) KSD_GET_PTR(mbx.km_curthread) #define _ksd_readandclear_tmbx KSD_READANDCLEAR_PTR(mbx.km_curthread) int _ksd_create(struct ksd *ksd, void *base, int size); void _ksd_destroy(struct ksd *ksd); int _ksd_getprivate(struct ksd *ksd, void **base, int *size); int _ksd_setprivate(struct ksd *ksd); For archs other than i386, this is probably just simple malloc() and free() with a way to set the thread pointer register. libpthread/arch/i386/include/atomic_ops.h atomic_swap_long(long *dst, long val, long *res); libpthread/arch/i386/include/pthread_md.h You can define THR_GETCONTEXT and THR_SETCONTEXT to be the system calls if you want. i386 has optimized them to be userland, but it should be fine if they are syscalls. Basically, all you really need is THR_ALIGNBYTES and THR_ALIGN(). We don't currently use GET_STACK_*, but may at some point to check for inappropriate longjmps()/setcontexts() (thread's shouldn't be jumping out of their own stacks, but if they want to, I guess why should we stop them!). libpthread/arch/i386/i386: The functions thr_enter_uts() and thr_switch(). These are really the only functions that require much thought. For someone who knows the ucontext for ia64 and how it is saved and restored, it's probably a snap. There may be more MD header files than we need, but it was a first shot at abstracting out the MD stuff. There shouldn't be any MD stuff other than in libpthread/arch. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 20:42:40 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EA69837B401 for ; Thu, 19 Jun 2003 20:42:40 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id F3AD343F3F for ; Thu, 19 Jun 2003 20:42:39 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5K3gZXh018386; Thu, 19 Jun 2003 23:42:36 -0400 (EDT) Date: Thu, 19 Jun 2003 23:42:35 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Mike Makonnen In-Reply-To: <20030620032237.PMLR27254.pop017.verizon.net@kokeb.ambesa.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: agapon@cv-nj.com cc: threads@freebsd.org Subject: Re: Removal of bogus gethostbyaddr_r() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 03:42:41 -0000 On Thu, 19 Jun 2003, Mike Makonnen wrote: > On Thu, 19 Jun 2003 23:01:57 -0400 (EDT) > Daniel Eischen wrote: > > > > > We should try to handle it the same way (libthr, libkse) > > if possible. Shoot me some email if you ever want to > > address the problem. > > I don't like the added complexity that they would introduce in the pthreads > libraries, so I don't plan on working on it. I think it can be solved > differently, but I'll have to do some work before I can say for sure. I don't > object to someone else implementing it, though :-) Sure, if it can be done without pthread keys, that's fine. As long as it's not TLS! :-) -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 20:47:19 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B256A37B401; Thu, 19 Jun 2003 20:47:19 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 005F843F75; Thu, 19 Jun 2003 20:47:19 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K3klDZ073156; Thu, 19 Jun 2003 20:46:47 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K3kl17002394; Thu, 19 Jun 2003 20:46:47 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5K3klWL002393; Thu, 19 Jun 2003 20:46:47 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 20:46:47 -0700 From: Marcel Moolenaar To: deischen@freebsd.org Message-ID: <20030620034647.GB2260@dhcp01.pn.xcllnt.net> References: <20030620031326.GA2260@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 03:47:20 -0000 On Thu, Jun 19, 2003 at 11:39:42PM -0400, Daniel Eischen wrote: > > > Q2: Given that libthr has been ported to ia64, what needs to be done > > to port libkse to ia64 (roughtly)? > > libpthread/arch/i386/include/ksd.h: > libpthread/arch/i386/include/atomic_ops.h > libpthread/arch/i386/include/pthread_md.h > libpthread/arch/i386/i386: No kernel code? Hmmm, looks like something that's worth giving a shot. If we have libkse on ia64, we can more easily work on the issues related to TLS... -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 20:55:22 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 19F5B37B408; Thu, 19 Jun 2003 20:55:22 -0700 (PDT) Received: from exchhz01.viatech.com.cn (ip-167-164-97-218.anlai.com [218.97.164.167]) by mx1.FreeBSD.org (Postfix) with ESMTP id B58AB43FDD; Thu, 19 Jun 2003 20:55:07 -0700 (PDT) (envelope-from davidxu@viatech.com.cn) Received: from davidw2k (ip-240-1-168-192.rev.dyxnet.com [192.168.1.240]) by exchhz01.viatech.com.cn with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id M4G4C1MA; Fri, 20 Jun 2003 11:38:46 +0800 Message-ID: <006f01c336e0$231d8fe0$f001a8c0@davidw2k> From: "David Xu" To: "Marcel Moolenaar" , References: <20030620031326.GA2260@dhcp01.pn.xcllnt.net> <20030620034647.GB2260@dhcp01.pn.xcllnt.net> Date: Fri, 20 Jun 2003 11:57:58 +0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4807.1700 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4910.0300 cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 03:55:22 -0000 ----- Original Message -----=20 From: "Marcel Moolenaar" To: Cc: ; "Julian Elischer" Sent: Friday, June 20, 2003 11:46 AM Subject: Re: Implementing TLS: step 1 > On Thu, Jun 19, 2003 at 11:39:42PM -0400, Daniel Eischen wrote: > >=20 > > > Q2: Given that libthr has been ported to ia64, what needs to be = done > > > to port libkse to ia64 (roughtly)? > >=20 > > libpthread/arch/i386/include/ksd.h: > > libpthread/arch/i386/include/atomic_ops.h > > libpthread/arch/i386/include/pthread_md.h > > libpthread/arch/i386/i386: >=20 > No kernel code? Hmmm, looks like something that's worth giving a > shot. If we have libkse on ia64, we can more easily work on the > issues related to TLS... >=20 Can you also look /sys/sys/kse.h, some structures have integer members, and we use fuword() and suword() in kernel, these functions take a long integer parameter, does it work on ia64 ?=20 > --=20 > Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to = "freebsd-threads-unsubscribe@freebsd.org" >=20 From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 20:57:09 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A24EB37B401; Thu, 19 Jun 2003 20:57:09 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id DCE1043F93; Thu, 19 Jun 2003 20:57:08 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5K3v4Xh020487; Thu, 19 Jun 2003 23:57:04 -0400 (EDT) Date: Thu, 19 Jun 2003 23:57:04 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Marcel Moolenaar In-Reply-To: <20030620034647.GB2260@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: davidxu@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 03:57:09 -0000 On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > On Thu, Jun 19, 2003 at 11:39:42PM -0400, Daniel Eischen wrote: > > > > > Q2: Given that libthr has been ported to ia64, what needs to be done > > > to port libkse to ia64 (roughtly)? > > > > libpthread/arch/i386/include/ksd.h: > > libpthread/arch/i386/include/atomic_ops.h > > libpthread/arch/i386/include/pthread_md.h > > libpthread/arch/i386/i386: > > No kernel code? Hmmm, looks like something that's worth giving a I'm not too sure about the kernel code. If you have KSEs working for libthr, then I assume there is very little extra kernel code needed. You do need to have get_mcontext() and set_mcontext() implemented in machdep.c, though. It looks like you do (although nothing is done with clear_ret in get_mcontext()). David, Julian, care to comment? > shot. If we have libkse on ia64, we can more easily work on the > issues related to TLS... -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 21:02:39 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 22BE237B405 for ; Thu, 19 Jun 2003 21:02:38 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 685F543FAF for ; Thu, 19 Jun 2003 21:02:37 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5K42WXh021247; Fri, 20 Jun 2003 00:02:33 -0400 (EDT) Date: Fri, 20 Jun 2003 00:02:32 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Marcel Moolenaar In-Reply-To: <20030620034647.GB2260@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 04:02:39 -0000 On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > On Thu, Jun 19, 2003 at 11:39:42PM -0400, Daniel Eischen wrote: > > > > > Q2: Given that libthr has been ported to ia64, what needs to be done > > > to port libkse to ia64 (roughtly)? > > > > libpthread/arch/i386/include/ksd.h: > > libpthread/arch/i386/include/atomic_ops.h > > libpthread/arch/i386/include/pthread_md.h > > libpthread/arch/i386/i386: > > No kernel code? Hmmm, looks like something that's worth giving a > shot. If we have libkse on ia64, we can more easily work on the > issues related to TLS... I forgot. You also need signalcontext() in libc//gen/. All it does is add a signal frame to an existing ucontext. (It lets the UTS munge a ucontext to add a signal handler on top of it.) I assume you already have makecontext(). -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 21:17:58 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B9EBD37B401; Thu, 19 Jun 2003 21:17:58 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id DB11143F3F; Thu, 19 Jun 2003 21:17:57 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K4HPDZ073294; Thu, 19 Jun 2003 21:17:25 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K4HP17060227; Thu, 19 Jun 2003 21:17:25 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5K4HOnW060217; Thu, 19 Jun 2003 21:17:24 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 21:17:24 -0700 From: Marcel Moolenaar To: David Xu Message-ID: <20030620041724.GB28472@dhcp01.pn.xcllnt.net> References: <20030620034647.GB2260@dhcp01.pn.xcllnt.net> <006f01c336e0$231d8fe0$f001a8c0@davidw2k> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <006f01c336e0$231d8fe0$f001a8c0@davidw2k> User-Agent: Mutt/1.5.4i cc: deischen@freebsd.org cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 04:17:59 -0000 On Fri, Jun 20, 2003 at 11:57:58AM +0800, David Xu wrote: > > Can you also look /sys/sys/kse.h, some structures have integer members, > and we use fuword() and suword() in kernel, these functions take a long > integer parameter, does it work on ia64 ? On ia64, fuword and suword operate on 64-bit entities (ie long). If we use fuword and suword to access integer fields, we have a problem. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 21:33:42 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 29DE637B401; Thu, 19 Jun 2003 21:33:42 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2A77943FAF; Thu, 19 Jun 2003 21:33:41 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K4X9DZ073370; Thu, 19 Jun 2003 21:33:09 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K4X817094224; Thu, 19 Jun 2003 21:33:09 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5K4CYhl046343; Thu, 19 Jun 2003 21:12:34 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 21:12:34 -0700 From: Marcel Moolenaar To: deischen@freebsd.org Message-ID: <20030620041234.GA28472@dhcp01.pn.xcllnt.net> References: <20030620034647.GB2260@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org cc: davidxu@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 04:33:42 -0000 On Thu, Jun 19, 2003 at 11:57:04PM -0400, Daniel Eischen wrote: > > I'm not too sure about the kernel code. If you have KSEs > working for libthr, then I assume there is very little extra > kernel code needed. You do need to have get_mcontext() and > set_mcontext() implemented in machdep.c, though. It looks > like you do (although nothing is done with clear_ret in > get_mcontext()). Yes, {g|s}et_mcontext() are implemented, as is makecontext(3). We cannot do anything with clear_ret, because it's based on assumptions that don't hold in ia64. BTW: there's no race that can't be plugged if TP doesn't point to the mailbox. All we need is an atomic compare-exchange and a retry loop... -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 21:40:45 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7F69A37B401; Thu, 19 Jun 2003 21:40:45 -0700 (PDT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 14ED243F75; Thu, 19 Jun 2003 21:40:45 -0700 (PDT) (envelope-from davidxu@freebsd.org) Received: from davidw2k (davidxu@localhost [127.0.0.1]) by freefall.freebsd.org (8.12.9/8.12.9) with SMTP id h5K4edUp066708; Thu, 19 Jun 2003 21:40:40 -0700 (PDT) (envelope-from davidxu@freebsd.org) Message-ID: <009b01c336e6$85045ee0$f001a8c0@davidw2k> From: "David Xu" To: , "Marcel Moolenaar" References: Date: Fri, 20 Jun 2003 12:43:34 +0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4807.1700 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4910.0300 cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 04:40:45 -0000 ----- Original Message -----=20 From: "Daniel Eischen" To: "Marcel Moolenaar" Cc: ; ; "Julian Elischer" = Sent: Friday, June 20, 2003 11:57 AM Subject: Re: Implementing TLS: step 1 > On Thu, 19 Jun 2003, Marcel Moolenaar wrote: >=20 > > On Thu, Jun 19, 2003 at 11:39:42PM -0400, Daniel Eischen wrote: > > >=20 > > > > Q2: Given that libthr has been ported to ia64, what needs to be = done > > > > to port libkse to ia64 (roughtly)? > > >=20 > > > libpthread/arch/i386/include/ksd.h: > > > libpthread/arch/i386/include/atomic_ops.h > > > libpthread/arch/i386/include/pthread_md.h > > > libpthread/arch/i386/i386: > >=20 > > No kernel code? Hmmm, looks like something that's worth giving a >=20 > I'm not too sure about the kernel code. If you have KSEs > working for libthr, then I assume there is very little extra > kernel code needed. You do need to have get_mcontext() and > set_mcontext() implemented in machdep.c, though. It looks > like you do (although nothing is done with clear_ret in > get_mcontext()). >=20 > David, Julian, care to comment? >=20 Yes, we don't have so much MD code in kernel. MD functions used by kse kernel code: cpu_thread_setup - not kse related, all process need it, should already work. cpu_set_upcall_kse - kse code, set userland return address to upcall func entry, and push its=20 mailbox parameter on user stack. cpu_thread_clean - not kse related, should already work. cpu_set_upcall - not kse related, make a new thread's=20 initial context so cpu_switch() can load it, it may inherits original thread's pcb and userland context (syscall trap frame). I think it might already work on ia64. get_mcontext - used by kse code, but is general function, and any application can use it, not just kse app. set_mcontext - same as above Planed MD function: void thread_siginfo(int sig, u_long code, siginfo_t *info); Construct a siginfo structure based on current user trap frame and parameter sig and MD fault code, fill fault address in this structure. The function is used by kse to deliver sync signal to userland, sample implement on ia32 is : /* * Build siginfo_t for SA thread */ void thread_siginfo(int sig, u_long code, siginfo_t *si) { struct proc *p; struct thread *td; td =3D curthread; p =3D td->td_proc; PROC_LOCK_ASSERT(p, MA_OWNED); bzero(si, sizeof(*si)); si->si_signo =3D sig; si->si_code =3D code; si->si_addr =3D (void *)td->td_frame->tf_err; } > > shot. If we have libkse on ia64, we can more easily work on the > > issues related to TLS... >=20 > --=20 > Dan Eischen >=20 > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to = "freebsd-threads-unsubscribe@freebsd.org" >=20 From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 22:08:53 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0028937B401; Thu, 19 Jun 2003 22:08:52 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id DCA7F43FE3; Thu, 19 Jun 2003 22:08:51 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5K58kXh000586; Fri, 20 Jun 2003 01:08:46 -0400 (EDT) Date: Fri, 20 Jun 2003 01:08:46 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Marcel Moolenaar In-Reply-To: <20030620041234.GA28472@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: davidxu@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 05:08:53 -0000 On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > On Thu, Jun 19, 2003 at 11:57:04PM -0400, Daniel Eischen wrote: > > > > I'm not too sure about the kernel code. If you have KSEs > > working for libthr, then I assume there is very little extra > > kernel code needed. You do need to have get_mcontext() and > > set_mcontext() implemented in machdep.c, though. It looks > > like you do (although nothing is done with clear_ret in > > get_mcontext()). > > Yes, {g|s}et_mcontext() are implemented, as is makecontext(3). > > We cannot do anything with clear_ret, because it's based on > assumptions that don't hold in ia64. How do return values from syscalls get passed back? > BTW: there's no race that can't be plugged if TP doesn't point > to the mailbox. All we need is an atomic compare-exchange and > a retry loop... Ok, the only problem might be something being deallocated out from under you. For instance, a KSE goes away (gets deallocated) while your thread is continued on another KSE and you are still dereferencing something that may no longer be valid. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 22:20:45 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2FB5E37B401; Thu, 19 Jun 2003 22:20:45 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4658643F75; Thu, 19 Jun 2003 22:20:44 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K5KCDZ073570; Thu, 19 Jun 2003 22:20:12 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K5KB17058493; Thu, 19 Jun 2003 22:20:11 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5K5KBRs058481; Thu, 19 Jun 2003 22:20:11 -0700 (PDT) (envelope-from marcel) Date: Thu, 19 Jun 2003 22:20:10 -0700 From: Marcel Moolenaar To: Daniel Eischen Message-ID: <20030620052010.GC28472@dhcp01.pn.xcllnt.net> References: <20030620041234.GA28472@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org cc: davidxu@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 05:20:45 -0000 On Fri, Jun 20, 2003 at 01:08:46AM -0400, Daniel Eischen wrote: > > > set_mcontext() implemented in machdep.c, though. It looks > > > like you do (although nothing is done with clear_ret in > > > get_mcontext()). > > > > We cannot do anything with clear_ret, because it's based on > > assumptions that don't hold in ia64. > > How do return values from syscalls get passed back? trapframe, as normal. The point is that return registers are not part of the context and are not saved in the trapframe on entry to the kernel. The trapframe basicly contains garbage that we don't save. Hence, clearing is meaningless. > > BTW: there's no race that can't be plugged if TP doesn't point > > to the mailbox. All we need is an atomic compare-exchange and > > a retry loop... > > Ok, the only problem might be something being deallocated > out from under you. For instance, a KSE goes away (gets > deallocated) while your thread is continued on another > KSE and you are still dereferencing something that may no > longer be valid. But isn't that a generic problem and not specific to whether the thread pointer points to the curthread mailbox? -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 23:32:15 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9D70937B401 for ; Thu, 19 Jun 2003 23:32:15 -0700 (PDT) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id DFD7D43FA3 for ; Thu, 19 Jun 2003 23:32:14 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-uinj93o.dialup.mindspring.com ([165.121.164.120] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19TFRN-0001Y2-00; Thu, 19 Jun 2003 23:32:06 -0700 Message-ID: <3EF2AA23.89CCD315@mindspring.com> Date: Thu, 19 Jun 2003 23:30:59 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Marcel Moolenaar References: <20030619202013.GA833@dhcp01.pn.xcllnt.net> <20030619223608.GB1273@dhcp01.pn.xcllnt.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4e550267a68a927fc0c6f81d80ede8f58350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 06:32:15 -0000 [ ... hacked up Marcel's text in the post a bit ... ] Marcel Moolenaar wrote: > > Call me stupid but can you draw a picture of what you mean? > > (it's worth a thoudsand words you know :-) > > Not easily. Let me try with words again. Let me know if it's > more clear or not. If not, I'll see if there's a graphical > representation on the net. [ ... ] > The linker combines the .tdata and .tbss sections in the same way > it combines the .data and .bss sections. The end result is an > executable (or library) that contains both global data and TLS. ---------------------------------------- Program image file on disk ---------------------------------------- Ordinary TLS enabled ,-------. ,-------. | code | | code | `-------' `-------' ,-------. ,-------. | data | | data | `-------' `-------' ,-------. ,-------. | bss | | bss | `-------' `-------' ,-------. | tbss | `-------' ,-------. | tdata | `-------' ---------------------------------------- > When code contains thread local variables (by way of defining them > with the __thread modifier), the compiler will reserve the space > for them in the .tdata section (for initialized data) or the .tbss > section (for uninitialized data or data initialized to zero). This > is exactly like how the compiler reserves space for global data > (using .data and .bss sections), except of course that the intend > of the TLS is that each thread has its own instance. ------------------------------------------------------------ How the "__thread" C language extension works ------------------------------------------------------------ declaration where it ends up int foo1; ---> bss int foo2 = 37; ---> data __thread int foo3; ---> tbss __thread int foo4 = 37; ---> tdata ------------------------------------------------------------ > The global data is normally loaded by the kernel at program load > because there's one instance per process. For each thread, the > thread library has to create the TLS instance by copying the TLS > image present in the executable (or constructed by the rtld). > Hence the use of template. ------------------------------------------------------------ How things look when loaded into memory ------------------------------------------------------------ Thread #1 Disk Image Thread #N ,-------. code----reference---->| code |<----reference---code `-------' ,-------. data----reference---->| data |<----reference---data `-------' ,-------. bss----reference---->| bss |<----reference----bss `-------' ,-------. ,-------. ,-------. | tdata |<-----copy-----| tdata |-----copy----->| tdata | (templated) `-------' `-------' `-------' ,-------. ,-------. .-------. | tbss |<-----copy-----| tbss |-----copy----->| tbss | (templated) `-------' `-------' `-------' ------------------------------------------------------------ > The compiler generates access sequences according to the runtime > specification which in general means that all offsets to the TLS > are based on some TLS base address. On ia64 the thread pointer > points to the TLS and serves as the TLS base address. On other > architectures there may be an indirection. This means that on ia64 > the lack of TLS still requires us to allocate something for the > thread pointer to point to. On other architectures this may not be > the case. Implementation defined access mechanisms are outside the scope of this discussions, since they have not yet been selected. But the above means that the compiler "magically" knows to implement code to reference "__thread" attributed data through the locally defined access mechanism, whatever that may be. Note(1): I have no idea how this applies to things like function pointers with this attribute pointed to functions without it; I assume it will "do the right thing", and make seperate data elements for the pointers, as directed, *AND* generate code to make the calls relative to the TLS for the active thread, which could make the implementeion very complicated. Note(2): For external global references, one would assume that there are scoping issues, i.e. that the external declaration with the "__thread" qualifier language extension *MUST* be in scope at the time, or, at bes, the symbol decorations will not match, or, at worst, everyone who references an out of scope variable like this, or, if forced to have a reference in scope, the reference fails to also have the "__thread" qualifier, they would get the first thread's instance... or even worse, the template instance. > > I need to go out to the car and get my copy of the TLS proposal.... > > this supports exec-time linking but does it support run-time (i.e after > > exec has begun) linking? > > Yes. The rtld will dynamicly construct the TLS template from the > images in the ELF files in the startup set and pass this in > AT_TLS_* by overriding the values (at least that was the idea). This is where I personally have a problem with lazy intialization of per thread TLS. Specifically, when a thread exits, you have to know what you have and have not instanced, on a per dynamic object, per thread basis, as a minimum granularity, in order to be able to clean it up, without trying to clean up things you have not yet instanced in that particular thread. This strikes me as being unable to use the %gs "single instruction" shortcuts, which means that code generation for a dynamically linked object module would nee to know, _apriori_, what kind of references it needed to be generating, OR *all* references would have to be via function and pointer indirection... meaning that the "single instruction" optimization is an illusion that can never happen in reality. -- Terry From owner-freebsd-threads@FreeBSD.ORG Thu Jun 19 23:37:38 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BCB9C37B401 for ; Thu, 19 Jun 2003 23:37:38 -0700 (PDT) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id 35F0443F3F for ; Thu, 19 Jun 2003 23:37:38 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-uinj93o.dialup.mindspring.com ([165.121.164.120] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19TFWh-0002Hj-00; Thu, 19 Jun 2003 23:37:36 -0700 Message-ID: <3EF2AB6D.67145D17@mindspring.com> Date: Thu, 19 Jun 2003 23:36:29 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Julian Elischer References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a40cbb8dba2b51d8854231f6ba502e3c69350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c cc: threads@FreeBSD.org cc: Marcel Moolenaar Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 06:37:39 -0000 Julian Elischer wrote: > It doesn't see the userland threads and knows nothing aboyut them so it > is physically incapable of setting the register to point to them. > > (though it COULD save and restore the value for threads that have > enterred the kernel I guess, (blindly hoping that they make sense)) This adds the assumption that, following an involuntary context switch, when the next quantum is allocated to the KSE, the thread that was running under that KSE is the one which will be activated, regardless of priority relative to other threads. It further assumes that the thread will not be migrated to another KSE (e.g. on another CPU which is assigning quantum to the process, while the KSE in question is the only one in the "runnable" state in the process at the time of the assignment). Neither of these are valid assumptions. The first one breaks already, in the current implementation, if you are not running a hacked user space scheduler. I don't think this is a reasonable direction to go. Your own suggestion that a second indirection is needed to get the actual thread (and TLS context pointer) is correct, IMO. -- Terry From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 00:03:54 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0161737B401; Fri, 20 Jun 2003 00:03:54 -0700 (PDT) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5E27E43F93; Fri, 20 Jun 2003 00:03:53 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-uinj93o.dialup.mindspring.com ([165.121.164.120] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19TFw6-00052T-00; Fri, 20 Jun 2003 00:03:51 -0700 Message-ID: <3EF2B18E.FB346477@mindspring.com> Date: Fri, 20 Jun 2003 00:02:38 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Marcel Moolenaar References: <20030620031326.GA2260@dhcp01.pn.xcllnt.net> <20030620034647.GB2260@dhcp01.pn.xcllnt.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4c92f172431006ea57215f4fed25826ee666fa475841a1c7a350badd9bab72f9c350badd9bab72f9c cc: deischen@freebsd.org cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 07:03:54 -0000 Marcel Moolenaar wrote: > On Thu, Jun 19, 2003 at 11:39:42PM -0400, Daniel Eischen wrote: > > > Q2: Given that libthr has been ported to ia64, what needs to be done > > > to port libkse to ia64 (roughtly)? > > > > libpthread/arch/i386/include/ksd.h: > > libpthread/arch/i386/include/atomic_ops.h > > libpthread/arch/i386/include/pthread_md.h > > libpthread/arch/i386/i386: > > No kernel code? Hmmm, looks like something that's worth giving a > shot. If we have libkse on ia64, we can more easily work on the > issues related to TLS... You guys are missing something. Daniel's earlier point about the IA64 documentation not specifying that the TP point to TLS, rather than a data structure that contains a pointer to TLS is correct (from my reading of both references posted by Marcel). On the other hand, zeroing the TP register in user space to indicate that upcalls are disabled is probably illegal, from the point of vie of that same documentation, so there needs to be another way to do the deed. The problem I forsee here is that it's possible that the SMT cores assign special significance to this register when doing instruction reordering inside the pipeline, and this would definitely break any benefits you would otherwise have obtained from using the register for real threads, if that's the case. Note that this predominantly applies only on UP machines; any MP machines will have a statistical chance of 1 in of breaking the pipelining anyway (IMO, Intel is amining SMT machines at the UP market, more than the SMP market, and SMP users should not expect much benefit: negaffinity will win you more cycles than 1/2-a-CPU-core-when-the-other-1/2-is-idle). -- Terry From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 00:06:40 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AE15D37B401; Fri, 20 Jun 2003 00:06:40 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 09D8443F3F; Fri, 20 Jun 2003 00:06:40 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5K76ZXh016802; Fri, 20 Jun 2003 03:06:35 -0400 (EDT) Date: Fri, 20 Jun 2003 03:06:34 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Marcel Moolenaar In-Reply-To: <20030620052010.GC28472@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: davidxu@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 07:06:41 -0000 On Thu, 19 Jun 2003, Marcel Moolenaar wrote: > On Fri, Jun 20, 2003 at 01:08:46AM -0400, Daniel Eischen wrote: > > > > set_mcontext() implemented in machdep.c, though. It looks > > > > like you do (although nothing is done with clear_ret in > > > > get_mcontext()). > > > > > > We cannot do anything with clear_ret, because it's based on > > > assumptions that don't hold in ia64. > > > > How do return values from syscalls get passed back? > > trapframe, as normal. The point is that return registers are not > part of the context and are not saved in the trapframe on entry > to the kernel. The trapframe basicly contains garbage that we don't > save. Hence, clearing is meaningless. > > > > BTW: there's no race that can't be plugged if TP doesn't point > > > to the mailbox. All we need is an atomic compare-exchange and > > > a retry loop... > > > > Ok, the only problem might be something being deallocated > > out from under you. For instance, a KSE goes away (gets > > deallocated) while your thread is continued on another > > KSE and you are still dereferencing something that may no > > longer be valid. > > But isn't that a generic problem and not specific to whether the > thread pointer points to the curthread mailbox? Not currently because current KSE access is atomic. When a KSE goes away, it is done under a lock and all of its threads have either gone away also, or have had their "what KSE am I currently running on" pointers migrated to the main (initial) KSE. So there are no references to the KSE any longer (at least, that's the idea). Depending on how one were to implement setting the KSE mailbox on ia64 and how TLS, TCB, and KSE pointers were set up, it might be possible to reference a KSE mailbox after it was deallocated. I don't know what you have in mind, so it may not be a problem. Also note that there are both thread and KSE mailboxen. The km_curthread that must be set to NULL is in the KSE mailbox. So if all you have is TP(offset 8) pointing to the thread TCB/mailbox, you still have another pointer from the thread to the KSE (includes the mailbox). So you have to atomically set curthread->curkse->km_curthread to NULL. The thing that can change out from under you is curthread->curkse, not curthread->curkse->km_curthread. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 00:18:03 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D611637B401 for ; Fri, 20 Jun 2003 00:18:03 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id CE5E843F3F for ; Fri, 20 Jun 2003 00:18:02 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K7HUDZ074059; Fri, 20 Jun 2003 00:17:30 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K7HTR8016117; Fri, 20 Jun 2003 00:17:30 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5K7HTvm016116; Fri, 20 Jun 2003 00:17:29 -0700 (PDT) (envelope-from marcel) Date: Fri, 20 Jun 2003 00:17:29 -0700 From: Marcel Moolenaar To: Terry Lambert Message-ID: <20030620071729.GA16066@dhcp01.pn.xcllnt.net> References: <20030619202013.GA833@dhcp01.pn.xcllnt.net> <20030619223608.GB1273@dhcp01.pn.xcllnt.net> <3EF2AA23.89CCD315@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3EF2AA23.89CCD315@mindspring.com> User-Agent: Mutt/1.5.4i cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 07:18:04 -0000 On Thu, Jun 19, 2003 at 11:30:59PM -0700, Terry Lambert wrote: > [ ... hacked up Marcel's text in the post a bit ... ] Cool graphics. Thanks! > > The compiler generates access sequences according to the runtime > > specification which in general means that all offsets to the TLS > > are based on some TLS base address. On ia64 the thread pointer > > points to the TLS and serves as the TLS base address. On other > > architectures there may be an indirection. This means that on ia64 > > the lack of TLS still requires us to allocate something for the > > thread pointer to point to. On other architectures this may not be > > the case. > > Implementation defined access mechanisms are outside the scope > of this discussions, since they have not yet been selected. In fact, they are already architected as part of the psABI (which is an extention to the psABI in most cases). It's our job to implement our threading models within the psABI. > Note(1): I have no idea how this applies to things like function > pointers with this attribute pointed to functions without it; There's no problem. It's no different than having a function pointer on the stack or anywhere else in memory. > I assume it will "do the right thing", and make seperate data > elements for the pointers, as directed, *AND* generate code to > make the calls relative to the TLS for the active thread, which > could make the implementeion very complicated. Function pointers are always absolute. The call sequence may need to construct a displacement, but that's dependent on the IP, not on where the function pointer was obtained from. > Note(2): For external global references, one would assume that > there are scoping issues, i.e. that the external declaration with > the "__thread" qualifier language extension *MUST* be in scope at > the time, or, at bes, the symbol decorations will not match, or, > at worst, everyone who references an out of scope variable like > this, or, if forced to have a reference in scope, the reference > fails to also have the "__thread" qualifier, they would get the > first thread's instance... or even worse, the template instance. All thread local variables must have TLS specific relocations attached to them. It the linkers job and also the rtlds job to validate this. A program is invalid is there's an inconsistency. > > > I need to go out to the car and get my copy of the TLS proposal.... > > > this supports exec-time linking but does it support run-time (i.e after > > > exec has begun) linking? > > > > Yes. The rtld will dynamicly construct the TLS template from the > > images in the ELF files in the startup set and pass this in > > AT_TLS_* by overriding the values (at least that was the idea). > > This is where I personally have a problem with lazy intialization > of per thread TLS. Specifically, when a thread exits, you have to > know what you have and have not instanced, on a per dynamic object, > per thread basis, as a minimum granularity, in order to be able to > clean it up, without trying to clean up things you have not yet > instanced in that particular thread. This strikes me as being > unable to use the %gs "single instruction" shortcuts, which means > that code generation for a dynamically linked object module would > nee to know, _apriori_, what kind of references it needed to be > generating, OR *all* references would have to be via function and > pointer indirection... meaning that the "single instruction" > optimization is an illusion that can never happen in reality. This is where the DTV comes in. It's basicly a vector of TLS block pointers and each pointer/index corresponds to the TLS block of a shared library. At cleanup you iterate over the vector and clean all the non-NULL pointers. This is specific to the dynamic TLS model, BTW. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 00:23:50 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D031137B401; Fri, 20 Jun 2003 00:23:50 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id E1B5C43F3F; Fri, 20 Jun 2003 00:23:49 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K7NIDZ074095; Fri, 20 Jun 2003 00:23:18 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K7NIR8016143; Fri, 20 Jun 2003 00:23:18 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5K7NI4V016142; Fri, 20 Jun 2003 00:23:18 -0700 (PDT) (envelope-from marcel) Date: Fri, 20 Jun 2003 00:23:17 -0700 From: Marcel Moolenaar To: Terry Lambert Message-ID: <20030620072317.GB16066@dhcp01.pn.xcllnt.net> References: <20030620031326.GA2260@dhcp01.pn.xcllnt.net> <20030620034647.GB2260@dhcp01.pn.xcllnt.net> <3EF2B18E.FB346477@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3EF2B18E.FB346477@mindspring.com> User-Agent: Mutt/1.5.4i cc: deischen@freebsd.org cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 07:23:51 -0000 On Fri, Jun 20, 2003 at 12:02:38AM -0700, Terry Lambert wrote: > > You guys are missing something. > > Daniel's earlier point about the IA64 documentation not > specifying that the TP point to TLS, rather than a data > structure that contains a pointer to TLS is correct (from > my reading of both references posted by Marcel). Most of the discussions in the psABI describe dynamic TLS. As I emphasized in the first post in this thread, step 1 is about implementing the static TLS model. There's hardly any reference to that in the psABI, other than the access sequence from which it is implied that the thread pointer points to the TLS (with an offset of -16). -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 00:32:40 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4E0E437B401; Fri, 20 Jun 2003 00:32:40 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 699B043F75; Fri, 20 Jun 2003 00:32:39 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K7W8DZ074132; Fri, 20 Jun 2003 00:32:08 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5K7W7R8016162; Fri, 20 Jun 2003 00:32:07 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5K7W73j016161; Fri, 20 Jun 2003 00:32:07 -0700 (PDT) (envelope-from marcel) Date: Fri, 20 Jun 2003 00:32:07 -0700 From: Marcel Moolenaar To: deischen@freebsd.org Message-ID: <20030620073207.GC16066@dhcp01.pn.xcllnt.net> References: <20030620052010.GC28472@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org cc: davidxu@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 07:32:40 -0000 On Fri, Jun 20, 2003 at 03:06:34AM -0400, Daniel Eischen wrote: > > Also note that there are both thread and KSE mailboxen. > The km_curthread that must be set to NULL is in the > KSE mailbox. So if all you have is TP(offset 8) pointing > to the thread TCB/mailbox, you still have another pointer > from the thread to the KSE (includes the mailbox). So > you have to atomically set curthread->curkse->km_curthread > to NULL. The thing that can change out from under you > is curthread->curkse, not curthread->curkse->km_curthread. I intend to put the TCB at a negative offset from the thread pointer on ia64. This works for both thread models and for both thread libraries. TP(8) is then free in all cases to point to something else (the KSE in this case). I have to play with it to make sure I'm not overlooking something. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 00:37:21 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3A50037B401 for ; Fri, 20 Jun 2003 00:37:21 -0700 (PDT) Received: from puffin.mail.pas.earthlink.net (puffin.mail.pas.earthlink.net [207.217.120.139]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5865E43FAF for ; Fri, 20 Jun 2003 00:37:20 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-uinj93o.dialup.mindspring.com ([165.121.164.120] helo=mindspring.com) by puffin.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19TGSP-0003NK-00; Fri, 20 Jun 2003 00:37:14 -0700 Message-ID: <3EF2B952.64E731F9@mindspring.com> Date: Fri, 20 Jun 2003 00:35:46 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Julian Elischer References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4704f3b1f64e4b4fc8a7f44aa2c236866667c3043c0873f7e350badd9bab72f9c350badd9bab72f9c cc: threads@freebsd.org cc: Marcel Moolenaar Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 07:37:21 -0000 Julian Elischer wrote: > The big cost is that a processin x86 can only have a limited number of > segments set up in teh Local descriptor table (LDT). > I forget the actual number but i vaguely remember that it is 16383 or > 8191 or something. 65536, but it includes the recursive mapping and the call trap, etc., so it's reall 65531. This is one reason we don't use the TSS for context switching, and why Linux, which used to use the TSS, doesn't use it any more. -- Terry From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 02:34:17 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E8F6637B401 for ; Fri, 20 Jun 2003 02:34:17 -0700 (PDT) Received: from stork.mail.pas.earthlink.net (stork.mail.pas.earthlink.net [207.217.120.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id 337FB43F75 for ; Fri, 20 Jun 2003 02:34:17 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-uinj93o.dialup.mindspring.com ([165.121.164.120] helo=mindspring.com) by stork.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19TIHU-000757-00; Fri, 20 Jun 2003 02:34:06 -0700 Message-ID: <3EF2D3DE.8C7CA97D@mindspring.com> Date: Fri, 20 Jun 2003 02:29:02 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Marcel Moolenaar References: <20030619202013.GA833@dhcp01.pn.xcllnt.net> <20030619223608.GB1273@dhcp01.pn.xcllnt.net> <3EF2AA23.89CCD315@mindspring.com> <20030620071729.GA16066@dhcp01.pn.xcllnt.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a40aa2bba9f6afd9c1148c93649b71ad1f666fa475841a1c7a350badd9bab72f9c350badd9bab72f9c cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 09:34:18 -0000 Marcel Moolenaar wrote: > > Implementation defined access mechanisms are outside the scope > > of this discussions, since they have not yet been selected. > > In fact, they are already architected as part of the psABI (which is > an extention to the psABI in most cases). It's our job to implement > our threading models within the psABI. Right. There's a menu from which we can select, but it we can't decide on something not on the menu. That's what I meant by it being "outside the scope"... we're not allowed to reinvent some new crazy thing. > > Note(1): I have no idea how this applies to things like function > > pointers with this attribute pointed to functions without it; > > There's no problem. It's no different than having a function pointer > on the stack or anywhere else in memory. I was worried that the compiler might not generate the correct code for a function pointer dereference for a function pointer that was __thread. E.g.: __thread int (*funcp)(int i); ... { ... x = (*funcp)( 3); ... x = funcp( 7); ... You'd *hope* it would do the right thing, but mixing code and data references didn't seem to be covered... > > Note(2): For external global references, one would assume that > > there are scoping issues, i.e. that the external declaration with > > the "__thread" qualifier language extension *MUST* be in scope at > > the time, or, at bes, the symbol decorations will not match, or, > > at worst, everyone who references an out of scope variable like > > this, or, if forced to have a reference in scope, the reference > > fails to also have the "__thread" qualifier, they would get the > > first thread's instance... or even worse, the template instance. > > All thread local variables must have TLS specific relocations > attached to them. It the linkers job and also the rtlds job to > validate this. A program is invalid is there's an inconsistency. OK, this could be a problem with the GNU toolchain, secifically with linking shared. You would expect that the relocations would be trated as RTLD_NOW at link time and RTLD_LAZY at runtime; that is, at link time, you would not be permitted to have symbols that referenced a library function that referenced symbols which did not resolve in the set of objects and binaries you were linking. This is not the case. Specifically, the linker doesn't enforce the reference counting in the "referenced/references" case, and you can end up with a runtime unresolved symbol, if it's referenced by a library you are linked against that expects you to link against a second party library. The way you are *supposed* to ensure against this, according to the linker's assumptions, is that all ELF binaries are linked shared, and all shared libraries that depend on other shared libraries are linked against them explicitly. This is not the case on FreeBSD, which supports static linking, and as you probably already know, static ELF libraries can't be linked against other ELF libraries that way, so that they get drug in automatically. Archie Cobbs ran into this with a second order dependency on a database library when writing a JNI shared object that linked against a third party library in Kaffe, bck in the Whistle days. I tried to fix this back then, but it required tunneling state across two functions calls deep. Possible, but too much work; we just linked the main program against the library required by the modeule as a workaround. 8-(. My relatively recent dive into the idea of a static libdl told me that things haven't changes in there. > > This is where I personally have a problem with lazy intialization > > of per thread TLS. Specifically, when a thread exits, you have to > > know what you have and have not instanced, on a per dynamic object, > > per thread basis, as a minimum granularity, in order to be able to > > clean it up, without trying to clean up things you have not yet > > instanced in that particular thread. > > This is where the DTV comes in. It's basicly a vector of TLS block > pointers and each pointer/index corresponds to the TLS block of > a shared library. At cleanup you iterate over the vector and clean > all the non-NULL pointers. This is specific to the dynamic TLS > model, BTW. This is the 256 entry pointer table from the spec., right? I still don't think a program can know apriori what kind of a reference to generate in its code: how does it know, at the time an individual object is compiled, that you are going to link it static or dynamic, so it should generate static vs. dynamic references? Or are you saying this only applies to dlopen, and not to linking against a dynamic library... and that ld.so will need to know which kind of thing it's link-loading, and act differently? Thanks for your incredible patience with us, -- Terry From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 04:00:33 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 70E6137B40A for ; Fri, 20 Jun 2003 04:00:33 -0700 (PDT) Received: from park.rambler.ru (park.rambler.ru [81.19.64.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2683C43F75 for ; Fri, 20 Jun 2003 04:00:27 -0700 (PDT) (envelope-from is@rambler-co.ru) Received: from is.park.rambler.ru (is.park.rambler.ru [81.19.64.102]) by park.rambler.ru (8.12.6/8.12.6) with ESMTP id h5KB0KmF046357; Fri, 20 Jun 2003 15:00:20 +0400 (MSD) Date: Fri, 20 Jun 2003 15:00:20 +0400 (MSD) From: Igor Sysoev X-Sender: is@is To: threads@freebsd.org In-Reply-To: <3EF2B952.64E731F9@mindspring.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 11:00:34 -0000 On Fri, 20 Jun 2003, Terry Lambert wrote: > Julian Elischer wrote: > > The big cost is that a processin x86 can only have a limited number of > > segments set up in teh Local descriptor table (LDT). > > I forget the actual number but i vaguely remember that it is 16383 or > > 8191 or something. > > 65536, but it includes the recursive mapping and the call trap, etc., > so it's reall 65531. LDT can have 8192 descriptors. GDT can have 8191 usefull descriptors. The first descriptor is null one. Igor Sysoev http://sysoev.ru/en/ From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 04:40:50 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 69D0337B401 for ; Fri, 20 Jun 2003 04:40:50 -0700 (PDT) Received: from park.rambler.ru (park.rambler.ru [81.19.64.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id 634A743F75 for ; Fri, 20 Jun 2003 04:40:49 -0700 (PDT) (envelope-from is@rambler-co.ru) Received: from is.park.rambler.ru (is.park.rambler.ru [81.19.64.102]) by park.rambler.ru (8.12.6/8.12.6) with ESMTP id h5KBemmF047031 for ; Fri, 20 Jun 2003 15:40:48 +0400 (MSD) Date: Fri, 20 Jun 2003 15:40:48 +0400 (MSD) From: Igor Sysoev X-Sender: is@is To: threads@freebsd.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 11:40:50 -0000 If we could sacrifice our current thread compatibility in 5.0-5.1 then we could change kse_mailbox from struct kse_mailbox { int km_version; /* Mailbox version */ struct kse_thr_mailbox *km_curthread; /* Current thread */ ... to struct kse_mailbox { struct kse_thr_mailbox *km_curthread; /* Current thread */ int km_version; /* Mailbox version */ ... then x86's gs would still point to kse_mailbox, and gs:[0] would be and TP pointer. Also we need to modify struct kse_thr_mailbox { + void *tls; ucontext_t tm_context; /* User thread context */ ... And the static TLS must be allocated before kse_thr_mailbox. Igor Sysoev http://sysoev.ru/en/ From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 10:48:14 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5647137B404 for ; Fri, 20 Jun 2003 10:48:14 -0700 (PDT) Received: from sccrmhc13.attbi.com (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3EC7A43F85 for ; Fri, 20 Jun 2003 10:48:13 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc13) with ESMTP id <20030620174812016009t8g3e>; Fri, 20 Jun 2003 17:48:12 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id KAA57815; Fri, 20 Jun 2003 10:48:09 -0700 (PDT) Date: Fri, 20 Jun 2003 10:48:08 -0700 (PDT) From: Julian Elischer To: Igor Sysoev In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 17:48:14 -0000 On Fri, 20 Jun 2003, Igor Sysoev wrote: > On Fri, 20 Jun 2003, Terry Lambert wrote: > > > Julian Elischer wrote: > > > The big cost is that a processin x86 can only have a limited number of > > > segments set up in teh Local descriptor table (LDT). > > > I forget the actual number but i vaguely remember that it is 16383 or > > > 8191 or something. > > > > 65536, but it includes the recursive mapping and the call trap, etc., > > so it's reall 65531. > > LDT can have 8192 descriptors. > GDT can have 8191 usefull descriptors. The first descriptor is null one. > yes, that's what I remembered.. thanks > > Igor Sysoev > http://sysoev.ru/en/ > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 10:49:31 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C478437B401 for ; Fri, 20 Jun 2003 10:49:31 -0700 (PDT) Received: from sccrmhc13.attbi.com (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id 218AE43FA3 for ; Fri, 20 Jun 2003 10:49:31 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (sccrmhc13) with ESMTP id <20030620174930016009rmiae>; Fri, 20 Jun 2003 17:49:30 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id KAA57823; Fri, 20 Jun 2003 10:49:29 -0700 (PDT) Date: Fri, 20 Jun 2003 10:49:29 -0700 (PDT) From: Julian Elischer To: Igor Sysoev In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 17:49:32 -0000 On Fri, 20 Jun 2003, Igor Sysoev wrote: > If we could sacrifice our current thread compatibility in 5.0-5.1 > then we could change kse_mailbox from > > struct kse_mailbox { > int km_version; /* Mailbox version */ > struct kse_thr_mailbox *km_curthread; /* Current thread */ > ... > > to > > struct kse_mailbox { > struct kse_thr_mailbox *km_curthread; /* Current thread */ > int km_version; /* Mailbox version */ > ... > > then x86's gs would still point to kse_mailbox, and gs:[0] would be > and TP pointer. Also we need to modify > > struct kse_thr_mailbox { > + void *tls; > ucontext_t tm_context; /* User thread context */ > ... > > And the static TLS must be allocated before kse_thr_mailbox. Yes. This is what I was thinking... > > > Igor Sysoev > http://sysoev.ru/en/ > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 11:27:14 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D721337B401 for ; Fri, 20 Jun 2003 11:27:14 -0700 (PDT) Received: from park.rambler.ru (park.rambler.ru [81.19.64.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6B67B43FBF for ; Fri, 20 Jun 2003 11:27:13 -0700 (PDT) (envelope-from is@rambler-co.ru) Received: from is.park.rambler.ru (is.park.rambler.ru [81.19.64.102]) by park.rambler.ru (8.12.6/8.12.6) with ESMTP id h5KIR8mF052312; Fri, 20 Jun 2003 22:27:08 +0400 (MSD) Date: Fri, 20 Jun 2003 22:27:08 +0400 (MSD) From: Igor Sysoev X-Sender: is@is To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 18:27:15 -0000 On Fri, 20 Jun 2003, Julian Elischer wrote: > On Fri, 20 Jun 2003, Igor Sysoev wrote: > > > If we could sacrifice our current thread compatibility in 5.0-5.1 > > then we could change kse_mailbox from > > > > struct kse_mailbox { > > int km_version; /* Mailbox version */ > > struct kse_thr_mailbox *km_curthread; /* Current thread */ > > ... > > > > to > > > > struct kse_mailbox { > > struct kse_thr_mailbox *km_curthread; /* Current thread */ > > int km_version; /* Mailbox version */ > > ... > > > > then x86's gs would still point to kse_mailbox, and gs:[0] would be > > and TP pointer. Also we need to modify > > > > struct kse_thr_mailbox { > > + void *tls; > > ucontext_t tm_context; /* User thread context */ > > ... > > > > And the static TLS must be allocated before kse_thr_mailbox. > > Yes. > This is what I was thinking... I'm wrong, this scheme does not allow to use the 1:1 model in libkse when kse_mailbox.km_curthread is always NULL. We can implement such scheme on x86: gs -> [ TP ] ---> [ TLS ] [ struct kse_mailbox ] +-> [ struct kse_thr_mailbox ] [ .km_curthread ] -+ When UTS would switch to the next thread it should set thread's TLS: kse_mailbox.km_curthread = NULL; gs:[0] = next_thr_tls; kse_mailbox.km_curthread = next_kse_thr_mailbox; kse_mailbox can be accessed via gs register. On amd64 scheme is the same except the use of fs register instead of gs. However on sparc64, ia64 and alpha TP is in the register but not in the memory so we need introduce the new field tm_kse in kse_thr_mailbox to find kse_mailbox: TP -> [ TLS ] [ struct kse_thr_mailbox ] [ .tm_kse ] ---> [ struct kse_mailbox ] By the way how was kse_mailbox being found before gs register was used ? Igor Sysoev http://sysoev.ru/en/ From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 11:36:00 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2E97037B401 for ; Fri, 20 Jun 2003 11:36:00 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.bayarea.net [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 19E0543FA3 for ; Fri, 20 Jun 2003 11:35:59 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5KIZQDZ077394; Fri, 20 Jun 2003 11:35:26 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5KIZQR8018730; Fri, 20 Jun 2003 11:35:26 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5KIZPJd018729; Fri, 20 Jun 2003 11:35:25 -0700 (PDT) (envelope-from marcel) Date: Fri, 20 Jun 2003 11:35:25 -0700 From: Marcel Moolenaar To: Terry Lambert Message-ID: <20030620183525.GA18628@dhcp01.pn.xcllnt.net> References: <20030619202013.GA833@dhcp01.pn.xcllnt.net> <20030619223608.GB1273@dhcp01.pn.xcllnt.net> <3EF2AA23.89CCD315@mindspring.com> <20030620071729.GA16066@dhcp01.pn.xcllnt.net> <3EF2D3DE.8C7CA97D@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3EF2D3DE.8C7CA97D@mindspring.com> User-Agent: Mutt/1.5.4i cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 18:36:00 -0000 On Fri, Jun 20, 2003 at 02:29:02AM -0700, Terry Lambert wrote: > > > Note(1): I have no idea how this applies to things like function > > > pointers with this attribute pointed to functions without it; > > > > There's no problem. It's no different than having a function pointer > > on the stack or anywhere else in memory. > > I was worried that the compiler might not generate the correct > code for a function pointer dereference for a function pointer > that was __thread. E.g.: > > __thread int (*funcp)(int i); I don't think this is meaninful, unless you'd actually put the code of the function in the TLS. The __thread keyword cannot be applied to code. If you put a buffer in TLS, fill it with machine code and write an assembly stub that jumps to the buffer in TLS you have it, but the compiler doesn't know about it. > > > Note(2): For external global references, one would assume that > > > there are scoping issues, i.e. that the external declaration with > > > the "__thread" qualifier language extension *MUST* be in scope at > > > the time, or, at bes, the symbol decorations will not match, or, > > > at worst, everyone who references an out of scope variable like > > > this, or, if forced to have a reference in scope, the reference > > > fails to also have the "__thread" qualifier, they would get the > > > first thread's instance... or even worse, the template instance. > > > > All thread local variables must have TLS specific relocations > > attached to them. It the linkers job and also the rtlds job to > > validate this. A program is invalid is there's an inconsistency. > > OK, this could be a problem with the GNU toolchain, secifically > with linking shared. You would expect that the relocations would > be trated as RTLD_NOW at link time and RTLD_LAZY at runtime; that > is, at link time, you would not be permitted to have symbols that > referenced a library function that referenced symbols which did not > resolve in the set of objects and binaries you were linking. This > is not the case. It's also not meaningful. There are no guarantees that the library you load at runtime is in fact the library that was given to the linker at link time. The only thing the linker has to do is to make sure all TLS symbols have TLS relocations. This guatantees that all ELF objects are consistent. The rtld then should not bind a TLS symbol to a non-TLS symbol. > The way you are *supposed* to ensure against this, according to the > linker's assumptions, is that all ELF binaries are linked shared, > and all shared libraries that depend on other shared libraries are > linked against them explicitly. Yes. > This is not the case on FreeBSD, which supports static linking, > and as you probably already know, static ELF libraries can't be > linked against other ELF libraries that way, so that they get > drug in automatically. But a static library is just another way of listing object files on the link line. They are part of the object your're linking. >From a dynamic linking poin of view they're not even libraries. Thus, if there's a dependency on some library L from within an object file in the archive library, you're supposed to put that library L on the link line as well. If L is a static library, this will be recursive. > > > This is where I personally have a problem with lazy intialization > > > of per thread TLS. Specifically, when a thread exits, you have to > > > know what you have and have not instanced, on a per dynamic object, > > > per thread basis, as a minimum granularity, in order to be able to > > > clean it up, without trying to clean up things you have not yet > > > instanced in that particular thread. > > > > This is where the DTV comes in. It's basicly a vector of TLS block > > pointers and each pointer/index corresponds to the TLS block of > > a shared library. At cleanup you iterate over the vector and clean > > all the non-NULL pointers. This is specific to the dynamic TLS > > model, BTW. > > This is the 256 entry pointer table from the spec., right? Yes. > I still don't think a program can know apriori what kind of a > reference to generate in its code: how does it know, at the > time an individual object is compiled, that you are going to > link it static or dynamic, so it should generate static vs. > dynamic references? A compiler generally won't know. So, one solution is to always generate dynamic TLS sequences. This however requires that the static binaries also contain __tls_get_addr(). GCC has tied this to -fpic. If you compile without -fpic, the code is unsuitable for shared libraries and static TLS sequences are generated. Thus, if you compile a staticly linked binary containing TLS with -fpic, you need __tls_get_addr() in your startup code or libc or where-ever. Note that the use of the __thread keyword does not mean you'll actually link against the thread library. > Or are you saying this only applies to dlopen, and not to linking > against a dynamic library... and that ld.so will need to know > which kind of thing it's link-loading, and act differently? ELF objects containing TLS will have a flag that is set when the ELF object contains static TLS access sequences. The rtld should reject loading such ELF objects. > Thanks for your incredible patience with us, No problem. The consequences of having the _thread keyword are tricky. We need to make sure we grasp it, that includes me. I may have been close to the fire, but that does not mean I actually know it all. Talking about it is one way to make sure that we don't rush things and consequently end up having it wrong. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 12:40:45 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1C6EA37B401 for ; Fri, 20 Jun 2003 12:40:45 -0700 (PDT) Received: from rwcrmhc11.attbi.com (rwcrmhc11.comcast.net [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 87E1C43F75 for ; Fri, 20 Jun 2003 12:40:44 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([12.233.125.100]) by attbi.com (rwcrmhc11) with ESMTP id <20030620194043013008rh0de>; Fri, 20 Jun 2003 19:40:44 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id MAA58558; Fri, 20 Jun 2003 12:40:43 -0700 (PDT) Date: Fri, 20 Jun 2003 12:40:42 -0700 (PDT) From: Julian Elischer To: Igor Sysoev In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 19:40:45 -0000 On Fri, 20 Jun 2003, Igor Sysoev wrote: > On Fri, 20 Jun 2003, Julian Elischer wrote: > > > On Fri, 20 Jun 2003, Igor Sysoev wrote: > > > > > If we could sacrifice our current thread compatibility in 5.0-5.1 > > > then we could change kse_mailbox from > > > > > > struct kse_mailbox { > > > int km_version; /* Mailbox version */ > > > struct kse_thr_mailbox *km_curthread; /* Current thread */ > > > ... > > > > > > to > > > > > > struct kse_mailbox { > > > struct kse_thr_mailbox *km_curthread; /* Current thread */ > > > int km_version; /* Mailbox version */ > > > ... > > > > > > then x86's gs would still point to kse_mailbox, and gs:[0] would be > > > and TP pointer. Also we need to modify > > > > > > struct kse_thr_mailbox { > > > + void *tls; > > > ucontext_t tm_context; /* User thread context */ > > > ... > > > > > > And the static TLS must be allocated before kse_thr_mailbox. > > > > Yes. > > This is what I was thinking... > > I'm wrong, this scheme does not allow to use the 1:1 model in libkse > when kse_mailbox.km_curthread is always NULL. I think we can still have a value at kse_mailbox.km_curthread even for 1:1 type threads.. > > We can implement such scheme on x86: > > gs -> [ TP ] ---> [ TLS ] > [ struct kse_mailbox ] +-> [ struct kse_thr_mailbox ] > [ .km_curthread ] -+ > > When UTS would switch to the next thread it should set thread's TLS: > > kse_mailbox.km_curthread = NULL; > gs:[0] = next_thr_tls; > kse_mailbox.km_curthread = next_kse_thr_mailbox; yes and the last line is atomic.. But remember having a NULL curhtread pointer stops upcalls but it is not the ONLY thing that stops upcalls.. A flag TMF_NOUPCALLS (spelling?) in the mailbox will also inhibit any upcalls. 1:1 threads (BOUND) threads, (system scope threads?) set this bit, but they still can have a mailbox for other purposes. (e.g. setting mode flags and stuff). If you are talking about libthr when you say 1:1 then they have gs:0 pointing to an array of pointers each of which points to a thread structure.. (they have the same indirection, but there is no KSE mailbox at teh indirection point, just the pointer.) (in _setcurthread.c ) void *ldt_entries[MAXTHR]; (these are set to point to thread structures as they are needed and %gs:0 points to an entry in this array) There is a small race we must guard against when accessing TLS.. %gs-->KSE--->TLS however the thread can be preemted between any two machine instructions, and unless the TMF_NOUPCALLS bit is set, it may start executing again under a DIFFERENT KSE. this means that we can not do: lea gs:0, %esi movl (%esi),%esi to find the TLS as at teh time of the 2nd command, we may have been pre-empted and %gs may point to a different place.. HOWEVER ensuring that we get past teh gs and into the actual thread-specific stuff in one instruction, e.g. movl gs:0, %esi ;%esi now points to a thread-specific thing.. should get around this.. > > kse_mailbox can be accessed via gs register. On amd64 scheme is the same > except the use of fs register instead of gs. > > However on sparc64, ia64 and alpha TP is in the register but not > in the memory so we need introduce the new field tm_kse in kse_thr_mailbox > to find kse_mailbox: > > TP -> [ TLS ] > [ struct kse_thr_mailbox ] > [ .tm_kse ] ---> [ struct kse_mailbox ] > > > By the way how was kse_mailbox being found before gs register was used ? > > > Igor Sysoev > http://sysoev.ru/en/ > > From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 13:43:16 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9773437B401 for ; Fri, 20 Jun 2003 13:43:16 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.bayarea.net [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4DCEA43FBF for ; Fri, 20 Jun 2003 13:43:15 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5KKhFDZ077892 for ; Fri, 20 Jun 2003 13:43:15 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5KKhER8019187 for ; Fri, 20 Jun 2003 13:43:14 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5KKhExk019186 for threads@FreeBSD.org; Fri, 20 Jun 2003 13:43:14 -0700 (PDT) (envelope-from marcel) Date: Fri, 20 Jun 2003 13:43:14 -0700 From: Marcel Moolenaar To: threads@FreeBSD.org Message-ID: <20030620204314.GA19111@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.4i Subject: TLS: defining the problem space X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 20:43:16 -0000 Gang, After the first round of discussions, it seems beneficial to draw the problem space. As far as I can see it, we have a 3-dimensional problem space: X. complete vs shared executable. Complete means static, but is a term used in certain environments. The advantage of complete in this context is that it allows us to use static to mean something else. The big difference between complete and shared is the presence (or absence) of RTLD. Y. static vs dynamic. This of course refers to the TLS model in use. A process can have both models in use at the same time!, but only if the process belongs to a complete binary (ie: you cannot have the static TLS model in shared executables, but you can have the dynamic TLS model in complete executables. The big difference between static and dynamic is the use of __tls_get_addr() to get the virtual address of a thread local variable (or not). Z. with pthread vs without pthread. This means whether a threads library (libc_r, libthr or libkse) is present and/or in use. The existence of the __thread keyword does not imply or mean that the process will be multi-threaded. This means that we have to deal with TLS access outside the context of RTLD or pthread. The big difference between pthread and without pthread is the ability to actually have multiple threads. Examples of problems we need to solve are: o In a complete executable, without pthread and using dynamic TLS, what defines __tls_get_addr() and how is it defined? o In a complete executable, without pthread and using static TLS, where is the thread pointer initialized? Examples of answers we have, but may need to revise: o In a complete executable, with pthread and using static TLS, the thread pointer is defined by the pthread library. o In a complete executable, with pthread and using dynamic TLS, __tls_get_addr() is defined by the pthread library. The intend of the posting is to come up with all questions and make sure we have them all answered before we do anything else. Note that answers may have tricky consequences, such as: If pthread defines __tls_get_addr() in complete executables and RTLD defines __tls_get_addr() in shared executables, we have duplicate definitions for shared executables, with pthread (using dynamic TLS). Which takes precedence? [answer: RTLD] It probably helps to have a authoritative list of questions and answers. I can probably create a trivial page somewhere where we list it... -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 13:46:07 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 496CB37B401 for ; Fri, 20 Jun 2003 13:46:07 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 741F843FA3 for ; Fri, 20 Jun 2003 13:46:06 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5KKk0Xh020715; Fri, 20 Jun 2003 16:46:00 -0400 (EDT) Date: Fri, 20 Jun 2003 16:46:00 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Igor Sysoev In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org cc: Julian Elischer Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 20:46:07 -0000 On Fri, 20 Jun 2003, Igor Sysoev wrote: > > I'm wrong, this scheme does not allow to use the 1:1 model in libkse > when kse_mailbox.km_curthread is always NULL. > > We can implement such scheme on x86: > > gs -> [ TP ] ---> [ TLS ] > [ struct kse_mailbox ] +-> [ struct kse_thr_mailbox ] > [ .km_curthread ] -+ > > When UTS would switch to the next thread it should set thread's TLS: > > kse_mailbox.km_curthread = NULL; FYI, The above has to be atomic. > gs:[0] = next_thr_tls; > kse_mailbox.km_curthread = next_kse_thr_mailbox; FYI2, The above need not be atomic since upcalls are disabled by the first line. > kse_mailbox can be accessed via gs register. On amd64 scheme is the same > except the use of fs register instead of gs. > > However on sparc64, ia64 and alpha TP is in the register but not > in the memory so we need introduce the new field tm_kse in kse_thr_mailbox > to find kse_mailbox: > > TP -> [ TLS ] > [ struct kse_thr_mailbox ] > [ .tm_kse ] ---> [ struct kse_mailbox ] > > > By the way how was kse_mailbox being found before gs register was used ? We've used %gs ever since we supported multiple KSEs. Before, it was just a port of libc_r with everything global and only 1 KSE. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 13:50:47 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1239237B401 for ; Fri, 20 Jun 2003 13:50:47 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5C09743FB1 for ; Fri, 20 Jun 2003 13:50:46 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5KKojXh021529; Fri, 20 Jun 2003 16:50:45 -0400 (EDT) Date: Fri, 20 Jun 2003 16:50:45 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Marcel Moolenaar In-Reply-To: <20030620204314.GA19111@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: TLS: defining the problem space X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 20:50:47 -0000 On Fri, 20 Jun 2003, Marcel Moolenaar wrote: > If pthread defines __tls_get_addr() in complete executables and RTLD > defines __tls_get_addr() in shared executables, we have duplicate > definitions for shared executables, with pthread (using dynamic TLS). > Which takes precedence? [answer: RTLD] You can have one __tls_get_addr() be a weak definition, and lib{thr,pthread,c_r} can define a non-weak definition. BTW, we can probably drop libc_r from the picture (mark it for deprecation). -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 13:59:47 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E3CDA37B401 for ; Fri, 20 Jun 2003 13:59:47 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.bayarea.net [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id AB74043FE0 for ; Fri, 20 Jun 2003 13:59:45 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5KKxjDZ077959; Fri, 20 Jun 2003 13:59:45 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5KKxjR8019249; Fri, 20 Jun 2003 13:59:45 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5KKxjLI019248; Fri, 20 Jun 2003 13:59:45 -0700 (PDT) (envelope-from marcel) Date: Fri, 20 Jun 2003 13:59:45 -0700 From: Marcel Moolenaar To: Daniel Eischen Message-ID: <20030620205945.GA19212@dhcp01.pn.xcllnt.net> References: <20030620204314.GA19111@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org Subject: Re: TLS: defining the problem space X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 20:59:48 -0000 On Fri, Jun 20, 2003 at 04:50:45PM -0400, Daniel Eischen wrote: > On Fri, 20 Jun 2003, Marcel Moolenaar wrote: > > If pthread defines __tls_get_addr() in complete executables and RTLD > > defines __tls_get_addr() in shared executables, we have duplicate > > definitions for shared executables, with pthread (using dynamic TLS). > > Which takes precedence? [answer: RTLD] > > You can have one __tls_get_addr() be a weak definition, and > lib{thr,pthread,c_r} can define a non-weak definition. Of course. The intend of the question was not to ask how, merely to ask which. The answer is not simple, because what if we also have a (weak) definition in libc to deal with the complete, without pthread and dynamic TLS case (if we want to support that)? Then we have 3 definitions. If both libc and pthread provide weak definitions, then how do we guarantee correct binding in the complete, with pthread and dynamic TLS case. If libc and RTLD provide weak definitions, then how do we guarantee proper binding in the shared, without pthread and dynamic TLS case? That's my point of having the problem space defined: we have a lot of cases to cover and I want it nailed down and documented before we do anything else. > BTW, we can probably drop libc_r from the picture (mark it > for deprecation). Yay! :-) -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 14:05:39 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6AF1E37B417 for ; Fri, 20 Jun 2003 14:05:39 -0700 (PDT) Received: from h132-197-179-27.gte.com (h132-197-179-27.gte.com [132.197.179.27]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8584C43F93 for ; Fri, 20 Jun 2003 14:05:38 -0700 (PDT) (envelope-from ak03@gte.com) Received: from kanpc.gte.com (ak03@localhost [127.0.0.1]) h5KL5bTO069076; Fri, 20 Jun 2003 17:05:37 -0400 (EDT) (envelope-from ak03@kanpc.gte.com) Received: (from ak03@localhost) by kanpc.gte.com (8.12.9/8.12.9/Submit) id h5KL5biG069075; Fri, 20 Jun 2003 17:05:37 -0400 (EDT) Date: Fri, 20 Jun 2003 17:05:37 -0400 From: Alexander Kabaev To: Marcel Moolenaar Message-Id: <20030620170537.0e03161b.ak03@gte.com> In-Reply-To: <20030620204314.GA19111@dhcp01.pn.xcllnt.net> References: <20030620204314.GA19111@dhcp01.pn.xcllnt.net> Organization: Verizon Data Services X-Mailer: Sylpheed version 0.9.0claws25 (GTK+ 1.2.10; i386-portbld-freebsd5.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit cc: threads@freebsd.org Subject: Re: TLS: defining the problem space X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 21:05:39 -0000 On Fri, 20 Jun 2003 13:43:14 -0700 Marcel Moolenaar wrote: > > Y. static vs dynamic. This of course refers to the TLS model in use. > A process can have both models in use at the same time!, but only > if the process belongs to a complete binary (ie: you cannot have > the static TLS model in shared executables, Unless I am missing something, this is not true. One can have static TLS in a dynamically linked binary as long all all modules using static access are part of the process startup set of libraries. One cannot dynamically load shared libraries compiled with static TLS model, but it is OK to link with them. > but you can have the > dynamic TLS model in complete executables. The big difference > between static and dynamic is the use of __tls_get_addr() to get > the virtual address of a thread local variable (or not). __tls_get_addr() use is a characteristic of the TLS model the module has compiled in, and not whether or not application is static or dynamic. > > Z. with pthread vs without pthread. This means whether a threads > library (libc_r, libthr or libkse) is present and/or in use. The > existence of the __thread keyword does not imply or mean that > the process will be multi-threaded. This means that we have to > deal with TLS access outside the context of RTLD or pthread. The > big difference between pthread and without pthread is the ability > to actually have multiple threads. > > Examples of problems we need to solve are: > o In a complete executable, without pthread and using dynamic TLS, > what defines __tls_get_addr() and how is it defined? How about weak libc symbol which will be overridden by rtld definotion? > o In a complete executable, without pthread and using static TLS, > where is the thread pointer initialized? There has to be a hook defined in our crt startup files. I have mentioned this initialization function before. > Examples of answers we have, but may need to revise: > o In a complete executable, with pthread and using static TLS, > the thread pointer is defined by the pthread library. > o In a complete executable, with pthread and using dynamic TLS, > __tls_get_addr() is defined by the pthread library. > > The intend of the posting is to come up with all questions and make > sure we have them all answered before we do anything else. Note that > answers may have tricky consequences, such as: > > If pthread defines __tls_get_addr() in complete executables and RTLD > defines __tls_get_addr() in shared executables, we have duplicate > definitions for shared executables, with pthread (using dynamic TLS). > Which takes precedence? [answer: RTLD] Can __tls_get_addr be included only in a static library. All dynamic libraries by necessity will have to depend on a declaration provided by the rtld then. Otherwise a weak symbol trick can be used as described above. This is no different from a case of libc providing dummy dlfcn.h stubs today. > It probably helps to have a authoritative list of questions and > answers. I can probably create a trivial page somewhere where we list > it... Good idea. -- Alexander Kabaev From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 14:08:18 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2589537B401 for ; Fri, 20 Jun 2003 14:08:18 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 834E043F75 for ; Fri, 20 Jun 2003 14:08:17 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5KL8GXh024323; Fri, 20 Jun 2003 17:08:16 -0400 (EDT) Date: Fri, 20 Jun 2003 17:08:16 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Marcel Moolenaar In-Reply-To: <20030620205945.GA19212@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: TLS: defining the problem space X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 21:08:18 -0000 On Fri, 20 Jun 2003, Marcel Moolenaar wrote: > On Fri, Jun 20, 2003 at 04:50:45PM -0400, Daniel Eischen wrote: > > On Fri, 20 Jun 2003, Marcel Moolenaar wrote: > > > If pthread defines __tls_get_addr() in complete executables and RTLD > > > defines __tls_get_addr() in shared executables, we have duplicate > > > definitions for shared executables, with pthread (using dynamic TLS). > > > Which takes precedence? [answer: RTLD] > > > > You can have one __tls_get_addr() be a weak definition, and > > lib{thr,pthread,c_r} can define a non-weak definition. > > Of course. The intend of the question was not to ask how, merely to > ask which. The answer is not simple, because what if we also have a > (weak) definition in libc to deal with the complete, without pthread > and dynamic TLS case (if we want to support that)? Then we have 3 > definitions. > If both libc and pthread provide weak definitions, then how do we > guarantee correct binding in the complete, with pthread and dynamic > TLS case. If libc and RTLD provide weak definitions, then how do we > guarantee proper binding in the shared, without pthread and dynamic > TLS case? libthr,pthread don't have to supply __tls_get_addr(). They can tell rtld that they are there and pass in a differently named function (during rtld_lock_init()). So now you're back down to two definitions. Does that help? > That's my point of having the problem space defined: we have a lot > of cases to cover and I want it nailed down and documented before we > do anything else. Of course. I'm just offering a possibility. Feel free to toss it if there are better ways. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 14:17:14 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8399337B401 for ; Fri, 20 Jun 2003 14:17:14 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id DD47043F93 for ; Fri, 20 Jun 2003 14:17:13 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.8/8.12.1) with ESMTP id h5KLHCXh025650; Fri, 20 Jun 2003 17:17:12 -0400 (EDT) Date: Fri, 20 Jun 2003 17:17:12 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Marcel Moolenaar In-Reply-To: <20030620205945.GA19212@dhcp01.pn.xcllnt.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: TLS: defining the problem space X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 21:17:14 -0000 On Fri, 20 Jun 2003, Marcel Moolenaar wrote: > If both libc and pthread provide weak definitions, then how do we > guarantee correct binding in the complete, with pthread and dynamic > TLS case. If libc and RTLD provide weak definitions, then how do we > guarantee proper binding in the shared, without pthread and dynamic > TLS case? I don't know if it helps, but here's some info from a Solaris 9 box (UltraSparc): gpz $ nm /usr/lib/libc.so.1 | grep tls_ gpz $ nm /usr/lib/ld.so.1 | grep tls_ [829] | 0| 0|FUNC |GLOB |0 |UNDEF |Dbg_tls_modactivity [875] | 0| 0|FUNC |GLOB |0 |UNDEF |Dbg_tls_static_block [568] | 221816| 4|OBJT |LOCL |0 |15 |fptr_tls_modadd [569] | 221820| 4|OBJT |LOCL |0 |15 |fptr_tls_modrem [570] | 221824| 4|OBJT |LOCL |0 |15 |fptr_tls_statmods [220] | 85632| 84|FUNC |LOCL |0 |10 |tls_assign_soffset [140] | 85176| 96|FUNC |LOCL |0 |10 |tls_freemodid [116] | 84808| 368|FUNC |LOCL |0 |10 |tls_getmodid [361] | 85348| 284|FUNC |LOCL |0 |10 |tls_modactivity [64] | 85716| 688|FUNC |LOCL |0 |10 |tls_report_modules [398] | 85272| 76|FUNC |LOCL |0 |10 |tls_setroutines [572] | 221832| 4|OBJT |LOCL |0 |15 |tls_static_size gpz $ nm /usr/lib/libthread.so.1 | grep tls_ [986] | 90128| 60|FUNC |GLOB |0 |10 |__tls_get_addr [287] | 85584| 228|FUNC |LOCL |0 |10 |__tls_mod_add [71] | 85812| 72|FUNC |LOCL |0 |10 |__tls_mod_remove [178] | 85260| 324|FUNC |LOCL |0 |10 |__tls_static_mods [448] | 85884| 392|FUNC |LOCL |0 |10 |slow_tls_get_addr [403] | 86488| 296|FUNC |LOCL |0 |10 |tls_exit [52] | 86784| 148|FUNC |LOCL |0 |10 |tls_free [611] | 167144| 24|OBJT |LOCL |0 |17 |tls_lock [410] | 172176| 8|OBJT |LOCL |0 |20 |tls_modinfo [252] | 168712| 40|OBJT |LOCL |0 |19 |tls_rtldinfo [407] | 86276| 212|FUNC |LOCL |0 |10 |tls_setup gpz $ nm /usr/lib/libpthread.so.1 | grep tls_ libthread seems to be the only one defining __tls_get_addr(). -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 14:21:51 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 01D8A37B401 for ; Fri, 20 Jun 2003 14:21:51 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.bayarea.net [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 868FB43F3F for ; Fri, 20 Jun 2003 14:21:48 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5KLLiDZ078111; Fri, 20 Jun 2003 14:21:45 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5KLLiR8019314; Fri, 20 Jun 2003 14:21:44 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5KLLiu8019313; Fri, 20 Jun 2003 14:21:44 -0700 (PDT) (envelope-from marcel) Date: Fri, 20 Jun 2003 14:21:44 -0700 From: Marcel Moolenaar To: Alexander Kabaev Message-ID: <20030620212144.GB19212@dhcp01.pn.xcllnt.net> References: <20030620204314.GA19111@dhcp01.pn.xcllnt.net> <20030620170537.0e03161b.ak03@gte.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030620170537.0e03161b.ak03@gte.com> User-Agent: Mutt/1.5.4i cc: threads@freebsd.org Subject: Re: TLS: defining the problem space X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 21:21:51 -0000 On Fri, Jun 20, 2003 at 05:05:37PM -0400, Alexander Kabaev wrote: > On Fri, 20 Jun 2003 13:43:14 -0700 > Marcel Moolenaar wrote: > > > > Y. static vs dynamic. This of course refers to the TLS model in use. > > A process can have both models in use at the same time!, but only > > if the process belongs to a complete binary (ie: you cannot have > > the static TLS model in shared executables, > > Unless I am missing something, this is not true. One can have static > TLS in a dynamically linked binary as long all all modules using static > access are part of the process startup set of libraries. One cannot > dynamically load shared libraries compiled with static TLS model, but it > is OK to link with them. Ah, yes. True. You can make it work by having the thread pointer point to the TLS of the startup as well as have them accessable through the DTV. Excellent point. > > but you can have the > > dynamic TLS model in complete executables. The big difference > > between static and dynamic is the use of __tls_get_addr() to get > > the virtual address of a thread local variable (or not). > > __tls_get_addr() use is a characteristic of the TLS model the module has > compiled in, and not whether or not application is static or dynamic. Yes. My wording may have been sloppy enough to suggest an implication. They are correctly identified as different dimensions in the problem space. That should be considered authoritative. > > Examples of problems we need to solve are: > > o In a complete executable, without pthread and using dynamic TLS, > > what defines __tls_get_addr() and how is it defined? > > How about weak libc symbol which will be overridden by rtld definotion? My thought as well, but we may also need a definition in the threads library as well as in rtld. This means that we can have a case in which we only have weak definitions and thus no guranteed correct binding. > > If pthread defines __tls_get_addr() in complete executables and RTLD > > defines __tls_get_addr() in shared executables, we have duplicate > > definitions for shared executables, with pthread (using dynamic TLS). > > Which takes precedence? [answer: RTLD] > > Can __tls_get_addr be included only in a static library. All dynamic > libraries by necessity will have to depend on a declaration provided by > the rtld then. Otherwise a weak symbol trick can be used as described > above. This is no different from a case of libc providing dummy dlfcn.h > stubs today. That's a possibility. But I don't want to think too much in terms of implementation at this stage. I want to have a conceptual view as well as have implementation alternatives. The conceptual view should help us to resolve conflicts caused by implementation assumptions. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 14:25:06 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7E3BF37B401; Fri, 20 Jun 2003 14:25:06 -0700 (PDT) Received: from h132-197-179-27.gte.com (h132-197-179-27.gte.com [132.197.179.27]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9B73643F75; Fri, 20 Jun 2003 14:25:05 -0700 (PDT) (envelope-from ak03@gte.com) Received: from kanpc.gte.com (ak03@localhost [127.0.0.1]) h5KLP4TO069387; Fri, 20 Jun 2003 17:25:04 -0400 (EDT) (envelope-from ak03@kanpc.gte.com) Received: (from ak03@localhost) by kanpc.gte.com (8.12.9/8.12.9/Submit) id h5KLP4tq069386; Fri, 20 Jun 2003 17:25:04 -0400 (EDT) Date: Fri, 20 Jun 2003 17:25:04 -0400 From: Alexander Kabaev To: deischen@freebsd.org Message-Id: <20030620172504.4b35f4be.ak03@gte.com> In-Reply-To: References: <20030620205945.GA19212@dhcp01.pn.xcllnt.net> Organization: Verizon Data Services X-Mailer: Sylpheed version 0.9.0claws25 (GTK+ 1.2.10; i386-portbld-freebsd5.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit cc: threads@freebsd.org cc: Marcel Moolenaar Subject: Re: TLS: defining the problem space X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 21:25:06 -0000 On Fri, 20 Jun 2003 17:17:12 -0400 (EDT) Daniel Eischen wrote: > tls_setroutines ^^^^^ I bet this is the function libpthread calls to register its double underscored functions with ld.so :) -- Alexander Kabaev From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 14:27:15 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D305C37B401; Fri, 20 Jun 2003 14:27:14 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.bayarea.net [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0607C43FAF; Fri, 20 Jun 2003 14:27:14 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from dhcp01.pn.xcllnt.net (dhcp01.pn.xcllnt.net [192.168.4.201]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5KLRDDZ078163; Fri, 20 Jun 2003 14:27:13 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from dhcp01.pn.xcllnt.net (localhost [127.0.0.1]) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5KLRDR8019340; Fri, 20 Jun 2003 14:27:13 -0700 (PDT) (envelope-from marcel@dhcp01.pn.xcllnt.net) Received: (from marcel@localhost) by dhcp01.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5KLRDCt019339; Fri, 20 Jun 2003 14:27:13 -0700 (PDT) (envelope-from marcel) Date: Fri, 20 Jun 2003 14:27:13 -0700 From: Marcel Moolenaar To: deischen@freebsd.org Message-ID: <20030620212713.GC19212@dhcp01.pn.xcllnt.net> References: <20030620205945.GA19212@dhcp01.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.4i cc: threads@freebsd.org Subject: Re: TLS: defining the problem space X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 21:27:15 -0000 On Fri, Jun 20, 2003 at 05:08:16PM -0400, Daniel Eischen wrote: > > > > > > You can have one __tls_get_addr() be a weak definition, and > > > lib{thr,pthread,c_r} can define a non-weak definition. > > > > Of course. The intend of the question was not to ask how, merely to > > ask which. The answer is not simple, because what if we also have a > > (weak) definition in libc to deal with the complete, without pthread > > and dynamic TLS case (if we want to support that)? Then we have 3 > > definitions. > > If both libc and pthread provide weak definitions, then how do we > > guarantee correct binding in the complete, with pthread and dynamic > > TLS case. If libc and RTLD provide weak definitions, then how do we > > guarantee proper binding in the shared, without pthread and dynamic > > TLS case? > > libthr,pthread don't have to supply __tls_get_addr(). They > can tell rtld that they are there and pass in a differently > named function (during rtld_lock_init()). So now you're > back down to two definitions. Does that help? Yes, but it is yet again at the level of the implementation. At the conceptual level there are still 3 different definitions and it's our job to find the implementation that deals with all the cases without ambiguity. Again: we need to keep close to the conceptual level to keep a clear view of things. Once we sink down to the implementation level, things get messy pretty quickly. We need to have something to fall back (up) to if we want to avoid loosing the overview. -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net From owner-freebsd-threads@FreeBSD.ORG Fri Jun 20 15:33:35 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 95B9237B405; Fri, 20 Jun 2003 15:33:35 -0700 (PDT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 25D3B43F75; Fri, 20 Jun 2003 15:33:35 -0700 (PDT) (envelope-from davidxu@freebsd.org) Received: from tiger (davidxu@localhost [127.0.0.1]) by freefall.freebsd.org (8.12.9/8.12.9) with SMTP id h5KMXVUp070110; Fri, 20 Jun 2003 15:33:33 -0700 (PDT) (envelope-from davidxu@freebsd.org) Message-ID: <002b01c3377c$6ebcb880$0701a8c0@tiger> From: "David Xu" To: "Igor Sysoev" , "Julian Elischer" References: Date: Sat, 21 Jun 2003 06:36:44 +0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: David Xu List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 22:33:36 -0000 ----- Original Message -----=20 From: "Igor Sysoev" To: "Julian Elischer" Cc: Sent: Saturday, June 21, 2003 2:27 AM Subject: Re: Implementing TLS: step 1 > On Fri, 20 Jun 2003, Julian Elischer wrote: >=20 > > On Fri, 20 Jun 2003, Igor Sysoev wrote: > >=20 > > > If we could sacrifice our current thread compatibility in 5.0-5.1 > > > then we could change kse_mailbox from > > >=20 > > > struct kse_mailbox { > > > int km_version; /* Mailbox = version */ > > > struct kse_thr_mailbox *km_curthread; /* Current = thread */ > > > ... > > >=20 > > > to > > > =20 > > > struct kse_mailbox {=20 > > > struct kse_thr_mailbox *km_curthread; /* Current = thread */ > > > int km_version; /* Mailbox = version */ > > > ... > > >=20 > > > then x86's gs would still point to kse_mailbox, and gs:[0] would = be > > > and TP pointer. Also we need to modify > > >=20 > > > struct kse_thr_mailbox { > > > + void *tls; > > > ucontext_t tm_context; /* User thread = context */ > > > ... > > >=20 > > > And the static TLS must be allocated before kse_thr_mailbox. > >=20 > > Yes. > > This is what I was thinking... >=20 > I'm wrong, this scheme does not allow to use the 1:1 model in libkse > when kse_mailbox.km_curthread is always NULL. >=20 > We can implement such scheme on x86: >=20 > gs -> [ TP ] ---> [ TLS ] > [ struct kse_mailbox ] +-> [ struct kse_thr_mailbox ] > [ .km_curthread ] -+ >=20 > When UTS would switch to the next thread it should set thread's TLS: >=20 > kse_mailbox.km_curthread =3D NULL; > gs:[0] =3D next_thr_tls; > kse_mailbox.km_curthread =3D next_kse_thr_mailbox; >=20 > kse_mailbox can be accessed via gs register. On amd64 scheme is the = same > except the use of fs register instead of gs. >=20 > However on sparc64, ia64 and alpha TP is in the register but not > in the memory so we need introduce the new field tm_kse in = kse_thr_mailbox > to find kse_mailbox: >=20 > TP -> [ TLS ] > [ struct kse_thr_mailbox ] > [ .tm_kse ] ---> [ struct kse_mailbox ] >=20 >=20 > By the way how was kse_mailbox being found before gs register was used = ? >=20 In libpthread, a LDT entry is allocated and %gs is loaded as this value when libpthread is being initialized. For newly created kse, when an upcall is made at first time for that kse, in userland upcall handler, the %gs will be loaded with its new value which libpthread already allocated for it and put it in kse structure, keep in mind when=20 kernel is doing upcall, it will always push current kse pointer to user stack and then calls userland upcall handler, so the handler can always find its kse, and find its %gs value in kse structure. >=20 > Igor Sysoev > http://sysoev.ru/en/ >=20 > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to = "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Sat Jun 21 02:50:21 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D658E37B401 for ; Sat, 21 Jun 2003 02:50:21 -0700 (PDT) Received: from park.rambler.ru (park.rambler.ru [81.19.64.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3DB2B43F75 for ; Sat, 21 Jun 2003 02:50:20 -0700 (PDT) (envelope-from is@rambler-co.ru) Received: from is.park.rambler.ru (is.park.rambler.ru [81.19.64.102]) by park.rambler.ru (8.12.6/8.12.6) with ESMTP id h5L9oEmF068762; Sat, 21 Jun 2003 13:50:14 +0400 (MSD) Date: Sat, 21 Jun 2003 13:50:14 +0400 (MSD) From: Igor Sysoev X-Sender: is@is To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: Implementing TLS: step 1 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Jun 2003 09:50:22 -0000 On Fri, 20 Jun 2003, Julian Elischer wrote: > > We can implement such scheme on x86: > > > > gs -> [ TP ] ---> [ TLS ] > > [ struct kse_mailbox ] +-> [ struct kse_thr_mailbox ] > > [ .km_curthread ] -+ > > > > When UTS would switch to the next thread it should set thread's TLS: > > > > kse_mailbox.km_curthread = NULL; > > gs:[0] = next_thr_tls; > > kse_mailbox.km_curthread = next_kse_thr_mailbox; > > yes and the last line is atomic.. But remember having a NULL curhtread > pointer stops upcalls but it is not the ONLY thing that stops upcalls.. > A flag TMF_NOUPCALLS (spelling?) in the mailbox will also inhibit any > upcalls. 1:1 threads (BOUND) threads, (system scope threads?) set this > bit, but they still can have a mailbox for other purposes. (e.g. setting > mode flags and stuff). So NULL curthread is the short term (in UTS only) and atomic method to disable upcalls while KMF_NOUPCALL flag is the long term and non-atomic (we can not atomically update bit masks in general) method ? Igor Sysoev http://sysoev.ru/en/ From owner-freebsd-threads@FreeBSD.ORG Sat Jun 21 04:06:22 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 980B837B401; Sat, 21 Jun 2003 04:06:22 -0700 (PDT) Received: from obsecurity.dyndns.org (adsl-64-169-104-32.dsl.lsan03.pacbell.net [64.169.104.32]) by mx1.FreeBSD.org (Postfix) with ESMTP id CF8F243F93; Sat, 21 Jun 2003 04:06:21 -0700 (PDT) (envelope-from kris@obsecurity.org) Received: from rot13.obsecurity.org (rot13.obsecurity.org [10.0.0.5]) by obsecurity.dyndns.org (Postfix) with ESMTP id 7261266CFB; Sat, 21 Jun 2003 04:06:21 -0700 (PDT) Received: by rot13.obsecurity.org (Postfix, from userid 1000) id 3A19E5D6; Sat, 21 Jun 2003 04:06:21 -0700 (PDT) Date: Sat, 21 Jun 2003 04:06:20 -0700 From: Kris Kennaway To: sparc64@FreeBSD.org Message-ID: <20030621110620.GA72595@rot13.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="r5Pyd7+fXNt84Ff3" Content-Disposition: inline User-Agent: Mutt/1.4.1i cc: threads@FreeBSD.org Subject: MozillaFirebird with libthr on sparc64 dies with SIGILL X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Jun 2003 11:06:23 -0000 --r5Pyd7+fXNt84Ff3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline I thought I'd try libthr to see if it fixes the crashes/hangs I'm seeing with MozillaFirebird on sparc64, but it dies with a SIGILL during startup. libpthread is unimplemented on sparc64. Any ideas? Kris # /etc/libmap.conf # # candidate mapping # libc_r.so.5 libthr.so.1 # Everything uses 'libthr' libc_r.so libthr.so > gdb53 MozillaFirebird MozillaFirebird-bin.core GNU gdb 5.3 (FreeBSD) Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "sparc64-portbld-freebsd5.1"..."/usr/X11R6/bin/MozillaFirebird": not in executable format: File format not recognized Core was generated by `MozillaFirebird-bin'. Program terminated with signal 4, Illegal instruction. #0 0x0000000041a64d20 in ?? () (gdb) disassemble 0x0000000041a64d20 No function contains specified address. (gdb) bt #0 0x0000000041a64d20 in ?? () Cannot access memory at address 0x7fdffed9fb8 --r5Pyd7+fXNt84Ff3 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (FreeBSD) iD8DBQE+9DwsWry0BWjoQKURAuvjAJ4ugc9hyNc/FG8sivaEGAPy92fbHwCdGv9+ sVRaWj8fWWdYeJ0JSIt/ZbA= =P54d -----END PGP SIGNATURE----- --r5Pyd7+fXNt84Ff3-- From owner-freebsd-threads@FreeBSD.ORG Sat Jun 21 10:22:35 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ACDAF37B401; Sat, 21 Jun 2003 10:22:35 -0700 (PDT) Received: from pop018.verizon.net (pop018pub.verizon.net [206.46.170.212]) by mx1.FreeBSD.org (Postfix) with ESMTP id 641FD43F85; Sat, 21 Jun 2003 10:22:34 -0700 (PDT) (envelope-from mtm@identd.net) Received: from kokeb.ambesa.net ([138.88.140.205]) by pop018.verizon.net (InterMail vM.5.01.05.33 201-253-122-126-133-20030313) with ESMTP id <20030621172233.XILJ11703.pop018.verizon.net@kokeb.ambesa.net>; Sat, 21 Jun 2003 12:22:33 -0500 Date: Sat, 21 Jun 2003 13:22:32 -0400 From: Mike Makonnen To: Kris Kennaway In-Reply-To: <20030621110620.GA72595@rot13.obsecurity.org> References: <20030621110620.GA72595@rot13.obsecurity.org> X-Mailer: Sylpheed version 0.8.10 (GTK+ 1.2.10; i386-portbld-freebsd5.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Authentication-Info: Submitted using SMTP AUTH at pop018.verizon.net from [138.88.140.205] at Sat, 21 Jun 2003 12:22:33 -0500 Message-Id: <20030621172233.XILJ11703.pop018.verizon.net@kokeb.ambesa.net> cc: threads@freebsd.org cc: sparc64@freebsd.org Subject: Re: MozillaFirebird with libthr on sparc64 dies with SIGILL X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Jun 2003 17:22:36 -0000 On Sat, 21 Jun 2003 04:06:20 -0700 Kris Kennaway wrote: > I thought I'd try libthr to see if it fixes the crashes/hangs I'm > seeing with MozillaFirebird on sparc64, but it dies with a SIGILL > during startup. libpthread is unimplemented on sparc64. Any ideas? > [snip] > > Core was generated by `MozillaFirebird-bin'. > Program terminated with signal 4, Illegal instruction. > #0 0x0000000041a64d20 in ?? () > (gdb) disassemble 0x0000000041a64d20 > No function contains specified address. > > (gdb) bt > #0 0x0000000041a64d20 in ?? () > Cannot access memory at address 0x7fdffed9fb8 I was looking into libthr and sparc64 the the other day. I know that all the necessary functions are implemented, but I don't know if they need tweaking or not. OTOH I have seen similar results when linked against C++ applications. Sorry I can't be of more help at the moment :( Cheers. -- Mike Makonnen | GPG-KEY: http://www.identd.net/~mtm/mtm.asc mtm@identd.net | D228 1A6F C64E 120A A1C9 A3AA DAE1 E2AF DBCC 68B9 mtm@FreeBSD.Org| FreeBSD - The Power To Serve From owner-freebsd-threads@FreeBSD.ORG Sat Jun 21 10:40:53 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4927537B401 for ; Sat, 21 Jun 2003 10:40:53 -0700 (PDT) Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by mx1.FreeBSD.org (Postfix) with SMTP id 9A84143FD7 for ; Sat, 21 Jun 2003 10:40:51 -0700 (PDT) (envelope-from tmoestl@gmx.net) Received: (qmail 10006 invoked by uid 65534); 21 Jun 2003 17:40:50 -0000 Received: from p508E545F.dip.t-dialin.net (EHLO galatea.local) (80.142.84.95) by mail.gmx.net (mp006) with SMTP; 21 Jun 2003 19:40:50 +0200 Received: from tmm by galatea.local with local (Exim 4.20 #1) id 19TmFf-0000zi-71; Sat, 21 Jun 2003 19:34:11 +0200 Date: Sat, 21 Jun 2003 19:34:10 +0200 From: Thomas Moestl To: Kris Kennaway Message-ID: <20030621173410.GB656@crow.dom2ip.de> Mail-Followup-To: Kris Kennaway , sparc64@freebsd.org, threads@freebsd.org References: <20030621110620.GA72595@rot13.obsecurity.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030621110620.GA72595@rot13.obsecurity.org> User-Agent: Mutt/1.4.1i Sender: Thomas Moestl cc: threads@freebsd.org cc: sparc64@freebsd.org Subject: Re: MozillaFirebird with libthr on sparc64 dies with SIGILL X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Jun 2003 17:40:53 -0000 On Sat, 2003/06/21 at 04:06:20 -0700, Kris Kennaway wrote: > I thought I'd try libthr to see if it fixes the crashes/hangs I'm > seeing with MozillaFirebird on sparc64, but it dies with a SIGILL > during startup. libpthread is unimplemented on sparc64. Any ideas? > > Kris > > # /etc/libmap.conf > # > # candidate mapping > # > libc_r.so.5 libthr.so.1 # Everything uses 'libthr' > libc_r.so libthr.so > > > gdb53 MozillaFirebird MozillaFirebird-bin.core > GNU gdb 5.3 (FreeBSD) > Copyright 2002 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "sparc64-portbld-freebsd5.1"..."/usr/X11R6/bin/MozillaFirebird": not in executable format: File format not recognized MozillaFirebird is just a shell script; the core seems to have been generated by /usr/X11R6/lib/firebird/mozilla-1.4b/MozillaFirebird-bin. - Thomas -- Thomas Moestl http://www.tu-bs.de/~y0015675/ http://people.FreeBSD.org/~tmm/ PGP fingerprint: 1C97 A604 2BD0 E492 51D0 9C0F 1FE6 4F1D 419C 776C From owner-freebsd-threads@FreeBSD.ORG Sat Jun 21 14:05:13 2003 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5F5CA37B401; Sat, 21 Jun 2003 14:05:13 -0700 (PDT) Received: from obsecurity.dyndns.org (adsl-64-169-104-32.dsl.lsan03.pacbell.net [64.169.104.32]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7DAB243F3F; Sat, 21 Jun 2003 14:05:12 -0700 (PDT) (envelope-from kris@obsecurity.org) Received: from rot13.obsecurity.org (rot13.obsecurity.org [10.0.0.5]) by obsecurity.dyndns.org (Postfix) with ESMTP id 31B3866BE5; Sat, 21 Jun 2003 14:05:12 -0700 (PDT) Received: by rot13.obsecurity.org (Postfix, from userid 1000) id B4F15796; Sat, 21 Jun 2003 14:05:11 -0700 (PDT) Date: Sat, 21 Jun 2003 14:05:11 -0700 From: Kris Kennaway To: Morten Rodal Message-ID: <20030621210511.GA82582@rot13.obsecurity.org> References: <20030621110620.GA72595@rot13.obsecurity.org> <20030621132803.GA1608@atlantis.rodal.no> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="cNdxnHkX5QqsyA0e" Content-Disposition: inline In-Reply-To: <20030621132803.GA1608@atlantis.rodal.no> User-Agent: Mutt/1.4.1i cc: threads@FreeBSD.org cc: sparc64@FreeBSD.org Subject: Re: MozillaFirebird with libthr on sparc64 dies with SIGILL X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Jun 2003 21:05:13 -0000 --cNdxnHkX5QqsyA0e Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jun 21, 2003 at 03:28:03PM +0200, Morten Rodal wrote: > On Sat, Jun 21, 2003 at 04:06:20AM -0700, Kris Kennaway wrote: > > > gdb53 MozillaFirebird MozillaFirebird-bin.core > > GNU gdb 5.3 (FreeBSD) > > Copyright 2002 Free Software Foundation, Inc. > > GDB is free software, covered by the GNU General Public License, and yo= u are > > welcome to change it and/or distribute copies of it under certain condi= tions. > > Type "show copying" to see the conditions. > > There is absolutely no warranty for GDB. Type "show warranty" for deta= ils. > > This GDB was configured as "sparc64-portbld-freebsd5.1"..."/usr/X11R6/b= in/MozillaFirebird": not in executable format: File format not recognized >=20 > Maybe you should try to run gdb with > "/usr/X11R6/lib/firebird/lib/mozilla-1.4b/MozillaFirebird-bin" instead > of "MozillaFirebird" (since that is a shell script)? Heh, oops..that's what I get for trying to debug at 4AM! (gdb) disassemble Dump of assembler code for function _ctx_start: 0x41a64d20 <_ctx_start>: call %g1 0x41a64d24 <_ctx_start+4>: mov %g2, %l0 0x41a64d28 <_ctx_start+8>: call 0x41b94320 <__isthreaded+17712> 0x41a64d2c <_ctx_start+12>: mov %l0, %o0 0x41a64d30 <_ctx_start+16>: illtrap 0 0x41a64d34 <_ctx_start+20>: b,a %xcc, 0x41a64d40 0x41a64d38 <_ctx_start+24>: nop 0x41a64d3c <_ctx_start+28>: nop End of assembler dump. (gdb) bt #0 0x0000000041a64d20 in _ctx_start () from /usr/lib/libc.so.5 Looks more sane :) Kris --cNdxnHkX5QqsyA0e Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (FreeBSD) iD8DBQE+9MiHWry0BWjoQKURAgyGAJ9OePZXLlwz2KuKVDQ+V8qq2FSlvwCguQGc CRzIgKr5HSZvDid2WJJujBk= =3ihY -----END PGP SIGNATURE----- --cNdxnHkX5QqsyA0e--