Date: Thu, 10 Feb 2005 18:28:51 -0800 From: "Medi Montaseri" <mmontaseri@amcc.com> To: Sean Chittenden <sean@gigave.com> Cc: threads@FreeBSD.org Subject: Re: Strange backtrace from amd64 + mysqld... Message-ID: <420C1863.1010609@amcc.com> In-Reply-To: <20050211001252.GY1060@sean.gigave.com> References: <20050211001252.GY1060@sean.gigave.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I have also been having problem with my multi-threaded app on FreeBSD 5.3.
My code starts up, initializes ok, and then creates a new thread. Inside
the thread
I do some OpenSSL initializations and based on gdb I seg fault in
SSL_libraray_init().
I see that your gdb is showing fault activity at
pthread_mutexattr_init(). I also see
this on one of my threads. I'm not sure if they are related but provided
here for those
who know more about threads..
medit64# gdb dapid
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
(gdb) run -c dapid.cfg
Starting program: /aux/swg/medi/src/64_530_emt/gigapath/sw/dapi/dapid -c
dapid.cfg
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 3 (LWP 100119)]
ssl_auth_init (soap=0x7fffffede980) at stdsoap2_cpp.cpp:2381
2381 SSL_library_init();
(gdb) bt
#0 ssl_auth_init (soap=0x7fffffede980) at stdsoap2_cpp.cpp:2381
#1 0x000000000056fb4e in soap_ssl_server_context (soap=0x7fffffede980,
flags=0,
keyfile=0x6fe3d8 "server.pem", password=0x7006f8 "dapi", cafile=0x0,
capath=0x0, dhfile=0x0,
randfile=0x0, sid=0x700ef8 "medit64") at stdsoap2_cpp.cpp:2349
#2 0x00000000004e5f82 in NetMgr::process (this=0x70d1a0) at
basic_string.h:1452
#3 0x000000000049ea25 in startNetMgr () at DapiMgr.h:66
#4 0x0000000800c7d039 in pthread_create () from /usr/lib/libpthread.so.1
#5 0x0000000801108034 in makecontext () from /lib/libc.so.5
#6 0x0000000000000000 in ?? ()
#7 0x0000000000709800 in ?? ()
#8 0x000000000049ea00 in startAenMgr () at DapiMgr.cpp:273
#9 0x0000000000000000 in ?? ()
#10 0x0000000000000000 in ?? ()
#11 0x0000000000000000 in ?? ()
#12 0x0000000000000000 in ?? ()
Error accessing memory address 0x7fffffeee000: Bad address.
(gdb) thread 2
[Switching to thread 2 (Thread 1 (sleeping))]#0 0x0000000800c88557 in
pthread_mutexattr_init ()
from /usr/lib/libpthread.so.1
(gdb) bt
#0 0x0000000800c88557 in pthread_mutexattr_init () from
/usr/lib/libpthread.so.1
#1 0x0000000800c8396f in _nanosleep () from /usr/lib/libpthread.so.1
#2 0x000000080110ac50 in sleep () from /lib/libc.so.5
#3 0x0000000800c79846 in sleep () from /usr/lib/libpthread.so.1
#4 0x000000000049f123 in DapiMgr::process (this=0x7014d8) at
DapiMgr.cpp:218
#5 0x0000000000406dba in main (argc=3, argv=0x7fffffffead8) at dapid.cpp:57
(gdb)
Sean Chittenden wrote:
>Howdy. I'm running into a strange issue with MySQL 4.1.9 wherein its
>eating itself every 24hrs. I grabbed a back trace, but haven't
>recompiled MySQL with debugging given the strangeness of the backtrace
>(can't tell if this is gdb or if the call stack is massively corrupt
>due to some error). Given I've watched this repeat itself a few times
>now and it seems to be dying in a pthread call, I'm posting here.
>MySQL is the only process having problems, but it's also the most
>heavily threaded and is getting pounded pretty hard. Without going
>back to MySQL (esp given the backtrace), I want to verify that this
>isn't a pthreads issue. Can someone comment on whether or not they've
>seen this before?
>
>What really bothers me about this, is if I restart mysqld by hand (ie,
>not from mysqld_safe), the database comes up and behaves correctly for
>~24hrs before it starts to puke again. When it does, it cores every
>few minutes until I bounce it by hand. In the meantime, because it's
>dumping out core files, the system becomes reasonably unusable.
>Thoughts, or is this there some coincidental behavior going on?
>
>FreeBSD host.example.com 5.3-STABLE FreeBSD 5.3-STABLE #1: Mon Feb 7 17:35:07 PST 2005 root@host.example.com:/usr/obj/usr/src/sys/CONFIG amd64
>
>src is pulled from Feb-07 too. -sc
>
>
>GNU gdb 6.1.1 [FreeBSD]
>Copyright 2004 Free Software Foundation, Inc.
>GDB is free software, covered by the GNU General Public License, and you are
>welcome to change it and/or distribute copies of it under certain conditions.
>Type "show copying" to see the conditions.
>There is absolutely no warranty for GDB. Type "show warranty" for details.
>This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)...
>Core was generated by `mysqld'.
>Program terminated with signal 11, Segmentation fault.
>Reading symbols from /lib/libz.so.2...(no debugging symbols found)...done.
>Loaded symbols for /lib/libz.so.2
>Reading symbols from /usr/lib/libwrap.so.3...(no debugging symbols found)...done.
>Loaded symbols for /usr/lib/libwrap.so.3
>Reading symbols from /lib/libcrypt.so.2...(no debugging symbols found)...done.
>Loaded symbols for /lib/libcrypt.so.2
>Reading symbols from /usr/lib/libstdc++.so.4...(no debugging symbols found)...done.
>Loaded symbols for /usr/lib/libstdc++.so.4
>Reading symbols from /lib/libm.so.3...(no debugging symbols found)...done.
>Loaded symbols for /lib/libm.so.3
>Reading symbols from /usr/lib/libpthread.so.1...(no debugging symbols found)...done.
>Loaded symbols for /usr/lib/libpthread.so.1
>Reading symbols from /lib/libc.so.5...(no debugging symbols found)...done.
>Loaded symbols for /lib/libc.so.5
>Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done.
>Loaded symbols for /libexec/ld-elf.so.1
>#0 0x0000000801017529 in pthread_mutexattr_init () from /usr/lib/libpthread.so.1
>(gdb) bt
>#0 0x0000000801017529 in pthread_mutexattr_init () from /usr/lib/libpthread.so.1
>#1 0x0000000801017875 in pthread_mutexattr_init () from /usr/lib/libpthread.so.1
>#2 0x00000008010078f7 in pthread_create () from /usr/lib/libpthread.so.1
>#3 0x0000000000488079 in ?? ()
>#4 0x0000000000488655 in ?? ()
>#5 0x0000000000487cfb in ?? ()
>#6 0x0000000000404e8e in ?? ()
>#7 0x00000008008ad000 in ?? ()
>#8 0x0000000000000000 in ?? ()
>#9 0x0000000000000000 in ?? ()
>#10 0x0000000000000007 in ?? ()
>#11 0x00007fffffffec48 in ?? ()
>#12 0x00007fffffffec62 in ?? ()
>#13 0x00007fffffffec77 in ?? ()
>#14 0x00007fffffffec8f in ?? ()
>#15 0x00007fffffffec9c in ?? ()
>#16 0x00007fffffffecce in ?? ()
>#17 0x00007fffffffecda in ?? ()
>#18 0x0000000000000000 in ?? ()
>#19 0x00007fffffffecf3 in ?? ()
>#20 0x00007fffffffed01 in ?? ()
>#21 0x00007fffffffed12 in ?? ()
>#22 0x00007fffffffed32 in ?? ()
>#23 0x00007fffffffed3c in ?? ()
>#24 0x00007fffffffed50 in ?? ()
>#25 0x00007fffffffed58 in ?? ()
>#26 0x00007fffffffed67 in ?? ()
>#27 0x00007fffffffed77 in ?? ()
>#28 0x00007fffffffed87 in ?? ()
>#29 0x00007fffffffed9a in ?? ()
>#30 0x00007fffffffeda5 in ?? ()
>#31 0x00007fffffffedd7 in ?? ()
>#32 0x00007fffffffede2 in ?? ()
>#33 0x00007fffffffedf0 in ?? ()
>#34 0x00007fffffffedfd in ?? ()
>#35 0x00007fffffffee08 in ?? ()
>#36 0x00007fffffffee2d in ?? ()
>#37 0x00007fffffffee39 in ?? ()
>#38 0x00007fffffffee4a in ?? ()
>#39 0x00007fffffffeeb5 in ?? ()
>#40 0x00007fffffffeecc in ?? ()
>#41 0x00007fffffffeee3 in ?? ()
>#42 0x00007fffffffeefd in ?? ()
>#43 0x00007fffffffef0d in ?? ()
>#44 0x00007fffffffef1c in ?? ()
>#45 0x00007fffffffef2b in ?? ()
>#46 0x00007fffffffef50 in ?? ()
>#47 0x00007fffffffef62 in ?? ()
>#48 0x00007fffffffef95 in ?? ()
>#49 0x00007fffffffefaa in ?? ()
>#50 0x0000000000000000 in ?? ()
>#51 0x0000000000000003 in ?? ()
>#52 0x0000000000400040 in ?? ()
>#53 0x0000000000000004 in ?? ()
>#54 0x0000000000000038 in ?? ()
>#55 0x0000000000000005 in ?? ()
>#56 0x0000000000000005 in ?? ()
>#57 0x0000000000000006 in ?? ()
>#58 0x0000000000001000 in ?? ()
>#59 0x0000000000000008 in ?? ()
>#60 0x0000000000000000 in ?? ()
>#61 0x0000000000000009 in ?? ()
>#62 0x0000000000404e10 in ?? ()
>#63 0x0000000000000007 in ?? ()
>#64 0x000000080088a000 in ?? ()
>#65 0x0000000000000000 in ?? ()
>#66 0x0000000000000000 in ?? ()
>#67 0x0000000000000000 in ?? ()
>#68 0x0000000000000000 in ?? ()
>#69 0x0000000000000000 in ?? ()
>#70 0x0000000000000000 in ?? ()
>#71 0x0000000000000000 in ?? ()
>#72 0x0000000000000000 in ?? ()
>#73 0x0000000000000000 in ?? ()
>#74 0x0000000000000000 in ?? ()
>#75 0x0000000000000000 in ?? ()
>#76 0x0000000000000000 in ?? ()
>#77 0x0000000000000000 in ?? ()
>#78 0x0000000000000000 in ?? ()
>#79 0x0000000000000000 in ?? ()
>#80 0x0000000000000000 in ?? ()
>#81 0x636f6c2f7273752f in ?? ()
>#82 0x786562696c2f6c61 in ?? ()
>#83 0x6c7173796d2f6365 in ?? ()
>#84 0x657361622d2d0064 in ?? ()
>#85 0x7273752f3d726964 in ?? ()
>#86 0x2d006c61636f6c2f in ?? ()
>#87 0x726964617461642d in ?? ()
>#88 0x62642f7261762f3d in ?? ()
>#89 0x2d006c7173796d2f in ?? ()
>#90 0x796d3d726573752d in ?? ()
>#91 0x69702d2d006c7173 in ?? ()
>#92 0x2f3d656c69662d64 in ?? ()
>#93 0x6d2f62642f726176 in ?? ()
>#94 0x7777772f6c717379 in ?? ()
>#95 0x612d796e6e65702e in ?? ()
>#96 0x6f632e6564616372 in ?? ()
>#97 0x2d2d006469702e6d in ?? ()
>#98 0x3033333d74726f70 in ?? ()
>#99 0x6b636f732d2d0036 in ?? ()
>#100 0x2f706d742f3d7465 in ?? ()
>#101 0x6f732e6c7173796d in ?? ()
>#102 0x5f4f445553006b63 in ?? ()
>#103 0x313030313d444947 in ?? ()
>#104 0x5059544843414d00 in ?? ()
>#105 0x776f6e6b6e753d45 in ?? ()
>#106 0x4c435f485353006e in ?? ()
>#107 0x2e34363d544e4549 in ?? ()
>#108 0x33372e39362e3138 in ?? ()
>#109 0x3220303739343520 in ?? ()
>#110 0x723d524553550032 in ?? ()
>#111 0x4c49414d00746f6f in ?? ()
>#112 0x616d2f7261762f3d in ?? ()
>#113 0x006e6165732f6c69 in ?? ()
>#114 0x00333d4c564c4853 in ?? ()
>#115 0x753d524f444e4556 in ?? ()
>#116 0x48006e776f6e6b6e in ?? ()
>#117 0x6d6f682f3d454d4f in ?? ()
>#118 0x50006e6165732f65 in ?? ()
>#119 0x5341424154414447 in ?? ()
>#120 0x5300747365743d45 in ?? ()
>#121 0x2f3d5954545f4853 in ?? ()
>#122 0x707974742f766564 in ?? ()
>#123 0x3d52454741500031 in ?? ()
>#124 0x414d51007373656c in ?? ()
>#125 0x2f3d43455053454b in ?? ()
>#126 0x61636f6c2f727375 in ?? ()
>#127 0x2f65726168732f6c in ?? ()
>#128 0x6570736b6d2f7471 in ?? ()
>#129 0x62656572662f7363 in ?? ()
>#130 0x47002b2b672d6473 in ?? ()
>#131 0x6165733d50554f52 in ?? ()
>#132 0x555f4f445553006e in ?? ()
>#133 0x00313030313d4449 in ?? ()
>#134 0x3d454d414e474f4c in ?? ()
>#135 0x53564300746f6f72 in ?? ()
>#136 0x00323d4b53414d55 in ?? ()
>#137 0x454d4f485f575757 in ?? ()
>#138 0x2f2f3a707474683d in ?? ()
>#139 0x6968632e6e616573 in ?? ()
>#140 0x2e6e65646e657474 in ?? ()
>#141 0x4f4c42002f67726f in ?? ()
>#142 0x4b3d455a49534b43 in ?? ()
>#143 0x74783d4d52455400 in ?? ()
>#144 0x6f6c6f632d6d7265 in ?? ()
>#145 0x2f3d485441500072 in ?? ()
>#146 0x69622f3a6e696273 in ?? ()
>#147 0x732f7273752f3a6e in ?? ()
>#148 0x7273752f3a6e6962 in ?? ()
>#149 0x73752f3a6e69622f in ?? ()
>#150 0x3a73656d61672f72 in ?? ()
>#151 0x636f6c2f7273752f in ?? ()
>#152 0x3a6e6962732f6c61 in ?? ()
>#153 0x636f6c2f7273752f in ?? ()
>#154 0x2f3a6e69622f6c61 in ?? ()
>#155 0x523131582f727375 in ?? ()
>#156 0x682f3a6e69622f36 in ?? ()
>#157 0x6e6165732f656d6f in ?? ()
>#158 0x4d4552006e69622f in ?? ()
>#159 0x3d54534f4845544f in ?? ()
>#160 0x39362e31382e3436 in ?? ()
>#161 0x4f4455530033372e in ?? ()
>#162 0x444e414d4d4f435f in ?? ()
>#163 0x63742f6e69622f3d in ?? ()
>#164 0x3d54534f48006873 in ?? ()
>#165 0x6e6e65702e777777 in ?? ()
>#166 0x6564616372612d79 in ?? ()
>#167 0x454853006d6f632e in ?? ()
>#168 0x2f6e69622f3d4c4c in ?? ()
>#169 0x4455530068736374 in ?? ()
>#170 0x733d524553555f4f in ?? ()
>#171 0x5954534f006e6165 in ?? ()
>#172 0x42656572463d4550 in ?? ()
>#173 0x4f52535643004453 in ?? ()
>#174 0x2f7777772f3d544f in ?? ()
>#175 0x6f6374736f637461 in ?? ()
>#176 0x632f6d6f632e6f6c in ?? ()
>#177 0x002f746f6f727376 in ?? ()
>#178 0x7261762f3d445750 in ?? ()
>#179 0x7173796d2f62642f in ?? ()
>#180 0x4f435f485353006c in ?? ()
>#181 0x4e4f495443454e4e in ?? ()
>#182 0x362e31382e34363d in ?? ()
>#183 0x3934352033372e39 in ?? ()
>#184 0x31312e3833203037 in ?? ()
>#185 0x33312e3832322e33 in ?? ()
>#186 0x5054460032322030 in ?? ()
>#187 0x455649535341505f in ?? ()
>#188 0x45593d45444f4d5f in ?? ()
>#189 0x595454534f480053 in ?? ()
>#190 0x42656572463d4550 in ?? ()
>#191 0x0000000000004453 in ?? ()
>#192 0x0000000000000000 in ?? ()
>#193 0x0000000000000000 in ?? ()
>#194 0x0000000000000000 in ?? ()
>#195 0x0000000000000000 in ?? ()
>#196 0x0000000000000000 in ?? ()
>#197 0x0000000000000000 in ?? ()
>#198 0x0000000000000000 in ?? ()
>#199 0x0000000000000000 in ?? ()
>#200 0x0000000000000000 in ?? ()
>#201 0x0000000000000000 in ?? ()
>#202 0x0000000000000000 in ?? ()
>#203 0x0000000000000000 in ?? ()
>#204 0x0000000000000000 in ?? ()
>#205 0x0000000000000000 in ?? ()
>#206 0x0000000000000000 in ?? ()
>#207 0x0000000000000000 in ?? ()
>#208 0x0000000000000000 in ?? ()
>#209 0x0000000000000000 in ?? ()
>#210 0x0000000000000000 in ?? ()
>#211 0x0000000000000000 in ?? ()
>#212 0x0000000000000000 in ?? ()
>#213 0x0000000000000000 in ?? ()
>#214 0x0000000000000000 in ?? ()
>#215 0x0000000000000000 in ?? ()
>#216 0x0000000000000000 in ?? ()
>#217 0x0000000000000000 in ?? ()
>#218 0x0000000000000000 in ?? ()
>#219 0x0000000000000000 in ?? ()
>#220 0x0000000000000000 in ?? ()
>#221 0x0000000000000000 in ?? ()
>#222 0x0000000000000000 in ?? ()
>#223 0x0000000000000000 in ?? ()
>#224 0x0000000000000000 in ?? ()
>#225 0x0000000000000000 in ?? ()
>#226 0x0000000000000000 in ?? ()
>#227 0x0000000000000000 in ?? ()
>#228 0x0000000000000000 in ?? ()
>#229 0x0000000000000000 in ?? ()
>#230 0x0000000000000000 in ?? ()
>#231 0x0000000000000000 in ?? ()
>#232 0x0000000000000000 in ?? ()
>#233 0x0000000000000000 in ?? ()
>#234 0x0000000000000000 in ?? ()
>#235 0x0000000000000000 in ?? ()
>#236 0x0000000000000000 in ?? ()
>#237 0x0000000000000000 in ?? ()
>#238 0x0000000000000000 in ?? ()
>#239 0x0000000000000000 in ?? ()
>#240 0x0000000000000000 in ?? ()
>#241 0x0000000000000000 in ?? ()
>#242 0x0000000000000000 in ?? ()
>#243 0x0000000000000000 in ?? ()
>#244 0x0000000000000000 in ?? ()
>#245 0x0000000000000000 in ?? ()
>#246 0x0000000000000000 in ?? ()
>#247 0x0000000000000000 in ?? ()
>#248 0x0000000000000000 in ?? ()
>#249 0x0000000000000000 in ?? ()
>#250 0x0000000000000000 in ?? ()
>#251 0x0000000000000000 in ?? ()
>#252 0x0000000000000000 in ?? ()
>#253 0x0000000000000000 in ?? ()
>#254 0x0000000000000000 in ?? ()
>#255 0x0000000000000000 in ?? ()
>#256 0x0000000000000000 in ?? ()
>#257 0x0000000000000000 in ?? ()
>#258 0x0000000000000000 in ?? ()
>#259 0x0000000000000000 in ?? ()
>#260 0x0000000000000000 in ?? ()
>#261 0x0000000000000000 in ?? ()
>#262 0x0000000000000000 in ?? ()
>#263 0x0000000000000000 in ?? ()
>#264 0x0000000000000000 in ?? ()
>#265 0x0000000000000000 in ?? ()
>#266 0x0000000000000000 in ?? ()
>#267 0x0000000000000000 in ?? ()
>#268 0x0000000000000000 in ?? ()
>#269 0x0000000000000000 in ?? ()
>#270 0x0000000000000000 in ?? ()
>#271 0x0000000000000000 in ?? ()
>#272 0x0000000000000000 in ?? ()
>#273 0x0000000000000000 in ?? ()
>#274 0x0000000000000000 in ?? ()
>#275 0x0000000000000000 in ?? ()
>#276 0x0000000000000000 in ?? ()
>#277 0x0000000000000000 in ?? ()
>#278 0x0000000000000000 in ?? ()
>#279 0x0000000000000000 in ?? ()
>#280 0x0000000000000000 in ?? ()
>#281 0x0000000000000000 in ?? ()
>#282 0x0000000000000000 in ?? ()
>#283 0x0000000000000000 in ?? ()
>#284 0x0000000000000000 in ?? ()
>#285 0x0000000000000000 in ?? ()
>#286 0x0000000000000000 in ?? ()
>#287 0x0000000000000000 in ?? ()
>#288 0x0000000000000000 in ?? ()
>#289 0x0000000000000000 in ?? ()
>#290 0x0000000000000000 in ?? ()
>#291 0x0000000000000000 in ?? ()
>#292 0x0000000000000000 in ?? ()
>#293 0x0000000000000000 in ?? ()
>#294 0x0000000000000000 in ?? ()
>#295 0x0000000000000000 in ?? ()
>#296 0x0000000000000000 in ?? ()
>#297 0x0000000000000000 in ?? ()
>#298 0x0000000000000000 in ?? ()
>#299 0x0000000000000000 in ?? ()
>#300 0x0000000000000000 in ?? ()
>#301 0x0000000000000000 in ?? ()
>#302 0x0000000000000000 in ?? ()
>#303 0x0000000000000000 in ?? ()
>#304 0x0000000000000000 in ?? ()
>#305 0x0000000000000000 in ?? ()
>#306 0x0000000000000000 in ?? ()
>#307 0x0000000000000000 in ?? ()
>#308 0x0000000000000000 in ?? ()
>#309 0x0000000000000000 in ?? ()
>#310 0x0000000000000000 in ?? ()
>#311 0x0000000000000000 in ?? ()
>#312 0x0000000000000000 in ?? ()
>#313 0x0000000000000000 in ?? ()
>#314 0x0000000000000000 in ?? ()
>#315 0x0000000000000000 in ?? ()
>#316 0x0000000000000000 in ?? ()
>#317 0x0000000000000000 in ?? ()
>#318 0x0000000000000000 in ?? ()
>#319 0x0000000000000000 in ?? ()
>#320 0x0000000000000000 in ?? ()
>#321 0x0000000000000000 in ?? ()
>#322 0x0000000000000000 in ?? ()
>#323 0x0000000000000000 in ?? ()
>#324 0x0000000000000000 in ?? ()
>#325 0x0000000000000000 in ?? ()
>#326 0x0000000000000000 in ?? ()
>#327 0x0000000000000000 in ?? ()
>#328 0x0000000000000000 in ?? ()
>#329 0x0000000000000000 in ?? ()
>#330 0x0000000000000000 in ?? ()
>#331 0x0000000000000000 in ?? ()
>#332 0x0000000000000000 in ?? ()
>#333 0x0000000000000000 in ?? ()
>#334 0x0000000000000000 in ?? ()
>#335 0x0000000000000000 in ?? ()
>#336 0x0000000000000000 in ?? ()
>#337 0x0000000000000000 in ?? ()
>#338 0x0000000000000000 in ?? ()
>#339 0x0000000000000000 in ?? ()
>#340 0x0000000000000000 in ?? ()
>#341 0x0000000000000000 in ?? ()
>#342 0x0000000000000000 in ?? ()
>#343 0x0000000000000000 in ?? ()
>#344 0x0000000000000000 in ?? ()
>#345 0x0000000000000000 in ?? ()
>#346 0x0000000000000000 in ?? ()
>#347 0x0000000000000000 in ?? ()
>#348 0x0000000000000000 in ?? ()
>#349 0x0000000000000000 in ?? ()
>#350 0x0000000000000000 in ?? ()
>#351 0x0000000000000000 in ?? ()
>#352 0x0000000000000000 in ?? ()
>#353 0x0000000000000000 in ?? ()
>#354 0x0000000000000000 in ?? ()
>#355 0x0000000000000000 in ?? ()
>#356 0x0000000000000000 in ?? ()
>#357 0x0000000000000000 in ?? ()
>#358 0x0000000000000000 in ?? ()
>#359 0x0000000000000000 in ?? ()
>#360 0x0000000000000000 in ?? ()
>#361 0x0000000000000000 in ?? ()
>#362 0x0000000000000000 in ?? ()
>#363 0x0000000000000000 in ?? ()
>#364 0x0000000000000000 in ?? ()
>#365 0x0000000000000000 in ?? ()
>#366 0x0000000000000000 in ?? ()
>#367 0x0000000000000000 in ?? ()
>#368 0x0000000000000000 in ?? ()
>#369 0x0000000000000000 in ?? ()
>#370 0x0000000000000000 in ?? ()
>#371 0x0000000000000000 in ?? ()
>#372 0x0000000000000000 in ?? ()
>#373 0x0000000000000000 in ?? ()
>#374 0x0000000000000000 in ?? ()
>#375 0x0000000000000000 in ?? ()
>#376 0x0000000000000000 in ?? ()
>#377 0x0000000000000000 in ?? ()
>#378 0x0000000000000000 in ?? ()
>#379 0x0000000000000000 in ?? ()
>#380 0x0000000000000000 in ?? ()
>#381 0x0000000000000000 in ?? ()
>#382 0x0000000000000000 in ?? ()
>#383 0x0000000000000000 in ?? ()
>#384 0x0000000000000000 in ?? ()
>#385 0x0000000000000000 in ?? ()
>#386 0x0000000000000000 in ?? ()
>#387 0x0000000000000000 in ?? ()
>#388 0x0000000000000000 in ?? ()
>#389 0x0000000000000000 in ?? ()
>#390 0x0000000000000000 in ?? ()
>#391 0x0000000000000000 in ?? ()
>#392 0x0000000000000000 in ?? ()
>#393 0x0000000000000000 in ?? ()
>#394 0x0000000000000000 in ?? ()
>#395 0x0000000000000000 in ?? ()
>#396 0x0000000000000000 in ?? ()
>#397 0x0000000000000000 in ?? ()
>#398 0x0000000000000000 in ?? ()
>#399 0x0000000000000000 in ?? ()
>#400 0x0000000000000000 in ?? ()
>#401 0x0000000000000000 in ?? ()
>#402 0x0000000000000000 in ?? ()
>#403 0x0000000000000000 in ?? ()
>#404 0x0000000000000000 in ?? ()
>#405 0x0000000000000000 in ?? ()
>#406 0x0000000000000000 in ?? ()
>#407 0x0000000000000000 in ?? ()
>#408 0x0000000000000000 in ?? ()
>#409 0x0000000000000000 in ?? ()
>#410 0x0000000000000000 in ?? ()
>#411 0x0000000000000000 in ?? ()
>#412 0x0000000000000000 in ?? ()
>#413 0x0000000000000000 in ?? ()
>#414 0x0000000000000000 in ?? ()
>#415 0x0000000000000000 in ?? ()
>#416 0x0000000000000000 in ?? ()
>#417 0x0000000000000000 in ?? ()
>#418 0x0000000000000000 in ?? ()
>#419 0x0000000000000000 in ?? ()
>#420 0x0000000000000000 in ?? ()
>#421 0x0000000000000000 in ?? ()
>#422 0x0000000000000000 in ?? ()
>#423 0x0000000000000000 in ?? ()
>#424 0x0000000000000000 in ?? ()
>#425 0x0000000000000000 in ?? ()
>#426 0x0000000000000000 in ?? ()
>#427 0x0000000000000000 in ?? ()
>#428 0x0000000000000000 in ?? ()
>#429 0x0000000000000000 in ?? ()
>#430 0x0000000000000000 in ?? ()
>#431 0x0000000000000000 in ?? ()
>#432 0x0000000000000000 in ?? ()
>#433 0x0000000000000000 in ?? ()
>#434 0x0000000000000000 in ?? ()
>#435 0x0000000000000000 in ?? ()
>#436 0x0000000000000000 in ?? ()
>#437 0x0000000000000000 in ?? ()
>#438 0x0000000000000000 in ?? ()
>#439 0x0000000000000000 in ?? ()
>#440 0x0000000000000000 in ?? ()
>#441 0x0000000000000000 in ?? ()
>#442 0x0000000000000000 in ?? ()
>#443 0x0000000000000000 in ?? ()
>#444 0x0000000000000000 in ?? ()
>#445 0x0000000000000000 in ?? ()
>#446 0x0000000000000000 in ?? ()
>#447 0x0000000000000000 in ?? ()
>#448 0x0000000000000000 in ?? ()
>#449 0x0000000000000000 in ?? ()
>#450 0x0000000000000000 in ?? ()
>#451 0x0000000000000000 in ?? ()
>#452 0x0000000000000000 in ?? ()
>#453 0x0000000000000000 in ?? ()
>#454 0x0000000000000000 in ?? ()
>#455 0x0000000000000000 in ?? ()
>#456 0x0000000000000000 in ?? ()
>#457 0x0000000000000000 in ?? ()
>#458 0x0000000000000000 in ?? ()
>#459 0x0000000000000000 in ?? ()
>#460 0x0000000000000000 in ?? ()
>#461 0x0000000000000000 in ?? ()
>#462 0x0000000000000000 in ?? ()
>#463 0x0000000000000000 in ?? ()
>#464 0x0000000000000000 in ?? ()
>#465 0x0000000000000000 in ?? ()
>#466 0x0000000000000000 in ?? ()
>#467 0x0000000000000000 in ?? ()
>#468 0x0000000000000000 in ?? ()
>#469 0x0000000000000000 in ?? ()
>#470 0x0000000000000000 in ?? ()
>#471 0x0000000000000000 in ?? ()
>#472 0x0000000000000000 in ?? ()
>#473 0x0000000000000000 in ?? ()
>#474 0x0000000000000000 in ?? ()
>#475 0x0000000000000000 in ?? ()
>#476 0x0000000000000000 in ?? ()
>#477 0x0000000000000000 in ?? ()
>#478 0x0000000000000000 in ?? ()
>#479 0x0000000000000000 in ?? ()
>#480 0x0000000000000000 in ?? ()
>#481 0x0000000000000000 in ?? ()
>#482 0x0000000000000000 in ?? ()
>#483 0x0000000000000000 in ?? ()
>#484 0x0000000000000000 in ?? ()
>#485 0x0000000000000000 in ?? ()
>#486 0x0000000000000000 in ?? ()
>#487 0x0000000000000000 in ?? ()
>#488 0x0000000000000000 in ?? ()
>#489 0x0000000000000000 in ?? ()
>#490 0x0000000000000000 in ?? ()
>#491 0x0000000000000000 in ?? ()
>#492 0x0000000000000000 in ?? ()
>#493 0x0000000000000000 in ?? ()
>#494 0x0000000000000000 in ?? ()
>#495 0x0000000000000000 in ?? ()
>#496 0x0000000000000000 in ?? ()
>#497 0x0000000000000000 in ?? ()
>#498 0x0000000000000000 in ?? ()
>#499 0x0000000000000000 in ?? ()
>#500 0x0000000000000000 in ?? ()
>#501 0x0000000000000000 in ?? ()
>#502 0x0000000000000000 in ?? ()
>#503 0x0000000000000000 in ?? ()
>#504 0x0000000000000000 in ?? ()
>#505 0x0000000000000000 in ?? ()
>#506 0x0000000000000000 in ?? ()
>#507 0x0000000000000000 in ?? ()
>#508 0x0000000000000000 in ?? ()
>#509 0x0000000000000000 in ?? ()
>#510 0x0000000000000000 in ?? ()
>#511 0x0000000000000000 in ?? ()
>#512 0x0000000000000000 in ?? ()
>#513 0x0000000000000000 in ?? ()
>#514 0x0000000000000000 in ?? ()
>
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?420C1863.1010609>