Date: Sat, 15 Apr 2000 22:15:45 -0400 From: "Jerry Bell" <jerry@bellnetworks.net> To: <questions@FreeBSD.ORG> Cc: <stable@FreeBSD.ORG> Subject: ps, top, et. al problems with 4.0 stable Message-ID: <008c01bfa749$aca13340$f7bbb1d0@netrex.com>
next in thread | raw e-mail | index | archive | help
This is somewhat long - I apologize in advance.
db# top
kvm_open: proc size mismatch (28188 total, 1056 chunks)
top: Out of memory.
db# ps -x
ps: proc size mismatch (24012 total, 1056 chunks)
db#
Summary of my findings (detail follows):
Starting with a fresh 4.0 RELEASE installation. cvsup 4.0 stable source.
make world, make and install new kernel.
The 4.0 stable copy of libkvm appears to not work on my system (built today
from source supped today). The 4.0 release copy of libkvm does work.
The 4.0 stable copy of top does not work on my system (built today from
source supped today). The 4.0 release copy of top does work.
The 4.0 stable copy of ps does not work on my system (built today from
source supped today). The 4.0 release copy of ps does work.
The 4.0 stable copy of iostat does not work on my system (built today from
source supped today). The 4.0 release copy of iostat also does not work.
This *is not* due to an inconsistancy between the binaries and the kernel.
(Please don't email me telling me to build a new kernel)
I have rebuilt and installed libkvm. I still get the same problem.
I've done this about 10 times on 2 different computers with the exact same
results.
I have run mergemaster - it did not help.
I did copy the new MAKEDEV in and remade all devices.
I have tried copying /dev/null on top of utmp and wtmp.
I have tried removing procfs from the kernel (same problem)
I have tried removing procfs from fstab, reboot, add it back, reboot (same
problem).
I am REALLY REALLY sure that I am running on the new kernel.
I am REALLY REALLY sure that I am running the new binaries.
I am sorry if I'm coming off as rude. I posted an email a few weeks ago
about the same problem and got about 50 responses telling me to build a new
kernel and to make sure I wasn't running an old copy of ps. (the old copy
works, anyway)
I really need some help getting this straightened out. Any ideas?
Here's the situation: I installed 4.0 release on a box. It works
wonderfully. I cvsup the source with the following supfile (supped
4/15/2000):
*default host=cvsup4.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4
*default delete use-rel-suffix
src-all
Now, I do the following:
cd /usr/src
make world -DNOGAMES -DNOAOUT -DNOPROFILE -DNOINFO
<this runs and finishes fine>
cd /usr/src/sys/i386/conf
config GENERIC
cd ../../compile/GENERIC
make depend
make
make install
<this runs and finished without a problem>
reboot
After it boots, running on the new kernel, most everything is good, except
ps, top and iostat.
Here is an example of what I am seeing:
db# top
kvm_open: proc size mismatch (28188 total, 1056 chunks)
top: Out of memory.
db# ps -x
ps: proc size mismatch (24012 total, 1056 chunks)
db#
iostat 1 1000
tty da0 da0 cpu
tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id
0 323 0.00 0 0.00 0.00 0 0.00 0 0 0 0100
0 60 0.00 0 0.00 0.00 0 0.00 0 0 0 0100
351121 0.00 0 0.00 0.00 0 0.00 3 0 6 1 90
074525 0.00 4253280890 0.00 0.00 4253280890 0.00 4 0 5 2 90
046553 0.00 0 0.00 0.00 0 0.00 2 0 8 2 88
129533 0.00 18263922208111388672 0.00 0.00 18263922208111388672 0.00
5 0 9 0 86
0 93 3.81 901459873129 3355227269.53 3.81 901459873129 3355227269.53
0 0 1 0 99
0 93 0.00 0 3255805582.43 0.00 0 3255805582.43 0 0 0 0100
0 75 0.00 0 431896853.51 0.00 0 431896853.51 0 0 0 0100
0 73 0.00 0 0.00 0.00 0 0.00 0 0 0 0100
0 60 0.00 0 0.00 0.00 0 0.00 0 0 0 0100
0 60 0.00 0 0.00 0.00 0 0.00 0 0 0 0100
^C
db#
<note the 2 occurances of da0 in the list of disks in the iostat output>
Here is the kernel I'm running:
db# ls -l /
total 7514
-rw-r--r-- 2 root wheel 574 Mar 20 21:32 .cshrc
-rw-r--r-- 2 root wheel 251 Mar 20 21:32 .profile
-r--r--r-- 1 root wheel 4735 Mar 20 21:32 COPYRIGHT
drwxr-xr-x 2 root wheel 1024 Apr 16 00:13 bin
drwxr-xr-x 3 root wheel 512 Apr 9 01:18 boot
lrwxr-xr-x 1 root wheel 11 Apr 9 01:19 compat -> /usr/compat
drwxr-xr-x 3 root wheel 12288 Apr 16 00:51 dev
drwxr-xr-x 15 root wheel 2048 Apr 16 01:11 etc
lrwxr-xr-x 1 root wheel 9 Apr 9 01:32 home -> /usr/home
-r-xr-xr-x 1 root wheel 2574609 Apr 16 00:48 kernel
-rwxr-xr-x 1 root wheel 2777025 Mar 20 22:50 kernel.GENERIC
-rwxr-xr-x 1 root wheel 2274463 Apr 8 08:16 kernel.old
drwxr-xr-x 2 root wheel 512 Mar 20 21:26 mnt
drwxr-xr-x 2 root wheel 2048 Apr 9 00:46 modules
dr-xr-xr-x 1 root wheel 512 Apr 16 02:04 proc
drwxr-xr-x 2 root wheel 512 Apr 9 01:33 root
drwxr-xr-x 2 root wheel 2048 Apr 16 00:17 sbin
drwxr-xr-x 4 root wheel 512 Apr 9 00:42 stand
lrwxr-xr-x 1 root wheel 11 Apr 16 00:11 sys -> usr/src/sys
drwxrwxrwt 2 root wheel 512 Apr 16 00:52 tmp
drwxr-xr-x 18 root wheel 512 Apr 16 00:11 usr
drwxr-xr-x 19 root wheel 512 Apr 15 15:55 var
db# uname -a
FreeBSD db.bellnetworks.net 4.0-STABLE FreeBSD 4.0-STABLE #0: Sun Apr 16
00:47:48 GMT 2000 jbell@db.bellnetworks.net:/usr/src/sys/compile/GENERIC
i386
db# ls -l /bin/ps /usr/sbin/iostat /usr/bin/top
-r-xr-xr-x 1 root wheel 222648 Apr 16 00:13 /bin/ps
-r-xr-sr-x 1 root kmem 34240 Apr 16 02:08 /usr/bin/top
-r-xr-sr-x 1 root kmem 10084 Apr 16 02:10 /usr/sbin/iostat
All of these programs functioned normally under 4.0 release on this system.
So, after some research, I find that libkvm is probably the culprit behind
top not working. Several threads in the mail archives suggest building and
installing libkvm. So I go to the libkvm directory under /usr/src, run make
clean, make all, make install. Same thing.
Ahhh, but since I've now rebuilt this computer approximately 10000000 times
trying to solve this problem, I know to keep a backup copy of all the 4.0
release binaries :) So, I copy the 4.0 RELEASE (yes 4.0 release) version of
/usr/lib/libkvm* back in. Guess what? top works! But the output looks
like this:
last pid: 528; load averages: 0.00, 0.00, 0.00
up 0+01:22:09 02:13:14
4 processes: 1 starting, 1 running
CPU states: % user, % nice, % system, % interrupt, %
idle
Mem: 5908K Active, 28M Inact, 12M Wired, 52K Cache, 29M Buf, 141M Free
Swap: 400M Total, 400M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
0 news -22 52481 0K 0K ? 20 57:18 1429.93% 1429.93% <>
0 daemon -22 52489 3279M 513G ? -32 0:00 0.00% 138.23% <>
528 root 30 0 0K 32K RUN 0:00 1.00% 0.05% top
0 root -22 -52 0K 0K START 0:00 0.00% 157443857.81% <>
If I run the 4.0 release top binary with the 4.0 release libkvm's, top works
properly.
Unfortunately, ps and iostat (and who knows what else) still doesn't work -
same problem.
Ok, so now I focus on ps. This is the message I get:
db# ps -x
ps: proc size mismatch (26100 total, 1056 chunks)
db#
Ok, so lets to a bit of research:
db# truss /bin/ps
syscall ioctl(1,TIOCGWINSZ,0xbfbfefe0)
returns 0 (0x0)
syscall readlink("/etc/malloc.conf",0xbfbfeeb8,63)
errno 2 'No such file or directory'
syscall mmap(0x0,4096,0x3,0x1002,-1,0x0)
returns 671588352 (0x2807a000)
syscall break(0x8087000)
returns 0 (0x0)
syscall break(0x8088000)
returns 0 (0x0)
syscall open("/dev/null",0,00)
returns 3 (0x3)
syscall fstat(3,0xbfbfeeb8)
returns 0 (0x0)
syscall open("/dev/null",0,00)
returns 4 (0x4)
syscall break(0x8089000)
returns 0 (0x0)
syscall getuid()
returns 0 (0x0)
syscall __sysctl(0xbfbfef58,0x4,0x0,0xbfbfef24,0x0,0x0)
returns 0 (0x0)
syscall break(0x8092000)
returns 0 (0x0)
syscall __sysctl(0xbfbfef58,0x4,0x8089000,0xbfbfef24,0x0,0x0)
returns 0 (0x0)
ps: syscall write(2,0xbfbfe8b8,4)
returns 4 (0x4)
proc size mismatch (27144 total, 1056 chunks)syscall write(2,0xbfbfe8d8,45)
returns 45 (0x2d)
syscall write(2,0xbfbfe8b8,1)
returns 1 (0x1)
syscall exit(0x1)
process exit, rval = 256
db#
Nothing looks out of the ordinary. But wait. I still have a copy of the
4.0 release copy of ps. lets try it:
db# /var/build/tmp/bin/ps -x
PID TT STAT TIME COMMAND
0 ?? DLs 0:00.00 (swapper)
1 ?? ILs 0:00.02 /sbin/init --
2 ?? DL 0:00.00 (pagedaemon)
3 ?? DL 0:00.00 (vmdaemon)
4 ?? DL 0:00.00 (bufdaemon)
5 ?? DL 0:00.15 (syncer)
73 ?? Is 0:00.05 syslogd
94 ?? Is 0:00.03 inetd -wW
96 ?? Is 0:00.02 cron
99 ?? Is 0:00.02 sendmail: accepting connections on port 25
(sendmail)
186 ?? Ss 0:00.24 telnetd
245 ?? Is 0:00.41 telnetd
224 p0 S 0:00.08 _su (csh)
286 p0 R+ 0:00.00 /var/build/tmp/bin/ps -x
248 p1 I+ 0:00.06 _su (csh)
219 v0 Is+ 0:00.01 /usr/libexec/getty Pc ttyv0
156 v1 Is 0:00.05 -csh (csh)
216 v1 I+ 0:00.04 _su (csh)
157 v2 Is+ 0:00.01 /usr/libexec/getty Pc ttyv2
158 v3 Is+ 0:00.01 /usr/libexec/getty Pc ttyv3
159 v4 Is+ 0:00.01 /usr/libexec/getty Pc ttyv4
160 v5 Is+ 0:00.01 /usr/libexec/getty Pc ttyv5
161 v6 Is+ 0:00.01 /usr/libexec/getty Pc ttyv6
162 v7 Is+ 0:00.01 /usr/libexec/getty Pc ttyv7
134 con- I 0:00.01 /bin/sh /usr/local/bin/safe_mysqld --user=mysql
db#
Woah! pretty interesting. The old copy of ps works. So, lets look at this:
truss /var/build/tmp/bin/ps -x
syscall ioctl(1,TIOCGWINSZ,0xbfbfefdc)
returns 0 (0x0)
syscall readlink("/etc/malloc.conf",0xbfbfeed4,63)
errno 2 'No such file or directory'
syscall mmap(0x0,4096,0x3,0x1002,-1,0x0)
returns 671567872 (0x28075000)
syscall break(0x8083000)
returns 0 (0x0)
syscall break(0x8084000)
returns 0 (0x0)
syscall break(0x8085000)
returns 0 (0x0)
syscall open("/dev/null",0,00)
returns 3 (0x3)
syscall fstat(3,0xbfbfeec4)
returns 0 (0x0)
syscall open("/dev/null",0,00)
returns 4 (0x4)
syscall getuid()
returns 0 (0x0)
syscall __sysctl(0xbfbfef64,0x4,0x0,0xbfbfef30,0x0,0x0)
returns 0 (0x0)
syscall break(0x808e000)
returns 0 (0x0)
syscall __sysctl(0xbfbfef64,0x4,0x8085000,0xbfbfef30,0x0,0x0)
returns 0 (0x0)
syscall break(0x8090000)
returns 0 (0x0)
syscall __sysctl(0xbfbfee88,0x2,0xbfbfee90,0xbfbfee84,0x80712e8,0x17)
returns 0 (0x0)
syscall __sysctl(0xbfbfee90,0x2,0x8077d70,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall break(0x8091000)
returns 0 (0x0)
syscall break(0x8092000)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfeeac,0x2,0xbfbfeea4,0xbfbfeea8,0x0,0x0)
returns 0 (0x0)
syscall break(0x80d3000)
returns 0 (0x0)
syscall break(0x80d4000)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall __sysctl(0xbfbfef08,0x4,0x8090000,0xbfbfef04,0x0,0x0)
returns 0 (0x0)
syscall fstat(1,0xbfbfec04)
returns 0 (0x0)
syscall break(0x80d5000)
returns 0 (0x0)
syscall ioctl(1,TIOCGETA,0xbfbfec38)
returns 0 (0x0)
PID TT STAT TIME COMMAND
syscall write(1,0x80d4000,34)
returns 34 (0x22)
0 ?? DLs 0:00.00 (swapper)
syscall write(1,0x80d4000,37)
returns 37 (0x25)
1 ?? ILs 0:00.02 /sbin/init --
syscall write(1,0x80d4000,40)
returns 40 (0x28)
2 ?? DL 0:00.00 (pagedaemon)
syscall write(1,0x80d4000,40)
returns 40 (0x28)
3 ?? DL 0:00.00 (vmdaemon)
syscall write(1,0x80d4000,38)
returns 38 (0x26)
4 ?? DL 0:00.00 (bufdaemon)
syscall write(1,0x80d4000,39)
returns 39 (0x27)
5 ?? DL 0:00.16 (syncer)
syscall write(1,0x80d4000,36)
returns 36 (0x24)
73 ?? Ss 0:00.05 syslogd
syscall write(1,0x80d4000,34)
returns 34 (0x22)
94 ?? Is 0:00.03 inetd -wW
syscall write(1,0x80d4000,36)
returns 36 (0x24)
96 ?? Is 0:00.02 cron
syscall write(1,0x80d4000,31)
returns 31 (0x1f)
99 ?? Is 0:00.02 sendmail: accepting connections on port 25
(sendmail)
syscall write(1,0x80d4000,80)
returns 80 (0x50)
186 ?? Ss 0:00.24 telnetd
syscall write(1,0x80d4000,34)
returns 34 (0x22)
245 ?? Is 0:00.41 telnetd
syscall write(1,0x80d4000,34)
returns 34 (0x22)
syscall break(0x80d6000)
returns 0 (0x0)
syscall stat("/var/run/dev.db",0xbfbfee24)
returns 0 (0x0)
syscall open("/var/run/dev.db",0,00)
returns 5 (0x5)
syscall fcntl(0x5,0x2,0x1)
returns 0 (0x0)
syscall read(0x5,0x80d5000,0x104)
returns 260 (0x104)
syscall break(0x80d8000)
returns 0 (0x0)
syscall lseek(5,0x4000,0)
returns 16384 (0x4000)
syscall read(0x5,0x80d6000,0x2000)
returns 8192 (0x2000)
224 p0 S 0:00.08 _su (csh)
syscall write(1,0x80d4000,36)
returns 36 (0x24)
290 p0 R+ 0:00.01 truss /var/build/tmp/bin/ps -x
syscall write(1,0x80d4000,57)
returns 57 (0x39)
291 p0 R+ 0:00.00 /var/build/tmp/bin/ps -x
syscall write(1,0x80d4000,51)
returns 51 (0x33)
syscall break(0x80da000)
returns 0 (0x0)
syscall lseek(5,0x2000,0)
returns 8192 (0x2000)
syscall read(0x5,0x80d8000,0x2000)
returns 8192 (0x2000)
248 p1 I+ 0:00.06 _su (csh)
syscall write(1,0x80d4000,36)
returns 36 (0x24)
219 v0 Is+ 0:00.01 /usr/libexec/getty Pc ttyv0
syscall write(1,0x80d4000,54)
returns 54 (0x36)
156 v1 Is 0:00.05 -csh (csh)
syscall write(1,0x80d4000,37)
returns 37 (0x25)
216 v1 I+ 0:00.04 _su (csh)
syscall write(1,0x80d4000,36)
returns 36 (0x24)
157 v2 Is+ 0:00.01 /usr/libexec/getty Pc ttyv2
syscall write(1,0x80d4000,54)
returns 54 (0x36)
158 v3 Is+ 0:00.01 /usr/libexec/getty Pc ttyv3
syscall write(1,0x80d4000,54)
returns 54 (0x36)
159 v4 Is+ 0:00.01 /usr/libexec/getty Pc ttyv4
syscall write(1,0x80d4000,54)
returns 54 (0x36)
160 v5 Is+ 0:00.01 /usr/libexec/getty Pc ttyv5
syscall write(1,0x80d4000,54)
returns 54 (0x36)
161 v6 Is+ 0:00.01 /usr/libexec/getty Pc ttyv6
syscall write(1,0x80d4000,54)
returns 54 (0x36)
162 v7 Is+ 0:00.01 /usr/libexec/getty Pc ttyv7
syscall write(1,0x80d4000,54)
returns 54 (0x36)
134 con- I 0:00.01 /bin/sh /usr/local/bin/safe_mysqld --user=mysql
syscall write(1,0x80d4000,74)
returns 74 (0x4a)
syscall exit(0x0)
process exit, rval = 0
db#
So, the 4.0 release copy of ps works on the 4.0 stable system. The 4.0
release copy of iostat produces the same results as the 4.0 stable (see
above).
Thanks for your time!
Jerry
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?008c01bfa749$aca13340$f7bbb1d0>