Date: Thu, 15 Feb 1996 10:49:04 -0500 (EST) From: Brian Tao <taob@io.org> To: "Miguel A.L. Paraz" <map@iphil.net> Cc: freebsd-questions@freebsd.org, FREEBSD-ISP-L <freebsd-isp@freebsd.org> Subject: 128+ users on one system? (was Re: It feels so good!) Message-ID: <Pine.BSF.3.91.960215095838.24170f-100000@zip.io.org> In-Reply-To: <199602150904.RAA09887@marikit.iphil.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 15 Feb 1996, Miguel A.L. Paraz wrote:
>
> Brian Tao wrote:
>
> > 2:05AM up 8 days, 12:13, 132 users, load averages: 2.84, 3.29, 4.05
>
> Wow! Care to share your system configuration, kernel and
> system and tuning tricks, et cetera?
There's nothing extraordinary about the hardware or the kernel.
This is an example of what FreeBSD can do "out of the box", which is
rather impressive (at least to me ;-) ).
Hardware is a 133-MHz Pentium system, ASUS P/I-P55TP4XEG
motherboard with 512K pipeline burst cache and 128M of RAM. Tip: if
you're spending this kind of money on a system, go out and buy PCI
versions of everything! Our supplier shipped the systems with a cheap
Trident PCI VGA card, an NCR53c810 SCSI controller, and an SMC
EtherPower 9332 10/100Mbps NIC. The EtherPower 8432 (I think that's
the number, it's the 10Mbps-only version) also works well.
Hardware installation was literally "plug and play". The PCI
firmware sorts out all the nastiness normally associated with ISA
peripherals. IRQ's are settable through the BIOS setup screen (the
defaults worked for me) and the 2.1.0-R install kernel correctly
identified all the pieces without prior fiddling with jumpers and
DOS-based config utilities. :)
Total cost of the system came to around C$6000, or around US$4500.
The killer is the RAM, of course (C$4200 for four 60ns 32MB SIMM's),
but the software was free. :) That works out to around US$45 per
user, if you limit yourself to 100 online at a time.
At 100+ interactive users, I've found that the bottleneck is at
the 10Mbps Ethernet interface, not the CPU or memory. With that many
users typing away at their terminals, de0 is sending and receiving 400
to 500 packets per second. Home directories are NFS mounted, which
doesn't help matters. This results in lagged keyboard response and
screen updates. Putting the machine on a 100Mbps port may eliminate
this bottleneck. Swap usage hovers around 10 to 20% and the CPU
doesn't appear to be working particularly hard with 100+ users. I'd
guess that it can handle 150 interactive users comfortably, bringing
the costs down to a mere US$30 per user.
More tips... don't forget to configure and MAKEDEV all 256 pty's!
I let my users run 'screen' here, and it was configured at compile
time to use pty[PQRS] (the upper 128 pty's) so that [pt]ty[pqrs] were
available for telnetd and rlogind. Also, be sure to have enough mbufs
around to avoid locking up the network on that machine if they ever
run out. This happened once with NMBCLUSTERS=2048, so now I have it
set to 4096. I've seen the high-water mark hit 1952 clusters since
then.
I've included the dmesg output and kernel config file below. As I
said, there isn't anything out of the ordinary with the configuration.
All the options are documented or well-known. We have two such
systems, with a third one arriving today (whee!). A CNAME shuffle
record in our DNS is used to spread the user load around (trying an
"nslookup shell.io.org"). Although I might be able to get away with
having only a single login server, having more available gives me the
flexibility to take one of them down for maintenance, knowing the
remaining ones will be able to handle the load. All run stock
2.1.0-RELEASE with no additional patches.
>>>>>
FreeBSD 2.1.0-RELEASE #0: Sat Dec 30 14:19:27 EST 1995
taob@flinch.io.org:/src/2.1.0-RELEASE/sys/compile/SHELL
CPU: 133-MHz Pentium 735\90 or 815\100 (Pentium-class CPU)
Origin = "GenuineIntel" Id = 0x52b Stepping=11
Features=0x1bf<FPU,VME,PSE,MCE,CX8,APIC>
real memory = 134217728 (131072K bytes)
avail memory = 130035712 (126988K bytes)
Probing for devices on the ISA bus:
vt0 at 0x60-0x6f irq 1 on motherboard
vt0: ega, mono, 8 scr, mf2-kbd, [R3.20-b24]
sio0 at 0x3f8-0x3ff irq 4 on isa
sio0: type 16550A
sio1 at 0x2f8-0x2ff irq 3 on isa
sio1: type 16550A
fdc0 at 0x3f0-0x3f7 irq 6 drq 2 on isa
fdc0: NEC 72065B
fd0: 1.44MB 3.5in
npx0 on motherboard
npx0: INT 16 interface
Probing for devices on the PCI bus:
chip0 <Intel 82437 (Triton)> rev 2 on pci0:0
chip1 <Intel 82371 (Triton)> rev 2 on pci0:7
ncr0 <ncr 53c810 scsi> rev 2 int a irq 10 on pci0:11
ncr0 waiting for scsi devices to settle
(ncr0:1:0): "QUANTUM FIREBALL1080S 1Q09" type 0 fixed SCSI 2
sd0(ncr0:1:0): Direct-Access
sd0(ncr0:1:0): FAST SCSI-2 100ns (10 Mb/sec) offset 8.
1042MB (2134305 512 byte sectors)
sd0(ncr0:1:0): with 3835 cyls, 4 heads, and an average 139 sectors/track
de0 <Digital DC21140 Fast Ethernet> rev 18 int a irq 11 on pci0:12
de0: DC21140 [10-100Mb/s] pass 1.2 Ethernet address 00:00:c0:80:46:c8
de0: enabling 10baseT UTP port
machine "i386"
cpu "I586_CPU"
ident CABAL
maxusers 128
options INET
options FFS
options NFS
options MSDOSFS
options PROCFS
options QUOTA
options "COMPAT_43"
options "SCSI_DELAY=5"
options SCSIDEBUG
options SCSI_REPORT_GEOMETRY
options UCONSOLE
options SYSVSHM
options SYSVSEM
options SYSVMSG
options PROBE_VERBOSE
options "NMBCLUSTERS=4096"
options "MAXMEM=131072"
config kernel root on sd0
controller isa0
controller pci0
controller ncr0
controller scbus0
device sd0
controller fdc0 at isa? port "IO_FD1" bio irq 6 drq 2 vector fdintr
disk fd0 at fdc0 drive 0
device vt0 at isa? port "IO_KBD" tty irq 1 vector pcrint
options "PCVT_FREEBSD=210"
device npx0 at isa? port "IO_NPX" irq 13 vector npxintr
device sio0 at isa? port "IO_COM1" tty irq 4 vector siointr
device sio1 at isa? port "IO_COM2" tty irq 3 vector siointr
device de0
pseudo-device loop
pseudo-device ether
pseudo-device log
pseudo-device pty 256
pseudo-device bpfilter 4
--
Brian Tao (BT300, taob@io.org)
Systems Administrator, Internex Online Inc.
"Though this be madness, yet there is method in't"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.3.91.960215095838.24170f-100000>