From owner-freebsd-questions Thu Feb 15 07:51:21 1996 Return-Path: owner-questions Received: (from root@localhost) by freefall.freebsd.org (8.7.3/8.7.3) id HAA05440 for questions-outgoing; Thu, 15 Feb 1996 07:51:21 -0800 (PST) Received: from zip.io.org (root@zip.io.org [198.133.36.80]) by freefall.freebsd.org (8.7.3/8.7.3) with SMTP id HAA05434 Thu, 15 Feb 1996 07:51:17 -0800 (PST) Received: (from taob@localhost) by zip.io.org (8.6.12/8.6.12) id KAA27113; Thu, 15 Feb 1996 10:49:04 -0500 Date: Thu, 15 Feb 1996 10:49:04 -0500 (EST) From: Brian Tao To: "Miguel A.L. Paraz" cc: freebsd-questions@freebsd.org, FREEBSD-ISP-L Subject: 128+ users on one system? (was Re: It feels so good!) In-Reply-To: <199602150904.RAA09887@marikit.iphil.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-questions@freebsd.org Precedence: bulk On Thu, 15 Feb 1996, Miguel A.L. Paraz wrote: > > Brian Tao wrote: > > > 2:05AM up 8 days, 12:13, 132 users, load averages: 2.84, 3.29, 4.05 > > Wow! Care to share your system configuration, kernel and > system and tuning tricks, et cetera? There's nothing extraordinary about the hardware or the kernel. This is an example of what FreeBSD can do "out of the box", which is rather impressive (at least to me ;-) ). Hardware is a 133-MHz Pentium system, ASUS P/I-P55TP4XEG motherboard with 512K pipeline burst cache and 128M of RAM. Tip: if you're spending this kind of money on a system, go out and buy PCI versions of everything! Our supplier shipped the systems with a cheap Trident PCI VGA card, an NCR53c810 SCSI controller, and an SMC EtherPower 9332 10/100Mbps NIC. The EtherPower 8432 (I think that's the number, it's the 10Mbps-only version) also works well. Hardware installation was literally "plug and play". The PCI firmware sorts out all the nastiness normally associated with ISA peripherals. IRQ's are settable through the BIOS setup screen (the defaults worked for me) and the 2.1.0-R install kernel correctly identified all the pieces without prior fiddling with jumpers and DOS-based config utilities. :) Total cost of the system came to around C$6000, or around US$4500. The killer is the RAM, of course (C$4200 for four 60ns 32MB SIMM's), but the software was free. :) That works out to around US$45 per user, if you limit yourself to 100 online at a time. At 100+ interactive users, I've found that the bottleneck is at the 10Mbps Ethernet interface, not the CPU or memory. With that many users typing away at their terminals, de0 is sending and receiving 400 to 500 packets per second. Home directories are NFS mounted, which doesn't help matters. This results in lagged keyboard response and screen updates. Putting the machine on a 100Mbps port may eliminate this bottleneck. Swap usage hovers around 10 to 20% and the CPU doesn't appear to be working particularly hard with 100+ users. I'd guess that it can handle 150 interactive users comfortably, bringing the costs down to a mere US$30 per user. More tips... don't forget to configure and MAKEDEV all 256 pty's! I let my users run 'screen' here, and it was configured at compile time to use pty[PQRS] (the upper 128 pty's) so that [pt]ty[pqrs] were available for telnetd and rlogind. Also, be sure to have enough mbufs around to avoid locking up the network on that machine if they ever run out. This happened once with NMBCLUSTERS=2048, so now I have it set to 4096. I've seen the high-water mark hit 1952 clusters since then. I've included the dmesg output and kernel config file below. As I said, there isn't anything out of the ordinary with the configuration. All the options are documented or well-known. We have two such systems, with a third one arriving today (whee!). A CNAME shuffle record in our DNS is used to spread the user load around (trying an "nslookup shell.io.org"). Although I might be able to get away with having only a single login server, having more available gives me the flexibility to take one of them down for maintenance, knowing the remaining ones will be able to handle the load. All run stock 2.1.0-RELEASE with no additional patches. >>>>> FreeBSD 2.1.0-RELEASE #0: Sat Dec 30 14:19:27 EST 1995 taob@flinch.io.org:/src/2.1.0-RELEASE/sys/compile/SHELL CPU: 133-MHz Pentium 735\90 or 815\100 (Pentium-class CPU) Origin = "GenuineIntel" Id = 0x52b Stepping=11 Features=0x1bf real memory = 134217728 (131072K bytes) avail memory = 130035712 (126988K bytes) Probing for devices on the ISA bus: vt0 at 0x60-0x6f irq 1 on motherboard vt0: ega, mono, 8 scr, mf2-kbd, [R3.20-b24] sio0 at 0x3f8-0x3ff irq 4 on isa sio0: type 16550A sio1 at 0x2f8-0x2ff irq 3 on isa sio1: type 16550A fdc0 at 0x3f0-0x3f7 irq 6 drq 2 on isa fdc0: NEC 72065B fd0: 1.44MB 3.5in npx0 on motherboard npx0: INT 16 interface Probing for devices on the PCI bus: chip0 rev 2 on pci0:0 chip1 rev 2 on pci0:7 ncr0 rev 2 int a irq 10 on pci0:11 ncr0 waiting for scsi devices to settle (ncr0:1:0): "QUANTUM FIREBALL1080S 1Q09" type 0 fixed SCSI 2 sd0(ncr0:1:0): Direct-Access sd0(ncr0:1:0): FAST SCSI-2 100ns (10 Mb/sec) offset 8. 1042MB (2134305 512 byte sectors) sd0(ncr0:1:0): with 3835 cyls, 4 heads, and an average 139 sectors/track de0 rev 18 int a irq 11 on pci0:12 de0: DC21140 [10-100Mb/s] pass 1.2 Ethernet address 00:00:c0:80:46:c8 de0: enabling 10baseT UTP port machine "i386" cpu "I586_CPU" ident CABAL maxusers 128 options INET options FFS options NFS options MSDOSFS options PROCFS options QUOTA options "COMPAT_43" options "SCSI_DELAY=5" options SCSIDEBUG options SCSI_REPORT_GEOMETRY options UCONSOLE options SYSVSHM options SYSVSEM options SYSVMSG options PROBE_VERBOSE options "NMBCLUSTERS=4096" options "MAXMEM=131072" config kernel root on sd0 controller isa0 controller pci0 controller ncr0 controller scbus0 device sd0 controller fdc0 at isa? port "IO_FD1" bio irq 6 drq 2 vector fdintr disk fd0 at fdc0 drive 0 device vt0 at isa? port "IO_KBD" tty irq 1 vector pcrint options "PCVT_FREEBSD=210" device npx0 at isa? port "IO_NPX" irq 13 vector npxintr device sio0 at isa? port "IO_COM1" tty irq 4 vector siointr device sio1 at isa? port "IO_COM2" tty irq 3 vector siointr device de0 pseudo-device loop pseudo-device ether pseudo-device log pseudo-device pty 256 pseudo-device bpfilter 4 -- Brian Tao (BT300, taob@io.org) Systems Administrator, Internex Online Inc. "Though this be madness, yet there is method in't"