Tech Insider					     Technology and Trends


			      USENET Archives

From: ur...@usa.net
Subject: 2.2.13ac3 crashes under high network load
Date: 1999/12/15
Message-ID: <837hq1$hat$1@nnrp1.deja.com>
X-Deja-AN: 560862255
X-Http-Proxy: 1.0 x35.deja.com:80 (Squid/1.1.22) for client 199.95.209.163
Organization: Deja.com - Before you buy.
X-Article-Creation-Date: Wed Dec 15 08:00:02 1999 GMT
X-MyDeja-Info: XMYDJUIDray450
Newsgroups: comp.os.linux.networking
X-Http-User-Agent: Mozilla (X11; I; Linux 2.0.32 i586)

Alan C/Dave M/Ingo M/Dan K et. al ...

I'm trying to build a "ultra high-performance" webserver
which needs to handle at least 500 (preferably 1000) sustained
hits/second (my entire webserver farm needs to handle excess
of 1 billion hits/day already, soon to be 2 billion/day).
Yes I've read the C10K page at http://www.kegel.com/c10k.html
and tried some of the experimental webservers, most of which
are way too experimental (and lacking features :) which I need.
The issue isn't with Apache not cutting it, but rather with
some kernel (and/or network driver) issues I think ...

Here is my hardware and software configuration:

Compaq 6400R (rackmount):
  dual PIII-550 MHz processors
  2048 MByte ECC memory
  Intel EtherExpress Pro/100 NIC

Linux 2.2.13:
  with Alan Cox's "2.2.13ac3" super-patch applied
  using Donald Becker's eepro100.c:v1.09l (8/7/99)
  "BIGMEM" option enabled in kernel configuration
  using raid0145 (raidtools-0.90) (in ac3 patchset)
  system on single 9.1G Ultra-Wide 10K SCSI drive
  web content on RAID-0 across 2 identical UWSCSI drives
  modify some SysCtl parameters via script (see below)

I have the above setup on 10 servers running Apache 1.3.9
(modified with an updated version of his TOP_FUEL patch)
in a cluster, managed by a Cisco LocalDirector 420. Under
test conditions, I can pull content fast enough from Apache,
to saturate the 100MBit/s full-duplex [switched] interface
on each webserver (according to `ab`, anyhow).

HOWEVER, after 2-3 days of uptime, handling on average
between 50 and 200 hits/second (per server) which means
from 100 to 400 (and up) open connections per server ...
these machines will start to fail. Usually this happens,
when the number of active TCP/IP connections is high,
say about 1000 for a brief period of time. I did have to
increase the route-cache limits in /proc/sys/net/ipv4
per a posting from Alexey Kuznetsov in fa.linux.kernel,
else I was getting tons of "dst_cache_overflow" errors)

The server becomes unreachable (cannot be PING'ed)
and on the console (finally figured out how to disable
the damned screen-blanker/powersave via a little script
during init, also pasted below). First I see some stuff
about INODE already being cleared (seems like RAID
related problems?) and shortly after, different messages
start looping on the console:

  <[8010b37d]> <[80162067]> <[80162140]> \
  <[8016f168]> <[801511d4]> <[80161fa6]>

  wait_on_bh, CPU 3:
  irq:	0 [0 0]
  bh:	1 [0 0]

(repeats about once per second)

I've compiled in the 'Magic SysRq' feature into the kernel,
so upon pressing <ALT> + <SysRq> + P, we see the following:
(there might be a few mistakes, hand-copied!)

  >SysRq: Show Regs

  EIP: 0010:[<8010b38c>]  EFLAGS: 00000202
  EAX: 00000001  EBX: fae00e80  ECX: f0aeff14  EDX: aee2c21d
  ESI: 00000000  EDI: f0aee000  EBP: f0aee000  DS: 0018  ES: 0018
  CR0: 8005003b  CR2: 2aac1000  CR3: 69e5f000

Pressing <ALT> + <SysRq> + M shows the usual,
nothing unusual about usage and no swap in use.
I did some searches on DejaNews about wait_on_bh,
and found references from January, March 1999
from David Miller, Alan Cox, saying this was
SMP-related ("wait on bottom-half") and fixed,
in 2.2.3 (?) Apparently it's not really fixed :/

On other servers in the same cluster, I'm not seeing the
"wait_on_bh" error above, but rather some resource-starvation
issue (I think); the console displays the usual /etc/issue
message and a login: prompt, but when I try to login (any
user) there is a message displayed [very fast/unreadable]
and I'm kicked back to the login screen again. Suspect that
the message is about "Resource temporarily unavailable".

One one of these machines which would not let me login,
after reducing the network load (bled off some traffic)
I could finally login interactively. I went ahead and
pressed <ALT> + <SysRq> + P on all the machines stuck
at the login screen, and they mostly all were at EIP
"80107a71" (cpu_idle, according to my System.map).
The output from <ALT> + <SysRq> + M was more interesting,
some of these servers had NEGATIVE "buffer hash" values,
e.g. "buffer hash: -191435", "buffer hash:-178450", etc).

In all cases I was able to sync via <ALT> + <SysRq> + S
(some servers I had to <ALT> + <SysRq> + E first, though)
and Unmount/reBoot via SysCtl also in most of the crashes.

For the time being, I've edited /etc/lilo.conf, and appended
"nosmp noapic" to the active kernel entry, to force non-SMP
mode; I'm going to run the webserver cluster for several days,
and see if the same problems occur ...

Any help/patches/advice on the above is greatly appreciated.
Thanks in advance ...

RW

----------------------------------------------------------------------
/etc/rc.d/init.d/noblank
----------------------------------------------------------------------
#!/bin/sh
#
# Prevents situations where kernel crashes,
# but we cannot see any console error messages
# because the screen was blanked earlier.

IFS=' '
TTYS='1 2 3 4 5 6'

for n in ${TTYS}
do
    echo "eval `/usr/bin/setterm -blank 0`"         > /dev/tty${n}
    echo "eval `/usr/bin/setterm -powersave off`"   > /dev/tty${n}
done

exit 0

---------------------------------------------------------------------
/etc/rc.d/init.d/proctune
----------------------------------------------------------------------
#!/bin/bash
#
# /etc/rc.d/init.d/proctune.linuxcare
#
# chkconfig: 345 80 20
# description: SysCtl (proc) tunings from LinuxCare \
#              based on research of Jim Dennis
#
# processname: proctune.linuxcare

# Source function library.
. /etc/rc.d/init.d/functions

#
#       See how we were called.
#
case "$1" in
  start)
        echo "Running proctune.linuxcare:"
        #
        echo -n "  /proc/sys/fs/file-max  . . . . . . . . . . . "
        echo '16384' > /proc/sys/fs/file-max
        cat /proc/sys/fs/file-max
        #
        echo -n "  /proc/sys/fs/inode-max . . . . . . . . . . . "
        echo '65536' > /proc/sys/fs/inode-max
        cat /proc/sys/fs/inode-max
        #
        echo -n "  /proc/sys/net/ipv4/ip_local_port_range . . . "
        echo "32768 65535" > /proc/sys/net/ipv4/ip_local_port_range
        cat /proc/sys/net/ipv4/ip_local_port_range
        #
        echo -n "  /proc/sys/net/ipv4/route/gc_elasticity . . . "
        echo '2' > /proc/sys/net/ipv4/route/gc_elasticity
        cat /proc/sys/net/ipv4/route/gc_elasticity
        #
        echo -n "  /proc/sys/net/ipv4/route/gc_min_interval . . "
        echo '1' > /proc/sys/net/ipv4/route/gc_min_interval
        #echo '0' > /proc/sys/net/ipv4/route/gc_min_interval
        cat /proc/sys/net/ipv4/route/gc_min_interval
        #
        #echo -n "  /proc/sys/net/ipv4/route/gc_thresh . . . . ."
        #echo '256' > /proc/sys/net/ipv4/route/gc_thresh
        #echo '512' > /proc/sys/net/ipv4/route/gc_thresh
        #cat /proc/sys/net/ipv4/route/gc_thresh
        #
        echo -n "  /proc/sys/net/ipv4/route/max_size  . . . . . "
        #echo '4096' > /proc/sys/net/ipv4/route/max_size
        echo '8192' > /proc/sys/net/ipv4/route/max_size
        cat /proc/sys/net/ipv4/route/max_size
        #
	#
        echo -n "  /proc/sys/vm/bdflush . . . . . . . . . . . . "
        echo '98 100 128 256 15 500 1884 2 2' > /proc/sys/vm/bdflush
        cat /proc/sys/vm/bdflush
        #
	#
        echo -n "  /proc/sys/vm/bdflush . . . . . . . . . . . . "
        echo '98 100 128 256 15 500 1884 2 2' > /proc/sys/vm/bdflush
        cat /proc/sys/vm/bdflush
        #
        #
        echo -n "  /proc/sys/vm/buffermem . . . . . . . . . . . "
        echo '90 10 98' > /proc/sys/vm/buffermem
        cat /proc/sys/vm/buffermem
        #
        #echo -n "  /proc/sys/vm/overcommit_memory . . . . . . ."
        #echo '1' > /proc/sys/vm/overcommit_memory
        #cat /proc/sys/vm/overcommit_memory
        #
        echo -n "  /proc/sys/vm/page-cluster  . . . . . . . . . "
        echo '5' > /proc/sys/vm/page-cluster
        cat /proc/sys/vm/page-cluster
        #
        echo -n "  /proc/sys/vm/pagecache . . . . . . . . . . . "
        echo '80 30 95' > /proc/sys/vm/pagecache
        cat /proc/sys/vm/pagecache
        #
        ;;
  stop)
        :
        ;;
  *)
        echo "Usage: /etc/rc.d/init.d/proctune.linuxcare {start|stop}"
        exit 1
esac

exit 0


Sent via Deja.com http://www.deja.com/
Before you buy.

			      USENET Archives


The materials and information included in this website may only be used
for purposes such as criticism, review, private study, scholarship, or 
research.


Electronic mail:			       WorldWideWeb:
   tech-insider@outlook.com			  http://tech-insider.org/