Newsgroups: comp.os.linux.development.system
Path: pad-thai.cam.ov.com!bloom-beacon.mit.edu!gatech!newsxfer.itd.umich.edu!
agate!hpg30a.csc.cuhk.hk!news.ust.hk!usenet
From: cc...@cs.ust.hk (Chi-Chung Hui)
Subject: Suggestion on the memory management of Linux
Message-ID: <1995Jan16.085103.17348@uxmail.ust.hk>
Keywords: Linux, Memory Management, Suggestion
Sender: use...@uxmail.ust.hk (usenet account)
Nntp-Posting-Host: cssu135.cs.ust.hk
Reply-To: cc...@cs.ust.hk
Organization: Computer Science Department, HKUST
Date: Mon, 16 Jan 1995 08:51:03 GMT
Lines: 111
I would like to share my experience on using Linux with you, and I hope I can
give some useful suggestions about its memory management.
First of all, I am running Linux v1.1.73 with 16M RAM and 8M swap space.
When I am using GCC under X (with olvwm + 2 textedits + 2 xterms + 2 tcshs),
the system begins to thrash. This does not happen when X is not running.
When I sum up the memory usage (got by 'ps -aux'), I found out that the
total memory utilization is 104%, and every process (no matter it is idle
for a long time or not) utilized certain amount of memory:
-------------------------
linux:cchui:1>ps -aux
USER PID %CPU %MEM SIZE RSS TTY STAT START TIME COMMAND
...
cchui 599 13.2 3.6 379 560 pp0 S 11:46 0:00 -tcsh
cchui 610 2.0 9.6 286 1488 pp0 S 12:31 0:00 textedit
cchui 612 13.0 8.3 222 1288 pp0 S 12:31 0:00 textedit
cchui 602 0.0 1.5 80 236 pp0 R 11:46 0:00 ps -aux
root 1 0.0 1.4 48 216 ? S Dec 20 0:03 init auto
root 5 0.0 0.8 24 124 ? S Dec 20 0:00 bdflush (daemon)
root 6 0.0 0.8 24 128 ? S Dec 20 0:00 update (bdflush)
root 24 0.0 0.8 24 128 ? S Dec 20 0:00 update (bdflush)
root 38 0.0 1.5 57 232 ? S Dec 20 0:02 /usr/sbin/syslogd
root 40 0.0 1.2 36 200 ? S Dec 20 0:00 /usr/sbin/klogd
root 42 0.0 1.4 68 220 ? S Dec 20 0:00 /usr/sbin/inetd
root 44 0.0 1.3 64 204 ? S Dec 20 0:00 /usr/sbin/lpd
root 49 0.0 2.5 251 396 ? S Dec 20 0:00 sendmail: accepting c
root 54 0.0 1.5 88 232 v04 S Dec 20 0:00 /sbin/getty tty4 3840
root 55 0.0 1.5 88 232 v05 S Dec 20 0:00 /sbin/getty tty5 3840
root 56 0.0 1.5 88 232 v06 S Dec 20 0:00 /sbin/getty tty6 3840
root 201 0.0 1.5 88 232 v02 S Dec 20 0:00 /sbin/getty tty2 3840
root 205 0.0 1.5 88 232 v03 S Dec 20 0:00 /sbin/getty tty3 3840
root 224 0.0 1.5 88 232 v01 S Dec 20 0:00 /sbin/getty tty1 3840
root 598 3.4 1.7 78 264 ? S 11:46 0:00 in.telnetd
...
-------------------------
All the processes that should be idle (e.g., sendmail and getty's) are
still occupying memories (I am running under X, the getty's must be idle.
Sendmail should be idle also since I have no network connection). These
processes should be swapped out so that an extra 11.5% of memories can
be allocated to GCC. If I do not use textedit during compilation, the
memories occupied by the two textedits can be swapped out temporarily
to free an extra 17.9% of memory. I am sure that no swapping will occur
anymore when GCC has an extra 29.4% of memory (~= 4.7Mbyte).
Unfortunatly, I found out that these processes are not swapped out and the
disk I/O is very heavy.
I would like to compare this with SunOS 4.1.3. I got the following ps output
from a SPARCstation:
-------------------------
sparc:home:15>ps -aux
USER PID %CPU %MEM SZ RSS TT STAT START TIME COMMAND
...
cchui 28792 0.0 4.9 120 1384 co S 11:44 0:00 clock -Wp 240 240 -Ws 13
cchui 29074 0.0 1.7 248 488 p1 R 12:45 0:00 ps -aux
cchui 28744 0.0 14.1 2980 3956 co S 11:44 0:29 /usr/openwin/bin/xnews :
cchui 28816 0.0 0.0 284 0 p2 IW 11:45 0:00 -bin/tcsh (tcsh)
cchui 27849 0.0 0.0 268 0 co IW 17:34 0:00 -tcsh (tcsh)
cchui 28852 0.0 0.0 220 0 p5 IW 11:45 0:00 cmdtool -Wp 0 0 -Ws 539
cchui 28780 0.0 1.4 276 384 p1 S 11:44 0:00 -bin/tcsh (tcsh)
root 27448 0.0 0.0 356 0 ? IW Nov 15 6:04 automount
root 29030 0.0 0.0 36 0 ? IW 12:37 0:00 in.telnetd
cchui 28832 0.0 0.0 192 0 co IW 11:45 0:00 textedit -Wp 549 79 -Ws
cchui 28825 0.0 0.0 192 0 co IW 11:45 0:00 textedit -Wp 0 79 -Ws 53
csrico 28706 0.0 0.0 48 0 p7 IW 11:14 0:00 rlogin csl3su10
cchui 28760 0.0 0.9 216 252 co S 11:44 0:00 olwm -3
cchui 28779 0.0 2.3 228 644 p1 S 11:44 0:02 cmdtool -Wp 548 408 -Ws
cchui 28739 0.0 0.0 32 0 co IW 11:44 0:00 /bin/sh /usr/openwin/bin
cchui 28772 0.0 3.5 692 992 co S 11:44 0:04 mailtool -Wp 549 79 -Ws
cchui 28773 0.0 0.0 92 0 co IW 11:44 0:00 olwmslave
...
-------------------------
Only the activating processes are occupying memories (i.e., I am sending
email to you <mailtool>. The clocking is running continuously <clock>.
And I am running olwm <xnews and olwm>, and <ps> under <tcsh>). All the
idle processes, including the textedit's, cmdtool, and the deamons are
swapped out. The total memory utilization is only 36%, although there
are three persons logging in.
If I guess correctly, Linux uses part of the memory in storing the shared
libraries, and the remaining for the process and disk buffer. When all
the buffers are used, and *only* when there are new processes coming, the
kernel begins to thrash in order to make free memories.
And after that, every new process introduce some levels of thrashing
which degrade the system performance (response time) dramatically.
What I want to suggest is that, is it possible to introduce a machanism
to ensure that there are sufficient free memory pages at any time? There
are several mechanisms to achieve this:
1) Add a swapper (by "swapper" I mean "the process with
pid 0", which function is to swap the idle processes to the swap space)
to swap out the idle processes (e.g., not activate for
more than 1 minute), and the corresponding pages are put at the end
of the free list. However, the content of these pages are not cleared
so that it can be reused by the idle processes immediately when they
are activated.
2) Begin to flush the disk buffer onto the disk as long as the disk is idle.
In this way, the disk buffer can be reused without flushing anymore.
The effect of the above mechanisms is to make sure that at least x% of
the memory are ready to be used (x should be at least 20). In this way,
we can ensure the response time at a satisfactory level.
-------
C.C.Hui
The Hong Kong University of Science and Technology
Path: nntp.gmd.de!Germany.EU.net!EU.net!nova.puug.pt!news.ist.utl.pt!
ci.ist.utl.pt!l39801
From: l39...@ci.ist.utl.pt (Highlander C H B M)
Newsgroups: comp.os.linux.development.system
Subject: Re: Suggestion on the memory management of Linux
Date: 2 Feb 1995 12:54:11 GMT
Organization: Instituto Superior Tecnico
Lines: 46
Message-ID: <3gqklj$qb3@alfa.ist.utl.pt>
References: <1995Jan16.085103.17348@uxmail.ust.hk> <D2KLqn.29J@pe1chl.ampr.org>
<RAJU.95Feb1182047@darbari.toi.ernet.in>
NNTP-Posting-Host: ci.ist.utl.pt
X-Newsreader: TIN [version 1.2 PL2]
Raj Mathur (r...@darbari.toi.ernet.in) wrote:
: >>>>> "Rob" == Rob Janssen <r...@pe1chl.ampr.org> writes:
: In article <D2KLqn....@pe1chl.ampr.org> r...@pe1chl.ampr.org (Rob Janssen)
writes:
: [snip]
: Rob> A hint to improve things a bit: after you have booted the
: Rob> system and started X, run a program that uses a lot of
: Rob> memory. It will force out the idle processes, and when it
: Rob> terminates there is a lot of room. The system will be much
: Rob> faster after that. Indeed it would be nice when this is
: Rob> somehow done automatically, like you propose. Sounds like
: Rob> you have a nice project to try!
: Rob> Rob
: Rob> example program:
: [snip]
: I just give the command
: swapoff -a ; swapon -a
: from time to time, which clears up the swap and in the process the
: buffers (BTW, this is while in X). If my luck is bad, it also kills my
: Emacs. If my luck absolutely sux, it hangs the system :-) However, on
: the whole it does beat having to reboot the system in order to free up
: RAM.
: So am I doing the right thing? What advantage (if any) would I gain by
: running your program instead of the swap{off,on} combination?
[snip]
Why not just write a little C like :
while(1) { if ( malloc(2048) == NULL ) return(0) }
this would hog all the space it could and then exit netly returning it .
( MHO )
--
+---------------------------------+-------------------------------------------+
| Carlos Hugo Baptista Morgado | |
| e-mail: l39...@alfa.ist.utl.pt | "God is real unless declered integer" |
| l39...@ci.ist.utl.pt | |
| PGP key on finger +-------------------------------------------+
| Instituto Superior Tecnico de Lisboa -- Universidade Tecnica de Lisboa |
| ( Technical Superior Institute of Lisbon - Portugal ) |
+-----------------------------------------------------------------------------+
Path: nntp.gmd.de!dearn!esoc!linuxed1!peernews.demon.co.uk!btnet!bt!pipex!
howland.reston.ans.net!
lamarck.sura.net!hookup!swrinde!sgiblab!sgigate.sgi.com!fido.asd.sgi.com!
slovax!lm
From: l...@slovax.engr.sgi.com (Larry McVoy)
Newsgroups: comp.os.linux.development.system
Subject: Re: Suggestion on the memory management of Linux
Date: 3 Feb 1995 04:58:01 GMT
Organization: Silicon Graphics Inc., Mountain View, CA
Lines: 39
Message-ID: <3gsd4p$asb@fido.asd.sgi.com>
References: <1995Jan16.085103.17348@uxmail.ust.hk> <D2KLqn.29J@pe1chl.ampr.org>
<RAJU.95Feb1182047@darbari.toi.ernet.in> <3gqklj$qb3@alfa.ist.utl.pt>
Reply-To: l...@slovax.engr.sgi.com
NNTP-Posting-Host: slovax.engr.sgi.com
X-Newsreader: TIN [version 1.2 PL2]
Highlander C H B M (l39...@ci.ist.utl.pt) wrote:
: Why not just write a little C like :
: while(1) { if ( malloc(2048) == NULL ) return(0) }
: this would hog all the space it could and then exit netly returning it .
I have added some system calls that do this in my version of Linux.
They work like so
swapout(pid_t pid) - swapout the process completely
shrink_buffers() - sync all buffers and free those that can be freed
They aren't ready for prime time yet, but I can give you early
results: a 16MB machine is a far more pleasant place to work. When I
start up and get into X things get kinda full (and I like to have the
Linux logo and xfishtank going too - things got real tight). Top looks
like:
Mem: 15128K av, 12732K used, 2396K free, 5648K shrd, 2904K buff
Swap: 30036K av, 2316K used, 27720K free
After running swapout on all the processes
Mem: 15128K av, 7048K used, 8080K free, 732K shrd, 3168K buff
Swap: 30036K av, 9564K used, 20472K free
And after running shrink
Mem: 15128K av, 4848K used, 10280K free, 788K shrd, 860K buff
Swap: 30036K av, 9556K used, 20480K free
Going from 2.3MB free to 10.2MB free, yeah, I like that. I'm using 5MB,
running X and xfishtank and top. I can live with that.
As soon as I get these a bit polished, I'll ship them to Linus and see if
he'll put them in the base kernel.
--
---
Larry McVoy (415) 390-1804 l...@sgi.com
Newsgroups: comp.os.linux.development.system
From: ja...@purplet.demon.co.uk (Mike Jagdis)
Path: nntp.gmd.de!dearn!esoc!linuxed1!peernews.demon.co.uk!purplet!jaggy
Subject: Re: Suggestion on the memory management of Linux
Organization: FidoNet node 2:252/305 - The Purple Tentacle, Reading
X-Posting-Host: purplet.demon.co.uk
Date: Sat, 4 Feb 1995 00:24:00 +0000
Message-ID: <809.2F33FD61@purplet.demon.co.uk>
Sender: use...@demon.co.uk
Lines: 21
* In message <3gsd4p$...@fido.asd.sgi.com>, Larry McVoy said:
LM> I have added some system calls that do this in my version of
LM> Linux.
LM> They work like so
LM> swapout(pid_t pid) - swapout the process completely
LM> shrink_buffers() - sync all buffers and free those
LM> that can be freed
Hmmm... I took the approach of a bdflush() like syscall that wakes up when
pages start running low and pages things out until a high water mark is
reached again. When the swap daemon is running processes don't swap
themselves - they just hang on a semaphore and are woken up when a page
becomes available. If the swap daemon isn't running the normal eveyone
thrash at once rule applies :-).
I haven't done a lot of tuning with this either. I was waiting for 1.3
before I do too much...
Mike