Tech Insider					     Technology and Trends


			      USENET Archives

Newsgroups: comp.arch.storage
Path: sparky!uunet!van-bc!ubc-cs!newsserver.sfu.ca!sfu.ca!vanepp
From: van...@fraser.sfu.ca (Peter Van Epp)
Subject: Mainframe as high transaction rate database engine (longish)
Message-ID: <vanepp.704949857@sfu.ca>
Keywords: TPF Mainframes
Sender: ne...@sfu.ca
Organization: Simon Fraser University, Burnaby, B.C., Canada
Date: Mon, 4 May 1992 03:24:17 GMT
Lines: 143

Since a couple of people have asked to hear more about how the program that
airlines and banks use for reservations (in the case of airlines) and 
check clearing (in the case of banks),  I'll do a fast (and possibly not
dead accurate, I've haven't done it in 4 years -:)) description of the 
database/disk farm (as I said, I wouldn't call it a file system -:)) and
if you don't want to hear anymore I'll shut up -:).
	The following description applies to TPF1, I expect the general
architecture (and almost certainly the database layout) is still the same
in whatever the current version is (TPF 2.4 is the last one I know of).
	The reason that I say there is no file system is that everything
has to fit into a disk block, one of three (or 4) sizes fits all, 128 bytes
381 bytes, 1055 bytes and in later systems 4096 bytes. Other than the 
basic kernel (some 128k bytes as I recall) all programs and data exist in 
one of these blocks. In order to allow you to change the makeup of the 
underlying disk farm, there is actually a translation mechanism that accepts
a "block number" (a 4 byte hex value) and the system translates that (one
to one) to a disk and track cylindar sector. All blocks have a chain word
so when you run out of space in the current block, you can request a new one 
from one of two pools of free blocks, chain it to the current block and
continue doing whatever you were doing (in the case of a program, when
you run out of a 1055 byte block, you have to call a new program to get
more space (and split the source code so it maps between the two blocks
by hand!).
	As I said above, there are two pools of free blocks, short term
and long term, each pool contains all types of blocks (maybe not 128's
I'm not sure anymore -:)). The difference between the two is how the 
block directories work, and the amount of time you will need the block for
short term is only good for three to four hours and long term is yours for
as long as you don't release it. The short term pool area is intended for
use on a single transaction (or booking) basis, use it for up to an hour
or so and then release it. There are a series of bitmaps for short term
pool and a system call that will return you the address of a block of the 
requested size attached to your task control block and backed by a block
sized piece of main storage. When a directory is full, the system steps into
the next directory (setting all of the blocks in the directory to "not in 
use" whether they are or not!), the directories are set such that this takes
between 4 and 12 hours depending on how busy the system is. If an application
is still using a pool block that has been deallocated by teh directory rolling
over, then a system error occurs, the contents of main storage that are 
considered "interesting or useful" are dumped to a channel connected 
mag tape (that you want to go fast, believe me -:)) while the system "freezes"
for the time needed for the dump to get a consistant snapshot. Obviously you
want to dump the minimum amount of memory you can to the fastest tape device
you can -:)), once the dump finishes, an error message is sent to the user
(typically "system error - redo transaction", this is called use initiated
errpr correction -:)).
	This is all very fine for short term data, but since most airline
customers wish to book their flight somewhat more than 4 hours before flight
time we need something with little longer lifetime -:) This comes in the 
form of "fixed file" and long term pool, There is an area of the disk that
is like a preallocated file system, where you can request a block by a 
sort of file name (a hex number again -:)), this block serves as the 
anchor for a chain of long term pool blocks where the data about a flight,
passengers, and whatever other data you have is stored. One item to note
(to do with performance) is that successive records (of for instance 
passengers booked on a flight) are spread across the disks in the disk
farm (and there are usually lots of disk channels -:)). This is so that
when flight departure is near and lots of agents are making queries against
the same flight records, the single threadedness of tings is minimized
(ie. an agent changing Mr Aaaa's reservation doesn't block access to another
agent changing Mr Gggg's reservation, although it would briefly block Mr Abaa's
reservation from being changed). Another part of the performance secret is that
a single transaction should not execute for more than a few 10's to 100's
of milliseconds, so a booking is broken up into a long series of "transactions"
each being very short, and all of them modifying a block in short term pool
until all the data is in and the transation is "ended" and the data is written
out to the correct place in long term pool (again in a short transaction).
	Like short term pool, there are a series of directories for long
term pool blocks (bitmaps), unlike short term pool, these directories are
never recycled, and when a long term pool block is released, the directory
entry is not set to "free", it is left at "in use", all that happens is that
a log record is written to the log tape saying "block xyz has been released".
Understandably, over time more and more of the blcoks in long term storage
become "in use" and if left long enough, you would run out and everything 
would grind to a halt (very Unix like, well I set the return code on this
write that just failed, but you ignored it, so that must be what you wanted
to do! -:)), and it does to the great consternation of all. The proper
course of action (before or after running out of pool -;)) is to run something
called "recoup", this is effectivly a garbage collection pass, which starts
at the fixed records (one at a time) and chases all the chains of blocks that
are connected to the fixed records (marking them as "in use" as it goes).
Anything that isn't found connected to a block is considered released and 
will be reused. Offline (on another system, typically MVS), the release 
records written to the log tape are compared to the directories generated
from the online garbage collection and differences are printed out in a 
report that the systems programmer then sweats over to attemt to make sure
that things have worked correctly, and the directories generated really 
reflect the state of the online system. Once all looks reasonable, then
comes the exciting part, recoup rollin (or toss the bones and bet your
job -:)), at the point that the online garbage collection finished, the
system keeps track of which fixed directories it has modified at the 
point of rollin, the changes are added to the directories generated offline
and the live directories are overwritten, if all is ok, releasing for use
all of the long term pool blocks released since the last recoup and 
everybody is happy. The reason for the "toss the bones and bet your job"
comment above is the downside of this procedure, if something goes wrong
somewhere in recoup (ie. there was a system crash at a bad time and a 
bunch of directories that shouldn't have got released), when you roll in 
the new directories, instead of freeing the unused blocks like it should
it releases all the live database data (can you say "excuse me but does
this 23rd story window open" in 18 languages? Most people running recoup
can -:)). After the white faces, the denial, and the screaming, one gets
out the backup tapes and starts a restore, (the entire data base is backed
up every other night, since even several day old data represents a huge
loss in bookings), this is an exciting (and time consuming) procedure, one
that I have luckily never participated in, I heard a report of an american
carrier that lost their reservation system for some 14 hours to a mistake
that sounded from afar to be a bad recoup being rolled in. This is a disaster
since our up time target was something like 99.5% (ie. 2 or 3 hours per
month, sheduled and unsheduled), system downtime was unofficially estimated
to be around $3000 per minute on a Monday morning ... Exciting enough to 
justify having a spare $4,000,000 mainframe in case the live one broke
(it was of course used to run other systems on, but the size was always
selected to be large enough to run the res system on).
	Given the consequence of data loss, and downtime the backup schemes
are also interesting. As a normal event, the input transaction is written 
to a log tape, as well the old data from any record that the transaction 
changes also gets written out to the log tape (given that we were doing 
80 transactions per second, you again want fast tape drives!). This means
that during a restore, you can roll forward to any particular transaction
since the last backup if some thing goes wrong (like a software error 
destroying the database after some particular time for instance).
	The backup itself is a block by block (track by track actually)
copy of all the data on the disks while the system still runs. In order
for the backup to be consistant, anything written to a disk that has  
already been backed up (other than short term pool records as I recall)
are also written to the log tape (which for your sake better be on a
different channel and control unit than the dump drives, which are 
themselves perferably each on their own channel and control unit).
	A restore involved dumping all of the backup tapes back onto the
disks, then running the dump log tapes in sequence until the data base
is at a consistant state, then running in the transaction log tapes one at 
time until the database has been rolled forward to the desired point, then
you get to run a recoup and roll it in (hopefully right this time if thats
what you screwed up last time!) to get back a checked an consistant system.
This is something that you shouldn't try at home (nor at work if there is
any way to avoid it -:)).
	Thats the basics, feel free to ask questions (and any of the IBM
guys out there that know TPF, correct my mistakes -:)), or say thats all
very nice but I'm not interested so shut up -:)

Peter Van Epp / van...@sfu.ca

Newsgroups: comp.arch.storage
Path: sparky!uunet!usc!rpi!batcomputer!cornell!uw-beaver!ubc-cs!
newsserver.sfu.ca!sfu.ca!vanepp
From: van...@fraser.sfu.ca (Peter Van Epp)
Subject: Re: Mainframe as high transaction rate database engine (longish)
Message-ID: <vanepp.705127031@sfu.ca>
Keywords: TPF Mainframes
Sender: ne...@sfu.ca
Organization: Simon Fraser University, Burnaby, B.C., Canada
References: <vanepp.704949857@sfu.ca>
Date: Wed, 6 May 1992 04:37:11 GMT
Lines: 64

A couple of points of were raised via e-mail that may be of more general 
interest, one is what is the configuration that this thing runs under?

Here are the general specs (again from memory after 4 or 5 years)

CPU (main and backup) Amdahl 5860 (11 MIPS?) IBM3081KX (17 mips?)
 	(TPF could only use one of the 2 processors on the 3081)
16 channels in use in both cases (32 and 64 on the machine as I recall for
	VM and/or MVS)
   12 channels to 12 disk controllers to 160 Amdahl 6880 disks running in 
			3350 mode, (~500 megs/disk around 300 megs in use
			due to seek time limitations).
   2 channels to two controllers for tapes
   2 byte mux channels supporting ~ 140 sync lines and some 5000 terminals
		across five continents.

32 Megs of main memory of which TPF only used the bottom 16 megs
	further subdivided into the bottom ~2 megs being used for
	the various core blocks and the kernel and pool directory
	structures and the other 14 megs as a write through cache
	into the disk farm to further reduce the real I/O rate. 

This configuration supported 80 transactions per second with a response
time guarantee of 90% of the transactions completing in under 3 seconds 
(including round trip comms times on 2400 to 9600 baud lease lines).

As a size comparison (this is a small tpf system!) the large American
carriers were reported to have transaction rates in the 2000 transaction
per second range on the largest CPUs IBM made (3090-600j at that point).

Several incidental points of interest, the later IBM (and Amdahl) cpus
were being tuned to running MVS and VM, a TPF system typically achieved
30% few MIPS than the system was rated for on MVS or VM (and you could
buy a special  machine called a 9090-xxxx that has unspecified changes
to get those MIPs back under TPF). The other one is that Amdahl != IBM,
of you are running TPF, we tripped over several differences that didn't
bother MVS or VM and lest this be taken as critisism of Amdahl, they
were quickly overcome in all cases with Amdahl's assistance and of course
having the source to the operating system and not being afraid to change
it.

	The point was made by someone else that he was sure the airlines 
would like to scrap it and get something more modern. While probably
true, there are a few barriers, performance, you need enough power to
maintain the response rate (a suggested change that would double the
response time prompted one of the bosses to calculate that we would need
a 2/3 increase in the number of reservations agents for the same booking
load in the main reservations area, note that this ignored the effect
on checking passengers in to planes at airports ....). There is a huge
investment in applications software (I added up many of the object files
on the system one day and as I recall it was in the 2 to 3 megabyte range)
heavily modified for the work habits of the airline involved plus a huge
base of trained people at remote sites (some 8,000 to 10,000 employees
many of whom were trained to use some portion of the res system). Remember
that all these applications are in 370 assembler contrived to fit within
1055 byte blocks and dealing with data in 1055 byte chunks, moving it
to C, for instance would be a huge job even before considering the performance
issues  this however is getting fairly far away from storage ... 
	I will close this by pointing out that there is much the same (if
on a slightly more reasonable scale) problem with many other mainframe 
shops. I expect to be long dead before that last MVS shop shuts down. 

Peter Van Epp / van...@sfu.ca

Newsgroups: comp.arch.storage
Path: sparky!uunet!haven.umd.edu!darwin.sura.net!uvaarpa!murdoch!
fermi.clas.Virginia.EDU!gl8f
From: gl...@fermi.clas.Virginia.EDU (Greg Lindahl)
Subject: Re: Mainframe as high transaction rate database engine (longish)
Message-ID: <1992May7.165119.15764@murdoch.acc.Virginia.EDU>
Keywords: TPF Mainframes
Sender: use...@murdoch.acc.Virginia.EDU
Organization: Department of Astronomy, University of Virginia
References: <vanepp.704949857@sfu.ca> <vanepp.705127031@sfu.ca>
Date: Thu, 7 May 1992 16:51:19 GMT
Lines: 51

In this article, I will probably show a near-complete lack of facts.
Accept this as hand-waving.

In article <vanepp.7...@sfu.ca> van...@fraser.sfu.ca (Peter Van Epp) writes:

>Here are the general specs (again from memory after 4 or 5 years)
>
>CPU (main and backup) Amdahl 5860 (11 MIPS?) IBM3081KX (17 mips?)
> 	(TPF could only use one of the 2 processors on the 3081)
>16 channels in use in both cases (32 and 64 on the machine as I recall for
>	VM and/or MVS)
>   12 channels to 12 disk controllers to 160 Amdahl 6880 disks running in 
>			3350 mode, (~500 megs/disk around 300 megs in use
>			due to seek time limitations).
[...]
>This configuration supported 80 transactions per second with a response
>time guarantee of 90% of the transactions completing in under 3 seconds 
>(including round trip comms times on 2400 to 9600 baud lease lines).

That's a total of 48 gigabytes of disk storage. My guess is that this
whole setup is limited by the seek time of the disks; it's easy to buy
a CPU that's 3 times faster, but you can't get disks that have 1/3 the
seek time.

The total transfer rate through the channels seems to be fairly small.
80 transactions per second -- how much I/O per transaction? Well,
let's assume we can buy much more RAM than 16 megabytes these days. If
so, we might be able to get disk I/O down to 10kbyte per transaction,
or under a megabyte per second. An older mainframe channel runs at
around 3 megabytes per second. So they're using only 1/36 of their
possible transfer rate. Again, seek times on their disks is killing
them.

If you were building such a system from scratch today, you might be
tempted to buy 48 gigabytes of ECC ramdisk (!), since that should cost
you around $50/megabyte, or a mere $2.4 million. Back it up with
$100,000 of cheap SCSI disks, and log transactions to cheap SCSI
disks. Relatively inexpensive RISC CPU's could provide sufficient CPU
horsepower, and since the total transfer rate is low, the poor design
of most RISC memory subsystems for this kind of transfer rate would
not cause a problem. The system ought to have a more consistant
response since there is no seeking. It should also scale up fairly
easily.

Of course, this is mostly a pipe-dream, since I don't have very many
details about the original system, and since there are probably
aspects about the setup I proposed that would cause problems. However,
it is interesting to note that the rapidly-falling price of dram can
provide a revolution in transaction processing -- what happens when
dram is $5/megabyte? Sure, disks will be even cheaper, but seek times will
still be large.

Newsgroups: comp.arch.storage
Path: sparky!uunet!van-bc!ubc-cs!newsserver.sfu.ca!sfu.ca!vanepp
From: van...@fraser.sfu.ca (Peter Van Epp)
Subject: Re: Mainframe as high transaction rate database engine (longish)
Message-ID: <vanepp.705456278@sfu.ca>
Keywords: TPF Mainframes
Sender: ne...@sfu.ca
Organization: Simon Fraser University, Burnaby, B.C., Canada
References: <vanepp.704949857@sfu.ca> <vanepp.705127031@sfu.ca> 
<1992May7.165119.15764@murdoch.acc.Virginia.EDU>
Date: Sun, 10 May 1992 00:04:38 GMT
Lines: 125

gl...@fermi.clas.Virginia.EDU (Greg Lindahl) writes:

>In this article, I will probably show a near-complete lack of facts.
>Accept this as hand-waving.

>In article <vanepp.7...@sfu.ca> van...@fraser.sfu.ca (Peter Van Epp) writes:

>[...]

>That's a total of 48 gigabytes of disk storage. My guess is that this
>whole setup is limited by the seek time of the disks; it's easy to buy
>a CPU that's 3 times faster, but you can't get disks that have 1/3 the
>seek time.

I should point out that this is only 24 gigs of unique storage, since each
disk is mirrored to another disk down a separate channel and disk controller
(no single point of failure), and a write from the cache goes to both disks,
a read to cache comes from which ever disk has the least queue to increase
performance slightly.
	You are also correct that this is seek limited, that is why only 
2/3 of the disk is usable. As I recall 3380s are worse for this than 3350s
as the 3380s tended to have many shortish tracks compared to a 3350 and 
you could only use about 20% of the disk. While you can't get shorter seek
time disks, you can spread the data over more quite quick (even by todays
standards) disks. 3350s have a published seek time up in the 16 msec region
and for limited distances would do in the 10 to 12 range (remembering this is
a disk that has been obsoleted by the 3380 for 10 years or so!), as I recall
the 6880's were down in the 10 to 12 msec range published and over short 
distances could get down to the 6 to 8 msec range (these being 3380A models
which are now also obsolete)

>The total transfer rate through the channels seems to be fairly small.
>80 transactions per second -- how much I/O per transaction? Well,

As I recall 7 I/Os per transaction but I don't remember if this was physical
to the disk, (it likely was) or I/O to the cache. 

>let's assume we can buy much more RAM than 16 megabytes these days. If
>so, we might be able to get disk I/O down to 10kbyte per transaction,

You could then to, as I said there was more ram than we could use on each
machine, the problem at that point was (and likely still is) the architecture.
Since ACP (the predecessor of TPF) started out on 360 machines, and all the
application code is written in assembler, neither XA mode (more than 24 bits
of address) nor DAT (virtual memory) was usable by TPF. At that point, the 
latest version I know of had go XA and current ones are probably ESA, but
application code had to be copied below the 16 meg line to run (unless it 
was a new or rewritten application, taking a performace hit).
	The other point of interest here, is as I pointed out, this is a 
small to medium sized airline, the largest carriers had an order of magnitude
large transaction rate (80 transactions /sec -> ~2000 transactions a second) 
running the same applications and presumably the same number of I/Os per
transaction. The TPF machine was typically the smallest of the machines in 
house, the MVS/IMS machine was typically 4 times as large/fast for around
(I think!) 10 transactions a second on IMS ( a relational database running
under MVS).


>or under a megabyte per second. An older mainframe channel runs at
>around 3 megabytes per second. So they're using only 1/36 of their
>possible transfer rate. Again, seek times on their disks is killing
>them.

Correct as far as the disk I/O channel goes. I think the actual transfer 
rate is closer to 600k /sec, reponse time due (to I/O queuing) starts to
die when the channel is around 30% busy (ie. your 1 Mbyte/ sec) and we
were running between 15% and 20% busy. However, this 15 to 20% is per
channel, and the I/O is very evenly spread across the channels so (unlike
an MVS or VM system) all 12 of those channels were busy at the same time
(during a peak period, ie. day time) of course. Now our 600k I/O rate
on a single channel has climbed to a total into main memory I/O rate
of 7 megs/sec (and this excludes the I/O on the byte mux channels and 
Tape channels that are adding another couple of megs per second to the
I/O rate, still not a killer, but up there. 
	Now consider this, assume a server with a 17 meg/second VME bus
for I/O, and a fast internal CPU and a typical Dram SIMM main memory,
I would suggest that you are going to start seeing instruction starvation
on the CPU as its cache fights for main memory bandwidth with the I/O 
subsystem. I will point out that this is how the situation seems to me,
and that none of the Unix vendors we talked to could give me a convincing
reason as to why I might be wrong (this being in the context of an NFS
file server), with the exception of Auspex (which is as I have said before,
is why we are a happy Auspex customer). At this point, having experience with
Auspex, Unix I/O and backup issues, and NFS, given the decsion to make again
I'd recommend the Auspex even more strongly for reasons other than the 
apparant I/O performance (although that is still a good reason in itself
in my mind!). 
	I however am no more sure now, than when I talked to the various unix 
vendors, that my insistance that there could be an I/O problem using a standard
Unix server as an NFS server, where the single CPU and I/O bus are fielding all
the ethernet, and disk interrupts and request processing all over a 
(for instance) 17 meg/ second VME bus was going to fly. We could obviously do 
it on a set of such severs, but that isn't what was being quoted (presumably 
for cost reasons). This being the case, I would welcome some more discussion
along the lines "you ignorant mainframe bigot, of course it would work and 
here's why" (I got a lot of the first, but nobody ever came up with what I 
thought were reasonable numbers and a working site, for the second!) or 
responses of the form "hey he may just be right, what can we do about this?
(one assumes that telling your customer that your NFS server won't do the
job, that they better buy an Auspex wouldn't go over so well ...) As I say,
our decsion is made, and I for one am happy with it, but there are other
mainframe shops that are considering a move (sometimes, like us, not
voluntarily!) to Unix who would like to know that answer. We thought that 
we were going down a well trodden path from a mainframe to Unix, but either
we didn't ask the right people (and the vendors wouldn't admit to knowing the 
right people or we were talking to the wrong vendors which is very possible!),
or we are doing something quite strange, because answers to these types of
questions seem hard to come by.

>If you were building such a system from scratch today, you might be
>tempted to buy 48 gigabytes of ECC ramdisk (!), since that should cost
>you around $50/megabyte, or a mere $2.4 million. Back it up with
>$100,000 of cheap SCSI disks, and log transactions to cheap SCSI

The killer here is the "from scratch today", and I agree, something along
these lines would work except for the installed base of non portable 
applications, and the cost and risk of making such a change (since you 
really are "betting the business" on the change). The real killer (even
for IBM) is that this is a small market, I would guess that you are not
talking more than around 200 sites total, and the return is just not 
worth it when there is a solution (even though an expensive on) in 
place. 

Peter Van Epp / van...@sfu.ca

			        About USENET

USENET (Users’ Network) was a bulletin board shared among many computer
systems around the world. USENET was a logical network, sitting on top
of several physical networks, among them UUCP, BLICN, BERKNET, X.25, and
the ARPANET. Sites on USENET included many universities, private companies
and research organizations. See USENET Archives.

		       SCO Files Lawsuit Against IBM

March 7, 2003 - The SCO Group filed legal action against IBM in the State 
Court of Utah for trade secrets misappropriation, tortious interference, 
unfair competition and breach of contract. The complaint alleges that IBM 
made concentrated efforts to improperly destroy the economic value of 
UNIX, particularly UNIX on Intel, to benefit IBM's Linux services 
business. See SCO v IBM.

The materials and information included in this website may only be used
for purposes such as criticism, review, private study, scholarship, or
research.

Electronic mail:			       WorldWideWeb:
   tech-insider@outlook.com			  http://tech-insider.org/