From: Matt Welsh < mdw@TC.Cornell.EDU>
Subject: New FAQ on kernel panics
Date: Sat, 16 Jan 1993 00:28:40 +0200
Linus, can we get a Q/A in the next FAQ about how to track down kernel
panic messages with nm? I see a lot of these, and if people would be so
kind as to tell us where the kernel is panicing (instead of just giving us
the panic message) it would help a lot. Basically, a Q/A describing the
contents of a standard panic message, what it means, etc. would be great.
Thanks.
mdw
From: torvalds@cs.Helsinki.FI (Linus Torvalds)
Subject: Re: New FAQ on kernel panics
Date: Sat, 16 Jan 1993 02:19:37 +0200
Matt Welsh: "New FAQ on kernel panics" (Jan 15, 17:28):
>
> Linus, can we get a Q/A in the next FAQ about how to track down kernel
> panic messages with nm? I see a lot of these, and if people would be so
> kind as to tell us where the kernel is panicing (instead of just giving us
> the panic message) it would help a lot. Basically, a Q/A describing the
> contents of a standard panic message, what it means, etc. would be great.
Ok. Hope somebody can make a FAQ out of this...
The panic message essentially consists of:
- possible debugging messages printed out by the code before the panic:
these may be important to tell more closely why the kernel decided to
panic.
- one line of "Kernel panic: " and a small reason string (eg "unable to
mount root")
- if the panic happened while running the swapper task (aka idle task,
aka dummy task), you finally get a line that tells you that the
kernel is unable to sync any devices ("In swapper task - not
syncing"). This generally means that the panic happened in a
interrupt handler, as the idle task should never really panic on it's
own.
After a kernel panic, the machine is essentially dead: keyboard
interrupts may still be working, so that you can switch VC's and press
ctrl-alt-del, but no tasks are running.
More interesting than the actual panic message is usually the debugging
messages prior to a panic. The debugging messages can happen without
the panic, but most debug messages that remain are pretty severe, so a
panic may be likely. The most interesting debugging messages have the
form:
- possible extended explanation (eg "unable to handle kernel paging
request at address xxxxx").
- one line of reason + possible error code. This can look like
"General protection fault: 0000" or similar: it tells which exception
happened, and gives the error code. The error code is mostly zero,
but can sometimes be non-zero, which usually makes them more
interesting.
- the place the error was reported, in the form "EIP: 0008:xxxxxxxx".
This is important: it should be used to later check up in which
kernel routine the error happened. The 0008 tells that it happened
in the kernel code segment (it can be something else, but it probably
shouldn't happen), and the "xxxxxxxx" is the offset of the offending
instruction,
- the value of the 'fs' segment at the time of the exception: this is
usually 0017, and isn't really interesting any more (it's a leftover
from much earlier debugging sessions).
- the base and limit of the current code segment. These too are mostly
leftovers from older kernel versions: in the current kernels these
are unlikely to have anything important in them (but do report them
anyway for completeness).
- the pid of the current process and the value of the task register at
this point. Not generally of any importance.
- ten hexadecimal values representing the offending instruction. These
can be used to hand-disassemble what the offending instruction was,
and sometimes helps pinpoint it a bit more easily than just telling
where it happened. This is useful.
When doing a panic report (or a report of just a "normal" kernel error
without an actual panic), the thing to do is:
(a) write down the above debugging info exactly. Especially the EIP
and instruction hex-dump values are important, and need to be
correct for any kind of debugging.
(b) find out where the exception happened. With earlier kernels (0.12
and below), the address was generally enough for me: all the
kernels were generally the same, and I could look at my kernel
binary to find out where the error occurred. With newer kernels
that is no longer possible, so the person who reports the error
will have to pinpoint it a bit closer with respect to his
particular kernel version.
There are several ways to find out where the error happened, but the
simplest one is generally the following:
- get the kernel namelist with 'nm' and sort it according to address.
This is most easily done with the commands
# nm /usr/src/linux/tools/system | sort > namelist
where you have to make sure that the tools/system file actually
corresponds with the kernel that paniced.
- search for the place that seems to contain the offending
instructions. 'grep' is not really an option, as the exact address
is unlikely to be in the output of 'nm', so you'll have to eyeball
it. This is easy enough in a editor or using 'less'.
- send along about 10 lines of the nm output from around the offending
instruction. Assuming the EIP value reported by the panic was
00012345, the output of nm that is interesting might look like this:
00011fd4 T _sys_ssetmask
00011ff4 T _sys_sigpending
00012024 T _sys_sigsuspend
00012084 T _sys_signal
00012114 T _sys_sigaction
00012204 T _do_signal
000124ac T _kernel_mktime
000124ac t gcc2_compiled.
000124ac t mktime.o
00012560 t _get_long
00012560 t gcc2_compiled.
where the 00012345 address is in the _do_signal() function that seems
to extend from 00012204 to 000124ac. Note the "seems" - I prefer to
have a couple of lines of context around the offending place as that
can help pinpoint it a bit more: there may be static functions in the
kernel between the two addresses that won't show up in the namelist
or similar. Also, sending a couple of lines of context means that
bogus lines can safely be ignored (things like the "gcc2_compiled"
and "mktime.o" in the example). But don't try to prune out the bogus
lines yourself unless you know that you know what you are doing.
So, the result of it all? A bug-report with only the register dumps and
no other info is generally pretty useless - although if it also tells
what was going on that resulted in the error the bug might still be
possible to find. Together with a pinpoint where it happened, it's
generally much easier then to find exactly what went wrong, and fix it.
There are some circumstances where even all the above information won't
help: under some circumstances (a kernel jump to a nonexistent address
etc), the debugging info is simply bogus and not enough. So always try
to make the bugreport as complete as possible: if you can re-create the
error so that somebody else also can test it, please include that kind
of info ("if I do this, then that, then the kernel will crash with this
error").
Linus
From: eric@tantalus.nrl.navy.mil (Eric Youngdale)
Subject: New FAQ on kernel panics
Date: Sat, 16 Jan 1993 07:59:32 +0200
> - send along about 10 lines of the nm output from around the offending
> instruction. Assuming the EIP value reported by the panic was
> 00012345, the output of nm that is interesting might look like this:
>
> 00011fd4 T _sys_ssetmask
> 00011ff4 T _sys_sigpending
> 00012024 T _sys_sigsuspend
> 00012084 T _sys_signal
> 00012114 T _sys_sigaction
> 00012204 T _do_signal
Here is an item that is probably only of interest to kernel hackers.
The beta version of gas, 1.93, is capable of generating listing files. It will
print out the offsets of each instruction, the hexadecimal bytes that each
instruction is translated to, and the source code that was responsible for
the assembly code. I am enclosing a short example at the end of this message.
To generate the listings, you need to use the '-a' switch, and the
listing goes to stdout.
-Eric
GAS LISTING seagate.s page 50
1556 00000100
1556 0000
857:seagate.c **** printk("disconnecting..");
1557 .stabd 68,0,857
1558 0f42 68790600 pushl $LC25
1558 00
1559 0f47 E8B4F0FF call _printk
1559 FF
1560 0f4c 83C404 addl $4,%esp
858:seagate.c **** if(!SCint) panic("SCint == NULL after REQ_MSGIN1");
1561 .stabd 68,0,858
1562 0f4f 833D5612 cmpl $0,_SCint
1562 000000
1563 0f56 7510 jne L110
1564 0f58 68890600 pushl $LC26
1564 00
1565 0f5d E89EF0FF call _panic
1565 FF
1566 0f62 83C404 addl $4,%esp
1567 0f65 909090 .align 4,0x90
1568 L110:
859:seagate.c **** current_data = data; /* WDE add */
1569 .stabd 68,0,859
1570 0f68 8B4DF8 movl -8(%ebp),%ecx
1571 0f6b 890D7C12 movl %ecx,_current_data
1571 0000
860:seagate.c **** current_bufflen = len; /* WDE add */
1572 .stabd 68,0,860
1573 0f71 8B55FC movl -4(%ebp),%edx
1574 0f74 89158012 movl %edx,_current_bufflen
1574 0000
861:seagate.c **** #if (DEBUG & (PHASE_RESELECT | PHASE_MSGIN))
862:seagate.c **** printk("scsi%d : disconnected.\n", hostno);
863:seagate.c **** #endif
864:seagate.c **** done=1;
1575 .stabd 68,0,864
1576 0f7a C745EC01 movl $1,-20(%ebp)
1576 000000
865:seagate.c **** break;
1577 .stabd 68,0,865
1578 0f81 E9CA0000 jmp L108
1578 00
1579 0f86 9090 .align 4,0x90
1580 L111:
866:seagate.c **** case COMMAND_COMPLETE :
867:seagate.c **** #if (DEBUG & PHASE_MSGIN)
868:seagate.c **** printk("scsi%d : command complete.\n", hostno);
869:seagate.c **** #endif
870:seagate.c **** if(!SCint) panic("SCint == NULL after REQ_MSGIN2");
1581 .stabd 68,0,870
1582 0f88 833D5612 cmpl $0,_SCint
1582 000000
1583 0f8f 750F jne L112
1584 0f91 68A80600 pushl $LC27
1584 00
1585 0f96 E865F0FF call _panic
1585 FF