Total/db User's Guide

Simple Kernel Debugger - SKDB

Overview

The Simple Kernel Debugger (SKDB) is a machine-level symbolic debugger. This chapter provides an overview of SKDB, instructions on how to install and remove it, and how to start it after a kernel crash, and also lists details of SKDB commands.

SKDB is designed to support debugging of LynxOS kernel internals, primarily device drivers. It allows you to perform the following operations interactively in LynxOS kernel space:

• Setting breakpoints

• Examining memory and registers

• Changing memory contents

• Displaying kernel data structures.

In addition, you can use SKDB to determine the cause of a kernel crash or a kernel panic. LynxOS GDB uses SKDB as the target agent for kernel/device driver debugging.

To use SKDB, the user needs a solid understanding of LynxOS internals, including its memory model, scheduling, interrupt handling, and so forth. SKDB is not designed for user process debugging; use GDB for user process debugging instead.

SKDB has the following characteristics by design:

• SKDB is a tool that you use to debug new device drivers and similar components after the kernel has started and is running fairly stably. It is not intended to be used for porting the LynxOS kernel to a new platform; although SKDB may be still useful for LynxOS kernel porting, it requires a fairly stable kernel and it is not effective in the early stages of the kernel's start-up until the kernel internally installs and initializes SKDB.

• While SKDB is in operation (at its prompt) the entire operating system is paused and no kernel services are available.

Installing/Removing SKDB

In order to use SKDB, install the SKDB module into the kernel. It is possible install and/or remove SKDB support after initial installation of LynxOS.

Installing SKDB

To install, execute the following:

/usr/bin/Install.skdb

The installation script will prompt to choose the default SKDB port for SKDB to use when it starts automatically in case of a kernel crash or a kernel panic. SKDB uses this port to break in to SKDB with the hot key. The video console, if one exists, always accepts break-ins with the hot key. For more information about starting SKDB automatically, see "Using SKDB".

After installation, reboot the system to make SKDB effective.

Removing SKDB

To remove SKDB, execute the following:

/usr/bin/Uninstall.skdb

Reboot the target system after removal of SKDB.

Using SKDB

SKDB Prompt

Whenever the operating system is in SKDB, it shows an asterisk (*) as its prompt. The entire operating system is paused and no kernel services are available while in SKDB.

Starting SKDB Automatically after a Kernel Crash or Panic

If installed, SKDB is automatically started by a kernel crash, such as kernel memory access fault or a panic situation. In this case, the kernel is usually unable to resume operation, but it is possible to determine the cause and the location of the kernel fault or the panic. For example, the following commands may be useful for analyzing the cause:

• p - process/thread table display

• t - stack trace display

• r - register contents display

• m - memory contents display.

Breaking into SKDB with Hot Key

Once SKDB has been installed, it can invoked by pressing ShiftCtrlMinus (the "hot key") on the keyboard of an SKDB-ready port while the operating system is up and running. Some keyboards (mostly video consoles) may use CtrlMinus instead.

Note: To break into SKDB with a hot key from a serial port, the port must have been explicitly opened by a process. Having a login process or a dummy process such as cat running on the port will suffice for this.

The default hot key combination for SKDB can be changed by using the z command within SKDB.

1. At the SKDB prompt, enter the following command:

* z

SKDB will prompt for a new key combination.

2. Press the desired key combination.

SKDB will prompt for the same key combination for confirmation.

3. Press the same key combination again.

4. To cancel the change, press something else.

The hot key combination is set per port, therefore different key combinations can be set for different ports.

The new hot key combination is not preserved across operating system reboots; it returns to the default combination after each reboot and it needs to be again.

Note: To use SKDB for remote kernel debugging with GDB or TotalView, do not change the hot key combination on the serial port. These debuggers have the hot key combination hardcoded.

Kernel Status Display

At each invocation, SKBD prints a line like the following:

DELOK:pid.tid@trapcode, slevel, econtext, PSW, PC checksum

DEL The ASCII Delete character (usually invisible)

OK: The literal string

pid Current process ID

tid Current thread ID

trapcode Trap code; trap code is the same as the architecture's exception code with the addition of -1 for invocation from keyboard and -2 for a panic situation

slevel Last slevel; slevel is the kernel preemption level (0: user, 1: kernel, 2: no context switching, 3: no interrupts)

econtext Econtext address; econtext is the per-thread register stack into which the kernel saves registers

PSW Processor status word (PSW); called flags or status register in some architectures

PC Last program counter

checksum String checksum.

For example, breaking into SKDB with the "hot key" would display something similar to the following:

OK:0.0@-1,1,DB0AB19C,000199A0,00000207 C961

The above is interpreted that the operating system was running the null process code (process 0) at address 0x207 with context switching enabled when the break-in occurred.

Kernel Status Redisplay

To redisplay the above information, press Ctrl-B then ? and Return. This option is currently available on serial terminals only, not on video consoles.

Stack Trace Display

The t command displays a traceback or the "history" of nested function calls of a thread within the kernel. One can determine the "path" to the current breakpoint, panic location, or kernel fault location where the kernel entered SKDB. Tracing stops as soon as the stack frame appears to be out of the valid kernel address range.

Give the t command the process ID to determine the process's main thread, or the thread ID with a preceding - (minus sign). The default is the current thread.

Verbose Trace Mode

Turning on the verbose trace mode with the v command makes the
t command display the contents of each stack frame as well as the offset values from the frame pointer (or the stack pointer in the case of the PowerPC).

Process, Thread, and Other Displays

The p command displays the contents of the kernel's process table and thread table. The s command with options displays the contents of a variety of the kernel's internal data structures.

Resuming the Kernel

To exit SKDB and resume the operating system, press the Esc key; the kernel will continue running until the following occurs:

• Hitting a kernel breakpoint

• Being interrupted by a hot key

• Getting a kernel crash or panic.

As discussed, it may not be possible to resume the kernel if the kernel was in SKDB due to a kernel crash or a kernel panic.

Setting Breakpoints

SKDB can set up to 10 breakpoints in the kernel including device drivers. When the CPU reaches the instruction at a breakpoint, the control is trapped into SKDB. The breakpoints remain set until explicitly unset by the
u command.

SKDB may refuse to set a breakpoint on some instructions that are critical to its operation. These instructions include those handling the processor status word (PSW) register.

Caution! Do not set a breakpoint in the user process space from SKDB. Such a breakpoint will not be recognized by SKDB and thus will cause an unexpected termination of the user process.

Single-Stepping

Pressing x and Return single-steps the current thread (the thread that caused to enter SKDB.) It is not, however, possible to single-step the following:

• The thread "broken-in" with the hot key

• Some machine instructions that are critical for SKDB's operation (generally those handling the processor status word (PSW) register)

• A crashed or panicked kernel.

  Caution! It is not recommended to attempt to trace (setting breakpoints in and/or single-stepping) the "core" portions of the LynxOS kernel, such as those handling context switching and interrupt control, because such an attempt may severely interfere with the LynxOS kernel operation. Also, the instructions that manipulate the processor's status register cannot be safely single-stepped. Though SKDB detects, warns about, and prevents such an attempt, casual tracing of such critical code may result in an unexpected system freeze.

Disassembly

The d command disassembles 10 instructions from the specified address or the current address. The current address is updated to the next text location after each disassembly. The current address is also updated to the breakpoint or the fault location whenever the kernel stops and reenters SKDB.

Note: The current x86 version of SKDB uses the Intel-style syntax of disassembly, not the GNU (AT&T) style.

Setting Watchpoints

Some CPU architectures support hardware debug registers to implement watchpoints. The following LynxOS ports support SKDB watchpoints:

• x86 - up to 4 watchpoints

• PowerPC - up to 1 watchpoint

  Note: The availability of watchpoint operation for the PowerPC depends on the type of the CPU.

The B command can set as many watchpoints as the target CPU architecture allows. The B command takes two mandatory arguments-the watchpoint number and the watchpoint location address-plus the following optional arguments:

• Access mode - r for read access and w for write access; the default is w

• Ignore PC addresses - up to 10 text addresses after "!" for the program counter to be ignored for watchpoints hit. This is useful to avoid stopping at know kernel locations where the watchpoint is accessed.

For example, if one wants to catch all write accesses to currtptr but does not want to stop at resched+0x24 which is considered a normal access:

* B 1 currtptr w ! resched+0x24

Caution! "Ignore PC addresses" is implemented by software: all accesses to a watchpoint memory location actually cause CPU exception handling that is captured by SKDB. SKDB examines the cause of the exception and the program counter value to determine whether to resume the kernel silently or to stop and report the hit to the user. This may result in a significant speed penalty if the watchpoint is frequently accessed but ignored. SKDB uses the virtual address for setting the debug register. Depending on the CPU (MMU) architecture, this may result in watchpoint misses if the page is aliased (mapped at different address locations) and the alias addresses are accessed.

To remove a watchpoint, use the U command with the watchpoint number.

SKDB Commands

SKDB Commands 

Command Format	Example	Description
Examine Memory
hex-addr [size]	* 0xdb100000	Examines (displays) the 4 or size bytes at the location of addr
$sym1 [size]	* $currpid 10	Examines the 4 or size bytes at the location of sym
m addr [size]	* m currpid 256	Examines size or 64 bytes at addr
T addr [pid]	* T 0xdb100000 9	Translates virtual address addr to physical using pid or current process' mapping
+ [size]	* +	Examines the next 32 or size bytes of memory
- [size]	* - 10	Examines the last 32 or size bytes of memory
Change Memory
c addr data	* c 0xdb100000 0x200	Stores data as a long word (4 bytes) at addr
Find Symbol
f addr	* f 0xdb100000	Displays closest symbol with offset to addr
&sym	* &currpid	Displays the address of sym
Display Data Structure
s st [addr2|tid]	* s st 5	Displays contents of st_entry structure for thread ID tid, address addr, or current thread
s proc [addr2|pid]	* s proc	Displays contents of pentry structure at address addr, for process ID pid, or for the current process
s pss [addr2|pid]	* s pss 0xdb100000	Displays contents of pssentry structure at address addr, for process ID pid, or for the current process
s inode {addr2|num}	* s inode 45	Displays contents of inode_entry structure at address addr or at index num
s block {addr2|num}	* s block 0xdb100000	Displays contents of buf_entry structure at address addr or at index num
s ihead {addr2|num}	* s ihead 30	Displays contents of ihead_entry structure at address addr or at index num
s file {addr2|num}	* s file 0xdb100000	Displays contents of file structure at address addr or at index num
s fifo {addr2|num}	* s fifo 49	Displays contents of fifo structure at address addr or at index num
s cdev {addr2|num}	* s cdev 0xdb100000	Displays contents of cdevsw_entry structure at address addr or at index num
s bdev {addr2|num}	* s bdev 0	Displays contents of bdevsw_entry structure at address addr or at index num
s fdentry addr	* s fdentry 0xdb100000	Displays contents of fdentry structure at address addr
s {+|-}	* s +	Displays contents of the next or the last (in memory) data structure of the type being displayed
s {next|prev}	* s next	Displays contents of the data structure pointed to by the next or prev (or equivalent) field of the currently-displayed data structure
Stack Trace
t [pid|-tid]	* t -5	Symbolic stack trace of process pid, thread tid, or the current thread
v	* v	Toggles verbose mode for trace
Display Registers, Processes & Set Priority
r [pid|-tid]	* r	Displays CPU registers of process pid's main thread, thread tid, or the current thread3
p [count]	* p 20	Displays process table (all or count lines worth)
P prio tid	* P 15 8	Changes priority of thread tid to prio divided by 2
Breakpoints
b	* b	Shows all breakpoints set
b num addr	* b 1 0xdb100000	Sets breakpoint num at the memory location addr
u num	* u 5	Unsets breakpoint num
Watchpoints
B	* B	Shows all watchpoints set
B num addr [r|w|rw] [! iaddr ...]	* B 1 currtptr w	Sets watchpoint num at the memory location addr for read, write or read/write accesses but ignores accesses by the instruction at iaddr
U num	* U 5	Unsets watchpoint num
Single-Stepping
x	* x	Single-steps current thread
Disassembly
d [addr]	* d resched+10	Disassembles at addr or the current PC
Miscellaneous
R	* R	Restarts the operating system
h	* h	Displays a description of all available commands
?	* * ?	Same as h
PowerPC Specific
S	* S	Sees segment register

1 On the PowerPC, a text symbol is preceded by a "." (dot). A symbol without the preceding dot refers to the corresponding TOC entry. 2 The address value must point to a valid table entry. 3 Some architecture may not save all registers upon context switching.

General Notes

Parameter Validation

SKDB performs little validation for command arguments. Although SKDB catches most memory access faults resulting from SKDB commands, improper arguments may result in a system freeze.

Symbol Information

SKDB uses the kernel symbol table that is loaded at the startup time for symbol lookup. SKDB cannt do interactive symbolic debugging with a stripped kernel.

Address Expressions

SKDB accepts simple address expressions with symbolic notations for most commands that accept memory address parameters. The syntax is as follows:

• Number - hexadecimal if starting with "0x"; octal if starting with "0"; or otherwise decimal

• Symbol - the symbol's absolute virtual address value (note the PowerPC re quires a preceding dot for text symbols)

• Register - the CPU register of the current thread. The following mnemonics work as common aliases for all architectures: %pc, %fp, %sp. Other register mnemonics depend on the CPU architecture

• Operator - + and - represent addition and subtraction respectively. Operations are performed left to right without precedence or associatively.

For example, the following sets a breakpoint at the current PC address plus 20 bytes:

* b 1 %pc+0x14

Default Virtual Address Space

The LynxOS memory model assigns a separate virtual address space to each process (kernel threads belong to process 0 <zero>). Although all processes share the same kernel text, kernel data, and kernel heap in the kernel, each supervisor stack still belongs to its respective process's virtual address space. To access a memory location of a non-current process, use the T command to get the memory location's PHYSBASE address.

The PHYSBASE address is the region of kernel address space where a mirror image of the system's physical memory is mapped (aliased). Since any page that the kernel may access is found in this region and the page is visible to all processes at the same virtual address, SKDB uses PHYSBASE for quick memory reference in a non-current process's virtual address space.

Remote Debugger Interface Protocol

SKDB supports a communication protocol for interfacing with a remote kernel debugger such as LynxOS GDB. For more information on how to debug the LynxOS kernel at the source level, see LynxOS GDB Enhancements, See "LynxOS GDB Enhancements"

LynuxWorks, Inc. 855 Branham Lane East San Jose, CA 95138 http://www.lynuxworks.com 1.800.255.5969