Chapter 3 LynxOS GDB Enhancements

LynxOS GDB extends and enhances the functionality of the GNU debugger for debugging various LynxOS targets. This chapter is intended to supplement the previous chapter, See "Debugging with GDB" Readers are advised to read the chapter prior to this in order to familiarize themselves with GDB.

Overview

LynxOS GDB supports debugging of a variety of LynxOS targets including, but not limited to, the following areas:

Debugging POSIX Threads

LynxOS user threads fully conform to the POSIX/IEEE 1003.1c threads model. A LynxOS process consists of one or more threads each of which is scheduled by the kernel. LynxOS GDB provides full support of multiple thread debugging including


Note: GDB can debug a single process per debug session. To debug more than one process, start as many GDB sessions as the number of the processes. Each GDB session works independently from the others and there is no mutual synchronization mechanism available between GDB sessions.

Understanding Thread Numbers

Each debugged process may contain any number of threads. GDB manages these threads with internal thread numbers that are unique within the process, and within the GDB session. Note that these thread numbers are different from LynxOS thread IDs that are assigned by the LynxOS kernel and are unique throughout the operating system. GDB maintains the mapping between its thread numbers and LynxOS thread IDs.

Browsing and Switching Threads

The first column of a thread list is the GDB thread number. The asterisk indicates the current thread. The second and third numbers (8, 38, and 32 in this example) are the LynxOS process ID and thread IDs, respectively.

Current Thread

Whenever GDB stops and returns to its prompt, it maintains its concept of the current thread. GDB can only focus on one thread at a time, which is referred to as the current thread. By default, any GDB command uses the current thread if it implicitly uses threadspecific parameters such as

At the start-up of the debugged process, the initial thread is the current thread. When a thread hits a breakpoint or watchpoint, that thread becomes the current thread. When the target process is interrupted by GDB, then the interrupted thread becomes the current thread.

Use the thread command with the new thread number to switch the focus from one thread to another:

Setting a Breakpoint

You can make a breakpoint common to all the threads, so that any thread in the target process will stop at a hit on it, or specific to a particular thread, so that only the specified thread will stop. The default is any thread.

To set a thread-specific breakpoint, use the break command with the thread modifier:

When a thread stops because of a breakpoint or any other reason, all threads in the target process will stop immediately, not just the thread that encountered the stop condition. Likewise, when a thread is resumed, all threads in the debug process are resumed. Whenever GBD is at its prompt, the entire process is stopped and all variables can be determined statically.


Note: Thread-specific breakpoints are implemented via simulation: All breakpoints are actually threadinsensitive to the operating system. When GDB detects a hit at a thread-specific breakpoint by a thread that is not specified for the breakpoint, GDB immediately resumes the target without reporting it to the user. So, the operation is transparent, but some speed penalty may be seen if there are many uninteresting hits.

Resuming Threads

The continue command resumes the operation of the target process. If the target process has more than one thread, all the threads are resumed, not just the current thread.

Likewise, single-stepping actually resumes all the threads in the target process. The step and stepi commands merely guarantee that the current thread executes at most one line of code and one instruction respectively, while other threads with the same or higher priorities than the current thread may execute any amount of code before control returns to GDB. It is even possible that the current thread may not have a chance to complete single-stepping if some other thread runs first and hits a breakpoint or receives a signal.

To single-step the current thread, raise the thread's priority to a value higher than any other thread in the target process by using the setprio command at the shell level.

The thread's original priority must be restored after completing the exclusive single-stepping for normal operation of the application.


Note: When a multithreaded target process is resumed, the threads are actually resumed one after another by the LynxOS kernel, not all at once. The order of resumption and subsequent execution of threads is determined by the scheduling algorithm of the LynxOS kernel based on priorities.

Debugging Embedded Applications Remotely

Using the Target Command

Use the target command to choose the appropriate remote debug target and protocol (communication channel). In the following syntax, the first argument specifies the remote debug target and the second specifies the protocol.

Remote Debug Targets
Target	Target name(s)
Remote user process	remote and extended-remote
Kernel/device driver	skdb

Remote Debug Protocols
Protocol	Example
TCP/IP	foo:12345
Serial line	/dev/ttya
Remotely started gdbserver	foo:gdbserver
Proxy server	foo:sspp

The optional third argument args to the target command is only used by the remote gdbserver and proxy server protocols. Refer to the following sections for the details of args.

Debugging Remote Targets

Remote and Extended-Remote Targets

Both remote and extended-remote targets select debugging of a remote user process over a communication channel. There are small differences between remote and extended-remote targets in the way gdbserver handles termination of a debug session as shown in Table: Remote and Extended-Remote Targets.

remote and extended-remote targets support a full range of GDB features available in LynxOS GDB, including debugging of multithreaded processes, sending a signal to the target process, and attaching to a target process.

Device Driver/Kernel Target (skdb)

LynxOS GDB supports debugging of kernel code including device drivers over a serial line communication. For more information, see "Debugging Kernel/Device Drivers".

Supported Protocols for Remote and ExtendedRemote Targets

TCP Port

If there is TCP/IP communication available between the debugging host and the remote debugging target, it can be used to get the best possible debugging speed and reliability.

To start remote debugging through a TCP port, the remote target must first start gdbserver with an unused TCP port number that is available for normal use.

The first example starts the target program /test/prog with an argument arg. The second example does not start a target program immediately but waits until it is told by the host GDB to attach an already-running process. For more information, see "Debugging Kernel/Device Drivers".

In either case, the first argument to gdbserver is the TCP port specification in the form of host:port. An unused TCP/IP port number must be specified for gdb. This is usually a large number between 1,024 and 65,534 inclusive; a range of 5,000 through 65,534 is recommended for better compatibility. The host name string before : can be anything and is ignored by gdbserver.

At the host machine, give the host GDB prompt the same port number associated with the target's host name or IP address:

Using a Serial Line

A serial communication line can be used for GDB remote debugging with gdbserver provided that a serial port is available on both the host and target machines. When using a serial port, make sure that no other processes are using the port on either side. Neither GDB nor gdbserver uses a lock file, but they must gain exclusive access to the ports; otherwise, GDB may report a communication error.

To start gdbserver with the target process on the target's serial port (/dev/com1):

The following starts gdbserver for later process attaching on the target's serial port (/dev/com2). The -b option is used to specify the serial port's communication speed explicitly.

In either case, /dev/com1 and /dev/com2 are the device files of the targets' serial ports that are connected to the GDB hosts.

At the host machine, you must give the GDB prompt the host's serial port name (/dev/ttya in this case) to which the target is connected:

To change the host's serial line speed, use the set remotebaud command before using the target command:

Starting gdbserver Remotely

With TCP/IP communication, you can start gdbserver remotely in a couple of different ways:

The last method has an advantage that, for example, you can start a debug session completely from GDB, but it, as well as the rsh method, requires that the remote shell (rsh) and the .rhosts file for your account on the target be set up so that the GDB user can use remote shell commands on the target from the GDB host.

The first example starts gdbserver on the remote target with the target program /test/prog and an argument of arg. The second starts gdbserver for later attachment of a target process. The gdbserver string after the colon ":" is optional; if it is present, it must be in all lower-case and cannot be preceded by a path prefix.

In both of the above cases, the host GDB process spawns a local rsh process to ask the remote rshd process to start gdbserver with appropriate arguments. The communication TCP port number is automatically determined and you do not need to specify it. The standard output and standard error paths of the remote target process are redirected to those of the local GDB process, while the standard input path of the remote target process is either closed or redirected to the local host's /dev/null. Therefore, an interactive program cannot be debugged in this way.

Using a Proxy Server

The serial line between the GDB host and the target usually limits the physical distance between the two machines. It is, however, often desirable to be able to debug a target that has only a serial line for communication from a geographically distant locale. LynxOS GDB provides a proxy server solution to this problem.

The proxy server program runs on a third computer that is actually connected to the target via a serial line. The proxy computer and the GDB host computer communicate over TCP/IP and the proxy server program redirects all the messages to and from the serial line to the TCP/IP connection. This way, one can debug the target from a local workstation placed anywhere on the globe as long as there is a TCP/IP connection with the proxy server.

To enable remote debugging with a proxy, start gdbserver on the target with the serial port name connected to the proxy computer first:

Here, myproxy is the proxy server's host name, which could be a fully qualified domain name (FQDN) such as myproxy.foo.com or an IP address such as 198.4.254.47; and /dev/ttya is the serial port name of the proxy computer to which the target machine is connected.

This proxy server is also useful when the target's serial port is connected to a "serial terminal server" computer. The terminal server can be configured to run the proxy for shared access to the target. For more details on the proxy server, see "Proxy Server".

Starting the Remote Target

Starting from gdbserver

To start the remote target process from gdbserver, give the target program name with optional arguments at the gdbserver command line:

The first and second examples start gdbserver on the remote target then make a connection to the gdbserver process from GDB. The third example starts gdbserver remotely from GDB over a TCP/IP connection.

In any case, as soon as the communication is established, one will see something similar to the following:

(gdb) target remote mytarget: /test/prog arg
Process /test/proc created; pid = 17
Connected to 198.4.254.217...
Remote debugging using 198.4.254.217: /test/prog arg
Kernel supports MTD ptrace requests.
gdbserver: passed 0.0.0.0:3304 using addr:port=198.4.254.131:3304
0x10001000 in __start ()
(gdb)

At this point, the target process has started and is stopped at the very first user instruction of the program, which is in this case labeled as __start. To let the target process run, use the continue command (after setting breakpoints or other desired debugger commands), but do not use the run command which would start a local process.

Attaching to a Running Process

In a multiprocess application where the target process is forked by another process on the target, it is not possible for gdbserver to start the target process with the target program specification given in the GDB command line. In such a case, GDB can tell gdbserver to attach to an already-running process.

To start gdbserver for attaching to a process, give no target program specification to it:

To attach to a remote process, one needs to obtain the target process ID. To find out the process ID, execute the ps command on the target. With LynxOS GDB, you can run the ps command remotely from your GDB host with the rshell GDB command.

Then, use the GDB attach command with the process ID to attach the target process:

Ready to attach to a process
gdbserver: passed 0.0.0.0:1050 using addr:port=198.4.254.47:1050
Connected to mytarget...
Remote debugging using mytarget:

Attaching to remote program `/usr/home/joe/test/prog' (process 44)...
Kernel supports MTD ptrace requests.
Process 44 has threads 39.
0x100029d4 in _trap_ ()

To resume the target, use the continue command (after setting breakpoints, if necessary).

Target's Environment

Although GDB supports set environment to set the debug target's environment, currently this command has no effect on remote debugging. To set or reset the remote target process's environment, do so on the parent process (usually the shell) of gdbserver (if the application process is started by gdbserver) or the application process (if one attaches to the process later).

Postmortem Debugging of Dynamically Linked Programs

When loading a partial core file without either the data or the heap section created from a dynamically linked program, GDB is unable to debug the shared libraries used by the program. To obtain the shard libraries address for correct interpreting of the libraries symbolic information, GDB uses the DT_DEBUG symbol. The DT_DEBUG pointer maintained by the dynamic linker is located in the program data section of the core file and points to a special program heap area. This program heap area is allocated by the dynamic linker and holds the structures conatining the information on how the dynamic linker loads the libraries needed for the program execution. In the case when the core file is configured not to contain either the data or the program heap section, GDB is unable to get the load information of the library and analyze the functions located in the shared libraries.

Should debuging of the shared libraries in a core file be required, the user must configure the core file to include the data and heap sections. Refer to the LynxOS User's Guide for details.

Debugging Shared Libraries

A shared library is a collection of library functions that are commonly used by multiple application programs at the same time. Instead of linking a library to each application program executable, a single copy of the shared library is loaded into memory and used by multiple programs. This sharing reduces the use of physical memory as well as disk storage requirements; therefore, it is especially useful for embedded applications where memory is tight.

LynxOS GDB can set breakpoints in, single-step, and trace shared library code just like any application program. When a breakpoint is set in the text segment of a shared library that is actually used by multiple processes, a copy of the page where the breakpoint is being set is made for the debugged process so that the breakpoint does not interfere with other processes that share the same shared library.

Creating a Shared Library for Debugging Purposes

To create a shared library for GDB debugging purposes, compile (gcc) the library source file(s) with the -g option.

Loading Shared Library Symbol Information

GDB automatically loads the necessary symbol files (shared library files) when one of the following GDB commands is executed:

The following add-symbol-file command can be used to load a shared library symbol file. This may be necessary if the file cannot be found in one of the automatically searched directory paths, or if you are performing postmortem (core file) debugging:

The first argument for the add-symbol-file command is the additional shared library file path name on the debug host's file system.

Application-Loaded (dlopen'ed) Shared Libraries

GDB automatically loads the necessary symbol file (shared library file) when an application-loaded shared library file is first opened by the application (dlopen()) and discards the symbol file when the shared library file is last closed by the application (dlclose()) for live debugging. The dlopen and dlclose library functions use special signals to notify GDB that a shared library has been opened or closed.

GDB also searches for application-loaded shared libraries and loads the necessary symbol files when it attaches to a live target process or when it starts analyzing a core file (postmortem debugging).

The following add-symbol-file command can be used to load a shared library symbol file manually. This may be necessary if GDB cannot locate the file in any of the automatically searched directory paths:

The first argument for the add-symbol-file command is the additional shared library file path name on the debug host's file system.

If a symbol file has been loaded manually with add-symbol-file, it must be discarded manually with the delete-symbol-file command after the shared library has been last closed by the dlclose() library call. GDB will prompt the user when this seems to be necessary.

The delete-symbol-file command takes one argument for the deleted symbol file name on the debug host's file system. The info symbol-file command shows the list of currently loaded symbol files.

From To Symbol file
0x00400a20 0x00400e1f /usr/lynx/3.1.0/mips/tmp/hello/hello
0x70000c20 0x7000450f /usr/lynx/3.1.0/mips/lib/shlib/libdl.so
0x70027f40 0x7005987f /usr/lynx/3.1.0/mips/lib/shlib/libc.so
0x70452ec0 0x704649ff /usr/lynx/3.1.0/mips/lib/shlib/libgcc.so
0x70475240 0x70480aef /usr/lynx/3.1.0/mips/lib/shlib/libm.so

Deferred Breakpoints

When setting a breakpoint with symbolic information such as a function name, GDB has to resolve the breakpoint specification into the target process's virtual address immediately. This is impossible if the breakpoint being set will be found in a shared library and the shared library symbol file has not been loaded to GDB either because the target process has not yet started or because GDB has not yet attached to the target process.

GDB for LynxOS supports deferred breakpoints that allow breakpoint addresses to remain unresolved. When a deferred breakpoint is set, GDB immediately tries to set it as a "real" breakpoint: If the breakpoint setting is successful, the breakpoint will work as a regular breakpoint; if it fails, GDB does not print an error message but will remember the breakpoint specification by keeping it in the deferred breakpoint list. When the target process starts or is attached to GDB and a shared library is detected and loaded, GDB will try to set the deferred breakpoints as real breakpoints.

Deferred Breakpoint Commands

The dbreak command sets a deferred breakpoint. breakpoint_spec is the specification of the breakpoint to be set; it can be any string that would be accepted by the "real" break command, including an optional if condition clause. If the breakpoint is successfully set as a real breakpoint, the breakpoint will work just like any other regular breakpoints, except that it is also registered in the deferred breakpoint list; otherwise, breakpoint_spec merely remains registered in the deferred breakpoint list. An attempt will be made to set the deferred breakpoint as a real breakpoint when the target process is started or attached by one of the following commands:

If the attempt to convert a deferred breakpoint to a real breakpoint fails, the deferred breakpoint will remain registered for another attempt in the future and you will see no error message.


Note: The dbreak command does not parse or check the syntax of breakpoint_spec; it simply passes the whole string to the breakpoint command executive. So, "failure to set as a real breakpoint" may indicate a syntactical error in breakpoint_spec.

Optionally, you may give the break command an unresolvable breakpoint specification. If the break command finds it cannot resolve the breakpoint specification to an address immediately, it will fall into the dbreak command function after confirmation.

The ddelete command removes a deferred breakpoint from the deferred breakpoint list. dbreak_num is the deferred breakpoint number shown by the info dbreak command. If dbreak_num is not given, ddelete will remove all deferred breakpoints after confirmation. If the deferred breakpoint being removed is currently set as a real breakpoint, ddelete will prompt for confirmation to remove the real breakpoint as well.


Note: The deferred breakpoint list uses separate numbering from "real breakpoints." Those two series of breakpoint numbers should not be confused.

The info dbreak command displays the information about the deferred breakpoint designated by dbreak_num or all deferred breakpoints.

Deferred Breakpoints for Application-Loaded (dlopen'ed) Shared Libraries

In addition to the above features of deferred breakpoints that are applicable to both kernel-loaded and application-loaded shared libraries, application-loaded (dlopen'ed) shared libraries benefit some more from deferred breakpoints.

Automatic Promotion for dlopen

When a new application-loaded (dlopen'ed) shared library is loaded by the target process and its symbol file is loaded to GDB, GDB will automatically try to set the "pending" deferred breakpoints as real breakpoints. Those deferred breakpoints that are successfully converted to real breakpoints are marked "busy," while other pending ones will remain pending for later attempts. GDB will not remove the successfully converted deferred breakpoints from the list; they will remain registered until explicitly removed by the ddelete command. This is because those breakpoints may need to be set again when the shared library is closed and then reopened in the future.

Automatic Demotion for dlclose

When an application-loaded shared library is last closed and its symbol file is removed from GDB, GDB automatically removes all breakpoints that were set for the shared library. These breakpoints include both those that were automatically set "real" when the shared library was loaded and those that were set manually by the break command.

It is important to know that a stale breakpoint is deleted but not disabled even if the breakpoint belonged to an application-loaded shared library that may be reopened in the future. This is because the breakpoint's address value is no longer valid and there is no guarantee that the same address will be used for the given deferred breakpoint specification when the shared library is reopened. Therefore, the corresponding real breakpoint number will become invalid as well, and a new breakpoint number will be assigned when the shared library is reopened.

Shared Library File Path Names

In remote debugging, the debug host and the debug target may not necessarily have the same directory layout; for example, one may be developing a shared library foo.so in the debug host's directory /home/joe/proj1/usr/lib/shlib/, but the library may be supposed to be loaded by the target process from /usr/lib/shlib/ on the debug target's file system. GDB has to resolve this directory path difference particularly for shared library symbol file loading because:

In general, it is recommended to have (a subset of) the target's file system image under a host file system directory pointed to by the ENV_PREFIX environment variable, but this may not always be the case. LynxOS GDB handles this shared library file path resolution issue in the following ways for convenience and flexibility.

Automatic Shared Library Symbol File Loading

The .dynamic section of the application's executable file will contain the shared library file names and optionally the directory information (on the target's file system). GDB uses the _host_shlib_dirs GDB variable and the debug host's ENV_PREFIX environment variable to search for the shared library symbol files on the debug host's file system.

_host_shlib_dirs is a colon-separated list of host directory names in which the shared library file is expected to reside. GDB uses these directories to override the following file name path composition rules. _host_shlib_dirs can be set by the set command as follows:

The above example implies the shared library symbol files are supposed to be found in /home/joe/proj1/usr/lib/shlib or /home/joe/proj1/testlib.

Otherwise if _host_shlib_dirs is not set, GDB will resolve shared library path names in two stages:

If GDB cannot find the library in the preceding steps, GDB will prefix the composed file path with the ENV_PREFIX environment variable (if it exists). GDB uses ENV_PREFIX as the "virtual mount point" of the target's file system on the debug host. ENV_PREFIX should point to a directory on the debug host's file system which contains a duplicate image of the target's file system.

For example, if the target application uses a shared library foo.so with the following conditions:

GDB will look for the corresponding symbol file on the debug host in the following sequence:

	Caution! Although the ELF specification allows a relative shared library path name to the application process's current working directory and LynxOS kernel-loaded shared library honors this, the current release of LynxOS GDB does not support this for automatic symbol file loading.

Application-Loaded Shared (dlopen'ed) Libraries

The LynxOS dlopen library resolves any application-loaded shared library path name passed to the library as an argument into a "clean" absolute path beginning with forward slash (/). Here, "clean" means that the path contains no single-dot ( . ) and double-dot ( .. ) elements representing the current and parent directories respectively. GDB will try to locate the corresponding symbol file on the debug host's file system in the following sequence (for /prod/shlib/foo.so):

Manual Shared Library Symbol Loading/Unloading
(add-symbol-file/delete-symbol-file)

For the add-symbol-file command, GDB accepts a file path name in the debug host's notation. GDB will use the exact given file path name for loading the file. For the platforms that support dynamic shared library loading, GDB will ask the target about the library's run-time loading address with the following rule:

Note: Path names are compared literally: GDB will not resolve traverses in the directory hierarchy with the current and parent directory notations ("." and ".."). For example, /usr/./abc and /usr/abc will not match.

If any of the above inquiries fails, GDB considers that the shared library does not have a dynamic loading address, but is loaded statically: That is, the loading address is 0 (zero).

For each manually loaded shared library file, GDB displays a message like the following:

(gdb) add-symbol-file ./file.so 0
add symbol table from file "./file.so" at text_addr = 0x0
(y or n) y
Loading symbols for ./file.so with address offset 0x12345678...

If the target platform supports dynamic loading of shared libraries and the "address offset" is zero (0), it usually means GDB failed to determine the loading address. Check the file path for add-symbol-file.

Symbol Table

GDB loads and unloads symbol information by the symbol file, but it looks up the symbol table for a symbol entry by using the source file name. If two shared libraries have been built using the same source file or source files with the same name containing the same function name, and if those shared libraries are loaded at the same time, only one of the multiple functions (with the same name) is visible to GDB.

For example, if shared libraries x.so and y.so were built from a.c and b.c, and b.c and c.c respectively, a reference to function foo in b.c (b.c:foo) would find only one of the two functions, although there are two copies of function foo at different addresses. GDB currently has no way to specify the shared library file name for a symbol lookup.

Single-Stepping into a Shared Library Function

Because of the symbol scope sensitivity, the step command for a shared library function call may unexpectedly act like the next command; execution does not stop at the entry of the shared library function but it stops after returning the function call. This happens if the function belongs to a different symbol scope (another shared library) from the current one. To step into the function with stopping, set a breakpoint at the function before executing the step command.

Summary of Additional Commands for Shared Library Support

Debugging Kernel/Device Drivers

GDB can be used to debug LynxOS custom device drivers. Although GDB can be used as a tool for porting the LynxOS kernel to a new platform, it may not be very useful because LynxOS GDB kernel debugging requires a fairly stable LynxOS target kernel to operate.

Requirements

Building a Kernel for Debug Purposes

To build a LynxOS kernel for debug purposes at the source level, compile the device driver (or whatever code that will be debugged at the source level) with the g option. Edit your Makefile to include the -g option for compiling (the linker does not need -g).

LynuxWorks includes a second set of kernel libraries to help debug the kernel even more. These libraries include not only the symbols necessary for linking, but also the full debug information. By linking the kernel image with these libraries, even though the LynxOS kernel source code may not be available, one can browse some useful source level information such as calling parameters in a function call chain (stack trace).

To build a kernel image with libraries that can be debugged fully, run the make utility in sys/lynx.os with SYS_DEBUG=true:

The use of the libraries with full debug information, however, increases the size of the resulting kernel image (fat image) as well as the run time memory requirement, typically by several mega bytes. If the target's memory is tight, it is possible to strip the debug information off the fat image being loaded into the target, while still using the fat image for referencing debug information by GDB on the host.


Note: A stripped kernel is not capable of dynamically loading device drivers because it requires resolution of kernel symbols.To use dynamically loaded device drivers on a system where the target's memory is tight, use the traditional kernel libraries. (Without SYS_DEBUG=true, you can still include full debug information for a specific device driver incurring a small to moderate increase of memory requirement.)

Debugging the Kernel

Virtually all the normal GDB features are available for kernel debugging purposes:

One big difference is that kernel debugging must always be performed in the form of remote debugging. A separate host computer to run LynxOS GDB on is required to debug the LynxOS target kernel. There is no self or local kernel debugging.

Another difference is that it is impossible to start and terminate the target kernel; GDB always interrupts a kernel that is already running to start a debug session and releases it to finish the session. This is similar to attach and detach in user process debugging. To start and finish debug sessions, see "Starting Kernel Debugging" and "Finishing Kernel Debugging".


	Caution! Since kernel debugging must stop the entire operating system, the operating system will not respond when the kernel is at a breakpoint or is stopped by the debugger. In some circumstances, even though one lets the system "go," the system may respond considerably slower or have no response at all because the debugger internally keeps single-stepping the kernel.

Simple Kernel Debugger-SKDB

LynxOS GDB on the debug host actually talks to SKDB, which is embedded in the kernel of the target with a special protocol. SKDB works as an agent and performs the following basic operations to requests made by GDB:

To build a LynxOS kernel with SKDB installed, use the Install.skdb utility script with the optional SYS_DEBUG=true flag as follows:

Threads vs. Processes

In a kernel debugging session, GDB is virtually unaware of processes at all. Though it still reports the current process ID, every thread running on the target is visible to GDB, unlike user process debugging in which only the target process's threads are visible. Therefore, a nonthread-specific breakpoint can be hit by any thread of any process on the target, including kernel threads. The info thread command may display a long list of threads.

Setting Up Serial Ports

Though it is possible to share the target serial port for both regular terminal use and kernel debugging, a dedicated serial port for kernel debugging is strongly recommended for reliable communication. The target serial port must have matching parameters with the host's, such as bit rate speed and parity bit. These parameters are usually configured with the target's ttyinfo.c file, but you can override the default values with the
stty command on the target. To change the communication speed of GDB, use the set remotebaud command.

Starting Kernel Debugging

To start a kernel debug session, use the target command with the target of skdb and an appropriate protocol (serial port or sspp proxy).

Note: For SKDB to accept a "break-in" by GDB, the serial port must have been opened by a process. This may be done by starting a login process on the port by editing /etc/ttys or by running such a program as cat on the port. No user can actually log in at the port because the port is connected to the debug host. Also, the cat process should not output to the port. Keep the port open but quiet.

Note: For SKDB to accept a "break-in" by GDB, the serial port must have been opened by a process. This may be done by starting a login process on the port by editing /etc/ttys or by running such a program as cat on the port. No user can actually log in at the port because the port is connected to the debug host. Also, the cat process should not output to the port. Keep the port open but quiet.

Once communication is established, GDB interrupts the target's kernel and reports the location where the kernel was interrupted, usually in the null process. In the above example, an error message was displayed because GDB could not find the source file main.c in its default source file search path while the null process module was compiled with full debug information. If the source code file(s) are available, use the dir command to add the appropriate directory to the search path list.

try the target command again. If the error persists, it may be due to wrong communication parameters or wrong target configuration such as missing SKDB or a wrong port.

Now set breakpoints at desired kernel locations and use the continue command to resume the kernel. Once the kernel hits a breakpoint and stops, it is possible to examine variables and the call stack chain, single-step, continue, and so forth as one would do for user process debugging.


	Caution! Although kernel mode debugging is powerful and GDB lets the user manipulate any and all memory contents in the target system, not only those in the kernel space but also those in user process spaces, it is not advisable to set breakpoints in the user process space (user program). The LynxOS kernel handles kernel-mode breakpoints completely different from user-mode breakpoints; a user-mode breakpoint set by kernel debugger (GDB) will not be captured by GDB running in kernel debug mode, and thus it will cause an unexpected termination of the process.

Interrupting the Kernel

To interrupt a running kernel, press Ctrl+C at GDB, while it is waiting for the target to stop as one would do for user process debugging. LynxOS GDB sends the break-in character to stop the target kernel.

Single-Stepping the Kernel

The step and stepi command single-step the current thread in the kernel. Unlike user process debugging, however, only the current thread can be single-stepped; the thread command has no effect on single-stepping (you cannot change the current thread for singlestepping).


	Caution! It is not recommended to attempt to trace (setting breakpoints in and/or single-stepping) the "core" portions of the LynxOS kernel, such as those handling context switching and interrupt control, because such an attempt may severely interfere with the LynxOS kernel operation. Also, those instructions that manipulate the processor's status register cannot be safely single-stepped. Though LynxOS GDB detects, warns about, and prevents such an attempt, casual tracing of such critical code may result in an unexpected system freeze.

Finishing Kernel Debugging

To finish a debug session and to let the target kernel resume freely, kill the target at the GDB prompt or quit GDB. Despite the command name, the target's kernel is not killed, but is resumed freely.

(gdb)

Caution! If a kernel debugging session is accidentally terminated due to a communication error or some other unexpected reason, GDB may not have had a chance to remove breakpoints before the termination. If this happens and a new kernel debug session is started, the kernel may be trapped at a breakpoint forever until the original instruction at the breakpoint location is restored by hand with a command like print or until the kernel is reloaded (restarted). GDB currently provides no convenient way to restore the original instructions.To reload the kernel and restart LynxOS, use the R SKDB command. (See "Raw SKDB Commands") or reset the target's hardware with the reset switch or a power-cycle.

	Caution! If a kernel debugging session is accidentally terminated due to a communication error or some other unexpected reason, GDB may not have had a chance to remove breakpoints before the termination. If this happens and a new kernel debug session is started, the kernel may be trapped at a breakpoint forever until the original instruction at the breakpoint location is restored by hand with a command like print or until the kernel is reloaded (restarted). GDB currently provides no convenient way to restore the original instructions.To reload the kernel and restart LynxOS, use the R SKDB command. (See "Raw SKDB Commands") or reset the target's hardware with the reset switch or a power-cycle.

Loading Device Drivers Dynamically

Device drivers can be dynamically loaded into memory at runtime, instead of build-time, while the system is up and running. Device drivers can be also removed from memory and reloaded later. This facility is very convenient for device driver development.

To load a device driver dynamically, use the drinstall LynxOS command with the device driver's *.o file. To add symbol information for a dynamically loaded device driver, use the add-symbol-file GDB command with the driver's *.o file name and its loading address as reported by the drinstall or drivers LynxOS command.

Raw SKDB Commands

GDB is a generic debugger. It knows little about the LynxOS kernel. To explore the LynxOS kernel in more detail, such as the process table, thread structures, and so on, you can pass a raw SKDB command from LynxOS GDB using the skdb command followed by the desired SKDB command string.

See Chapter 5, "Simple Kernel Debugger-SKDB" for information on the SKDB command.


	Caution! GDB does not keep track of any operation performed with the skdb command. For example, if you set a breakpoint with the skdb command, it would be unknown to GDB but still cause a break, and therefore the breakpoint would confuse GDB and SKDB. It is generally not advisable to use raw SKDB commands to alter the target's state. It is possible to change the target's memory contents with a raw SKDB command (although it is not recommended). When doing so, it is important to clear GDB's internal data cache with the set remotecache command after changing the memory contents.

Proxy Server

If GDB is set up for remote debugging over a serial line such as RS-232, the serial line usually limits the physical distance between the target and the host running GDB. LynxOS GDB extends this distance infinitely by using a proxy server-a third computer-between the target and the host. The proxy server redirects the serial line communication to a TCP/IP connection such as a local area network (LAN) or the Internet. Now you can use GDB from another room, another building, or even another country to debug the target.

The proxy server program sspp is a simple and small program supplied in the form of source code so that it can be ported to different platforms. The proxy server computer is either a dedicated or shared machine running a variant of UNIX including LynxOS, or it can be a terminal server that multiplexes a number of serial port connections.

The sspp program is transparent to the target, so it can be used for both user process debugging (target remote) and device driver/kernel debugging (target skdb).

To run the sspp proxy server program, the following items are required on the proxy server computer:

These should be available on most modern UNIX workstations including LynxOS workstations.

Syntax

The following example starts a user process debugging session (remote) using the sspp proxy server program (sspp) on the proxy server computer myproxy, whose serial port /dev/ttya is connected to the LynxOS target:

If the proxy program is not installed in the default search path on the server, pass its full path name to GDB. To specify the serial port communication speed, use sspp's -b option.

The next example shows a device driver/kernel debugging session (skdb) using the sspp proxy server program loaded at /local/bin/sspp on the proxy server computer myproxy, whose serial port /dev/com2 is connected to the LynxOS target at 19,200 bps:

Installation

sspp comes in source code for easy porting to a variety of platforms. To port sspp, you need an ANSI-compliant C compiler (gcc preferred).

Compiling sspp.c

The associated Makefile is simple. Give its compile command line the desired macro definitions in the form of -DMACRO=1:

Installing sspp

sspp must be installed as a set-uid executable to be able to access the serial port device file and the lock file. This requires root (super user) privilege. The Makefile uses the set-uid user of uucp. If your site has a different user ID for this purpose, change it appropriately.

After installation, try starting sspp from a remote machine for testing. If the results are similar to the following, the installation was successful:

When modifying sspp's source code for a custom environment, it can be debugged by using double colons when starting a debug session from GDB. The message transactions and some more useful information are displayed:

Minicom Terminal Server

LynuxWorks uses dedicated terminal servers in its product test area. These terminal servers run the Minicom terminal server program written by Miquel van Smoorenburg on LynxOS, and multiplex serial port connections to a number of test platforms. The Minicom terminal server program is usually installed as the user's login shell and thus it is impossible for the user to use remote shell for the sspp proxy. The sspp.arb script arbitrates between Minicom and sspp by checking if a login is interactive (Minicom) or remote (sspp) with a simple time-out mechanism.

To use this arbitrator script, first edit the script so that the function gominicom points to a correct path for Minicom. Then install it as the user's login shell (put the script's path into the login shell field of the user's passwd entry).

General Tips and Miscellaneous Issues

The following sections provide you with general tips and other useful information:

Reading and Writing Large Memory Blocks

The memget and memput commands let you transfer large blocks of memory contents from and to, respectively, the target process's address space very efficiently. Table: Reading and Writing Large Memory Blocks shows the syntax for memget and memput commands.

Reading and Writing Large Memory Blocks
Syntax	Description
memget localfile address size	localfile is the GDB host's local file to which the memory block is transferred. address is the absolute address in the debugged process's address space in either decimal, hexadecimal, or octal (no symbolic address is allowed). size is the transferred memory size in number of bytes in decimal.
memget ">localfile" address size	Same as the first syntax except that the surrounding quotation marks are mandatory.
memget "\|localcommand args" address size	Same as the first syntax except that it redirects the memory bytes to localcommand's standard input through pipe. The surrounding quotation marks are mandatory.
memput localfile address size	Same as the first syntax except that localfile is the GDB host's local file from which the memory block is transferred. size is optional and the default is localfile's size.
memput "<localfile" address size	Same as the previous syntax. size is optional and the default is localfile's size. The surrounding quotation marks are mandatory.

Browsing Target Process's Environment

If the debug target process is not available the info environment command displays the debugger's or gdbserver's environment, depending on the mode of debugging (local or remote).

Executing Remote Shell Commands

The rshell command lets one execute a shell command on the remote target machine. It is similar to the BSD rsh (remote shell) command, but rshell works even over a serial connection without TCP/IP. Therefore, it is useful for browsing the embedded target's directories, processes, and so forth.

The rshell command handles quotation marks and other shell meta characters properly if the target has a shell program installed; otherwise, it does not.


	Caution! Do not execute an interactive program or a program that reads the standard input with the rshell command. The standard input of the remote program is redirected to /dev/null.

In remote debugging, the rshell command becomes effective only after a connection with the target is established with the target command. If the rshell command is used before a connection is established or after connection is lost, it works for the local host because the default target is the local host.

Function Calls in a Multithreaded Process

To manually execute a function call, such as a command line like
"print foo(1)" at the GDB prompt, GDB performs the following:

It should be noted that if a function is called by hand in a target process that has more than one thread, not only the current thread but also all other sibling threads in the target process are subject to scheduling, as described in "Resuming Threads". This usage may result in unexpected side effects if some other thread actually runs while the manual function call is running. To prevent this, raise the current thread's priority temporarily as described in "Resuming Threads" provided that the function call does not involve a blocking system call.

Functions Calls after Ctrl+C

If the current thread is in a blocking system call and you type Ctrl+C from GDB to interrupt the thread, the target process stops and control returns to GDB. If you then try to call a function in the target program by hand with the print command or a similar command, GDB appears to be hung and the function will not be executed.

This is because the target process has to complete the blocking system call before running the function. The hung function may be interrupted by entering another Ctrl+C. Then, the continue command resumes the target process to reenter the system call.

Resuming after a Blocking System Call

If the current thread stops at a system call instruction due to a breakpoint and you try to resume the application process with the continue command, GDB may report the following warning:

In this case, GDB regains control without executing the blocking system call. To resume the process continuously, remove the cause of the blocking.

This is a result of GDB's internal mechanism to resume a thread from a breakpoint. When GDB resumes a process that has stopped due to a breakpoint hit by one of its threads, GDB:

In order to ensure an atomic operation, LynxOS GDB uses special single-stepping which guarantees that only the thread in the process runs while all other sibling threads in the same target process remain stopped. Otherwise, some other thread in the same process may run while the original instruction at the breakpoint location is restored and that take-over thread may miss the breakpoint.

In this scenario, if the breakpoint were set at a system call instruction that waits for a resource that is locked by another thread in the same target process (for example, mutex) the system call would never complete because no other threads in the target process can run, and thus this would cause a dead lock. The LynxOS kernel prevents such a dead lock by detecting the situation and breaking the blocking with a special internal signal (SIGNONBLOCK).

Debugging a Signal-Intensive Process

LynxOS relies on the UNIX/POSIX 1 synchronous signal mechanism for debugging a process: Any signal to a traced (debugged) process is captured by the kernel's process trace code, causing the traced process to stop. The stop is then reported to the debugger. A breakpoint or single-stepping merely generates and sends a special signal to the traced process.

Although GDB can be configured with the handle command to pass and not print a signal when it is sent to the target (traced) process, GDB implements this signal handling by software: Any signal sent the target process still causes the process to stop; when GDB detects the target process's stop, it examines the signal code that caused the stop; If the signal code is configured to pass, GDB silently resumes the target process without reporting it to the user.

Obviously, the above processing involves software overhead in both the kernel and GDB. If the target process is designed to receive signals frequently, its execution speed may noticeably slow down under a debugger, even though the signals are not explicitly reported by GDB. Remote debugging makes the situation even worse because of the additional overhead for the remote communication transaction for each signal reception.

Total/db User's Guide

LynxOS GDB Enhancements