April 23, 2007
Multi-threaded Debugging TechniquesNotification on Thread Creation
When GDB detects that a new thread is created, it displays a message specifying the thread's identification on the current system. This identification, known as the systag, varies from platform to platform. Here is an example of this notification:
Starting program: /home/user/threads [Thread debugging using libthread_db enabled] [New Thread -151132480 (LWP 4445)] [New Thread -151135312 (LWP 4446)]
Keep in mind that the systag is the operating system's identification for a thread, not GDB's. GDB assigns each thread a unique number that identifies it for debugging purposes.
Getting a List of All Threads in the ApplicationGDB provides the generic info command to get a wide variety of information about the program being debugged. It is no surprise that a subcommand of info would be info threads. This command prints a list of threads running in the system:
(gdb) info threads 2 Thread -151135312 (LWP 4448) 0x00905f80 in vfprintf () from /lib/tls/libc.so.6 * 1 Thread -151132480 (LWP 4447) main () at threads.c:27
The info threads command displays a table that lists three properties of the threads in the system: the thread number attached to the thread by GDB, the systag value, and the current stack frame for the current thread. The currently active thread is denoted by GDB with the * symbol. The thread number is used in all other commands in GDB.
Setting Thread-specific Breakpoints
GDB allows users that are debugging multi-threaded applications to choose whether or not to set a breakpoint on all threads or on a particular thread. The much like the info command, this capability is enabled via an extended parameter that's specified in the break command. The general form of this instruction is:
break linespec thread threadnum
where linespec is the standard gdb syntax for specifying a breakpoint, and threadnum is the thread number obtained from the info threads command. If the thread threadnum arguments are omitted, the breakpoint applies to all threads in your program. Thread-specific breakpoints can be combined with conditional breakpoints:
(gdb) break buffer.c:33 thread 7 if level > watermark
Note that stopping on a breakpoint stops all threads in your program. Generally speaking this is a desirable effect-it allows a developer to examine the entire state of an application, and the ability to switch the current thread. These are good things.
Developers should keep certain behaviors in mind, however, when using breakpoints from within GDB. The first issue is related to how system calls behave when they are interrupted by the debugger. To illustrate this point, consider a system with two threads. The first thread is in the middle of a system call when the second thread reaches a breakpoint. When the breakpoint is triggered, the system call may return early. The reason-GDB uses signals to manage breakpoints. The signal may cause a system call to return prematurely. To illustrate this point, let's say that thread 1 was executing the system call sleep(30). When the breakpoint in thread 2 is hit, the sleep call will return, regardless of how long the thread has actually slept. To avoid unexpected behavior due to system calls returning prematurely, it is advisable that you check the return values of all system calls and handle this case. In this example, sleep() returns the number of seconds left to sleep. This call can be placed inside of a loop to guarantee that the sleep has occurred for the amount of time specified. This is shown in Listing 1.8.
int sleep_duration = 30;
do
{
sleep_duration = sleep(sleep_duration);
} while (sleep_duration > 0);
Listing 1.8: Proper Error Handling of System Calls
The second point to keep in mind is that GDB does not single step all threads in lockstep. Therefore, when single-stepping a line of code in one thread, you may end up executing a lot of code in other threads prior to returning to the thread that you are debugging. If you have breakpoints in other threads, you may suddenly jump to those code sections. On some OSs, GDB supports a scheduler locking mode via the set scheduler-locking command. This allows a developer to specify that the current thread is the only thread that should be allowed to run.
Switching Between Threads
In GDB, the thread command may be used to switch between threads. It takes a single parameter, the thread number returned by the info threads command. Here is an example of the thread command:
gdb) thread 2
[Switching to thread 2 (Thread -151135312 (LWP 4549))]#0 PrintThreads (num=0xf6fddbb0) at threads.c:39
39 {
(gdb) info threads
* 2 Thread -151135312 (LWP 4549) PrintThreads (num=0xf6fddbb0) at threads.c:39
1 Thread -151132480 (LWP 4548) main () at threads.c:27
(gdb)
In this example, the thread command makes thread number 2 the active thread.
Applying a Command to a Group of Threads
The thread command supports a single subcommand apply that can be used to apply a command to one or more threads in the application. The thread numbers can be supplied individually, or the special keyword all may be used to apply the command to all threads in the process, as illustrated in the following example:
gdb) thread apply all bt
Thread 2 (Thread -151135312 (LWP 4549)):
#0 PrintThreads (num=0xf6fddbb0) at threads.c:39
#1 0x00b001d5 in start_thread () from /lib/tls/libpthread.so.0
#2 0x009912da in clone () from /lib/tls/libc.so.6
Thread 1 (Thread -151132480 (LWP 4548)):
#0 main () at threads.c:27
39 {
(gdb)
The GDB backtrace (bt) command is applied to all threads in the system. In this scenario, this command is functionally equivalent to: thread apply 2 1 bt.
Key Points
This article described a number of general purpose debugging techniques for multi-threaded applications. To sum up:
|
 |
|
|||||||||||||||||||||||||||||
|
|
|
