Return to Training Programs


Course-Number: ATR-003

Course-Description: Advanced Linux – Kernel Debugging

Duration of course: 3 days

Course Description:

As this course title indicates, this is an advanced course on Linux Kernel Internals and can only be taken by people with at least 6 months to 1 year of programing experience with the linux kernel.

The course concentrates on the various techniques involved in debugging kernel code. Ordinary training programs in Linux merely discuss using printk and the proc file system. This is not sufficient for the advanced linux kernel programmer who has to deal with unpredictable hardware, unexpected interrupts and kernel crashes.

Debugging a running kernel is often treated like black-magic, known only to those few who are termed "kernel hackers". This course de-mystifies the entire treatment of debugging the kernel like debugging any other piece of complex code and in the process, the participant also learns to appreciate the finer points of the linux kernel architecture.

The course covers the following important concepts:

Target Audience:

The course is ideal for intermediate level systems programmers and device driver writers under Linux who wish to further enhance their skills in writing kernel code.


Completion of course ATR-001 and ATR-002 ( Linux Kernel Internals, Linux Device Drivers ) is a must for doing this course.






Introduction to kernel debugging – issues & considerations

1.5 hrs

Types of problems involved in debugging the kernel. The big picture must be seen first to then understand the various tools involved in kernel debugging

Using simple tools – printk, proc file system

1.5 hrs

Quick intro to printk() and console log levels. Writing a proc file system interface to kernel code – examples

Using strace() and ksymoops

1.5 hrs

Using the system call trace for profiling execution, and a detailed treatment of OOPS messages, using klogd and ksymoops

Using ioctl for specific querying of kernel code

1.5 hrs

Review of ioctl, usage guidelines and its role in debugging the kernel


The SysRq key

1.5 hrs

Enabling support for SysRq in the kernel, usage guidelines, options and examples.

Using gdb

1.5 hrs

Understanding the limited options available when using gdb, compiling for gdb support, usage guidelines and examples

The Kernel Debugger ( kdb )

1.5 hrs

Patching the kernel source for kdb support, commands, usage guidelines and examples

LTT - The Linux Trace Toolkit

1.5 hrs

Overview - Event Tracing mechanisms.


The kgdb patch

1.5 hrs

Using gdb on the linux kernel with a second system using a serial cable.

LKCD – Linux Kernel Crash Dump analyzers

3.0 hrs

Dumping system state on a crash into a special dump device.

IBM’s DProbes ( Dynamic Probes )

1.5 hrs

Inserting probes ( ‘instrumenting’) into the running kernel.

(C) 2005 -   All Rights Reserved.

Return to Training Programs