Linux Basics for UNIX Developers

In this class, you will learn the basics of developing in Linux. We assume development experience on another UNIX system, and focus on the differences and unique features of Linux.

Outline:

Getting started
Development environment
Tracing, debugging, and performance
Finding and installing software
Kernel development
Useful Linux tools
Learning more

Set up your environment

Web browser

If you're reading this, you're probably already using either Mozilla or Firefox. Konqueror is an option as well.

Graphical terminal

Click on the start menu in the bottom left hand corner, go to "System Tools", and select "Terminal". This will get you a gterm - if you don't like it, type "xterm" to get an xterm. gterm has some nifty features. In gterm, change the font by right clicking and choosing "Edit current profile". In xterm, change the font by holding down control and right-clicking.

Shell

The default shell in Linux is bash (/bin/sh is a link to /bin/bash). It features command and filename completion (hit tab), command history (hit up arrow), command history search (Ctrl-R then type in the string to search for, hit Ctrl-R again to cycle through matches), and much more. The man page is complete but difficult to read in one sitting. The main alternative is tcsh, which is functionally equivalent but slightly different in syntax and key bindings.

Choose an editor

The eternal editor wars exist in Linux as well; choose your poison, emacs or vi(m). Type either "emacs" or "vi" at the prompt. If you use emacs, you can get a lot of convenient configuration, including the right defaults for the C coding style of the Linux kernel, from my configuration file (download here). A simple graphical editor is gedit. There is some kind of tutorial for each editor:

For emacs, it escape, type "help", then type "t" for tutorial
For vim, run "vimtutor"
For gedit, click on the "help" button

Some quick references:

Finding installed software

A simple first method is to guess the beginning of the name of a program and then hit tab in your shell and see what is available in your path. For example, in the GNOME window environment, many programs start with "g" and are followed by a common word. Let's say I want to download some photos from my camera. I might try typing "g" followed by "pho" and then hitting tab:

$ gpho[tab]
$ gphoto2

You can learn about programs by reading the man page:

$ man gphoto2

Or by getting a help message:

$ gphoto2 --help

Or looking in /usr/share/doc/<program name and version>:

$ less /usr/share/doc/gphoto2-2.1.0

Or by doing a web search. There are vast amounts of Linux documentation available online.

Another method is to use the keyword search feature of man:

$ man -k photo

You can use the menu of the window manager you are using to track down programs as well - just click on the start menu icon, usually located in about the same place as the Windows start menu.

Development environment

Compiling programs

gcc is the compiler. cc is a link to gcc. You can learn more about gcc by typing "info gcc".

Let's compile a hello world program. Download this program and run:

$ cc -o hello hello.c
$ ./hello
Hello, world!

Run only the preprocessor on it:

$ gcc -E hello.c

Generate assembly output in hello.s:

$ gcc -S hello.c

Makefiles

Makefiles in Linux are processed using gmake, which has a few slight differences from many commercial makes. As usual, make is an alias for gmake. Copy this Makefile into the same directory as your hello program and run make. Most of the make documentation is available using "info make".

Tracing, debugging, and performance tools

System call tracing

The most useful and reliable tracing tool in Linux is strace, a tool for tracing system calls. Try it on hello world:

$ strace ./hello
execve("./hello", ["./hello"], [/* 33 vars */]) = 0
uname({sys="Linux", node="localhost.localdomain", ...}) = 0
brk(0)                                  = 0x8ef7000
[...]

As you can see, a whole lot goes on during program startup and teardown that you aren't interested in. Let's limit the output of strace to write() system calls:

$ strace -e write ./hello
write(1, "Hello, world!\n", 14Hello, world!
)         = 14

The output of the program ends up mixed in with the output of strace. Send the program's output to /dev/null:

$ strace -e write ./hello 1>/dev/null
write(1, "Hello, world!\n", 14)         = 14

Library call tracing

We can trace library calls made by a program using ltrace:

$ ltrace ./hello
__libc_start_main(0x8048368, 1, 0xfee7f254, 0x804839c, 0x80483f0 
printf("Hello, world!\n"Hello, world!
)                        = 14
+++ exited (status 0) +++

Memory allocation tracing

We can trace memory allocations made using glibc malloc(), but we need to modify the program source to do so. Copy your hello world program to hello_mtrace.c (or download it) and add the following lines to your program:

#include <stdlib.h>
#include <mcheck.h>

mtrace();
(void) malloc(1000);
muntrace();

Recompile:

$ make hello_mtrace

Then you must specify a file for the output:

$ export MALLOC_TRACE=mtrace.out

Now run the program and process the output:

$ ./hello_mtrace
Hello, world!
$ mtrace hello_mtrace mtrace.out

Memory not freed:
-----------------
   Address     Size     Caller
0x09a2b378    0x3e8  at 0x804844a

That's nice, but what do those addresses mean? Let's recompile with debugging information on so we can get line numbers.

$ cc -g -o hello_mtrace hello_mtrace.c
$ ./hello_mtrace
Hello, world!
$ ./mtrace hello_mtrace mtrace.out

Memory not freed:
-----------------
   Address     Size     Caller
0x0985b378    0x3e8  at /home/val/class/web/hello_mtrace.c:12

Using gdb

gdb is the primary Linux debugger, but it tends to be poorly supported and crash a lot.

Let's introduce a bug that causes a core dump. Copy hello.c to hello_core.c (or download it) and add the following line:

int i = * (int *) 0;

Recompile with debugging information:

$ gcc -g -o hello_core hello_core.c

Most likely, you will not get a core dump unless you increase the core dump limit:

$ su
Password:
# ulimit -c unlimited
# su 
$ ./hello_core
Segmentation fault (core dumped)

Now run gdb with both the binary and the core file:

$ gdb hello_core core.*
[...]
#0  main () at hello_core.c:6
6               int i = * (int *) 0;
(gdb)

Seems like line 6 of hello_core.c would be a good place to look for a bug. Get a backtrace with the "bt" command:

(gdb) bt
#0  main () at hello_core.c:6

The online help can be accessed with the "help" command.

Profiling with gcov

gcov use static instrumentation produced by gcc to profile the program. First, we need to add a loop to our program so that we can get some interesting data. Add the following just prior to the printf statement (or download here):

int i;
for (i = 0; i < 10000; i++)

Now recompile the program to produce profile data:

$ cc -pg -g -fprofile-arcs -ftest-coverage -o hello_gcov hello_gcov.c

Now run the program. This generates the profiling data and puts it into files in the local directory.

$ ./hello_gcov

Now run gcov:

$ gcov hello_gcov.c
File `hello_gcov.c'
Lines executed:100.00% of 5
hello_gcov.c:creating `hello_gcov.c.gcov'

Looks like we have 100% test coverage - every line was executed at least once. The file gives us the following information:

        -:    0:Source:hello_gcov.c
        -:    0:Graph:hello_gcov.gcno
        -:    0:Data:hello_gcov.gcda
        -:    0:Runs:1
        -:    0:Programs:1
        -:    1:#include 
        -:    2:
        -:    3:int
        -:    4:main(void)
function main called 1 returned 100% blocks executed 100%
        1:    5:{
        1:    6:        int i;
        -:    7:
    10001:    8:        for (i = 0; i < 10000; i++)
    10000:    9:                printf("Hello, world!\n");
        1:   10:        return (0);
        -:   11:}

On the performance side, it looks like we executed line 8 10001 times and line 9 10000 times... perhaps this would be a good place to optimize the program. Later on, after we have recompiled the kernel, we will use oprofile, another profiling tool.

Finding and installing software

If you can think of a piece of software, probably someone in the open source community has already written it and put it up for download and free use somewhere on the web. Here are a few ways to find that software.

Packaged software

Prepackaged software is the easiest to install. On Red Hat, the package manager is called rpm; package files in this format are called rpms.

The best way to find an rpm you want is to search for it on http://rpmfind.net. We are going to search for the package for yum, a software installer which is much easier to use than rpm. Type in "yum" to the search box. You get a lot of results, click on "yum-2.2.1-0.fc3.noarch.html". Click on the big link at the top of the page for "yum-2.2.1-0.fc3 RPM for noarch" and download the file. Now we will install it - make sure you are root.

$ rpm -ivv yum-2.2.1-0.fc3.noarch.rpm

rpm includes a query interface. Find out what version of yum is installed:

$ rpm -q yum
yum -2.2.1-0.fc3

Find out which rpm provided a particular file:

$ rpm -q --file /sbin/init
SysVinit-2.85-34

There are many other options for querying. If you decide you don't want yum installed anymore, uninstall it:

$ rpm -e yum

Now that we have yum, we can often skip looking at the rpmfind.net website and just install what we want using just the name of the package. We'll install xsnow:

$ yum install xsnow
  Traceback (most recent call last):
  File "/usr/bin/yum", line 6, in ?
    import yummain
  File "/usr/share/yum-cli/yummain.py", line 23, in ?
    import yum
ImportError: No module named yum

Whoa! That didn't work. Let's follow rule number one for when something doesn't work in Linux: cut and paste the most unique part of the error message into Google. Clicking on the first link gets someone who has the same problem. Click on "Next message" at the bottom and we get an answer:

rhel3 doesn't have python 2.3 - which the yum 2.2.0 noarch rpm was built
for. You need to rebuild the src.rpm for rhel3 and the problem will go
away.

So now we remove the yum package we installed:

$ rpm -e yum

Go back and get the source rpm for this version of yum and install it:

$ rpm -ivv yum-2.2.1-0.fc3.src.rpm

Cd to the rpm specification directory and build the rpm:

$ cd /usr/src/redhat/SPECS
$ rpmbuild -bb yum.spec

Now install the rpm file we just built:

$ rpm -ivv /usr/src/redhat/RPMS/noarch/yum-2.2.1-0.fc3.noarch.rpm

Now install xsnow:

$ yum install xsnow

We get a message about importing GPG keys. Using Google, we find that we can import the keys we need thusly:

$ rpm --import http://www.fedora.us/FEDORA-GPG-KEY
$ rpm --import http://www.fedora.us/FEDORA-LEGACY-GPG-KEY

Try again:

$ yum install xsnow
Setting up Install Process
Setting up Repos
No Repositories Available to Set Up

Argh! Still foiled! But reading the man page for yum tells us to look at http://linux.duke.edu/yum/, which tells us... that there are simply too many repositories to list any more, so ask a mailing list or a friend. In this case, we will copy my list of repositories, which I got from people in LinuxChix.

# cd /
# tar xzvf yum_repos.tar.gz

Try again:

$ yum install xsnow

Now run it:

$ xsnow

And let it snow!

Some documentation on setting up repositories is here.

Unpackaged software

If you can't find a packaged version of software for your use, you will have to download it and compile it yourself. This is done in the same basic way as for other UNIXes.

$ tar xzvf source.tar.gz
$ cd source/
$ ./configure
$ make
$ make install

Download the gnuplot source from the Sourceforge web site and follow the above steps to compile and install it.

Software distribution sites

The major open source software distribution sites are:

Linux kernel development

Online resources

There are many online resources available to help people who need to change the Linux kernel, but most of them are out of date.

There are many books available as well, but they are even more out of date. In general, reader beware. Often the best documentation is in the kernel source tree itself, in /usr/src/linux/Documentation.

Compiling the kernel

The canonical source for the Linux kernel is available at http://www.kernel.org in the form of compressed tarballs. You can also get it in packaged form (RPM) or through a CVS repository clone. Fortunately for us, the source is already installed.

$ cd /usr/src/linux-<version>

Normally, you would copy a pre-made configuration file into the local directory:

$ cp configs/kernel-2.4.21-i686.config .config

However, this configuration file has almost everything turned on, and so compiling the kernel takes a very long time (around an hour). We will use a cut-down configuration file instead.

The kernel is configured by running "make menuconfig".

$ make menuconfig

Turn on profiling support so we can run oprofile later. Select "Profile support" and then turn on Profiling support and hit "y" to include OProfile system profiling. Exit saving your configuration. Now start compiling. This will take a long time.

$ make bzImage && make modules && make modules_install

While the kernel is compiling, set up the bootloader for a new kernel. Edit /etc/grub.conf and copy the last 4 lines. Change the title and path to the kernel and initrd:

title My Linux
	root (hd0,0)
	kernel /my_vmlinuz ro root=LABEL=/
	initrd /my_initrd

Now put your entry before the old entry so that it will boot automatically.

After your kernel and modules are finished compiling, make a new initrd and copy your new kernel and System.map into place:

$ mkinitrd /boot/my_initrd 2.4.21-27.ELcustom
$ cp arch/i386/boot/bzImage /boot/my_vmlinuz
$ cp System.map /boot/System.map-2.4.21-27.ELcustom

Now reboot! If the system does not boot with your new kernel, select the old entry in the grub screen shortly after boot.

Writing your own module

We will now write and compile a minimal kernel module. Follow this link for directions.

Useful Linux tools

This section is a random assortment of useful Linux tools.

dmesg

A lot of information is printed to the kernel message log.

$ dmesg | less
$ dmesg | grep eth0

/proc

/proc is a pseudo-file system that contains many interesting pieces of information that can be read using cat or less.

$ cat /proc/meminfo

/proc/sys

/proc/sys is another pseudo-file system that allows you to read and write the values of many kernel variables.

$ cat /proc/sys/net/ipv4/ip_forward
0
$ echo 1 > /proc/sys/net/ipv4/ip_forward
$ cat /proc/sys/net/ipv4/ip_forward
1

This machine will now forward third party IP packets. To set these values permanently, edit /etc/sysctl.conf (there is a man page for more information).

System Request key (magic sysrq key)

You can ask the system to print out useful information such as current memory usage or a list of tasks and their status with a direct request to the kernel using the "magic sysrq key". First you must enable it:

$ echo 1 > /proc/sys/kernel/sysrq

Now ask it to do something. Originally, it was easiest to use the sysrq on the keyboard, but now it is often overloaded as something else, so the easiest way is to use the /proc/sysrq-trigger:

$ echo h > /proc/sysrq-trigger
$ tail dmesg
[...]
SysRq : HELP : loglevel0-8 reBoot Crash tErm kIll saK showMem powerOff showPc unRaw Sync showTasks Unmount

Try a few of the less dangerous ones, like t to show all tasks:

$ echo t > /proc/sysrq-trigger
$ dmesg | less

lscpi

When trying to discover what hardware is installed on your system, lspci can at least tell you the manufacturer and name of all the PCI devices on your system.

$ lspci
$ lspci -vvv

Kernel command line

A great deal of the kernel's behavior depends on the kernel command line - the arguments the kernel is booted with. To alter the command line, edit the "kernel" line in /etc/grub.conf - everything after the kernel image filename is the kernel command line.

My favorite is a way to get root access on most Linux machines. You should of course only do this if it is legal, moral, ethical, etc. Reboot the Linux machine, and stop the automatic grub boot. Edit the command line by typing "a" in the grub boot screen and add to the command line:

init=/bin/bash

Then boot. This will drop you directly into a shell when init begins. First we will mount the /proc file system, since nothing other that the root file system is mounted at this point and /proc is necessary for most commands to work:

# mount /proc /proc -t proc

Now remount the root file system read-write and edit /etc/shadow:

# mount -o remount,rw /
# vi /etc/shadow

Your editor will complain about the file being read only, but that is fine. Make a copy of the line for root, comment it out, and remove the password field from the uncommented line so it looks like this:

root::12734:0:99999:7:::
#root:[encrypted password]:12734:0:99999:7:::

Reboot the machine and you will be able to login as root without a password. Be sure to put it back the way you found it as a courtesy. (The password program will complain about easy to guess passwords but will allow you to change it to one anyway.)

# passwd

You can do a lot more than get a login using the kernel command line. You can set the maximum amount of memory or cpus, set the root device, set debugging level, and much more. See the file Documentation/kernel-parameters.txt in your kernel source directory for details.

Learning more

Linux has a wealth of freely available documentation. Here are a few options for learning more.

Extras

Pthreads

Threading in Linux has always used the POSIX threads (pthreads) specification interface (with some minor changes for correctness). (Solaris has an older threading interface and now supports the POSIX threads interface as well.) For a very simple example of program using pthreads, download and extract this file.

Oprofile

oprofile is one of the most powerful profiling tools but also one of the most difficult to use. It uses a kernel facility to take samples periodically of what is going on the system. This data is then analyzed by various commands.

Start oprofile with the location of the uncompressed kernel image:

$ opcontrol --start --vmlinux=/usr/src/linux-2.4/vmlinux

Now dump the collected information for analysis:

$ opcontrol --dump

We have a variety of tools to use at this point - try typing "op" and hitting tab to see a few. Let's start with timing information:

$ op_time

Now let's profile a user application to see how much time it spends in each part of the program. If you haven't already done so, download and extract the newmemory program. Cd into the new directory:

$ cd newmemory

Edit the Makefile to build the application with debugging information:

newmemory: newmemory.c
        gcc -g -o newmemory -Wall -lpthread newmemory.c

Rebuild the application:

$ make clean
$ make newmemory

Before we start running the application, reset the oprofile information and restart it:

$ opcontrol --reset
$ opcontrol --start

Now run the applicaiton for a while so we can gather some samples:

$ ./newmemory -s 100 -f -t 50

Wait for a few minutes to get a significant number of samples, then dump the information:

$ opcontrol --dump

Check to see how many samples we got that were in our application:

$ op_time
[...]
45244     94.4709  0.0000 /root/newmemory/newmemory

Good, that's a lot of samples. Now we will annotate the source file with information about how often the program was executing at that line when oprofile sampled it.

$ op_to_source --source-dir /root/newmemory/ --output-dir /root/oprofile_out /root/newmemory/newmemory

Now look at the file in the output directory:

$ ls /root/oprofile_out
newmemory.c
$ less /root/oprofile_out/newmemory.c

Let's look at the first function:

               :static int
               :rand_num(int max)
     30 0.985% :{^M /* rand_num total:     273 8.971% */
    216 7.098% :        return (random() % max);
     27 0.887% :}

On the left are the number of samples and percentage of total samples that were taken at that line. All samples for this function are aggregrated and reported at the comment at the beginning of the function. As we can see, the program spends about 9% of its time in this function, most of it spent in the random() library call.

We can sort the lines of output to find out where most of the time is spent:

$ sort -r -n /root/oprofile_out/newmemory.c | head

You should see this line at the top of the list with something like 65% of total samples::

if (loc != *loc) {

This program is intended to find memory corruption (usually at the software level), so it steps through memory randomly and reads each location. Since the access pattern is random, each time it reads a memory location it almost certainly is not in cache (or in the TLB) and so it is expected that we are spending most of our time in this part of the code.

See /usr/share/doc/oprofile-0.5.4/oprofile.html for more information on oprofile.