Overview of Build Systems

This is a very short overview of build systems used in Linux or in general
GNU softwares. These tools are also ported to Windows, but generally only GNU
tools ported to windows, which makes use of cygwin or mingw use these build
systems. Tools aimed at windows only systems usually use Visual Studio build
environments.

So quick description of basic terminologies:

Build: It usually means compiling the source code and generating binary
(executable) file. Optionally it contains features to scan host system (system
on which code is being compiled) and make changes in source code accordingly.
Example would be GUI in Linux. Linux supports multiple GUI libraries eg: GTK,
QT. So build system may scan host to determine which library should be used for
compiling this code.

Installer: This program installs the generated executables, shared libraries and
other build outputs to some location, so that OS can find it easily while
executing the programs. Sometimes installer is part of build system itself,
 while some times build and installer comes in pair.

 Now let us see some very popular build systems:
 1. Shell Script: Used for very small and very simple tools or utterly
 complex builds. For simple builds, it just contains few lines to compile the
 code files and generate and install binaries.

 2. Makefile: Most common build system on unix and related environment. Makefile
 contains rules about source files and their dependency with each other. It
 effectively makes a dependency tree. Also, once built, it can detect which
 files have been changed in subsequent builds and compile only those.

 3. CMake: I am not too familier with it, so I'll just mention that it is
 another build system.

 4. SCons: Build system created in python. Gives you flexibility of specifying
 rules like Makefile and also add python code to make very complex build
 systems.

 5. Configure Scripts: These are not truelly build systems, but rather pair up
 usually with Makefiles. These scripts are very versatile in the sense they detect
 host environment and change Makefiles accordingly. Can be modified to make a
 generic build system, which will fit multiple host environments very smoothly.

 6. Automake and Autoconf: These are tools to generate Makefile and configure
 scripts from specification files. In rest of this post, we'll talk about these
 two with few examples.

Building source code:
 These steps are common for most of the codes that make use of configure script 
 and Makefile for building. Change in command line might be needed to change
 default build options.

 1. Make configure script executable if it is not already: chmod +x ./configure
 2. ./configure (optionally --prefix=/custom/install/folder)
 3. make
 4. make install
 5. make clean (only if you want to rebuild everything next time)


Autoconf: As name indicates, it is a tool used to generate configure scripts
from some options file. Usually this options file is named configure.ac or
configure.in. It'll have multiple macros having "AC_" prefix. Most interesting
ones are AC_ARG_ENABLE and AC_ARG_WITH. Here is what GNU manual says about theses
macros:
AC_ARG_ENABLE: "If a software package has optional compile-time features, the
user can give configure command line options to specify whether to compile them.
The options have one of these forms:
--enable-feature[=arg]
--disable-feature"

Structure of this macro is: AC_ARG_ENABLE(feature, help-string, [action-if-given],
[action-if-not-given]). Here action-if-not-given simply means default value.

AC_ARG_WITH: "Some packages require, or can optionally use, other software
packages that are already installed. The user can give configure command line
options to specify which such external software to use. The options have one of
these forms:
--with-package[=arg]
--without-package"

Structure of this macro is: AC_ARG_WITH(package, help-string, [action-if-given],
[action-if-not-given]). Here action-if-not-given simply means default value.

So let's see first example of Autoconf options file:

1. Bash:
I am using bash 4.2 source for this example. Source availble at
http://ftp.gnu.org/gnu/bash/bash-4.2.tar.gz. It has a configure.in file. Lets
open it. As I said, it contains lots of AC_ macros. Description of those can be
found at GNU site. First AC_ARG_WITH macro is present on line number 106. We can
see which optional packages bash has like afs, bash-malloc, gnu-malloc etc. Help
string is the string displayed as description about that package. First
AC_ARG_ENABLE macro is found on line 211. Features like alias, array-variables, debugger are specified here. Interestingly none of these macros had default
value specified. Lets find out where it is specified. eg To find default value
of debugger feature, let's see action associated with it. Third argument of
macro is opt_debugger=$enableval, which means enable value of this feature is
stored in opt_debugger. Simple grep command 'grep -n "opt_debugger"
configure.in |less' on configure.in shows its default value is set on line
number 184 as yes. In configure.in file surrounding lines set default value for
other features. Similar search on curses package shows default package values
are initialized around line 58.

2. Gnash:
Now we'll see confgure file for gnash, the open source flash player. I am
using version 0.8.9 from ftp://ftp.gnu.org/gnu/gnash/0.8.9/. It has a
configure.ac file. Simple grep shows it has 36 features and 13 packages
configureble through config script. Few of them have default value in macro,
others use shell scripting lines to decide values.

3. Nautilus:
The reason behind understanding structure of autoconf file is that all features
and packages may not have enough description. So if one wants to probe for
enabling/disabling specific functionality he can easily look into these macros
and corresponding shell variables to see its effect on output of configure file.

So, suppose we want to add configure option of enabling/disabling debug flag
during compilation of nautilus , we can add following lines to corresponding
configure.in files and generate configure file from it.

I have added following option to line 110 of configure.in after option for
enabling debug.

AC_ARG_ENABLE(
[debug],
[  --enable-debug            Enable Debug Flags (default: enabled)],
[case "${enableval}" in
      yes) CFLAGS="$CFLAGS -g -O0" ;;
    no) CFLAGS="$CFLAGS -g -O3" ;;
    *) AC_MSG_ERROR([bad value ${enableval} for --enable-debug option]) ;;
esac],
CFLAGS="$CFLAGS -g -O0")

This option when set, makes CFLAGS(used when compiling .c files) to be -g and
-O0 thus enabling debug output and disabling all optimizations.
Generate configure script using 'autoconf'. Now if we type ./configure --help,
we can see enable-debug option in it. If we do ./configure --enable-debug, and
make, CLFAGS used while compiling .c files will have -g -O0 arguments in it.
Thus debug version of nautilus will be built.

If we want to show  enbaled-disabled debug feature after completion of configure
script, we can add following lines at end of configure.in file. That part
usually contains echo statements for other features. We can add status of our
own feature by adding

if test x"echo $CFLAGS | grep \"-g -O0"= "x"; then
        echo "  Debug disabled"
else
        echo "  Debug enabled "
fi

at very end of the file. Generate configure file again, and configure it. On
successful configuration it'll show the message Debug disabled/enabled on last
line of output.

Automake: As you can guess from name, this is used to generate a Makefile from
provided template, usually specified in Makefile.am file.

Important thing to note here is nomenclature of macros defined in Makefile.am
files. It is usually specified as "folder name"_"description" or "file
name"_"description" where file/folder name is name of file to be built or name
of the folder relevent files should be copied into.

So e.g. "bin_PROGRAM = asd" means binary executable named asd should be built
(PROGRAM) and copied into "bin" folder.
Here are few other common macros:
1. bin_PROGRAM : binary name to be built and copied to bin folder.
2. libexec_PROGRAM : binary name to be built and copied to libexec folder.
3. lib_LIBRARIES : library name to be built and copied to lib folder.
4. lib_LTLIBRARIES : shared library name to be built and copied to lib folder.
5. noinst_PROGRAM : just build program but don't copy it anywhere
6. asd_SOURCES : Now this important one. Here we specify source files for
building asd (binary)
7. asd_a_SOURCES : sources for building asd as static library.
8. asd_la_SOURCES : sources for building asd as shared library.
9. EXTRA_asd_SOURCES : Source files usually for conditional compilation.
10. asd_LDFLAGS : linker flags for building asd.
11. asd_CFLAGS : C flags for building asd.
12. asd_LDADD : libraries to be linked while building asd
13. EXTRA_DIST : copy all mentioned files/folders to installation path.
14. SUBDIRS : List of subdirectories to be traversed while 'Making' package. May
or may not contain Makefile.am files.

As we can see from these macros, it is very easy to alter Makefiles by finding
required macros in Makefile.am and tweaking few macros to change flags, add or
remove sources or do something else.

Important thing is to regenerate Makefile using automake, you must go to parent
folder which contains config.ac or config.in and then execute following
commands:
aclocal
automake

Lets see few example Makefile.am now.

1. AS31
First package we will see here is AS31 an assembler and compiler for 8051
code. It is available on http://www.pjrc.com/tech/8051/.
So Makefile.am in root source dir contains just SUBDIR macro pointing to 'as31
example' to indicate these folders will contain Makefile.am or Makefile for
actual code and examples.

Now Makefile.am in as31 folder contains bin_PROGRAMS having value as31 and
conditionally as31gtk. Both these binary targets have their source file macro
defined with list of code files to be compiled for its generation. There is also
one LDADD macro to indicate linking of libraries.

Makefile.am in examples directory only contains EXTRA_DIST macro to move those
files to installation location.

2. Extreme Tux Racer:
A 3D game available at http://extremetuxracer.com/.
As with previous case its root Makefile.am contains SUBDIR and EXTRA_DIST macro.

Makefile.am in src is also trivial. Contains one bin_PROOGRAM and huge list of
source code files for it.

Now data directory does not contain Makefile.am at all but simple Makefile
indicating it is not to be regenerated through Automake. Modifying such Makefile
has to be done manually as far as I know.

3. Nautilus:
File manager for GNOME. Availalble at
http://ftp.gnome.org/pub/GNOME/sources/nautilus/.

As expected root Makefile.am contains list of subdirectories along with few
extra dist folders.

In libnautilus-extension, we see a new directive named includedir and
include_HEADER which means header files under $(includedir) should be installed
or copied to install location. Usually done for installing header along with
libraries for developers.

In src folder, there is a directive libexec_PROGRAM indicating binary to be
copied to libexec folder rather than usual bin folder.

So that's all I have for basic build info on open source softwares. Please
comment if there is any specific topic missing here or any peculiar
configuration files you came across and wish to be explained here.

                                                                                                              ASD

Tools

    Obviously first post will be tools needed for reading codes. Earlier I simply
used to untar the source and start reading from some random files, but now
I think tools are equally necessary at-least for beginners. So here is list of
my favorite tools. Most of these are also available on windows with some good
additions (like Visual Studio 2010 editor + symbols search).

1. Geany: http://www.geany.org/ My editor of choice. It is very light-wight,
shows functions, structure, classes and globals on left hand panel. It has a
drawback that it doesn't show symbols from other files, but with it works for me
with combination of other tools.

    I used to use Vim before, and at some point may again go back to it. I still use
it as default browser for Cscope. Vim with plugins may have all capabilities of
geany (with infinite advanced features of-course).

2. Cscope: http://cscope.sourceforge.net/ An awesome tool to extract almost
anything from your code. Works best with C, but as we go to C-like languages
(C++, Java and othrs) it starts giving hitches. There is a tool to generate
python index for Cscope. I haven't tried it yet, but if it is good enough,
Cscope should work smoothly for python too.

3. Cscope-Vim plugin: http://cscope.sourceforge.net/cscope_vim_tutorial.html. Vim already has support for Cscope, but this plugin reduces commands to just a
combination of keys. Also keeps track of your jumps in code.

4. Doxygen: http://www.stack.nl/~dimitri/doxygen/ Call graph generator for all
languages. Perticularly useful when we need to trace path between two points in
code path.

5. grep/perl/python: Very useful for search/modifications which are easily
codable and difficult to do from above tools. I am not very familier with perl,
but it is part of my current in-progress-learnings.

6.GDB: Default linux debugger. Easy to use once we know basic commands. As with
all debuggers, it may miss breakpoints due to non-loaded libraries,
optimizations, multithreading etc.  I perticularly like putting breakpoints
using "INT 3" so that none of them are missed.

About

    This is my first tutorial ever. My inspiration was my own struggle to read src
codes. I am still bad at it, and I hope that I'll find better methods through
comments. But I have learned a few things by now, and I hope sharing them would be useful since as far as I know there are no good articles on this topic yet.

    I'll try to cover how to read src code of any tool or in general any src code
available. Rather than reading whole code in one shot, I'll focus on some feature
(of my interest) and try to tweak it a little or sometimes add new feture
altogether. Given that most of the codes for Linux are written in C/C++, most
of the tutorials will  focus on them. But I'd like to find some python code and
work on it. All suggestions on this are welcome. As I said I'll take source from
linux tools, but if any interesting Windows src is there, I'd love to work on it.

P.S. The name of blog is illogical expansion of my favorite character string ASD.

                                                                                                   ASD
                                                                             (Anonymous Source Dissector)