ModularDevelopersGuide

The X.Org Modular Tree Developer's Guide

Contents

  1. The X.Org Modular Tree Developer's Guide
    1. Introduction
    2. Required Tools
      1. Minimum requirements
      2. Suggested packages
      3. Additional requirements for developers
    3. Getting the modular tree
      1. From the Git repository
      2. From tarballs
        1. Mixing 7.1 and 7.0 tarballs
    4. Building the modular tree
      1. Environment variables
      2. Understanding dependencies
      3. Building all modules with the build scripts
      4. Building all modules with jhbuild
      5. Building individual module components by hand
      6. Building only specific modules
      7. Compiling after updates
      8. Making tarballs
      9. Standard configuration options
    5. Using the modular tree
      1. X libraries
      2. X applications
      3. Running the X server
    6. Crosscompiling
    7. New Modules

Introduction

This guide is intended to be a repository of useful information that builders and developers need to know when working on the modular X.Org source tree. As new guidelines are developed, they should be added to this doc.

A more summarized guide to building X.org is at CompileXserverManually.

Required Tools

This section lists the tools required for various people involved with the project.

Minimum requirements

The following tools are required for all builders and developers:

(click here for a per-distribution list)

Suggested packages

The following external dependencies are recommended for a fully functional build:

If there is a compile error in libXFont, it is likely because libXfont-X11R7.1-1.1.0 is incompatible with recent versions of freetype2 libraries. Installing http://xorg.freedesktop.org/releases/individual/lib/libXfont-1.2.0.tar.bz2 will fix that.

If you're building Xorg with XCB support and want to use xcb-utils then you need GNU gperf 3.0.1 or newer.

Additional requirements for developers

In addition to the minimum requirements listed above, the following tools are required for developers:

You can check the autoconf version by running autoconf --version and similarly for the rest.

Getting the modular tree

In this section and the next, we assume you are working in /tmp with the sources in /tmp/src and the installed tree in /tmp/modular.

From the Git repository

Note: X.Org has moved all currently maintained modules to the Git repository. See the GitPage for more information and UsingGit on how to use Git.

The build.sh script is provided in the xorg/util/modular repository. It provides a --clone flag to get the modules required. To get the initial checkout of all modules, run: 

$> cd /tmp/src
$> git clone git://anongit.freedesktop.org/git/xorg/util/modular util/modular
$> ./util/modular/build.sh --clone -a -n /tmp/modular

This will clone all modules into the right directory structure. The -a flags tells the script to skip automake and hence the build will fail. The -n flag tells the script to continue regardless. Both flags together will result in a clean checkout of every module.

Alternatively, run the above command without the -a and -n flags and the script will build everything into the prefix provided.

From tarballs

If you are building from the release tarballs, we assume that you have downloaded the tarballs into /tmp/tarballs and will be installing everything into /tmp/modular. The tarballs for the release are arranged into a hierarchy similar to the CVS modules. For example, you can find the tarballs for each of the app module components in the app subdirectory. In addition, there is another directory called everything that contains all of the tarballs included in the other modules for convenience.

For each tarball, there are two versions: one compressed with gzip (e.g., util-macros-1.0.0.tar.gz) and one compressed with bzip2 (e.g., util-macros-1.0.0.tar.bz2). You will only need one, so to save yourself some download time, you can download either the .gz or the .bz2 files. The tarball will need to be unpacked first.

To unpack a gzipped tarball: 

$ gzip -d < xorg-macros-1.0.0.tar.gz | tar xf -

To unpack a bzip2'd tarball: 

$ bzip2 -d < xorg-macros-1.0.0.tar.bz2 | tar xf -

The modular tree should now be ready to build.

IMPORTANT NOTE: If you are building 7.1 from tarballs, make sure you have all of 7.0 available too. build-from-tarballs.sh relies on files that are not present in the 7.1 source release directory.

Mixing 7.1 and 7.0 tarballs

If you have obtained the everything directories from the 7.1 and the 7.0 distributions, and you have the build-from-tarballs.sh script, you can use http://wiki.x.org/wiki/x-merge.scm?action=raw to merge the files into a single directory. Note that you'll need to rename the x-merge.scm script if you download it with wget. (Tip: wget --user-agent="Mozilla/5.0" <url>)

To use, be in a directory holding the build-from-tarballs.sh file and two subdirectories 7.1 and 7.0 with the tarballs for each release, and then run guile -s x-merge.scm. You will finish up with a new subdirectory called build-dir which you can go into and then run the build-from-tarballs.sh script as explained below.

Alternatively you can edit the top of the x-merge.scm file for your own setup.

Building the modular tree

There are multiple ways to build the module components.

For each of these build methods, you may build either from the Git tree or the release tarballs. In the sections below we will explain how to build, install and package the modular tree components using each of these methods.

Environment variables

The following environment variables should be setup before building: 

PKG_CONFIG_PATH=/tmp/modular/lib/pkgconfig
PATH=/tmp/modular/bin:$PATH
ACLOCAL="aclocal -I /tmp/modular/share/aclocal"
LD_LIBRARY_PATH=/tmp/modular/lib/

Failure to set up these variables may result in build errors or building against system-installed headers. In some cases this may result in a loss of functionality.

Understanding dependencies

It is important to note that the module components have dependencies either on other module components or external packages. These dependencies determine the build order that you must follow. For example, all the module components depend on the macros module component (util/macros in Git or util-macros from the pre-packaged tarballs), so it must be built before any of the other module components. The build script and jhbuild both take care of these dependencies for you.

If you insist on building the modules by hand, you must do the dependency ordering yourself. It is recommended that you inspect the build scripts to determine the proper order. This is a lot of work, so building by hand is not recommended.

Building all modules with the build scripts

The first build method uses a script to automate the build process for each module component in the order required by their various interdependencies. For details on what each of these steps are, see the Building individual module components by hand section below.

If you are building from Git and have a complete tree checked out as outlined above, then you can build using the build.sh script in the util/modular directory. There are several options for this script (see below) but the only one required is the prefix that will be passed to each module component through the --prefix= command-line option. The options to the build.sh script are: 

Usage: ./build.sh [options] prefix
  where options are:
  -a : do NOT run auto config tools (autogen.sh, configure)
  -b : use .build.unknown build directory
  -c : run make clean in addition to others
  -d : run make distcheck in addition to others
  -D : run make dist in addition to others
  -f file: append module being built to file. The last line of this
           file can be used for resuming with -r.
  -g : build with debug information
  -n : do not quit after error; just print error message
  -o module/component : build just this component
  -p : run git pull on each component
  -r module/component : resume building with this component
  -s sudo-command : sudo command to use
  --clone : clone non-existing repositories (uses $GITROOT if set)
  --check : run make check in addition to others

Two of the more commonly used options are -d and -n. We recommend specifying the -d option to run make distcheck on each module component. By default the build.sh script will exit when it encounters a build failure, but with the -n command-line option, you can have the script print an error message and continue on with the next build step instead of immediately exiting.

This will build the entire source tree in the appropriate build order and it will run make distcheck on each module component. If any errors occur, it will print an error message instead of exiting. You can search the log (e.g., grep '*****' build.log) to see if any errors occurred during the build.

Note: as of 2009-01-18 (commit titled "Kill off --with-mesa-source completely"), the server will pick up the installed headers it needs from mesa.

By default, build.sh always starts from the first module when invoked. The -r option resumes from the given module. E.g. to resume building from libX11, run 

$> build.sh -r lib/libX11 /tmp/modular

It is recommended to use the -f flag to record the last module building into a file. This allows for easy resume after a build failure. The following command records the built module into a file "built.modules" and resumes from the last component recorded in that module. 

$> build.sh -f built.modules -r `tail -n 1 built.modules` /tmp/modular

Re-run this command repeatedly until all build errors have been fixed.

If you are building from the released tarballs and have downloaded all of the tarballs, then you can build the release with the build-from-tarballs.sh script. You can find the most recent version of the script here.

Most of the options to this script are the same as the build.sh script described above. The options to the build-from-tarballs.sh script are: 

Usage: ./build-from-tarballs.sh [options] prefix
  where options are:
  -d : run make distcheck in addition to others
  -D : run make dist in addition to others
  -c : run make clean in addition to others
  -m path-to-mesa-sources-for-xserver : full path to Mesa sources
  -n : do not quit after error; just print error message
  -s sudo-command : sudo command to use
  -bz2 : use tarballs with bzip2 compression (default)
  -gz : use tarballs with gzip compression
  -e : build from witin the 'everything' dir instead of module dirs

By default the build-from-tarballs.sh script will build the packages from the module subdirectories (e.g., app, data, doc, etc.); however, if you downloaded the everything dir instead of the individual module subdirs, then you can use the -e option to build from the everything dir. By default it will build the packages from the module subdirectories. Note that when you are building from the everything dir, you need to run the build-from-tarballs.sh script from within the everything dir, and when you are building from the module subdirs, you need to run the script from the directory containing the module subdirs.

The other important option is to specify whether you want to use the gzippped or the bzip2'd tarballs. By default it will use the bzip2 tarballs, but you can override this with the -gz option.

For example, If you have downloaded all of the gzipped tarballs in the module subdirs into /tmp/tarballs as well as the build-from-tarballs.sh script, then you can build the release as follows: 

$ cd /tmp/tarballs
$ PATH=/tmp/modular/bin:$PATH ./build-from-tarballs.sh -m /tmp/Mesa-6.4.1 -n -gz /tmp/modular > build.log 2>&1

Building all modules with jhbuild

The second build method uses jhbuild, a program by James Henstridge that can be used to automatically download module components from Git or from tarballs and then build them in the correct dependency order.

JhBuildInstructions explains how to build the modular tree from Git/CVS or from the tarballs.

Building individual module components by hand

The third build method is to do everything manually. This is especially useful when you already have a modular Xorg installed and only want to build some newer modules from source. See below for more information.

As noted above, everything depends on the Xorg m4 macros, so we will explain how to build it first.

If you are building from Git and followed the instructions above, you have a modular tree ready to use in /tmp/src, the macros module components can be built, installed and packaged as follows: 

$ cd /tmp/src/util/macros
$ ./autogen.sh --prefix=/tmp/modular
$ make
$ make install

The first step above ./autogen.sh --prefix=/tmp/modular does two things: it creates the configure script and then it runs it using the command line options given to autogen.sh. This will create all of the Makefile's used to build, install and package the files. The --prefix=/tmp/modular option is a standard configuration option to tell the autotools where to install the files. This is very similar to defining ProjectRoot in your host.def file for monolithic tree builds. By default, the prefix is /usr/local, but since we are installing everything in /tmp/modular, we must override it with the --prefix option. There are many other configure options (some of which are unique to each individual module component). You can list these configure options by running ./configure --help after running autogen.sh to create the configure script.

Normally, the make step would build everything in that module from the source files. For modules like app or lib, this step will build the application or library from the sources. However, for the macros module component, there is nothing to build, since the m4 macro files are simple text files that need to be installed. make it will say that there is nothing to be done. This is normal for components that only need to install.

The next step is to install the files we built in the previous step. make install will create the directories in /tmp/modular as needed and then install the macros.

If you are building from the tarballs, then you must first unpack the util-macros tarball using the methods listed in the previous section, and then the package can built and installed as follows: 

$ cd util-macros-1.0.0
$ ./configure --prefix=/tmp/modular
$ make
$ make install

The first step is to change to the directory that you just unpacked the util-macros tarball to. Note that the directory you change to can easily be determined from the tarball name. For example, if the tarball is named package-1.2.3.tar.gz, then the directory to change to is package-1.2.3.

The second step is very similar to the ./autogen.sh step when building from Git. In this case, it is not necessary to run autogen.sh since the configure script was already built when the package was created. The options to the configure script are the same as the ones used if building from Git. Also, as noted above, there are many options to configure, which can be listed by running ./configure --help. The remaining two steps will build and install the package as described above when building from Git.

Before building subsequent modules, two environment variables need to be set: 

$ export PKG_CONFIG_PATH=/tmp/modular/lib/pkgconfig
$ export ACLOCAL="aclocal -I /tmp/modular/share/aclocal"

Subsequent module components can now be built by running: 

$ cd theComponentDirectory
$ ./autogen.sh --prefix=/tmp/modular    ### or if using tarballs: ./configure --prefix=/tmp/module
$ make
$ make install

If you're missing any dependencies you'll get an error message during the configuration stage.

You'll generally want to start with the 'low-level' proto and lib modules before building the xserver. See the build.sh script for a detailed build ordering for the components.

For larger components such as libX11 and the X server you may need additional autogen/configure options. After running autogen.sh you may want to run ./configure --help to see if there's other important options. You might also look at the build.sh script for other tips.

Building only specific modules

If you already have a modular Xorg installed you don't need to rebuild everything from source. If, for example, you only want to work with a specific driver module there is no need to rebuild all the libraries, fonts, documentation etc.

Assuming that a modular Xorg comes with your distribution, make sure that you have the development packages installed. They provide SDK headers and pkgconfig files.

Driver module ABIs may have changed since the Xorg version that you have installed. For example, the video driver ABI changed between Xorg 7.0 and 7.1. In that case, in order to build and test a new video driver module, you would have to build and install a new version of the Xserver first. That, in turn, probably requires updated versions of a few proto-packages. Running autogen.sh or configure will tell you which versions of which modules you are missing.

Compiling after updates

If you update your source code with git pull to get the latest changes you can often recompile the module by just running make (and make install). If the changes are more extensive (files added/removed) you'll need to re-run autogen.sh or configure first.

Making tarballs

As part of the development process, it's important to create tarballs to be sure things work properly.

Creating the packages for the various components is very easy with the autotools — simply run make dist or make distcheck. We always recommend using distcheck since it will both create the package as well as test it to make sure that it works properly. It can also run a testsuite, if one is available for it. Since the configure.ac file has been set up to build both gzip and bzip2 tarballs, the packages created are util-macros-1.0.0.tar.gz and util-macros-1.0.0.tar.bz2 in the example above.

Note that if you are only interested in using the component and not actually developing it, then you can skip this step, but for all developers, this step is required.

Standard configuration options

The standard configure script options (passed to ./configure or ./autogen.sh) are: 

Configuration:
  -h, --help              display this help and exit
      --help=short        display options specific to this package
      --help=recursive    display the short help of all the included packages
  -V, --version           display version information and exit
  -q, --quiet, --silent   do not print `checking...' messages
      --cache-file=FILE   cache test results in FILE [disabled]
  -C, --config-cache      alias for `--cache-file=config.cache'
  -n, --no-create         do not create output files
      --srcdir=DIR        find the sources in DIR [configure dir or `..']
Installation directories:
  --prefix=PREFIX         install architecture-independent files in PREFIX
                          [/usr/local]
  --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
                          [PREFIX]
By default, `make install' will install all the files in
`/usr/local/bin', `/usr/local/lib' etc.  You can specify
an installation prefix other than `/usr/local' using `--prefix',
for instance `--prefix=$HOME'.
For better control, use the options below.
Fine tuning of the installation directories:
  --bindir=DIR            user executables [EPREFIX/bin]
  --sbindir=DIR           system admin executables [EPREFIX/sbin]
  --libexecdir=DIR        program executables [EPREFIX/libexec]
  --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
  --sharedstatedir=DIR    modifiable architecture-independent data [PREFIX/com]
  --localstatedir=DIR     modifiable single-machine data [PREFIX/var]
  --libdir=DIR            object code libraries [EPREFIX/lib]
  --includedir=DIR        C header files [PREFIX/include]
  --oldincludedir=DIR     C header files for non-gcc [/usr/include]
  --datarootdir=DIR       read-only arch.-independent data root [PREFIX/share]
  --datadir=DIR           read-only architecture-independent data [DATAROOTDIR]
  --infodir=DIR           info documentation [DATAROOTDIR/info]
  --localedir=DIR         locale-dependent data [DATAROOTDIR/locale]
  --mandir=DIR            man documentation [DATAROOTDIR/man]
  --docdir=DIR            documentation root [DATAROOTDIR/doc/util-macros]
  --htmldir=DIR           html documentation [DOCDIR]
  --dvidir=DIR            dvi documentation [DOCDIR]
  --pdfdir=DIR            pdf documentation [DOCDIR]
  --psdir=DIR             ps documentation [DOCDIR]

Program names:
  --program-prefix=PREFIX            prepend PREFIX to installed program names
  --program-suffix=SUFFIX            append SUFFIX to installed program names
  --program-transform-name=PROGRAM   run sed PROGRAM on installed program names

Optional Packages:
  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
  --with-release-version=STRING
                          Use release version string in package name

More information about the configure script can be found from the autoconf info pages.

Using the modular tree

This section describes how to use the newly compiled X libraries, programs and X server.

X libraries

The X libraries which are built will be in /tmp/modular/lib/.

To use them you have to configure the linker to find them. On GNU/Linux systems you do: 

export LD_LIBRARY_PATH=/tmp/modular/lib

Alternately, you might edit the /etc/ld.so.conf file and run ldconfig. There may be other ways to configure the linker on other systems.

Once this is done X applications should use the newly compiled libraries.

X applications

The X client programs which are built will be found in /tmp/modular/bin/. Generally, these programs can be run directly in the normal way.

Running the X server

The X server needs to be SUID root. This should happen automatically if you tell the build script to use sudo, but if make install runs as a normal user then this won't happen automatically. To do this: 

$ su
Password: <enter your root password here>
# chown root /tmp/modular/bin/Xorg
# chmod 4711 /tmp/modular/bin/Xorg
# exit

To run the new X server you can use the startx program:

XXX Verify this is correct 

startx -- /tmp/modular/bin/Xorg

Crosscompiling

There is a separate page dedicated to issues around CrossCompilingXorg.

New Modules

When adding modules to the tree, you may want to read the NewModuleGuidelines.