Setting up a Jenkins server

NOTE: This documentation is out of date.
The recent move of all the TEI development from SourceForge to GitHub has made the setup script below (which was already a bit out of date) no longer functional. It can be used as a guide to setting up your own server, but it will not work out of the box. This page will be updated with a new working script in due course.

---

This page contains instructions and a script for setting up a Jenkins Continuous Integration Server for automated building of all of the TEI products.

What is a Jenkins server?
Jenkins is a continuous integration server, a tool from modern software engineering. Its role is to notice whenever a change is made to the work-in-progress text of the TEI standard and perform a range of tasks, including building building each of the language variants and a number of tests. Jenkins catches a broad range of technical issues and complains very loudly about them to the users it thinks caused them.

When it detects a change (because someone has edited one of the source code files), it downloads the changes and starts building the codebase to create an updated version of the project, whatever that is. In the case of the TEI, there are seven different sub-projects, each of which is built separately: the OxGarage conversion server, the Roma schema-building tool, the TEI Stylesheets, an XSLT 1.0 version of the Stylesheets, TEI P5, TEI P5 Documentation (the Guidelines etc.), and some TEI P5 test cases. Some of these jobs depend on others -- for instance, all of the TEIP5-* projects are built using the stylesheets created from the Stylesheets project.

Why would I want to create a Jenkins server?
A Jenkins server is useful to project administrators because it will automatically create new builds whenever the codebase changes, so the admins don't have to remember to do that themselves. It will also notify you when a build fails, perhaps because a recent change to the source contained an error. Programmers working on the codebase can use this feature to check the changes they have made to make sure they're OK, and roll them back if they don't work, without having to have a complete build environment set up on their own computers.

The TEI's main Jenkins server, at the University of Victoria, is at http://jenkins.tei-c.org/. There you can see the multiple build targets and look at their status -- when they were last built, when the last build failure occurred, etc. The University of Oxford also has a Jenkins server running at http://jenkins2.tei-c.org/. There is a third backup server running at http://jenkins-paderborn.tei-c.org/.

You might want to set up your own Jenkins server for a number of reasons:


 * To add extra redundancy to the TEI project build process
 * To add your own build targets for specific purposes
 * To learn about Jenkins so that you can help administer one of the main servers

How do I go about creating a Jenkins server?
There are four stages:

1. Build a basic headless Ubuntu server, using the Ubuntu Precise 12.04 (Long-Term Support) edition.

2. Install an Oxygen licence in the right location (some of the TEI projects require Oxygen).

3. Run our customization script to install all of the TEI and Jenkins code.

4. Make some final tweaks to your setup (setting up security for Jenkins).

You will need to be comfortable with installing Linux and doing a couple of things at the command line. If you've never done this kind of thing before, this project is probably not for you.

Please note that the script is long and quite complicated, and it depends upon a variety of external services which may perhaps not be functioning correctly at the time when you run it. As a result, the script may fail. This is not really disastrous; you can open it in a text editor to see what it does, and run each command manually until the setup is complete. The script is intended as much to be a thorough documentation of how the TEI Jenkins servers work, and what is required in order to set one up, as it is to be a functional build script (although it has been thoroughly tested, and it does work).

Stage one
Download the latest distribution of the Ubuntu Precise (12.04) server ISO from the Ubuntu download site (http://www.ubuntu.com/download/ubuntu/alternative-download).

Install the distribution on the machine you're going to use as your server (or on a virtual machine, if you don't have a piece of hardware to dedicate to the job). Choose "Internet server" when you're asked what type of computer you're setting up, during the install process. Then do all the updates to make sure your Precise installation is fully updated.

Stage two
Log into your server, and place a copy of a valid Oxygen license file, licensekey.txt, in /usr/share/oxygen/. The file should contain the nine lines of text of the Oxygen license key (located between the license key start and end markers). A valid Oxygen license is required in order to build some of the TEI projects.

Stage three
Download our Jenkins_builder_script.sh script from the TEI SourceForge repository. Get the script onto your server somehow. Make it executable, and run it as root. You'll see some basic explanatory information as it starts up. When the script completes, you should have a working Jenkins machine. The Jenkins GUI will be running on its default port of 8080, and if you connect to your server on that port, you should see it start to run the first of the seven TEI jobs. The first run takes a long time, because it has to download all the source code from SVN, but subsequent builds triggered by changes should be much quicker.

Stage four
Next, you need to set up some security for your Jenkins installation, by using the Jenkins management interface on port 8080 There are various ways to do this, using different authentication methods, but this is the simplest:


 * Click on Manage Jenkins, then Configure System.
 * Check "Enable security".
 * Choose "Jenkins's own user database".
 * Choose "Matrix-based security".
 * Create a new user (called, for instance, "tei"), and give it full privileges (check everything for that user).
 * For the Anonymous user, check only the Read boxes in Overall and Job.
 * Click on Save at the bottom of the page to save your changes.

Now log in as the new user ("tei") and set a password for yourself. Now anyone can see the progress of jobs, but only the tei user can log in and make changes.

What does your script actually do?
Basically, the builder script adds some repositories and installs some software from them, including the Jenkins server software. Then it downloads some configuration files from the TEI SourceForge repository, tweaks them a bit, and restarts Jenkins. You can read the script itself in the SVN repository.