Setting up a Jenkins server

This page contains instructions and a script for setting up a Jenkins Continuous Integration Server for automated building of all of the TEI products.

What is a Jenkins server?

Jenkins is a continuous integration server, a tool from modern software engineering. It's role is to notice whenever a change is made to the work-in-progress text of the TEI standard and perform a range of tasks, including building building each of the language variants and a number of tests. Jenkins catches a broad range of technical issues and complains very loudly about them to the users it thinks caused them.

When it detects a change (because someone has edited one of the source code files), it downloads the changes and starts building the codebase to create an updated version of the project, whatever that is. In the case of the TEI, there are seven different sub-projects, each of which is built separately: the OxGarage conversion server, the Roma schema-building tool, the TEI Stylesheets, an XSLT 1.0 version of the Stylesheets, TEI P5, TEI P5 Documentation (the Guidelines etc.), and some TEI P5 test cases. Some of these jobs depend on others -- for instance, all of the TEIP5-* projects are built using the stylesheets created from the Stylesheets project.

Why would I want to create a Jenkins server?

A Jenkins server is useful to project administrators because it will automatically create new builds whenever the codebase changes, so the admins don't have to remember to do that themselves. It will also notify you when a build fails, perhaps because a recent change to the source contained an error. Programmers working on the codebase can use this feature to check the changes they have made to make sure they're OK, and roll them back if they don't work, without having to have a complete build environment set up on their own computers.

The TEI's main Jenkins server, in Oxford, is at http://tei.oucs.ox.ac.uk/jenkins/. There you can see the seven build targets and look at their status -- when they were last built, when the last build failure occurred, etc. The Humanities Computing and Media Centre at the University of Victoria also has a Jenkins server runnin at http://teijenkins.hcmc.uvic.ca:8080/

You might want to set up your own Jenkins server for a number of reasons:

• To add extra redundancy to the TEI project build process
• To add your own build targets for specific purposes
• To learn about Jenkins so that you can help administer one of the main servers

How do I go about creating a Jenkins server?

There are four stages:

1. Build a basic headless Ubuntu server, using the Ubuntu Precise 12.04 (Long-Term Support) edition.

2. Install an Oxygen licence in the right location (some of the TEI projects require Oxygen).

3. Run our customization script to install all of the TEI and Jenkins code.

4. Make some final tweaks to your setup (setting up security for Jenkins).

You will need to be comfortable with installing Linux and doing a couple of things at the command line. If you've never done this kind of thing before, this project is probably not for you.

Stage one

Download the latest distribution of the Ubuntu Precise (12.04) server ISO from the Ubuntu download site (http://www.ubuntu.com/download/ubuntu/alternative-download).

Install the distribution on the machine you're going to use as your server (or on a virtual machine, if you don't have a piece of hardware to dedicate to the job). Choose "Internet server" when you're asked what type of computer you're setting up, during the install process. Then do all the updates to make sure your Precise installation is fully updated.

Stage two

Log into your server, and place a copy of a valid Oxygen license file, licensekey.txt, in /usr/share/oxygen/. The file should contain the nine lines of text of the Oxygen license key (located between the license key start and end markers). A valid Oxygen license is required in order to build some of the TEI projects.

Stage three

Download our Jenkins_builder_script.sh.zip script. Unzip it, and get the script onto your server somehow. Make it executable, and run it as root. You'll see some basic instructions as it starts up. Agree to all EULAs along the way. When the script completes, you should have a working Jenkins machine. The Jenkins GUI will be running on its default port of 8080, and if you connect to your server on that port, you should see it start to run the five TEI jobs. The first run takes a long time, because it has to download all the source code from SVN, but subsequent builds triggered by changes should be much quicker.

Stage four

First, make sure you add a valid oXygen license key file in the appropriate location, so that your use of oXygen on this machine is legal. Put your oXygen licence key in a file called licensekey.txt in the oXygen directory (/usr/share/oxygen/).

Next, set up some security for your Jenkins installation, by using the Jenkins management interface on port 8080 There are various ways to do this, using different authentication methods, but this is the simplest:

• Click on Manage Jenkins, then Configure System.
• Check "Enable security".
• Choose "Jenkins's own user database".
• Choose "Matrix-based security".
• Create a new user (called, for instance, "tei"), and give it full privileges (check everything for that user).
• For the Anonymous user, check only the Read boxes in Overall and Job.
• Click on Save at the bottom of the page to save your changes.

Now log in as the new user ("tei") and set a password for yourself. Now anyone can see the progress of jobs, but only the tei user can log in and make changes.

What does your script actually do?

You can have a look for yourself:

#!/bin/bash
#The Mighty Jenkins Builder Script.
#Note that this should be run as root (with sudo).

echo ""
echo "Entering the Mighty Jenkins Builder Script."

uid=$(/usr/bin/id -u) && [ "$uid" = "0" ] ||
{ echo "This script must be run as root."; exit 1; }

echo "Running as root: good."
echo ""

if grep -q 10.04 /etc/lsb-release 
then echo "Running on Ubuntu Lucid. Good."
else
  echo "This script needs to be run on Ubuntu Lucid Server."
  echo "According to /etc/lsb-release, you don't seem to be running that version of Ubuntu."
  echo "The script will now terminate."
  exit
fi

echo ""
echo "Using netstat to check whether any service is currently running on port 8080."
echo ""
netstat -tulpan | grep 8080
if [ $? -eq 0 ] 
then echo "Another service appears to be running on port 8080, which is the default port for Jenkins."
  echo "You can either continue, and then change the port on which Jenkins runs later, or "
  echo "stop now, and move that service to another port."
  echo "Press return to continue, or Control+c to stop."
  read
fi

echo ""
echo "*******************************************"
echo "The purpose of this script is to set up a working
Jenkins Continuous Integration Server which will check
out and build a range of TEI products, including the 
P5 Guidelines (in various formats) and the Roma schema 
generation tool."
echo ""
echo "This script is designed to be run on a fully-updated
install of Ubuntu Lucid Lynx (Ubuntu 10.04). Lucid was 
chosen because it is a Long-Term Support edition, and 
will be available for around two years from the time 
of writing the script."
echo "The script may work on other versions of Ubuntu,
but only Lucid has been tested."
echo "*******************************************"
echo ""
echo "Do you want to continue? Press return to continue,
Control+c to stop."
read

echo ""
echo "*******************************************"
echo "Throughout the following process, you will be 
asked to agree to various EULAs and licences. Just
agree to everything, by selecting 'OK', 'Yes' etc."
echo "*******************************************"
echo ""

echo "Press return to continue"
read

#First do updates.
echo "Doing system updates before starting on anything else."
apt-get update
apt-get upgrade

#Now add the repositories we want.
echo "Backing up repository list."
cp /etc/apt/sources.list /etc/apt/sources.list.bak

#Uncomment partner repos.
echo "Uncommenting partner repositories on sources list, so we can get Sun Java."
sed -i -re '/partner/ s/^#//' /etc/apt/sources.list

#First Jenkins
echo "Adding Jenkins repository."
wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add -
echo "deb http://pkg.jenkins-ci.org/debian binary/" > /etc/apt/sources.list.d/jenkins.list

#Next TEI.
echo "Adding TEI Debian repository."
gpg --keyserver wwwkeys.uk.pgp.net --recv-keys FEA4973F86A9A497
apt-key add ~/.gnupg/pubring.gpg
echo "deb http://tei.oucs.ox.ac.uk/teideb/binary ./" > /etc/apt/sources.list.d/tei.list

#Now we can start installing packages.
echo "Updating for new repositories."
apt-get update

echo "Installing Sun Java JDK."
apt-get install sun-java6-jdk &&
echo "Installing core packages we need."
apt-get install openssh-server libxml2 libxml2-utils devscripts xsltproc debhelper subversion trang &&

#TEI packages
echo "Installing TEI packages."
apt-get install psgml xmlstarlet debiandoc-sgml linuxdoc-sgml jing jing-trang-doc libjing-java rnv texlive-xetex &&
apt-get install trang-java tei-p5-doc tei-p5-database tei-p5-source tei-schema tei-emacs saxon nxml-mode-tei tei-p5-xsl tei-p5-xsl2 tei-roma onvdl tei-oxygen zip &&

#I don't believe the following step is necessary, so it's been commented out for the moment.
#Waiting for info from SR and SY about why it was in the instructions.
#echo "Removing things that cause problems for TEI."
#apt-get remove `apt-cache search gcj | grep gcj | awk '{print $1}'`

#Setting up configuration for oXygen
mkdir /root/.com.oxygenxml
chmod a+x /root/.com.oxygenxml
mkdir /root/.java
chmod a+x /root/.java
echo "Don't forget to put your licensekey.txt file in the folder /usr/share/oxygen so that oXygen is registered."

#More packages needed
echo "Installing packages needed for building TEI source."

#Various fonts and the like.
echo "Installing fonts we need."
apt-get install ttf-dejavu msttcorefonts ttf-arphic-ukai ttf-arphic-uming ttf-baekmuk ttf-junicode ttf-kochi-gothic ttf-kochi-mincho
echo "The Han Nom font is not available in repositories, so we have to download it from SourceForge."
cd /usr/share/fonts/truetype
mkdir hannom
cd hannom
wget -O hannom.zip http://downloads.sourceforge.net/project/vietunicode/hannom/hannom%20v2005/hannomH.zip
unzip hannom.zip
find . -iname "*.ttf" | rename 's/\ /_/g'
rm hannom.zip
fc-cache -f -v

#Jenkins
apt-get install jenkins

#Configuration for Jenkins
echo "Starting configuration of Jenkins."
echo "Getting the Hudson log parsing rules from TEI SVN."
cd /var/lib/jenkins
svn export https://tei.svn.sourceforge.net/svnroot/tei/trunk/P5/Utilities/hudson-log-parse-rules
chown jenkins hudson-log-parse-rules
echo "Getting all the job data from TEI SVN."
#Don't bring down the config.xml file for now; that contains security settings specific to 
#Sebastian's setup, and will prevent anyone from logging in. We leave the server unsecured,
#and make it up to the user to secure it.
#svn export https://tei.svn.sourceforge.net/svnroot/tei/trunk/Documents/Editing/Jenkins/config.xml
#chown jenkins config.xml
svn export --force https://tei.svn.sourceforge.net/svnroot/tei/trunk/Documents/Editing/Jenkins/jobs/ jobs
chown -R jenkins jobs
echo "Installing Jenkins plugins."
cd plugins
wget --no-check-certificate http://updates.jenkins-ci.org/latest/copyartifact.hpi
chown jenkins copyartifact.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/emotional-hudson.hpi
chown jenkins emotiosudnal-hudson.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/greenballs.hpi
chown jenkins greenballs.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/jobConfigHistory.hpi
chown jenkins jobConfigHistory.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/plot.hpi
chown jenkins plot.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/log-parser.hpi
chown jenkins log-parser.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/scp.hpi
chown jenkins scp.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/WebSVN2.hpi
chown jenkins WebSVN2.hpi

echo "Restarting Jenkins server, so that it finds and initializes all the new plugins."
/etc/init.d/jenkins restart

echo "OK, we should be done. Now you have to:"
echo "1. Put your oXygen licence key in a file called licensekey.txt in the oXygen directory (/usr/share/oxygen/)."
echo "2. Go to the Jenkins interface on http://localhost:8080, and set up authentication. Read the Jenkins docs."
echo "That's it!"
read
exit