Setting up a Jenkins server

This page contains instructions and a script for setting up a Jenkins Continuous Integration Server for automated building of all of the TEI products.

What is a Jenkins server?
The purpose of a Jenkins server is to watch a code repository waiting for changes in the code. When it detects a change (because someone has edited one of the source code files), it downloads the changes and starts building the codebase to create an updated version of the project, whatever that is. In the case of the TEI, there are seven different sub-projects, each of which is built separately: the OxGarage conversion server, the Roma schema-building tool, the TEI Stylesheets, an XSLT 1.0 version of the Stylesheets, TEI P5, TEI P5 Documentation (the Guidelines etc.), and some TEI P5 test cases. Some of these jobs depend on others -- for instance, all of the TEIP5-* projects are built using the stylesheets created from the Stylesheets project.

Why would I want to create a Jenkins server?
A Jenkins server is useful to project administrators because it will automatically create new builds whenever the codebase changes, so the admins don't have to remember to do that themselves. It will also notify you when a build fails, perhaps because a recent change to the source contained an error. Programmers working on the codebase can use this feature to check the changes they have made to make sure they're OK, and roll them back if they don't work, without having to have a complete build environment set up on their own computers.

The TEI's main Jenkins server, in Oxford, is at http://tei.oucs.ox.ac.uk/jenkins/. There you can see the five build targets and look at their status -- when they were last built, when the last build failure occurred, etc. The Humanities Computing and Media Centre at the University of Victoria is in the process of setting up a second Jenkins server for the TEI. As soon as this is running, the URI will be included in this page. You might want to set up your own Jenkins server for a number of reasons:


 * To add extra redundancy to the TEI project build process
 * To add your own build targets for specific purposes
 * To learn about Jenkins so that you can help administer one of the main servers

How do I go about creating a Jenkins server?
There are three stages:

1. Build a basic headless Ubuntu server, using the Ubuntu Lucid (Long-Term Support) edition.

2. Run our customization script to install all of the TEI and Jenkins code.

3. Make some final tweaks to your setup (adding an oXygen license and setting up security for Jenkins).

You will need to be comfortable with installing Linux and doing a couple of things at the command line. If you've never done this kind of thing before, this project is probably not for you.

Stage one
Download the latest distribution of the Ubuntu Lucid (10.04) server ISO from the Ubuntu download site (http://www.ubuntu.com/download/ubuntu/alternative-download).

Install the distribution on the machine you're going to use as your server (or on a virtual machine, if you don't have a piece of hardware to dedicate to the job). Choose "Internet server" when you're asked what type of computer you're setting up, during the install process. Then do all the updates to make sure your Lucid installation is fully updated.

Stage two
Download our Jenkins_builder_script.sh.zip script. Unzip it, and get the script onto your server somehow. Make it executable, and run it as root. You'll see some basic instructions as it starts up. Agree to all EULAs along the way. When the script completes, you should have a working Jenkins machine. The Jenkins GUI will be running on its default port of 8080, and if you connect to your server on that port, you should see it start to run the five TEI jobs. The first run takes a long time, because it has to download all the source code from SVN, but subsequent builds triggered by changes should be much quicker.

Stage three
First, make sure you add a valid oXygen license key file in the appropriate location, so that your use of oXygen on this machine is legal. Put your oXygen licence key in a file called licensekey.txt in the oXygen directory (/usr/share/oxygen/).

Next, set up some security for your Jenkins installation, by using the Jenkins management interface on port 8080 There are various ways to do this, using different authentication methods, but this is the simplest:


 * Click on Manage Jenkins, then Configure System.
 * Check "Enable security".
 * Choose "Jenkins's own user database".
 * Choose "Matrix-based security".
 * Create a new user (called, for instance, "tei"), and give it full privileges (check everything for that user).
 * For the Anonymous user, check only the Read boxes in Overall and Job.
 * Click on Save at the bottom of the page to save your changes.

Now log in as the new user ("tei") and set a password for yourself. Now anyone can see the progress of jobs, but only the tei user can log in and make changes.

What does your script actually do?
You can have a look for yourself:

echo "" echo "Entering the Mighty Jenkins Builder Script." uid=$(/usr/bin/id -u) && [ "$uid" = "0" ] || { echo "This script must be run as root."; exit 1; } echo "Running as root: good." echo "" if grep -q 10.04 /etc/lsb-release then echo "Running on Ubuntu Lucid. Good." else echo "This script needs to be run on Ubuntu Lucid Server." echo "According to /etc/lsb-release, you don't seem to be running that version of Ubuntu." echo "The script will now terminate." exit fi echo "" echo "Using netstat to check whether any service is currently running on port 8080." echo "" netstat -tulpan | grep 8080 if [ $? -eq 0 ] then echo "Another service appears to be running on port 8080, which is the default port for Jenkins." echo "You can either continue, and then change the port on which Jenkins runs later, or " echo "stop now, and move that service to another port." echo "Press return to continue, or Control+c to stop." read fi echo "" echo "*******************************************" echo "The purpose of this script is to set up a working Jenkins Continuous Integration Server which will check out and build a range of TEI products, including the P5 Guidelines (in various formats) and the Roma schema  generation tool." echo "" echo "This script is designed to be run on a fully-updated install of Ubuntu Lucid Lynx (Ubuntu 10.04). Lucid was chosen because it is a Long-Term Support edition, and  will be available for around two years from the time  of writing the script." echo "The script may work on other versions of Ubuntu, but only Lucid has been tested." echo "*******************************************" echo "" echo "Do you want to continue? Press return to continue, Control+c to stop." read echo "" echo "*******************************************" echo "Throughout the following process, you will be asked to agree to various EULAs and licences. Just agree to everything, by selecting 'OK', 'Yes' etc." echo "*******************************************" echo "" echo "Press return to continue" read echo "Doing system updates before starting on anything else." apt-get update apt-get upgrade echo "Backing up repository list." cp /etc/apt/sources.list /etc/apt/sources.list.bak echo "Uncommenting partner repositories on sources list, so we can get Sun Java." sed -i -re '/partner/ s/^#//' /etc/apt/sources.list echo "Adding Jenkins repository." wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add - echo "deb http://pkg.jenkins-ci.org/debian binary/" > /etc/apt/sources.list.d/jenkins.list echo "Adding TEI Debian repository." gpg --keyserver wwwkeys.uk.pgp.net --recv-keys FEA4973F86A9A497 apt-key add ~/.gnupg/pubring.gpg echo "deb http://tei.oucs.ox.ac.uk/teideb/binary ./" > /etc/apt/sources.list.d/tei.list echo "Updating for new repositories." apt-get update echo "Installing Sun Java JDK." apt-get install sun-java6-jdk && echo "Installing core packages we need." apt-get install openssh-server libxml2 libxml2-utils devscripts xsltproc debhelper subversion trang && echo "Installing TEI packages." apt-get install psgml xmlstarlet debiandoc-sgml linuxdoc-sgml jing jing-trang-doc libjing-java rnv texlive-xetex && apt-get install trang-java tei-p5-doc tei-p5-database tei-p5-source tei-schema tei-emacs saxon nxml-mode-tei tei-p5-xsl tei-p5-xsl2 tei-roma onvdl tei-oxygen zip && mkdir /root/.com.oxygenxml chmod a+x /root/.com.oxygenxml mkdir /root/.java chmod a+x /root/.java echo "Don't forget to put your licensekey.txt file in the folder /usr/share/oxygen so that oXygen is registered." echo "Installing packages needed for building TEI source." echo "Installing fonts we need." apt-get install ttf-dejavu msttcorefonts ttf-arphic-ukai ttf-arphic-uming ttf-baekmuk ttf-junicode ttf-kochi-gothic ttf-kochi-mincho echo "The Han Nom font is not available in repositories, so we have to download it from SourceForge." cd /usr/share/fonts/truetype mkdir hannom cd hannom wget -O hannom.zip http://downloads.sourceforge.net/project/vietunicode/hannom/hannom%20v2005/hannomH.zip unzip hannom.zip find. -iname "*.ttf" | rename 's/\ /_/g' rm hannom.zip fc-cache -f -v apt-get install jenkins echo "Starting configuration of Jenkins." echo "Getting the Hudson log parsing rules from TEI SVN." cd /var/lib/jenkins svn export https://tei.svn.sourceforge.net/svnroot/tei/trunk/P5/Utilities/hudson-log-parse-rules chown jenkins hudson-log-parse-rules echo "Getting all the job data from TEI SVN." svn export --force https://tei.svn.sourceforge.net/svnroot/tei/trunk/Documents/Editing/Jenkins/jobs/ jobs chown -R jenkins jobs echo "Installing Jenkins plugins." cd plugins wget --no-check-certificate http://updates.jenkins-ci.org/latest/copyartifact.hpi chown jenkins copyartifact.hpi wget --no-check-certificate http://updates.jenkins-ci.org/latest/emotional-hudson.hpi chown jenkins emotiosudnal-hudson.hpi wget --no-check-certificate http://updates.jenkins-ci.org/latest/greenballs.hpi chown jenkins greenballs.hpi wget --no-check-certificate http://updates.jenkins-ci.org/latest/jobConfigHistory.hpi chown jenkins jobConfigHistory.hpi wget --no-check-certificate http://updates.jenkins-ci.org/latest/plot.hpi chown jenkins plot.hpi wget --no-check-certificate http://updates.jenkins-ci.org/latest/log-parser.hpi chown jenkins log-parser.hpi wget --no-check-certificate http://updates.jenkins-ci.org/latest/scp.hpi chown jenkins scp.hpi wget --no-check-certificate http://updates.jenkins-ci.org/latest/WebSVN2.hpi chown jenkins WebSVN2.hpi echo "Restarting Jenkins server, so that it finds and initializes all the new plugins." /etc/init.d/jenkins restart echo "OK, we should be done. Now you have to:" echo "1. Put your oXygen licence key in a file called licensekey.txt in the oXygen directory (/usr/share/oxygen/)." echo "2. Go to the Jenkins interface on http://localhost:8080, and set up authentication. Read the Jenkins docs." echo "That's it!" read exit
 * 1) !/bin/bash
 * 2) The Mighty Jenkins Builder Script.
 * 3) Note that this should be run as root (with sudo).
 * 1) First do updates.
 * 1) Now add the repositories we want.
 * 1) Uncomment partner repos.
 * 1) First Jenkins
 * 1) Next TEI.
 * 1) Now we can start installing packages.
 * 1) TEI packages
 * 1) I don't believe the following step is necessary, so it's been commented out for the moment.
 * 2) Waiting for info from SR and SY about why it was in the instructions.
 * 3) echo "Removing things that cause problems for TEI."
 * 4) apt-get remove `apt-cache search gcj | grep gcj | awk '{print $1}'`
 * 1) Setting up configuration for oXygen
 * 1) More packages needed
 * 1) Various fonts and the like.
 * 1) Jenkins
 * 1) Configuration for Jenkins
 * 1) Don't bring down the config.xml file for now; that contains security settings specific to
 * 2) Sebastian's setup, and will prevent anyone from logging in. We leave the server unsecured,
 * 3) and make it up to the user to secure it.
 * 4) svn export https://tei.svn.sourceforge.net/svnroot/tei/trunk/Documents/Editing/Jenkins/config.xml
 * 5) chown jenkins config.xml