Getting Started with Open Source Probo.CI

By mrbagnall | 11 August 2017

Of the many challenges we face as software developers, getting our code through the code review and QA/regression testing process is often one of the more challenging. Moving and copying assets and databases between production, development and staging environments can be tedious, time consuming, and not always successful.

One of the newer tools to come out to help in this sphere of continuous integration and collaboration is Probo. Its singular purpose is to bridge the gap between branches of code to allow you to test specific instances of change within your repository.

In my proposed development model, you have a git repository with production branches of code, staging branches, sprint branches and issue/feature branches.

When used in conjunction with JIRA, what Probo allows us to do is not only test our individual ticket, features and bug fixes in them against our sprints, but also then test those for regressions against our development and production branches. Each branch generating it's own Docker container instance and URL that you can use in a standard web browser.

So, to put it simply, every pull request gets it's own unique URL with the contents only of the changes in that pull request. Complete with database.

Probo comes in two distinct flavors. There is a SaaS model hosted by the parent company Zivtech that runs on cutting edge hardware and is extraordinarily fast. The space you get for builds is somewhat limited, so if you have to have multiple pull requests on a particularly large repository, it may make sense to build your own in-hose Probo server.

Now there are some serious considerations to going the "roll your own" route. This is a technical challenge, requires some knowledge of NodeJS and some configuration hocus pocus - all of which this article will guide you through.

To get started, I used a base server installation of CentOS 7 with very little attached to it. I like to do this to keep things lean, but you can do as you please. Just remember that your mileage may vary if you are using a different distribution of Linux such as Debian or Ubuntu.

For those who wish to follow along in setting up your own Probo server using CentOS, be sure to create a user called 'probo' that had administrative privileges. This way you could use the sudo command with that user. This is a very important step for following along.

Credit where credit is due - some of the steps and information here come directly from the Quickstart README documents for the various Probo projects. My attempt here is to put the puzzle together and provide a walk-through of my experience in the hope it may help others in need or who are looking to get started with their own instance of Probo.

The first step was to install a package managed set of core software I found necessary to be sure that I was able to complete the compilation and running of Probo, git and Docker. Not to mention screen which will become very important later.

sudo wget http://download.rethinkdb.com/centos/7/`uname -m`/rethinkdb.repo \
          -O /etc/yum.repos.d/rethinkdb.repo
sudo yum install epel-release
sudo yum update
sudo yum makecache fast
sudo yum install nodejs
sudo yum install node-gyp
sudo yum install mocha
sudo yum install nodejs-should
sudo yum install git
sudo yum install screen
sudo yum install rethinkdb

The next step was I created a folder for my log files. By default, the Probo executables do not create log files, but we will be creating our own instead of outputting to the stdout. So create the folder by issuing:

mkdir logs

The next part involves opening up our external ports on our CentOS iptables firewall. It took me a while to figure this out. You might be stuck on this step or you might not be depending on how you installed CentOS. If you get a message saying that firewalld doesn't exist or isn't enabled, then you likely do not need to do this step.

sudo firewall-cmd --zone=public --add-port=3010/tcp --permanent
sudo firewall-cmd --zone=public --add-port=3050/tcp --permanent
sudo firewall-cmd --zone=public --add-port=3070/tcp --permanent
sudo systemctl restart firewalld

Now we checkout the Probo code and other packages covered in this walkthru. Note that you currently need to get my version and branch of the Probo code that includes a patch that fixes a few changes with Probo that are currently needed to work with Probo Proxy. As of this writing, there is an approved pull request to merge this into Probo, but it has not been merged yet. As this changes, I will be updating this entry. Be sure to follow me on Twitter at @mbagnall17 for the latest and to keep up with changes here and other interesting and not-so-interesting things.

git clone -b buildid-containerstop https://github.com/ElusiveMind/probo.git
git clone https://github.com/ProboCI/probo-asset-receiver.git
git clone https://github.com/ProboCI/probo-loom.git
git clone https://github.com/ProboCI/probo-proxy.git
git clone https://github.com/ProboCI/probo-reaper.git

The next step in the process involves downloading and installing the Docker container system.

wget -qO- https://get.docker.com/ | sudo sh

Now we need to add our probo user to the docker group so it can utilize the Docker daemon.

sudo usermod -aG docker $USER

Now we start Docker and ensure it starts at system start up.

sudo systemctl enable docker
sudo systemctl start docker

Now we need to pull the relevant Docker containers that can be used by our system. Here are the main ones. There are more, but these are the ones that are used most often. It is important to note that the default one does not specify the version of PHP. If you want to default to a specific version of PHP, you will need to make a change in your .probo.yaml file in your specific project or wait for another blog entry on some advanced things I will cover later. You may need to log out and log back in before you will be able to issue these commands if you get Docker daemon errors.

sudo docker pull proboci/ubuntu-14.04-lamp
sudo docker pull proboci/ubuntu-14.04-lamp:php-5.6
sudo docker pull proboci/ubuntu-14.04-lamp:php-7.0
sudo docker pull proboci/ubuntu-14.04-lamp:php-7.1

Deployment

Several components of the system require full DNS names to function properly and cannot be addressed at localhost. For this tutorial, we assume that the server your install is running on is publically routable on {YOUR-DOMAIN}, and that all services are directly accessible on their ports as well (such as http://{YOUR-DOMAIN):3050). You can also use a sub-domain of an existing domain if you wish.

In order to view the results of the builds ensure that {YOUR-DOMAIN} is also pointing to your server.

Note that this configuration is suitable for development, but do NOT use for production. Reverse-proxying and SSL configuration will be covered in a future blog entry.

GitHubHandler

The GitHubHandler (GHH) processes GitHub hook events each time a pull request is created or updated. It then triggers builds through the Container Manager. The GitHub Handler also sends commit status updates back to GitHub.

It is implemented as a plugin of the Probo repository. There is also a Bitbucket plugin that is in a separate repository and it's function and configuration will be discussed in a future blog entry as there are conditions and parameters outside the scope of this document which must be taken into consideration before you can successfully run Bitbucket.

cd probo
npm install

Create a file ggh.yaml with the following contents:

# Port for the server to listen on
port: 3010
hostname: 0.0.0.0

# Github hook and credentials
githubWebhookPath: '/github-webhook'
githubWebhookSecret: '{YOUR_CHOSEN_SECURE_SECRET_PHRASE}'
githubAPIToken: '{GITHUB_PERSONAL_TOKEN}'

# Container Manager API server
api:
  url: "http://localhost:3020"

Of the defaults above, githubAPIToken must be set to your token. You can generate a personal token at [https://github.com/settings/tokens]. A token created from an OAuth flow will also work here. Probo requires the response.

The githubWebhookSecret value should be modified to a secure string as well. This is a random token containing a string value that you select and will need to be used when you configure your webhook in GitHub.

Execution

node ./bin/probo github-handler -c ghh.yaml > ../logs/ghh.log &

Now add a webhook for your repository in GitHub to your server under Settings -> Webhooks & services. Direct link to the configuration page: https://github.com/OWNER/REPO/settings/hooks.

Set the following properties:
Payload URL: http://{YOUR-DOMAIN}:3010/github-webhook
Secret: {The value from `githubWebhookSecret` in the config file}

Next, under "Which events would you like to trigger this webhook?", select "Let me select individual events", and select "Pull Request"

If you see a green checkmark next to your new webhook, you're all set. GitHub can successfully send requests to your handler.

Container Manager

The Container Manager (CM) manages Docker containers and kicks off builds.

It is implemented as a plugin of the probo repository.

Configure

Create a file cm.yaml with the following contents:

hostname: localhost
port: 3020

instanceName: '{YOUR CHOSEN INSTANCE NAME}'

api:
  url: "http://localhost:3010"

loom:
  url: "http://localhost:3060"

assets:
  url: "http://{YOUR_DOMAIN}:3070"

buildUrl: "http://{{buildId}}.{YOUR_DOMAIN}:3050/"

You will want to be sure to modify the InstanceName as this is what is prefixed in your GitHub status updates as tests are conducted. This will differentiate those tests from those done by the ProboCI SASS service a little more clearly. Also be sure to provide your domain name in the proper space. Leave {{buildId}} as it is. That is an important token used by the container manager to build URL's to your containers. If you modify the port of your probo-proxy daemon (such as to port 80), you will need to change the port in the buildUrl (or remove it) manually as well.

Execution:

node ./bin/probo container-manager -c cm.yaml > ../logs/cm.log &

Build Probo Proxy

The build proxy maps an external host/port to a build container's port 80 to view the built web application.

cd ..
cd probo-proxy
npm install

Configure

Create a file proxy.yaml with the following contents:

# port that the proxy server is running on
port: 3050

# Host for the container lookup service that maps a build id 
# to a host/port to proxy to
containerLookupHost: "http://localhost:3020"

Be sure that containerLookupHost has the same port number as your container manager instance. As you can imagine - that's pretty important!

node ./index.js -c proxy.yaml > ../logs/proxy.log &

Loom

Loom is the task output aggregation service that records and plays back live log streams. Loom currently requires RethinkDB as a backing store.

Before starting this process, keep in mind that you will need to start the RethinkDB daemon before starting loom. This is an important step not only for the installation of Probo, but also for subsequent restarts. Currently RethinkDB does not come with a systemd startup script and the instructions for creating one are beyond the scope of this document.

cd ../
cd probo-loom
rethinkdb --daemon
npm install
node ./bin/loom > ../logs/loom.log &

Asset Receiver

The Probo Asset Receiver is really an important piece for uploading items such as databases and other things you might need when wanting to use this for existing Drupal and WordPress web sites. While designed for use with Probo Uploader, you do not need to use Probo Uploader to use it and this document does not cover use of Probo Uploader. In fact, the use of Probo Asset Receiver is covered in another blog post coming soon as it has its own set of rules to be considered for efficient use.

I use AmazonS3 to store my files because it is quick, convenient, and I want to save my disc space for builds and not large database files. You can configure it to store files on the Probo Asset Receiver. We will discuss that in a future blog post.

Configure your Probo Asset Receiver with a configuration file as follows modifying the defaults.config.yaml file in the git repository you downloaded with the values below:

# The host and port to listen on.
host: 0.0.0.0
port: 3070

# The directory in which to store the LevelDB database.
databasePlugin: LevelDB
databaseConfig:
  databaseDataDirectory: db

# Determines the cipher used to encrypt the assets.
# See https://www.openssl.org/docs/manmaster/apps/ciphers.html for options.
encryptionCipher: 'aes-256-cbc'
encryptionPassword: {YOUR CHOSE A SECRET PASS PHRASE}
recipheredOutputDir: null
 
# API tokens for creating all routes except asset upload (disabled by default)
tokens: null
uploadsPaused: false
 
# Amazon S3 File Storage Configuration Settings
fileStoragePlugin: AwsS3Storage
fileStorageConfig:
  awsAccessKeyId: {YOUR AWS ACCESS KEY}
  awsSecretAccessKey: {YOUR AWS SECRET ACCESS KEY}
  awsBucket: {FILE BUCKET}

Next we start the asset receiver:

cd ..
cd probo-asset-receiver
npm install
node ./bin/probo-asset-receiver --c default.config.yaml > ../logs/probo-asset-receiver.log &

In the asset receiver, bucket names must be calculated in order to work with .probo.yaml files. The asset id must be the same as the asset file name. As such if your database file is database.sql.gz both the asset name and asset id specified in the upload string must be database.sql.gz

The bucket name must be in the format of organization-repository_name. For example, if I was to use the cmp_build repository in my ElusiveMind organization, the bucket name must be: Elusivemind-cmp_build. We will provide code examples for all of this in a future blog post complete with example URL's and real-life scenario's and specific project .probo.yaml files that match.

Probo Server QuickStart Commands

For future restarts, you will want to use the following sets of commands in this order to restart your Probo server. I attempted a systemd startup system which works except for the pesk RethinkDB issue with systemd. It begins by logging in as the probo user you created, copying and pasting these commands and going to town.

One item of note here. I STRONGLY recommend running these item in a separate console. If you run them as background processes and then log out they will eventually die. My example includes this screen command, which is recommended, but optional.

screen
cd
cd probo
node ./bin/probo github-handler -c ghh.yaml > ../logs/ghh.log &
node ./bin/probo container-manager -c cm.yaml > ../logs/cm.log &

cd ../probo-asset-receiver
node ./bin/probo-asset-receiver -c default.config.yaml > ../logs/asset.log &

cd ../probo-loom
rethinkdb --daemon
node ./bin/loom  > ../logs/loom.log  &

cd ../probo-proxy
node ./index.js -c proxy.yaml > ../logs/proxy.log &

cd ..
Press CTRL-A
Press CTRL-D

Probo Reaper

The Reaper is a housekeeping process used to kill off old builds when pull requests have been merged, declined or otherwise go away. It tears down and destroys docker builds and can be generally thought of as garbage collection. I have found this task best set up on a cron to run about every 10 minutes. It is a light weight process and, of course, depends on how heavily used your server is.

The Reaper "defaults.yaml" file can be pretty much left untouched for standard performance. The only change you will need to be aware of is that if you change from GitHub to Bitbucket for repository handling the "github" under "codeHostingHandler" will need to be changed to "bitbucket" using the same URL.

A sample of the cron command might look something like this if you're user is named "probo"

*/10 * * * * node /home/probo/probo-reaper/bin/probo-reaper reap -c /home/probo/probo-reaper/defaults.yaml

Conclusion

I hope this helps someone out there get started with Probo. If you are looking for a starting point for using Probo with Bitbucket, understand there is a greater curve and more considerations for getting that going and it is covered in another article.

Be sure to checkout the Probo.CI SASS service as it is a tremendous service by the great folks at Zivtech and may serve your needs well without going through the roll-your-own scenario.

https://probo.ci
https://zivtech.com

Be sure to follow me on Twitter for updates and news as things become available and besides, I could use some followers! I try to keep it fun, but I take my hockey pretty seriously!

https://twitter.com/mbagnall17
https://michaelbagnall.com