Adding continuous integration (CI) to your workflow
This post is aimed at web development teams and is not tied to a specific technology. We will aim to not get more technical than is needed, but rather to explore what Continuous integration (CI) is, and how it can help save teams money within a month of it being set up.
What is continuous integration?
Although several definitions of CI have been proposed, we will use the following definition in the context of this post:
Cotinuous integration (CI) is the practice of running any number of tests, automatically, on a project, periodically and/or whenever the code changes. For CI practitioners, the number one priority is for tests to always be passing.
A simple example, please
Here is the very simplest example I can think of:
Let’s say you’re maintaining an old-school HTML website (no fancy stuff like databases or PHP), your team may decide to set up CI to make sure a file called “index.html” exists in your codebase: if it exists, your test passes; if it is absent, your test fails.
Checks may be run every time your code is changed.
Your team might store code on GitHub, and link a cloud CI provider such as CircleCI to your codebase, having it trigger every time your code changes.
You will then define a script which is your definition of “what it means for your your codebase to pass”: checking for the existence of “index.html” is a one-line script.
A more complex example
Although the example above has value, it is very simple, and you may soon find yourself wanting to add higher-value tests to your script. This ability to add complexity over time is a powerful feature of CI: getting started is simple, and you can add as many tests as you want over time depending on your available resources.
Let’s say your team is maintaining a Drupal or Wordpress codebase with lots of complex code, your team may set up a CI server that:
- checks for broken links on the live environment every so often;
- checks every few minutes that the live environment is responding and has some expected keywords on its front page;
- every so often, checks that the live environment adheres to certain Accessibility standards;
- every so often, checks that the live environment is not reporting any errors;
- on every code change, perform some static analysis on custom PHP code: for example, that a function which expects an array as an argument is never called with a string.
- on every code change, make sure PHP code adheres to coding standards (for example, functions should have comments; and indenting should be correct).
- on every code change, create a dummy Drupal or Wordpress site with a dummy database and make sure your site fires up, and run some end-to-end tests against it.
- etc., etc.
A cloud-based tool such as CircleCI can work well to check the codebase when it is changed; and a hosted tool such as Jenkins might be a good fit for running periodic checks (such as a sanity check making sure the production environment works).
The above example corresponds to real-world checks I perform on lost of projects I maintain; and both CircleCI and Jenkins are tools I have been using for years.
So how much does all this cost?
“How much does this cost?” is actually the wrong question. “How much can I save?” is a better way of putting it. Consider the following graph, the horizontal axis is time, and the vertical axis is cumulative project cost.
- The red line is business as usual: because we are not maintaining CI scripts or setting up tests, the up-front cost is low. But eventually you’ll lose control of your codebase and spend all your time putting out fires (I call this the “technical debt wall”).
- The blue line is the CI approach, higher up-front cost to set things up, but eventually you’ll get less errors.
- Where the two lines intersect, I call the “sweet spot”. That’s when you start saving money. Your “sweet spot” is not months or years away: I firmly believe it should happen within a month. If it takes longer than a month, you’re overengineering your CI system.
So what are these up-front costs?
The up-front costs are:
- Creating a simple script which defines what it means for your code “to work”. If you find this intimidating, just have your script check for a file that must be present, as in the simple example presented earlier.
- Make sure your code is tracked in GitHub or BitBucket.
- Make sure your entire team accepts the principle that making tests pass is the number one priority. This is crucial. If you start accepting failing tests, then CI becomes a useless burden. This also means every member of your team must agree with every test that is performed. If a test is not important enough to warrant dropping everything when it fails, then you should not have that test in your codebase.
- Integrate a simple, free CI cloud provider like CircleCI and make sure it works.
All of the above, together, can take between an hour and a day.
How about the ongoing costs?
Ongoing costs are closely relate to the complexity of your CI setup. If you are just testing for an “index.html” file, your ongoing costs are close to zero, but may include:
- dealing with errors and updates in the CI script itself. Don’t forget the CI script is computer code, and like any computer code, it needs to be maintained.
- updating the CI script to deal with API changes in the cloud CI provider.
- fixing false negatives. For example, someone may change the filename from index.html to index.htm, which might require you to fix your test script to also test for index.htm in addition to index.html.
- onboarding new team members to understand the importance of making sure tests always are passing.
If your tests are super simple (such as checking that an “index.html” file exists), the above costs are low, probably less than one hour a month.
If your tests are complex (as in our second example, above), you might set aside 5 to 10 hours a month for ongoing costs.
Obviously, if your ongoing costs are higher than your savings, then you are “over-testing”.
So what are the benefits?
The fundamental trick of CI is to keep your benefits higher than your costs. Let’s go back to our simple “index.html” example:
- We have already established that there are minimal up-front and ongoing costs.
- There are also ongoing savings: once you know that your index.html file is guaranteed to exist, your manual testing time decreases.
- The cost in lost revenue, lost confidence, and debugging time in case someone accidentally deletes index.html from your website would be considerable high.
Based on the above, you can conclude whether it’s worth implementing CI.
Continuous improvement of your CI setup
Checking for “index.html” is probably of very low value, but once you’ve done that, you’ve also set up the foundation to improve your script. Every time you feel your CI script has a positive cost-benefit ratio, it is time to improve your CI script. In practice, I have found that in projects under active development, the CI setup gets constantly improved.
Specifically, any time a problem makes its way to production, it should be a gut reaction to introduce a fix, along with a test to make sure the problem never happens again.
The key is making incremental improvements, making sure your cost-benefit ratio is always positive.
Docker and containerization
Docker, and containerization generally, embed software and configuration in computer code along with your project code.
The widespread adoption of Docker and containerization in recent years has been crucial for CI. Without containerization, let’s say you want to run PHP static analysis, start a database with a Drupal site, run end-to-end tests, you need to install a bunch of software on your CI server (or your laptop), make sure the versions and configuration are in sync with your local development setups. This is simply too expensive.
Docker makes all this easy: simply put, Docker abstracts all the software and configuration, making software act the same on any computer that has Docker installed.
If you are not using Docker and you’d like to see how simple this makes things, install and launch Docker Desktop on your computer, give it 6Gb RAM instead of the default 2Gb in its preferences, then you’ll be able to run all tests on my Drupal Starterkit project, without any additional fiddling with configuration of software:
cd ~/Desktop && git clone https://github.com/dcycle/starterkit-drupalsite.git
cd starterkit-drupalsite
./scripts/ci.sh
It should take about 10 minutes to run all tests and it will not add any software to your computer; everything is done on throwaway “containers”. (In general, tests become a lot more frustrating to developers as they take longer to run; which is why I have a policy of not accepting tests which take more than 20 minutes to run.)
The amount of software packages and configuration required to run all the tests in this example is enormous: database servers and configuration, passwords, permissions, PHPUnit, the right version of PHP and Apache or Nginx…; however it’s all defined in Docker files and in code, not on host computers.
Which is why you can run the tests in three lines of code!
This makes it possible to run these complex tests on your computer without installing any software other than Docker.
This also makes it possible to run these exact tests, sans extra configuration, on CircleCI or other CI providers which support virtual machines with Docker preinstalled. In fact, that’s exactly what we’re doing with the Drupal Starterkit. CircleCI even provides a cute badge to indicate whether tests are passing.
Click on the badge below to see test results on CircleCI, which should be identical to the results on your computer if you ran the the above script (you’ll need to log in with your GitHub or BitBucket account).
Security
Whether you are using a cloud service such as CircleCI, or hosting your own CI server with Jenkins or other software, be aware that it adds a potential attack vector for hackers, especially because by design, CI software needs access to your codebase.
In early 2021, a vulnerability was discovered in JetBrains TeamCity (Widely Used Software Company May Be Entry Point for Huge U.S. Hacking, New York Times, January 6th, 2021) in relation to the major SolarWinds hack.
Make sure you have a solid security policy, including the Principle of Least Privilege (POLP) and other industry-standard security approaches; also make sure your codebase, even if it’s private, does not contain any sensitive data, including API keys.
Conclusion
With continuous integration (CI), you can let computers do the grunt work of looking for bugs in your codebase, liberating your developers to do more productive work, reducing the number of bugs that make it into production, and increasing the level of confidence of all stakeholders in your software, and deploying frequently.
And, above all, saving money.
CI can be as simple or as complex as you need: start small, then let your CI process grow as your team becomes more comfortable with it.