The name of the game in technology is doing more with less. It is the core element in the drive for profitability. In today’s market, that means finding ways to reduce both the number of employees in an organization and improve the processes that allow for such a reduction.
The days of paying $80,000 for an employee with limited knowledge, only ETL or only XML, are waning or dead. Even more importantly, the cost of software bugs in 2016 reached $1.1 trillion.
Reducing labor costs and reducing the cost of software bugs means producing efficient, clean, easy to follow, and effective systems backed by quality processes (word play with quality assurance intended). This article dives into how to use Bamboo, Flask, and Bitbucket to achieve continuous integration and continuous deployment to a test environment.
The concepts in this article can be ported to a distributed environment using ngrok but hopefully a more secure option is available. The ngrok solution is explained well on the Bitbucket website but promotes non-free software and external providers. This article intends to explain how to do this for free without linking to an external provider.
An automated test environment can be set up with:
- An Ubuntu or Linux Server for the purposes of this article
Bitbucket is a powerful repository based on git. Part of the repositories power is the ability to add webhooks. These are essentially triggers that perform POST requests to a webserver on specified events.
Webhooks require a web server. Flask is a lightweight and easy to maintain, single threaded framework. This is perfect for most repositories that do not see a significant number of pull requests.
Other hooks, including to Jenkins, are available as well.
Another powerful feature of Bamboo worth noting is the ability to organize repositories into projects. This suits companies with well defined architectures extremely well.
Bamboo is a continuous integration (CI) tool, a tool that allows code to be continually built and tested from a Bitbucket repository with minimal human interaction. This tool integrates directly with Jira, Bitbucket, and nearly every other Atlassian product.
Automating a System
Most companies with a forward facing website have production and test environments. Integrating and automating the environment is made possible with the use of a continuous integration (CI) environment and webhooks.
Bamboo can be set up to automatically synchronize code when a commit or merge is made in the remote repository. However, jobs in this CI run in an isolated way with no access to write to the local machine beyond SCP.
While a CI ensures that code runs correctly and can be set to notify users of failed builds, the synchronization issue outside of the isolated build environment is solved with webhooks. Calls to web servers specified in these hooks are triggered whenever a certain event is performed in the repository, allowing for more automated and instantaneous regression and integration testing.
Step 1: Setting Up SSH in Bitbucket
Any automated pull requires using ssh instead of http protocol as the means to obtain code from the remote repository. This requires setting up an ssh key on Bitbucket and on the local machine where your test environment is configured.
Setting up an rsa key on Ubuntu is simple:
- run ssh-keygen -t rsa
- enter only the name of the key
- copy the public key (.pub) to your clipboard
- create or open the config file in your home’s .ssh directory
- add your bitbucket host on a new line Host <host ip address>
- on a new tab spaced line enter the name of your private key file generated in steps 1-2 as Identity File <private key file name>
- log in to Bitbucket
- click on your avatar
- click on Manage Account
- navigate to SSH Keys on the left hand menu
- click on add key towards the center of the page and simply paste your key
Step 2: Creating a Webhook in Flask
A webook can be setup simply in Flask. James Innes of Ogma Development wrote a terrific article on how to setup a webhook in Flask. After creating the project as specified, run the code using:
It is possible to change the application host by simply modifying app.run() in your hook as follows:
app.run(host=<host>, port=[unreserved port])
Note the host, port, and token if applicable when your application is running and perform a GET request with the appropriate token from the machine or container running your Bitbucket repository.
Step 3: Setting up Webhooks in Bitbucket
Webhooks in Bitbucket can occur at the project or repository level. For the best control, this article explains how to create a hook at the repository level. Imagine triggering a multitude of unnecessary webhooks when modifying a single repository.
Setting up a webhook on a Bitbucket repository is simple:
- Log in to Bitbucket
- navigate to your target project
- Navigate to your repository
- Click on the settings gear in the left hand menu
- Click on webhooks in the left hand menu
- Enter a name for your hook
- Enter the url of your webhook server (of the form http[s]://webhook[?verify_token=<token>] in this context)
- In the options menus, ensure that push and merged are clicked
- Click save at the bottom of the screen
Currently, Bitbucket tests these connections with an empty POST request. This is not handled by our code and empty POST requests are generally discouraged anyway.
It is now possible to simply execute a pull request from a module under the webhook function in your Flask application on every merge in or push to the repository. This can be used to synchronize code and even keep a test site constantly ready for testing if you are using Django. As Flask is single threaded, it is better suited for the task of creating hooks.
Create a Build in Bamboo
With our repository thoroughly connected to the remote server, we can create build plans using Bamboo. Bamboo organizes the build flow underneath a plan into jobs which maintain tasks.
For an annual fee of just ten dollars, it is possible to obtain ten free jobs with an unlimited number of tasks.
Future articles will explain how to integrate applications built with Django, Celery, and PyTorch. Still, even databases created using Djang are recreatable using the only the models and shell commands from the Script task.
As stated before, Bamboo can be set up to notify an appropriate developer on a build failure.
With these elements in place, it is possible to keep running test code up to date and continue to run builds against this environment with minimal human interaction. This greatly reduces labor and thus cost which can be passed to the end user in the form of savings.