Bamboo and Webhooks: Automating CI and Deployment

process_auto

The name of the game in technology is doing more with less. It is the core element in the drive for profitability. In today’s market, that means finding ways to reduce both the number of employees in an organization and improve the processes that allow for such a reduction.

The days of paying $80,000 for an employee with limited knowledge, only ETL or only XML, are waning or dead. Even more importantly, the cost of software bugs in 2016 reached $1.1 trillion.

Reducing labor costs and reducing the cost of software bugs means producing efficient, clean, easy to follow, and effective systems backed by quality processes (word play with quality assurance intended). This article dives into how to use Bamboo, Flask, and Bitbucket to achieve continuous integration and continuous deployment to a test environment.

The concepts in this article can be ported to a distributed environment using ngrok but hopefully a more secure option is available. The ngrok solution is explained well on the Bitbucket website but promotes non-free software and external providers. This article intends to explain how to do this for free without linking to an external provider.

Environment

An automated test environment can be set up with:

  • Bitbucket
  • Flask
  • Bamboo
  • An Ubuntu or Linux Server for the purposes of this article

Bitbucket is a powerful repository based on git. Part of the repositories power is the ability to add webhooks. These are essentially triggers that perform POST requests to a webserver on specified events.

Webhooks require a web server. Flask is a lightweight and easy to maintain, single threaded framework. This is perfect for most repositories that do not see a significant number of pull requests.

Other hooks, including to Jenkins, are available as well.

Another powerful feature of Bamboo worth noting is the ability to organize repositories into projects. This suits companies with well defined architectures extremely well.

Bamboo is a continuous integration (CI) tool, a tool that allows code to be continually built and tested from a Bitbucket repository with minimal human interaction. This tool integrates directly with Jira, Bitbucket, and nearly every other Atlassian product.

Automating a System

Most companies with a forward facing website have production and test environments. Integrating and automating the environment is made possible with the use of a continuous integration (CI) environment and webhooks.

Bamboo can be set up to automatically synchronize code when a commit or merge is made in the remote repository. However, jobs in this CI run in an isolated way with no access to write to the local machine beyond SCP.

While a CI ensures that code runs correctly and can be set to notify users of failed builds, the synchronization issue outside of the isolated build environment is solved with webhooks. Calls to web servers specified in these hooks are triggered whenever a certain event is performed in the repository, allowing for more automated and instantaneous regression and integration testing.

Step 1: Setting Up SSH in Bitbucket

Any automated pull requires using ssh instead of http protocol as the means to obtain code from the remote repository. This requires setting up an ssh key on Bitbucket and on the local machine where your test environment is configured.

Setting up an rsa key on Ubuntu is simple:

  1. run ssh-keygen -t rsa
  2. enter only the name of the key
  3. copy the public key (.pub) to your clipboard
  4. create  or open the config file in your home’s .ssh directory
  5. add your bitbucket host on a new line Host <host ip address>
  6. on a new tab spaced line enter the name of your private key file generated in steps 1-2 as Identity File <private key file name>
  7. log in to Bitbucket
  8. click on your avatar
  9. click on Manage Account
  10. navigate to SSH Keys on the left hand menu
  11. click on add key towards the center of the page and simply paste your key

Step 2: Creating a Webhook in Flask

A webook can be setup simply in Flask. James Innes of Ogma Development wrote a terrific article on how to setup a webhook in Flask. After creating the project as specified, run the code using:

flask run

It is possible to change the application host by simply modifying app.run() in your hook as follows:

app.run(host=<host>, port=[unreserved port])

Note the host, port, and token if applicable when your application is running and perform a GET request with the appropriate token from the machine or container running your Bitbucket repository.

Step 3: Setting up Webhooks in Bitbucket

Webhooks in Bitbucket can occur at the project or repository level. For the best control, this article explains how to create a hook at the repository level. Imagine triggering a multitude of unnecessary webhooks when modifying a single repository.

Setting up a webhook on a Bitbucket repository is simple:

  1. Log in to Bitbucket
  2. navigate to your target project
  3. Navigate to your repository
  4. Click on the settings gear in the left hand menu
  5. Click on webhooks in the left hand menu
  6. Enter a name for your hook
  7. Enter the url of your webhook server (of the form http[s]://webhook[?verify_token=<token>] in this context)
  8. In the options menus, ensure that push and merged are clicked
  9. Click save at the bottom of the screen

Currently, Bitbucket tests these connections with an empty POST request. This is not handled by our code and empty POST requests are generally discouraged anyway.

It is now possible to simply execute a pull request from a module under the webhook function in your Flask application on every merge in or push to the repository. This can be used to synchronize code and even keep a test site constantly ready for testing if you are using Django. As Flask is single threaded, it is better suited for the task of creating hooks.

Create a Build in Bamboo

With our repository thoroughly connected to the remote server, we can create build plans using Bamboo. Bamboo organizes the build flow  underneath a plan into jobs which maintain tasks.

For an annual fee of just ten dollars, it is possible to obtain ten free jobs with an unlimited number of tasks.

Future articles will explain how to integrate applications built with Django, Celery, and PyTorch. Still, even databases created using Djang are recreatable using the only the models and shell commands from the Script task.

As stated before, Bamboo can be set up to notify an appropriate developer on a build failure.

Conclusion

With these elements in place, it is possible to keep running test code up to date and continue to run builds against this environment with minimal human interaction. This greatly reduces labor and thus cost which can be passed to the end user in the form of savings.

Advertisements

Agile is Flexible: Deploying Agile in a Team Environment

Agile is not strict. In fact, agile is meant to be flexible. The concept comes with choices. Does your team use Scrum or Kanban? Are your retrospectives long or short?

This article examines the deployment of Agile, knowledge gleaned from deployment in multiple organizations.



Agile

The principles of agile are meant to provide flexibility and understanding. A few telling signs of this are forthcoming in the agile manifesto, which promotes:

  • simplicity by maximizing efficiency at work
  • early and continuous release of product
  • cross-organizational communication
  • working software as the primary sign of progress
  • embracing change

Deployment Headaches

The agile concepts and practices are some of the easiest to learn but maximizing efficiency does not mean having seven hours of meetings. It does not mean creating an unproductive comradery either.

Some of the pitfalls of deployment include:

  • meetings that eat away at productivity rather than help due to their length
  • a barrage of questions from various project managers that further eat away at productivity
  • stakeholder meetings that degrade into requirements analysis if there is no product to release yet
  • tickets that move into a Sprint or up a Kanban board in a way that slows the process down

Headache Exemplified

Issues in deployment can cause a backlog to grow uncontrollably. The amount of work can grow if agile is not deployed correctly and flexibly.

For instance, at a recent company I worked for,  the daily barrage of questions and meetings for the better part of a morning or day caused a significant slowdown in release times. The backlog grew by up to twenty tickets each week as pressure to complete tickets created more bugs.

Mounting pressure nearly created legal problems. In fact, as a new developer, I was the scapegoat for at least one problem, having a ticket offloaded onto myself only after a client demanded resolution and threatened a lawsuit.

Alleviating the Headaches

It is possible to alleviate the pain of deployment with a few simple measures including:

  • finding effective project managers and team leads for each team that are the sole source from which questions and tickets are generated
  • keeping meeting times small and encourage team members to find problems and solve them by embracing analysis of problems and their solutions at the appropriate meeting (some organizations look at an attempt to find the source of an issue as making excuses)
  •  finding an effective meeting length, often dedicating less time than recommended
  • planning stakeholder meetings effectively around the release of tangible product and using the project manager wisely to keep in contact with and update clients
  • utilizing software appropriately, keep teams as separated as needed

Clearly, there are decisions to be made regarding separation and time. Failing to address these problems can be deadly.

Tinkering with these resolutions, I have found that stakeholder meetings happen sporadically at first, ensuing as recommended after a product becomes substantial. Requirements gathering is itself a continuous project. I have also found that early stage projects can often make better use of Kanban, allowing for more flexibility, while established projects make better use of Scrum, when tickets are defined at the backlog grooming and retrospective.



Conclusion

While agile methodologies are effective, they are by no means inflexible. They were not intended to be. Using your time effectively is at the root of any project management philosophy.

Proper use of agile with a precise application for your organization is the key to maximizing effectiveness.