Bamboo and Webhooks: Automating CI and Deployment

process_auto

The name of the game in technology is doing more with less. It is the core element in the drive for profitability. In today’s market, that means finding ways to reduce both the number of employees in an organization and improve the processes that allow for such a reduction.

The days of paying $80,000 for an employee with limited knowledge, only ETL or only XML, are waning or dead. Even more importantly, the cost of software bugs in 2016 reached $1.1 trillion.

Reducing labor costs and reducing the cost of software bugs means producing efficient, clean, easy to follow, and effective systems backed by quality processes (word play with quality assurance intended). This article dives into how to use Bamboo, Flask, and Bitbucket to achieve continuous integration and continuous deployment to a test environment.

The concepts in this article can be ported to a distributed environment using ngrok but hopefully a more secure option is available. The ngrok solution is explained well on the Bitbucket website but promotes non-free software and external providers. This article intends to explain how to do this for free without linking to an external provider.

Environment

An automated test environment can be set up with:

  • Bitbucket
  • Flask
  • Bamboo
  • An Ubuntu or Linux Server for the purposes of this article

Bitbucket is a powerful repository based on git. Part of the repositories power is the ability to add webhooks. These are essentially triggers that perform POST requests to a webserver on specified events.

Webhooks require a web server. Flask is a lightweight and easy to maintain, single threaded framework. This is perfect for most repositories that do not see a significant number of pull requests.

Other hooks, including to Jenkins, are available as well.

Another powerful feature of Bamboo worth noting is the ability to organize repositories into projects. This suits companies with well defined architectures extremely well.

Bamboo is a continuous integration (CI) tool, a tool that allows code to be continually built and tested from a Bitbucket repository with minimal human interaction. This tool integrates directly with Jira, Bitbucket, and nearly every other Atlassian product.

Automating a System

Most companies with a forward facing website have production and test environments. Integrating and automating the environment is made possible with the use of a continuous integration (CI) environment and webhooks.

Bamboo can be set up to automatically synchronize code when a commit or merge is made in the remote repository. However, jobs in this CI run in an isolated way with no access to write to the local machine beyond SCP.

While a CI ensures that code runs correctly and can be set to notify users of failed builds, the synchronization issue outside of the isolated build environment is solved with webhooks. Calls to web servers specified in these hooks are triggered whenever a certain event is performed in the repository, allowing for more automated and instantaneous regression and integration testing.

Step 1: Setting Up SSH in Bitbucket

Any automated pull requires using ssh instead of http protocol as the means to obtain code from the remote repository. This requires setting up an ssh key on Bitbucket and on the local machine where your test environment is configured.

Setting up an rsa key on Ubuntu is simple:

  1. run ssh-keygen -t rsa
  2. enter only the name of the key
  3. copy the public key (.pub) to your clipboard
  4. create  or open the config file in your home’s .ssh directory
  5. add your bitbucket host on a new line Host <host ip address>
  6. on a new tab spaced line enter the name of your private key file generated in steps 1-2 as Identity File <private key file name>
  7. log in to Bitbucket
  8. click on your avatar
  9. click on Manage Account
  10. navigate to SSH Keys on the left hand menu
  11. click on add key towards the center of the page and simply paste your key

Step 2: Creating a Webhook in Flask

A webook can be setup simply in Flask. James Innes of Ogma Development wrote a terrific article on how to setup a webhook in Flask. After creating the project as specified, run the code using:

flask run

It is possible to change the application host by simply modifying app.run() in your hook as follows:

app.run(host=<host>, port=[unreserved port])

Note the host, port, and token if applicable when your application is running and perform a GET request with the appropriate token from the machine or container running your Bitbucket repository.

Step 3: Setting up Webhooks in Bitbucket

Webhooks in Bitbucket can occur at the project or repository level. For the best control, this article explains how to create a hook at the repository level. Imagine triggering a multitude of unnecessary webhooks when modifying a single repository.

Setting up a webhook on a Bitbucket repository is simple:

  1. Log in to Bitbucket
  2. navigate to your target project
  3. Navigate to your repository
  4. Click on the settings gear in the left hand menu
  5. Click on webhooks in the left hand menu
  6. Enter a name for your hook
  7. Enter the url of your webhook server (of the form http[s]://webhook[?verify_token=<token>] in this context)
  8. In the options menus, ensure that push and merged are clicked
  9. Click save at the bottom of the screen

Currently, Bitbucket tests these connections with an empty POST request. This is not handled by our code and empty POST requests are generally discouraged anyway.

It is now possible to simply execute a pull request from a module under the webhook function in your Flask application on every merge in or push to the repository. This can be used to synchronize code and even keep a test site constantly ready for testing if you are using Django. As Flask is single threaded, it is better suited for the task of creating hooks.

Create a Build in Bamboo

With our repository thoroughly connected to the remote server, we can create build plans using Bamboo. Bamboo organizes the build flow  underneath a plan into jobs which maintain tasks.

For an annual fee of just ten dollars, it is possible to obtain ten free jobs with an unlimited number of tasks.

Future articles will explain how to integrate applications built with Django, Celery, and PyTorch. Still, even databases created using Djang are recreatable using the only the models and shell commands from the Script task.

As stated before, Bamboo can be set up to notify an appropriate developer on a build failure.

Conclusion

With these elements in place, it is possible to keep running test code up to date and continue to run builds against this environment with minimal human interaction. This greatly reduces labor and thus cost which can be passed to the end user in the form of savings.

Advertisements

Discovering Sharable Resources in a Microservices Environment

mealsharingapp1

At times, it seems wise to have applications share resources. This extends to microservices

In the last article, I examined where boundaries might be placed between microservices. This article continues this examination by discussing which resources might be shared.

Related Articles:

Segmenting Microservices

Security in a Microservices Environment

When to Share

In this case, sharing means turning a set of potential microservices into a single microservice. Our experience suggests that services are cominable when:

  • there is no chance of a producing a security risk
  • services share more than just a database backend
  • services require similar resources OR
  • services use different resources on the same system with different levels of intensity
  • services will not require scaling on their own
  • services are maintained in the same manner and share code

Always consider resource usage and security. Any system set to scale beyond current resource usage or where a hardware gap is recommendable should be separated.

If done appropriately, combining microservices can:

  • reduce cost
  • reduce resource usage
  • increase speed
  • increase ease of maintenance

Example

Consider a set of services related to licensing and access grants, not passwords. These services are often symbiotic as a license often carries what oauth considers scopes, rights to use certain components of the system.

These services can easily be combined and shared with tokenized access for any frontend. When using frameworks such as Django, the service avoids constant bombardment as well.

Conclusion

Combining microservices helps reduce cost. Knowing what to share is critical. Sharing is caring unless it hurts.

 

The Case for Microservices, Where To Segment

micro

There is a growing need for microservices and shared services in the increasingly complex and vibrant set of technologies a true IT firm runs. Licensing, authentication, database services, ETL, reporting, analytics, information management, and the plethora of tasks being completed on the backend are impossible to accomplish in only a single application.

This article examines boundaries discovered through my own company’s experience in building microservice related applications.

Related Articles:

Discovering Sharable Resources in a Microservices Environment

Security in a Microservices Environment

Segment On Need and Resource Usage

To be fair, where segmentation of systems occurs depends on the need for each service. Clients may need a core set of tasks to be completed in one area or another. Where those needs diverge is a perfect boundary for establishing a service.

For instance, our clients need ETL, secured cloud file storage, data sharing, text management, FERPA/HIPP and legally compliant storage of data, analytics, data streaming, surveying, and reporting. Each of these areas encompasses one company or another but is cheaper done under a single roof to the tune of $7000 in savings per employee per year at a small to medium sized company.

Our boundaries are specified directly around needs, security, and resource costs. ETL encompasses one boundary due to computation costs, cloud storage another for security reasons, logging for legal compliance another, analytics takes up another service due to computational costs, stream and survey intake and initial analysis comprises another more vulnerable piece, and reporting yet another. Overlapping everything is a service for authorization and the authentication of access rights through oauth2.

The different services were chosen for one of the following factors:

  • resource cost
  • shared tasks and resources
  • legal compliance and security

Segmenting for Security

The modern world is growing increasingly security and privacy conscious. Including authentication systems and the storage of information on the same system as a web server is not recommended.

Microservices allow for individual applications to be separated and controlled. Access can be granted to specific clusters based on a firewall and authentication. Even user access control is easier to maintain. Hardware boundaries can be easily established between vulnerable pieces of a system.

Essentially, never stick a vulnerable frontend, streaming, or survey application on the same hardware as your potentially identifying initial file storage and always have some sort of authentication and access rights mechanism.

Results

Our boundaries are helping us scale. Simplr Insites, LLC dedicates individual resources as needed to each service. It also allows the company to offer a pricing scheme offering variable levels of services tailored to a customers needs more easily.

Some clients do not need an ETL system and only want case note management. That is possible. At the same time, granting GPU resources to the analytics cluster while giving our reporting cluster more RAM is as well.

In essence, Simplr Insites was able to reduce the cost of running systems in a 42 U shared space, possibly by as much as $5000 per month for our small company, while remaining more secure and delivering faster and tailored results based on the needs of clients through a single web frontend based SAAS application.

Conclusion

Discovering where to place microservice boundaries is critical to the success of an application. It relies on many factors ranging from resource cost, to the ability to share resources, and even legal compliance and security. Appropriate splitting of services can reduce cost and increase speed.

Can Romer-G Switches make you More Productive at Work?

The past few years have brought large cost increases to basic computing accessories. E-sports, tariffs, and, to an increasingly lesser extent, the saturation of computing have driven up prices.

The cost of a keyboard and mouse in my area is now over $100, sending me on a distracting morning quest when I should have been working. It is my fault for stepping on my mouse and keyboard, obliterating the USB housing and contacts.

The result of replacing my AST keyboard with a Logitech G413 is an 8 of 10 at a cost of $59.99 at MicroCenter.

The Romer-G’s are cheaper than their domed counterparts, last longer by 10-15 million key presses, are potentially more productive, and are more enjoyable. Not spending $99-$120 on an ergonomic keyboard, a wrist pad is only $8.99, was a great decision.



Ergonomics

These keys are actually ergonomic. Logitech claims that the indentation on the key caps help fingers slide into position.

This is true. Where my hands used to lie flat, I now find myself typing with the curl my 8th grade typing teacher would slap a gold star on.

Typing Speed

Logitech claims that the lower actuation point than domed and design can increase typing speed by 25 percent.

While the target consumer is a gamer, this concept should work both ways without producing redundant letters.

Never taking anything on face value, I ran a test.

I chose the Aesop Fables test on https://www.typingtest.com. The results:

  • Dome switch keyboard typing speed: 77
  • Romer-G  speed: 86

According to the test, my speed increased a whopping 9 words per minute. The test claims that my speed went from fast to pro.

This seems embarrassing considering I know someone who can type at 110 wpm.  Still, while this keyboard only gives a slight edge, less than 25 percent, the boost is almost noticeable. Also, my typing goddess of a friend is a former semi-professional Star Craft player, the one with all of the Koreans.



Click Factor and Pleasure

No review of a mechanical switch is complete without the ASMR factor. These keys are clicky. They feel better than domed switches. The ergonomic factor plays a role in this as my posture is never great.

Overall, these keys are faster, cheaper than most of their domed and Cherry counterparts, and tend to wear out fingers less. Modern mechanical keyboards use a more human friendly plastic.

Conclusion

The Logitech keyboard works well and the switches have a better feel than their rubber domed counterparts. While not as great as their Cherry counterparts, the Romer-G supposedly lasts longer and costs less than the ergonomic domed key keyboards. Being more responsive than a typical keyboard, a Romer-G keyboard can also increasing typing speed while helping combat carpel tunnel disease.