Better Key Storage With Blackbox, RSA, Redis, and the Fernet Algorithm in Django

safe

Python lacks a proper key store.  This is an unnerving issue when trying to build a secure application. More troubling is the plain text storage of RSA keys. This article examines a process for storing keys in an encrypted manner on black box as well as the storage of keys using the Fernet algorithm and encryption through RSA in Django using redis for speed.

Problem

Unlike Java which already has a key store, Python lacks the ability to store keys for data encryption. Python developers are left with only basic methods for storing keys and this often means doing so in plain text.

That method is inexplicably terrible when working with FERPA/HIPPA and especially the increasingly difficult state guidelines for storing sensitive information.

Solution

One solution, of many, is to use Stack Exchange black box to store keys and the Fernet algorithm to encrypt the keys in a cache. In this way keys are stored in an encrypted format in a hidden file as well as in a secure format in memory.

Black Box

Stack Exchange’s Black Box offers a perfect storage solution for keys using a gpg keyring to encrypt data. The tool was made to store secrets in a git repository.

Check out my Python API for reading files from black box. It is possible to add a user to the administrator file in order to avoid entering a password each time.

Storing Encrypted Keys in Django

Once the keys are encrypted and accessible, a large application needs to ensure speed. To help alleviate sluggishness, it is possible to store keys using the Fernet algorithm in any cache that Django provides.

It is possible to use the cryptography package for this task.

from cryptography.fernet import Fernet
from django.core.cache import cache

key = Fernet.generate_key()
f = Fernet(key)
token = f.encrypt(b"my deep dark secret")
cache.set('my_token', token)

Conclusion

It is possible to recreate a secure keystore using a mix of Stack Exchange Black Box and the Fernet algorithm when creating a Django application. The implementation above may not be production ready but is a proof of concept.

Security in a Microservices Environment

immunity

Too often, security is overlooked in a system. It is not enough to separate hardware or merely have an SSL certificate. There are some tricks to helping build secure microservices.

This article examines some of the security mechanisms available for helping prevent unauthorized access to a system. Notice the title related to security as nothing is ever 100 percent secured.

Related Articles:

Discovering Sharable Resources in a Microservices Environment

Segmenting Microservices

General Principles

While this article does not cover every potential security measure, a few are presented.

In general, however, a security policy and the following implementation should:

  • cover the entire system as a unit and examine each component
  • provide mechanisms for common logging, bug tracking, and monitoring
  • be weary of data visibility
  • analyze your data
  • provide a mechanism for security auditing and recommendations
  • include penetration testing
  • include policies regarding self-checking, monitoring, and violations
  • make everyone paranoid (cheers!)

Self-Checking

The most vulnerable point in any organization is the employee. Security experts are good but Lola at the front desk probably doesn’t really care as much whether her password is cat or randomized into a million pieces. Of course, long obscure passwords were found to be about as secure as cat recently so maybe Lola is a tad on the efficient side.

It is wise to:

  • phish employees (offer to take them fishing)
  • attack your system with brute force and deploy other mechanisms for evil
  • check for backdoors
  • penetration test using other means

There have been software companies where standard passwords are never changed while proprietary information is routed through the system. Imagine having the ability to type guest and admin into a console, setup a RabbitMQ router and instantly have access to every proprietary real estate listing.

Hardware Separation

Hardware separation is critical for the success of a system. Externally facing applications should never be placed on the same hardware as sensitive services meant for internal consumption.

In general the following should be separated:

  • a web application
  • data storage
  • services accessing data storage
  • services maintaining sensitive information

Simplr Insites, LLC, of which I am 50 percent owner, separates each of these types of services. With the increasing scrutiny placed on IT firms regarding personal information, privacy, and planning, this is not just a good thing but required by law. Colorado’s recent educational privacy laws demand that a security plan be presented for approval before an organization becomes a partner with a school district.

Password Protection

The most obvious point is to protect services via password on any forward facing component.  That means utilizing the most appropriate form of protection and staying abreast of security vulnerabilities.

Blowfish is broken but pbkdf is a recommended algorithm. Make NIST a regular part of your reading material.

Licenses and Tokens

It is not enough to secure front end applications, any access within the system should be authorized. Giving external services tokens for access and utilizing appropriate oauth2 toolkits helps to make unauthorized access more difficult. Again, nothing is impossible.

There is a difference between a license and a token in our systems. A license is often used to give general access to a system. A token grants access to components in the system. In this way, we can revoke specific access rights without invalidating a token or invalidate access to a system altogether.

Always remember to store tokens and licenses correctly and not in plain text anywhere within the system. Using an encryption algorithm such as Fernet encryption is a good start.

Encryption

Some data needs to be protected. This includes identifying information and licenses. It also means obtaining a trusted SSL certificate for each microservice.

Knowing which forms of encryption are considered secure is yet another reason to support the Boulder hippie coalition at NIST. They may be about peace and free love but are still concerned for their privacy.

When secured data needs to be transmitted, store a RSA private key at an appropriate place on the service and ensure that the front end maintains the public key. When data needs to be stored, Fernet encryption might be more recommendable. Of course, be weary of the date this article was published.

Monitoring Networks

Monitoring networks for unusual behavior goes well beyond intuition and Sentry, Zabbix, or the Elastic APM (used as an application logger) these days. These tools are terrific and must be deployed to tackle breaches alongside a strong firewall.

However, advances in neural networks and pattern matching allow security analysts to find anomalies. It might be possible to use a LSTM to detect and block unwanted users in real time.

To this end, companies such as Dark Trace are promising.

A combined approach is recommended as someone will be able to trick any security system. If it is possible, it will happen. Do not drink the cool-aid regarding cyber-immunity but do notice the word play.

Accounting

Related to monitoring a network, log everything that is feasible. If a user accesses a service or causes a job to be scheduled, log that job. If a service generates an error, log the error.

The Elastic APM is a terrific tool for logging, monitoring and bug reporting. Zabbix is available for monitoring hardware. Other companies I have worked for utilize Sentry for bug reporting as well.

Conclusion

A multi-pronged approach to security is a necessity in a micro-services environment. A decent system bridges the gap between services, monitors all traffic as a whole, allows learned security experts to find issues, tracks bugs in a single location, and provides much more.

Discovering Sharable Resources in a Microservices Environment

mealsharingapp1

At times, it seems wise to have applications share resources. This extends to microservices

In the last article, I examined where boundaries might be placed between microservices. This article continues this examination by discussing which resources might be shared.

Related Articles:

Segmenting Microservices

Security in a Microservices Environment

When to Share

In this case, sharing means turning a set of potential microservices into a single microservice. Our experience suggests that services are cominable when:

  • there is no chance of a producing a security risk
  • services share more than just a database backend
  • services require similar resources OR
  • services use different resources on the same system with different levels of intensity
  • services will not require scaling on their own
  • services are maintained in the same manner and share code

Always consider resource usage and security. Any system set to scale beyond current resource usage or where a hardware gap is recommendable should be separated.

If done appropriately, combining microservices can:

  • reduce cost
  • reduce resource usage
  • increase speed
  • increase ease of maintenance

Example

Consider a set of services related to licensing and access grants, not passwords. These services are often symbiotic as a license often carries what oauth considers scopes, rights to use certain components of the system.

These services can easily be combined and shared with tokenized access for any frontend. When using frameworks such as Django, the service avoids constant bombardment as well.

Conclusion

Combining microservices helps reduce cost. Knowing what to share is critical. Sharing is caring unless it hurts.

 

The Case for Microservices, Where To Segment

micro

There is a growing need for microservices and shared services in the increasingly complex and vibrant set of technologies a true IT firm runs. Licensing, authentication, database services, ETL, reporting, analytics, information management, and the plethora of tasks being completed on the backend are impossible to accomplish in only a single application.

This article examines boundaries discovered through my own company’s experience in building microservice related applications.

Related Articles:

Discovering Sharable Resources in a Microservices Environment

Security in a Microservices Environment

Segment On Need and Resource Usage

To be fair, where segmentation of systems occurs depends on the need for each service. Clients may need a core set of tasks to be completed in one area or another. Where those needs diverge is a perfect boundary for establishing a service.

For instance, our clients need ETL, secured cloud file storage, data sharing, text management, FERPA/HIPP and legally compliant storage of data, analytics, data streaming, surveying, and reporting. Each of these areas encompasses one company or another but is cheaper done under a single roof to the tune of $7000 in savings per employee per year at a small to medium sized company.

Our boundaries are specified directly around needs, security, and resource costs. ETL encompasses one boundary due to computation costs, cloud storage another for security reasons, logging for legal compliance another, analytics takes up another service due to computational costs, stream and survey intake and initial analysis comprises another more vulnerable piece, and reporting yet another. Overlapping everything is a service for authorization and the authentication of access rights through oauth2.

The different services were chosen for one of the following factors:

  • resource cost
  • shared tasks and resources
  • legal compliance and security

Segmenting for Security

The modern world is growing increasingly security and privacy conscious. Including authentication systems and the storage of information on the same system as a web server is not recommended.

Microservices allow for individual applications to be separated and controlled. Access can be granted to specific clusters based on a firewall and authentication. Even user access control is easier to maintain. Hardware boundaries can be easily established between vulnerable pieces of a system.

Essentially, never stick a vulnerable frontend, streaming, or survey application on the same hardware as your potentially identifying initial file storage and always have some sort of authentication and access rights mechanism.

Results

Our boundaries are helping us scale. Simplr Insites, LLC dedicates individual resources as needed to each service. It also allows the company to offer a pricing scheme offering variable levels of services tailored to a customers needs more easily.

Some clients do not need an ETL system and only want case note management. That is possible. At the same time, granting GPU resources to the analytics cluster while giving our reporting cluster more RAM is as well.

In essence, Simplr Insites was able to reduce the cost of running systems in a 42 U shared space, possibly by as much as $5000 per month for our small company, while remaining more secure and delivering faster and tailored results based on the needs of clients through a single web frontend based SAAS application.

Conclusion

Discovering where to place microservice boundaries is critical to the success of an application. It relies on many factors ranging from resource cost, to the ability to share resources, and even legal compliance and security. Appropriate splitting of services can reduce cost and increase speed.

Secure Your Data Series: Protecting from SQL Injection

I have been collecting data and building databases for a while. As I do so, I have come across a wide variety of mistakes in thought and code varying from the notion that a captcha works to a failure to adequately protect from SQL injection. Therefore, I am going to publish fixes to what I find here with examples utilizing Groovy or Spring and Java.

First up, the SQL injection attack. Attempts to stop these attacks range from  using JavaScript to setting variables. I recently came across a site that made an attempt to stop these attacks by limiting wildcards and querying with a variable. However, changing the header by resetting the variable actually gave me access to everything on the site in one fell swoop.

Suggestions

A few suggestions follow:

  • Do not check anything with JavaScript, this is too easy to get around by simply rewriting the page.
  • Write rules in the back end to protect from unauthorized access using SQL. Check variable combinations.
  • Eliminate any suspicious non-allowable wildcards and look for key terms like LIKE, ILIKE,%, and = that are common for WHERE clauses. Select statements in SQL/POSTGRESQL follow a form similar to SELECT –data list– FROM –table– WHERE –variable— ILIKE ‘%answer%’
  • Track IP Addresses and web pages using a symbol table (HashMap and the like) to eradicate any attempts to plainly just post to a server when it is not an API. This should be done in addition to any viewstate tracking and I strongly encourage the use of event validation.
  • Use event validation if available or come up with your own.

Implementation

Most requests occur through a POST request and most of the requests are handled in Java using spring or a format such as aspx with Groovy Grails starting to become popular.

  
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.web.context.request.RequestAttributes;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;

@Controller
@RequestMapping("/path")
public class PostController{
      //RequestMethod.GET also exists
      @RequestMapping(value="/path",method=RequestMethod.POST)
      public ResponseEntity newAnswer(@RequestParam(value="stringq", required=true) String stringq,@RequestParam(value="",required=true) boolean wildcard){
           //prep response
           HttPHeaders headers=new HttpHeaders();
           headers.set("Content-Type","text/html; charset=utf-8");
           
           //get IP
           String ip=requestAttributes.getRequest().getRemoteAddr();

           String data=null;           

           //check for appropriate response page, assumes post-like nature with single url
           //makes a call to a HashMap entity setup at the beginning of the program
           // and accessible throughout the servers existance
           if(IPmap.get(ip).compareTo('appropriatePageURL')==0){
           
               //create variable data here with response JSON or string
           }else{
               //create innappropriate use message
           }
           return new ResponseEntity(data, headers, HttpStatus.CREATED);
      }
}

The hashmap in this case is a static hash map. However, for security (even if using package level access modifiers) a file may be the solution. This will add time to the response though.