Private Immutable Configuration Hack in Python

Pyhton is notoriously not secure by default. However, it is still possible to generate a level of security through name mangling and other means. This article explains how to use name mangling, the singleton pattern, and class methods to create more secure access to configuration in Python3.

Singleton and Class Methods

Setting up a singleton in Python is simple:

class ChatConfig():

    __config = None

    class __Config:

        def __init__(self, config):
            self.config = config

    @classmethod
    def set_config(cls, config):
        if cls.__config is None:
            cls.__config = config


    @classmethod
    def get_config(cls):
        return cls.__config

 

The internal class and variable are mangled so as to make the variable itself private. This allows a single configuration to exist across the different packages while keeping the internal variable private and allowing for the variable to be set only once.

Conclusion

Python is not entirely insecure. It only takes code. This article offers an example of a way to set a variable once and share the setup among multiple packages.

Advertisements

XMPP v. AMQP

There are a few old debates with little interest given to them. This article examines the effectiveness of the XMPP and AMQP protocols and provides recommendations for areas of usage.

My own startup is using both for different reasons, XMPP serves as our bi-directional communications and multimedia backbone while AMQP through RabbittMQ serves as our content provider, uni-directional communications tool, and task queue engine (through Celery and our custom backend tool.

XMPP

Xmpp is now going on two decades old. It is solid and broad. However, it is XML based so it may be slow. Still, Simplr Insites, LLC chose XMP over MTQQ since it can handle both audio/video and chat communications.

XMPP communicates using stanzas which wrap different XML tags containing a wide array of information. The link above provides an excellent starting resource for learning XMPP.

XMPP offers:

  • bi-directional communications
  • p2p and multi-user communications
  •  information regarding presence and status
  • supports video and audio through Jingle or another client
  • large scale servers such as eJabberd and Openfire
  • has the strophe.js library for implementation

XMPP is not:

  • a great solution for the producer consumer pattern
  • requires decent hardware and is easy to abuse (overuse of the presence component)

AMQP

While we found XMPP to be a great tool for communications and multi-media. We also moved to task queuing upon finding the actor method to be too cumbersome in Python.

XMPP requires passing a large amount of XML and has no subscribe-able queues. AMQP libraries, these are protocols, offer robust structures and implement many enterprise patterns which are useful in data processing.

AMQP solutions tend to offer:

  • robust enterprise patterns
  • robust flow processing
  • task queuing libraries with speeds slightly slower than Akka (the best-in-class for everything library)

AMQP solutions tend to not offer:

  • state and presence support without extensive implementation by the developer
  • robust bi-directional queues for large scale communications (there are bi-directional queues)
  • audio/video support

Conclusion

If considering moving away from Scala/Java and thus the actor model and towards different platforms while requiring proven and robust real-time frameworks, XMPP libraries and AMQP libraries should be considered for their advantages. XMPP solutions work well in communications and real-time, non-flow based processing. AMQP solutions work well, not as well as Akka, in flow processing.

Opinion: Why Self-Driving Cars Will Fail in Customer Service

All to often, we are faced with a technology we feel is innovative and will break down an entire economy. That seems nearly certain with delivery, right? Wrong. If anything, the next 20 years will see a critical failure in an industry better suited to long haul trucking, train driving, and other even more mundane or menial tasks.

So many decent initiatives in technology fail because they do not consider the most important end games, people, security, and cost. In self-driving delivery game, that comes in the base pay vs. maintenance and LIDAR debate, security, and the customer services side.

There is an important reason to consider delivery as the improper industry for self driving cars, customer service.

No matter how many times a person is called, how hard, the door is knocked, or how friendly the front desk is. 25% or more of deliveries require problem solving well beyond route finding to provide both quality customer service or just complete a sale. This is well beyond the basic people enjoy interactions with others more than vehicles, another crucially important element.

Even a UPS driver will find that person just too completely unaware of their surroundings or un-trusting of their phone to respond to the door.

There are also the slew of factors not capable of being solved by a car that cannot fit through the front entrance, the factors that can drive the average tip over $5.

  • some high end establishments require talking to the front desk and arranging pickup due to ‘security’ reasons
  • some people require two or three approaches to reach
  • there are an increasing number of gated neighborhoods hiding miles of housing
  • People need an outlet to complain and the busy storefront will just hang up on them. They accomplish satisfaction by lodging a complaint with your driver.
  • stores tend to shift blame for failed quality control to drivers as near independent contractors to avoid the blame being placed on the store
  • many people, whether they consciously realize it or not, actually order delivery because someone comes to their door with good etiquette, a smile, and an assurance that their order is 100% correct
  • people tend to feel more secure when a trusted agent is in control of their goods and makes this fact known.
  • wrong orders made by everyone and everything are caught with good quality control and pizza chains follow CASQ better than most IT companies to achieve near 95% success

Imagine if every store lost even 15% of their business. Chains and restaurants paying drivers are already stretched to their capacity in a delivery radius that cannot be changed. That 15% will hurt and possibly close a store.

Consider, next, the lesser factor, maintenance and vehicle costs.

The cost of maintaining an electric car is, in fact more expensive than the cost of maintaining a fleet of drivers. If the average drive is paid $9.50 on average ($7.25  if tips are not equal to this price or you work for a better store attracting better quality employees + $10.25 per hour in store + $1 per mile), and the cost of LIDAR maintenance is roughly equal in addition to the cost of additional IT support staff and technicians, the store loses money.

That is not to mention the expenditure per car. The base delivery vehicle in the tech company’s target industry is $0, the driver provides the vehicle. The cost per vehicle for  a fleet of 13 cars running constantly is much higher even if by the time of writing this that cost goes down significantly. There is the $17000 base per vehicle, the breakdown of even electric vehicles and their maintenance at $120 per month, any electricity at $30 per vehicle per month, the cost of wifi at $100 per month per vehicle, $500 for an entire phone line setup for your fleet per month, and the cost of  wear and tear at $.50 per mile due to the technical nature of the vehicle. Gasoline costs will range between $20 and $40 per night if the vehicle is fueled by gasoline. A new car will be required every 4 years. Repairs can add thousands of dollars per year not considering LIDAR as the vehicle ages.

Finally, let’s consider security. In the past few years, the military lost at least 2 drones to Iran who simply launched a DDOS attack on them. How hard is it to lob requests at a cars network? Not difficult. Ask Charlie Miller.

If every drone carried $1000 in cash on a good day and there were dozens of drones to hack, that is more money than even a help desk employee makes in an entire year. Most attacks can be bought from the dark web. There isn’t as much skill involved in hacking as there used to be.

In sum total, delivery is the wrong industry to target self-driving vehicles towards.

There is a reason delivery drivers in my city, Denver, can earn $21- $25+ per hour, this is my part time go to when starting a company so I know this is true. That reason is not the store which usually pays $9 per hour on average per driver. The reason is the bizarre nuances of the delivery game.

It does not matter whether you are driving for brown (UPS), FedEx, or Papa Johns. That extra mile can make your business.

The right industry will always be long haul trucking, train engineering, and, to a lesser extent, air travel. Anywhere the task is more monotonous and there is no customer service, people are more replaceable. For the food industry, that means the back of house.

A New Kind of CRM

CRM software is not IT, creative, technology, or manufacturing oriented. Aside from a plugin for JIRA, there is not a lot dedicated towards project and lead generation in the CRM space. This article introduces an effort my company Simplr Insites, LLC is committed to which will create a CRM capable of not only separating clients from infrastructure project workflow tools but that can adequately group tickets for generic project creation and provide useful statistics.

This project is being deployed by Simplr Insites, LLC for use with our clients. This tool will likely have other features built in to supplement JIRA such as work-flow test integration with our backend tool.

The Problem

IT solutions in the CRM, manufacturing, construction, and technology space are oddly lacking. There are project management tools and a few high priced options but most stick to the old chat server systems and barely scrape the surface of lead generation.

So what is needed in this space:

  • the ability to converse which is already beat to death
  • the ability to manage a project which is fairly well done
  • the ability to spot similar tickets, problems, and issues to limit redundancy
  • the ability to generate group leads based on these tickets
  • the ability to share samples
  • role separation directly related to the space

The Solution

Simplr Insites, LLC is creating an open source attempt to get around this issue. After all, we are a low profit company controlling non-profits and public goods companies. We aim to tackle all of these issues using Python, Flask, JavaScript, JQuery, Bootstrap and connections to Celery ETL.

This project:

  • creates the conversational and ticket management tools we all love
  • adds the analytics we need for lead generation and genericism through NLP and AI
  • separates clients, project managers, developers, and observers with varying levels of access
  • allows sample sharing
  • the ability to kick of data validation tasks on Celery ETL because we need it and not because it is related
  • other tasks

It is 2018, let’s create something useful.

Conclusion

Want to help, join us on Github. While your at it, check out CeleryETL which is an entire distributed system which Simplr Insites, LLC currently uses to handle ETL, analytics, backend, and frontend tasks.

IT Suffers from a Personality Problem

I have been running a low profit company offering services to non-profits that is normally breaking even or paying little for about one year now. That means that if I cannot provide generic, low cost, results that continue to yield dividends through modularity, automation, infrastructure-as-code, and autonomation, my firm will cease trading.

To be certain, our products are also finding increasingly lucrative markets and I have learned who needs IT solutions and who has managerial or other issues when dealing with clients. For instance, one of the non-profits we first worked with showed some tell-tale signs of needing top down revision before they could consider looking at their support infrastructure. Counselors hardly perform their work and barely record information and there is a significant communications issue, all of which they know they need to address. That pales in comparison to our established, grant winning partners.

While the insight helps, there is another lurking problem I have discovered among IT firms that limits growth and productivity. There is a serious culture problem failing to promote skill and often leading to a lack of will power to tackle serious issues. I may be speaking for Denver only but with 2/3 of my last salaried and contract positions showing signs of this strain, it is an issue.

This article addresses several of the key problems I have seen over the last 6 years that have led to my desperate attempt to turn my B-corp into a source of steady income. These are:

  • leadership without knowledge or care
  • failure in leadership to base decisions on actual results
  • a bad hiring process leading to unskilled labor
  • focusing on what looks good and not what makes good
  • a fixation on things that are easy to learn instead of on skill
  • a lack of will power from the resulting complexity

Not every organization I worked for, with the exception of one that I spent too long at, exhibited these traits. However, any one of them can kill growth or an entire firm.

The remainder of this article offers examples of these practices and their detrimental outcomes.

Leadership Problems

Quality comes from the top. Every program and test in QA and QC offers this essential knowledge. That means projecting values and enforcing middle management practices if you decide not to have a hands on approach. An open door policy that addresses issues is a must. Brushing things under the rug or simply firing complaining employees is not a good solution. Letting employees complain constantly and disrespect everyone while barely working as a glorified computer operator whose skill is eaten by his tools is another.

One of my recent contracts displayed a clear lack of capability in this regard. The leadership was absent, unwilling to make an attempt at understanding its field, and hardly presented clear objectives or goals. Decisions were simply based on a gut feeling with seemingly few objectives. There was no quality policy and an over-reliance on SAAS. Worse, there was no attempt made at actually understanding employee capabilities. The employees flying by the seat of their pants were more highly regarded because of their culture fit than the ones able to actually perform skilled tasks. This led to an extremely high turnover rate.

The result was a 16 person company doing the work of 4, 1/20 the number of clients, and a company whose employees complained constantly. While I do not condone sites like Kubuntu, the 1.9/5 quality rating from employees is a good indicator of how this company is performing. The mess being created will likely destroy this firm as they rely 100% on a few contacts from a previous life and continually fail to pick up new business. This is in part from a lack of understanding of the organizations issues and capabilities. As someone who would never pick up stock in a company like this based on the potential for failure, this company already seems doomed. Their revenue less than 1/7 than that of their nearest competitors. They offer much less.

A Gut Full of Fat

Fat can block useful processes in the body and cause heart failure. The same goes for a company. Over time, our biases preclude us from acting with reason. We often grow more irrational because we fail to see the truth and try to rely only on instinct. If that instinct is horrible from the start, that is another problem.

Again resorting back to the previous example. In this instance, the gut of my boss led to elaborate spending on SAAS, an absolute bias against building simple modular and healthy programs to do work, and an absolutely atrocious instinct leaving him to promote individuals who take years to do what my team or even myself were completing in a matter of 30 minutes in anywhere from 1-2 years.

The result is a doomed company as already stated.

Failing to Focus on the Necessary Systems

We all like things that look good but that can kill a company as quickly as bad instinct and a lack of care. Focusing on a website and neglecting the back-end is a true problem.

Consider another firm I worked for. Their focus on the front-end as priority led to a backend with hundreds of tickets per team, spaghetti code, and a lack of progress from highly skilled individuals. Failing to maintain basic software engineering principals is finding a company that would have grown to thousands of clients stuck at 200.

What needs to happen is a focus on the irreplaceable systems. Web frameworks are nice in that they are interchangeable. Pick one up today and you can exchange it for something better tomorrow. Your back-end needs to last and should take the same approach as creating a lasting framework. Good components are singular in mass and capable of great flexibility, scale, and expansion with ease. I have already cut enough costs out of the process following this method to have an entire development team and pay for our server resources for up to 200 clients. Tickets are solved in minutes with solid configuration and good code practices instead of needing to resolve 8 layers of abstraction while a client is threatening legal action on someone who just picked up your ticket and has only been with your company for 2 months.

Wrapped Up in Unskilled Fads and SAAS

On the whole, fads are way too popular. We are basing our idea of skill on something that erases skill. Not long ago it seemed that Docker was more valued than Python or being able to actually administer a system. The sad truth is that anyone with an IQ of 80 can probably learn Docker in 5-10 minutes well enough to deploy it in the field. I was deploying an entire test system in Docker in that time, building from no knowledge and creating a few scripts to run my growing distributed processing system built from the failures of the aforementioned companies. The same is true of ETL systems.

Now consider what being wrapped up in a fad does. It builds over-complexity and an unskilled labor force. Labor needs to be able to work with systems flexibly and interchangeably. Folks that are relevant for only one reason are a waste of resources. This is why here today, gone tomorrow is the new normal.

Consider the first example. This company hired individuals because they can work on Alteryx. This is an easy to learn tool, much easier then even Pentaho. However, these individuals cannot even program. I actually wrote an internal document on what an API was. This also left me explaining things as simple as escape characters, employees who looked at you without a clue as to how to perform a network request, and people taking weeks to learn basic regex. Regular expression can be learned on the fly with instant results. It also left me laughing, at home of course, about a mid-tier employee with no data skills who had difficulty understanding basic regression. This same employee decided to try working with a neural network, obviously not a math/CS background or even GS1550 capable person here, on extremely dirty data where it was impossible to even cover cases regarding names and addresses, the fill rate was so bad it was impossible to obtain an accurate model for predicting anything.

Obviously, the result is employees who could be replaced by a freshman in college or high school student with a few AP or community college courses under their belt. That hurts everyone. The other detriment besides a skills deficit was a reliance on $200000 of hardware and software where $45000 was required. The $45000 is the cost of the 2 Kahu bricks with storage and backup costs. Add the 12 extra employees doing less work at $50000 – $75000 and you start to see the problem. With 2 people running a similar number of employees, the inefficient company is running an over $800,000 deficit. As my own business picks up, this deficit will likely come down to $500,000 but that is not a small number.

Hiring on Likability Without Testing Skill

Hiring is an intimate process. It is expensive to have high turnover. People need to grow and develop at an organization. You need to make sure that they are capable of doing so.

Consider the individual who was not given a code test that would have led to him not being hired. He was improperly using loops, failing to understand XML and other forms of configuration, and generally 10-15 times slower than anyone else.

This one should be obvious. It leads to the same issues explained before.

Will Power

Combining any of these factors creates overly complex systems that fail to achieve operational efficiency. It slows down work, creates a significant cost burden, and ultimately can lead to the downfall of an organization. It also reduces will power, reflected in complaints and the quality assessment scale.

When a system becomes overly complex, employee productivity shrinks. At eight levels of abstraction, a simple task is done and the employee leaves for quite some time. I have worked at firms where the level of stress generated by complexity led some workers to drink and smoke weed at work. Honestly, I blamed the system built on a lack of care. They were attempting, through capable yet busy hands, to monkey patch the system into something easier and more powerful but the damage being done is approaching irreversible.

Conclusion

These issues, summed up before, are severe hindrances to progress. They are also easy to overcome:

  • rely on fact
  • hire for generic capability, knowledge, and skill as much as cultural fit
  • do not rely solely on your gut
  • use software engineering principals
  • analyze your company, find out what will need to be solid and what can be placed onto a framework, and use principals like modularity
  • think before tackling a fad, do not get wrapped up in it as the concept is likely more important
  • display quality leadership
  • etc.

Simple, yet way too often not performed.

Opinion: FERPA, HIPPA and State Compliance for Sensitive Data, Follow NIST Dammit!

nistident_fleft_300ppi

Legal frameworks for data security are needed now more than ever. Recognizing this long after the journalistic debate giving rise to FERPA and HIPPA, states are creating a range of protection. It is, in my opinion, a sad fact that legislatures are the force driving this in the wake of breaches at Target and the rise of the Silk Road.

Navigating these frameworks and avoiding being placed in front of your local school firing squad sputtering garbage is actually quite simple. Use NIST, follow it well, and add some common sense.

My state of Colorado recently enacted some of the toughest data protection laws in the country. The important pieces for tech companies are:

  • Partners need to be verified by the school. Third parties can be vouched for but should also seek verification
  • Data can only be used within the confines of the objective stated in the contract
  • Companies working with information should de-identify student data for external programs unless working with a trusted party and should only work under the confines of their contract with allowed parties
  • Information cannot be used for direct marketing purposes
  • A security plan must be presented, in full, and actually deployed
    • Use access roles, proper password practices; etc. There are a few companies I can think of that really don’t
  • Keep information in audit tables
  • A breech will result in a public hearing and intense scrutiny as well as a possible black-listing

Other states, Washington in particular, already have similar laws. Hawaii, a state we are working within through a partner, actually verifies organizations as research partners.

FERPA and HIPPA compliance is really no different:

  • Certain data cannot be given out even under FOIA
  • Protect against threats using reasonable practices
  • Use common sense

Nothing, of course is secure and nothing ever will be. We, as developers, strive to achieve a level of difficulty that dismays.

So, how do you protect data. The government answered that too through NIST:

NIST even maintains a set of acceptable hashes.

As always, remember that security is fluid. Use common sense, patch known breaches, don’t use your database system full of customer information to run your HVAC. You know, the basics.

Getting a Dictionary from ConfigParser

Something that might be little known to the community and yet immensely powerful is the ability to obtain a dictionary  from the configuration sections. I will simply leave this tidbit here for anyone interested.

config = None
if file and os.path.exists(file):
    with open(file) as fp:
        config = configparser.ConfigParser()
        config.readfp(fp)
        if config:
            config = config._sections()
            for section in config:
                ordered_dict = config[section]
                config[section] = dict(ordered_dict)

I hope that helps. Cheers!