Avoid Race Conditions with a Python3 Manager

 

race

In this short article, we examine the potential for race conditions in Python’s multiprocessing queue and how to potentially avoid this problem.

Race Condition

At times, the standard multi-processing queue can infinitely block despite the success of similar code. It appears that the reader-writer lock issue plagues the standard multiprocessing Queue. There is a bad egg at the table.

The following code runs without issue:

class OutMe(object):

    def __init__(self, q):
        self.q = q

    def do_put(self):
        self.q.put("Received")

    def rec(self):
        pass


class QMe(OutMe):

    def __init__(self, q, iq):
        self.q = q
        self.iq = iq
        OutMe.__init__(self, q)

    def rec(self):
        msg = self.iq.get(timeout=20)
        return msg

    def run(self):
        pass


class MProc(Process, QMe):

    def __init__(self, queue, oqueue):
        self.queue = queue
        self.oqueue = oqueue.get('oq')
        print(self.queue)
        Process.__init__(self)
        QMe.__init__(self, self.oqueue, self.queue)

    def run(self):
        while True:
            msg = self.rec()
            print(msg)
            self.do_put()


if __name__ == "__main__":
    mq = Queue()
    oq = {'oq': Queue()}
    mp = MProc(mq, oq)
    mp.start()
    while True:
        mq.put('Put')
        sleep(random.randint(0,3))
        print(oq.get('oq').get())

However, there are similar circumstances where the Queue blocks despite having a queue size larger than 0. This issue often appears when inheriting other objects and was submitted as a bug to the Python Foundation in 2013.

Python Manager

A python Manager coordinates data between different processes.  The Manager is a server process that is better capable of handling race conditions.

Where the above approach fails, it is more likely that the following will succeed:

if __name__ == "__main__":
    mgr = Manager()
    mgr2 = Manager()
    mq = mgr.Queue()
    oq = {'oq': mgr2.Queue()}
    mp = MProc(mq, oq)
    mp.start()
    while True:
        mq.put('Put')
        sleep(random.randint(0,3))
        print(oq.get('oq').get())

In this example, the queue from the previous code is created using a manager. The manager contains a proxy object in place of the queue.  The server process handles access to the queue.

Of note, the actual queue may contain nested yet shared items since Python 3.6.

Conclusion

Python is decades old.  This creates some problems. The standard multi-processing queue is plagued by race conditions. A python Manager helps resolve this issue.

Running a Gevent StreamServer in a Thread for Maximum Control

There are times when serving requests needs to be controlled without forcibly killing a running server instance. It is possible to do this with Python’s gevent library. In this article, we will examine how to control the gevent StreamServer through an event.

All code is available on my Github through an actor system project which I intend to use in production (e.g. it will be completed).

Greenlet Threads

Gevent utilizes green threads through the gevent.greenlet package. A greenlet, a Green thread in gevent, utilizes a similar API to the Python asyncio library but with more control over scheduling.

The greenlet is scheduled by the program instead of the OS and works more akin to threading in the JVM. Greenlet threads run code concurrently but not in parallel, although this could be achieved by starting greenlets in a new process.

In this manner, it is possible to schedule Greenlet threads on a new operating system based thread or, more likely due to the GIL, in a new process to achieve a similar level of concurrency and even parallelism to Java. Of course, the cost of parallelism must be considered.

StreamServer

Gevent maintains a server through gevent.server.StreamServer. This server utilizes greenlet threads to concurrently handle requests.

A StreamServer takes an address and port and can optionally be given a pool for controlling the number of connections created:

pool = Pool(MAX_THREADS)
server = StreamServer((self.host, self.port), handle_connect, spawn=pool)

This concurrency allows for faster communication in the bi-directional way that troubles asyncio.

 Gracefully Exiting a StreamServer

In instances where we want to gracefully exit a server while still running code, it is possible to use the gevent.event.Event class to communicate between threads and force the server to stop.

As a preliminary note, it is necessary to call the gevent.monkey.patch_all method in the native thread to allow for cross-thread communications.

from gevent import monkey

monkey.patch_all()

Instead of using the serve_forever method, we must utilize the start and stop methods. this must also be accompanied by the use of the Event class:

evt = Event()
pool = Pool(MAX_THREADS)
server = StreamServer((self.host, self.port), handle_connect, spawn=pool)
server.start()
gevent.signal(signal.SIGQUIT, self.evt.set)
gevent.signal(signal.SIGTERM, self.evt.set)
gevent.signal(signal.SIGINT, self.evt.set)
self.signal_queue.put(ServerStarted())
atexit.register(self.__server.stop)
evt.wait()
server.stop(10)

For good measure, termination signals are handled through gevent.signal to allow the server to close and kill the thread in case of user based termination. This hopefully leaves no loose ends.

The event can be set externally to force the server to stop with the preset 10 second timeout:

evt.set()

Conclusion

In this article, we examined how a greenlet thread or server can be controlled using a separate thread for graceful termination without exiting a running program. An Event was used to achieve our goal.

WebFlow Designer: Mediocre at Best

disappointed-face_1f61e

Web Flow is one of several visual HTML editors to come on the market in the hopes of replacing Komodo or Dream Weaver. While promising, it may be best to wait for Framer X before purchasing a visual editor.

Since the creation of Wix and the corresponding Wix editor, the idea of giving designers a no-code way to develop web pages has taken root in the development community. The reason such tools have not gained much in the way of popularity is easily apparent with Web Flow.

To be certain, Web Flow contains quite a bit of promise as both a design tool and CMS for simpler websites. The live content editors in particular are a nice to have for teams of any size.

The pros of Web Flow can be summed up as follows:

  • simple, fast, and live development for simple websites
  • basic editing support
  • basic script and style support through an embed tag
  • can be cheaper than a tool such as Dream Weaver for smaller organizations
  • can connect to article-style and other forms of data not requiring heavy back-end support

Just from the pros, it is obvious that a visual editor is perfect for the rapid prototyping associated with the design sprint concept. This is also not a difficult tool to create when considering the current power of Javascript and JQuery, especially when supporting a Mozilla based browser.

For each of the positive characteristics of Web Flow, there is at least one downside:

  • only allows style and script editing in an embed tag
  • contains almost no support for external CSS files
  • cannot be installed on site and does not use a company database
  • fails to provide advanced CSS features beyond a box shadow or border radius
  • class editing cannot be done separately from the generation of an element
  • a parent element must first be created to generate a new class
  • promotes sloppy, terrifying CSS that a middle-ware developer will quit over
  • is not drag and drop and has almost no such support in the editor

Basically, Web Flow and even tools such as Framer area far from ready for the production of modern websites.  Still, if you need to give a designer access to any form of tool that allows for inheritance based web editing or need slightly more than Wikipedia or Wix, Web Flow is decent. Still, Framer X promises to be a much better tool.

This tool, in the current form, is at best a 5/10. It is an average effort from what I hope is not an arrogant fool as can be the case with people who don’t want to actually work on their product that I at least hope will continue to improve to an 8 or 9 of 10.

 

 

 

Fixing Floating Parent Divs

It is frustrating  to float a child element only to find that every element appears to take the same positioning or find the child spilling beyond the borders of the parent. This brief article presents a modern solution to this issue using the display attribute in CSS.

Problem

At times when using the float attribute, elements placed in siblings  to a parent tag or even the siblings appear to inherit the floating properties of the target element. Elements may also appear to expand beyond the boundaries of a parent tag.

There are a few root causes of an element causing the parent’s siblings to appear to float including:

  • the zero height problem where an element is larger than its parent
  • the width of an ancestor is specified and violated by the child element

Solution

An older solution to the issue is to use the clear fix hack. However, with modern browsers, the solution is simpler:

.textedit-stylesheet-attr-div{
    display: flow-root;
}

Specifying flow-root causes the block format to be followed for the elements in the parent. This will maintain the float property of the children and cause the parent to ‘fit’ the child elements.

Popular Technologies

It is the fourth of July here in the US. Happy fourth. A few articles here continue to see plenty of readers per month years after being and some see practically none.

Out of interest, I decided to share  my statistics today. Each article I write is on a specific piece of a technology or about the technology in general.

Here are the sites most popular technologies on my blog:

  • PDFMiner
  • PostgreSQL
  • OpenCV
  • Edge Detection
  • Web Scraping
  • Browser tools such as JCEF and Java FX
  • Big Data
  • Security
  • Akka
  • Celery
  • Web Development Related Technologies

Scraping continues to be the most popular topic, possibly due to a link from PDFMiner. However, with 60 percent as many views, PostgreSQL takes second place. Scraping and PostgreSQL make up nearly every article with over 2,000 views. One PDF Miner article has had 25,000 views in the past several years alone.

I blog on things I use at work including topics such as:

  • concurrent programming
  • databases
  • mathematics
  • image recognition
  • scraping
  • browser tools
  • web development
  • big data
  • Java and Scala
  • Python

Who said java is dead?

Agile is Flexible: Deploying Agile in a Team Environment

Agile is not strict. In fact, agile is meant to be flexible. The concept comes with choices. Does your team use Scrum or Kanban? Are your retrospectives long or short?

This article examines the deployment of Agile, knowledge gleaned from deployment in multiple organizations.



Agile

The principles of agile are meant to provide flexibility and understanding. A few telling signs of this are forthcoming in the agile manifesto, which promotes:

  • simplicity by maximizing efficiency at work
  • early and continuous release of product
  • cross-organizational communication
  • working software as the primary sign of progress
  • embracing change

Deployment Headaches

The agile concepts and practices are some of the easiest to learn but maximizing efficiency does not mean having seven hours of meetings. It does not mean creating an unproductive comradery either.

Some of the pitfalls of deployment include:

  • meetings that eat away at productivity rather than help due to their length
  • a barrage of questions from various project managers that further eat away at productivity
  • stakeholder meetings that degrade into requirements analysis if there is no product to release yet
  • tickets that move into a Sprint or up a Kanban board in a way that slows the process down

Headache Exemplified

Issues in deployment can cause a backlog to grow uncontrollably. The amount of work can grow if agile is not deployed correctly and flexibly.

For instance, at a recent company I worked for,  the daily barrage of questions and meetings for the better part of a morning or day caused a significant slowdown in release times. The backlog grew by up to twenty tickets each week as pressure to complete tickets created more bugs.

Mounting pressure nearly created legal problems. In fact, as a new developer, I was the scapegoat for at least one problem, having a ticket offloaded onto myself only after a client demanded resolution and threatened a lawsuit.

Alleviating the Headaches

It is possible to alleviate the pain of deployment with a few simple measures including:

  • finding effective project managers and team leads for each team that are the sole source from which questions and tickets are generated
  • keeping meeting times small and encourage team members to find problems and solve them by embracing analysis of problems and their solutions at the appropriate meeting (some organizations look at an attempt to find the source of an issue as making excuses)
  •  finding an effective meeting length, often dedicating less time than recommended
  • planning stakeholder meetings effectively around the release of tangible product and using the project manager wisely to keep in contact with and update clients
  • utilizing software appropriately, keep teams as separated as needed

Clearly, there are decisions to be made regarding separation and time. Failing to address these problems can be deadly.

Tinkering with these resolutions, I have found that stakeholder meetings happen sporadically at first, ensuing as recommended after a product becomes substantial. Requirements gathering is itself a continuous project. I have also found that early stage projects can often make better use of Kanban, allowing for more flexibility, while established projects make better use of Scrum, when tickets are defined at the backlog grooming and retrospective.



Conclusion

While agile methodologies are effective, they are by no means inflexible. They were not intended to be. Using your time effectively is at the root of any project management philosophy.

Proper use of agile with a precise application for your organization is the key to maximizing effectiveness.

 

Using :before or :after in CSS, adding a border to a clipped path

Clip path is great but there is an issue. The border property stops working. Triangles, polygons, and nearly any shape can easily be produced. However, the borders are cut.

This short article examines how to resolve this problem with pseudo images, introducing before and after.

Problem

CSS renders the clip path last no matter how it is placed in a CSS class or in what order classes are arranged. This means that margins, borders, and other features are lost when the object is clipped.

Before and After

The solution to the issue is to use CSS to superimpose one div element on another through the use of a pseudo image. Pseudo images can be created using :before or using a standard class followed by :after.

Before and after render content as expected:

.small-tab-bl:before{
        content: '';
        width: 378px;
        height: 178px;
        display: block;
        position: absolute;
        background: #f9fafc;
        top: 1px;
        left: 1px;
        clip-path:
            polygon(
                 0% 0%,
                 100% 0%,
                 100% 100%,
                 7.5% 100%,
                 0% 87%);
        -webkit-clip-path:
            polygon(
                 0% 0%,
                 100% 0%,
                 100% 100%,
                 7.5% 100%,
                 0% 87%)
        -moz-clip-path:
            polygon(
                 0% 0%,
                 100% 0%,
                 100% 100%,
                 7.5% 100%,
                 0% 87%)
        z-index: 1;
    }

    .small-tab{
       position: absolute;
       width: 380px;
       height: 180px;
       background: grey;
       clip-path:
            polygon(
                 0% 0%,
                 100% 0%,
                 100% 100%,
                 7.5% 100%,
                 0% 87%);
        -webkit-clip-path:
            polygon(
                 0% 0%,
                 100% 0%,
                 100% 100%,
                 7.5% 100%,
                 0% 87%)
        -moz-clip-path:
            polygon(
                 0% 0%,
                 100% 0%,
                 100% 100%,
                 7.5% 100%,
                 0% 87%)
       z-index: -1;
}

This class clips a div element to create a tab missing the bottom right corner. Two tabs are generated, one positions a single pixel off of the other to create a border. Before must include the gradient and content should be provided.

Conclusion

Before and after are useful tools for placing content and alleviating the problems with clip path.

Can Romer-G Switches make you More Productive at Work?

The past few years have brought large cost increases to basic computing accessories. E-sports, tariffs, and, to an increasingly lesser extent, the saturation of computing have driven up prices.

The cost of a keyboard and mouse in my area is now over $100, sending me on a distracting morning quest when I should have been working. It is my fault for stepping on my mouse and keyboard, obliterating the USB housing and contacts.

The result of replacing my AST keyboard with a Logitech G413 is an 8 of 10 at a cost of $59.99 at MicroCenter.

The Romer-G’s are cheaper than their domed counterparts, last longer by 10-15 million key presses, are potentially more productive, and are more enjoyable. Not spending $99-$120 on an ergonomic keyboard, a wrist pad is only $8.99, was a great decision.



Ergonomics

These keys are actually ergonomic. Logitech claims that the indentation on the key caps help fingers slide into position.

This is true. Where my hands used to lie flat, I now find myself typing with the curl my 8th grade typing teacher would slap a gold star on.

Typing Speed

Logitech claims that the lower actuation point than domed and design can increase typing speed by 25 percent.

While the target consumer is a gamer, this concept should work both ways without producing redundant letters.

Never taking anything on face value, I ran a test.

I chose the Aesop Fables test on https://www.typingtest.com. The results:

  • Dome switch keyboard typing speed: 77
  • Romer-G  speed: 86

According to the test, my speed increased a whopping 9 words per minute. The test claims that my speed went from fast to pro.

This seems embarrassing considering I know someone who can type at 110 wpm.  Still, while this keyboard only gives a slight edge, less than 25 percent, the boost is almost noticeable. Also, my typing goddess of a friend is a former semi-professional Star Craft player, the one with all of the Koreans.



Click Factor and Pleasure

No review of a mechanical switch is complete without the ASMR factor. These keys are clicky. They feel better than domed switches. The ergonomic factor plays a role in this as my posture is never great.

Overall, these keys are faster, cheaper than most of their domed and Cherry counterparts, and tend to wear out fingers less. Modern mechanical keyboards use a more human friendly plastic.

Conclusion

The Logitech keyboard works well and the switches have a better feel than their rubber domed counterparts. While not as great as their Cherry counterparts, the Romer-G supposedly lasts longer and costs less than the ergonomic domed key keyboards. Being more responsive than a typical keyboard, a Romer-G keyboard can also increasing typing speed while helping combat carpel tunnel disease.

Differences in Tablets and Browsers Over the Last Few Generations

We all have to do this at some point, create an all encompassing, GUI program that works between the different generations of IPad and even up to 1920 x insane. Increasingly, as of 2018, it doesn’t seem to matter whether you are working in the back or front end, at some point you are creating some sort of front end. Making matters more complicated, a slew of useful and enticing features are starting to become available.

This article is geared towards examining the increasing scale of screen widths in tablets and the differences in available features over time.

Features

Browser features have increasingly grown more powerful. Just a few newer features are:

The links above lead to the MDN pages including browser support. In general, Microsoft Edge or Explorer nearly never supports modern features. However, Microsoft’s browser usage is dropping steadily with Edge taking less than 3 percent of the market and Explorer taking less than 10 percent.

Solid, responsive web applications can be built for Opera, Safari, Chrome, and Firefox without entirely alienating all users but with a slightly outmoded design using CSS hacks or even web frameworks through header parsing.

Firefox, Chrome, and Safari continue to lead the pack in terms of support for newer features.

Device Usage

In terms of devices, the variety of tablet brands is growing. This is leading to a growth in the use of Android devices. This will mean that browsers such as Chrome and Firefox will grow increasingly popular.

Apple continues to lose market share while Microsoft is gaining ground. This could be due to the poor practices of Apple in relation to copyrights and development.


Reactive Trend

The overall trend is towards responsive web development. Each page scales to nearly any resolution required with the exception of mobile which requires a separate site.

This trend carries towards features as well. The days of using stateful web design, jumping from page to page, are nearly dead. Each page is practically a

Future development and certainly my own are also abandoning the typical grid system. Divisions can now become polygons. It is possible to draw complex shapes with d3js.

Web Applications in Unexpected Places

Anything and everything is becoming possible. Web applications are achieving the same power as desktop and mobile applications. However, security will still be a concern.

This power includes on the monitoring side of the IOT sphere. With the ability to launch a browser through tools such as QT and PyQT,  web applications will start showing up in unexpected places.

Consequently, web sockets, RTC, MQPP, and XMPP  will likely grow in popularity.


Screen Sizes

Screens have grown in resolution at the low and high end as expected. Since 2012, screen resolutions have grown from 960 x 720 for an IPAD to 1024 x 768 as well as up to 1366 x 768 on desktop, up from 1280 x 800.

A basic website should be safe coding, in 2018 using:

  • 960 x 720: Older Generation IPads
  • 1024 x 768: Newer Generation of IPads and older desktop screens
  • 1280 x 800: For older screens
  • 1366 x 768: The most common resolution of 2018
  • 1920 x 1080: The future most popular resolution

To stay relevant, plan on coding for both Ipad resolutions as low as 960 x 702 and desktop resolutions as high as 1920 x 1080. My own sites split this resolution into 7 different tiers.

Of course, create a mobile site as well at about 420 width.

The tiers I use are:

  • 900 – 999 width
  • 1000 – 1199 width
  • 12000 – 12299 width (some computers have bizarre resolutions in this range)
  • 1300-1399 width
  • 1400-1599 width
  • 1600 – 1799 width
  • 1800+ width

Frameworks

Frameworks have not changed much but have gotten better. Django now supports channels, Spring has been fully coupled with boot for some time, and Flask is becoming more secure.

More interestingly, React has gained ground due to the power it maintains in building responsive applications. This is really nothing new.

Personally, I utilize Django these days with Flask for micro-services. This allows me to  maintain as single stack for both my front and back-end which utilizes Celery,  Thespian, PyTorch, and Python’s other powerful data tools.

 

””

Product Review: Canon Rebel T7i is a Great Starter Camera

We live in a world where phone cameras have roughly as many pixels as a digital camera. That said, there are an unrivaled number of products available for an SLR that greatly enhance picture quality and allow for high quality imagery without spending for a 250 mega-pixel camera or one with 4k video support.

The Rebel T7i is showing its age proudly and offers a great and relatively cheap way to move from your phone or handheld digital camera to producing high quality images. It may not offer a stock focal length at the level of a Nikon D3400 but is easier to grasp, more durable, and has the terrific support of nearly every local camera shop and Canon itself. With a wide variety of available lenses, I still recommend this camera over the Nikon D3400.

Well everyone, I am starting to add product reviews. In full disclosure, I am reviewing products from time to time (maybe 10 to 20 percent of my content), out of interest and profit.

My first rating here 7/10 (comparable to a Nikon D3400 but loses a point for the stock kit)



Rebel TI Qualities

  • Price of Camera and Accessories is an unbeatable $639
  • Supports a great number of lenses and features that allow a user to produce images better than can be obtained from a handheld device or digital camera
  • Produces high quality images
  • Decent Non-4k Video Support
  • Terrific AI
  • Durable
  • Lower stock lens focal length than the Nikon D3400

Overall Quality

Stock image quality on the Rebel Ti is no longer discernible from your typical high definition Samsung camera phone. However, mega-pixels are no longer a good way to define the quality of a camera.

The stock qualities of the Rebel Ti are actually what leads to the lower score. If you want  a camera that works better out of the gate and are not looking to dive into the world of photography, the Nikon is a slightly better option.

However, Canon is a well-established company with more visibility than Nikon. It is easier to find lenses in your camera store, equally easy to find lenses and accessories online, and of equal quality to it’s counterparts once you decide to accessorize. With a high quality lens, the T6 can even serve as a durable backup for a professional photographer or the main camera for someone just getting their feet wet.

All of the photos I sell were shot using either an older Rebel T6io or newer Rebel T7i, the newest mountain images were shot with the Rebel T7i.

The EOS also offers terrific video quality. Of course, the camera is not rated at 4k.

Finally, the backing of Canons terrific default settings make the Camera terrific for anyone wanting to try an SLR without knowing how to fiddle with aperture and iso. I particularly find the portrait and landscape settings great for comparing against my own skill set.

Durability

This may seem odd but, as an adventurer or photographer, you will need to find interesting photographs to survive. The old days of going down to the dock and snapping a quick picture are not going to cut the mustard.

Canon cameras and lenses are durable. My Rebel T6i still works despite being taken on dozens of backcountry ski trips, up many mountains, on top rope routes, on boat rides, on plane trips, and on many road trips. In short, I can mistreat the Rebel and still take a picture at the end of the day. That said, treat the camera with care. No technology lasts as long as Teflon.

This camera and even the lenses can take more of a beating than others.

Accessories and Lenses

The Rebel Camera is established with a large number of available lenses that are terrific for getting a hang of digital photography, starting to sell images, or making high quality imagery. Canon cameras receive high remarks for their accessories, glass, and tools that can easily help

The 18-135mm lens is a must for anyone looking to produce high quality lenses. At the time of writing a full kit including a telephoto lens and the wonderful 18-135mm lens is selling for $900 on Amazon. In the age of high-megapixels, it is creativity and glass that makes a photographer stick out.



Canon customer support is also terrific. The factory helps with repairs and the staff is knowledgeable.

Verdict

It is difficult to give this camera a 7/10. However, with some shortcomings, the result is inevitable. This is still my go to starter camera.