This is more of a call to action for the Python programming community which I am moving back into after a long hiatus. Yes, I have reviewed PEP8 standards since my last employment code tests. There is still a desperate need by serious and innovative production environments in ETL, ingestion, AI, analytics, and IOT to address the GIL with an alternative at the moment and create a strong distributed framework. This call to action supports that.
Spark is not an end all and while the Apache environment is far superior in terms of distribution its infrastructure offerings the Java environment is laggard in tool support.
Despite the strong tool support, Python was the injured man at the party not long ago. It was easy to learn but terrible to run. Even through Python 3.2, the GIL had a tendency to cause thrashing and third party libraries such as twisted entered the lingo of concurrency. Python was for pet projects and a novelty used at a few companies for large systems but Java was king and Scala was popular. Distribution was nowhere to be seen until PySpark came out and even then it was a secondary tool for quite a while with less than cutting edge features.
That has changed. Since Python 3.5 (late 2015 to late 2016), asynchronous behavior started to take center stage and transform the way Python works. In 3.2, the GIL was reworked and the folks at the Python Foundation seem to be pushing for its eradication. The wonderful people at the Spark project began supporting Python in early releases as well.
Sadly, a turn away from Scala by many projects has dashed the hopes I have for this extremely strong language and support network to become the JVM equivalent of Python. I still believe strongly that Scala will become a top tier language with the backing of groups such as Lightbend and a powerful quasi-open source foundation of its own. However, the hopes of a Numpy-like library from ND4J and first class Open CV support are floundering. It is also difficult to teach compared to Python.
Therefore, to support large projects benefiting from the constant uptime and serious concurrency of actors, I have created CompAktor. The aim is to support nearly all of the features of Akka in addition to strong peer grid support. It will hopefully grow into a strong tool for scaling enormous clusters performing everything from ingestion and camera feed processing to string manipulation. Come help Python grow.
We have a Trello based scrum board so email me at email@example.com if you want to participate. I will shoot you an invite. Thanks