Why and How I Switched from Python to Erlang

WHY AND HOW I SWITCHED FROM python TO ERLANG

A PYTHON PROGRAMMER’S ADVENTURE

Version 1.0 | 7 Jul 2016

Farsheed Ashouri | CTO/RashaVas LTD.

Choosing The Right Framework

FULL STACK PYTHON Frameworks Are Honey Pots

PYTHON Frameworks Are Also Honey Pots!

I Don't Have Any Of These Problems! Go On…

I have solved all of these issues. What’s wrong with it?

Summary

In this article, I am explaining my trip from Python to Erlang. If you are not a Python developer (+ probably with a deep understanding of Python based web services), or you don’t need/want to scale thing to a very huge scale, you won’t be able to find this article much useful. If you don’t want to develop an infrastructure for your business or If you develop simple blogs, small asset management systems or Hello World-ish websites, This article won't help you at all and If you are about to choose a language to start, please do not decide based on my words. I am going to tell you what problems I encountered using Python and how Erlang is able to solve those specific problems for me.

I will start with a brief history and I will end with my conclusion. If you find yourself suffering from my reasonings, let’s discuss about it! That's the whole purpose of sharing my experience.

15 Years from Day 1

I started programming with MEL ( Maya Embedded Language ). Then I found a job and got my first pay check. Quickly I switched to Python for more serious development opportunities and I completed reading K&R for developing extensions for Python with C . Years past and I was increasingly interested in Web stuff. I quit Animation industry for good ( my demo ) and I was employed in a famous Tech company here in Tehran .

Recently I have developed Appido.IR , A Video/Music streaming service with Python.

Lets explain what my problems were.

Choosing The Right Framework

Everyone loves Django . I hate it for no reason! Probably because I helped Massimo Developing Web2py or maybe simplicity of Web2py made it impossible for me to select another full stack framework. I finally fully tried Django later on a boring project.

FULL STACK PYTHON Frameworks Are Honey Pots So, what's wrong with Django or even Web2py? Nothing! Until you start using Bottle / Falcon with some templating engines and database ORMs [you read Mako and SQLAlchemy] and you feel like those enterprise frameworks are very slow. For a simple RESTful API service, you have to waste your CPU cycles for no good reason. And for complex API services, you need to find a way to jailbreak and put together a completely new architecture inside your so called full-stack framework. Here is an example:

In Appido.ir Streaming Technology, we implemented Dash protocol with the help of FFMPEG and tons of other open-source tools. Appido has it’s own OAuth2 server, an Authorization system, a workflow engine, scheduling, monitoring, logging, debugging, reporting, accounting, ... You name it! After few weeks of RnD and struggling with Web2py/Django, It was obvious it’s not going to work with a single framework. Creating some folders and trying to be MVC or having an database admin panels in BIG FRAMEWORKS won’t help you scale! And finally some of those unwanted features will bit you! Sooner or later. Yes, there is some honey there. But you must be careful of angry bees . So I invented my own Falcon based framework.

PYTHON Frameworks Are Also Honey Pots!

So you start using Bottle/Falcon/Flask… And you find yourself in need to install task queues and scheduling modules ( Celery , RQ for example). Why do you need it? Because every request that takes more than 500ms needs to be stateful! Thats an unwritten rule. You need to give user status. Your clients can’t wait for your calculations. You need to put the heavy burden of sending emails, converting images, etc on Celery (or your own invented customized multi-process queue). What’s wrong with it? Let's see:

Imagine you have a streaming service. Client uploads 20Gb 4K raw video files, you convert files to 10 different resolutions and email him/her that results are ready.

You use celery with 40 workers. Videos start to convert and your server overloads and gets slow and slower. So you find a genius solution! Install another server, with streaming code and tools on it using Celery as a worker. Cool, Problem solved! NO! Not so fast!! In the middle of the night, You find out that 5 or your 6 servers have 0 CPU usage and the 6th has 100%. WHY? It turns out the Redis has a timing problem with Celery that prevents workers to pick jobs. (It has a solution of-course: Welcome to world of fanout-patterns and visibility-timeouts), Installing RabbitMQ would solved your problem (And would introduce other new strange problems). (If you developed such a system and did not find any of mentioned problems, Congratulations! You are the luckiest guy on earth). You found honey without getting bit by bees.

I Don't Have Any Of These Problems! Go On… So your service is working great and what you need is to increase your web service RPS (Requests Per Second) eit