On the importance of using Docker 🐳

On the importance of using Docker 🐳

   June 29, 2020       Wajdi Ben Saad

---

I have always talked about the importance of adopting agile methodology in analytics projects.

With the fast pace of data acquisition and the immense volume of data points collected every hour in certain domains,  being  able to iterate and deploy very fast is a vital necessity in today's cutting-edge era.

 fish swallows an Egyptian soldier in a mosaic scene depicting the splitting of the Red Sea from the Exodus story

I wanted to share some thoughts on a very important set of technologies that are widely used in certain organizations in the automation, the deployment and  the running of robust and stable data products.


I am most certainly sure that you have heard of Docker! That big blue smiling whale  🐳 

Below, I will boil down in simple terms what is it and why I believe it is an important tool in the field of data analytics.

What is Docker ?

Simply put, Docker is an open-source application that utilizes the container paradigm at its core functioning. It allows developers to eliminate hours of work every year, lost in doing repetitive work of setting up environments, installing OS and other applications and therefor allowing them to be more efficient and providing them with precious time that should be spent in more added-value tasks.

It allows the packaging of an application and its dependencies in a virtualized container, permitting its deployment in other environment without the hassle of thinking about compatibility and runtime issues.


Docker gives the possibility of making sure the application could be launched in an isolated environments, therefore providing the huge possibility of deploying it on premise, in a public or private cloud...etc

Docker is not the equivalent of a virtual machine because it does not require the user to worry about setting up the OS and its different dependencies. Its technology relies of utilizing the Linux kernel and the different host system resources (CPU, memory, storage...etc) to provide an isolated environment relative to the application.

Benefits of using Docker

Several benefits could be gained by utilizing Docker in an enterprise context. It goes without mention that the success of this technology comes at no surprise considering the gains it allows us to have >

Flexibility & Portability :

One of the major pain points for IT professionals while working with different sets of applications is the extreme complexity they face to guarantee that applications are smoothly deployed and ran on their production environment.

This complexity is doubled when there are constraints within the company if they have different sets of environments for development, testing and production. It is not always possible to have an iso-system for all these environments and sometimes upgrading the python version in a server might have disastrous consequences on other applications using the older version.

Due to its technology, Docker permits organizations to worry less about this issue and provides a certain flexibility and portability to its users. The core system and its dependencies are not effected by one application, each Docker container has its owns requirements installed within the container.

This makes the task of spinning up a Docker container and deploying it anywhere , a piece of cake! You don't need to think about whether or not the application dev team has thought of making sure the exact system requirements are respected, and offers this flexibility for the dev team to only focus on how to build a running software without the hassle of prerequisites installation and version compatibility .

Efficiency & Scalability :

You can not fully grasp the fact that with all its features, Docker provides a certain amount of efficiency. Gaining time that could have been spent in prerequisites verification and installation is a major efficiency feature. No more time lost due to inadequate configuration, error of compatibility and issues we all could face.

Docker's isolation feature allows developers to pick and choose their technological stack and use the most efficient version of any software to include in their application,  which means that is you believe that a certain programing language is more user friendly than other, or if you are more efficient in working with R better than python, Docker provides you with the possibility of using the best tools for you.

And because launching an application with Docker is a matter of a couple of clicks ( which can be automated) spinning up a container in few seconds provides a major scalability feature that could be utilized when demand is very high and there is a need to scale up or down a certain application.

Security:

Lastly and most importantly, the security feature provided by Docker is a less-known feature. By design, Docker ensures that every application running inside a container is isolated from other applications and containers, even if they are deployed on the same system/server.

Docker ensures that every container is completely segregated and isolated and from an architecture point of view, the system administrator have complete control over the flow of data and the interactions between different system.

Final thoughts:

I believe that data scientist and software engineers are better off focusing on enhancing their code, building efficient and user friendly application. Time lost in setting up infrastructure and dealing with compatibility issues is for me a lost investment .

Using Docker is in today's extremely competitive world is a must. It because some sort of a filter criteria for me to evaluate how efficient and competitive a certain company is, based on their techno stack and whether or not they utilize this type con technologies.

I should mention that the container based solutions are not limited to Docker. There are various other tools that can substitute it.

It goes without saying that some people have brought up the disadvantages of Docker in certain contexts. I am fully aware that in different situations, there should be an adapted solution/ tool.

Docker might have few drawback, however I believe that for a well structured organization, taking advantage of this tool, while keeping in mind its constrains (and doing something about them) is the answer here.


Feel free to share your thoughts about your experience with Docker in the comment section! If you need any advice about how to use Docker in Data Science applications, hit me up on Twitter!

Read More

Surviving a pandemic... Why some companies will not make it ?

Surviving a pandemic... Why some companies will not make it ?

   June 21, 2020       Wajdi Ben Saad

---

Over the course of history, the world has been through various crisis and chocs. One of the oldest recorded economic crisis was the one that happened about 2000 years ago, in the Roman empire.

It was simply caused by the loss of trade ships that sank in the red sea and pushed the prices of goods like ostrich feather and ivory, the crisis expanded to touch real estate, trade and agriculture.

Laurens Mort de Tibère

About the same time the great house of Malchus and Co. of Tyre with branches at Antioch and Ephesus, suddenly became bankrupt as a result of a strike among their Phoenician workmen and the embezzlement of a freedman manager.

These failures affected the Roman banking house, Quintus Maximus and Lucious Vibo. A run commenced on their bank and spread to other banking houses that were said to be involved.  This situation caused the banks to push borrowers to pay back their loans. Real estate value crashed because there were no liquidity available and the rest is history.


This happened again and again over the course of the following 2000 years. The reactions were almost always similar and the results were as catastrophic as you would imagine. However, something history proves to us is that humans don't learn from past events.

Today..

The current pandemic situation is one of those crisis. It is a big one, and with what economies around the world are implementing to survive it, are a proof that the hit will be hard and will change the course of history, again.

Let me clarify one thing : when I say it will hit hard, it does not mean it will be sudden, or fast. One thing we should all remember is,  for every action there is an equal and opposite reaction.

The world will soon pass "the peak" days of the virus. However, true recovery will take years, and the effects will be seismic especially if a "second wave" of the virus hits again.

Why seismic?

because the current crisis is similar to a sudden "earthquake". It has hit in an expected time, no country was prepared to deal with it and most importantly, no country has ever been prepared to deal with the "aftershock".

Currently the impact is immense, economies are on hold. Entire populations are in lock-down  or were on one ( almost 3 billion people) and issues in certain economies with weak and unprepared corporate structure are pushing the catastrophe even further down the rabbit hole.

In this article, I will lay down why I believe that some companies will not make it through this crisis, and I will talk briefly about the 2 most impactful factors of this decline.

What has happened / is happening ?

As many people were joking about it online, but this pandemic has driven innovation and digital transformation within various organizations. It has accelerated trends that were already reshaping and transforming our world as we know it.

The shift

Let us be honest, for those working in 1000+ employees corporations ( and also the smaller ones ), do you remember what was the "work from home" policies in before march 2020?

It was almost impossible in certain companies/countries to have permission for a half-day remote work.

Culturally, with the industrial age, the physical presence of "resources" in the factory was mandatory. You cannot get paid if you take some pieces of machinery and assemble the product at home or somewhere else and bring them the other day. It was not possible to do remote work.

This culture has evolved with time and with the evolution of technology. However, technology moves faster than "laws" or "rules".

The rule was : You have to be there, your boss has to see you work and should be able to check and control what you are doing.

Yes you can deliver, however, you have to do it in the corporate headquarters, from 8 to 17, with a "programmed" lunch break that you have to respect.

And what happened with the pandemic was the following :

Being physically present in the office became some sort of a risk. Yes, coming to work meant you will leave your home, commute in public transportation in some cases,  be in contact with other people on your way, probably get infected (or not, but who's up to take the bet) and this would make you a possible carrier of the virus.

Companies couldn't risk that. Why? Well,  companies didn't want to jeopardize all the resources they have in the time of crisis. So an "emergency-transition" and new work-schedules were put in place to distribute the working force into teams that would alternate between being present and being away ( till now, we haven't talked about remote work yet).

My reasoning here apply to almost all sectors of activity. I am not including technology-based corporations because these"emergency-transition" schedules were also applies in shops, restaurants, various types of business that had to reduce somehow the number of present people at once.

 Business continuity was the top priority, and risking one infected employee coming to work and spreading panic within the 1000+ other workers would cause panic, and a sudden stop of the activity. That would also cause panic amongst customers and providers, and no one was up to take the risk of it happening.

Companies were actually "pushed", yes pushed because it was not their first option in the first place to start a new "work from home" policy if it was not for the pandemic.  Announcement from governments came along to ask for their citizen to stay at home for the next couple of week.

The results of that were amusing : in a heartbeat, all possible and impossible resources were made available to all employees to do remote work. Again,  it was out of the question to allow any risk of stopping the activity.

VPN and video conference apps were bought and paid for, laptops were distributed and complimentary wide LCD screens were given to employees.

This was a precedent in the speed of technology adoption in modern economic history.

The point I am trying to make here is that this crisis has ( for one reason or the other) made a crack into the existing "stone-age" rules system.

Companies struggling to adopt technologies such as video conferencing were pushed to make a quick shift to adopt and use technologies that would benefit them during this strange time where business has to continue despite the distance constraints. Workers that had the habit of "resisting technological change" were obliged to use these tools.

Why some companies will not make it ?

There are various factors that are changing the future of the economy post-pandemic. 2 forces are in my opinion, the most impactful ones:  An major change in demand and the disruption of the workforce.

1.  Metamorphosis of demand

The sudden and very fast change in the demand trends has immediately impacted the top line results and made companies feel the immensity of the actual issues that is going to happen.

Consumers have drastically reduced their expenditure on certain products and services, while increasing the share of money spent on others.

No one understood why T.P was the go-to product of this pandemic

Restaurants, concert venues and other business are the typical example of "money not spent" by consumers. Delivery services and online shopping services are the opposite example of a booming business during the outbreak. However this is not the focus of my analysis.

What I'd like to focus on here is B2B demand.

It is this self-destructive corporate behavior that causes companies and economies to collapse and initiates a never ending domino effect on the economy ( remember what happened in 2008, right? )


By self destructive behavior I mean all the actions companies took to cut costs, arbitrarily, stop purchase orders, cancel service contracts and sometimes fire a whole bunch of employees, without thinking of the repercussions of such irrational behavior on the well-being of their companies in the mid and long terms.

I understand that for operational costs, there should be something to be changed when there are no operations ( AKA: no business, no money coming in ),  However it has touched costs related to long term investments that already have a budget set from 2019 and are sometimes an essential investment to insure service stability and quality.

Upgrading production ressources, enhancing existing process or hiring new talent are mandatory investments which have a great deal of impact on any company's performance down the road.


While rationalizing expenditure might seem like the go-to strategy, I would really like to say it with big huge letters:

 IT IS NOT A STRATEGY TO CUT COSTS !

Even the least experienced of people can think of this:

We need to safeguard our cash, well let's cancel that X order, and let's ask the other vendor to reduce its prices, and if they don't do that, we can cancel it as well.
This has prompted various companies to seek alternatives, either as vendors or as clients. These alternatives has always amounted to either spending a fraction of the provisioned budget, or not spending it at all.

In its essence, cost cutting is a double-edge sword, if done properly it can help struggling companies survive difficult times and manage to sustain operations while dealing with a declining business.

If done badly, it will backfire causing sometimes irreversible damage.


Layoffs especially, are a tough one to deal with. Going from a certain setup of the working-force and workload distribution, to facing new challenges of more work (with probably a salary reduction due to reduced working hours) wouldn't be much appreciated by the employees.

Effects of this type of actions would be reflected on employee performance. People would be less motivated, less efficient and  one hour of their work would be less valuable under these circumstances.

Without going very deep into this subject, I would refer you to this short write-up from 2006 ( yes issues happened before and it's not just the coronavirus . Idiots survive crisis, just like cockroaches survive radiations).

Idiotic examples of corporate cost cutting

Now let's move on.

 2. Disruption of the work force:

What we are observing currently is a drastic shift in skilled labor demand. Due to the sudden change of consumer behavior, some industries has moved their weight towards the digital space. This means that more talent is required with skills such as digital marketing, logistics and IT , just to name a few.

The existing pre-pandemic talent pool was already under high demand and more people have made the transition from other specialties to the digital world already.

With the boom in e-commerce, and the new / renewed preference for doing every possible (and reasonable) purchase from an online plateforme, companies are struggling to adopt more intuitive sales tunnels. This is/was not an easy task for most unprepared corporations, when we take into consideration the technical difficulties these unprepared companies are facing.

Shifting to online is not only about technology.. it is about the experience as well

Not finding the right talent would probably push companies to force this transition to an all-online business without the proper execution.

Throwing money at this problem wouldn't do it any good, considering the fact that online customer experience is highly influenced by how smooth the buying experience is.


As I thoroughly explained above, remote working has became somehow the norm now. Although some companies are going back on their decision to make WFH possible, employees are now taking notice of how important this flexibility to them, especially during uncertain times where traveling and free movement could be restricted at any time without a prior heads-up.

Nothing is going back to normal. Please understand this!

Talent is a very volatile "ressource"

If certain companies wont adapt to offer flexible work setups, others will. When a company X refuses to offer a certain amount of flexibility, the good guys will leave, taking with them a heart full of love for the idiocy that made them do a career change.

Losing talent to competitors is something regarded as "not an actual issue" by large corporations that don't value the humain intelligence and regard talent as a replaceable ressource, much like any other ressources they can buy with money.

However,  and in times of crisis, hiring and recruiting talent has proven to be a very challenging task when a company is facing other challenges with its sales and revenue. Most of the times, budget cuts start by freezing all open positions, putting ongoing hiring process on hold and for extreme cases, firing people recently hired, profiting from the trial period where they can fire new hires without repercussions .


This may be a strong factor in the prominent decline of certain corporations. And we all know (Except those idiots in certain places) that talent is NOT REPLACEABLE !

There are definitely more complex reasons and factors on this article' subject, however boring my readers is the last thing I want to do now.

Read More