My Style'Z web app [Style transfer in action]
Sometimes you start playing with an idea, time passes and you end up with a project that you need to finish.
I am sure this is not the case with most people, however this describes perfectly the process with which I've made the StyleZ app.
I have always been fascinated by the applications where neural networks are used, outside of the traditional context. One of the ideas that were proposed to utilize such technology is style transfer.
 |
| The Scream by Edvard Munch |
In this post, I will explain the steps I followed to implement and deploy a Neural Network Style Transfer App. The ideas is not to make a technical post, but to lay out the process to take a very small project from an idea to production
Neural Style Transfer
Neural style transfer is an optimization technique used to take two images, the first a content image such as the image of a person, a building or any other image or photo, and the second a style reference image (such as an artwork by a famous painter for example), and blend them together so the output image looks like the content image, but “styled or painted” in the same style of the style reference image.
This is implemented by optimizing the output image to match the content statistics of the content image and the style statistics of the style reference image.
These statistics are extracted from the images using a convolutional neural network.
In summary, the algorithm takes the content image and the style image that we want to match. it transforms the base input image by minimizing the content and style distances (losses) with back-propagation, creating an image that matches the content of the content image and the style of the style image.
I will not go into further detail, however here is a link to the article where the authors explain all the math details and the steps that goes into achieving this task of style transfer
 |
| The different styles applied to our Eminem Image |
Building the front-end
I don't see myself as a super-star professional graphic designer per se, however along the years, I had to learn to do graphics work for my NGO, build the website, make brochures and posters for events... etc
So I think I got that experience to thank for my way with graphical design.
For this app, I wasn't a complex job once you figure out the macro-idea, So I laid out the prerequisites for the web application. I wanted to have a page that allows the user to :
- upload a photo
- choose a style from a predefined styles list
- submit and see the result
Those were the basic needs to make the overall concept works, however I wanted to add more swagg in the app. So the user should also be able to:
- go back after seeing the result to choose an other style
- have some useful information about the artwork
- have links to the technical resources behind the style transfer
The trio : HTML, CSS and Javascript
I wanted the web app to have a soul. Being an admirer of harmony, I seemed like I would make sense for a style transfer app to have a certain "Style" for itself. So the main page was split into 3 parts:
Top section:
This is where the user will land at first. So it needed to be attractive and very easy to understand and use.
There are 3 component in this section:
- The first one is a slider to showcase the possible styles a user could apply to a photo.
- Just below that, the same concept but with the names of the styles with all of the available styles laid out in a row.
- And the final sub-section is the upload form. In this part of the top section, I tried to keep it simple, with a button to choose the photo from a local directory, a drop down menu for style selection, and finally a submit button.
Center section:
This is really an extra part. I was just set and done with keeping it simple with the top part of the page, which is enough to do the work.
However, I really wanted to share an appreciation for the art that inspired the Style Transfer algorithm.
So I have selected some of the famous paintings and artwork and linked to their Wikipedia pages or other pages that could provide sufficient information on the painting, the artist and the history behind it.
Bottom section:
This was also an optional part. However, if we are going to talk about the art, we need to also talk about the algorithm that made it happen.
So I linked to the Style Transfer original paper , and I linked to Google's Tensorflow tutorial about style transfer so the user (the curious ones) can see step by step the construction process.
Front-end of the results page:
This was also necessary to think about. The main thing here was based on 2 prerequisites:
- Keep the coherent style the same
- Provide a way for the user to try an other style
There was also a their point but I had to let it go. It was the possibility of making an automatic download button for the result image.
I had to et it go because it was a bit tricky to work on the Javascript, some browsers had limitations for allowing users to download something and I didn't have the time to dig deeper in that.
So the final results page has an upper section with both the original and the styled images, and a button to allow users to test an other style.
Responsive design!!
An important thing I need to mention here is the fact that nowadays, all websites are opened on mobile platforms. For certain cases, mobile penetration is way higher for particular websites.
People are always on the go and I can't assume that a future user should be on their laptop to use my app, so I had to make sure my front-end is compliant with this criteria.
Building the back-end
Since the whole project was about doing something fun and not too complex, I made the choice of keeping the complexity of the work limited to building the style transfer model.
That being said, the simplest backend choice that I could make was to build everything around Flask, and that is what I exactly dit.
For those of you unfamiliar with Flask, Flask is a micro web framework written in Python. It is classified as a micro-framework because it does not require particular tools or libraries. It has no database abstraction layer, form validation, or any other components where pre-existing third-party libraries provide common functions
Testing everything
There were few test case scenarios that I needed to keep in mind where the user, intentionally or not does one of the following things:
- upload a non-photo file (pdf , or zip for example)
- upload a very small image, a grey or black an white or an image with only one color
- adversarial attacks wanting to reveal the code, model or the backend
- other security risks
For the first two, testing wasn't very difficult to implement, and taking into account both scenarios, I was able to correct and enhance the overall performance with warning messages displayed to the user in case they missed and uploaded the wrong file type.
For the adversarial attacks and other security risks, since it is a free access app, the only thing I wanted to avoid is to have the app broken or taken off line, other than that most of the codes are available online and I don't really have to worry about people trying to "hack" it, so I did what I knew was possible to make sure the app and users are safe.
That's why the app does not store any original image or result image, as soon as the user refreshes the page, everything is gone.
It was also important for me to implement this feature of not conserving anything, because otherwise the hosting fees will cost me a fortune. So security for users is guaranteed and my pockets wont hurt as much.
Deploying
This wasn't a difficult thing to decide, since I had 2 conditions to respect:
- Finding a free hosting tier
- Easy to deploy without the hassle of too much configs
Amazon and Heroku were my to test options. The only concern I had an I needed to check was the hosting space. I have worked with both solutions in a professional / production setting, so most of the thinking about sizing, redundancy and availability was at a very different scale.
This app is supposed to be shared to my network and used probably intensively during the first week of me sharing the link and the blogpost publicly, and the usage is supposed to go down from there to a more stable level.
I shouldn't expect a 5000 hits/second ( unless something happens and a neural networks app goes viral for some reason)
Also, to make sure I wont get hit by a sudden crash due to high usage, I intended not to post any link on Hacker-news or any other public website.
So If I did the math right, based on my viewers ad interactions numbers on my blog, website landing page and my 2 social network accounts, Twitter and Linkedin, I calculated that in the event I choose to post on a "virality- prone" timing, on all of them, at the same time, I shouldn't be worried about going above the free tier limits of both platforms.
So considering all the above and other reasons, I decided to go with Heroku and did some testing from various locations around the world to check for latency and availability and all in all, everything was good to go.
To sum everything:
It was a nice fun project to do, outside of the do-stuff-for-work mindset. I took my time with it and had fun figuring out some of the parts I didn't have time before to work on deeply, especially javascript and CSS .
The front-end design was a pleasant part because it took me back to the days when I used to do graphic design for my NGO, I actually spent more time working on the front-end than what I spent doing my website landing page.
I wont say I didn't learn much or I learnt too much, It was fun and I was able to do something with an EMINEM photo on it.. So I'm happy, Slim Shady is happy and we all Good!
Next Steps?
Not sure what to do next, either add more styles inspired by other art works, add other features (ideas are yet to come) or probably think about an other thing to do ( while writing this blogpost, I already had an idea for the next weekend project, and I like it!)