The Hollycoaster
A Style Transfer Model
About The Model
The Hollycoaster is a style transfer system which converts old black and white movie posters into modern colorized posters and was made during the Computational Creativity module at the Queen Mary University of London. Given a content and style image the system transfers the style on the content image. The system allows the user to select black and white movie posters from a list of posters and results in a coloured and stylized output image.
The system implements two pretrained models to complete the stylization process. The first model is the Machine Vision Model (MVM), and the second model is that of Fast Style Transfer for Arbitrary Styles (FSTAS) model from the TF hub.
My main motivation behind this project was to be able to process the old movie posters into a new look such that fans who love the black and white cinema can see some of their favourite movies' posters in a new flavour mixed with colour and stylization. This appealed to me as a good idea to preserve the old with a flavour of new. This project is thus stylization with a purpose of bringing the joy and thrill of reliving the past.
Process of The Whole Implementation
The process for artefact generation is divided into the following steps:
1. User selects the names for the content and style image. Both the images are loaded and displayed to the user.
2. Internally, the system loads the VGG-19 CNN in the file.
3. For the model to process the images both the images are resized and pre-process by converting it into numpy array, adding an extra batch dimension and converting it to BGR. The data is then zero-centred and converted into tensor input.
4. The loss function and the loss and gradient function is calculated. The total loss for given images calculates the total loss for a given content, style and combination image based on content and style losses. While the loss and gradient function calculates the loss and gradient based on the loss function.
5. A user can adjust the weights of the content and style images to control the effects of each on the final output.
6. The Optimization and loss function is compiled in the model. The optimizer used is a Stochastic Gradient Descent(SGD) having learning rate, decay steps and decay rate.
7. The Optimization loop is run for a particular number of iterations; it calculates the loss and gradient and applies it to the image. After every 50 iterations, the stylized image is displayed for comparison. The system performs a de-processing step to convert the images back to RGB and add back the mean value of RGB. Thus, return an image as output for visualization.
8. For the second stylization stage, the user selects a style image from a dropdown. This style is then applied by the FSTAS model on the output obtained from the MVM model.
9. The stylized output from the FSTAS model and the MVM model are then stacked using the tf.stack() function and the Euclidean norm of the two images is calculated along an axis.
10. The final stylized image is obtained by adjusting the brightness, contrast, and saturation of the stylized image.
Outputs
These are some outputs obtained after running both the models over certain iterations and applying stacking.
Note: Due to the code's confidentiality agreement, I am unable to share it outside the university premises.
