Presentation about Transfer Learning

Last week I did a presenation at an internal conference at Bouvet, where I work, about how transfer learning in some cases could help achieve better performance for machine learning models faster and cheaper, using less data and processing power.

You can find my slides here (PDF, Norwegian).

Very simplified, transfer learning is about exploiting the fact that deep neural networks learn generic information in early layers, which is then built upon hierarchically in later layers that learn increasingly more specialized knowledge. This means that one can strip away the most specialized layers, keep the "general knowledge" from earlier layers, and use this to learn new specialized knowledge on previously unseen data - provided that the problem the original model was trained for and the new problem are somewhat similar.

As part of the presentation, I demoed how transfer learning could easily be applied using TensorFlow and a pretrained model (Inception v3), creating a classifier that could identify various humanoid species from the Star Trek universe.

The original pretrained model had learned to classify 1000 clases from ImageNet, from tens of thousands examples from each class (like "umbrella", "cheeta", "space shuttle", etc.) over a period of two weeks, running on a cluster 50 NVIDIA Kepler GPUs - achieving a "top-5 error rate" ("how often the model fails to predict the correct answer as one of their top 5 guesses") of 3.46%.

Using transfer learning, my Star Trek Humanoid classifier achieved an accuracy between about 79 and 95 percentage after about five and a half minutes on my laptop's CPU - without even spending any time tweaking hyperparameters.

The code I demonstrated in a notebook (plus a PDF of the presentation) is available here.


The presentation and notebook linked to in this post has been updated slightly, as I held an updated version of this presentation at a department meeting at Bouvet.

Newer post
Hypothesis- and Goal-Driven Development
Older post
A few thoughts about Electron