Over the last couple of years we have witnessed an explosion of novel mobile applications advancing the developments in deep learning research; some of which include real-time translation, word prediction, object detection and classification.

There are various applications of this technology ranging from entertainment apps like Snapchat, that are simply recreational, to apps that can save lives by being able to recognise cancerous skin, such as SkinVision. These are only a few examples of how deep learning augments mobile phone capabilities.

Most deep-learning-based mobile apps employ a so-called cloud-based deep inference approach. In short, this means that the data is captured on mobile devices but then shared and processed on remote powerful servers. The process requires an internet connection, and sees the data being shifted from various places in order to be processed.
There are certainly many legitimate use cases when cloud-based approach works best. However, there are also many limitations of this approach such as latency issues, privacy concerns and dependence on an internet connection. These issues have resulted in the active field of research that aims to bring certain deep-learning tasks directly to our phones.

Modern smartphones have become powerful enough to run deep learning inference in a matter of milliseconds, which opens a whole new world of possibilities. This new approach is called device-based deep inference and it’s not only applicable to mobile phones but also to drones, autonomous cars, augmented reality headsets, IoT devices, etc. The technology is an exciting progression that could revolutionise the way our devices process data, by limiting the dependency on external elements.

Cortexica’s Martin Peniak has developed a prototype application, which he believes is a perfect demonstration of both approaches. The app is called Lens and it uses on-device deep inference to detect dozens of clothing categories such as t-shirts, shirts, cardigans, coats, jackets, hoodies, sweaters, trousers, shorts, shoes, hats, bags and many others. Users can then click on a detected item of interest and shop for visually similar products amongst millions thanks to our FindSimilar service running on the AWS cloud. The application of on-device deep inference makes the app much more responsive, with slicker user experience, as brand partners do not need to send every single frame to AWS for analysis. Cortexica’s model only uses AWS when a user decides to search for visually similar products, creating a much more efficient and accurate visual search process.

Check the video below that demonstrates this application and get in touch if you would like to know more!