[email protected] +36 1 385 1949

Facial Recognition Explained, Part 1: How It Works

Magyar változat

What is facial recognition, and how does it work?

Facial recognition has become an everyday concept: just think of unlocking your phone or the access gates at an airport. But what exactly does „facial recognition” mean? And how does a system get to the point of recognizing your face? In this post, we show how image- and video-based facial recognition works in practice.

Not all facial recognition is the same

First, it is worth clarifying that facial recognition is not a single method but a technology toolbox, with several tools and approaches.

How does facial recognition work?

  • Simple image-based facial recognition: works with a single frame, which can be a photo or a snapshot from a video. It is fast, but sensitive to quality and external conditions (for example, lighting or a turned-away face).
  • Multi-frame approach based on video: the system does not use a single frame but a short video clip, so it works with several face positions and angles. This makes the method less sensitive to conditions, and it also enables liveness detection, that is, checking whether we are dealing with a live person rather than a photograph.
  • 3D or lidar-based facial recognition: for example, the Face ID system of iPhones, which processes not only the camera image but also depth information. This technology is particularly secure, but hardware-intensive.

The process of image- and video-based facial recognition

Most commercial and industrial solutions today use camera-based facial recognition. It works in the following few steps:

  1. Face detection: the system first finds the faces in the image. There can be several, even in motion.

  2. Face cropping and alignment: the system isolates the given face, rotates it and scales it to obtain a standard format (for example, a frontal view).

  3. Face template extraction: from the prepared face, the system creates a so-called face template (in technical terms, an embedding). A face template is a set of numbers, a vector of features produced by the algorithm from the face. It is not interpretable to the human eye, but for the system it is the face’s unique „fingerprint”.

The face recognition pipeline: face detection, cropping and transformation, then conversion into a face template by a deep neural network
The facial recognition process: detecting, cropping and transforming the face, then converting it into a face template with a deep neural network. Face templates are data that software can already compare and classify. (source: https://cmusatyalab.github.io/openface/)

This face template contains the facial characteristics that the program considers important for identification to be possible. That is, templates from photos of the same person should be more similar to each other, while templates from different people should be more different.

Such a template might look like this, for example:

[0.12, -0.85, 0.46, …, 1.03]

This sequence can contain hundreds of numbers; typical lengths are 128, 256 or 512. It will be different for every face, even if the photos differ only slightly. This achieves the goal that we can compare these face templates with each other, which is the key step of identification.

In the next part, we examine how it can be used, and for what, for example in a business environment.

Would you like to use facial recognition in your service? Learn about our VeriDome service, and let us contact you!