Facial recognition systems can be disrupted by a specially designed pattern, as a live demonstration at the Defcon hacker convention has shown. Bill Swearingen, a cybersecurity professional and founder of the Kansas City security community SecKC, stood in front of a live camera feed while a person-detection system analysed him. On the screen behind him, the software’s confidence score cleared 0.75, the threshold required to declare that a human being was in the frame.
Swearingen then raised a flat panel covered in a bizarre black-and-white pattern. The score began to fall, eventually slipping below the threshold and landing at 0.21. The screen announced “No person detected” in bright green letters, even though he remained plainly visible to everyone in the room. The software was still receiving the camera image, but it no longer detected a person above the configured confidence threshold.
Swearingen has spent the past year searching for patterns that can confuse the computer-vision systems used to identify people. His project, called noRecognition, aims to create clothing that makes the wearer harder for AI surveillance systems to detect. It remains a work in progress.
The camera was not trying to identify him
Although this kind of technology is often called facial recognition, surveillance systems can involve several separate layers of AI-based detection. A person detector asks whether a human body is in the frame. A face detector finds and isolates a face. Facial recognition then compares that face with a database and asks whether it recognises the person. Each of these performs a different job, though each step can depend on the one before it.
The demonstration targeted the first step. The system did not mistake Swearingen for somebody else or decide the room was empty. It simply stopped reporting a person detection above the threshold set for the demonstration. If the first link is broken, the rest of the surveillance chain may never begin. Should a camera fail to detect a person, it may never crop out that person’s face, send it to an identity database, or track them across a series of images.
An AI detector does not “see” a person in the way humans do. It generates thousands of guesses about what might be in an image, assigns a score to each and discards anything that fails to clear a chosen confidence threshold. Swearingen was still there; the detector’s mathematics simply was not sure he was a person.
How the patterns are created
Swearingen did not draw the patterns by hand. He built a program to create them. Security researchers normally use tools called fuzzers to bombard software with strange inputs in an attempt to break it. Swearingen’s fuzzer does something similar with computer vision. It creates a pattern, then digitally places it on an image of a computer-generated person to test how effectively the pattern lowers the detection system’s confidence.
The technique demonstrated at Defcon lowered the AI camera’s confidence score for detecting a person from above 0.75 to 0.21.
