Researchers found a way to make pictures and videos talk

Recording audio from an image might only happen in science fiction, but a scientist has found a way to do it using artificial intelligence.

Professor Kevin Fu, a professor of electrical and computer engineering from Northeastern University, has succeeded in developing a machine learning tool called Side Eye that can make images speak.

By applying Side Eye to a still image, he and his colleagues were able to determine the gender of a speaker in the room where the photo was taken. They can also use this tool for silent videos.

“Imagine someone posted a video on Tik Tok that is completely silent,” Fu said. Are you curious to know what this video really says?

Side Eye also uses image stabilization technology found in most smartphone cameras. Smartphone cameras have springs that prevent it from shaking. These springs combine with sensors and an electromagnet to push the lens in the opposite direction of any shake to stabilize the image.

When a person speaks next to the camera lens while taking a picture, small vibrations occur in the springs and the light is subtly bent. Although it is somewhat impossible to extract audio frequencies from these vibrations, it is possible due to the type of shutter used by most cameras.

Side Eye can have positive uses and can be used as a type of digital evidence for crime investigations. Of course, if a more advanced version of it falls into the hands of criminals, it may become a cyber security.