Using reflections to see the world in a different way with the ORCa technique

A new computer vision system turns any shiny object into a kind of camera, allowing the observer to see around corners or obstacles. Just as a car is moving along a street and the reflection of light from parked vehicles or their side mirrors can help the driver see things that would otherwise be hidden from view, such as perhaps child drivers in the car. The sidewalk plays behind the parked cars, they can’t see.

Building on this vision, researchers at MIT and Rice University have developed a computer vision technique called the ORCa technique that uses reflections for imaging. Their method uses reflections to turn shiny objects into cameras, allowing the user to see the world as if looking through the lens of everyday objects, such as a ceramic mug.

According to this technique, based on the images of the object taken from different angles, only its surface becomes a virtual sensor that records the reflections; But the AI ​​system maps these reflections in such a way that they can estimate the depth of the scene and capture new views that are only visible from the perspective of the object. In fact, this technique can be used to see corners or objects that block the observer’s view.

Interestingly, this method is very useful for self-driving vehicles. For example, it allows a self-driving car to use the reflection of objects it passes on the side of the road to see around a parked truck. In fact, with this method any surface can be turned into a sensor, which turns objects into virtual pixels and sensors. This research will be presented at the Computer Vision and Pattern Recognition conference.

A closer look at reflections in the ORCa technique

The protagonists of crime TV shows often magnify and amplify surveillance footage to capture reflections; But in real life, using these reflections is not so easy, because according to Dave, reflections give us a distorted view of the world. In fact, the ORCa technique depends on two things: the shape of the object and the world the object is reflecting, both of which researchers may have incomplete information about.

This technique works in three steps:

• In the first stagethey photograph the same object from multiple viewpoints to have multiple reflections.

• In the second stageFor each image from a real camera, ORCa uses machine learning to transform the object’s surface into a virtual sensor that captures the light and reflections that hit each virtual pixel on the object’s surface.

• In the third stagethe system uses virtual pixels on the surface of the object to model the 3D environment.

Taking rays in the ORCa technique

Interestingly, imaging the same object from different angles enables ORCa to record multi-view reflections. By doing this, in addition to estimating the shape of the shiny object, the system also uses it to estimate the depth between the shiny object and other objects in the scene. In fact, ORCa can transform the scene into a five-dimensional radiation field and, by doing so, obtain more information about the intensity and direction of the light rays. This information helps ORCa to estimate the depth accurately so that the user can see its hidden features.

After ORCa captures this 5D field, the user can place a virtual camera anywhere in the scene and composite what the camera sees.

Detailed evaluations of the ORCa technique

The researchers evaluated the ORCa technique with other methods that model reflections. Their method worked well in separating the true color of an object from the reflections. They compared the system’s depth estimation with simulated data on the actual distance between objects in the scene and found that ORCa’s predictions were reliable, and therefore wanted to apply the technique to drone imaging so that ORCa could detect weak reflections. Use the objects that a drone flies over to recreate a scene from the ground. Also, with ORCa enhancement, shadows are used to reconstruct hidden information or combine reflections of two objects to image new parts of a scene.

It should be noted that this research was partially supported by the activities of advanced information research projects and the National Science Foundation.