From the Series
The global phenomenon that is Pokémon Go gained a massive following because of its ability to blend virtual content and the real world through augmented reality. It provides a sense that the game’s characters exist in the same space as the player. As real as it may seem, these virtual characters can’t touch, move or change things in the real world. Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) aim to change that by developing a video format that allows augmented characters like Pokémon Go’s Pikachu to appear to affect what's happening in the real world.
Created by PhD student Abe Davis, interactive dynamic video (IDV) uses traditional cameras and algorithms to capture tiny vibrations recorded on video. They then calculate the different frequencies with which an object can move in real life. For instance, the way a bush might sway in the wind or the way a woman’s hair might change when touched. The technique provides a cheaper way to move objects in a realistic way in video.
"This technique lets us capture the physical behaviour of objects, which gives us a way to play with them in virtual space,” says Davis. “By making videos interactive, we can predict how objects will respond to unknown forces and explore new ways to engage with videos.”
Davis believes IDV has applications in various sectors. The tool could be used in engineering, entertainment, and more. In movies, the tool could be used to replace expensive CGI effects, making virtual characters realistically interact with their real-world environments. Engineers could also use the system to simulate how an old building or bridge would respond to strong winds or an earthquake.
“The ability to put real-world objects into virtual models is valuable for not just the obvious entertainment applications, but also for being able to test the stress in a safe virtual environment, in a way that doesn’t harm the real-world counterpart,” says Davis.
He applied the tool to the popular game to demonstrate how the technology could change how we currently experience gaming. According to MIT News, Davis says that he is also eager to see other applications emerge, from studying sports film to creating new forms of virtual reality.
