The future is here, and it’s only getting more advanced. People behind AI technology are increasingly coming up with new ideas, and NVIDIA researchers are no different. NVIDIA’s new AI technology builds interactive environments from real-life video, crafting true-to-life environments with relative ease.
At this year’s NeurIPS AI conference in Montreal, NVIDIA will demonstrate the new technology by showcasing footage of a virtual city using elements from real-world video, which you can watch below. The new way this AI captures real video could help developers create more realistic environments in video games. It would also make things significantly easier for game developers: instead of spending large amounts of time crafting objects with intricate detail, they could rely on this new AI technology that will do the job for them.
“Neural networks — specifically generative models — will change how graphics are created,” said Bryan Catanzano, NVIDIA’s president of applied deep learning, in a statement. “This will enable developers, particularly in gaming and automotive, to create scenes at a fraction of the traditional cost.“
In order to create the video demonstrated at the conference, researchers trained their neural model by using dashcam videos taken from self-driving cars, driving around for a week with one of the company’s DGX-1 supercomputers in tow. They also added in thousands of videos from Cityscapes and Apolloscapes datasets into the AI.
The research team then used the Unreal 4 engine to create a semantic map of the footage they obtained. From here, pixels taken from the photos were assigned a label, which the AI network would then use to create a car from what it was taught a car would look like.
Researchers used another example of what the technology could do by—what else—digitally creating a video of one of their team members doing the dance found in the Gangnam Style music video. Here, the AI captured footage from a dancer. It then managed to remember the way the dancer moved, turning them into a rudimentary stick figure. It then rendered the team member’s appearance on top of the stick figure.
Of course, this kind of technology could always be used for nefarious means, such as deepfakes or other methods of misinformation. But speaking to Engadget, Catanzaro is looking at the more positive aspects of what technology like this could bring.
“People really enjoy virtual experiences,” he said. “Most of the time they’re used for good things. We’re focused on the good applications.”
