Holodeck: Star Trek technology becomes reality

Inspired by the holodeck from the Star Trek universe, researchers have used artificial intelligence to develop a tool that can create 3D environments.

A team of scientists at the University of Pennsylvania, along with collaborating researchers at Stanford University, the University of Washington, and the Allen Institute for Artificial Intelligence (AI2), have recreated a Star Trek holodeck using ChatGPT and video game assets. The system, called Holodeck, creates interactive 3D virtual environments.

In the science fiction series Star Trek: The Next Generation, crew members can use interactive holodeck technology to transport themselves to locations of their choice, such as alien landscapes or historical sites. This fictional holodeck scenario is a system for creating interactive 3D environments. Scientists have now been able to recreate it in real life. To recreate its equivalent, they used advanced artificial intelligence such as ChatGPT and a database of video game assets.

Powerful language models (LLMs) like ChatGPT interpret user requests and translate them into detailed virtual environments. The process behind the Holodeck's operation generates an almost unlimited variety of interior spaces, using artificial intelligence to interpret the user's desires. Similar to Star Trek, users describe their desired environments using voice input.

For example, a user might request a cozy apartment with a cat. It then has the Holodeck create the floor plan, furnishings, and other items based on the description provided. To achieve this, the holodeck fills the virtual space with appropriate objects. To do this, it uses Objaverse, a massive dataset containing more than 800,000 annotated 3D objects.

Holodeck leverages the knowledge embedded in large language models (LLMs), the systems underpinning ChatGPT and other chatbots. Essentially, Holodeck works by engaging LLM in a conversation, using a carefully structured series of queries to break down user requests into specific parameters.

Robots receive worlds training

Yue Yang, a graduate student and computer and information science student at the University of Pennsylvania, points out the practical use of the holodeck. Become accordingly “Virtual interactive environments are also used to train robots before real use in a process called “Sim2Real.”. It seems that until now there have been virtual interactive environments “Surprisingly in short supply”Yang said.

“This lack of virtual environments is a problem if you want to teach robots how to navigate the real world in all its complexity. Neural networks, the systems driving today's AI revolution, require massive amounts of data, in this case a simulation of the physical world.“

Chris Callison Burch, associate professor of computer and information science at the University of Pennsylvania, adds:

“Generative AI systems like ChatGPT are trained on trillions of words, and image generators like Midjourney and DALLE are trained on billions of images. We only have a small fraction of that amount of 3D environments to train so-called ‘embodied AI.’ If we want to use technologies Generative AI To develop robots that can safely navigate real environments, we need to create millions or billions of simulated environments.

Mark Yatskar of the University of Pennsylvania knows:

“The ultimate test of the holodeck is to help robots interact more safely in environments they have never seen before.”

Training AI agents demonstrates the high potential of Holodeck

Finally, the researchers used the generated scenes to create the embodied AI agent “fine tune”. One task involved training these agents to find a piano in the music room. First, the scientists trained the test subject using ProcTHOR, a framework for procedural generation of embodied AI environments.

The agent was only able to complete the task in about 6% of cases. With 100 music rooms created by Holodeck, they have already succeeded in more than 30% of the cases.

The team will present the holodeck at the Institute of Electrical and Electronics Engineers (IEEE) and Computer Vision Foundation (CVF) 2024 Computer Vision and Pattern Recognition (CVPR) conference in Seattle, Washington.