Researchers at the University of Pennsylvania have developed a real-life version of the holodeck from “Star Trek: The Next Generation.” This virtual reality room can generate various environments using voice commands, resembling the futuristic concept seen on the show.
Holodeck creates tailored 3D virtual word
Penn researchers have developed a new system called “Holodeck,” inspired by Star Trek, that employs AI to interpret basic language and create lifelike 3D virtual worlds according to user specifications. Similar to Captain Picard’s requests on the show, users can ask for environments like a 1940s detective’s office or a 1-bedroom apartment with a cat. Holodeck swiftly generates detailed settings, including furniture and realistic details such as a cat tower.
Yue Yang, a doctoral student involved in developing Holodeck, explains that language can be utilized to command the system. Users can describe desired environments and train AI agents effectively, according to a university announcement.
The practice of preparing robots in virtual environments before deploying them in reality, termed “Sim2Real,” has traditionally faced challenges due to the time-consuming nature of creating these virtual training arenas. Artists personally craft these settings, sometimes dedicating a week to constructing just one.
Researchers have developed Holodeck, which enables the rapid creation of millions of unique virtual spaces for training robots across various scenarios at a significantly reduced time and cost. This facilitates the ingestion of massive datasets by the robots’ neural networks, crucial for their true intelligence development.
Holodeck leverages LLMs to create virtual reality environment
Chris Callison-Burch, an associate professor at Penn, highlights the need for extensive simulated environments, similar to the vast datasets used in generative AI systems like Midjourney, ChatGPT, and DALLE, to train robots effectively for real-world navigation.
Holodeck creates virtual worlds by leveraging the extensive knowledge of large language models (LLMs), the same technology behind conversational assistants like ChatGPT. Through a dialogue with the LLM, Holodeck parses user text descriptions into queries about objects, colors, layouts, and other parameters. It then searches a vast library of 3D objects and employs algorithms to arrange them appropriately, ensuring realistic spatial design.
