A new model for planning and controlling humanoid movements in 3D environments

A new model for planning and controlling humanoid movements in 3D environments

Path planning and motion control PlaMo simulates a human in a complex 3D scene. The input consists of a physically simulated 3D scene and a set of textual instructions that define high-level navigation cues and the type of motion (“Crouching from tree to swing”). The scene can include a variety of terrain (gravel, stairs), 3D obstacles, and dynamic obstacles (here, a green ball). The output is a set of motor actuations that control a human character. PlaMo produces a planned path with textual guidance (the type of motion) and a head height and speed profile that matches the constraints of the motion controller. Source: Hallak et al.

Humanoids, robotic or virtual systems with body structures resembling the human body, have a wide range of applications in the real world. Because their limbs and bodies mirror those of humans, they can mimic a wide range of human movements, such as walking, crouching, jumping, swimming, etc.

Computational generation of realistic motions for virtual humanoid characters could have interesting implications for the development of video games, animated films, virtual reality (VR) experiences, and other media content. However, the environments depicted in video games and animations are often highly dynamic and complex, which can make motion planning more challenging for humanoids introduced into these environments.

Researchers at NVIDIA Research in Israel recently introduced PlaMo (Plan and Move), a new computational approach for planning the movements of humanoids in complex, 3D, physically simulated worlds. Their approach, presented in a paper published Open arXiv It consists of a preprint server, a scene-aware path planner, and a strong control policy.

“Controlling humanoids in complex physically simulated worlds is a long-standing challenge that has numerous applications in gaming, simulation, and visual content creation,” Assaf Hallak, Gal Dalal, and colleagues wrote in their paper.

“In our setup, given a rich and complex 3D scene, the user provides a list of instructions consisting of target locations and types of movements. To solve this task, we present PlaMo, a scene-aware path planner and a robust physics-based controller.”

Most previous work aimed at planning the movements of humanoid characters in simulated 3D environments has focused on developing either a planner or a controller, rather than both. Because the tasks these two types of models perform—that is, planning and executing the movements of a humanoid—are interconnected, Hallak, Dalal, and their colleagues set out to design a computational approach that addresses both.

“The path planner generates a set of motion paths by considering various constraints that the scene imposes on the motion, such as position, height, and speed,” Hallak, Dalal, and their colleagues wrote. “Our control policy, which complements the planner, produces rich and realistic physical motion while adhering to the plan.”

The researchers tested PlaMo in a series of simulations, using it to plan and execute the movements of the SMPL humanoid, a humanoid virtual agent with a neutral body structure (i.e., no facial features, hair, clothing, etc.). They ran their simulations in IsaacGym, a physics-based simulation environment for reinforcement learning developed at NVIDIA.

The results of these tests were quite promising, as the PlaMo approach was found to effectively plan and execute the movements of SMPL humanoids in complex simulated landscapes by following textual instructions. In particular, the planner component of the approach was found to account for the humanoid character’s movements on uneven terrain and static and dynamic obstacles in the environment.

The motion controller was shown to reliably follow the paths planned by the planner, executing complex and scene-specific movements for the humanoid creature, such as passing under low ceilings or moving quickly to avoid an oncoming obstacle.

Overall the combination of the two modules was quite effective and produced realistic movements in response to changes in the environment.

“PlaMo focuses on planning and control,” Hallak, Dalal, and their colleagues wrote. “It creates opportunities to combine this with modern language models and 3D scene understanding. With these expansions in mind, we see PlaMo as a stepping stone to a much larger system where non-playable characters (NPCs) are assigned roles and create a rich simulated virtual world.”

More information:
Assaf Hallak et al., PlaMo: Plan and Move in Rich 3D Physical Environments, arXiv (2024). DOI: 10.48550/arxiv.2406.18237

Journal information:

© 2024 Science X Network

Quotation: A new model for planning and controlling humanoid movements in 3D environments (2024, July 10) Retrieved July 11, 2024, from

This document is subject to copyright. No part may be reproduced without written permission, except for any fair use for private study or research. The content is provided for informational purposes only.