NVIDIA Develops Advanced Vision-Language Model for Robotics, Demonstrates in Real-World Application

In a significant technological advancement, NVIDIA has announced the development of Vision-Language Models (VLMs) that merge the strengths of large language models with the vision capabilities of vision transformers (ViTs). The VLMs project both text and images into the same embedding space, allowing them to effectively handle and reason over unstructured multimodal data and return outputs in a structured way.

Building on its extensive base of pre-trained models, NVIDIA asserts these VLMs can be easily adapted for various vision-related tasks by introducing new prompts or performing parameter-efficient fine-tuning. They can also be integrated with live data sources and tools, enabling them to request more information or take action based on the data they process. This capacity allows VLMs and large language models (LLMs) to function as agents, aiding robots in performing complex and often hard-to-define tasks.

Expanding on this technology, NVIDIA has showcased the use of VLMs and LLMs on NVIDIA Jetson Orin devices in a previous demonstration, highlighting capabilities such as zero-shot object detection, video captioning, and text generation on edge devices.

The latest development, known as ReMEmbR (Retrieval-augmented Memory for Embodied Robots), integrates these advancements and retrieval-augmented generation (RAG) to allow robots to understand and act on what they see during extended deployments over hours or days. ReMEmbR is fully open-source and runs on-device, addressing major challenges in using LLMs and VLMs in robotic applications, such as handling large contexts, reasoning over spatial memory, and building prompt-based agents to query more data until users’ questions are thoroughly answered.

ReMEmbR encompasses two main phases: memory-building and querying. The memory-building phase uses VLMs and vector databases to create a long-horizon semantic memory. The querying phase utilises an LLM agent to reason over the stored memory. Videos captured during deployment are captioned using the NVIDIA VILA captioning VLM and embedded into a MilvusDB vector database, along with timestamps and coordinate information, allowing efficient storage and querying of information.

On a real robot, ReMEmbR leverages tools such as NVIDIA Isaac ROS and Nova Carter, showing practical applications like guiding and answering questions in an office environment. The process involves building an occupancy grid map, running the memory-building protocol, activating the ReMEmbR agent, and integrating speech recognition for user queries.

Build steps detailed by NVIDIA include:

  1. Building an occupancy grid map: Creating a map of the environment to establish global location (pose) information.
  2. Running the memory builder: Populating the vector database with captions and pose information from the robot’s memory.
  3. Running the ReMEmbR agent: Utilizing the populated database to answer user queries with generated goal poses for the robot’s actions.
  4. Adding speech recognition: Incorporating WhisperTRT project for low-latency inference on NVIDIA Jetson Orin platforms, enabling intuitive voice commands.

NVIDIA’s demonstration featured the robot performing tasks like navigating to the nearest elevator or a snack location based on user commands, showcasing the effective application of ReMEmbR in a realistic scenario.

This project stands as a significant development in robotics, laying the groundwork for future advancements and practical deployments of autonomous systems enabled by AI.

Professional Backgrounds:

  • Abrar Anwar is a Ph.D. student at the University of Southern California and an intern at NVIDIA, focusing on the intersection of language and robotics.
  • John Welsh is a developer technology engineer at NVIDIA, with a background in electrical engineering concentrated on robotics and computer vision.
  • Yan Chang is a principal engineer at NVIDIA, leading the robotics mobility team with extensive experience in autonomous vehicle behavior models.

Upcoming Events: Further insights into NVIDIA’s advancements in robotics will be presented at RoboBusiness 2024, taking place on 16-17 October in Santa Clara, California. Key sessions will feature Amit Goel and Sandra Skaff from NVIDIA, who will discuss the future of robotics innovation and the impact of generative AI on robotics.

Source: Noah Wire Services