Skip to content

FAQ

Jump to bottom
Rob Royce edited this page Oct 24, 2024 · 8 revisions

General ROSA Questions

What are "Embodied Agents"?

Answer: An embodied agent is an Artificial Intelligence (AI) system that interacts with its environment through a physical or virtual body. They are able to perceive, act, and interact with their surroundings, and are often used in research and development for a variety of applications.

ROSA, in particular, is an agent that is embodied in physical or virtual (simulated) robotic systems. It can receive data from its operating environment, ROS/ROS2, or any other connected system to allow it to perceive its environment. ROSA uses Python functions (also called Tools) to perform actions within this environment in order to achieve the goals of its operators. All of this is done via natural language to simplify the act of developing and operating robotics.

What are the benefits of using AI Agents like ROSA?

Answer: AI agents like ROSA can help humans interact with robots more effectively by providing natural language understanding and generation capabilities. This can improve the user experience and make it easier for non-experts to interact with robots. Additionally, ROSA can be used to automate many of the tedious tasks and procedures needed to effectively develop, diagnose, debug, and troubleshoot complex robotic systems.

What's the major advantage of using ROSA over copy-pasting files into ChatGPT?

Answer: ROSA provides several advantages over traditional chat services like ChatGPT:

  1. Direct ROS Integration: ROSA can interact directly with the running ROS system, providing real-time information and control.
  2. Tool Selection and Parallel Execution: The agent can select from multiple tools, use them simultaneously, and even execute them in parallel.
  3. Reasoning-Action-Observation Loop: ROSA employs a loop that allows it to reason about a query, take actions, observe results, and continue using tools if needed.
  4. Custom Tools: You can integrate custom tools for tasks beyond information retrieval, such as robot control (e.g., the ROSA-Spot agent that can control a Spot robot).
  5. Safety Measures: ROSA includes built-in safety checks and constraints to mitigate risks of unwanted behavior.
  6. Context Awareness: ROSA maintains awareness of the specific robot and environment it's working with, thanks to custom prompts and system instructions.

How can I get started with ROSA?

Answer: To get started with ROSA:

  1. Ensure you have Python 3.9+ and ROS Noetic or higher installed.
  2. Install ROSA using pip: pip3 install jpl-rosa
  3. Set up your preferred LLM (e.g., OpenAI, Azure OpenAI, or a local model).
  4. Create a ROSA instance in your Python script:
from rosa import ROSA

llm = get_your_llm_here()
rosa = ROSA(ros_version=1, llm=llm)
rosa.invoke("Your query here")

Tip

For more detailed instructions, refer to the Developer Documentation and the Model Configuration wiki page.

Will robot-specific ROSA agents become available in the future?

Answer: Yes! We are working on creating robot-specific ROSA agents that are tailored to the capabilities and constraints of specific robots. In particular, we are focusing on agents for the robots listed below. Stay tuned for updates on these agents and their availability to the public.

What are ROSA's limitations?

Answer: ROSA has various limitations that can be grouped into the following categories:

  • Model Limitations:
    • ROSA may not work with all LLM models. See the Supported Models page for more information.
    • The chosen model may not understand certain ROS commands or may provide incorrect responses.
    • Each model has a context window limited to a certain number of tokens. If the context window is exceeded, the model may not provide accurate responses or may fail to respond at all.
  • Physical Capabilities:
    • The core ROSA agent is limited to text-based interactions and cannot interact with the physical world. To enable physical interactions, you will need to create custom tools and prompts.
  • Tool Limitations:
    • The tools available to ROSA may not cover all possible ROS commands.
    • The tools may not be able to handle all possible variations of ROS commands.
    • Some tools might require additional parameters or context to function correctly.

How can I adapt ROSA for my ROS-based robot?

Answer: To adapt ROSA for your robot:

  1. Create a custom agent by inheriting from the ROSA class or creating a new instance with custom parameters.
  2. Add custom tools specific to your robot's capabilities.
  3. Create custom prompts using the RobotSystemPrompts class to provide context about your robot.
  4. Implement safety checks and constraints for your robot's specific needs.

Tip

For detailed information, see the Custom Agents and Developer Documentation guides.

AI Model Questions

Which LLM models are supported for use with ROSA?

Answer: ROSA supports various LLM models, including:

  • OpenAI models (e.g., GPT-3.5, GPT-4)
  • Azure OpenAI models
  • Local models (e.g., using Ollama)

Tip

For a complete list and configuration details, see the Supported Models page.

Which model do you recommend using?

Answer: We recommend using GPT-4o due to its strong reasoning abilities and performance. However, the choice of model may depend on your specific needs, budget, and privacy requirements. For local deployment or increased privacy, consider using a local model through Ollama.

Safety and Guardrails

How can I ensure that ROSA does not perform unsafe actions?

Answer: To ensure ROSA's safety:

  1. Add safety checks to your custom tools to validate parameters and limit actions.
  2. Use the global retrieval filter (blacklist) to prevent access to sensitive topics, nodes, or services.
  3. Implement constraints in your tools to check the robot's state (e.g., battery level) before executing actions.
  4. Utilize the RobotSystemPrompts class to provide clear instructions and constraints for your robot.

Tip

See the Custom Agents guide for more information on implementing safety measures.

How can I prevent ROSA from using certain topics, nodes, services, logs, etc. that match a certain pattern?

Answer: ROSA implements a global retrieval filter in the form of a blacklist. To use this:

  1. Create a list of strings representing the patterns you want to block.
  2. Pass this list to the blacklist parameter when creating a new instance of the ROSA class:
blacklist = ["sensitive_topic", "private_node"]
rosa = ROSA(ros_version=2, llm=your_llm, blacklist=blacklist)

This blacklist is automatically passed to all tools registered with ROSA, preventing access to matching items.

How can I ensure that ROSA does not perform actions that are outside the robot's capabilities?

Answer: To prevent ROSA from exceeding the robot's capabilities:

  1. Implement constraints in your custom tools to check the robot's state and capabilities before executing actions.
  2. Use the RobotSystemPrompts class to provide clear information about the robot's capabilities and limitations.
  3. Add safety checks in your tools to validate parameters and prevent actions that could damage the robot.

Tip

Refer to the Custom Agents and Developer Documentation guides for detailed implementation strategies.

Development and Customization

How can I add custom tools to ROSA?

Answer: To add custom tools to ROSA:

  1. Create a new Python file for your custom tools.
  2. Define your tool functions using the @tool decorator from Langchain.
  3. When initializing ROSA, pass your custom tools or tool packages:
from rosa import ROSA
from your_custom_tools import custom_tool1, custom_tool2

rosa = ROSA(
    ros_version=2,
    llm=your_llm,
    tools=[custom_tool1, custom_tool2],
    tool_packages=['your_custom_tool_package']
)

Tip

For more details, see the Developer Documentation guide.

Can I use ROSA with both ROS 1 and ROS 2?

Answer: Yes, ROSA supports both ROS 1 and ROS 2. When initializing ROSA, specify the ROS version you're using:

# For ROS 1
rosa_ros1 = ROSA(ros_version=1, llm=your_llm)

# For ROS 2
rosa_ros2 = ROSA(ros_version=2, llm=your_llm)

Note

ROSA will automatically use the appropriate tools and commands for the specified ROS version.

How can I customize ROSA's behavior for my specific robot?

Answer: To customize ROSA for your robot:

  1. Create a RobotSystemPrompts instance with information about your robot:
from rosa import ROSA, RobotSystemPrompts

robot_prompts = RobotSystemPrompts(
    embodiment_and_persona="You are a TurtleBot3 robot...",
    about_your_capabilities="You can move forward, backward, and rotate...",
    constraints_and_guardrails="Never exceed a speed of 0.5 m/s..."
)

rosa = ROSA(ros_version=2, llm=your_llm, prompts=robot_prompts)
  1. Implement custom tools specific to your robot's capabilities.
  2. Add safety checks and constraints to your custom tools.

Tip

For more information, see the Custom Agents guide.

ROSA: Robot Operating System Agent

Copyright (c) 2024. Jet Propulsion Laboratory. All rights reserved.

Clone this wiki locally