Isaac Sim: Photorealistic Simulation and Synthetic Data Generation

Introduction to Isaac Sim

Isaac Sim is NVIDIA's high-fidelity simulation environment built on the Omniverse platform. It provides photorealistic rendering, accurate physics simulation, and powerful tools for generating synthetic data to train AI models. Isaac Sim bridges the gap between simulation and reality, enabling developers to create, test, and validate robotics applications in virtual environments that closely resemble real-world conditions.

Key Features of Isaac Sim

Photorealistic Rendering

• NVIDIA RTX real-time ray tracing
• Physically-based materials and lighting
• High-quality visual effects and post-processing
• Support for HDRP (High Definition Render Pipeline)

Physics Simulation

• PhysX 4.1 physics engine
• Accurate collision detection and response
• Realistic material properties and friction
• Multi-body dynamics simulation

Synthetic Data Generation

• Domain randomization capabilities
• Automatic annotation of training data
• Support for multiple sensor types
• Scalable data generation pipelines

Omniverse Integration

• Collaborative simulation environment
• USD (Universal Scene Description) format support
• Real-time collaboration capabilities
• Extensible through Omniverse Kit

Installing Isaac Sim

System Requirements

• NVIDIA GPU with Turing architecture or newer (RTX series recommended)
• CUDA 11.8 or newer
• 16GB+ RAM recommended
• Windows 10/11 or Ubuntu 20.04/22.04

Installation Process

1. Install NVIDIA Omniverse Launcher
2. Install Isaac Sim from the Omniverse app catalog
3. Install Isaac ROS bridge packages:

sudo apt update
sudo apt install ros-humble-isaac-ros-*  # All Isaac ROS packages

Setting Up Your First Isaac Sim Environment

Basic Scene Structure

Isaac Sim scenes follow the Universal Scene Description (USD) format:

/World
  /Robots
    /MyRobot
  /Environments
    /Office
    /Warehouse
  /Lights
  /Cameras

Creating a Simple Scene

Here's a basic USD stage setup:

import omni
from omni.isaac.core import World
from omni.isaac.core.utils.stage import add_reference_to_stage
from omni.isaac.core.utils.nucleus import get_assets_root_path
from omni.isaac.core.utils.prims import get_prim_at_path

# Initialize the world
world = World(stage_units_in_meters=1.0)

# Add a simple robot
assets_root_path = get_assets_root_path()
franka_asset_path = assets_root_path + "/Isaac/Robots/Franka/franka_instanceable.usd"
add_reference_to_stage(usd_path=franka_asset_path, prim_path="/World/Franka")

# Add ground plane
world.scene.add_default_ground_plane()

Photorealistic Environment Creation

Material System

Isaac Sim uses Physically-Based Rendering (PBR) materials:

from omni.isaac.core.materials import PreviewSurface
from pxr import Gf, UsdShade

# Create a custom material
def create_material(prim_path, color):
    material = PreviewSurface(
        prim_path=prim_path,
        color=color,
        metallic=0.0,
        roughness=0.5
    )
    return material

# Apply materials to objects
red_material = create_material("/World/Materials/Red", Gf.Vec3f(1.0, 0.0, 0.0))

Lighting Setup

Proper lighting is crucial for photorealistic rendering:

from omni.isaac.core.utils.prims import create_prim
from omni.isaac.core.utils.stage import get_current_stage

# Create dome light (environment light)
create_prim(
    prim_path="/World/DomeLight",
    prim_type="DomeLight",
    attributes={"color": (0.2, 0.2, 0.2), "intensity": 500}
)

# Create key light
create_prim(
    prim_path="/World/KeyLight",
    prim_type="DistantLight",
    attributes={"color": (1.0, 1.0, 1.0), "intensity": 1000},
    position=(5, 5, 5),
    orientation=(0, 0, 0, 1)
)

Camera Systems

Isaac Sim supports various camera types for synthetic data generation:

from omni.isaac.sensor import Camera
import numpy as np

# Create an RGB camera
camera = Camera(
    prim_path="/World/Camera",
    position=np.array([1.0, 1.0, 1.0]),
    orientation=np.array([0.0, 0.0, 0.0, 1.0])
)

# Set camera properties
camera.post_process_lights_enabled = True
camera.focal_length = 24.0
camera.focus_distance = 100.0
camera.f_stop = 0.0

# Enable different sensor types
camera.add_raw_sensor_data_to_frame("rgb")
camera.add_raw_sensor_data_to_frame("depth")
camera.add_raw_sensor_data_to_frame("semantic_segmentation")

Synthetic Data Generation

Domain Randomization

Domain randomization helps create robust AI models by varying environmental conditions:

import random
from pxr import Gf

class DomainRandomizer:
    def __init__(self):
        self.light_range = (300, 1500)
        self.color_range = (0.1, 1.0)

    def randomize_lighting(self, light_prim):
        """Randomize lighting conditions"""
        intensity = random.uniform(*self.light_range)
        color = Gf.Vec3f(
            random.uniform(0.8, 1.2),  # R
            random.uniform(0.8, 1.2),  # G
            random.uniform(0.8, 1.2)   # B
        )

        # Apply randomization
        light_prim.GetAttribute("intensity").Set(intensity)
        light_prim.GetAttribute("color").Set(color)

    def randomize_materials(self, material_prims):
        """Randomize material properties"""
        for material in material_prims:
            # Randomize color
            color = Gf.Vec3f(
                random.uniform(0.0, 1.0),
                random.uniform(0.0, 1.0),
                random.uniform(0.0, 1.0)
            )
            material.color = color

            # Randomize roughness
            roughness = random.uniform(0.1, 0.9)
            material.roughness = roughness

# Usage
randomizer = DomainRandomizer()

Data Annotation Pipeline

Isaac Sim can automatically generate annotations for training data:

import omni.replicator.core as rep

with rep.new_layer():
    # Define randomization graph
    keys_and_values = rep.distribution.uniform((0, 0, 0), (1, 1, 1))
    rep.randomizer.get_global_randomization_graph().add_node(
        "RandomizeLightColor",
        inputs={"keys": keys_and_values}
    )

    # Annotators for synthetic data
    rgb_annotator = rep.AnnotatorRegistry.get_annotator("rgb")
    depth_annotator = rep.AnnotatorRegistry.get_annotator("distance_to_camera")
    seg_annotator = rep.AnnotatorRegistry.get_annotator("instance_segmentation")

    # Enable annotators
    rgb_annotator.attach([camera])
    depth_annotator.attach([camera])
    seg_annotator.attach([camera])

Synthetic Dataset Generation

Creating large datasets for AI training:

import asyncio
from omni.isaac.synthetic_utils import SyntheticDataHelper

class SyntheticDatasetGenerator:
    def __init__(self, output_dir):
        self.output_dir = output_dir
        self.frame_count = 0

    def generate_frames(self, num_frames=1000):
        """Generate synthetic frames with annotations"""
        for i in range(num_frames):
            # Randomize environment
            self.randomize_environment()

            # Step simulation
            world.step(render=True)

            # Capture data
            self.capture_frame_data()

            # Save with annotations
            self.save_annotated_data()

            self.frame_count += 1

            if self.frame_count % 100 == 0:
                print(f"Generated {self.frame_count} frames")

    def capture_frame_data(self):
        """Capture various sensor data"""
        # RGB image
        rgb_data = camera.get_rgb()

        # Depth image
        depth_data = camera.get_depth()

        # Semantic segmentation
        seg_data = camera.get_semantic_segmentation()

        # Store data
        self.current_frame_data = {
            'rgb': rgb_data,
            'depth': depth_data,
            'segmentation': seg_data
        }

    def save_annotated_data(self):
        """Save data with COCO-style annotations"""
        import json

        annotation = {
            "image_id": self.frame_count,
            "rgb_path": f"rgb/frame_{self.frame_count:06d}.png",
            "depth_path": f"depth/frame_{self.frame_count:06d}.png",
            "seg_path": f"seg/frame_{self.frame_count:06d}.png",
            "annotations": self.generate_coco_annotations()
        }

        # Save image data
        self.save_images()

        # Save annotation
        with open(f"{self.output_dir}/annotations.json", "a") as f:
            json.dump(annotation, f)
            f.write("\n")

# Usage
generator = SyntheticDatasetGenerator("/path/to/dataset")
generator.generate_frames(10000)  # Generate 10k frames

Isaac Sim Extensions

Custom Extensions

Creating custom functionality for Isaac Sim:

import omni.ext
import omni.kit.ui
from omni.isaac.core import World
from pxr import Sdf, UsdGeom

class CustomRobotExtension(omni.ext.IExt):
    def on_startup(self, ext_id):
        print("[my.robot.extension] Robot extension startup")

        # Create menu entry
        self._window = omni.kit.ui.get_test_window()
        self._menu = self._window.menu.add_item("My Robot Tools", self._menu_callback)

    def _menu_callback(self, menu, menu_idx):
        """Callback for menu item"""
        # Add custom robot to scene
        self.add_custom_robot()

    def add_custom_robot(self):
        """Add a custom robot to the scene"""
        stage = omni.usd.get_context().get_stage()

        # Create robot prim
        robot_prim = Sdf.PrimSpec(
            stage.GetPseudoRoot(),
            "CustomRobot",
            Sdf.SpecifierDef,
            "Xform"
        )

        # Add robot geometry
        cube_geom = Sdf.PrimSpec(
            robot_prim,
            "Base",
            Sdf.SpecifierDef,
            "Cube"
        )

        # Set properties
        cube_geom.specifier = Sdf.SpecifierDef
        cube_geom.typeName = "Cube"

    def on_shutdown(self):
        print("[my.robot.extension] Robot extension shutdown")

Isaac Sim Python API

Using the Isaac Sim Python API for automation:

from omni.isaac.core import World
from omni.isaac.core.utils.nucleus import get_assets_root_path
from omni.isaac.core.utils.stage import add_reference_to_stage
from omni.isaac.core.robots import Robot
import numpy as np

# Initialize world
world = World(stage_units_in_meters=1.0)

# Add robot
assets_root_path = get_assets_root_path()
robot_path = assets_root_path + "/Isaac/Robots/Turtlebot/turtlebot3_carter.usd"
add_reference_to_stage(robot_path, "/World/Robot")

# Get robot reference
robot = world.scene.get_object("Robot")

# Control robot
def move_robot():
    # Get current position
    current_position, current_orientation = robot.get_world_pose()

    # Define target position
    target_position = np.array([2.0, 0.0, 0.0])

    # Move robot
    robot.set_world_pose(position=target_position)

    # Step simulation
    world.step(render=True)

# Run simulation
for i in range(1000):
    move_robot()
    world.step(render=True)

Performance Optimization

Level of Detail (LOD)

Optimize complex scenes for performance:

class SceneOptimizer:
    def __init__(self):
        self.lod_distances = [10, 20, 50]  # meters
        self.lod_prims = {}

    def setup_lod(self, prim_path, lod_prims):
        """Setup LOD for a prim"""
        self.lod_prims[prim_path] = lod_prims

        # Initially show highest detail
        for i, lod_prim in enumerate(lod_prims):
            if i == 0:
                lod_prim.SetActive(True)
            else:
                lod_prim.SetActive(False)

    def update_lod(self, camera_position):
        """Update LOD based on camera distance"""
        for prim_path, lod_prims in self.lod_prims.items():
            # Calculate distance
            prim_position = self.get_prim_position(prim_path)
            distance = np.linalg.norm(camera_position - prim_position)

            # Determine appropriate LOD
            lod_level = 0
            for i, lod_distance in enumerate(self.lod_distances):
                if distance > lod_distance:
                    lod_level = i + 1
                else:
                    break

            # Activate appropriate LOD
            for i, lod_prim in enumerate(lod_prims):
                lod_prim.SetActive(i == lod_level)

Multi-GPU Support

Utilize multiple GPUs for larger simulations:

import torch

class MultiGPUManager:
    def __init__(self):
        self.num_gpus = torch.cuda.device_count()
        self.gpus = [torch.cuda.device(i) for i in range(self.num_gpus)]

    def distribute_simulation(self, scene_parts):
        """Distribute simulation across multiple GPUs"""
        if self.num_gpus > 1:
            # Assign different parts of the scene to different GPUs
            parts_per_gpu = len(scene_parts) // self.num_gpus

            for gpu_idx, gpu in enumerate(self.gpus):
                start_idx = gpu_idx * parts_per_gpu
                end_idx = start_idx + parts_per_gpu if gpu_idx < self.num_gpus - 1 else len(scene_parts)

                # Assign scene parts to GPU
                self.assign_to_gpu(scene_parts[start_idx:end_idx], gpu)

    def assign_to_gpu(self, scene_parts, gpu):
        """Assign scene parts to specific GPU"""
        with gpu:
            # Initialize parts on this GPU
            for part in scene_parts:
                part.to(gpu)

Integration with ROS/ROS 2

Isaac ROS Bridge

Connecting Isaac Sim to ROS 2:

from omni.isaac.ros_bridge.scripts import ros_bridge_helper
import rclpy
from sensor_msgs.msg import Image, CameraInfo
from geometry_msgs.msg import Twist

class IsaacROSIntegration:
    def __init__(self):
        # Initialize ROS
        rclpy.init()
        self.node = rclpy.create_node('isaac_sim_ros_bridge')

        # Publishers
        self.rgb_pub = self.node.create_publisher(Image, '/camera/rgb/image_raw', 10)
        self.depth_pub = self.node.create_publisher(Image, '/camera/depth/image_raw', 10)
        self.cmd_vel_sub = self.node.create_subscription(Twist, '/cmd_vel', self.cmd_vel_callback, 10)

        # Camera reference
        self.camera = None

    def publish_camera_data(self):
        """Publish camera data to ROS"""
        if self.camera:
            # Get image data
            rgb_image = self.camera.get_rgb()
            depth_image = self.camera.get_depth()

            # Convert to ROS messages
            rgb_msg = self.convert_to_ros_image(rgb_image, 'rgb8')
            depth_msg = self.convert_to_ros_image(depth_image, '32FC1')

            # Publish
            self.rgb_pub.publish(rgb_msg)
            self.depth_pub.publish(depth_msg)

    def cmd_vel_callback(self, msg):
        """Handle velocity commands from ROS"""
        # Convert ROS Twist to robot control
        linear_x = msg.linear.x
        angular_z = msg.angular.z

        # Apply to robot in simulation
        self.apply_robot_control(linear_x, angular_z)

    def convert_to_ros_image(self, image_data, encoding):
        """Convert Isaac image to ROS Image message"""
        # Implementation depends on Isaac Sim version
        pass

Best Practices for Isaac Sim

1. Start Simple: Begin with basic scenes and gradually add complexity
2. Use Appropriate Detail: Balance visual quality with performance
3. Leverage Domain Randomization: Vary conditions to improve model robustness
4. Validate Against Reality: Compare simulation results with real-world data
5. Optimize for Training: Focus on data quality and diversity for AI training
6. Monitor Performance: Keep frame rates high for interactive development

Exercise: Isaac Sim Challenge

Create an Isaac Sim environment that:

1. Implements a photorealistic warehouse scene
2. Includes domain randomization for lighting and materials
3. Generates synthetic data with automatic annotations
4. Integrates with ROS 2 for robot control
5. Demonstrates the sim-to-real transfer capability

Summary

In this section, we've explored Isaac Sim's capabilities:

• Photorealistic rendering and environment creation
• Synthetic data generation with automatic annotations
• Domain randomization for robust AI training
• Integration with ROS/ROS 2 systems
• Performance optimization techniques

The next section will cover Isaac ROS, focusing on hardware-accelerated perception and navigation.