Skip to content

Fix --video producing black frames without --viz kit (issue #5052)#5056

Open
pascal-roth wants to merge 3 commits intodevelopfrom
pascalr/fix-video-headless-render
Open

Fix --video producing black frames without --viz kit (issue #5052)#5056
pascal-roth wants to merge 3 commits intodevelopfrom
pascalr/fix-video-headless-render

Conversation

@pascal-roth
Copy link
Collaborator

Summary

  • Root cause: SimulationContext.render() was not calling omni.kit.app.get_app().update() when no active visualizer pumped the Kit app loop. Replicator render products (used by gym.wrappers.RecordVideo via render_mode="rgb_array") were never refreshed, producing black video frames.
  • Fix: Call app.update() in render() when has_kit() is True and no visualizer with pumps_app_update() == True is active. KitVisualizer already calls app.update() in its own step(), so we skip the call to avoid double-rendering when --viz kit is used.
  • Tests: Two regression tests added to test_simulation_context.py — one verifying app.update() IS called without a visualizer, one verifying it is NOT called when a visualizer already pumps the Kit loop.

Test plan

  • Regression test test_render_pumps_app_update_without_visualizer — fails without fix, passes with fix
  • Regression test test_render_skips_app_update_when_visualizer_pumps_it — passes with fix (no double-rendering)
  • Pre-commit checks pass (./isaaclab.sh -f)
  • Manual: ./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Velocity-Rough-Anymal-C-v0 --video should produce correct (non-black) video

Closes #5052

🤖 Generated with Claude Code

SimulationContext.render() was not calling omni.kit.app.get_app().update()
when running with Isaac Sim (Kit) and no active visualizer pumped the Kit
app loop. This meant replicator render products (used by
gym.wrappers.RecordVideo via render_mode="rgb_array") were never refreshed,
producing black video frames.

The fix calls app.update() in render() when has_kit() is True and no
visualizer with pumps_app_update() == True is active. KitVisualizer
already calls app.update() in its own step(), so we skip the call to
avoid double-rendering when --viz kit is used.

Adds two regression tests to verify:
1. render() calls app.update() when no visualizer pumps the Kit loop
2. render() does not call app.update() when a visualizer already does

Fixes #5052
@github-actions github-actions bot added bug Something isn't working documentation Improvements or additions to documentation asset New asset feature or request isaac-sim Related to Isaac Sim team isaac-mimic Related to Isaac Mimic team infrastructure labels Mar 18, 2026
@pascal-roth pascal-roth changed the base branch from main to develop March 18, 2026 13:32
@github-actions github-actions bot added the isaac-lab Related to Isaac Lab team label Mar 18, 2026
@nblauch
Copy link

nblauch commented Mar 18, 2026

Hi @pascal-roth . Thanks for this! I tested it out. It does fix the blank videos, but it seems the camera position is not updated properly according to the ViewerCfg. For the example listed in my issue (#5052 ), the video outputs are from a different perspective for --viz none (headless) vs. --viz kit. I also found this for other custom scripts using a DirectRLEnv setup.

@pascal-roth pascal-roth removed documentation Improvements or additions to documentation asset New asset feature or request isaac-sim Related to Isaac Sim team isaac-mimic Related to Isaac Mimic team infrastructure labels Mar 20, 2026
@pascal-roth
Copy link
Collaborator Author

Thanks for testing and reporting this, @nblauch!

The camera perspective mismatch you're seeing (headless --viz none vs --viz kit) is a separate issue from the black frames this PR addresses. In short: this PR only ensures app.update() is called so replicator render products are refreshed — it doesn't set the recording camera's position from ViewerCfg.

Good news: that exact problem is already being fixed in PR #5011 (Add perspective video recording via Newton GL viewer for Kitless backends; Fix for Kitfull backends camera position). That PR explicitly reads viewer.eye / viewer.lookat from the task config and passes them to the video recorder as camera_position / camera_target, so the recording matches the configured viewer perspective regardless of the visualizer backend.

To avoid duplicating the fix here and keeping both diffs focused, I'd suggest we track resolution in #5011. Once that lands, the ViewerCfg perspective will be honoured in all render modes.

Let me know if the scope there differs from what you experienced!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug Report] --video does not work in headless mode

2 participants