From 8218ac0c95248f8c1ca0afbc823fa61d9f7be9ba Mon Sep 17 00:00:00 2001 From: Igor Shovkun Date: Wed, 2 Sep 2026 09:10:04 -0700 Subject: [PATCH 1/8] Add TensorRT-LLM static FP8 cookbook Signed-off-by: Igor Shovkun --- README.md | 1 + .../cosmos3/generator/audiovisual/README.md | 10 + .../run_static_fp8_with_trt_llm.ipynb | 465 ++++++++++++++++++ 3 files changed, 476 insertions(+) create mode 100644 cookbooks/cosmos3/generator/audiovisual/run_static_fp8_with_trt_llm.ipynb diff --git a/README.md b/README.md index 8c6f265b..164119a6 100644 --- a/README.md +++ b/README.md @@ -1159,6 +1159,7 @@ We are building examples that show Cosmos 3 Super/Nano/Edge capabilities end to | Generator (audiovisual) with Diffusers | Generator | Text-to-image, plus text-to-video and image-to-video each with or without synchronized sound, via `Cosmos3OmniPipeline`. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb) | | Generator (audiovisual) with Cosmos Framework | Generator | Text-to-image, plus text-to-video and image-to-video each with sound on or off, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_cosmos_framework.ipynb) | | Generator (audiovisual) with TensorRT-LLM | Generator | Text-to-image, text-to-video and image-to-video with or without synchronized audio, and video-to-video against an OpenAI-compatible TensorRT-LLM VisualGen server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_trt_llm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_trt_llm.ipynb) | +| Generator static FP8 with TensorRT-LLM | Generator | Single-GPU text-to-image, text-to-video, image-to-video, and video-to-video generation from local ModelOpt-calibrated Cosmos3 Nano and Super FP8 checkpoints. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_static_fp8_with_trt_llm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_static_fp8_with_trt_llm.ipynb) | | Generator (audiovisual) with vLLM-Omni | Generator | Text-to-image, text-to-video, image-to-video, and video-to-video, with supported sound modes, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb) | | Generator (audiovisual) with NIM | Generator | Text2Video and Image2Video only, against the prebuilt `Cosmos3-Generator` NIM; requests use `POST /v1/infer` and decode JSON `b64_video` responses. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb) | | Generator (audiovisual) with SGLang | Generator | Text-to-image, plus text-to-video and image-to-video each with sound on or off, against an OpenAI-compatible SGLang server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_sglang.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_sglang.ipynb) | diff --git a/cookbooks/cosmos3/generator/audiovisual/README.md b/cookbooks/cosmos3/generator/audiovisual/README.md index 53062009..2aa7eec2 100644 --- a/cookbooks/cosmos3/generator/audiovisual/README.md +++ b/cookbooks/cosmos3/generator/audiovisual/README.md @@ -374,6 +374,16 @@ Super, FP8 dynamic quantization, CFG parallelism, Ulysses, and parallel VAE) live in the [shared environment setup guide](../../README.md#tensorrt-llm-generator). +### Static FP8 notebook + +[`run_static_fp8_with_trt_llm.ipynb`](./run_static_fp8_with_trt_llm.ipynb) +runs local, ModelOpt-calibrated Cosmos3 Nano and Super FP8 checkpoints directly +through TensorRT-LLM's offline Cosmos3 entry point. It contains the complete +single-GPU commands for text-to-image, text-to-video, image-to-video, and +video-to-video generation, followed by PNG and MP4 validation. The checkpoint +metadata supplies the calibrated weight and activation scales; no runtime +quantization flag is required. + ## Run with NIM ### Quickstart diff --git a/cookbooks/cosmos3/generator/audiovisual/run_static_fp8_with_trt_llm.ipynb b/cookbooks/cosmos3/generator/audiovisual/run_static_fp8_with_trt_llm.ipynb new file mode 100644 index 00000000..57de3519 --- /dev/null +++ b/cookbooks/cosmos3/generator/audiovisual/run_static_fp8_with_trt_llm.ipynb @@ -0,0 +1,465 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "markdown-00-95d2e8c3", + "metadata": {}, + "source": [ + "\n", + "\n", + "# Cosmos3 Generator static FP8 with TensorRT-LLM\n", + "\n", + "TensorRT-LLM can run ModelOpt-calibrated Cosmos3 Nano and Super FP8\n", + "checkpoints directly. The checkpoint metadata selects static FP8, including\n", + "its calibrated weight and activation scales, so no quantization CLI flag is\n", + "needed.\n", + "\n", + "This notebook covers the four validated single-GPU modes:\n", + "\n", + "| Mode | Prompt or condition | Output |\n", + "| -------------------- | ------------------------------------- | --------- |\n", + "| Text to image (T2I) | Structured text prompt | PNG image |\n", + "| Text to video (T2V) | Structured text prompt | MP4 video |\n", + "| Image to video (I2V) | Structured prompt and reference image | MP4 video |\n", + "| Video to video (V2V) | Structured prompt and reference video | MP4 video |\n", + "\n", + "FP8 Cosmos3 is single-GPU only. TensorRT-LLM rejects tensor parallelism,\n", + "Ulysses parallelism, context parallelism, CFG parallelism, or parallel VAE\n", + "sizes greater than one for these checkpoints. Use a BF16 checkpoint for a\n", + "multi-GPU deployment.\n", + "\n", + "## 1. Prerequisites\n", + "\n", + "Build and install TensorRT-LLM from its `main` branch by following the\n", + "[TensorRT-LLM source-build guide](https://nvidia.github.io/TensorRT-LLM/installation/build-from-source.html).\n", + "The source checkout supplies the Cosmos3 example, prompt files, and one-GPU\n", + "configuration files used below.\n", + "\n", + "Install `ffmpeg` so TensorRT-LLM can encode MP4 output. Install the Cosmos3\n", + "guardrail package, accept the gated\n", + "[`nvidia/Cosmos-1.0-Guardrail`](https://huggingface.co/nvidia/Cosmos-1.0-Guardrail)\n", + "terms, and authenticate to Hugging Face before generation:\n", + "\n", + "```shell\n", + "apt-get update && apt-get install -y ffmpeg\n", + "pip install cosmos_guardrail==0.3.0\n", + "pip uninstall -y opencv-python\n", + "pip install opencv-python-headless\n", + "hf auth login\n", + "```\n", + "\n", + "The pinned guardrail package performs prompt text checks and RetinaFace face\n", + "blurring. Its generated-frame content classifier is disabled upstream because\n", + "it produced too many false positives.\n", + "\n", + "Nano and Super publish their ModelOpt FP8 checkpoints on the `fp8` revision of\n", + "their Hugging Face repositories. Download each revision to a local directory;\n", + "TensorRT-LLM receives that directory through `--model`:\n", + "\n", + "```shell\n", + "hf download nvidia/Cosmos3-Nano \\\n", + " --revision fp8 \\\n", + " --local-dir checkpoints/Cosmos3-Nano-FP8\n", + "hf download nvidia/Cosmos3-Super \\\n", + " --revision fp8 \\\n", + " --local-dir checkpoints/Cosmos3-Super-FP8\n", + "```\n", + "\n", + "Set the paths used by the commands in this notebook. You may export these\n", + "variables before starting Jupyter or replace the placeholder defaults below:\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "python-01-bdb74719", + "metadata": {}, + "outputs": [], + "source": [ + "from pathlib import Path\n", + "import os\n", + "\n", + "\n", + "TRTLLM_ROOT = Path(os.environ.get(\"TRTLLM_ROOT\", \"/path/to/TensorRT-LLM\")).expanduser().resolve()\n", + "COSMOS3_NANO_FP8 = Path(\n", + " os.environ.get(\"COSMOS3_NANO_FP8\", Path.cwd() / \"checkpoints\" / \"Cosmos3-Nano-FP8\")\n", + ").expanduser().resolve()\n", + "COSMOS3_SUPER_FP8 = Path(\n", + " os.environ.get(\"COSMOS3_SUPER_FP8\", Path.cwd() / \"checkpoints\" / \"Cosmos3-Super-FP8\")\n", + ").expanduser().resolve()\n", + "COSMOS3_FP8_OUTPUTS = Path(\n", + " os.environ.get(\"COSMOS3_FP8_OUTPUTS\", Path.cwd() / \"outputs\" / \"tensorrt_llm_static_fp8\")\n", + ").expanduser().resolve()\n", + "\n", + "for name, path in {\n", + " \"TRTLLM_ROOT\": TRTLLM_ROOT,\n", + " \"COSMOS3_NANO_FP8\": COSMOS3_NANO_FP8,\n", + " \"COSMOS3_SUPER_FP8\": COSMOS3_SUPER_FP8,\n", + "}.items():\n", + " if not path.exists():\n", + " raise FileNotFoundError(f\"Set {name} to an existing path; got {path}\")\n", + "\n", + "COSMOS3_FP8_OUTPUTS.mkdir(parents=True, exist_ok=True)\n", + "for name, path in {\n", + " \"TRTLLM_ROOT\": TRTLLM_ROOT,\n", + " \"COSMOS3_NANO_FP8\": COSMOS3_NANO_FP8,\n", + " \"COSMOS3_SUPER_FP8\": COSMOS3_SUPER_FP8,\n", + " \"COSMOS3_FP8_OUTPUTS\": COSMOS3_FP8_OUTPUTS,\n", + "}.items():\n", + " os.environ[name] = str(path)\n", + " print(f\"{name}={path}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "markdown-02-5954b927", + "metadata": {}, + "source": [ + "\n", + "The checkpoints may carry a `diffusion_step_policy` in their quantization\n", + "metadata. TensorRT-LLM applies that checkpoint-owned policy automatically; do\n", + "not add a separate runtime override for it.\n", + "\n", + "## 2. Cosmos3 Nano FP8\n", + "\n", + "### Text to image\n", + "\n", + "The text-to-image config warms the 1024x1024, one-frame shape. The structured\n", + "prompt selects image mode, and `--output_type image` makes the expected output\n", + "explicit.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bash-03-0f16ab37", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "\n", + "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", + " --model \"$COSMOS3_NANO_FP8\" \\\n", + " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-t2i-1gpu.yaml\" \\\n", + " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/t2i.json\" \\\n", + " --output_type image \\\n", + " --output_path \"$COSMOS3_FP8_OUTPUTS/nano_t2i.png\"\n" + ] + }, + { + "cell_type": "markdown", + "id": "markdown-04-36f6f594", + "metadata": {}, + "source": [ + "\n", + "### Text to video\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bash-05-ca0df107", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "\n", + "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", + " --model \"$COSMOS3_NANO_FP8\" \\\n", + " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", + " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/t2v.json\" \\\n", + " --output_path \"$COSMOS3_FP8_OUTPUTS/nano_t2v.mp4\"\n" + ] + }, + { + "cell_type": "markdown", + "id": "markdown-06-4f217480", + "metadata": {}, + "source": [ + "\n", + "### Image to video\n", + "\n", + "The example I2V prompt names its reference image with an HTTPS URL. Use\n", + "`--image_path` to replace it with a local path, `file://` URL, HTTP(S) URL, or\n", + "`data:` URI.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bash-07-baadc896", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "\n", + "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", + " --model \"$COSMOS3_NANO_FP8\" \\\n", + " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", + " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/i2v.json\" \\\n", + " --output_path \"$COSMOS3_FP8_OUTPUTS/nano_i2v.mp4\"\n" + ] + }, + { + "cell_type": "markdown", + "id": "markdown-08-9c92db9f", + "metadata": {}, + "source": [ + "\n", + "### Video to video\n", + "\n", + "This command uses the Nano T2V output above as the reference video. By default,\n", + "the first five decoded pixel frames condition the output.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bash-09-ab023113", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "\n", + "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", + " --model \"$COSMOS3_NANO_FP8\" \\\n", + " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", + " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/v2v.json\" \\\n", + " --video_path \"$COSMOS3_FP8_OUTPUTS/nano_t2v.mp4\" \\\n", + " --output_path \"$COSMOS3_FP8_OUTPUTS/nano_v2v.mp4\"\n" + ] + }, + { + "cell_type": "markdown", + "id": "markdown-10-e177f899", + "metadata": {}, + "source": [ + "\n", + "## 3. Cosmos3 Super FP8\n", + "\n", + "The same one-GPU interfaces apply to the Super checkpoint. Do not use\n", + "`cosmos3-super-4gpu.yaml`; static FP8 rejects that parallel configuration.\n", + "\n", + "### Text to image\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bash-11-15a5a2ad", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "\n", + "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", + " --model \"$COSMOS3_SUPER_FP8\" \\\n", + " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-t2i-1gpu.yaml\" \\\n", + " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/t2i.json\" \\\n", + " --output_type image \\\n", + " --output_path \"$COSMOS3_FP8_OUTPUTS/super_t2i.png\"\n" + ] + }, + { + "cell_type": "markdown", + "id": "markdown-12-eb633740", + "metadata": {}, + "source": [ + "\n", + "### Text to video\n", + "\n", + "The file name `cosmos3-nano-1gpu.yaml` is historical: TensorRT-LLM documents\n", + "the configuration as the shared one-GPU config for both Nano and Super.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bash-13-7601d3d3", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "\n", + "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", + " --model \"$COSMOS3_SUPER_FP8\" \\\n", + " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", + " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/t2v.json\" \\\n", + " --output_path \"$COSMOS3_FP8_OUTPUTS/super_t2v.mp4\"\n" + ] + }, + { + "cell_type": "markdown", + "id": "markdown-14-8055a556", + "metadata": {}, + "source": [ + "\n", + "### Image to video\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bash-15-0056f0db", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "\n", + "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", + " --model \"$COSMOS3_SUPER_FP8\" \\\n", + " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", + " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/i2v.json\" \\\n", + " --output_path \"$COSMOS3_FP8_OUTPUTS/super_i2v.mp4\"\n" + ] + }, + { + "cell_type": "markdown", + "id": "markdown-16-b92c6894", + "metadata": {}, + "source": [ + "\n", + "### Video to video\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bash-17-0579fa3f", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "\n", + "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", + " --model \"$COSMOS3_SUPER_FP8\" \\\n", + " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", + " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/v2v.json\" \\\n", + " --video_path \"$COSMOS3_FP8_OUTPUTS/super_t2v.mp4\" \\\n", + " --output_path \"$COSMOS3_FP8_OUTPUTS/super_v2v.mp4\"\n" + ] + }, + { + "cell_type": "markdown", + "id": "markdown-18-674ba7b9", + "metadata": {}, + "source": [ + "\n", + "## 4. Verify the artifacts\n", + "\n", + "Decode the two PNG files and print their dimensions:\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bash-19-41172619", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "\n", + "python - \\\n", + " \"$COSMOS3_FP8_OUTPUTS/nano_t2i.png\" \\\n", + " \"$COSMOS3_FP8_OUTPUTS/super_t2i.png\" <<'PY'\n", + "from PIL import Image\n", + "import sys\n", + "\n", + "for path in sys.argv[1:]:\n", + " with Image.open(path) as image:\n", + " image.verify()\n", + " print(path, image.format, image.size)\n", + "PY\n" + ] + }, + { + "cell_type": "markdown", + "id": "markdown-20-39accea0", + "metadata": {}, + "source": [ + "\n", + "Inspect and fully decode every generated video stream:\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bash-21-b400c1f9", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "\n", + "for output in \\\n", + " \"$COSMOS3_FP8_OUTPUTS/nano_t2v.mp4\" \\\n", + " \"$COSMOS3_FP8_OUTPUTS/nano_i2v.mp4\" \\\n", + " \"$COSMOS3_FP8_OUTPUTS/nano_v2v.mp4\" \\\n", + " \"$COSMOS3_FP8_OUTPUTS/super_t2v.mp4\" \\\n", + " \"$COSMOS3_FP8_OUTPUTS/super_i2v.mp4\" \\\n", + " \"$COSMOS3_FP8_OUTPUTS/super_v2v.mp4\"; do\n", + " ffprobe -v error \\\n", + " -show_entries stream=codec_type,codec_name,width,height,r_frame_rate,nb_frames \\\n", + " -of json \\\n", + " \"$output\"\n", + " ffmpeg -v error -i \"$output\" -f null -\n", + "done\n" + ] + }, + { + "cell_type": "markdown", + "id": "markdown-22-334c3588", + "metadata": {}, + "source": [ + "\n", + "A successful encode/decode proves the interface and artifact contract; it is\n", + "not a claim that FP8 output quality matches BF16.\n", + "\n", + "## 5. Scope\n", + "\n", + "The Nano and Super FP8 checkpoints also contain an audio tower, so T2AV and\n", + "TI2AV requests run rather than being refused. Audio has not received the same\n", + "FP8 exercise as the four image/video modes above, and no FP8 audio-quality\n", + "claim is made here.\n", + "\n", + "For the complete TensorRT-LLM Cosmos3 example contract, including serving,\n", + "guardrails, media dependencies, and other checkpoint families, see the\n", + "[TensorRT-LLM Cosmos3 example](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/visual_gen/models/cosmos3).\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From b24a77f3c25c6c586dca9b2ed87735c8259ad51e Mon Sep 17 00:00:00 2001 From: Igor Shovkun Date: Wed, 2 Sep 2026 10:17:55 -0700 Subject: [PATCH 2/8] Document TensorRT-LLM static FP8 setup --- cookbooks/cosmos3/README.md | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/cookbooks/cosmos3/README.md b/cookbooks/cosmos3/README.md index 383d970c..33d29a1a 100644 --- a/cookbooks/cosmos3/README.md +++ b/cookbooks/cosmos3/README.md @@ -8,7 +8,7 @@ backend you want to run and follow that one section. | --- | --- | --- | | [Cosmos Framework](#cosmos-framework) | Native PyTorch inference, launched with `torchrun` | Reasoner, Generator (Audiovisual, Action, **Transfer**) | | [Diffusers](#diffusers) | Direct generation with `Cosmos3OmniPipeline` | Generator (Audiovisual) | -| [TensorRT-LLM Generator](#tensorrt-llm-generator) | OpenAI-compatible VisualGen server (image/video/audio generation) | Generator (Audiovisual) | +| [TensorRT-LLM Generator](#tensorrt-llm-generator) | Offline static-FP8 image/video generation and OpenAI-compatible VisualGen serving | Generator (Audiovisual) | | [TensorRT-LLM Reasoner](#tensorrt-llm-reasoner) | OpenAI-compatible image/video reasoning server | Reasoner | | [Transformers](#transformers) | Hugging Face Transformers inference | Reasoner | | [vLLM](#vllm) | OpenAI-compatible reasoning server (image/video understanding) | Reasoner | @@ -168,8 +168,9 @@ uv pip install --torch-backend=cu130 \ ## TensorRT-LLM Generator -OpenAI-compatible **VisualGen** server for Generator audiovisual text-to-image, -text-to-video, image-to-video, video-to-video, and synchronized audio examples. +TensorRT-LLM supports offline static-FP8 generation and an OpenAI-compatible +**VisualGen** server for Generator audiovisual text-to-image, text-to-video, +image-to-video, video-to-video, and synchronized audio examples. Initial Cosmos3 support was added in TensorRT-LLM PR [#14824](https://github.com/NVIDIA/TensorRT-LLM/pull/14824), synchronized audio in [#14827](https://github.com/NVIDIA/TensorRT-LLM/pull/14827), and @@ -221,6 +222,31 @@ pip install cosmos_guardrail==0.3.0 # pip uninstall opencv-python ``` +### Static FP8 (offline, single GPU) + +Cosmos3 Nano and Super publish ModelOpt-calibrated checkpoints on the `fp8` +revision of their Hugging Face repositories. These revisions contain the FP8 +weights, activation scales, and runtime policy. TensorRT-LLM reads that metadata +directly, so no quantization flag is required. + +```bash +hf download nvidia/Cosmos3-Nano \ + --revision fp8 \ + --local-dir checkpoints/Cosmos3-Nano-FP8 +hf download nvidia/Cosmos3-Super \ + --revision fp8 \ + --local-dir checkpoints/Cosmos3-Super-FP8 +``` + +Static FP8 supports one GPU only. Use the BF16 checkpoints for tensor, Ulysses, +context, or CFG parallelism, or for parallel VAE. The +[static-FP8 notebook](generator/audiovisual/run_static_fp8_with_trt_llm.ipynb) +runs Nano and Super through the offline TensorRT-LLM entry point for +text-to-image, text-to-video, image-to-video, and video-to-video generation and +then validates every PNG and MP4 artifact. + +### VisualGen server + Set the TensorRT-LLM source root for the shared VisualGen config YAMLs: ```bash From 8d48b34e02b71a54a14cdd1ec8649c3eeb0d7716 Mon Sep 17 00:00:00 2001 From: Igor Shovkun Date: Wed, 2 Sep 2026 10:18:34 -0700 Subject: [PATCH 3/8] Clarify TensorRT-LLM guardrail setup scope --- cookbooks/cosmos3/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/cookbooks/cosmos3/README.md b/cookbooks/cosmos3/README.md index 33d29a1a..673a6445 100644 --- a/cookbooks/cosmos3/README.md +++ b/cookbooks/cosmos3/README.md @@ -214,7 +214,8 @@ For Python-only changes, the upstream guide also documents installing the checkout in editable mode. Then install the Cosmos3 guardrail package in the same environment unless you -explicitly disable guardrails before starting the server: +explicitly disable guardrails before running offline inference or starting the +server: ```bash pip install cosmos_guardrail==0.3.0 From 86e0e091698438c7477b90e3b3f2861cc73f3c34 Mon Sep 17 00:00:00 2001 From: Igor Shovkun Date: Wed, 2 Sep 2026 10:22:58 -0700 Subject: [PATCH 4/8] Use Cosmos3 FP8 checkpoint naming --- README.md | 2 +- cookbooks/cosmos3/README.md | 12 ++++++------ cookbooks/cosmos3/generator/audiovisual/README.md | 4 ++-- ...with_trt_llm.ipynb => run_fp8_with_trt_llm.ipynb} | 11 ++++++----- 4 files changed, 15 insertions(+), 14 deletions(-) rename cookbooks/cosmos3/generator/audiovisual/{run_static_fp8_with_trt_llm.ipynb => run_fp8_with_trt_llm.ipynb} (97%) diff --git a/README.md b/README.md index 164119a6..4d3c9e39 100644 --- a/README.md +++ b/README.md @@ -1159,7 +1159,7 @@ We are building examples that show Cosmos 3 Super/Nano/Edge capabilities end to | Generator (audiovisual) with Diffusers | Generator | Text-to-image, plus text-to-video and image-to-video each with or without synchronized sound, via `Cosmos3OmniPipeline`. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb) | | Generator (audiovisual) with Cosmos Framework | Generator | Text-to-image, plus text-to-video and image-to-video each with sound on or off, through the `cosmos_framework.scripts.inference` entrypoint. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_cosmos_framework.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_cosmos_framework.ipynb) | | Generator (audiovisual) with TensorRT-LLM | Generator | Text-to-image, text-to-video and image-to-video with or without synchronized audio, and video-to-video against an OpenAI-compatible TensorRT-LLM VisualGen server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_trt_llm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_trt_llm.ipynb) | -| Generator static FP8 with TensorRT-LLM | Generator | Single-GPU text-to-image, text-to-video, image-to-video, and video-to-video generation from local ModelOpt-calibrated Cosmos3 Nano and Super FP8 checkpoints. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_static_fp8_with_trt_llm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_static_fp8_with_trt_llm.ipynb) | +| Cosmos3 Nano and Super FP8 with TensorRT-LLM | Generator | Single-GPU text-to-image, text-to-video, image-to-video, and video-to-video generation from local ModelOpt-calibrated Cosmos3 Nano and Super FP8 checkpoints. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb) | | Generator (audiovisual) with vLLM-Omni | Generator | Text-to-image, text-to-video, image-to-video, and video-to-video, with supported sound modes, against an OpenAI-compatible vLLM-Omni server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb) | | Generator (audiovisual) with NIM | Generator | Text2Video and Image2Video only, against the prebuilt `Cosmos3-Generator` NIM; requests use `POST /v1/infer` and decode JSON `b64_video` responses. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb) | | Generator (audiovisual) with SGLang | Generator | Text-to-image, plus text-to-video and image-to-video each with sound on or off, against an OpenAI-compatible SGLang server. | [Notebook](cookbooks/cosmos3/generator/audiovisual/run_with_sglang.ipynb) | [![Render with nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.org/github/nvidia/cosmos/blob/main/cookbooks/cosmos3/generator/audiovisual/run_with_sglang.ipynb) | diff --git a/cookbooks/cosmos3/README.md b/cookbooks/cosmos3/README.md index 673a6445..badc7c1f 100644 --- a/cookbooks/cosmos3/README.md +++ b/cookbooks/cosmos3/README.md @@ -8,7 +8,7 @@ backend you want to run and follow that one section. | --- | --- | --- | | [Cosmos Framework](#cosmos-framework) | Native PyTorch inference, launched with `torchrun` | Reasoner, Generator (Audiovisual, Action, **Transfer**) | | [Diffusers](#diffusers) | Direct generation with `Cosmos3OmniPipeline` | Generator (Audiovisual) | -| [TensorRT-LLM Generator](#tensorrt-llm-generator) | Offline static-FP8 image/video generation and OpenAI-compatible VisualGen serving | Generator (Audiovisual) | +| [TensorRT-LLM Generator](#tensorrt-llm-generator) | Offline FP8 image/video generation and OpenAI-compatible VisualGen serving | Generator (Audiovisual) | | [TensorRT-LLM Reasoner](#tensorrt-llm-reasoner) | OpenAI-compatible image/video reasoning server | Reasoner | | [Transformers](#transformers) | Hugging Face Transformers inference | Reasoner | | [vLLM](#vllm) | OpenAI-compatible reasoning server (image/video understanding) | Reasoner | @@ -168,7 +168,7 @@ uv pip install --torch-backend=cu130 \ ## TensorRT-LLM Generator -TensorRT-LLM supports offline static-FP8 generation and an OpenAI-compatible +TensorRT-LLM supports offline FP8 checkpoint generation and an OpenAI-compatible **VisualGen** server for Generator audiovisual text-to-image, text-to-video, image-to-video, video-to-video, and synchronized audio examples. Initial Cosmos3 support was added in TensorRT-LLM PR @@ -223,7 +223,7 @@ pip install cosmos_guardrail==0.3.0 # pip uninstall opencv-python ``` -### Static FP8 (offline, single GPU) +### Cosmos3 Nano and Super FP8 checkpoints (offline, single GPU) Cosmos3 Nano and Super publish ModelOpt-calibrated checkpoints on the `fp8` revision of their Hugging Face repositories. These revisions contain the FP8 @@ -239,9 +239,9 @@ hf download nvidia/Cosmos3-Super \ --local-dir checkpoints/Cosmos3-Super-FP8 ``` -Static FP8 supports one GPU only. Use the BF16 checkpoints for tensor, Ulysses, -context, or CFG parallelism, or for parallel VAE. The -[static-FP8 notebook](generator/audiovisual/run_static_fp8_with_trt_llm.ipynb) +These FP8 checkpoints support one GPU only. Use the BF16 checkpoints for tensor, +Ulysses, context, or CFG parallelism, or for parallel VAE. The +[FP8 checkpoint notebook](generator/audiovisual/run_fp8_with_trt_llm.ipynb) runs Nano and Super through the offline TensorRT-LLM entry point for text-to-image, text-to-video, image-to-video, and video-to-video generation and then validates every PNG and MP4 artifact. diff --git a/cookbooks/cosmos3/generator/audiovisual/README.md b/cookbooks/cosmos3/generator/audiovisual/README.md index 2aa7eec2..f4452022 100644 --- a/cookbooks/cosmos3/generator/audiovisual/README.md +++ b/cookbooks/cosmos3/generator/audiovisual/README.md @@ -374,9 +374,9 @@ Super, FP8 dynamic quantization, CFG parallelism, Ulysses, and parallel VAE) live in the [shared environment setup guide](../../README.md#tensorrt-llm-generator). -### Static FP8 notebook +### Cosmos3 Nano and Super FP8 notebook -[`run_static_fp8_with_trt_llm.ipynb`](./run_static_fp8_with_trt_llm.ipynb) +[`run_fp8_with_trt_llm.ipynb`](./run_fp8_with_trt_llm.ipynb) runs local, ModelOpt-calibrated Cosmos3 Nano and Super FP8 checkpoints directly through TensorRT-LLM's offline Cosmos3 entry point. It contains the complete single-GPU commands for text-to-image, text-to-video, image-to-video, and diff --git a/cookbooks/cosmos3/generator/audiovisual/run_static_fp8_with_trt_llm.ipynb b/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb similarity index 97% rename from cookbooks/cosmos3/generator/audiovisual/run_static_fp8_with_trt_llm.ipynb rename to cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb index 57de3519..fb91bb82 100644 --- a/cookbooks/cosmos3/generator/audiovisual/run_static_fp8_with_trt_llm.ipynb +++ b/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb @@ -8,11 +8,11 @@ "\n", "\n", - "# Cosmos3 Generator static FP8 with TensorRT-LLM\n", + "# Run Cosmos3 Nano and Super FP8 checkpoints with TensorRT-LLM\n", "\n", "TensorRT-LLM can run ModelOpt-calibrated Cosmos3 Nano and Super FP8\n", - "checkpoints directly. The checkpoint metadata selects static FP8, including\n", - "its calibrated weight and activation scales, so no quantization CLI flag is\n", + "checkpoints directly. The checkpoint metadata supplies the calibrated weight\n", + "and activation scales, so no quantization CLI flag is\n", "needed.\n", "\n", "This notebook covers the four validated single-GPU modes:\n", @@ -90,7 +90,7 @@ " os.environ.get(\"COSMOS3_SUPER_FP8\", Path.cwd() / \"checkpoints\" / \"Cosmos3-Super-FP8\")\n", ").expanduser().resolve()\n", "COSMOS3_FP8_OUTPUTS = Path(\n", - " os.environ.get(\"COSMOS3_FP8_OUTPUTS\", Path.cwd() / \"outputs\" / \"tensorrt_llm_static_fp8\")\n", + " os.environ.get(\"COSMOS3_FP8_OUTPUTS\", Path.cwd() / \"outputs\" / \"tensorrt_llm_fp8\")\n", ").expanduser().resolve()\n", "\n", "for name, path in {\n", @@ -248,7 +248,8 @@ "## 3. Cosmos3 Super FP8\n", "\n", "The same one-GPU interfaces apply to the Super checkpoint. Do not use\n", - "`cosmos3-super-4gpu.yaml`; static FP8 rejects that parallel configuration.\n", + "`cosmos3-super-4gpu.yaml`; these FP8 checkpoints reject that parallel\n", + "configuration.\n", "\n", "### Text to image\n", "\n" From 8cb525c62ecaea101f7096e2cd7cd38ab9f3f170 Mon Sep 17 00:00:00 2001 From: Igor Shovkun Date: Thu, 3 Sep 2026 10:24:37 -0700 Subject: [PATCH 5/8] Use structured prompts for FP8 image generation --- .../audiovisual/run_fp8_with_trt_llm.ipynb | 31 ++++++++++++++----- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb b/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb index fb91bb82..d8134550 100644 --- a/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb +++ b/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb @@ -33,8 +33,9 @@ "\n", "Build and install TensorRT-LLM from its `main` branch by following the\n", "[TensorRT-LLM source-build guide](https://nvidia.github.io/TensorRT-LLM/installation/build-from-source.html).\n", - "The source checkout supplies the Cosmos3 example, prompt files, and one-GPU\n", - "configuration files used below.\n", + "The source checkout supplies the Cosmos3 example and one-GPU configuration\n", + "files used below. The structured T2I prompt comes from this cookbook's shared\n", + "audiovisual assets.\n", "\n", "Install `ffmpeg` so TensorRT-LLM can encode MP4 output. Install the Cosmos3\n", "guardrail package, accept the gated\n", @@ -82,6 +83,20 @@ "import os\n", "\n", "\n", + "def find_repo_root(start: Path) -> Path:\n", + " for path in [start, *start.parents]:\n", + " if (path / \"README.md\").exists() and (path / \"cookbooks\").exists():\n", + " return path\n", + " return start\n", + "\n", + "\n", + "COSMOS_ROOT = find_repo_root(Path.cwd().resolve())\n", + "COSMOS3_AUDIOVISUAL_ROOT = (\n", + " COSMOS_ROOT / \"cookbooks\" / \"cosmos3\" / \"generator\" / \"audiovisual\"\n", + ")\n", + "COSMOS3_T2I_PROMPT = (\n", + " COSMOS3_AUDIOVISUAL_ROOT / \"assets\" / \"prompts\" / \"text2image\" / \"robot_draping.json\"\n", + ")\n", "TRTLLM_ROOT = Path(os.environ.get(\"TRTLLM_ROOT\", \"/path/to/TensorRT-LLM\")).expanduser().resolve()\n", "COSMOS3_NANO_FP8 = Path(\n", " os.environ.get(\"COSMOS3_NANO_FP8\", Path.cwd() / \"checkpoints\" / \"Cosmos3-Nano-FP8\")\n", @@ -97,6 +112,7 @@ " \"TRTLLM_ROOT\": TRTLLM_ROOT,\n", " \"COSMOS3_NANO_FP8\": COSMOS3_NANO_FP8,\n", " \"COSMOS3_SUPER_FP8\": COSMOS3_SUPER_FP8,\n", + " \"COSMOS3_T2I_PROMPT\": COSMOS3_T2I_PROMPT,\n", "}.items():\n", " if not path.exists():\n", " raise FileNotFoundError(f\"Set {name} to an existing path; got {path}\")\n", @@ -106,6 +122,7 @@ " \"TRTLLM_ROOT\": TRTLLM_ROOT,\n", " \"COSMOS3_NANO_FP8\": COSMOS3_NANO_FP8,\n", " \"COSMOS3_SUPER_FP8\": COSMOS3_SUPER_FP8,\n", + " \"COSMOS3_T2I_PROMPT\": COSMOS3_T2I_PROMPT,\n", " \"COSMOS3_FP8_OUTPUTS\": COSMOS3_FP8_OUTPUTS,\n", "}.items():\n", " os.environ[name] = str(path)\n", @@ -126,9 +143,9 @@ "\n", "### Text to image\n", "\n", - "The text-to-image config warms the 1024x1024, one-frame shape. The structured\n", - "prompt selects image mode, and `--output_type image` makes the expected output\n", - "explicit.\n", + "The text-to-image config warms the 1024x1024, one-frame shape. The cookbook's\n", + "structured T2I prompt matches the checkpoint's trained prompt format, and\n", + "`--output_type image` makes the expected output explicit.\n", "\n" ] }, @@ -145,7 +162,7 @@ "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", " --model \"$COSMOS3_NANO_FP8\" \\\n", " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-t2i-1gpu.yaml\" \\\n", - " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/t2i.json\" \\\n", + " --prompt_file \"$COSMOS3_T2I_PROMPT\" \\\n", " --output_type image \\\n", " --output_path \"$COSMOS3_FP8_OUTPUTS/nano_t2i.png\"\n" ] @@ -268,7 +285,7 @@ "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", " --model \"$COSMOS3_SUPER_FP8\" \\\n", " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-t2i-1gpu.yaml\" \\\n", - " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/t2i.json\" \\\n", + " --prompt_file \"$COSMOS3_T2I_PROMPT\" \\\n", " --output_type image \\\n", " --output_path \"$COSMOS3_FP8_OUTPUTS/super_t2i.png\"\n" ] From ec76cad2c24e8e7183492eae16574db529e493ab Mon Sep 17 00:00:00 2001 From: Igor Shovkun Date: Mon, 5 Oct 2026 11:02:42 -0700 Subject: [PATCH 6/8] Update FP8 TensorRT-LLM cookbook for merged static-FP8 support Use TensorRT-LLM main (static-FP8 #17476), cosmos_guardrail 0.3.2 without the NLTK workaround, --revision fp8 loading, and the shared audiovisual prompts and reference media. --- cookbooks/cosmos3/README.md | 33 +++-- .../cosmos3/generator/audiovisual/README.md | 7 +- .../audiovisual/run_fp8_with_trt_llm.ipynb | 132 +++++++++++------- 3 files changed, 103 insertions(+), 69 deletions(-) diff --git a/cookbooks/cosmos3/README.md b/cookbooks/cosmos3/README.md index 163c84ff..322047fa 100644 --- a/cookbooks/cosmos3/README.md +++ b/cookbooks/cosmos3/README.md @@ -291,19 +291,26 @@ video-content classification or suppress that warning. Cosmos3 Nano and Super publish ModelOpt-calibrated checkpoints on the `fp8` revision of their Hugging Face repositories. These revisions contain the FP8 weights, activation scales, and runtime policy. TensorRT-LLM reads that metadata -directly, so no quantization flag is required. - -```bash -hf download nvidia/Cosmos3-Nano \ - --revision fp8 \ - --local-dir checkpoints/Cosmos3-Nano-FP8 -hf download nvidia/Cosmos3-Super \ - --revision fp8 \ - --local-dir checkpoints/Cosmos3-Super-FP8 -``` - -These FP8 checkpoints support one GPU only. Use the BF16 checkpoints for tensor, -Ulysses, context, or CFG parallelism, or for parallel VAE. The +directly, so no quantization flag is required. Static-FP8 loading +([#17476](https://github.com/NVIDIA/TensorRT-LLM/pull/17476)) is merged on +TensorRT-LLM `main`; use a checkout at or after it rather than the older revision +pinned above. Select the FP8 weights with `--revision fp8`; TensorRT-LLM +downloads them on first use: + +```bash +python3 examples/visual_gen/models/cosmos3/cosmos3.py \ + --model nvidia/Cosmos3-Nano --revision fp8 \ + --visual_gen_args examples/visual_gen/configs/cosmos3-nano-1gpu.yaml \ + --prompt_file examples/visual_gen/models/cosmos3/prompts/t2v.json \ + --output_path nano_fp8_t2v.mp4 +``` + +Install `cosmos_guardrail==0.3.2` (not the `0.3.0` pin shown above) for these +offline runs; it needs no NLTK data setup. These FP8 checkpoints support one GPU +only, and the weights are about 21 GB for Nano and 70 GB for Super, so Super needs +a single GPU with enough memory for 70 GB of weights plus activations. Use the +BF16 checkpoints for tensor, Ulysses, context, or CFG parallelism, or for +parallel VAE. The [FP8 checkpoint notebook](generator/audiovisual/run_fp8_with_trt_llm.ipynb) runs Nano and Super through the offline TensorRT-LLM entry point for text-to-image, text-to-video, image-to-video, and video-to-video generation and diff --git a/cookbooks/cosmos3/generator/audiovisual/README.md b/cookbooks/cosmos3/generator/audiovisual/README.md index 3e77aeab..2f1f9b8d 100644 --- a/cookbooks/cosmos3/generator/audiovisual/README.md +++ b/cookbooks/cosmos3/generator/audiovisual/README.md @@ -424,10 +424,11 @@ live in the ### Cosmos3 Nano and Super FP8 notebook [`run_fp8_with_trt_llm.ipynb`](./run_fp8_with_trt_llm.ipynb) -runs local, ModelOpt-calibrated Cosmos3 Nano and Super FP8 checkpoints directly -through TensorRT-LLM's offline Cosmos3 entry point. It contains the complete +runs the ModelOpt-calibrated Cosmos3 Nano and Super FP8 checkpoints (the `fp8` +revision on Hugging Face) directly through TensorRT-LLM's offline Cosmos3 entry point. It contains the complete single-GPU commands for text-to-image, text-to-video, image-to-video, and -video-to-video generation, followed by PNG and MP4 validation. The checkpoint +video-to-video generation on this folder's shared prompts and reference media, +followed by PNG and MP4 validation. Super needs a Blackwell-class single GPU. The checkpoint metadata supplies the calibrated weight and activation scales; no runtime quantization flag is required. diff --git a/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb b/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb index d8134550..95221c51 100644 --- a/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb +++ b/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb @@ -33,9 +33,13 @@ "\n", "Build and install TensorRT-LLM from its `main` branch by following the\n", "[TensorRT-LLM source-build guide](https://nvidia.github.io/TensorRT-LLM/installation/build-from-source.html).\n", - "The source checkout supplies the Cosmos3 example and one-GPU configuration\n", - "files used below. The structured T2I prompt comes from this cookbook's shared\n", - "audiovisual assets.\n", + "Static-FP8 Cosmos3 loading\n", + "([#17476](https://github.com/NVIDIA/TensorRT-LLM/pull/17476)) is merged on\n", + "`main`, so a checkout at or after that change is required. The source checkout\n", + "supplies the Cosmos3 example script and one-GPU configuration files used\n", + "below. Prompts, negative prompts, the reference image, and the reference video\n", + "come from this cookbook's shared audiovisual assets, so the same inputs run\n", + "through every other backend in this folder.\n", "\n", "Install `ffmpeg` so TensorRT-LLM can encode MP4 output. Install the Cosmos3\n", "guardrail package, accept the gated\n", @@ -44,28 +48,25 @@ "\n", "```shell\n", "apt-get update && apt-get install -y ffmpeg\n", - "pip install cosmos_guardrail==0.3.0\n", + "pip install cosmos_guardrail==0.3.2\n", "pip uninstall -y opencv-python\n", "pip install opencv-python-headless\n", "hf auth login\n", "```\n", "\n", - "The pinned guardrail package performs prompt text checks and RetinaFace face\n", - "blurring. Its generated-frame content classifier is disabled upstream because\n", - "it produced too many false positives.\n", + "TensorRT-LLM downloads only the guardrail files it loads, and no NLTK data\n", + "workaround is needed. The guardrail package performs prompt text checks and\n", + "RetinaFace face blurring. Its generated-frame content classifier is disabled\n", + "upstream because it produced too many false positives.\n", "\n", "Nano and Super publish their ModelOpt FP8 checkpoints on the `fp8` revision of\n", - "their Hugging Face repositories. Download each revision to a local directory;\n", - "TensorRT-LLM receives that directory through `--model`:\n", - "\n", - "```shell\n", - "hf download nvidia/Cosmos3-Nano \\\n", - " --revision fp8 \\\n", - " --local-dir checkpoints/Cosmos3-Nano-FP8\n", - "hf download nvidia/Cosmos3-Super \\\n", - " --revision fp8 \\\n", - " --local-dir checkpoints/Cosmos3-Super-FP8\n", - "```\n", + "their Hugging Face repositories, next to the BF16 weights. The commands below\n", + "pass `--revision fp8` so TensorRT-LLM downloads that revision on first use. The\n", + "weights are about 21 GB for Nano and 70 GB for Super, and the weights cannot be\n", + "split across GPUs, so Super needs a single GPU with enough memory for 70 GB of\n", + "weights plus activations, such as a B200 or B300. To use a local copy instead,\n", + "`hf download` the `fp8` revision and point `COSMOS3_NANO_FP8` or\n", + "`COSMOS3_SUPER_FP8` at that directory; `--revision` is ignored for local paths.\n", "\n", "Set the paths used by the commands in this notebook. You may export these\n", "variables before starting Jupyter or replace the placeholder defaults below:\n", @@ -91,42 +92,46 @@ "\n", "\n", "COSMOS_ROOT = find_repo_root(Path.cwd().resolve())\n", - "COSMOS3_AUDIOVISUAL_ROOT = (\n", - " COSMOS_ROOT / \"cookbooks\" / \"cosmos3\" / \"generator\" / \"audiovisual\"\n", - ")\n", - "COSMOS3_T2I_PROMPT = (\n", - " COSMOS3_AUDIOVISUAL_ROOT / \"assets\" / \"prompts\" / \"text2image\" / \"robot_draping.json\"\n", - ")\n", + "ASSETS = COSMOS_ROOT / \"cookbooks\" / \"cosmos3\" / \"generator\" / \"audiovisual\" / \"assets\"\n", + "COSMOS3_T2I_PROMPT = ASSETS / \"prompts\" / \"text2image\" / \"robot_draping.json\"\n", + "COSMOS3_T2V_PROMPT = ASSETS / \"prompts\" / \"text2video\" / \"robot_kitchen.json\"\n", + "COSMOS3_T2V_NEGATIVE = ASSETS / \"negative_prompts\" / \"text2video\" / \"neg_prompt.json\"\n", + "COSMOS3_I2V_PROMPT = ASSETS / \"prompts\" / \"image2video\" / \"car_driving.json\"\n", + "COSMOS3_I2V_NEGATIVE = ASSETS / \"negative_prompts\" / \"image2video\" / \"neg_prompt.json\"\n", + "COSMOS3_I2V_IMAGE = ASSETS / \"images\" / \"image2video\" / \"car_driving.jpg\"\n", + "COSMOS3_V2V_VIDEO = ASSETS / \"videos\" / \"car_driving_plain.mp4\"\n", + "\n", "TRTLLM_ROOT = Path(os.environ.get(\"TRTLLM_ROOT\", \"/path/to/TensorRT-LLM\")).expanduser().resolve()\n", - "COSMOS3_NANO_FP8 = Path(\n", - " os.environ.get(\"COSMOS3_NANO_FP8\", Path.cwd() / \"checkpoints\" / \"Cosmos3-Nano-FP8\")\n", - ").expanduser().resolve()\n", - "COSMOS3_SUPER_FP8 = Path(\n", - " os.environ.get(\"COSMOS3_SUPER_FP8\", Path.cwd() / \"checkpoints\" / \"Cosmos3-Super-FP8\")\n", - ").expanduser().resolve()\n", + "# A Hugging Face repo ID (loaded at --revision fp8) or a local copy of the fp8 revision.\n", + "COSMOS3_NANO_FP8 = os.environ.get(\"COSMOS3_NANO_FP8\", \"nvidia/Cosmos3-Nano\")\n", + "COSMOS3_SUPER_FP8 = os.environ.get(\"COSMOS3_SUPER_FP8\", \"nvidia/Cosmos3-Super\")\n", "COSMOS3_FP8_OUTPUTS = Path(\n", " os.environ.get(\"COSMOS3_FP8_OUTPUTS\", Path.cwd() / \"outputs\" / \"tensorrt_llm_fp8\")\n", ").expanduser().resolve()\n", "\n", - "for name, path in {\n", + "inputs = {\n", " \"TRTLLM_ROOT\": TRTLLM_ROOT,\n", - " \"COSMOS3_NANO_FP8\": COSMOS3_NANO_FP8,\n", - " \"COSMOS3_SUPER_FP8\": COSMOS3_SUPER_FP8,\n", " \"COSMOS3_T2I_PROMPT\": COSMOS3_T2I_PROMPT,\n", - "}.items():\n", + " \"COSMOS3_T2V_PROMPT\": COSMOS3_T2V_PROMPT,\n", + " \"COSMOS3_T2V_NEGATIVE\": COSMOS3_T2V_NEGATIVE,\n", + " \"COSMOS3_I2V_PROMPT\": COSMOS3_I2V_PROMPT,\n", + " \"COSMOS3_I2V_NEGATIVE\": COSMOS3_I2V_NEGATIVE,\n", + " \"COSMOS3_I2V_IMAGE\": COSMOS3_I2V_IMAGE,\n", + " \"COSMOS3_V2V_VIDEO\": COSMOS3_V2V_VIDEO,\n", + "}\n", + "for name, path in inputs.items():\n", " if not path.exists():\n", " raise FileNotFoundError(f\"Set {name} to an existing path; got {path}\")\n", "\n", "COSMOS3_FP8_OUTPUTS.mkdir(parents=True, exist_ok=True)\n", - "for name, path in {\n", - " \"TRTLLM_ROOT\": TRTLLM_ROOT,\n", + "for name, value in {\n", + " **inputs,\n", " \"COSMOS3_NANO_FP8\": COSMOS3_NANO_FP8,\n", " \"COSMOS3_SUPER_FP8\": COSMOS3_SUPER_FP8,\n", - " \"COSMOS3_T2I_PROMPT\": COSMOS3_T2I_PROMPT,\n", " \"COSMOS3_FP8_OUTPUTS\": COSMOS3_FP8_OUTPUTS,\n", "}.items():\n", - " os.environ[name] = str(path)\n", - " print(f\"{name}={path}\")\n" + " os.environ[name] = str(value)\n", + " print(f\"{name}={value}\")" ] }, { @@ -145,7 +150,8 @@ "\n", "The text-to-image config warms the 1024x1024, one-frame shape. The cookbook's\n", "structured T2I prompt matches the checkpoint's trained prompt format, and\n", - "`--output_type image` makes the expected output explicit.\n", + "`--output_type image` makes the expected output explicit. `--revision fp8`\n", + "selects the FP8 weights.\n", "\n" ] }, @@ -161,6 +167,7 @@ "\n", "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", " --model \"$COSMOS3_NANO_FP8\" \\\n", + " --revision fp8 \\\n", " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-t2i-1gpu.yaml\" \\\n", " --prompt_file \"$COSMOS3_T2I_PROMPT\" \\\n", " --output_type image \\\n", @@ -189,8 +196,10 @@ "\n", "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", " --model \"$COSMOS3_NANO_FP8\" \\\n", + " --revision fp8 \\\n", " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", - " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/t2v.json\" \\\n", + " --prompt_file \"$COSMOS3_T2V_PROMPT\" \\\n", + " --negative_prompt_file \"$COSMOS3_T2V_NEGATIVE\" \\\n", " --output_path \"$COSMOS3_FP8_OUTPUTS/nano_t2v.mp4\"\n" ] }, @@ -202,9 +211,8 @@ "\n", "### Image to video\n", "\n", - "The example I2V prompt names its reference image with an HTTPS URL. Use\n", - "`--image_path` to replace it with a local path, `file://` URL, HTTP(S) URL, or\n", - "`data:` URI.\n", + "`--image_path` supplies the reference image as a local path, `file://` URL,\n", + "HTTP(S) URL, or `data:` URI.\n", "\n" ] }, @@ -220,8 +228,11 @@ "\n", "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", " --model \"$COSMOS3_NANO_FP8\" \\\n", + " --revision fp8 \\\n", " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", - " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/i2v.json\" \\\n", + " --prompt_file \"$COSMOS3_I2V_PROMPT\" \\\n", + " --negative_prompt_file \"$COSMOS3_I2V_NEGATIVE\" \\\n", + " --image_path \"$COSMOS3_I2V_IMAGE\" \\\n", " --output_path \"$COSMOS3_FP8_OUTPUTS/nano_i2v.mp4\"\n" ] }, @@ -233,8 +244,9 @@ "\n", "### Video to video\n", "\n", - "This command uses the Nano T2V output above as the reference video. By default,\n", - "the first five decoded pixel frames condition the output.\n", + "This command conditions on the cookbook's reference driving clip. Latent\n", + "indexes `[0, 1]` with `\"first\"` keep the first five decoded pixel frames as\n", + "clean conditioning, matching the other backends in this folder.\n", "\n" ] }, @@ -250,9 +262,13 @@ "\n", "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", " --model \"$COSMOS3_NANO_FP8\" \\\n", + " --revision fp8 \\\n", " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", - " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/v2v.json\" \\\n", - " --video_path \"$COSMOS3_FP8_OUTPUTS/nano_t2v.mp4\" \\\n", + " --prompt_file \"$COSMOS3_I2V_PROMPT\" \\\n", + " --negative_prompt_file \"$COSMOS3_I2V_NEGATIVE\" \\\n", + " --use_system_prompt \\\n", + " --video_path \"$COSMOS3_V2V_VIDEO\" \\\n", + " --extra_params '{\"condition_video_latent_indexes\": [0, 1], \"condition_video_keep\": \"first\"}' \\\n", " --output_path \"$COSMOS3_FP8_OUTPUTS/nano_v2v.mp4\"\n" ] }, @@ -284,6 +300,7 @@ "\n", "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", " --model \"$COSMOS3_SUPER_FP8\" \\\n", + " --revision fp8 \\\n", " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-t2i-1gpu.yaml\" \\\n", " --prompt_file \"$COSMOS3_T2I_PROMPT\" \\\n", " --output_type image \\\n", @@ -315,8 +332,10 @@ "\n", "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", " --model \"$COSMOS3_SUPER_FP8\" \\\n", + " --revision fp8 \\\n", " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", - " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/t2v.json\" \\\n", + " --prompt_file \"$COSMOS3_T2V_PROMPT\" \\\n", + " --negative_prompt_file \"$COSMOS3_T2V_NEGATIVE\" \\\n", " --output_path \"$COSMOS3_FP8_OUTPUTS/super_t2v.mp4\"\n" ] }, @@ -342,8 +361,11 @@ "\n", "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", " --model \"$COSMOS3_SUPER_FP8\" \\\n", + " --revision fp8 \\\n", " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", - " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/i2v.json\" \\\n", + " --prompt_file \"$COSMOS3_I2V_PROMPT\" \\\n", + " --negative_prompt_file \"$COSMOS3_I2V_NEGATIVE\" \\\n", + " --image_path \"$COSMOS3_I2V_IMAGE\" \\\n", " --output_path \"$COSMOS3_FP8_OUTPUTS/super_i2v.mp4\"\n" ] }, @@ -369,9 +391,13 @@ "\n", "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", " --model \"$COSMOS3_SUPER_FP8\" \\\n", + " --revision fp8 \\\n", " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", - " --prompt_file \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/prompts/v2v.json\" \\\n", - " --video_path \"$COSMOS3_FP8_OUTPUTS/super_t2v.mp4\" \\\n", + " --prompt_file \"$COSMOS3_I2V_PROMPT\" \\\n", + " --negative_prompt_file \"$COSMOS3_I2V_NEGATIVE\" \\\n", + " --use_system_prompt \\\n", + " --video_path \"$COSMOS3_V2V_VIDEO\" \\\n", + " --extra_params '{\"condition_video_latent_indexes\": [0, 1], \"condition_video_keep\": \"first\"}' \\\n", " --output_path \"$COSMOS3_FP8_OUTPUTS/super_v2v.mp4\"\n" ] }, From 578458eec440e2c0ac856507db06203d66d778c6 Mon Sep 17 00:00:00 2001 From: Igor Shovkun Date: Tue, 6 Oct 2026 12:20:59 -0700 Subject: [PATCH 7/8] Add FP8 Reasoner section to the TensorRT-LLM FP8 cookbook Serve the fp8 revision through trtllm-serve --hf_revision fp8 and send image and video understanding requests to Nano and Super. --- .../audiovisual/run_fp8_with_trt_llm.ipynb | 191 +++++++++++++++++- 1 file changed, 188 insertions(+), 3 deletions(-) diff --git a/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb b/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb index 95221c51..8d7005d9 100644 --- a/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb +++ b/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb @@ -15,7 +15,7 @@ "and activation scales, so no quantization CLI flag is\n", "needed.\n", "\n", - "This notebook covers the four validated single-GPU modes:\n", + "This notebook covers the four single-GPU generation modes and the Reasoner:\n", "\n", "| Mode | Prompt or condition | Output |\n", "| -------------------- | ------------------------------------- | --------- |\n", @@ -23,6 +23,7 @@ "| Text to video (T2V) | Structured text prompt | MP4 video |\n", "| Image to video (I2V) | Structured prompt and reference image | MP4 video |\n", "| Video to video (V2V) | Structured prompt and reference video | MP4 video |\n", + "| Reasoner | Image or video plus a question | Text |\n", "\n", "FP8 Cosmos3 is single-GPU only. TensorRT-LLM rejects tensor parallelism,\n", "Ulysses parallelism, context parallelism, CFG parallelism, or parallel VAE\n", @@ -478,15 +479,199 @@ "source": [ "\n", "A successful encode/decode proves the interface and artifact contract; it is\n", - "not a claim that FP8 output quality matches BF16.\n", + "not a claim that FP8 output quality matches BF16.\n" + ] + }, + { + "cell_type": "markdown", + "id": "markdown-23-reasoner-intro", + "metadata": {}, + "source": [ + "## 5. Reasoner: image and video understanding\n", + "\n", + "The same `fp8` revision also serves the Cosmos3 Reasoner through `trtllm-serve`,\n", + "TensorRT-LLM's OpenAI-compatible LLM server. It loads the language and vision\n", + "towers only: the text-layer linear layers run in FP8 with the checkpoint's\n", + "static scales, while the vision tower, embeddings, and output head stay in\n", + "BF16. The generator's diffusion step policy does not apply to this path.\n", + "\n", + "The helpers below start `trtllm-serve` in the background with\n", + "`--hf_revision fp8`, wait for `/health`, send image and video requests with the\n", + "`openai` client, and stop the server. The server listens on port 8001, the same\n", + "endpoint as the [Reasoner notebook](../../reasoner/run_with_tensorrt_llm.ipynb),\n", + "so every example there also runs against it. The first start downloads the\n", + "weights and can take several minutes.\n", + "\n", + "If `trtllm-serve` stalls at `worker_initialization_wait` on a Slurm compute\n", + "node, MPI cannot spawn its worker process there. Set\n", + "`COSMOS3_TRTLLM_LAUNCHER=\"mpirun -np 1 --bind-to none trtllm-llmapi-launch\"`\n", + "before starting Jupyter to launch the server under `mpirun` instead.\n" + ] + }, + { + "cell_type": "code", + "id": "python-24-reasoner-helpers", + "metadata": {}, + "source": [ + "import shlex\n", + "import subprocess\n", + "import time\n", + "import urllib.request\n", + "\n", + "import openai\n", + "\n", + "REASONER_ASSETS = COSMOS_ROOT / \"cookbooks\" / \"cosmos3\" / \"reasoner\" / \"assets\"\n", + "REASONER_PORT = 8001\n", + "REASONER_URL = f\"http://localhost:{REASONER_PORT}\"\n", + "VIDEO_KWARGS = {\n", + " \"media_io_kwargs\": {\"video\": {\"num_frames\": -1, \"fps\": 4}},\n", + " \"mm_processor_kwargs\": {\"do_sample_frames\": False},\n", + "}\n", + "# Optional launcher prefix. Leave unset on a workstation or in a container. On a\n", + "# Slurm node where MPI cannot spawn worker processes, set for example:\n", + "# COSMOS3_TRTLLM_LAUNCHER=\"mpirun -np 1 --bind-to none trtllm-llmapi-launch\"\n", + "REASONER_LAUNCHER = shlex.split(os.environ.get(\"COSMOS3_TRTLLM_LAUNCHER\", \"\"))\n", + "reasoner_server = None\n", + "\n", + "\n", + "def start_reasoner(model: str, timeout_s: int = 3600) -> None:\n", + " \"\"\"Start trtllm-serve for the FP8 Reasoner and wait until /health answers.\"\"\"\n", + " global reasoner_server\n", + " command = REASONER_LAUNCHER + [\n", + " \"trtllm-serve\", model,\n", + " \"--host\", \"127.0.0.1\", \"--port\", str(REASONER_PORT),\n", + " \"--max_num_tokens\", \"32768\",\n", + " ]\n", + " if not Path(model).exists():\n", + " command += [\"--hf_revision\", \"fp8\"] # Hub repo ID: select the FP8 branch\n", + " log_path = COSMOS3_FP8_OUTPUTS / \"reasoner_server.log\"\n", + " reasoner_server = subprocess.Popen(\n", + " command, stdout=log_path.open(\"w\"), stderr=subprocess.STDOUT\n", + " )\n", + " print(\" \".join(command))\n", + " deadline = time.time() + timeout_s\n", + " while time.time() < deadline:\n", + " if reasoner_server.poll() is not None:\n", + " raise RuntimeError(f\"trtllm-serve exited early; see {log_path}\")\n", + " try:\n", + " with urllib.request.urlopen(f\"{REASONER_URL}/health\", timeout=5):\n", + " print(\"Reasoner server ready\")\n", + " return\n", + " except OSError:\n", + " time.sleep(5)\n", + " stop_reasoner()\n", + " raise TimeoutError(f\"trtllm-serve was not ready after {timeout_s} s; see {log_path}\")\n", + "\n", + "\n", + "def stop_reasoner() -> None:\n", + " if reasoner_server is not None and reasoner_server.poll() is None:\n", + " reasoner_server.terminate()\n", + " try:\n", + " reasoner_server.wait(timeout=120)\n", + " except subprocess.TimeoutExpired:\n", + " reasoner_server.kill()\n", + "\n", + "\n", + "def ask(media: str, prompt: str) -> str:\n", + " client = openai.OpenAI(api_key=\"EMPTY\", base_url=f\"{REASONER_URL}/v1\")\n", + " kind = \"video_url\" if media.endswith(\".mp4\") else \"image_url\"\n", + " response = client.chat.completions.create(\n", + " model=client.models.list().data[0].id,\n", + " messages=[{\n", + " \"role\": \"user\",\n", + " \"content\": [\n", + " {\"type\": kind, kind: {\"url\": (REASONER_ASSETS / media).resolve().as_uri()}},\n", + " {\"type\": \"text\", \"text\": prompt},\n", + " ],\n", + " }],\n", + " max_tokens=4096,\n", + " seed=0,\n", + " extra_body=VIDEO_KWARGS if kind == \"video_url\" else {},\n", + " )\n", + " answer = response.choices[0].message.content\n", + " assert answer and answer.strip(), f\"empty answer for {media}\"\n", + " return answer\n", + "\n", + "\n", + "REASONER_EXAMPLES = [\n", + " (\"robot_153.jpg\", \"Caption the image in detail.\"),\n", + " (\"video_caption.mp4\", \"Describe the video in detail.\"),\n", + " (\"grounding_2d.png\", \"Locate the accurate bounding box of the load as a whole. Return a json.\"),\n", + "]\n", + "\n", + "\n", + "def run_reasoner_examples() -> None:\n", + " for media, prompt in REASONER_EXAMPLES:\n", + " print(f\"### {media}: {prompt}\")\n", + " print(ask(media, prompt), end=\"\\n\\n\")" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "markdown-25-reasoner-nano", + "metadata": {}, + "source": [ + "### Cosmos3 Nano FP8 Reasoner\n" + ] + }, + { + "cell_type": "code", + "id": "python-26-reasoner-nano", + "metadata": {}, + "source": [ + "start_reasoner(COSMOS3_NANO_FP8)\n", + "try:\n", + " run_reasoner_examples()\n", + "finally:\n", + " stop_reasoner()" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "markdown-27-reasoner-super", + "metadata": {}, + "source": [ + "### Cosmos3 Super FP8 Reasoner\n", "\n", - "## 5. Scope\n", + "Super's FP8 language model fits one Blackwell-class GPU; no tensor parallelism\n", + "is needed.\n" + ] + }, + { + "cell_type": "code", + "id": "python-28-reasoner-super", + "metadata": {}, + "source": [ + "start_reasoner(COSMOS3_SUPER_FP8)\n", + "try:\n", + " run_reasoner_examples()\n", + "finally:\n", + " stop_reasoner()" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "markdown-29-scope", + "metadata": {}, + "source": [ + "## 6. Scope\n", "\n", "The Nano and Super FP8 checkpoints also contain an audio tower, so T2AV and\n", "TI2AV requests run rather than being refused. Audio has not received the same\n", "FP8 exercise as the four image/video modes above, and no FP8 audio-quality\n", "claim is made here.\n", "\n", + "The Reasoner section exercises three representative requests per model; the\n", + "[Reasoner notebook](../../reasoner/run_with_tensorrt_llm.ipynb) has the full set\n", + "of image and video understanding examples, which run unchanged against the FP8\n", + "server.\n", + "\n", "For the complete TensorRT-LLM Cosmos3 example contract, including serving,\n", "guardrails, media dependencies, and other checkpoint families, see the\n", "[TensorRT-LLM Cosmos3 example](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/visual_gen/models/cosmos3).\n" From c2e057162f6b2c56ec4319e691739d362c620eee Mon Sep 17 00:00:00 2001 From: Igor Shovkun Date: Tue, 6 Oct 2026 12:42:36 -0700 Subject: [PATCH 8/8] Add FP8 audio generation and Reasoner serving docs Add text-to-video and image-to-video with audio for Nano and Super, check the audio streams, and document serving the FP8 Reasoner with trtllm-serve --hf_revision fp8. --- cookbooks/cosmos3/README.md | 28 +++- .../cosmos3/generator/audiovisual/README.md | 5 +- .../audiovisual/run_fp8_with_trt_llm.ipynb | 154 +++++++++++++++++- 3 files changed, 176 insertions(+), 11 deletions(-) diff --git a/cookbooks/cosmos3/README.md b/cookbooks/cosmos3/README.md index 322047fa..bd95381f 100644 --- a/cookbooks/cosmos3/README.md +++ b/cookbooks/cosmos3/README.md @@ -313,8 +313,10 @@ BF16 checkpoints for tensor, Ulysses, context, or CFG parallelism, or for parallel VAE. The [FP8 checkpoint notebook](generator/audiovisual/run_fp8_with_trt_llm.ipynb) runs Nano and Super through the offline TensorRT-LLM entry point for -text-to-image, text-to-video, image-to-video, and video-to-video generation and -then validates every PNG and MP4 artifact. +text-to-image, text-to-video, image-to-video, and video-to-video generation, +text-to-video and image-to-video with synchronized audio, validates every PNG +and MP4 artifact, and serves the FP8 Reasoner for image and +video understanding. ### VisualGen server @@ -481,6 +483,28 @@ The server exposes `/health` and the OpenAI-compatible API at [Reasoner TensorRT-LLM notebook](reasoner/run_with_tensorrt_llm.ipynb) for image and video requests. +**Cosmos3-Nano or Cosmos3-Super FP8** (single GPU, port 8001). The `fp8` +revision carries ModelOpt-calibrated weights and static activation scales; +`trtllm-serve` reads them from the checkpoint, so no quantization flag is +needed. This needs a TensorRT-LLM build from `main`, as described in the +[FP8 checkpoint section](#cosmos3-nano-and-super-fp8-checkpoints-offline-single-gpu), +and Super FP8 fits on one Blackwell-class GPU: + +```bash +CUDA_VISIBLE_DEVICES=0 \ +trtllm-serve nvidia/Cosmos3-Nano \ + --hf_revision fp8 \ + --host 0.0.0.0 \ + --port 8001 \ + --max_num_tokens 32768 +``` + +Replace `nvidia/Cosmos3-Nano` with `nvidia/Cosmos3-Super` for Super. On a Slurm +compute node where `trtllm-serve` stalls at `worker_initialization_wait`, +prefix the command with `mpirun -np 1 --bind-to none trtllm-llmapi-launch`. +The [FP8 checkpoint notebook](generator/audiovisual/run_fp8_with_trt_llm.ipynb) +starts this server and sends image and video requests to both models. + ## Transformers Local Python inference for the Cosmos3 Reasoner. This backend uses the diff --git a/cookbooks/cosmos3/generator/audiovisual/README.md b/cookbooks/cosmos3/generator/audiovisual/README.md index 2f1f9b8d..ae9ef3ce 100644 --- a/cookbooks/cosmos3/generator/audiovisual/README.md +++ b/cookbooks/cosmos3/generator/audiovisual/README.md @@ -427,8 +427,9 @@ live in the runs the ModelOpt-calibrated Cosmos3 Nano and Super FP8 checkpoints (the `fp8` revision on Hugging Face) directly through TensorRT-LLM's offline Cosmos3 entry point. It contains the complete single-GPU commands for text-to-image, text-to-video, image-to-video, and -video-to-video generation on this folder's shared prompts and reference media, -followed by PNG and MP4 validation. Super needs a Blackwell-class single GPU. The checkpoint +video-to-video generation, text-to-video and image-to-video with audio, on this folder's shared prompts and reference media, +followed by PNG and MP4 validation, plus image and video understanding through +the FP8 Reasoner served with `trtllm-serve`. Super needs a Blackwell-class single GPU. The checkpoint metadata supplies the calibrated weight and activation scales; no runtime quantization flag is required. diff --git a/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb b/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb index 8d7005d9..3a314dc2 100644 --- a/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb +++ b/cookbooks/cosmos3/generator/audiovisual/run_fp8_with_trt_llm.ipynb @@ -15,7 +15,8 @@ "and activation scales, so no quantization CLI flag is\n", "needed.\n", "\n", - "This notebook covers the four single-GPU generation modes and the Reasoner:\n", + "This notebook covers the single-GPU generation modes, with and without audio,\n", + "and the Reasoner:\n", "\n", "| Mode | Prompt or condition | Output |\n", "| -------------------- | ------------------------------------- | --------- |\n", @@ -23,6 +24,8 @@ "| Text to video (T2V) | Structured text prompt | MP4 video |\n", "| Image to video (I2V) | Structured prompt and reference image | MP4 video |\n", "| Video to video (V2V) | Structured prompt and reference video | MP4 video |\n", + "| Text to video with audio (T2AV) | Structured prompt | MP4 video with stereo audio |\n", + "| Image to video with audio (TI2AV) | Structured prompt and reference image | MP4 video with stereo audio |\n", "| Reasoner | Image or video plus a question | Text |\n", "\n", "FP8 Cosmos3 is single-GPU only. TensorRT-LLM rejects tensor parallelism,\n", @@ -101,6 +104,9 @@ "COSMOS3_I2V_NEGATIVE = ASSETS / \"negative_prompts\" / \"image2video\" / \"neg_prompt.json\"\n", "COSMOS3_I2V_IMAGE = ASSETS / \"images\" / \"image2video\" / \"car_driving.jpg\"\n", "COSMOS3_V2V_VIDEO = ASSETS / \"videos\" / \"car_driving_plain.mp4\"\n", + "COSMOS3_T2AV_PROMPT = ASSETS / \"prompts\" / \"text2video\" / \"robot_pouring_water_audio.json\"\n", + "COSMOS3_TI2AV_PROMPT = ASSETS / \"prompts\" / \"image2video\" / \"coastal_road_audio.json\"\n", + "COSMOS3_TI2AV_IMAGE = ASSETS / \"images\" / \"image2video\" / \"coastal_road_audio.jpg\"\n", "\n", "TRTLLM_ROOT = Path(os.environ.get(\"TRTLLM_ROOT\", \"/path/to/TensorRT-LLM\")).expanduser().resolve()\n", "# A Hugging Face repo ID (loaded at --revision fp8) or a local copy of the fp8 revision.\n", @@ -119,6 +125,9 @@ " \"COSMOS3_I2V_NEGATIVE\": COSMOS3_I2V_NEGATIVE,\n", " \"COSMOS3_I2V_IMAGE\": COSMOS3_I2V_IMAGE,\n", " \"COSMOS3_V2V_VIDEO\": COSMOS3_V2V_VIDEO,\n", + " \"COSMOS3_T2AV_PROMPT\": COSMOS3_T2AV_PROMPT,\n", + " \"COSMOS3_TI2AV_PROMPT\": COSMOS3_TI2AV_PROMPT,\n", + " \"COSMOS3_TI2AV_IMAGE\": COSMOS3_TI2AV_IMAGE,\n", "}\n", "for name, path in inputs.items():\n", " if not path.exists():\n", @@ -273,6 +282,66 @@ " --output_path \"$COSMOS3_FP8_OUTPUTS/nano_v2v.mp4\"\n" ] }, + { + "cell_type": "markdown", + "id": "markdown-09a-t2av", + "metadata": {}, + "source": [ + "\n", + "### Text to video with audio\n", + "\n", + "`--enable_audio` generates a synchronized stereo audio track with the video.\n", + "The FP8 checkpoints carry the audio tower, so no other change is needed.\n" + ] + }, + { + "cell_type": "code", + "id": "bash-09a-t2av", + "metadata": {}, + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "\n", + "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", + " --model \"$COSMOS3_NANO_FP8\" \\\n", + " --revision fp8 \\\n", + " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", + " --prompt_file \"$COSMOS3_T2AV_PROMPT\" \\\n", + " --enable_audio \\\n", + " --output_path \"$COSMOS3_FP8_OUTPUTS/nano_t2av.mp4\"" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "markdown-09a-ti2av", + "metadata": {}, + "source": [ + "\n", + "### Image to video with audio\n" + ] + }, + { + "cell_type": "code", + "id": "bash-09a-ti2av", + "metadata": {}, + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "\n", + "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", + " --model \"$COSMOS3_NANO_FP8\" \\\n", + " --revision fp8 \\\n", + " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", + " --prompt_file \"$COSMOS3_TI2AV_PROMPT\" \\\n", + " --image_path \"$COSMOS3_TI2AV_IMAGE\" \\\n", + " --enable_audio \\\n", + " --output_path \"$COSMOS3_FP8_OUTPUTS/nano_ti2av.mp4\"" + ], + "execution_count": null, + "outputs": [] + }, { "cell_type": "markdown", "id": "markdown-10-e177f899", @@ -402,6 +471,66 @@ " --output_path \"$COSMOS3_FP8_OUTPUTS/super_v2v.mp4\"\n" ] }, + { + "cell_type": "markdown", + "id": "markdown-17a-t2av", + "metadata": {}, + "source": [ + "\n", + "### Text to video with audio\n", + "\n", + "`--enable_audio` generates a synchronized stereo audio track with the video.\n", + "The FP8 checkpoints carry the audio tower, so no other change is needed.\n" + ] + }, + { + "cell_type": "code", + "id": "bash-17a-t2av", + "metadata": {}, + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "\n", + "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", + " --model \"$COSMOS3_SUPER_FP8\" \\\n", + " --revision fp8 \\\n", + " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", + " --prompt_file \"$COSMOS3_T2AV_PROMPT\" \\\n", + " --enable_audio \\\n", + " --output_path \"$COSMOS3_FP8_OUTPUTS/super_t2av.mp4\"" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "markdown-17a-ti2av", + "metadata": {}, + "source": [ + "\n", + "### Image to video with audio\n" + ] + }, + { + "cell_type": "code", + "id": "bash-17a-ti2av", + "metadata": {}, + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "\n", + "python \"$TRTLLM_ROOT/examples/visual_gen/models/cosmos3/cosmos3.py\" \\\n", + " --model \"$COSMOS3_SUPER_FP8\" \\\n", + " --revision fp8 \\\n", + " --visual_gen_args \"$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml\" \\\n", + " --prompt_file \"$COSMOS3_TI2AV_PROMPT\" \\\n", + " --image_path \"$COSMOS3_TI2AV_IMAGE\" \\\n", + " --enable_audio \\\n", + " --output_path \"$COSMOS3_FP8_OUTPUTS/super_ti2av.mp4\"" + ], + "execution_count": null, + "outputs": [] + }, { "cell_type": "markdown", "id": "markdown-18-674ba7b9", @@ -443,7 +572,8 @@ "metadata": {}, "source": [ "\n", - "Inspect and fully decode every generated video stream:\n", + "Inspect and fully decode every generated MP4, and confirm that the audio\n", + "outputs carry an audio stream:\n", "\n" ] }, @@ -463,12 +593,24 @@ " \"$COSMOS3_FP8_OUTPUTS/nano_v2v.mp4\" \\\n", " \"$COSMOS3_FP8_OUTPUTS/super_t2v.mp4\" \\\n", " \"$COSMOS3_FP8_OUTPUTS/super_i2v.mp4\" \\\n", - " \"$COSMOS3_FP8_OUTPUTS/super_v2v.mp4\"; do\n", + " \"$COSMOS3_FP8_OUTPUTS/super_v2v.mp4\" \\\n", + " \"$COSMOS3_FP8_OUTPUTS/nano_t2av.mp4\" \\\n", + " \"$COSMOS3_FP8_OUTPUTS/nano_ti2av.mp4\" \\\n", + " \"$COSMOS3_FP8_OUTPUTS/super_t2av.mp4\" \\\n", + " \"$COSMOS3_FP8_OUTPUTS/super_ti2av.mp4\"; do\n", " ffprobe -v error \\\n", " -show_entries stream=codec_type,codec_name,width,height,r_frame_rate,nb_frames \\\n", " -of json \\\n", " \"$output\"\n", " ffmpeg -v error -i \"$output\" -f null -\n", + "done\n", + "\n", + "# The audio outputs must also carry a decodable audio stream.\n", + "for output in \"$COSMOS3_FP8_OUTPUTS\"/*_t2av.mp4 \"$COSMOS3_FP8_OUTPUTS\"/*_ti2av.mp4; do\n", + " audio=$(ffprobe -v error -select_streams a:0 \\\n", + " -show_entries stream=codec_name,channels,sample_rate,duration -of csv=p=0 \"$output\")\n", + " [ -n \"$audio\" ] || { echo \"no audio stream in $output\"; exit 1; }\n", + " echo \"$(basename \"$output\") audio: $audio\"\n", "done\n" ] }, @@ -662,10 +804,8 @@ "source": [ "## 6. Scope\n", "\n", - "The Nano and Super FP8 checkpoints also contain an audio tower, so T2AV and\n", - "TI2AV requests run rather than being refused. Audio has not received the same\n", - "FP8 exercise as the four image/video modes above, and no FP8 audio-quality\n", - "claim is made here.\n", + "The FP8 checkpoints run on one GPU only, and no FP8-versus-BF16 quality\n", + "benchmark, for video or audio, is claimed here.\n", "\n", "The Reasoner section exercises three representative requests per model; the\n", "[Reasoner notebook](../../reasoner/run_with_tensorrt_llm.ipynb) has the full set\n",