Skip to content

Commit 9aa98e5

Browse files
Add ModelOpt FP8 Cosmos Framework inference (NVIDIA#301)
The change describes how to run FP8 ModelOpt checkpoints using cosmos-framwork It works for Cosmos3 Nano, Super, Super T2I, Super I2V, Super T2I 4steps, Super I2V 4steps --------- Signed-off-by: Krzysztof Tomala <ktomala@nvidia.com>
1 parent e7ad5e7 commit 9aa98e5

2 files changed

Lines changed: 155 additions & 0 deletions

File tree

‎cookbooks/cosmos3/generator/audiovisual/run_with_cosmos_framework.ipynb‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -624,6 +624,85 @@
624624
"execution_count": null,
625625
"outputs": []
626626
},
627+
{
628+
"cell_type": "markdown",
629+
"metadata": {},
630+
"source": [
631+
"## Nano FP8: Text to Image\n",
632+
"\n",
633+
"FP8 checkpoints are published in the same Hugging Face repositories as the standard checkpoints, under the `fp8` revision rather than in separate FP8 repositories. This example downloads `nvidia/Cosmos3-Nano` at revision `fp8`. To use another released FP8 checkpoint, change `FP8_MODEL_ID` and `FP8_CHECKPOINT` below; keep `FP8_REVISION=fp8`.\n",
634+
"\n",
635+
"### Create Payload"
636+
],
637+
"id": "fe1b922e"
638+
},
639+
{
640+
"cell_type": "code",
641+
"metadata": {},
642+
"source": [
643+
"t2i_fp8_payload, t2i_fp8_output, t2i_fp8_model = create_payload(\"t2i\", backend=\"pytorch_fp8\")"
644+
],
645+
"execution_count": null,
646+
"outputs": [],
647+
"id": "73ff907c"
648+
},
649+
{
650+
"cell_type": "markdown",
651+
"metadata": {},
652+
"source": [
653+
"### Run"
654+
],
655+
"id": "aaf87ed7"
656+
},
657+
{
658+
"cell_type": "code",
659+
"metadata": {},
660+
"source": [
661+
"%%bash\n",
662+
"set -euo pipefail\n",
663+
"\n",
664+
"cd \"$COSMOS3_REPO\"\n",
665+
"FP8_MODEL_ID=\"nvidia/Cosmos3-Nano\"\n",
666+
"FP8_REVISION=\"fp8\"\n",
667+
"FP8_CHECKPOINT=\"$COSMOS3_REPO/checkpoints/Cosmos3-Nano-FP8\"\n",
668+
"uvx hf@latest download \"$FP8_MODEL_ID\" \\\n",
669+
" --revision \"$FP8_REVISION\" \\\n",
670+
" --local-dir \"$FP8_CHECKPOINT\"\n",
671+
"\n",
672+
"CUDA_VISIBLE_DEVICES=\"$CUDA_VISIBLE_DEVICES\" LD_LIBRARY_PATH= \\\n",
673+
"\"$COSMOS3_UV_ENV/bin/torchrun\" \\\n",
674+
" --nproc-per-node=\"$COSMOS3_NUM_GPUS\" \\\n",
675+
" --master-addr=\"$COSMOS3_MASTER_ADDR\" \\\n",
676+
" --master-port=\"$COSMOS3_TEXT_MASTER_PORT\" \\\n",
677+
" -m cosmos_framework.scripts.inference \\\n",
678+
" --parallelism-preset=throughput \\\n",
679+
" -i \"$COSMOS3_PYTORCH_FP8_T2I_INPUT\" \\\n",
680+
" -o \"$COSMOS3_PYTORCH_FP8_T2I_OUTPUT\" \\\n",
681+
" --checkpoint-path \"$FP8_CHECKPOINT\" \\\n",
682+
" --seed=0\n"
683+
],
684+
"execution_count": null,
685+
"outputs": [],
686+
"id": "e0cbe358"
687+
},
688+
{
689+
"cell_type": "markdown",
690+
"metadata": {},
691+
"source": [
692+
"### View Results"
693+
],
694+
"id": "03e5c4b5"
695+
},
696+
{
697+
"cell_type": "code",
698+
"metadata": {},
699+
"source": [
700+
"view_run(t2i_fp8_output)"
701+
],
702+
"execution_count": null,
703+
"outputs": [],
704+
"id": "8970833a"
705+
},
627706
{
628707
"cell_type": "markdown",
629708
"metadata": {},

‎cookbooks/cosmos3/reasoner/run_with_cosmos_framework.ipynb‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -639,6 +639,82 @@
639639
" print(json.dumps(json.loads(benchmark_path.read_text()).get(\"average\", {}), indent=2))\n"
640640
]
641641
},
642+
{
643+
"cell_type": "markdown",
644+
"id": "da7c0e0a",
645+
"metadata": {},
646+
"source": [
647+
"### 7a. Optional: Run Nano FP8 Text Inference\n",
648+
"\n",
649+
"FP8 checkpoints are published in the same Hugging Face repositories as the standard checkpoints, under the `fp8` revision rather than in separate FP8 repositories. This example downloads `nvidia/Cosmos3-Nano` at revision `fp8`. To use another released FP8 checkpoint, change `FP8_MODEL_ID` and `FP8_CHECKPOINT` below; keep `FP8_REVISION=fp8`.\n",
650+
"\n",
651+
"Set `FP8_NUM_GPUS=1` to run without FSDP. To shard the model with FSDP, expose multiple GPUs in `CUDA_VISIBLE_DEVICES` and set `FP8_NUM_GPUS` to their count (for example, `CUDA_VISIBLE_DEVICES=0,1` and `FP8_NUM_GPUS=2`).\n",
652+
"\n",
653+
"Expected output file:\n",
654+
"\n",
655+
"```text\n",
656+
"packages/cosmos3/outputs/cookbooks/cosmos3/reasoner/nano/cosmos_framework_nano_fp8_text/nano_text/reasoner_text.txt\n",
657+
"```\n"
658+
]
659+
},
660+
{
661+
"cell_type": "code",
662+
"execution_count": null,
663+
"id": "228e450b",
664+
"metadata": {},
665+
"outputs": [],
666+
"source": [
667+
"%%bash\n",
668+
"set -euo pipefail\n",
669+
"\n",
670+
"cd \"$COSMOS3_REPO\"\n",
671+
"FP8_MODEL_ID=\"nvidia/Cosmos3-Nano\"\n",
672+
"FP8_REVISION=\"fp8\"\n",
673+
"FP8_CHECKPOINT=\"$COSMOS3_REPO/checkpoints/Cosmos3-Nano-FP8\"\n",
674+
"FP8_NUM_GPUS=\"${FP8_NUM_GPUS:-1}\"\n",
675+
"uvx hf@latest download \"$FP8_MODEL_ID\" \\\n",
676+
" --revision \"$FP8_REVISION\" \\\n",
677+
" --local-dir \"$FP8_CHECKPOINT\"\n",
678+
"\n",
679+
"COSMOS_TRAINING=false CUDA_VISIBLE_DEVICES=\"$CUDA_VISIBLE_DEVICES\" \\\n",
680+
".venv/bin/torchrun \\\n",
681+
" --nproc-per-node=\"$FP8_NUM_GPUS\" \\\n",
682+
" --master-addr=\"$COSMOS3_MASTER_ADDR\" \\\n",
683+
" --master-port=\"$COSMOS3_NANO_TEXT_MASTER_PORT\" \\\n",
684+
" -m cosmos_framework.scripts.inference \\\n",
685+
" --parallelism-preset=throughput \\\n",
686+
" --dp-shard-size=\"$FP8_NUM_GPUS\" \\\n",
687+
" --dp-replicate-size=1 \\\n",
688+
" --cp-size=1 \\\n",
689+
" --cfgp-size=1 \\\n",
690+
" --no-use-torch-compile \\\n",
691+
" -i \"$COSMOS3_INPUT_DIR/nano_text.json\" \\\n",
692+
" -o \"$COSMOS3_OUTPUT_ROOT/cosmos_framework_nano_fp8_text\" \\\n",
693+
" --checkpoint-path \"$FP8_CHECKPOINT\" \\\n",
694+
" --seed=0 \\\n",
695+
" --benchmark\n"
696+
]
697+
},
698+
{
699+
"cell_type": "code",
700+
"execution_count": null,
701+
"id": "7fdbcd0f",
702+
"metadata": {},
703+
"outputs": [],
704+
"source": [
705+
"from pathlib import Path\n",
706+
"import os, json\n",
707+
"\n",
708+
"output_root = Path(os.environ[\"COSMOS3_OUTPUT_ROOT\"])\n",
709+
"text_path = output_root / \"cosmos_framework_nano_fp8_text\" / \"nano_text\" / \"reasoner_text.txt\"\n",
710+
"benchmark_path = output_root / \"cosmos_framework_nano_fp8_text\" / \"benchmark.json\"\n",
711+
"\n",
712+
"print(text_path)\n",
713+
"print(text_path.read_text())\n",
714+
"if benchmark_path.exists():\n",
715+
" print(json.dumps(json.loads(benchmark_path.read_text()).get(\"average\", {}), indent=2))\n"
716+
]
717+
},
642718
{
643719
"cell_type": "markdown",
644720
"id": "5e7b2393",

0 commit comments

Comments
 (0)