ComfyUI now supports Nvidia Cosmos

ComfyUI has rolled out support for Nvidia Cosmos, introducing groundbreaking tools for AI video generation. This update includes the Cosmos 7B model, optimized for high efficiency on GPUs with 24GB VRAM and even functional on 12GB GPUs, albeit with slower render times. By lowering hardware demands, the integration promises broader access to advanced AI video technology.

The Cosmos 7B model supports up to 121 frames at a 704×704 resolution and shines with longer, detailed prompts. Users can now create vivid, expressive outputs while leveraging features like negative prompts, which simplify training compared to distilled models like Hunyuan Video.

Key Features of Nvidia Cosmos in ComfyUI:

Revolutionary VAE: The model’s video VAE is significantly more efficient, encoding and decoding 121 frames of 1280×704 video on a 12GB GPU without requiring tiling tricks, delivering a ~50x memory efficiency boost over Hunyuan Video.
Image-to-Video Functionality: Users can generate videos from the last or first frame, or even transition between two images, with prompt-based control for seamless creativity.
Enhanced Coherence: A new res_multistep sampler refines video generation, ensuring smoother sequences and improved quality.

Limitations and Considerations:

While the advancements are impressive, the update comes with some caveats:

The model is optimized for 121 frames, and deviations can affect performance.
The lowest supported resolution is 704×704.
Generating high-resolution videos with detailed prompts requires longer processing times, taking over 10 minutes for a 1280×704, 121-frame video on an NVIDIA RTX 4090.

This update solidifies ComfyUI’s position as a leader in accessible, high-quality AI video creation, empowering developers, artists, and researchers to explore new creative frontiers. For detailed instructions and further insights, visit the official ComfyUI website.