Overview

Pusa introduces a paradigm shift in video diffusion modeling through frame-level noise control (thus it has thousands of timesteps, rather than one thousand of timesteps), departing from conventional approaches. This shift was first presented in our FVDM paper. Leveraging this architecture, Pusa seamlessly supports diverse video generation tasks (Text/Image/Video-to-Video) while maintaining exceptional motion fidelity and prompt adherence with our refined base model adaptations. Pusa-V0.5 represents an early preview based on Mochi1-Preview. We are open-sourcing this work to foster community collaboration, enhance methodologies, and expand capabilities.

Model: https://huggingface.co/RaphaelLiu/Pusa-V0.5

Code: https://github.com/Yaofang-Liu/Pusa-VidGen

Training Toolkit: https://github.com/Yaofang-Liu/Mochi-Full-Finetuner

Dataset: https://huggingface.co/datasets/RaphaelLiu/PusaV0.5_Training

Some things are just meant for each other. Like Igorr and GenAI art.

Announcement: https://github.com/mit-han-lab/nunchaku/discussions/236

Abstract

Recent advancements in Unet-based diffusion models, such as ControlNet and IP-Adapter, have introduced effective spatial and subject control mechanisms. However, the DiT (Diffusion Transformer) architecture still struggles with efficient and flexible control. To tackle this issue, we propose EasyControl, a novel framework designed to unify condition-guided diffusion transformers with high efficiency and flexibility. Our framework is built on three key innovations. First, we introduce a lightweight Condition Injection LoRA Module. This module processes conditional signals in isolation, acting as a plug-and-play solution. It avoids modifying the base model weights, ensuring compatibility with customized models and enabling the flexible injection of diverse conditions. Notably, this module also supports harmonious and robust zero-shot multi-condition generalization, even when trained only on single-condition data. Second, we propose a Position-Aware Training Paradigm. This approach standardizes input conditions to fixed resolutions, allowing the generation of images with arbitrary aspect ratios and flexible resolutions. At the same time, it optimizes computational efficiency, making the framework more practical for real-world applications. Third, we develop a Causal Attention Mechanism combined with the KV Cache technique, adapted for conditional generation tasks. This innovation significantly reduces the latency of image synthesis, improving the overall efficiency of the framework. Through extensive experiments, we demonstrate that EasyControl achieves exceptional performance across various application scenarios. These innovations collectively make our framework highly efficient, flexible, and suitable for a wide range of tasks.

Paper: https://arxiv.org/abs/2503.07027

Code: https://github.com/Xiaojiu-z/EasyControl

Model: https://huggingface.co/Xiaojiu-Z/EasyControl/

Project Page: https://easycontrolproj.github.io/

Demo: https://huggingface.co/spaces/jamesliu1217/EasyControl/

Abstract

Diffusion models have achieved remarkable progress in the field of video generation. However, their iterative denoising nature requires a large number of inference steps to generate a video, which is slow and computationally expensive. In this paper, we begin with a detailed analysis of the challenges present in existing diffusion distillation methods and propose a novel efficient method, namely AccVideo, to reduce the inference steps for accelerating video diffusion models with synthetic dataset. We leverage the pretrained video diffusion model to generate multiple valid denoising trajectories as our synthetic dataset, which eliminates the use of useless data points during distillation. Based on the synthetic dataset, we design a trajectory-based few-step guidance that utilizes key data points from the denoising trajectories to learn the noise-to-video mapping, enabling video generation in fewer steps. Furthermore, since the synthetic dataset captures the data distribution at each diffusion timestep, we introduce an adversarial training strategy to align the output distribution of the student model with that of our synthetic dataset, thereby enhancing the video quality. Extensive experiments demonstrate that our model achieves 8.5x improvements in generation speed compared to the teacher model while maintaining comparable performance. Compared to previous accelerating methods, our approach is capable of generating videos with higher quality and resolution, i.e., 5-seconds, 720x1280, 24fps.

Paper: https://arxiv.org/abs/2503.19462

Code: https://github.com/aejion/AccVideo/

Model: https://huggingface.co/aejion/AccVideo

Project Page: https://aejion.github.io/accvideo/

Abstract

Personalizing image generation and editing is particularly challenging when we only have a few images of the subject, or even a single image. A common approach to personalization is concept learning, which can integrate the subject into existing models relatively quickly, but produces images whose quality tends to deteriorate quickly when the number of subject images is small. Quality can be improved by pre-training an encoder, but training restricts generation to the training distribution, and is time consuming. It is still an open hard challenge to personalize image generation and editing from a single image without training. Here, we present SISO, a novel, training-free approach based on optimizing a similarity score with an input subject image. More specifically, SISO iteratively generates images and optimizes the model based on loss of similarity with the given subject image until a satisfactory level of similarity is achieved, allowing plug-and-play optimization to any image generator. We evaluated SISO in two tasks, image editing and image generation, using a diverse data set of personal subjects, and demonstrate significant improvements over existing methods in image quality, subject fidelity, and background preservation.

Paper: https://arxiv.org/abs/2503.16025

Code: https://github.com/yairshp/SISO

Project Page: https://siso-paper.github.io/

Abstract

In this paper, we propose LSRNA, a novel framework for higher-resolution (exceeding 1K) image generation using diffusion models by leveraging super-resolution directly in the latent space. Existing diffusion models struggle with scaling beyond their training resolutions, often leading to structural distortions or content repetition. Reference-based methods address the issues by upsampling a low-resolution reference to guide higher-resolution generation. However, they face significant challenges: upsampling in latent space often causes manifold deviation, which degrades output quality. On the other hand, upsampling in RGB space tends to produce overly smoothed outputs. To overcome these limitations, LSRNA combines Latent space Super-Resolution (LSR) for manifold alignment and Region-wise Noise Addition (RNA) to enhance high-frequency details. Our extensive experiments demonstrate that integrating LSRNA outperforms state-of-the-art reference-based methods across various resolutions and metrics, while showing the critical role of latent space upsampling in preserving detail and sharpness. The code is available at this https URL.

Paper: https://arxiv.org/abs/2503.18446

Code: https://github.com/3587jjh/LSRNA (coming soon)

https://youtu.be/Xa9Sg-j62xY

Artstation: https://www.artstation.com/artwork/eR41JJ

Quickstart: https://github.com/bghira/SimpleTuner/blob/main/documentation/quickstart/LTXVIDEO.md

Release notes: https://github.com/bghira/SimpleTuner/releases/tag/v1.3.0

Abstract

Style transfer involves transferring the style from a reference image to the content of a target image. Recent advancements in LoRA-based (Low-Rank Adaptation) methods have shown promise in effectively capturing the style of a single image. However, these approaches still face significant challenges such as content inconsistency, style misalignment, and content leakage. In this paper, we comprehensively analyze the limitations of the standard diffusion parameterization, which learns to predict noise, in the context of style transfer. To address these issues, we introduce ConsisLoRA, a LoRA-based method that enhances both content and style consistency by optimizing the LoRA weights to predict the original image rather than noise. We also propose a two-step training strategy that decouples the learning of content and style from the reference image. To effectively capture both the global structure and local details of the content image, we introduce a stepwise loss transition strategy. Additionally, we present an inference guidance method that enables continuous control over content and style strengths during inference. Through both qualitative and quantitative evaluations, our method demonstrates significant improvements in content and style consistency while effectively reducing content leakage.

Paper: https://arxiv.org/abs/2503.10614

Code: https://github.com/000linlin/ConsisLoRA (coming soon)

Project Page: https://consislora.github.io/

FP8 Model: https://huggingface.co/Kijai/HunyuanVideo_comfy/tree/main

ComfyUI nodes (updated wrapper): https://github.com/kijai/ComfyUI-HunyuanVideoWrapper

Example ComfyUI workflow: https://github.com/kijai/ComfyUI-HunyuanVideoWrapper/blob/main/example_workflows/hyvideo_i2v_example_01.json

Discord Server: https://discord.com/invite/7tsKMCbNFC

Model Description

TensorArt-TurboX-SD3.5Large is a highly optimized variant of Stable Diffusion 3.5 Large, achieving 6x faster generation speed with minimal quality loss. It surpasses the official Stable Diffusion 3.5 Large Turbo in image detail, diversity, richness, and realism.

Repo: https://github.com/nygaard91/Pixel-Perfect-AI-Art-Converter

AI-generated pixel art aren’t truly pixel perfect, too oversized for game assets, or otherwise unsuitable for professional use—this tool provides a practical, manual approach that can better meet your needs.

Abstract

Recent advancements in text-to-image generative systems have been largely driven by diffusion models. However, single-stage text-to-image diffusion models still face challenges, in terms of computational efficiency and the refinement of image details. To tackle the issue, we propose CogView3, an innovative cascaded framework that enhances the performance of text-to-image diffusion. CogView3 is the first model implementing relay diffusion in the realm of text-to-image generation, executing the task by first creating low-resolution images and subsequently applying relay-based super-resolution. This methodology not only results in competitive text-to-image outputs but also greatly reduces both training and inference costs. Our experimental results demonstrate that CogView3 outperforms SDXL, the current state-of-the-art open-source text-to-image diffusion model, by 77.0% in human evaluations, all while requiring only about 1/2 of the inference time. The distilled variant of CogView3 achieves comparable performance while only utilizing 1/10 of the inference time by SDXL.

Paper: https://arxiv.org/abs/2403.05121

Code: https://github.com/THUDM/CogView4

Demo: https://huggingface.co/spaces/THUDM-HF-SPACE/CogView4

Model Weights: https://huggingface.co/THUDM/CogView4-6B/tree/main

Abstract

In 3D modeling, designers often use an existing 3D model as a reference to create new ones. This practice has inspired the development of Phidias, a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generalization ability, and controllability. Our model integrates three key components: 1) meta-ControlNet that dynamically modulates the conditioning strength, 2) dynamic reference routing that mitigates misalignment between the input image and 3D reference, and 3) self-reference augmentations that enable self-supervised training with a progressive curriculum. Collectively, these designs result in a clear improvement over existing methods. Phidias establishes a unified framework for 3D generation using text, image, and 3D conditions with versatile applications.

Paper: https://arxiv.org/abs/2409.11406

Code: https://github.com/3DTopia/Phidias-Diffusion

Models: https://huggingface.co/ZhenweiWang/Phidias-Diffusion/tree/main

Project Page: https://rag-3d.github.io/