NVlabs/DiffiT

Stars
443
Rank 98,504 (Top 2 %)
Language
Created almost 2 years ago
Updated 5 months ago

NVlabs/DiffiT

NVlabs

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

[ECCV 2024] Official Repository for DiffiT: Diffusion Vision Transformers for Image Generation

DiffiT: Diffusion Vision Transformers for Image Generation

Official PyTorch implementation of DiffiT: Diffusion Vision Transformers for Image Generation.

Code and pretrained DiffiT models will be released soon !

DiffiT achieves a new SOTA FID score of 1.73 on ImageNet-256 dataset !

In addition, DiffiT sets a new SOTA FID score of 2.22 on FFHQ-64 dataset !

We introduce a new Time-dependent Multihead Self-Attention (TMSA) mechanism that jointly learns spatial and temporal dependencies and allows for attention conditioning with finegrained control.

💥 News 💥

[12.04.2023] 🔥 DiffiT manuscript is now available on arXiv !

Benchmarks

Latent Space

ImageNet-256

Model	Dataset	Resolution	FID-50K	Inception Score
Latent DiffiT	ImageNet	256x256	1.73	276.49

ImageNet-512

Model	Dataset	Resolution	FID-50K	Inception Score
Latent DiffiT	ImageNet	512x512	2.67	252.12

Image Space

Model	Dataset	Resolution	FID-50K
DiffiT	CIFAR-10	32x32	1.95
DiffiT	FFHQ-64	64x64	2.22

instant-ngp

Instant neural graphics primitives: lightning fast NeRF and more

stylegan

StyleGAN - Official TensorFlow Implementation

stylegan2

StyleGAN2 - Official TensorFlow Implementation

SPADE

Semantic Image Synthesis with SPADE

stylegan3

Official PyTorch implementation of StyleGAN3

neuralangelo

Official implementation of "Neuralangelo: High-Fidelity Neural Surface Reconstruction" (CVPR 2023)

imaginaire

NVIDIA's Deep Imagination Team's PyTorch Library

stylegan2-ada-pytorch

StyleGAN2-ADA - Official PyTorch implementation

tiny-cuda-nn

Lightning fast C++/CUDA neural network framework

ffhq-dataset

Flickr-Faces-HQ Dataset (FFHQ)

eg3d

MUNIT

Multimodal Unsupervised Image-to-Image Translation

SegFormer

Official PyTorch implementation of SegFormer

nvdiffrec

Official code for the CVPR 2022 (oral) paper "Extracting Triangular 3D Models, Materials, and Lighting From Images".

VILA

VILA - a multi-image visual language model with training, inference and evaluation recipe, deployable from cloud to edge (Jetson Orin and laptops)

few-shot-vid2vid

Pytorch implementation for few-shot photorealistic video-to-video translation.

stylegan2-ada

StyleGAN2 with adaptive discriminator augmentation (ADA) - Official TensorFlow implementation

FUNIT

Translate images to unseen domains in the test time with few example images.

PWC-Net

PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume, CVPR 2018 (Oral)

noise2noise

Noise2Noise: Learning Image Restoration without Clean Data - Official TensorFlow implementation of the ICML 2018 paper

nvdiffrast

Nvdiffrast - Modular Primitives for High-Performance Differentiable Rendering

alias-free-gan

Alias-Free GAN project website and code

edm

Elucidating the Design Space of Diffusion-Based Generative Models (EDM)

prismer

The implementation of "Prismer: A Vision-Language Model with Multi-Task Experts".

FoundationPose

[CVPR 2024 Highlight] FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects

DG-Net

👫 Joint Discriminative and Generative Learning for Person Re-identification. CVPR'19 (Oral) 👫

VoxFormer

Official PyTorch implementation of VoxFormer [CVPR 2023 Highlight]

Deep_Object_Pose

Deep Object Pose Estimation (DOPE) – ROS inference (CoRL 2018)

BundleSDF

[CVPR 2023] BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects

NVAE

The Official PyTorch Implementation of "NVAE: A Deep Hierarchical Variational Autoencoder" (NeurIPS 2020 spotlight paper)

ODISE

Official PyTorch implementation of ODISE: Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models [CVPR 2023 Highlight]

FasterViT

[ICLR 2024] Official PyTorch implementation of FasterViT: Fast Vision Transformers with Hierarchical Attention

MambaVision

Official PyTorch Implementation of MambaVision: A Hybrid Mamba-Transformer Vision Backbone

GroupViT

Official PyTorch implementation of GroupViT: Semantic Segmentation Emerges from Text Supervision, CVPR 2022.

curobo

CUDA Accelerated Robot Library

sionna

Sionna: An Open-Source Library for Next-Generation Physical Layer Research

denoising-diffusion-gan

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs https://arxiv.org/abs/2112.07804

InstantSplat

InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds

GA3C

Hybrid CPU/GPU implementation of the A3C algorithm for deep reinforcement learning.

FB-BEV

Official PyTorch implementation of FB-BEV & FB-OCC - Forward-backward view transformation for vision-centric autonomous driving perception

genvs

DoRA

[ICML2024 (Oral)] Official PyTorch implementation of DoRA: Weight-Decomposed Low-Rank Adaptation

RADIO

Official repository for "AM-RADIO: Reduce All Domains Into One"

EmerNeRF

PyTorch Implementation of EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision

CALM

EAGLE

EAGLE: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

Dancing2Music

FourCastNet

Initial public release of code, data, and model weights for FourCastNet

planercnn

PlaneRCNN detects and reconstructs piece-wise planar surfaces from a single RGB image

pacnet

Pixel-Adaptive Convolutional Neural Networks (CVPR '19)

edm2

Analyzing and Improving the Training Dynamics of Diffusion Models (EDM2)

DeepInversion

Official PyTorch implementation of Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion (CVPR 2020)

FAN

Official PyTorch implementation of Fully Attentional Networks

GCVit

[ICML 2023] Official PyTorch implementation of Global Context Vision Transformers

intrinsic3d

Intrinsic3D - High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting (ICCV 2017)

nvdiffmodeling

Differentiable rasterization applied to 3D model simplification tasks

flip

A tool for visualizing and communicating the errors in rendered images.

nvdiffrecmc

Official code for the NeurIPS 2022 paper "Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising".

wetectron

Weakly-supervised object detection.

GLAMR

[CVPR 2022 Oral] Official PyTorch Implementation of "GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras”.

geomapnet

Geometry-Aware Learning of Maps for Camera Localization (CVPR2018)

LSGM

The Official PyTorch Implementation of "LSGM: Score-based Generative Modeling in Latent Space" (NeurIPS 2021)

timeloop

Timeloop performs modeling, mapping and code-generation for tensor algebra workloads on various accelerator architectures.

ssn_superpixels

Superpixel Sampling Networks (ECCV2018)

FreeSOLO

FreeSOLO for unsupervised instance segmentation, CVPR 2022

long-video-gan

Official PyTorch implementation of LongVideoGAN

trajdata

A unified interface to many trajectory forecasting datasets.

contact_graspnet

Efficient 6-DoF Grasp Generation in Cluttered Scenes

neuralrgbd

Neural RGB→D Sensing: Per-pixel depth and its uncertainty estimation from a monocular RGB video

selfsupervised-denoising

High-Quality Self-Supervised Deep Image Denoising - Official TensorFlow implementation of the NeurIPS 2019 paper

CF-3DGS

sim-web-visualizer

Web Based Visualizer for Simulation Environments

Taylor_pruning

Pruning Neural Networks with Taylor criterion in Pytorch

mimicgen

This code corresponds to simulation environments used as part of the MimicGen project.

metfaces-dataset

few_shot_gaze

Pytorch implementation and demo of FAZE: Few-Shot Adaptive Gaze Estimation (ICCV 2019, oral)

Hydra-MDP

splatnet

SPLATNet: Sparse Lattice Networks for Point Cloud Processing (CVPR2018)

VILA-archive

VILA - A multi-image visual language model with training, inference and evaluation recipe, deployable from cloud to edge (Jetson Orin and laptops)

RVT

Official Code for RVT-2 and RVT

Jupyter Notebook

MinVIS

CenterPose

Single-Stage Keypoint-based Category-level Object Pose Estimation from an RGB Image (ICRA 2022)

matchlib

SystemC/C++ library of commonly-used hardware functions and components for HLS.

Minitron

A family of compressed models obtained via pruning and knowledge distillation

DiffRL

[ICLR 2022] Accelerated Policy Learning with Parallel Differentiable Simulation

DiffPure

A new adversarial purification method that uses the forward and reverse processes of diffusion models to remove adversarial perturbations.

STEP

STEP: Spatio-Temporal Progressive Learning for Video Action Detection. CVPR'19 (Oral)

I2SB

SCOPS

SCOPS: Self-Supervised Co-Part Segmentation (CVPR'19)

UMR

Self-supervised Single-view 3D Reconstruction

cule

CuLE: A CUDA port of the Atari Learning Environment (ALE)

SSV

Pytorch implementation of SSV: Self-Supervised Viewpoint Learning from Image Collections (CVPR 2020)

NVBit

AFNO-transformer

Adaptive FNO transformer - official Pytorch implementation

6dof-graspnet

Implementation of 6-DoF GraspNet with tensorflow and python. This repo has been tested with python 2.7 and tensorflow 1.12.

latentfusion

LatentFusion: End-to-End Differentiable Reconstruction and Rendering for Unseen Object Pose Estimation

nvbio

NVBIO is a library of reusable components designed to accelerate bioinformatics applications using CUDA.

OmniDrive

UnseenObjectClustering

Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation

traffic-behavior-simulation