• Stars
    star
    498
  • Rank 87,744 (Top 2 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created about 1 year ago
  • Updated 8 days ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Scalable toolkit for efficient model alignment

NVIDIA NeMo-Aligner

Introduction

NeMo-Aligner is a scalable toolkit for efficient model alignment. The toolkit has support for state of the art model alignment algorithms such as SteerLM, DPO and Reinforcement Learning from Human Feedback (RLHF). These algorithms enable users to align language models to be more safe, harmless and helpful. Users can do end-to-end model alignment on a wide range of model sizes and take advantage of all the parallelism techniques to ensure their model alignment is done in a performant and resource efficient manner.

NeMo-Aligner toolkit is built using the NeMo Toolkit which allows for scaling training up to 1000s of GPUs using tensor, data and pipeline parallelism for all components of alignment. All of our checkpoints are cross compatible with the NeMo ecosystem; allowing for inference deployment and further customization.

The toolkit is currently in it's early stages, and we are committed to improving the toolkit to make it easier for developers to pick and choose different alignment algorithms to build safe, helpful and reliable models.

Key features

  • SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF.
  • Supervised Fine Tuning
  • Reward Model Training
  • Reinforcement Learning from Human Feedback using the PPO Algorithm
  • Direct Preference Optimization as described in paper

Learn More

Latest Release

For the latest stable release please see the releases page. All releases come with a pre-built container. Changes within each release will be documented in CHANGELOG.

Installing your own environment

Requirements

NeMo-Aligner has the same requirements as the NeMo Toolkit Requirements with the addition of PyTriton.

Installation

Please follow the same steps as the NeMo Toolkit Installation Guide but run the following after installing NeMo

pip install nemo-aligner

or if you prefer to install the latest commit

pip install .

Docker Containers

We provide an official NeMo-Aligner Dockerfile which is based on stable, tested versions of NeMo, Megatron-LM, and TransformerEngine. The goal of this Dockerfile is stability, so it may not track the very latest versions of those 3 packages. You can access our Dockerfile here

Alternatively, you can build the NeMo Dockerfile here NeMo Dockerfile and add RUN pip install nemo-aligner at the end.

Future work

  • Add Rejection Sampling support
  • We will continue improving the stability of the PPO learning phase.
  • Improve the performance of RLHF

Contributing

We welcome community contributions! Please refer to CONTRIBUTING.md for guidelines.

License

This toolkit is licensed under the Apache License, Version 2.0.

More Repositories

1

nvidia-docker

Build and run Docker containers leveraging NVIDIA GPUs
16,896
star
2

open-gpu-kernel-modules

NVIDIA Linux open GPU kernel module source
C
14,991
star
3

DeepLearningExamples

State-of-the-Art Deep Learning scripts organized by models - easy to train and deploy with reproducible accuracy and performance on enterprise-grade infrastructure.
Jupyter Notebook
13,169
star
4

NeMo

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)
Python
11,421
star
5

FastPhotoStyle

Style transfer, deep learning, feature transform
Python
11,020
star
6

TensorRT

NVIDIAยฎ TensorRTโ„ข is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
C++
10,455
star
7

Megatron-LM

Ongoing research training transformer models at scale
Python
9,813
star
8

vid2vid

Pytorch implementation of our method for high-resolution (e.g. 2048x1024) photorealistic video-to-video translation.
Python
8,482
star
9

apex

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch
Python
8,239
star
10

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
C++
8,047
star
11

pix2pixHD

Synthesizing and manipulating 2048x1024 images with conditional GANs
Python
6,488
star
12

cuda-samples

Samples for CUDA Developers which demonstrates features in CUDA Toolkit
C
5,704
star
13

FasterTransformer

Transformer related optimization, including BERT, GPT
C++
5,313
star
14

cutlass

CUDA Templates for Linear Algebra Subroutines
C++
5,100
star
15

DALI

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
C++
5,048
star
16

thrust

[ARCHIVED] The C++ parallel algorithms library. See https://github.com/NVIDIA/cccl
C++
4,900
star
17

tacotron2

Tacotron 2 - PyTorch implementation with faster-than-realtime inference
Jupyter Notebook
4,562
star
18

DIGITS

Deep Learning GPU Training System
HTML
4,105
star
19

NeMo-Guardrails

NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.
Python
3,924
star
20

warp

A Python framework for high performance GPU simulation and graphics
Python
3,908
star
21

nccl

Optimized primitives for collective multi-GPU communication
C++
3,086
star
22

flownet2-pytorch

Pytorch implementation of FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Python
2,938
star
23

ChatRTX

A developer reference project for creating Retrieval Augmented Generation (RAG) chatbots on Windows using TensorRT-LLM
TypeScript
2,635
star
24

k8s-device-plugin

NVIDIA device plugin for Kubernetes
Go
2,481
star
25

libcudacxx

[ARCHIVED] The C++ Standard Library for your entire system. See https://github.com/NVIDIA/cccl
C++
2,292
star
26

waveglow

A Flow-based Generative Network for Speech Synthesis
Python
2,133
star
27

GenerativeAIExamples

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
Python
2,052
star
28

MinkowskiEngine

Minkowski Engine is an auto-diff neural network library for high-dimensional sparse tensors
Python
2,007
star
29

Stable-Diffusion-WebUI-TensorRT

TensorRT Extension for Stable Diffusion Web UI
Python
1,886
star
30

nvidia-container-toolkit

Build and run containers leveraging NVIDIA GPUs
Go
1,785
star
31

semantic-segmentation

Nvidia Semantic Segmentation monorepo
Python
1,763
star
32

TransformerEngine

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilization in both training and inference.
Python
1,761
star
33

cub

[ARCHIVED] Cooperative primitives for CUDA C++. See https://github.com/NVIDIA/cccl
Cuda
1,668
star
34

DeepRecommender

Deep learning for recommender systems
Python
1,662
star
35

OpenSeq2Seq

Toolkit for efficient experimentation with Speech Recognition, Text2Speech and NLP
Python
1,511
star
36

stdexec

`std::execution`, the proposed C++ framework for asynchronous and parallel programming.
C++
1,483
star
37

CUDALibrarySamples

CUDA Library Samples
Cuda
1,468
star
38

VideoProcessingFramework

Set of Python bindings to C++ libraries which provides full HW acceleration for video decoding, encoding and GPU-accelerated color space and pixel format conversions
C++
1,292
star
39

deepops

Tools for building GPU clusters
Shell
1,236
star
40

open-gpu-doc

Documentation of NVIDIA chip/hardware interfaces
C
1,235
star
41

gpu-operator

NVIDIA GPU Operator creates/configures/manages GPUs atop Kubernetes
Go
1,226
star
42

aistore

AIStore: scalable storage for AI applications
Go
1,215
star
43

trt-samples-for-hackathon-cn

Simple samples for TensorRT programming
Python
1,211
star
44

Q2RTX

NVIDIAโ€™s implementation of RTX ray-tracing in Quake II
C
1,206
star
45

partialconv

A New Padding Scheme: Partial Convolution based Padding
Python
1,145
star
46

MatX

An efficient C++17 GPU numerical computing library with Python-like syntax
C++
1,132
star
47

cccl

CUDA Core Compute Libraries
C++
1,093
star
48

sentiment-discovery

Unsupervised Language Modeling at scale for robust sentiment classification
Python
1,055
star
49

nvidia-container-runtime

NVIDIA container runtime
Makefile
1,035
star
50

gpu-monitoring-tools

Tools for monitoring NVIDIA GPUs on Linux
C
974
star
51

jetson-gpio

A Python library that enables the use of Jetson's GPIOs
Python
898
star
52

modulus

Open-source deep-learning framework for building, training, and fine-tuning deep learning models using state-of-the-art Physics-ML methods
Python
893
star
53

retinanet-examples

Fast and accurate object detection with end-to-end GPU optimization
Python
885
star
54

flowtron

Flowtron is an auto-regressive flow-based generative network for text to speech synthesis with control over speech variation and style transfer
Jupyter Notebook
867
star
55

mellotron

Mellotron: a multispeaker voice synthesis model based on Tacotron 2 GST that can make a voice emote and sing without emotive or singing training data
Jupyter Notebook
852
star
56

cuda-python

CUDA Python Low-level Bindings
Python
835
star
57

gdrcopy

A fast GPU memory copy library based on NVIDIA GPUDirect RDMA technology
C++
832
star
58

BigVGAN

Official PyTorch implementation of BigVGAN (ICLR 2023)
Python
806
star
59

spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
Scala
782
star
60

libnvidia-container

NVIDIA container runtime library
C
779
star
61

nccl-tests

NCCL Tests
Cuda
760
star
62

dcgm-exporter

NVIDIA GPU metrics exporter for Prometheus leveraging DCGM
Go
753
star
63

nv-wavenet

Reference implementation of real-time autoregressive wavenet inference
Cuda
728
star
64

tensorflow

An Open Source Machine Learning Framework for Everyone
C++
719
star
65

MAXINE-AR-SDK

NVIDIA AR SDK - API headers and sample applications
C
671
star
66

DLSS

NVIDIA DLSS is a new and improved deep learning neural network that boosts frame rates and generates beautiful, sharp images for your games
C
667
star
67

nvvl

A library that uses hardware acceleration to load sequences of video frames to facilitate machine learning training
C++
665
star
68

gvdb-voxels

Sparse volume compute and rendering on NVIDIA GPUs
C
664
star
69

runx

Deep Learning Experiment Management
Python
633
star
70

NVFlare

NVIDIA Federated Learning Application Runtime Environment
Python
579
star
71

nvcomp

Repository for nvCOMP docs and examples. nvCOMP is a library for fast lossless compression/decompression on the GPU that can be downloaded from https://developer.nvidia.com/nvcomp.
C++
545
star
72

Dataset_Synthesizer

NVIDIA Deep learning Dataset Synthesizer (NDDS)
C++
530
star
73

jitify

A single-header C++ library for simplifying the use of CUDA Runtime Compilation (NVRTC).
C++
512
star
74

libglvnd

The GL Vendor-Neutral Dispatch library
C
493
star
75

multi-gpu-programming-models

Examples demonstrating available options to program multiple GPUs in a single node or a cluster
Cuda
481
star
76

AMGX

Distributed multigrid linear solver library on GPU
Cuda
474
star
77

nvbench

CUDA Kernel Benchmarking Library
Cuda
468
star
78

cuda-quantum

C++ and Python support for the CUDA Quantum programming model for heterogeneous quantum-classical workflows
C++
463
star
79

cuCollections

C++
463
star
80

enroot

A simple yet powerful tool to turn traditional container/OS images into unprivileged sandboxes.
Shell
459
star
81

hpc-container-maker

HPC Container Maker
Python
442
star
82

NeMo-Framework-Launcher

Provides end-to-end model development pipelines for LLMs and Multimodal models that can be launched on-prem or cloud-native.
Python
442
star
83

NeMo-Curator

Scalable data pre processing and curation toolkit for LLMs
Jupyter Notebook
439
star
84

MDL-SDK

NVIDIA Material Definition Language SDK
C++
438
star
85

PyProf

A GPU performance profiling tool for PyTorch models
Python
437
star
86

gpu-rest-engine

A REST API for Caffe using Docker and Go
C++
421
star
87

framework-reproducibility

Providing reproducibility in deep learning frameworks
Python
421
star
88

TensorRT-Model-Optimizer

TensorRT Model Optimizer is a unified library of state-of-the-art model optimization techniques such as quantization and sparsity. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM or TensorRT to optimize inference speed on NVIDIA GPUs.
Python
407
star
89

NvPipe

NVIDIA-accelerated zero latency video compression library for interactive remoting applications
Cuda
390
star
90

DCGM

NVIDIA Data Center GPU Manager (DCGM) is a project for gathering telemetry and measuring the health of NVIDIA GPUs
C++
354
star
91

cuQuantum

Home for cuQuantum Python & NVIDIA cuQuantum SDK C++ samples
Jupyter Notebook
328
star
92

data-science-stack

NVIDIA Data Science stack tools
Shell
317
star
93

ai-assisted-annotation-client

Client side integration example source code and libraries for AI-Assisted Annotation SDK
C++
306
star
94

video-sdk-samples

Samples demonstrating how to use various APIs of NVIDIA Video Codec SDK
C++
301
star
95

torch-harmonics

Differentiable spherical harmonic transforms and spherical convolutions in PyTorch
Jupyter Notebook
297
star
96

nvidia-settings

NVIDIA driver control panel
C
294
star
97

gpu-feature-discovery

GPU plugin to the node feature discovery for Kubernetes
Go
287
star
98

egl-wayland

The EGLStream-based Wayland external platform
C
280
star
99

cnmem

A simple memory manager for CUDA designed to help Deep Learning frameworks manage memory
C++
280
star
100

radtts

Provides training, inference and voice conversion recipes for RADTTS and RADTTS++: Flow-based TTS models with Robust Alignment Learning, Diverse Synthesis, and Generative Modeling and Fine-Grained Control over of Low Dimensional (F0 and Energy) Speech Attributes.
Roff
269
star