NVIDIA/nvvl NVIDIA/nvvl

  • Stars
    star
    665
  • Rank 67,810 (Top 2 %)
  • Language
    C++
  • License
    Other
  • Created over 6 years ago
  • Updated over 5 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
NVIDIA/nvvl
github

NVIDIA

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A library that uses hardware acceleration to load sequences of video frames to facilitate machine learning training

NVVL is part of DALI!

DALI (Nvidia Data Loading Library) incorporates NVVL functionality and offers much more than that, so it is recommended to switch to it. DALI source code is also open source and available on the GitHub. Up to date documentation can be found here. NVVL project will still be available on the GitHub but it won't be maintained. All issues and request for the future please submit in the DALI repository.

NVVL

NVVL (NVIDIA Video Loader) is a library to load random sequences of video frames from compressed video files to facilitate machine learning training. It uses FFmpeg's libraries to parse and read the compressed packets from video files and the video decoding hardware available on NVIDIA GPUs to off-load and accelerate the decoding of those packets, providing a ready-for-training tensor in GPU device memory. NVVL can additionally perform data augmentation while loading the frames. Frames can be scaled, cropped, and flipped horizontally using the GPUs dedicated texture mapping units. Output can be in RGB or YCbCr color space, normalized to [0, 1] or [0, 255], and in float, half, or uint8 tensors.

Note that, while we hope you find NVVL useful, it is example code from a research project performed by a small group of NVIDIA researchers. We will do our best to answer questions and fix small bugs as they come up, but it is not a supported NVIDIA product and is for the most part provided as-is.

Using compressed video files instead of individual frame image files significantly reduces the demands on the storage and I/O systems during training. Storing video datasets as video files consumes an order of magnitude less disk space, allowing for larger datasets to both fit in system RAM as well as local SSDs for fast access. During loading fewer bytes must be read from disk. Fitting on smaller, faster storage and reading fewer bytes at load time allievates the bottleneck of retrieving data from disks, which will only get worse as GPUs get faster. For the dataset used in our example project, H.264 compressed .mp4 files were nearly 40x smaller than storing frames as .png files.

Using the hardware decoder on NVIDIA GPUs to decode images significantly reduces the demands on the host CPU. This means fewer CPU cores need to be dedicated to data loading during training. This is especially important in servers with a large number of GPUs per CPU, such as the in the NVIDIA DGX-2 server, but also provides benefits for other platforms. When training our example project on a NVIDIA DGX-1, the CPU load when using NVVL was 50-60% of the load seen when using a normal dataloader for .png files.

Measurements that quantify the performance advantages of using NVVL are detailed in our super resolution example project.

Most users will want to use the deep learning framework wrappers provided rather than using the library directly. Currently a wrapper for PyTorch is provided (PR's for other frameworks are welcome). See the PyTorch wrapper README for documentation on using the PyTorch wrapper. Note that it is not required to build or install the C++ library before building the PyTorch wrapper (its setup scripts will do so for you).

Building and Installing

NVVL depends on the following:

  • CUDA Toolkit. We have tested versions 8.0 and later but earlier versions may work. NVVL will perform better with CUDA 9.0 or later1.
  • FFmpeg's libavformat, libavcodec, libavfilter, and libavutil. These can be installed from source as in the example Dockerfiles or from the Ubuntu 16.04 packages libavcodec-dev libavfilter-dev libavformat-dev libavutil-dev. Other distributions should have similar packages.

Additionally, building from source requires CMake version 3.8 or above and some examples optionally make use of some libraries from OpenCV if they are installed.

The docker directory contains Dockerfiles that can be used as a starting point for creating an image to build or use the NVVL library. The example's docker directory has an example Dockerfile that actually builds and installs the NVVL library.

CMake 3.8 and above provides builtin CUDA language support that NVVL's build system uses. Since CMake 3.8 is relatively new and not yet in widely used Linux distribution, it may be required to install a new version of CMake. The easiest way to do so is to make use of their package on PyPI:

pip install cmake

Alternatively, or if pip isn't available, you can install to /usr/local from a binary distribution:

wget https://cmake.org/files/v3.10/cmake-3.10.2-Linux-x86_64.sh
/bin/sh cmake-3.10.2-Linux-x86_64.sh --prefix=/usr/local

See https://cmake.org/download/ for more options.

Building and installing NVVL follows the typical CMake pattern:

mkdir build && cd build
cmake ..
make -j
sudo make install

This will install libnvvl.so and development headers into appropriate subdirectores under /usr/local. CMake can be passed the following options using cmake .. -DOPTION=Value:

  • CUDA_ARCH - Name of a CUDA architecture to generate device code for, seperated via a semicolon. Valid options are Kepler, Maxwell, Pascal, and Volta. You can also use specific architecture names such as sm_61. Default is Maxwell;Pascal;Volta.

  • CMAKE_CUDA_FLAGS - A string of arguments to pass to nvcc. In particular, you can decide to link against the static or shared runtime library using -cudart shared or -cudart static. You can also use this for finer control of code generation than CUDA_ARCH, see the nvcc documentation. Default is -cudart shared.

  • WITH_OPENCV - Set this to 1 to build the examples with the optional OpenCV functionality.

  • CMAKE_INSTALL_PREFIX - Install directory. Default is /usr/local.

  • CMAKE_BUILD_TYPE - Debug or Release build.

See the CMake documentation for more options.

The examples in doc/examples can be built using the examples target:

make examples

Finally, if Doxygen is installed, API documentation can be built using the doc target:

make doc

This will build html files in doc/html.

Preparing Data

NVVL supports the H.264 and HEVC (H.265) video codecs in any container format that FFmpeg is able to parse. Video codecs only store certain frames, called keyframes or intra-frames, as a complete image in the data stream. All other frames require data from other frames, either before or after it in time, to be decoded. In order to decode a sequence of frames, it is necessary to start decoding at the keyframe before the sequence, and continue past the sequence to the next keyframe after it. This isn't a problem when streaming sequentially through a video; however, when decoding small sequences of frames randomly throughout the video, a large gap between keyframes results in reading and decoding a large amount of frames that are never used.

Thus, to get good performance when randomly reading short sequences from a video file, it is necessary to encode the file with frequent key frames. We've found setting the keyframe interval to the length of the sequences you will be reading provides a good compromise between filesize and loading performance. Also, NVVL's seeking logic doesn't support open GOPs in HEVC streams. To set the keyframe interval to X when using ffmpeg:

  • For libx264 use -g X
  • For libx265 use -x265-params "keyint=X:no-open-gop=1"

The pixel format of the video must also be yuv420p to be supported by the hardware decoder. This is done by passing -pix_fmt yuv420p to ffmpeg. You should also remove any extra audio or video streams from the video file by passing -map v:0 to ffmpeg after the input but before the output.

For example to transcode to H.264:

ffmpeg -i original.mp4 -map v:0 -c:v libx264 -crf 18 -pix_fmt yuv420p -g 5 -profile:v high prepared.mp4

Basic Usage

This section describes the usage of the base C/C++ library, for usage of the PyTorch wrapper, see the README in the pytorch directory.

The library provides both a C++ and C interface. See the examples in doc/examples for brief example code on how to use the library. extract_frames.cpp demonstrates the C++ interface and extract_frames_c.c the C interface. The API documentation built with make doc is the canonical reference for the API.

The basic flow is to create a VideoLoader object, tell it which frame sequences to read, and then give it buffers in device memory to put the decoded sequences into. In C++, creating a video loader is straight forward:

auto loader = NVVL::VideoLoader{device_id};

You can then tell it which sequences to read via read_sequence:

loader.read_sequence(filename, frame_num, sequence_length);

To receive the frames from the decoder, it is necessary to create a PictureSequence to tell it how and where you want the decoded frames provided. First, create a PictureSequence, providing a count of the number of frames to receive from the decoder. Note that the count here does not need to be the same as the sequence_length provided to read_sequence; you can read a large sequence of frames and receive them as multiple tensors, or read multiple smaller sequences and receive them concatenated as a single tensor.

auto seq = PictureSequence{sequence_count};

You now create "Layers" in the sequence to provide the destination for the frames. Each layer can be a different type, have different processing, and contain different frames from the received sequence. First, create a PictureSequence::Layer of the desired type:

auto pixels = PictureSequence::Layer<float>{};

Next, fill in the pointer to the data and other details. See the documentation in PictureSequence.h for a description of all the available options.

float* data = nullptr;
size_t pitch = 0;
cudaMallocPitch(&data, &pitch,
                crop_width * sizeof(float),
                crop_height * sequence_count * 3);
pixels.data = data;
pixels.desc.count = sequence_count;
pixels.desc.channels = 3;
pixels.desc.width = crop_width;
pixels.desc.height = crop_height;
pixels.desc.scale_width = scale_width;
pixels.desc.scale_height = scale_height;
pixels.desc.horiz_flip = false;
pixels.desc.normalized = true;
pixels.desc.color_space = ColorSpace_RGB;
pixels.desc.stride.x = 1;
pixels.desc.stride.y = pitch / sizeof(float);
pixels.desc.stride.c = pixels.desc.stride.y * crop_height;
pixels.desc.stride.n = pixels.desc.stride.c * 3;

Note that here we have set the strides such that the dimensions are "nchw", we could have done "nhwc" or any other dimension order by setting the strides appropriately. Also note that the strides in the layer description are number of elements, not number of bytes.

We now add this layer to our PictureSequence, and send it to the loader:

seq.set_layer("pixels", pixels);
loader.receive_frames(seq);

This call to receive_frames will be asynchronous. receive_frames_sync can be used if synchronous reading is desired. When we are ready to use the frames we can insert a wait event into the CUDA stream we are using for our computation:

seq.wait(stream);

This will insert a wait event into the stream stream, causing any further kernels launched on stream to wait until the data is ready.

The C interface follows a very similar pattern, see doc/examples/extract_frames_c.c for an example.

Reference

If you find this library useful in your work, please cite it in your publications using the following BibTeX entry:

@misc{nvvl,
  author = {Jared Casper and Jon Barker and Bryan Catanzaro},
  title = {NVVL: NVIDIA Video Loader},
  year = {2018},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/NVIDIA/nvvl}}
}

Footnotes

[1] Specifically, with nvidia kernel modules version 384 and later, which come with CUDA 9.0+, CUDA kernels launched by NVVL will run asynchronously on a separate stream. With earlier kernel modules, all CUDA kernels are launched on the default stream. ↩

More Repositories

1

nvidia-docker

Build and run Docker containers leveraging NVIDIA GPUs
16,896
star
2

open-gpu-kernel-modules

NVIDIA Linux open GPU kernel module source
C
14,997
star
3

DeepLearningExamples

State-of-the-Art Deep Learning scripts organized by models - easy to train and deploy with reproducible accuracy and performance on enterprise-grade infrastructure.
Jupyter Notebook
13,339
star
4

NeMo

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)
Python
12,016
star
5

FastPhotoStyle

Style transfer, deep learning, feature transform
Python
11,020
star
6

TensorRT

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
C++
10,618
star
7

Megatron-LM

Ongoing research training transformer models at scale
Python
10,332
star
8

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
C++
8,542
star
9

vid2vid

Pytorch implementation of our method for high-resolution (e.g. 2048x1024) photorealistic video-to-video translation.
Python
8,482
star
10

apex

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch
Python
8,239
star
11

pix2pixHD

Synthesizing and manipulating 2048x1024 images with conditional GANs
Python
6,488
star
12

cuda-samples

Samples for CUDA Developers which demonstrates features in CUDA Toolkit
C
6,119
star
13

cutlass

CUDA Templates for Linear Algebra Subroutines
C++
5,519
star
14

FasterTransformer

Transformer related optimization, including BERT, GPT
C++
5,313
star
15

DALI

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
C++
5,048
star
16

thrust

[ARCHIVED] The C++ parallel algorithms library. See https://github.com/NVIDIA/cccl
C++
4,914
star
17

tacotron2

Tacotron 2 - PyTorch implementation with faster-than-realtime inference
Jupyter Notebook
4,562
star
18

warp

A Python framework for high performance GPU simulation and graphics
Python
4,206
star
19

DIGITS

Deep Learning GPU Training System
HTML
4,105
star
20

NeMo-Guardrails

NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.
Python
4,064
star
21

nccl

Optimized primitives for collective multi-GPU communication
C++
3,187
star
22

flownet2-pytorch

Pytorch implementation of FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Python
2,938
star
23

ChatRTX

A developer reference project for creating Retrieval Augmented Generation (RAG) chatbots on Windows using TensorRT-LLM
TypeScript
2,635
star
24

k8s-device-plugin

NVIDIA device plugin for Kubernetes
Go
2,481
star
25

libcudacxx

[ARCHIVED] The C++ Standard Library for your entire system. See https://github.com/NVIDIA/cccl
C++
2,294
star
26

GenerativeAIExamples

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
Python
2,192
star
27

nvidia-container-toolkit

Build and run containers leveraging NVIDIA GPUs
Go
2,171
star
28

waveglow

A Flow-based Generative Network for Speech Synthesis
Python
2,133
star
29

MinkowskiEngine

Minkowski Engine is an auto-diff neural network library for high-dimensional sparse tensors
Python
2,007
star
30

TransformerEngine

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilization in both training and inference.
Python
1,917
star
31

Stable-Diffusion-WebUI-TensorRT

TensorRT Extension for Stable Diffusion Web UI
Python
1,886
star
32

semantic-segmentation

Nvidia Semantic Segmentation monorepo
Python
1,763
star
33

gpu-operator

NVIDIA GPU Operator creates/configures/manages GPUs atop Kubernetes
Go
1,735
star
34

cub

[ARCHIVED] Cooperative primitives for CUDA C++. See https://github.com/NVIDIA/cccl
Cuda
1,679
star
35

DeepRecommender

Deep learning for recommender systems
Python
1,662
star
36

stdexec

`std::execution`, the proposed C++ framework for asynchronous and parallel programming.
C++
1,554
star
37

OpenSeq2Seq

Toolkit for efficient experimentation with Speech Recognition, Text2Speech and NLP
Python
1,511
star
38

CUDALibrarySamples

CUDA Library Samples
Cuda
1,468
star
39

VideoProcessingFramework

Set of Python bindings to C++ libraries which provides full HW acceleration for video decoding, encoding and GPU-accelerated color space and pixel format conversions
C++
1,303
star
40

deepops

Tools for building GPU clusters
Shell
1,252
star
41

open-gpu-doc

Documentation of NVIDIA chip/hardware interfaces
C
1,243
star
42

aistore

AIStore: scalable storage for AI applications
Go
1,233
star
43

Q2RTX

NVIDIA’s implementation of RTX ray-tracing in Quake II
C
1,217
star
44

trt-samples-for-hackathon-cn

Simple samples for TensorRT programming
Python
1,211
star
45

cccl

CUDA Core Compute Libraries
C++
1,200
star
46

MatX

An efficient C++17 GPU numerical computing library with Python-like syntax
C++
1,187
star
47

partialconv

A New Padding Scheme: Partial Convolution based Padding
Python
1,145
star
48

sentiment-discovery

Unsupervised Language Modeling at scale for robust sentiment classification
Python
1,055
star
49

nvidia-container-runtime

NVIDIA container runtime
Makefile
1,035
star
50

modulus

Open-source deep-learning framework for building, training, and fine-tuning deep learning models using state-of-the-art Physics-ML methods
Python
991
star
51

gpu-monitoring-tools

Tools for monitoring NVIDIA GPUs on Linux
C
974
star
52

jetson-gpio

A Python library that enables the use of Jetson's GPIOs
Python
898
star
53

dcgm-exporter

NVIDIA GPU metrics exporter for Prometheus leveraging DCGM
Go
886
star
54

retinanet-examples

Fast and accurate object detection with end-to-end GPU optimization
Python
885
star
55

flowtron

Flowtron is an auto-regressive flow-based generative network for text to speech synthesis with control over speech variation and style transfer
Jupyter Notebook
867
star
56

nccl-tests

NCCL Tests
Cuda
864
star
57

cuda-python

CUDA Python Low-level Bindings
Python
859
star
58

mellotron

Mellotron: a multispeaker voice synthesis model based on Tacotron 2 GST that can make a voice emote and sing without emotive or singing training data
Jupyter Notebook
852
star
59

gdrcopy

A fast GPU memory copy library based on NVIDIA GPUDirect RDMA technology
C++
832
star
60

libnvidia-container

NVIDIA container runtime library
C
818
star
61

BigVGAN

Official PyTorch implementation of BigVGAN (ICLR 2023)
Python
806
star
62

spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
Scala
800
star
63

nv-wavenet

Reference implementation of real-time autoregressive wavenet inference
Cuda
728
star
64

DLSS

NVIDIA DLSS is a new and improved deep learning neural network that boosts frame rates and generates beautiful, sharp images for your games
C
727
star
65

tensorflow

An Open Source Machine Learning Framework for Everyone
C++
719
star
66

gvdb-voxels

Sparse volume compute and rendering on NVIDIA GPUs
C
674
star
67

MAXINE-AR-SDK

NVIDIA AR SDK - API headers and sample applications
C
671
star
68

runx

Deep Learning Experiment Management
Python
633
star
69

NVFlare

NVIDIA Federated Learning Application Runtime Environment
Python
630
star
70

NeMo-Aligner

Scalable toolkit for efficient model alignment
Python
564
star
71

nvcomp

Repository for nvCOMP docs and examples. nvCOMP is a library for fast lossless compression/decompression on the GPU that can be downloaded from https://developer.nvidia.com/nvcomp.
C++
545
star
72

multi-gpu-programming-models

Examples demonstrating available options to program multiple GPUs in a single node or a cluster
Cuda
535
star
73

Dataset_Synthesizer

NVIDIA Deep learning Dataset Synthesizer (NDDS)
C++
530
star
74

TensorRT-Model-Optimizer

TensorRT Model Optimizer is a unified library of state-of-the-art model optimization techniques such as quantization, pruning, distillation, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM or TensorRT to optimize inference speed on NVIDIA GPUs.
Python
513
star
75

jitify

A single-header C++ library for simplifying the use of CUDA Runtime Compilation (NVRTC).
C++
512
star
76

nvbench

CUDA Kernel Benchmarking Library
Cuda
501
star
77

libglvnd

The GL Vendor-Neutral Dispatch library
C
501
star
78

NeMo-Curator

Scalable data pre processing and curation toolkit for LLMs
Jupyter Notebook
500
star
79

cuda-quantum

C++ and Python support for the CUDA Quantum programming model for heterogeneous quantum-classical workflows
C++
496
star
80

AMGX

Distributed multigrid linear solver library on GPU
Cuda
474
star
81

cuCollections

C++
470
star
82

enroot

A simple yet powerful tool to turn traditional container/OS images into unprivileged sandboxes.
Shell
459
star
83

NeMo-Framework-Launcher

Provides end-to-end model development pipelines for LLMs and Multimodal models that can be launched on-prem or cloud-native.
Python
459
star
84

hpc-container-maker

HPC Container Maker
Python
442
star
85

MDL-SDK

NVIDIA Material Definition Language SDK
C++
438
star
86

PyProf

A GPU performance profiling tool for PyTorch models
Python
437
star
87

framework-reproducibility

Providing reproducibility in deep learning frameworks
Python
424
star
88

gpu-rest-engine

A REST API for Caffe using Docker and Go
C++
421
star
89

DCGM

NVIDIA Data Center GPU Manager (DCGM) is a project for gathering telemetry and measuring the health of NVIDIA GPUs
C++
394
star
90

NvPipe

NVIDIA-accelerated zero latency video compression library for interactive remoting applications
Cuda
390
star
91

torch-harmonics

Differentiable signal processing on the sphere for PyTorch
Jupyter Notebook
386
star
92

cuQuantum

Home for cuQuantum Python & NVIDIA cuQuantum SDK C++ samples
Jupyter Notebook
344
star
93

data-science-stack

NVIDIA Data Science stack tools
Shell
317
star
94

ai-assisted-annotation-client

Client side integration example source code and libraries for AI-Assisted Annotation SDK
C++
308
star
95

video-sdk-samples

Samples demonstrating how to use various APIs of NVIDIA Video Codec SDK
C++
301
star
96

egl-wayland

The EGLStream-based Wayland external platform
C
299
star
97

nvidia-settings

NVIDIA driver control panel
C
292
star
98

NVTX

The NVIDIA® Tools Extension SDK (NVTX) is a C-based Application Programming Interface (API) for annotating events, code ranges, and resources in your applications.
C
290
star
99

go-nvml

Go Bindings for the NVIDIA Management Library (NVML)
C
288
star
100

gpu-feature-discovery

GPU plugin to the node feature discovery for Kubernetes
Go
286
star