LightSeq: A High Performance Library for Sequence Processing and Generation
Table Of Contents
Release Notes
[2022.10.25] Release v3.0.0 version, which supports int8 mixed-precision training and inference. [中文介绍]
[2021.06.18] Release v2.0.0 version, which supports fp16 mixed-precision training. [中文介绍]
[2019.12.06] Release v1.0.0 version, which supports fp16 mixed-precision inference. [中文介绍]
Introduction
LightSeq is a high performance training and inference library for sequence processing and generation implemented in CUDA. It enables highly efficient computation of modern NLP and CV models such as BERT, GPT, Transformer, etc. It is therefore best useful for machine translation, text generation, image classification, and other sequence related tasks.
The library is built on top of CUDA official library(cuBLAS, Thrust, CUB) and custom kernel functions which are specially fused and optimized for Transformer model family. In addition to model components, the inference library also provide easy-to-deploy model management and serving backend based on TensorRT Inference Server. With LightSeq, one can easily develop modified Transformer architecture with little additional code.
LightSeq training and inference is very fast. Below is the overall performance:
- LightSeq fp16 training achieves a speedup of up to 3x, compared to PyTorch fp16 training.
- LightSeq int8 training achieves a speedup of up to 5x, compared to PyTorch QAT (i.e., quantization aware training).
- LightSeq fp16 and int8 inference achieve a speedup of up to 12x and 15x, compared to PyTorch fp16 inference, respectively.
Support Matrix
LightSeq supports multiple features, which is shown in the table below.
Features | Support List |
---|---|
Model | Transformer, BERT, BART, GPT2, ViT, T5, MT5, XGLM, VAE, Multilingual, MoE |
Layer | embedding, encoder, decoder, criterion, optimizer |
Precision | fp32, fp16, int8 |
Mode | training, inference |
Compatibility | Fairseq, Hugging Face, DeepSpeed |
Decoding Algorithm | beam search, diverse beam search, sampling, CRF |
Others | gradient communication quantization, auto-tune GEMM algorithm |
The table below shows the running modes and precision currently supported by different models.
Models | fp16 Training | fp16 Inference | int8 Training | int8 Inference |
---|---|---|---|---|
Transformer | Yes | Yes | Yes | Yes |
BERT | Yes | Yes | Yes | Yes |
GPT2 | Yes | Yes | Yes | Yes |
BART | Yes | Yes | - | - |
T5 | - | Yes | - | - |
MT5 | - | Yes | - | - |
XGLM | - | Yes | - | - |
ViT | Yes | Yes | Yes | Yes |
VAE | - | Yes | - | - |
Multilingual | - | Yes | - | Yes |
MoE | - | Yes | - | - |
Performance
We test the speedup of LightSeq training and inference using both fp16 and int8 mix-precision on Transformer and BERT models. The baseline is PyTorch fp16 mix-precision. Training experiments are tested on one A100 GPU and inference experiments are tested on eight A100 GPUs.
More performance results are available here.
Speedup of Transformer Training
Batch Token Size | PyTorch QAT | LightSeq fp16 | LightSeq int8 |
---|---|---|---|
512 | 0.36 | 1.99 | 1.86 |
1024 | 0.37 | 1.78 | 1.69 |
2048 | 0.37 | 1.56 | 1.50 |
4096 | 0.39 | 1.47 | 1.44 |
8192 | 0.41 | 1.44 | 1.44 |
15000 | 0.43 | 1.44 | 1.44 |
Speedup of BERT Training
Batch Token Size | PyTorch QAT | LightSeq fp16 | LightSeq int8 |
---|---|---|---|
8 | 0.45 | 2.12 | 1.99 |
16 | 0.44 | 1.92 | 1.80 |
32 | 0.42 | 1.59 | 1.52 |
64 | 0.46 | 1.62 | 1.58 |
128 | 0.46 | 1.74 | 1.70 |
256 | 0.46 | 1.68 | 1.73 |
Speedup of Transformer Inference
Batch Size | Sequence Length | LightSeq fp16 | LightSeq int8 |
---|---|---|---|
1 | 8 | 8.00 | 9.33 |
1 | 32 | 6.48 | 7.38 |
1 | 128 | 6.24 | 6.19 |
8 | 8 | 9.38 | 10.71 |
8 | 32 | 8.24 | 8.75 |
8 | 128 | 6.83 | 7.28 |
32 | 8 | 11.82 | 14.44 |
32 | 32 | 9.68 | 11.15 |
32 | 128 | 6.68 | 7.74 |
Speedup of BERT Inference
Batch Size | Sequence Length | LightSeq fp16 | LightSeq int8 |
---|---|---|---|
1 | 8 | 9.22 | 9.87 |
1 | 32 | 10.51 | 11.30 |
1 | 128 | 9.96 | 10.85 |
8 | 8 | 9.88 | 10.33 |
8 | 32 | 7.79 | 8.22 |
8 | 128 | 4.04 | 4.35 |
32 | 8 | 10.60 | 11.02 |
32 | 32 | 8.11 | 8.85 |
32 | 128 | 1.82 | 2.04 |
Installation
Install from PyPI
You can install LightSeq from PyPI, which only supports Python 3.6 to 3.8 on Linux:
pip install lightseq
Build from Source
You can also build from source:
PATH=/usr/local/hdf5/:$PATH ENABLE_FP32=0 ENABLE_DEBUG=0 pip install -e $PROJECT_DIR
Detailed building introduction is available here.
Getting Started
We provide several samples here to show the usage of LightSeq. Refer to the complete user guide and examples for more details.
LightSeq Training from Scratch
You can use the modules provided by LightSeq to build your own models. The following is an example of building a Transformer encoder layer.
First, import LightSeq Transformer encoder module:
from lightseq.training import LSTransformerEncoderLayer
Then create an encoder configuration, and create a LightSeq Transformer encoder layer initialized with the configuration:
config = LSTransformerEncoderLayer.get_config(
max_batch_tokens=4096,
max_seq_len=512,
hidden_size=1024,
intermediate_size=4096,
nhead=16,
attn_prob_dropout_ratio=0.1,
activation_dropout_ratio=0.1,
hidden_dropout_ratio=0.1,
pre_layer_norm=True,
activation_fn="relu",
fp16=True,
local_rank=0,
)
layer = LSTransformerEncoderLayer(config)
In addition to encoder layers, the other modules can be created using similar methods, and then be trained as normal PyTorch models.
More usage is available here.
LightSeq Training from Fairseq
LightSeq integrates all the fast and lightning modules into Fairseq.
First install the two following requirements:
pip install fairseq==0.10.2 sacremoses
You can train a fp16 mix-precision translation task on wmt14 en2de dataset by:
sh examples/training/fairseq/ls_fairseq_wmt14en2de.sh
(Optional) Then you can start int8 mix-precision training on the basis of fp16 pre-training models by:
sh examples/training/fairseq/ls_fairseq_quant_wmt14en2de.sh
More usage is available here.
LightSeq Training from Hugging Face BERT
LightSeq replaces the encoder layers of Hugging Face BERT with LightSeq fast layers.
First you should install these requirements:
pip install transformers seqeval datasets
Before doing next training, you need to switch to the following directory:
cd examples/training/huggingface/bert
Then you can easily fine-tune BERT for different tasks. Taking named entity recognition task as an example, you can train the BERT with fp16 mixed-precision using:
python task_ner/run_ner.sh
(Optional) You can also start int8 mix-precision training on the basis of fp16 pre-training models by:
python task_ner/run_quant_ner.sh
More usage is available here.
LightSeq Inference from Fairseq
After training using the above scripts, you can quickly infer the models using LightSeq.
You should transform the fp16 PyTorch weights to LightSeq protobuf or HDF5:
python export/fairseq/ls_fs_transformer_export.py
(Optional) You can also transform the int8 PyTorch weights to LightSeq protobuf or HDF5:
python export/fairseq/ls_fs_quant_transformer_export.py
Once obtaining the LightSeq weights, you can quickly infer them using the following code:
import lightseq.inference as lsi
model = lsi.Transformer(MODEL_PATH, MAX_BATCH_SIZE)
results = model.infer([[63, 47, 65, 1507, 88, 74, 10, 2057, 362, 9, 284, 6, 2, 1]])
Here MODEL_PATH is the path of your LightSeq weights and MAX_BATCH_SIZE is the maximal batch size of your input sentences.
You can also quickly infer the int8 LightSeq weights by replacing the lsi.Transformer
with lsi.QuantTransformer
.
More usage is available here.
LightSeq Inference from Hugging Face BERT
We provide an end2end bert-base example to see how fast Lightseq is compared to original Hugging Face.
First you should install the requirements and locate to the specified directory:
pip install transformers
cd examples/inference/python
Then you can check the performance by simply running the following commands. hf_bert_export.py
is used to transform PyTorch weights to LightSeq protobuf or HDF5.
python export/huggingface/hf_bert_export.py
python test/ls_bert.py
More usage is available here.
LightSeq Deployment Using Inference Server
We provide a docker image which contains tritonserver and LightSeq's dynamic link library, and you can deploy an inference server by simply replacing the model file with your own model file.
sudo docker pull hexisyztem/tritonserver_lightseq:22.01-1
More usage is available here.
Cite Us
If you use LightSeq in your research, please cite the following papers.
@InProceedings{wang2021lightseq,
title = "{L}ight{S}eq: A High Performance Inference Library for Transformers",
author = "Wang, Xiaohui and Xiong, Ying and Wei, Yang and Wang, Mingxuan and Li, Lei",
booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers (NAACL-HLT)",
month = jun,
year = "2021",
publisher = "Association for Computational Linguistics",
pages = "113--120",
}
@article{wang2021lightseq2,
title={LightSeq2: Accelerated Training for Transformer-based Models on GPUs},
author={Wang, Xiaohui and Xiong, Ying and Qian, Xian and Wei, Yang and Li, Lei and Wang, Mingxuan},
journal={arXiv preprint arXiv:2110.05722},
year={2021}
}
We are Hiring!
The LightSeq team is hiring Interns and FTEs with backgrounds in deep learning system, natural language processing, computer vision, speech, etc. We are based in Beijing and Shanghai. If you are interested, please send your resume to [email protected].